Download Data API

NaicsSources dataclass

The four Census NAICS files as read, combined sector codes normalized.

build_descriptions(sources, examples_entries, redirections=None)

One row per code: title, description and its source, examples channel and exclusion channel.

Parameters:

Name Type Description Default
sources NaicsSources

The four Census files.

 required
examples_entries DataFrame

The index entries that form examples channels (entry_id, code, text); every other entry of a code with index entries is a query.

 required
redirections Optional[DataFrame]

The redirection table whose rows form the exclusion channel (redirections.exclusion_channel); built from sources with no row withheld when omitted.

 None

download_preprocess_data(cfg=None, *, force=False)

Build the descriptions, index-roles and redirection parquets from the Census files.

Every index entry takes its role from the frozen role table (cfg.index_roles_csv). A code's examples channel holds its examples-role entries only. A redirection row that a validation or test query leaks into is withheld from the exclusion channel, and no held-out query may match any training text or activity phrase (Req 3). Everything is checked before anything is written.

Parameters:

Name Type Description Default
cfg Optional[DownloadConfig]

Download configuration; conf/data/download.yaml when omitted.

 None
force bool

Overwrite a descriptions file that a configured supervision bundle pins.

 False

load_naics_sources(cfg)

Read the four files, from cfg.source_dir when set; the index file must match its pin.

naics_index_entries(sources)

The index file's entries for six-digit codes (entry_id, code, text).

naics_redirections(sources, held_out_queries=())

The redirection table of the Census files (redirections.build_redirections).

Rows are the cross-reference file's rows in file order, then the "Excluded" paragraphs harvested from descriptions in code order. A row that one of held_out_queries leaks into is withheld.

pinned_description_fingerprints(pinning_configs=PINNING_CONFIGS)

The description_fingerprint of each bundle a config names, by manifest path.

Each config names the manifest a run would read: the gitignored conf/local.yaml's value where it sets one (read_local_config), otherwise the tracked file's. Configs, keys and manifests that do not exist are skipped: they pin nothing.

refuse_pinned_overwrite(output, *, force, pinning_configs=PINNING_CONFIGS)

Refuse to overwrite the descriptions file a configured supervision bundle pins.

Training fails closed once the descriptions file no longer matches its bundle's description_fingerprint, so rewriting it would break every run against that bundle.

Raises:

Type Description
FileExistsError

If output is pinned and force is False.

verify_text_channels(descriptions)

Fail unless every channel holds real text or is null (Req 9, "Masking").

No channel may be an empty or blank string, since an absent channel is null, and none may contain the placeholder the tokenizer once substituted. A description's description_source must be set exactly when the description is.

Returns:

Type Description
Dict[str, int]

The number of present texts per channel.