Redirections API

The redirection table (Req 8, roadmap Stage 5): every Census cross-reference and harvested "Excluded" paragraph once, with the codes it names. Each code's exclusion channel is built from it.

The redirection table (Req 8): every Census cross-reference once, with where it sends an activity.

A cross-reference reroutes an activity rather than asserting that two codes are unrelated: "Growing soybeans--are classified in Industry 111110" sends soybean growing to 111110. The table holds one row per cross-reference row, in file order, then one per "Excluded" paragraph harvested from a description, in code order. Its columns:

  • reference_id: the row's position in the table, so cross-reference rows keep their file positions.
  • source: cross_reference or description.
  • code and text: the referencing code and the row's text.
  • activity: the text before -- or before " are/is classified" (or "included"), on a cross-reference row that names a code (supervision.activity.activity_phrase, which this module re-exports). Stage 7 trains on these phrases as queries, and the bundle loader recomputes each one.
  • named_codes: the codebook codes the text names other than its own code, in order of first appearance. On a cross-reference row these are its destinations.
  • lineal_codes: the named codes that are the row's code's ancestors or descendants. Lineal references stay text only and never act as negatives.
  • withheld: a held-out query leaks into one of the text's segments or into its activity phrase (Req 3), so the text leaves the exclusion channel and the activity phrase is dropped. The row stays in the table, and its named codes still count as exclusions.

The exclusion channel is built from the table: each code's texts that are not withheld, once each, joined in table order.

build_redirections(references, paragraphs, codes, held_out_queries=(), *, min_jaccard=NEAR_DUPLICATE_MIN_JACCARD)

The redirection table: references rows in order, then paragraphs rows in order.

Parameters:

Name Type Description Default
references DataFrame

The cross-reference file's rows (code, text), in file order.

 required
paragraphs DataFrame

The "Excluded" paragraphs harvested from descriptions (code, text), in code order.

 required
codes Set[str]

The codebook's codes.

 required
held_out_queries Sequence[str]

Validation and test queries; a row any of them leaks into is withheld.

 ()
min_jaccard Fraction

The near-duplicate threshold of the leakage check.

 NEAR_DUPLICATE_MIN_JACCARD

Returns:

Type Description
DataFrame

One row per input row, with the columns of REDIRECTIONS_SCHEMA.

exclusion_channel(redirections)

Each code's exclusion channel from the redirection table (Req 8(a)).

Returns:

Type Description
DataFrame

One row per code with a redirection row, sorted by code: excluded, the texts of its
DataFrame

rows that are not withheld, each once, joined by one space in table order (null when all
DataFrame

are withheld), and excluded_codes, the codes its rows name, withheld rows included,
DataFrame

each once in order of first appearance (null when none).

lineal_codes(code, named)

The named codes that are code's ancestors or descendants.

named_codes(code, text, codes)

The codebook codes text names other than code, in order of first appearance.