Export and Arm Encoder API

Export writes each code's coordinates in Req 2's form of its geometry arm: bounded tangents in the hyperbolic arm. The arm encoder reads queries through the checkpoint and codes from its matching table, both through the head's read map. Checkpoint objective req11-v1, bundle and preprocessing identities are checked before model loading or panel reads. Unit text curvature is part of the objective, with no curvature setting or legacy migration.

Export

Encoding an arm's codes and queries, and exporting its code table (spec 4.3).

encode_token_rows runs token rows through an arm's model: a code's cached channels, or a marked query (encode_query_texts). The HGCN feeder, the table export, the arm encoder and the training monitor all encode through it, so a code or a query embeds the same way wherever it is read. The last three default to batches of ENCODE_BATCH_SIZE.

export_code_table writes Req 2's form of an arm: code, index, level and e0 … e{d-1}, in the bundle's codebook order. They are each code's coordinates in its geometry arm's form (COORDINATES): the bounded tangent vector at the origin in the hyperbolic arm (R6, Req 13), v in the Euclidean arm and û in the spherical arm (Req 12). Its provenance ties the table to its checkpoint and names its geometry.

code_token_config(cfg)

The tokenization cache training reads: train hands it to NAICSDataModule.

The descriptions and the window are the streaming ones, the tokenizer is the tokenization one, and the path is the default. Export and reads therefore load the cache file that training built.

encode_query_texts(model, tokenizer, texts, max_length, batch_size=ENCODE_BATCH_SIZE)

Encode query texts through the model, each marked query: and tokenized at max_length.

The arm encoder and the training monitor both encode their queries here, so a read of the live model and a read of its exported arm put each query at the same point.

Parameters:

Name Type Description Default
model Module

As encode_token_rows.

 required
tokenizer Any

The token cache's tokenizer.

 required
texts Sequence[str]

The query texts, in output order.

 required
max_length int

The token cache's window.

 required
batch_size int

Queries per forward pass.

 ENCODE_BATCH_SIZE

Returns:

Type Description
Tensor

The queries' coordinates in the arm's export form (the head's tangent), (Q, d),
Tensor

float64 on the CPU.

Raises:

Type Description
ValueError

As encode_token_rows, if there are no texts.

encode_token_rows(model, rows, *, fields=CHANNELS, batch_size=ENCODE_BATCH_SIZE)

Encode token rows through the model in batches, in eval mode and without gradient.

A row maps each field to its tokens (input_ids, attention_mask and present), as the tokenization cache stores a code or tokenize_field returns a query. The batches go to the model's device, and the model is left in eval mode.

Parameters:

Name Type Description Default
model Module

A model whose forward returns tangent, embedding, radius and direction: the shared encoder, or the Lightning module that holds it.

 required
rows Sequence[Mapping[str, Mapping[str, Any]]]

The token rows, in output order.

 required
fields Sequence[str]

The fields read from each row.

 CHANNELS
batch_size int

Rows per forward pass.

 ENCODE_BATCH_SIZE

Returns:

Type Description
Dict[str, Tensor]

tangent (N, d), embedding (N, d + 1) in the hyperbolic arm and (N, d) in the flat
Dict[str, Tensor]

arms, radius (N,) and direction (N, d), float64 on the CPU, in row order.

Raises:

Type Description
ValueError

If there are no rows, or batch_size is not positive.

export_code_table(checkpoint_path, bundle, token_config, output_path, *, device='cpu', batch_size=ENCODE_BATCH_SIZE)

Export an arm's code table in Req 2's form, with its provenance beside it (spec 4.3).

Every code goes through the checkpoint's model in eval mode, without gradient. The table holds code, then index and level from the descriptions (Int64), then e0 … e{d-1} (float64): each code's coordinates in its arm's form (COORDINATES), in the bundle's codebook order. The provenance is <stem>_provenance.json; it names the arm's geometry and coordinates.

Parameters:

Name Type Description Default
checkpoint_path Union[str, Path]

The arm's Lightning checkpoint.

 required
bundle ValidatedSupervisionBundle

The configured supervision bundle.

 required
token_config TokenizationConfig

The token cache training read (code_token_config). Its descriptions_parquet is the arm's descriptions.

 required
output_path Union[str, Path]

The table's parquet path.

 required
device Union[str, device]

Where the model runs.

 'cpu'
batch_size int

Codes per forward pass.

 ENCODE_BATCH_SIZE

Returns:

Type Description
Path

The table's path.

Raises:

Type Description
ValueError

As load_arm_model; if the descriptions' (index, code) rows are not the codebook's (code_id, code) rows; or if coordinate_matrix refuses the table, which is then not written.

load_arm_model(checkpoint_path, bundle, *, summaries, device='cpu')

Load an arm's checkpoint for export or a read, refusing it before any weight loads.

The checkpoint's own hyperparameters rebuild its fusion, dimension and geometry, so its encoder record is never compared with a config (spec 4.4). It must have been trained under Req 11's objective, its supervision fields must match bundle, and its summaries summaries. Curvature is fixed at 1 with no parameter (spec 4.2), so there is none to check.

Parameters:

Name Type Description Default
checkpoint_path Union[str, Path]

The arm's Lightning checkpoint.

 required
bundle ValidatedSupervisionBundle

The configured supervision bundle.

 required
summaries Optional[str]

The sha256 of the summaries the read's token cache applies (summaries_identity of its tokenizer); keyword-only with no default.

 required
device Union[str, device]

Where the model runs.

 'cpu'

Returns:

Type Description
Tuple[NAICSContrastiveModel, CheckpointContract]

The model, in eval mode on device, and the checkpoint's saved contract.

Raises:

Type Description
ValueError

If the checkpoint was trained under another objective, such as every one saved before Stage 7 (spec 4.5; checked first), its supervision contract is not the bundle's, it was trained under other summaries, or it is of another encoder architecture. Nothing migrates such a checkpoint (D2).

Arm Encoder

An arm as the outcome panel reads it: queries through its checkpoint, codes from its table (spec 4.3).

ArmEncoder implements QueryCodeEncoder (panels/outcome.py):

  • A query is marked query: and goes through the checkpoint's model (encode_query_texts, the query path the training monitor's live encoder shares).
  • A code's vector is decoded from the table tools export-table wrote from the checkpoint.

Both pass through the head's float64 read map (read_points): the exp map at the origin in the hyperbolic arm, and the coordinates as they are in the flat arms (Req 12). The code points a read decodes against are therefore a fixed function of the table, and the matrix_fingerprint the read logs names them. Checkpoint, table, encoder and distance are the pieces of Stage 4's SeedArtifacts (decision/sweep.py).

ArmEncoder

QueryCodeEncoder for one arm: its checkpoint's model and the table exported from it.

Parameters:

Name Type Description Default
model Module

The arm's model, in eval mode (load_arm_model).

 required
tokenizer Any

The token cache's tokenizer.

 required
max_length int

The token cache's window.

 required
table DataFrame

The arm's exported table.

 required
checkpoint_sha256 str

The checkpoint file's SHA-256.

 required
batch_size int

Queries per forward pass.

 ENCODE_BATCH_SIZE

Attributes:

Name Type Description
head

The model's geometry head: its read map takes the queries' and the codes' coordinates to the points its distance reads.
distance

The head's decoding distance: 'lorentz', 'euclidean' or 'cosine' for the hyperbolic, Euclidean and spherical arms (Req 12).
table_fingerprint

The table's matrix_fingerprint, which a read logs as table.
checkpoint_sha256

The checkpoint's SHA-256, which a read logs as checkpoint.

encode_codes(codes)

The codes' table rows through the head's read map: float64, (C, d + 1) in the hyperbolic arm and (C, d) in the flat arms.

Raises:

Type Description
ValueError

If a code has no row in the table.

encode_queries(texts)

Marked query: texts through the model, then the head's read map: float64, (Q, d + 1) in the hyperbolic arm and (Q, d) in the flat arms.

from_files(checkpoint_path, table_path, bundle, token_config, *, device='cpu', batch_size=ENCODE_BATCH_SIZE) classmethod

The arm of a checkpoint and the table exported from it.

The table's provenance is checked before the model loads. It must name this checkpoint and this table file, and the window, tokenizer and summaries it records, which the table's codes were read under, must be the ones the queries will be read under: the arm has one preprocessing contract.

Parameters:

Name Type Description Default
checkpoint_path Union[str, Path]

The arm's Lightning checkpoint.

 required
table_path Union[str, Path]

The table tools export-table wrote from it.

 required
bundle ValidatedSupervisionBundle

The configured supervision bundle.

 required
token_config TokenizationConfig

The token cache training read (code_token_config): its tokenizer and window tokenize the queries. Its tokenizer, window and summaries must be the ones the table was exported under.

 required
device Union[str, device]

Where the model runs.

 'cpu'
batch_size int

Queries per forward pass.

 ENCODE_BATCH_SIZE

Raises:

Type Description
ValueError

If the table's provenance is no exported arm table's (it names no checkpoint, table hash or window), predates Stage 6b (it records no summaries or tokenizer), names another checkpoint, or the table file is not the one it names; if the table was exported at another window, with another tokenizer or under other summaries than token_config's; or as load_arm_model: another objective, as every checkpoint saved before Stage 7 has (spec 4.5), another supervision contract or summaries, or another encoder architecture (D2).
FileNotFoundError

If the checkpoint, the table or its provenance is missing.

read_outcome_validation(encoder, panel, purpose)

Score the arm on the outcome panel's validation split under its own distance.

The read is logged. Its detail names the table the codes were decoded from (table, the key Stage 4's sweep logs) and the checkpoint the queries went through (checkpoint). There is no test-split path here: Stage 12 opens that split.

Parameters:

Name Type Description Default
encoder ArmEncoder

The arm.

 required
panel OutcomePanel

The outcome panel of the arm's bundle.

 required
purpose str

Why the read happens; the selection log records it.

 required

Returns:

Type Description
DecodingResult

The decoding scores.