Window-Fitting Summaries API

Extractive summaries of the channel texts over the backbone's trained window (Req 9, roadmap Stage 6b): the units and the resolver every reader goes through, and the build behind naics-embedder data summaries.

Units, pin and resolver

Window-fitting summaries of over-long channel texts (Req 9, "Input windows"; roadmap Stage 6b).

A channel text whose marked form ('<field>: <text>', special tokens included) is over the backbone's trained window is read as its summary: whole units of the text, in source order, chosen once by naics-embedder data summaries (data/window_summaries.py) and committed. Every unit boundary is a boundary of the leakage segmenter (panels/leakage.py), so a summary's segments are a subset of its text's, and leakage needs no sealed read (spec 4.4).

WINDOW_SUMMARIES pins each backbone's artifact by sha256, and summaries_identity is the sha256 every identity site records: the token cache's sidecar, the checkpoint contract, the export and text-only provenances, and the decision store. resolve_channel_texts is the only place summaries enter. The token cache and the text-only builder call it, and it checks the artifact's invariants against the descriptions on every call that finds an over-window text. The module loads no model and imports no torch.

SummariesPin dataclass

A committed summaries artifact.

Attributes:

Name Type Description
path str

The artifact, relative to the repository root.
sha256 str

The artifact's sha256, which every identity site records.
window int

The trained window the summaries fit.

over_window(descriptions, count_marked, window)

The present channel texts whose marked form is over the window, sorted by channel, then code.

Parameters:

Name Type Description Default
descriptions DataFrame

Codes and their four channel texts; a null or blank text is absent.

 required
count_marked TokenCounter

Token counts, special tokens included.

 required
window int

The window.

 required

Returns:

Type Description
List[Tuple[str, str]]

(code, channel) pairs.

read_window_summaries(path)

Read a summaries artifact with SUMMARIES_SCHEMA, so that code stays a string.

Raises:

Type Description
ValueError

If a row names a channel other than the three, or a (code, channel) repeats.

resolve_channel_texts(descriptions, tokenizer, backbone, max_length, *, pin=DEFAULT)

The descriptions, each over-window channel text replaced by its pinned summary (spec 4.7).

Steps, raising at the first failure: (1) find the texts over the window, and return the descriptions unchanged when there are none; (2) require a pin for the window; (3) read the artifact and require the pin's sha256; (4) require one row per over-window text and no other; (5) require each row's source sha256 and window; (6) require each summary to be an extract of its source; (7) require every marked channel text to fit after substitution.

Parameters:

Name Type Description Default
descriptions DataFrame

Codes and their four channel texts.

 required
tokenizer Any

The backbone's tokenizer, which counts tokens as the backbone reads them.

 required
backbone str

The backbone, whose pin is the default.

 required
max_length int

The window texts are tokenized at.

 required
pin Union[SummariesPin, None, _PinDefault]

The summaries to read: DEFAULT looks up WINDOW_SUMMARIES[backbone] at call time, and None means no pin.

 DEFAULT

Returns:

Type Description
DataFrame

The descriptions with over-window texts replaced.

Raises:

Type Description
ValueError

At the first step that fails, naming the code and channel where there is one.

summaries_identity(backbone)

The sha256 of the backbone's pinned summaries, or None when it has no pin.

Parameters:

Name Type Description Default
backbone str

A Hugging Face model or tokenizer name.

 required

Returns:

Type Description
Optional[str]

The pin's sha256, or None.

summary_budget(count_marked, channel, window)

The tokens a channel's text may take: the window less its marker and special tokens.

Parameters:

Name Type Description Default
count_marked TokenCounter

Token counts, special tokens included.

 required
channel str

A channel name.

 required
window int

The trained window.

 required

Returns:

Type Description
int

The budget; 124 for every channel under MiniLM at 128.

text_sha256(text)

The sha256 of a text's UTF-8 bytes.

text_units(channel, text, count, budget)

A channel text's units, in source order, each a verbatim span of text (spec 4.2).

Every unit boundary is a boundary of the leakage segmenter. A title is one unit; the examples channel's units are its entries. A description or exclusion text's units are its sentences (level 1); a sentence over the budget is re-split at its clauses (level 2), and a clause over the budget at the segmenter's pieces (level 3).

Parameters:

Name Type Description Default
channel str

title, description, examples or excluded.

 required
text str

The channel text.

 required
count TokenCounter

Token counts without special tokens.

 required
budget int

The channel's budget (summary_budget).

 required

Returns:

Type Description
List[str]

The units.

Raises:

Type Description
ValueError

If a title, an examples entry or a level-3 piece is over the budget, or the channel is none of the four.

token_counter(tokenizer, *, special_tokens=True)

Token counts under tokenizer, without truncation.

The torch-free twin of utils.input_window.token_counter, which this module cannot import: naics_embedder.utils imports torch when the package loads.

Parameters:

Name Type Description Default
tokenizer Any

A Hugging Face tokenizer.

 required
special_tokens bool

Count [CLS] and [SEP], as the model reads a text.

 True

Returns:

Type Description
TokenCounter

A function from texts to their token counts.

write_window_summaries(rows, path)

Write summaries rows as the artifact: UTF-8 CSV, sorted by channel, then code.

Parameters:

Name Type Description Default
rows DataFrame

Rows with the columns of SUMMARIES_SCHEMA.

 required
path Path

The CSV to write.

 required

Returns:

Type Description
str

The sha256 of the bytes written.

The build

Build the window-fitting summaries (roadmap Stage 6b).

naics-embedder data summaries runs this once per backbone. Every channel text whose marked form is over the backbone's trained window is cut into units (panels.window_summaries), and the summary keeps the units whose pooled vectors best approximate the whole text's (centrality):

  1. Of the units whose marked summary fits the window on its own, keep the one whose weighted vector has the highest cosine with the target, the weighted sum of all the text's units.
  2. Repeatedly add the unit that most raises that cosine, among the units that keep the marked summary, joined in source order, within the window.
  3. Stop when no remaining unit fits, or none raises the cosine strictly.

The frozen backbone reads each unit unmarked and alone, on the CPU in float32; its last hidden state is mean-pooled over the attention mask and L2-normalized, and a unit's weight is its token count without special tokens. Exact ties go to the earlier unit. The kept units are emitted in source order.

The artifact is written to a temporary file and checked by resolve_channel_texts under a temporary pin before it is moved into place, so a failing invariant leaves no artifact. It is committed and pinned in WINDOW_SUMMARIES; neither data preprocess nor data all runs this.

Selection dataclass

The units a summary keeps.

Attributes:

Name Type Description
kept List[int]

Indices of the kept units, in source order.
plateau bool

True when the selection stopped because no unit that still fit raised the cosine.

backbone_embedder(model, tokenizer, *, batch_size=64)

Unit vectors from a frozen backbone: each unit read unmarked and alone, its last hidden state mean-pooled over the attention mask.

Parameters:

Name Type Description Default
model Any

A Hugging Face encoder returning last_hidden_state.

 required
tokenizer Any

Its tokenizer.

 required
batch_size int

Units per forward pass.

 64

Returns:

Type Description
UnitEmbedder

A function from units to their vectors (float64).

generate_window_summaries(descriptions_path, output_path, *, backbone, force=False, model=None, tokenizer=None, revision=None)

Summarize every over-window channel text, check the artifact, and write it and its provenance.

model and tokenizer default to load_backbone(backbone), from the local Hugging Face cache; tests pass small ones.

Parameters:

Name Type Description Default
descriptions_path Path

The descriptions parquet.

 required
output_path Path

The artifact (CSV); its provenance is written beside it.

 required
backbone str

The backbone, whose trained window the summaries fit.

 required
force bool

Overwrite an existing artifact.

 False
model Optional[Any]

The backbone's model.

 None
tokenizer Any

The backbone's tokenizer.

 None
revision Optional[str]

The backbone's snapshot revision.

 None

Returns:

Type Description
SummariesPin

The pin to commit in WINDOW_SUMMARIES.

Raises:

Type Description
FileExistsError

If the artifact exists and force is False.
ValueError

If a text cannot be summarized, or the artifact fails a resolver check.

joiner(channel)

What a summary's units are joined by: the examples separator, or one space.

select_units(vectors, weights, fits)

Greedy centrality selection over one text's units.

Parameters:

Name Type Description Default
vectors ndarray

One row per unit, in source order.

 required
weights ndarray

Each unit's token count, without special tokens.

 required
fits Callable[[List[List[int]]], List[bool]]

For each candidate (unit indices in source order), whether its marked summary fits the window.

 required

Returns:

Type Description
Selection

The kept units.

Raises:

Type Description
ValueError

If no unit fits the window on its own.

summary_rows(descriptions, tokenizer, embed, *, window)

One summary row per over-window channel text.

Identical source texts get identical summaries, one row per code. Each distinct unit is embedded once.

Parameters:

Name Type Description Default
descriptions DataFrame

Codes and their four channel texts.

 required
tokenizer Any

The backbone's tokenizer.

 required
embed UnitEmbedder

Unit vectors (backbone_embedder).

 required
window int

The trained window.

 required

Returns:

Type Description
DataFrame

The rows, with the columns of SUMMARIES_SCHEMA, and per channel the summaries that
Dict[str, int]

ended at the plateau stop.

Raises:

Type Description
ValueError

If a title is over the window, or a unit is over its channel's budget.