Input Window API

The backbone's trained input window (Req 9, roadmap Stage 5): no tokenizing path reads beyond it, a channel text over it is read as its window-fitting summary (roadmap Stage 6b), and the supervision bundle records each channel's texts beyond it.

The backbone's trained input window (Req 9, "Input windows"; Verification "Backbone input window").

Inputs fit the window the backbone was trained on, recorded here from the backbone's own documentation. For sentence-transformers/all-MiniLM-L6-v2, the model card at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 says that in training "the sequence length was limited to 128 tokens". Its 256 (sentence_bert_config.json's max_seq_length, the truncation it applies at inference) and 512 (config.json's max_position_embeddings) are not the trained window. Every tokenizing path refuses a max_length beyond the window. A channel text over it is read as its window-fitting summary (panels/window_summaries.py), so no channel text is truncated, and truncation stays a backstop for queries. The supervision bundle records each channel's share of texts beyond the window.

check_window(backbone, max_length)

The length to tokenize at: max_length, or the trained window when it is None.

Raises:

Type Description
ValueError

If max_length exceeds the backbone's trained window, or none is recorded.

overflow_shares(texts, count_tokens, window)

Each channel's share of present texts longer than window tokens.

Parameters:

Name Type Description Default
texts Mapping[str, Sequence[Optional[str]]]

Each channel's texts; a null or blank text is absent and not counted.

 required
count_tokens Callable[[List[str]], List[int]]

Token counts of a list of texts, special tokens included.

 required
window int

The trained window.

 required

Returns:

Type Description
Dict[str, Dict[str, Any]]

Per channel: present texts, over (longer than the window) and their share.

token_counter(tokenizer)

Token counts under tokenizer, special tokens included, without truncation.

trained_window(backbone)

The backbone's trained input window, in tokens.

Raises:

Type Description
ValueError

If no window is recorded for the backbone.