Tokenization Cache API

get_tokens(idx_code, cache)

Get tokens for a specific NAICS index or code from cache.

load_verified_tokenization_cache(cfg, *, description_fingerprint, codebook_fingerprint)

Load a tokenization cache whose sidecar matches the expected fingerprints.

Raises:

Type Description
RuntimeError

If the cache is missing or was built from other inputs.

tokenization_cache(cfg=TokenizationConfig(), *, description_fingerprint, codebook_fingerprint, use_locking=True)

Get tokenization cache, loading from disk or building if necessary.

A cache is reused only when its JSON sidecar records exactly the requested description and codebook fingerprints, tokenizer, max length, format, field markers and summaries; otherwise it is rebuilt, because its source text is independently reproducible.

This function is safe for multi-worker environments. It uses file locking to ensure only one worker builds the cache, while others wait and then load it.

Parameters:

Name Type Description Default
cfg TokenizationConfig

TokenizationConfig

 TokenizationConfig()
description_fingerprint str

SHA-256 identifying the descriptions input

 required
codebook_fingerprint str

Fingerprint of the code-ID assignment the cache keys follow

 required
use_locking bool

If False, skip locking (for fast reads when cache exists)

 True