Tokenizer/docs/offset_contract.md

Offset Contract

Every EncodedToken produced by Tokenizer.unified.DualTrackTokenizer and its sub-tokenizers carries start and end integer fields. This document specifies what those fields mean.

Coordinate system

• start and end are Python character offsets into the original input string passed to encode_with_spans(text). They are not byte offsets and not UTF-16 code-unit offsets.
• The slice text[token.start:token.end] should round-trip — when the tokenizer can do so — to a string that re-encodes to the same token.
• [start, end) is half-open. start == end is allowed for zero-width tokens such as inserted boundary markers.

Special cases

• BOS, EOS, and other synthetic specials use start = end = -1.
• Byte-fallback tokens point at the original character span that produced the bytes. A 3-byte emoji becomes 3 byte tokens that share the same (start, end) covering the source character.
• MorphBPE tokens cover the original-text character range, including any MVS / NNBSP / FVS control characters that fall inside the morpheme. When precise mapping is not possible the tokenizer uses conservative wider spans rather than raising.
• Image / video patch tokens use:
  • start, end = the span of the <image> (or <video>) placeholder in the original text, or -1, -1 if the placeholder was injected programmatically.
  • metadata = {"image_index": N} (or video_index) so consumers can correlate patches to a specific image in the images= argument.

Invariants

For every encoded sequence:

1. All non-special token spans satisfy 0 <= start <= end <= len(text).
2. The non-special spans are weakly monotonic: each start is >= the previous token's start.
3. len(input_ids) == len(tokens) == len(attention_mask).
4. special_tokens_mask[i] == 1 iff tokens[i].track == "special".

Why this matters

The contract enables:

• Re-rendering token-level model outputs back onto the source string (highlighting, NER, alignment).
• Sanity-checking that BPE merges never silently dropped or duplicated characters.
• Aligning multimodal placeholder ranges with raw image/video inputs.