# Offset Contract

Every `EncodedToken` produced by `Tokenizer.unified.DualTrackTokenizer` and
its sub-tokenizers carries `start` and `end` integer fields. This document
specifies what those fields mean.

## Coordinate system

- `start` and `end` are **Python character offsets** into the original
  input string passed to `encode_with_spans(text)`. They are not byte
  offsets and not UTF-16 code-unit offsets.
- The slice `text[token.start:token.end]` should round-trip — when the
  tokenizer can do so — to a string that re-encodes to the same token.
- `[start, end)` is half-open. `start == end` is allowed for zero-width
  tokens such as inserted boundary markers.

## Special cases

- BOS, EOS, and other synthetic specials use `start = end = -1`.
- Byte-fallback tokens point at the **original character span** that
  produced the bytes. A 3-byte emoji becomes 3 byte tokens that share
  the same `(start, end)` covering the source character.
- MorphBPE tokens cover the original-text character range, including any
  `MVS` / `NNBSP` / `FVS` control characters that fall inside the
  morpheme. When precise mapping is not possible the tokenizer uses
  conservative wider spans rather than raising.
- Image / video patch tokens use:
  - `start, end` = the span of the `<image>` (or `<video>`)
    placeholder in the original text, or `-1, -1` if the placeholder
    was injected programmatically.
  - `metadata = {"image_index": N}` (or `video_index`) so consumers can
    correlate patches to a specific image in the `images=` argument.

## Invariants

For every encoded sequence:

1. All non-special token spans satisfy `0 <= start <= end <= len(text)`.
2. The non-special spans are weakly monotonic: each `start` is `>=` the
   previous token's `start`.
3. `len(input_ids) == len(tokens) == len(attention_mask)`.
4. `special_tokens_mask[i] == 1` iff `tokens[i].track == "special"`.

## Why this matters

The contract enables:

- Re-rendering token-level model outputs back onto the source string
  (highlighting, NER, alignment).
- Sanity-checking that BPE merges never silently dropped or duplicated
  characters.
- Aligning multimodal placeholder ranges with raw image/video inputs.