File size: 2,198 Bytes
56f291f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | # Offset Contract
Every `EncodedToken` produced by `Tokenizer.unified.DualTrackTokenizer` and
its sub-tokenizers carries `start` and `end` integer fields. This document
specifies what those fields mean.
## Coordinate system
- `start` and `end` are **Python character offsets** into the original
input string passed to `encode_with_spans(text)`. They are not byte
offsets and not UTF-16 code-unit offsets.
- The slice `text[token.start:token.end]` should round-trip — when the
tokenizer can do so — to a string that re-encodes to the same token.
- `[start, end)` is half-open. `start == end` is allowed for zero-width
tokens such as inserted boundary markers.
## Special cases
- BOS, EOS, and other synthetic specials use `start = end = -1`.
- Byte-fallback tokens point at the **original character span** that
produced the bytes. A 3-byte emoji becomes 3 byte tokens that share
the same `(start, end)` covering the source character.
- MorphBPE tokens cover the original-text character range, including any
`MVS` / `NNBSP` / `FVS` control characters that fall inside the
morpheme. When precise mapping is not possible the tokenizer uses
conservative wider spans rather than raising.
- Image / video patch tokens use:
- `start, end` = the span of the `<image>` (or `<video>`)
placeholder in the original text, or `-1, -1` if the placeholder
was injected programmatically.
- `metadata = {"image_index": N}` (or `video_index`) so consumers can
correlate patches to a specific image in the `images=` argument.
## Invariants
For every encoded sequence:
1. All non-special token spans satisfy `0 <= start <= end <= len(text)`.
2. The non-special spans are weakly monotonic: each `start` is `>=` the
previous token's `start`.
3. `len(input_ids) == len(tokens) == len(attention_mask)`.
4. `special_tokens_mask[i] == 1` iff `tokens[i].track == "special"`.
## Why this matters
The contract enables:
- Re-rendering token-level model outputs back onto the source string
(highlighting, NER, alignment).
- Sanity-checking that BPE merges never silently dropped or duplicated
characters.
- Aligning multimodal placeholder ranges with raw image/video inputs.
|