Tokenizer/docs/external_implementation_notes.md

# External Implementation Notes

Design principles distilled from open-source tokenizers and multimodal
processors. We do not vendor code or weights; we record contracts and
patterns that shape this repo's API.

## HuggingFace tokenizers (`tokenizers.Encoding`)

- Source: https://huggingface.co/docs/tokenizers/api/encoding
- Encoding carries `ids`, `tokens`, `offsets`, `attention_mask`,
  `special_tokens_mask`, `type_ids`, `word_ids`, `sequence_ids`. We mirror
  this contract: `DualTrackResult` exposes `input_ids`, `tokens`,
  `attention_mask`, `special_tokens_mask` plus our routed `track`.
- Special tokens carry `offsets = (0, 0)` to signal "synthetic, not from
  text". We use `(-1, -1)` instead for BOS/EOS to make detection
  unambiguous against zero-length real spans; placeholder tokens (image
  patches) reuse the source `<image>` span and surface `image_index` in
  `metadata`.
- `char_to_token` / `token_to_chars` is feasible only when every token has
  a real character span. We document our offset contract (see
  `offset_contract.md`) so downstream layers can do the same lookups.

## tiktoken

- Source: https://github.com/openai/tiktoken
- BPE state is split into `mergeable_ranks` + a separate `special_tokens`
  dict; specials are matched out-of-band (regex split before BPE). We do
  the same: `vocab.SPECIAL_TOKENS` is reserved in segment 0 and routed
  before language detection.
- Rank-based BPE: merges are stored as `(piece -> rank)` and the lowest
  rank wins at each pair. Our `MorphBPETrainer` records merges in order
  and reconstructs ranks on load — the merge order *is* the rank.
- Byte-level fallback is guaranteed reversible & lossless. We adopt the
  same invariant in `generic_bpe.byte_fallback`: any input byte sequence
  encodes to vocab tokens and decodes back to the exact bytes; emoji
  round-trip is tested.

## SentencePiece

- Source: https://github.com/google/sentencepiece
- Trains directly from raw sentences; whitespace is a regular symbol
  (`▁`). We respect this: `▁` is reserved id 17, decoders treat it as a
  word boundary, and `_strip_hf_boundary_markers` normalizes `▁`/`Ġ`.
- NFKC normalization is part of the tokenizer, not a pre-step the caller
  must remember. Our Rust `normalize_to_nominal_unicode` plays the same
  role for Mongolian and is wired in via the CLI bridge so training
  pipelines cannot skip it.
- Vocabulary size is predetermined and the trainer optimizes inside that
  budget. Our `MorphBPETrainer.train(vocab_size=...)` stops at the target
  size and skips merges below `min_pair_freq` for stability.

## LLaVA processor

- Source: https://huggingface.co/docs/transformers/main/en/model_doc/llava
- The processor expands a single `<image>` placeholder into N image
  tokens based on `patch_size`, `image_seq_length`, and
  `num_additional_image_tokens` (CLS). We mirror this with
  `expand_image_placeholders_by_sizes` driven by `image_patch_count`.
- The processor validates that the text contains the right number of
  image placeholders for the supplied images. We raise `ValueError` when
  `image_sizes` length disagrees with `<image>` count.
- Image embeddings are merged into the LM after tokenization using the
  image-token positions. We therefore must return `image_token_spans` as
  `input_ids`-index ranges (not char ranges) so the model layer can locate
  the slot.

## Qwen2-VL processor

- Source: https://huggingface.co/docs/transformers/main/en/model_doc/qwen2_vl
- Vision tokens are bracketed by explicit start/end markers
  (`<|vision_start|>` / `<|vision_end|>`); the count between them depends
  on `image_grid_thw`. We adopt the same triple
  (`<image_start>`/`<image_patch>*N`/`<image_end>`) and document
  `image_token_spans = (start_index, end_index)` as the *inclusive* token
  range covering all three.
- Video is handled with the same scheme plus a temporal patch factor.
  Our `video_patch_count(num_frames, w, h, patch_size, temporal_patch_size,
  merge_size)` matches the formula
  `ceil(num_frames/temporal_patch_size) * (w/patch_size/merge_size) *
  (h/patch_size/merge_size)`.
- Dynamic resolution: the number of vision tokens is data-dependent.
  Processors must compute counts per-sample, not at config time. Our
  `MultimodalProcessor.__call__` accepts per-call `image_sizes` /
  `video_sizes` rather than baking patch counts into the tokenizer.

## vLLM multimodal processing

- Source: https://docs.vllm.ai/en/latest/features/multimodal_inputs.html
- vLLM enforces that the number of image placeholders in the prompt
  equals the number of supplied images (and exposes
  `limit_mm_per_prompt`). We validate the same invariant in the processor
  and surface a clear `ValueError`.
- The multi-modal data dict is keyed by modality (`"image"`, `"video"`)
  and accepts lists for batched inputs. Our processor accepts
  `images=[...]` / `videos=[...]` in the same shape so future vLLM-style
  serving is a thin wrapper.
- Allowed-media-domains and redirect controls are a security primitive,
  not a tokenizer concern. We note this for the future pipeline layer:
  the tokenizer must never fetch remote bytes itself; callers preload
  PIL images / bytes.