zero v1 — M7 lookup table, run p35w-2m-s2500

README.md

@@ -137,7 +137,8 @@ Precisely what that changes, by caller:
 |---|---|---|
 | `zero_encoder.py` (the reference path) | 512, no padding | unchanged — **byte-identical output**; it calls `no_padding()` and reads truncation from `config.json` |
 | `tokenizers` directly | every `encode` padded to 512 | ragged unless you enable padding yourself |
-| `transformers` / Sentence Transformers | `model_max_length` 32768, so `truncation=True` with no explicit length truncated at 32768 | 512 — but truncation and padding still happen only when the *call* asks for them |
+| `transformers` | `model_max_length` 32768, so `truncation=True` with no explicit length truncated at 32768 | 512 — but truncation and padding still happen only when the *call* asks for them |
+| Sentence Transformers | requests truncation itself and may impose its own `max_seq_length` | unchanged in that respect; only the underlying default moves to 512 |
 | fastembed's tokenizer loader | truncation 8000, fixed-512 padding kept | truncation 512, dynamic padding |
 
 The reference path is unaffected either way, so no published number changes; only what a
@@ -220,7 +221,7 @@ Amazon ESCI is Apache-2.0. The teacher, `NovaSearch/stella_en_400M_v5`, is MIT.
 | | |
 |---|---|
 | first published | 2026-09-03 — the frozen M7 bundle, with stella's tokenizer files copied verbatim |
-| this revision | 2026-09-03 — `tokenizer_config.json` `model_max_length`/`max_length` → 512, `tokenizer.json` `padding` → `null`, and one broken snippet in this card fixed |
+| this revision | 2026-09-03, commit `f5479985` — `tokenizer_config.json` `model_max_length`/`max_length` 32768/8000 → 512, `tokenizer.json` `padding` `Fixed(512)` → `null`, and one broken snippet in this card fixed |
 
 `model.npz` is byte-identical across both (sha `a7007b1a…`) and the reference encoder's output is
 unchanged, so **no published number differs between revisions**. Pass `revision=` to

zero_encoder.py

@@ -47,6 +47,10 @@ class ZeroQueryEncoder:
         self.variant = variant
 
         self.tokenizer = Tokenizer.from_file(str(d / "tokenizer.json"))
+        n = self.tokenizer.get_vocab_size(with_added_tokens=True)
+        if n != self.rows.shape[0]:
+            raise ValueError(f"tokenizer has {n} tokens but the table has {self.rows.shape[0]} "
+                             "rows; a token id outside the table would index off the end")
         self.tokenizer.enable_truncation(max_length=self.max_length)
         # stella's tokenizer.json ships with padding-to-512 enabled. Padding would put ~500
         # [PAD] rows into every bag; the frozen path (transformers, padding off) never sees one.