File size: 3,535 Bytes
7e96a85 fa41352 7e96a85 fa41352 7e96a85 fa41352 7e96a85 fa41352 7e96a85 fa41352 7e96a85 fa41352 361c586 fa41352 b779c03 fa41352 7e96a85 fa41352 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
language:
- en
library_name: transformers
tags:
- tokenizer
- language-modeling
license: apache-2.0
---
# v11-tokenizer
**adopted.** This is the current recommended tokenizer for this line.
## Loading
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
```
No `trust_remote_code`, no clone, no training code. That is verified against the
downloaded artifact at publish time, not asserted.
## Identity
**Identity is the content hash, not the Hub revision.** Re-pushing identical
bytes mints a new commit oid; a README edit does too. Join on this instead — it
is the same value a checkpoint records as `tokenizer_hash`:
| | |
|---|---|
| `tokenizer.json` sha256 | `10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae` |
| Vocabulary size | 71,260 |
| Source | [chrishayuk/v-tokenizers](https://github.com/chrishayuk/v-tokenizers) |
| Source commit | `3ed5f3cb7aea16b150588f27f24a7f31121b1792` |
This repo is immutable: a fix ships under a new name, never as a replacement.
## Special tokens
| Role | Token | ID |
|---|---|---|
| Unknown | `<unk>` | 1 |
| Beginning | `<s>` | 2 |
| End | `</s>` | 3 |
| Padding | `<pad>` | 0 |
## Using the Rust CLI
`v11.vocab.bin` is the native vocabulary, shipped here because `cargo install` delivers
the binary and not the data:
```bash
cargo install v11-cli
huggingface-cli download chrishayuk/v11-tokenizer v11.vocab.bin --local-dir .
v11 --model v11.vocab.bin vocab --blocks
v11 --model v11.vocab.bin encode --text "Once upon a time" --show-pieces
```
## Dormant blocks — read before sizing an embedding table
This vocabulary declares blocks that are **present as ids but masked out of the
projection matmul and optimizer state** at 115M parameters. Sizing an embedding
table from `len(tokenizer)` will disagree with the model's active slice, and no
vocabulary-length check will catch it.
See `dormant_blocks.json` for the machine-readable declaration.
| Block | Active @115M | Active @1B | Tokens frozen |
|---|---|---|---|
| dormant: T-cell | False | True | no (schema pinned, filled at TOK-5) |
| dormant: T-scale | False | True | no (schema pinned, filled at TOK-5) |
## Golden encodings
`golden_encodings.json` carries 116 frames from the MSI frame battery —
call operands, post-delimiter and mid-string-punctuation surfaces, not
"representative strings", because those are where segmentation actually breaks.
Every one is replayed against the downloaded artifact at publish time.
```python
import json, urllib.request
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
golden = json.load(urllib.request.urlopen(
"https://huggingface.co/chrishayuk/v11-tokenizer/resolve/main/golden_encodings.json"))
for name, case in golden.items():
assert tok(case["text"], add_special_tokens=False)["input_ids"] == case["ids"], name
```
## Provenance
`provenance.json` points at the systems that own these facts rather than copying
them: the corpus lives in the **chuk-datasets** catalog (dataset name, version,
content sha), and results live in the **chuk-experiments** ledger (run id). This
repo is a publication location, not a second source of truth.
## Intended use
Research and training of compact language models.
## Limitations
Corpus coverage, language coverage and normalisation behaviour are described in
the source repository. Dormant-block ids are exposed by this artifact but inactive at 115M — see above.
|