README.md

---
language:
  - en
library_name: transformers
tags:
  - tokenizer
  - language-modeling
license: apache-2.0
---

# v11-tokenizer

**adopted.** This is the current recommended tokenizer for this line.

## Loading

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
```

No `trust_remote_code`, no clone, no training code. That is verified against the
downloaded artifact at publish time, not asserted.

## Identity

**Identity is the content hash, not the Hub revision.** Re-pushing identical
bytes mints a new commit oid; a README edit does too. Join on this instead — it
is the same value a checkpoint records as `tokenizer_hash`:

| | |
|---|---|
| `tokenizer.json` sha256 | `10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae` |
| Vocabulary size | 71,260 |
| Source | [chrishayuk/v-tokenizers](https://github.com/chrishayuk/v-tokenizers) |
| Source commit | `3ed5f3cb7aea16b150588f27f24a7f31121b1792` |

This repo is immutable: a fix ships under a new name, never as a replacement.

## Special tokens

| Role | Token | ID |
|---|---|---|
| Unknown | `<unk>` | 1 |
| Beginning | `<s>` | 2 |
| End | `</s>` | 3 |
| Padding | `<pad>` | 0 |

## Using the Rust CLI

`v11.vocab.bin` is the native vocabulary, shipped here because `cargo install` delivers
the binary and not the data:

```bash
cargo install v11-cli
huggingface-cli download chrishayuk/v11-tokenizer v11.vocab.bin --local-dir .
v11 --model v11.vocab.bin vocab --blocks
v11 --model v11.vocab.bin encode --text "Once upon a time" --show-pieces
```

## Dormant blocks — read before sizing an embedding table

This vocabulary declares blocks that are **present as ids but masked out of the
projection matmul and optimizer state** at 115M parameters. Sizing an embedding
table from `len(tokenizer)` will disagree with the model's active slice, and no
vocabulary-length check will catch it.

See `dormant_blocks.json` for the machine-readable declaration.

| Block | Active @115M | Active @1B | Tokens frozen |
|---|---|---|---|
| dormant: T-cell | False | True | no (schema pinned, filled at TOK-5) |
| dormant: T-scale | False | True | no (schema pinned, filled at TOK-5) |

## Golden encodings

`golden_encodings.json` carries 116 frames from the MSI frame battery —
call operands, post-delimiter and mid-string-punctuation surfaces, not
"representative strings", because those are where segmentation actually breaks.
Every one is replayed against the downloaded artifact at publish time.

```python
import json, urllib.request
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
golden = json.load(urllib.request.urlopen(
    "https://huggingface.co/chrishayuk/v11-tokenizer/resolve/main/golden_encodings.json"))
for name, case in golden.items():
    assert tok(case["text"], add_special_tokens=False)["input_ids"] == case["ids"], name
```

## Provenance

`provenance.json` points at the systems that own these facts rather than copying
them: the corpus lives in the **chuk-datasets** catalog (dataset name, version,
content sha), and results live in the **chuk-experiments** ledger (run id). This
repo is a publication location, not a second source of truth.

## Intended use

Research and training of compact language models.

## Limitations

Corpus coverage, language coverage and normalisation behaviour are described in
the source repository. Dormant-block ids are exposed by this artifact but inactive at 115M — see above.