| --- |
| license: mit |
| language: |
| - it |
| - en |
| library_name: tiktoken |
| tags: |
| - tokenizer |
| - bpe |
| - italian |
| - english |
| - code |
| - fineweb-2 |
| - fineweb-edu |
| --- |
| |
| # ita-en-code-bpe-48k |
|
|
| A byte-level **BPE tokenizer**, vocab **49152 (48k)**, trained with |
| [`rustbpe`](https://github.com/karpathy/rustbpe) on a **40% Italian / 40% English |
| / 20% code** mix (~4B chars), so it is efficient across all three: |
|
|
| - **Italian** — [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) `ita_Latn` (filtered) |
| - **English** — [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample/100BT` (`int_score ≥ 3`) |
| - **code** — [github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean) |
|
|
| Inference uses [`tiktoken`](https://github.com/openai/tiktoken) — `tokenizer.pkl` |
| is a pickled `tiktoken.Encoding`. GPT-4 split pattern (cl100k). Italian stays |
| efficient (IT/EN share Latin script ⇒ shared merges) while English and code are |
| first-class. |
|
|
| ## Special tokens (aligned with DeepSeek-V4 / GLM-5.2) |
|
|
| 33 special tokens, ids **49119–49151**. Pretraining packs each doc `<|bos|> … <|eos|>`. |
|
|
| | group | tokens | |
| |---|---| |
| | core | `<\|bos\|>` (49119), `<\|eos\|>` (49120), `<\|pad\|>` (49121) | |
| | chat | `<\|system\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|observation\|>` | |
| | turn | `<\|eot\|>` | |
| | reasoning | `<think>`, `</think>` | |
| | tools | `<tool_call>`, `</tool_call>`, `<tool_response>`, `</tool_response>` | |
| | code FIM | `<\|fim_begin\|>`, `<\|fim_hole\|>`, `<\|fim_end\|>` | |
| | reserved | `<\|reserved_0\|>` … `<\|reserved_15\|>` | |
|
|
| ## Files |
| - `tokenizer.pkl` — pickled `tiktoken.Encoding`. |
| - `token_bytes.pt` — per-id UTF-8 byte length (0 for specials), for bits-per-byte eval. |
| - `summary.json` — training config + stats. |
|
|
| ## Usage |
| ```python |
| import pickle |
| from huggingface_hub import hf_hub_download |
| enc = pickle.load(open(hf_hub_download("procmarco/ita-en-code-bpe-48k", "tokenizer.pkl"), "rb")) |
| ids = enc.encode_ordinary("def somma(a, b):\n return a + b # Ciao") |
| print(len(ids), enc.decode(ids)) |
| ``` |
| `pip install tiktoken` runs it; training used `rustbpe`. Companion token dataset: |
| [procmarco/ita-en-code-tokens-48k](https://huggingface.co/datasets/procmarco/ita-en-code-tokens-48k). |
|
|
