---
license: mit
language:
- it
- en
library_name: tiktoken
tags:
- tokenizer
- bpe
- italian
- english
- code
- fineweb-2
- fineweb-edu
---

# ita-en-code-bpe-48k

A byte-level **BPE tokenizer**, vocab **49152 (48k)**, trained with
[`rustbpe`](https://github.com/karpathy/rustbpe) on a **40% Italian / 40% English
/ 20% code** mix (~4B chars), so it is efficient across all three:

- **Italian** — [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) `ita_Latn` (filtered)
- **English** — [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample/100BT` (`int_score ≥ 3`)
- **code** — [github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean)

Inference uses [`tiktoken`](https://github.com/openai/tiktoken) — `tokenizer.pkl`
is a pickled `tiktoken.Encoding`. GPT-4 split pattern (cl100k). Italian stays
efficient (IT/EN share Latin script ⇒ shared merges) while English and code are
first-class.

## Special tokens (aligned with DeepSeek-V4 / GLM-5.2)

33 special tokens, ids **49119–49151**. Pretraining packs each doc `<|bos|> … <|eos|>`.

| group | tokens |
|---|---|
| core | `<\|bos\|>` (49119), `<\|eos\|>` (49120), `<\|pad\|>` (49121) |
| chat | `<\|system\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|observation\|>` |
| turn | `<\|eot\|>` |
| reasoning | `<think>`, `</think>` |
| tools | `<tool_call>`, `</tool_call>`, `<tool_response>`, `</tool_response>` |
| code FIM | `<\|fim_begin\|>`, `<\|fim_hole\|>`, `<\|fim_end\|>` |
| reserved | `<\|reserved_0\|>` … `<\|reserved_15\|>` |

## Files
- `tokenizer.pkl` — pickled `tiktoken.Encoding`.
- `token_bytes.pt` — per-id UTF-8 byte length (0 for specials), for bits-per-byte eval.
- `summary.json` — training config + stats.

## Usage
```python
import pickle
from huggingface_hub import hf_hub_download
enc = pickle.load(open(hf_hub_download("procmarco/ita-en-code-bpe-48k", "tokenizer.pkl"), "rb"))
ids = enc.encode_ordinary("def somma(a, b):\n    return a + b  # Ciao")
print(len(ids), enc.decode(ids))
```
`pip install tiktoken` runs it; training used `rustbpe`. Companion token dataset:
[procmarco/ita-en-code-tokens-48k](https://huggingface.co/datasets/procmarco/ita-en-code-tokens-48k).