48k IT/EN/code BPE (40/40/20), frontier-aligned special tokens

README.md

@@ -0,0 +1,60 @@
+---
+license: mit
+language:
+- it
+- en
+library_name: tiktoken
+tags:
+- tokenizer
+- bpe
+- italian
+- english
+- code
+- fineweb-2
+- fineweb-edu
+---
+
+# ita-en-code-bpe-48k
+
+A byte-level **BPE tokenizer**, vocab **49152 (48k)**, trained with
+[`rustbpe`](https://github.com/karpathy/rustbpe) on a **40% Italian / 40% English
+/ 20% code** mix (~4B chars), so it is efficient across all three:
+
+- **Italian** — [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) `ita_Latn` (filtered)
+- **English** — [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample/100BT` (`int_score ≥ 3`)
+- **code** — [github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean)
+
+Inference uses [`tiktoken`](https://github.com/openai/tiktoken) — `tokenizer.pkl`
+is a pickled `tiktoken.Encoding`. GPT-4 split pattern (cl100k). Italian stays
+efficient (IT/EN share Latin script ⇒ shared merges) while English and code are
+first-class.
+
+## Special tokens (aligned with DeepSeek-V4 / GLM-5.2)
+
+33 special tokens, ids **49119–49151**. Pretraining packs each doc `<|bos|> … <|eos|>`.
+
+| group | tokens |
+|---|---|
+| core | `<\|bos\|>` (49119), `<\|eos\|>` (49120), `<\|pad\|>` (49121) |
+| chat | `<\|system\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|observation\|>` |
+| turn | `<\|eot\|>` |
+| reasoning | `<think>`, `</think>` |
+| tools | `<tool_call>`, `</tool_call>`, `<tool_response>`, `</tool_response>` |
+| code FIM | `<\|fim_begin\|>`, `<\|fim_hole\|>`, `<\|fim_end\|>` |
+| reserved | `<\|reserved_0\|>` … `<\|reserved_15\|>` |
+
+## Files
+- `tokenizer.pkl` — pickled `tiktoken.Encoding`.
+- `token_bytes.pt` — per-id UTF-8 byte length (0 for specials), for bits-per-byte eval.
+- `summary.json` — training config + stats.
+
+## Usage
+```python
+import pickle
+from huggingface_hub import hf_hub_download
+enc = pickle.load(open(hf_hub_download("procmarco/ita-en-code-bpe-48k", "tokenizer.pkl"), "rb"))
+ids = enc.encode_ordinary("def somma(a, b):\n    return a + b  # Ciao")
+print(len(ids), enc.decode(ids))
+```
+`pip install tiktoken` runs it; training used `rustbpe`. Companion token dataset:
+[procmarco/ita-en-code-tokens-48k](https://huggingface.co/datasets/procmarco/ita-en-code-tokens-48k).

summary.json

@@ -0,0 +1,34 @@
+{
+  "vocab_size": 49152,
+  "num_special_tokens": 33,
+  "ratios": {
+    "ita": 0.4,
+    "en": 0.4,
+    "code": 0.2
+  },
+  "char_budget": 4000000000,
+  "special_token_ids": {
+    "bos": 49119,
+    "eos": 49120,
+    "pad": 49121
+  },
+  "pad_distinct_from_eos": true,
+  "train_time_sec": 126.83898162841797,
+  "sanity": [
+    {
+      "text": "Ciao mondo, l'Italia \u00e8 bellissima.",
+      "num_tokens": 8,
+      "roundtrip_ok": true
+    },
+    {
+      "text": "The quick brown fox jumps over the lazy dog.",
+      "num_tokens": 11,
+      "roundtrip_ok": true
+    },
+    {
+      "text": "def add(a, b):\n    return a + b  # sum",
+      "num_tokens": 14,
+      "roundtrip_ok": true
+    }
+  ]
+}

tokenizer/token_bytes.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fecb6042d2eefc25c6dadcd30e4287f2d4c0466a9c45759a5487231a86634bd6
+size 198213

tokenizer/tokenizer.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e48d6a95c6e4e2b98eba5121dc0a8da77d3515613bcc0847fb742b86f995fdb9
+size 612162