Upload VLA tokenizer with model card

README.md

@@ -14,9 +14,10 @@ license: apache-2.0
 # VLA Tokenizer — Qwen3
 
 Extended Qwen3 tokenizer for the **FineVideo-VLA** multimodal dataset.
-Adds all VLA tokens (video, 3D pose, SNAC audio) on top of the Qwen3 base tokenizer.
+Adds all VLA tokens (video, 3D pose, SNAC audio, caption/speech wrappers) on
+top of the Qwen3 base tokenizer.
 
-**Vocab size: 257,897** (~151,669 Qwen3 base + 106,228 VLA tokens)
+**Vocab size: 257,901** (~151,669 Qwen3 base + 106,232 VLA tokens)
 
 > **Why Qwen3?** The Qwen3 family has strong multilingual + reasoning abilities
 > and native HuggingFace ecosystem support (vLLM, llama.cpp, transformers).
@@ -41,8 +42,9 @@ Adds all VLA tokens (video, 3D pose, SNAC audio) on top of the Qwen3 base tokeni
 | SNAC Level 1 even | `<snac_132362>` – `<snac_136457>` | 4,096 | 25 Hz fine audio (even frames) |
 | SNAC Level 1 odd | `<snac_144650>` – `<snac_148745>` | 4,096 | 25 Hz fine audio (odd frames) |
 | SNAC wrappers | `<snac>`, `</snac>` | 2 | Block delimiters |
+| Caption/speech wrappers | `<caption>`, `</caption>`, `<speech>`, `</speech>` | 4 | Inline caption/dialogue interleaving |
 
-**Total VLA tokens added: 106,228**
+**Total VLA tokens added: 106,232**
 
 ---
 
@@ -60,7 +62,7 @@ Adds all VLA tokens (video, 3D pose, SNAC audio) on top of the Qwen3 base tokeni
 from transformers import AutoTokenizer
 
 tok = AutoTokenizer.from_pretrained("EmpathicRobotics/tokenizer-vla-qwen3")
-print(len(tok))  # 257897
+print(len(tok))  # 257901
 
 # All VLA tokens are atomic
 print(tok.encode("<seed2_1137>",   add_special_tokens=False))  # single ID
@@ -113,7 +115,7 @@ tok.save_pretrained("tokenizer-vla-qwen3")
 # vocab size: 257,897
 ```
 
-Full script: `tools/build_tokenizers.py` in the
+Full script: `tools/tokenizer/build_tokenizers.py` in the
 [finevideo-vla](https://github.com/TieuDaoChanNhan/finevideo-vla) repo.
 
 ---

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6da92a63580d67a15989711c8a0817314e64df599610605dd3f8e3f518f84956
-size 31654989
+oid sha256:d97a3c88b39e2dffeee8d470d767d48e2777c0d1395aaeee32b3709cb35e68a5
+size 31655733