CMSManhattan
/

QwenRoboticsTokenizer

Mixture of Experts

Model card Files Files and versions

kgrabko commited on Jun 29

Commit

4501150

·

verified ·

1 Parent(s): 371b467

Update README.md

Files changed (1) hide show

README.md +6 -6

README.md CHANGED Viewed

@@ -18,9 +18,9 @@ tags:
 # 💎 Qwen Robotics & Routing Tokenizer
 - Qwen Pro Tokenizer . Fully compatible with Qwen3 models.
 - Size {
- Vocab size: 262251
- pad_token_id: 0
- eos_token_id: 1
  }
 - Compatible with Qwen 3 • Optimized for Code . Use resize function without adaptation, see examples below
 - It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
@@ -214,6 +214,6 @@ print("pad_token_id:", tok.pad_token_id)
 print("eos_token_id:", tok.eos_token_id)
 '
-Vocab size: 262251
-pad_token_id: 0
-eos_token_id: 1

 # 💎 Qwen Robotics & Routing Tokenizer
 - Qwen Pro Tokenizer . Fully compatible with Qwen3 models.
 - Size {
+  Vocab size: 151778
+  pad_token_id: 151643
+  eos_token_id: 151645
  }
 - Compatible with Qwen 3 • Optimized for Code . Use resize function without adaptation, see examples below
 - It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
 print("eos_token_id:", tok.eos_token_id)
 '
+Vocab size: 151778
+pad_token_id: 151643
+eos_token_id: 151645