README.md

---
license: apache-2.0
language:
  - multilingual
tags:
  - tokenizer
  - bpe
  - byte-level-bpe
  - chatml
  - routing
  - moe
  - robotics
  - jiarck
---

Enjoy — We extend the JiRack Models Ecosystem! 🚀

# JiRack Pro Tokenizer 128K

**JiRack Pro Tokenizer - 347 active language editions of Wikipedia** 

**High-performance production-grade Byte-Level BPE tokenizer** developed as part of the **JiRack Ternary Models** ecosystem.

This is the **Pro version** designed for maximum quality, compression, and precision in complex real-world applications.

- JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

### Open Robot platform 

- **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
- **Unitree g1**  https://a.co/d/0e4A8YVc
- **LimX Oli**  https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
- **ubtrobot** https://www.ubtrobot.com/en/
- **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
  


### Key Features

- **Algorithm**: Byte-Level BPE
- **Vocabulary Size**: **128,000** tokens — excellent balance between precision and efficiency
- **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation
- **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts

### Special Tokens Support

- Full **ChatML** dialogue format (`<|im_start|>`, `<|im_end|>`)
- FIM (Fill-in-the-Middle) support for code generation
- Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.)
- Extended robotics and control tokens

### Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Pro-Tokenizer-128K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)
```

### JiRack Pretrain Dataset
https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

```bash
python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]
```

### Benchmark for tokens  quality .


```bash
=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>

=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 30667, 395, 58912, 6643, 2299, 462, 8646, 141, 4, 326]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
326 -> '\n'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
... 
72 -> '__CODING__'
87 -> '__PYTHON__'
30667 -> ' Write'
58912 -> ' merge'
6643 -> ' sort'

=== GERMAN =====
=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 1818, 420, 17119, 4086, 5977, 1039, 140, 178, 800, 140, 165, 11028, 472, 462, 8646, 141, 4, 326]

=== Dekodierung Token für Token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
1818 -> ' Sch'
420 -> 're'
17119 -> 'ibe'
4086 -> ' eine'
5977 -> ' Mer'
1039 -> 'ge'
140 -> '-'
178 -> 'S'
800 -> 'ort'
140 -> '-'
165 -> 'F'
11028 -> 'unkt'
472 -> 'ion'
462 -> ' in'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
=== SPANISH =====

=== Texto después de la plantilla ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Escribe una función de ordenamiento por mezcla (merge sort) en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 4230, 12807, 1569, 38999, 444, 16400, 13573, 1441, 15031, 71301, 450, 113333, 6643, 136, 561, 8646, 141, 4, 326]

=== Decodificación token por token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
4230 -> ' Es'
12807 -> 'cribe'
1569 -> ' una'
38999 -> ' función'
444 -> ' de'
16400 -> ' orden'
13573 -> 'amiento'
1441 -> ' por'
15031 -> ' mez'
71301 -> 'cla'
450 -> ' ('
113333 -> 'merge'
6643 -> ' sort'
136 -> ')'
561 -> ' en'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== RUSSAIN =====

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 24549, 57864, 16351, 56848, 101013, 111998, 3263, 945, 1657, 1081, 822, 75733, 141, 4, 326]

=== Декодирование по токенам ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
24549 -> ' Нап'
57864 -> 'иши'
16351 -> ' функ'
56848 -> 'цию'
101013 -> ' сорт'
111998 -> 'ировки'
3263 -> ' сл'
945 -> 'ия'
1657 -> 'ни'
1081 -> 'ем'
822 -> ' на'
75733 -> ' python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== FRANCE =====

=== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 112537, 3439, 2834, 27517, 444, 3276, 33659, 561, 8646, 141, 4, 326]

=== Décodage token par token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
112537 -> ' Éc'
3439 -> 'ris'
2834 -> ' une'
27517 -> ' fonction'
444 -> ' de'
3276 -> ' tri'
33659 -> ' fusion'
561 -> ' en'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== CHINA =====

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 348, 2879, 8646, 348, 22739, 19808, 72775, 15454, 20847, 29714, 115881, 760, 4, 326]

=== 逐个 Token 解码 ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
348 -> ' '
2879 -> '用'
8646 -> ' Python'
348 -> ' '
22739 -> '写'
19808 -> '一个'
72775 -> '归'
15454 -> '并'
20847 -> '排'
29714 -> '序'
115881 -> '函数'
760 -> '。'
4 -> '<|im_end|>'
326 -> '
'

=== JAPAN =========

=== ChatMLテンプレート適用後のテキスト ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Pythonでマージソートの関数を書いてください。<|im_end|>


=== トークン (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 8646, 1183, 3911, 11941, 7947, 7691, 720, 6055, 108920, 6689, 7351, 3176, 5222, 99686, 760, 4, 326]

=== トークンごとのデコード ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
8646 -> ' Python'
1183 -> 'で'
3911 -> 'マ'
11941 -> 'ージ'
7947 -> 'ソ'
7691 -> 'ート'
720 -> 'の'
6055 -> '関'
108920 -> '数を'
6689 -> '書'
7351 -> 'いて'
3176 -> 'く'
5222 -> 'だ'
99686 -> 'さい'
760 -> '。'
4 -> '<|im_end|>'
326 -> '
'

=== ARABIC =======

=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 45789, 8459, 770, 24238, 6361, 1142, 5451, 5669, 1062, 450, 113333, 6643, 136, 7954, 27188, 88636, 2181, 1343, 141, 4, 326]

=== فك الترميز رمزا برمز ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
45789 -> ' اك'
8459 -> 'تب'
770 -> ' د'
24238 -> 'الة'
6361 -> ' فر'
1142 -> 'ز'
5451 -> ' بال'
5669 -> 'دم'
1062 -> 'ج'
450 -> ' ('
113333 -> 'merge'
6643 -> ' sort'
136 -> ')'
7954 -> ' بل'
27188 -> 'غة'
88636 -> ' باي'
2181 -> 'ث'
1343 -> 'ون'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
```

## 📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

- **Email:** grabko@cmsmanhattan.com
- **Phone:** +1 (516) 777-0945
- **Location:** New York, USA

## 📧 Copyright 2026  CMS Manhattan . All rights reserved