README.md

---
language:
- multilingual
tags:
- tokenizer
- bpe
- byte-level-bpe
- chatml
- routing
- moe
- robotics
- jiarck
license: other
license_name: jirack-tokenizer-v1.0
license_link: LICENSE
---

Enjoy — We extend the JiRack Models Ecosystem! 🚀

# JiRack Utra Pro Tokenizer 512

**JiRack Utra Pro Tokenizer - 347 active language editions of Wikipedia** 

**High-performance production-grade Byte-Level BPE tokenizer** developed as part of the **JiRack Ternary Models** ecosystem.

This is the **Ultra Pro version** designed for maximum quality, advanced compression vs 256k Ultra version for very large models , and precision in complex real-world applications.

- JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

### Open Robot platform 

- **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
- **Unitree g1**  https://a.co/d/0e4A8YVc
- **LimX Oli**  https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
- **ubtrobot** https://www.ubtrobot.com/en/
- **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
  


### Key Features

- **Algorithm**: Byte-Level BPE
- **Vocabulary Size**: **524,288** tokens — excellent balance between precision and efficiency
- **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation
- **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts

### Special Tokens Support

- Full **ChatML** dialogue format (`<|im_start|>`, `<|im_end|>`)
- FIM (Fill-in-the-Middle) support for code generation
- Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.)
- Extended robotics and control tokens

### CMS Manhattan Service & Support 

- Jirack patent guards your technology for competitors 
- Redesign Llama , Qwen , Gemma to Ternary model
- Re-tain and replace embeddings for  Llama , Qwen , Gemma  to extend langeages to 347
- Accelerate inference via high compression 512K tokenizer and replace multiplication with sum operations via Ternary weights

### Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-UltraPro-Tokenizer-512K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 524288
pad_token_id: 2
eos_token_id: 1

```

### JiRack Pretrain Dataset
https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

```bash
python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]
```


### Benchmark for tokens  quality .

```bash

=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 31576, 522, 66472, 6176, 2037, 576, 7637, 269, 4, 454]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
31576 -> ' Write'
522 -> ' a'
66472 -> ' merge'
6176 -> ' sort'
2037 -> ' function'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 196893, 7637, 476, 24410, 16482, 61950, 14333, 295880, 92252, 870, 4, 454]

=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
196893 -> ' 用'
7637 -> ' Python'
476 -> ' '
24410 -> '写'
16482 -> '一个'
61950 -> '归'
14333 -> '并'
295880 -> '排序'
92252 -> '函数'
870 -> '。'
4 -> '<|im_end|>'
454 -> '
'


== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 170797, 3433, 4484, 56203, 595, 3102, 34759, 720, 7637, 269, 4, 454]

=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
170797 -> ' Éc'
3433 -> 'ris'
4484 -> ' une'
56203 -> ' fonction'
595 -> ' de'
3102 -> ' tri'
34759 -> ' fusion'
720 -> ' en'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 115144, 18528, 6806, 256728, 268, 144869, 268, 386798, 592, 576, 7637, 269, 4, 454]

=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
115144 -> ' Schre'
18528 -> 'ibe'
6806 -> ' eine'
256728 -> ' Merge'
268 -> '-'
144869 -> 'Sort'
268 -> '-'
386798 -> 'Funkt'
592 -> 'ion'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 33139, 9120, 360514, 209685, 6513, 5867, 1187, 580, 121802, 6176, 264, 474504, 84721, 2518, 1435, 269, 4, 454]

=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33139 -> ' اك'
9120 -> 'تب'
360514 -> ' دالة'
209685 -> ' فرز'
6513 -> ' بال'
5867 -> 'دم'
1187 -> 'ج'
580 -> ' ('
121802 -> 'merge'
6176 -> ' sort'
264 -> ')'
474504 -> ' بلغة'
84721 -> ' باي'
2518 -> 'ث'
1435 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 35013, 91654, 492868, 430356, 174599, 302515, 467815, 874, 66017, 269, 4, 454]

=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35013 -> ' Нап'
91654 -> 'иши'
492868 -> ' функцию'
430356 -> ' сорти'
174599 -> 'ровки'
302515 -> ' слия'
467815 -> 'нием'
874 -> ' на'
66017 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'



```


## 📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

- **Email:** grabko@cmsmanhattan.com
- **Phone:** +1 (516) 777-0945
- **Location:** New York, USA

## 📧 Copyright 2026  CMS Manhattan . All rights reserved