--- language: - multilingual tags: - tokenizer - bpe - byte-level-bpe - chatml - routing - moe - robotics - jiarck license: other license_name: jirack-tokenizer-v1.0 license_link: LICENSE --- Enjoy — We extend the JiRack Models Ecosystem! 🚀 # JiRack Utra Pro Tokenizer 512 **JiRack Utra Pro Tokenizer - 347 active language editions of Wikipedia** **High-performance production-grade Byte-Level BPE tokenizer** developed as part of the **JiRack Ternary Models** ecosystem. This is the **Ultra Pro version** designed for maximum quality, advanced compression vs 256k Ultra version for very large models , and precision in complex real-world applications. - JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b ### Open Robot platform - **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html - **Unitree g1** https://a.co/d/0e4A8YVc - **LimX Oli** https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c- - **ubtrobot** https://www.ubtrobot.com/en/ - **x-humanoid** https://www.x-humanoid.com/detail/hskw.html ### Key Features - **Algorithm**: Byte-Level BPE - **Vocabulary Size**: **524,288** tokens — excellent balance between precision and efficiency - **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation - **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts ### Special Tokens Support - Full **ChatML** dialogue format (`<|im_start|>`, `<|im_end|>`) - FIM (Fill-in-the-Middle) support for code generation - Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.) - Extended robotics and control tokens ### CMS Manhattan Service & Support - Jirack patent guards your technology for competitors - Redesign Llama , Qwen , Gemma to Ternary model - Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347 - Accelerate inference via high compression 512K tokenizer and replace multiplication with sum operations via Ternary weights ### Usage ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-UltraPro-Tokenizer-512K") text = "__CODING__ __PYTHON__ Write a merge sort function in Python." tokens = tokenizer.tokenize(text) token_ids = tokenizer.encode(text) print("Tokens:", tokens) print("Token IDs:", token_ids) Vocab size: 524288 pad_token_id: 2 eos_token_id: 1 ``` ### JiRack Pretrain Dataset https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset ```bash python train_jirack_accelerate.py Processing jirack_pretrain_chunk_0.pt: 27%|█████████████████████▋ | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16] Processing jirack_pretrain_chunk_0.pt: 87%|███████████████████████████████████████████████████████████████████████ | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52] ``` ### Benchmark for tokens quality . ```bash === Text after ChatML Template === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|> === Tokens (IDs) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 31576, 522, 66472, 6176, 2037, 576, 7637, 269, 4, 454] === Decoding Token by Token === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 31576 -> ' Write' 522 -> ' a' 66472 -> ' merge' 6176 -> ' sort' 2037 -> ' function' 576 -> ' in' 7637 -> ' Python' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' === ChatML 模板处理后的文本 === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|> === Token (ID) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 196893, 7637, 476, 24410, 16482, 61950, 14333, 295880, 92252, 870, 4, 454] === 逐个 Token 解码 === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 196893 -> ' 用' 7637 -> ' Python' 476 -> ' ' 24410 -> '写' 16482 -> '一个' 61950 -> '归' 14333 -> '并' 295880 -> '排序' 92252 -> '函数' 870 -> '。' 4 -> '<|im_end|>' 454 -> ' ' == Texte après le modèle ChatML === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|> === Tokens (IDs) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 170797, 3433, 4484, 56203, 595, 3102, 34759, 720, 7637, 269, 4, 454] === Décodage token par token === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 170797 -> ' Éc' 3433 -> 'ris' 4484 -> ' une' 56203 -> ' fonction' 595 -> ' de' 3102 -> ' tri' 34759 -> ' fusion' 720 -> ' en' 7637 -> ' Python' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' === Text nach ChatML-Template === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|> === Token (IDs) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 115144, 18528, 6806, 256728, 268, 144869, 268, 386798, 592, 576, 7637, 269, 4, 454] === Dekodierung Token für Token === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 115144 -> ' Schre' 18528 -> 'ibe' 6806 -> ' eine' 256728 -> ' Merge' 268 -> '-' 144869 -> 'Sort' 268 -> '-' 386798 -> 'Funkt' 592 -> 'ion' 576 -> ' in' 7637 -> ' Python' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' === النص بعد تطبيق قالب ChatML === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|> === الرموز (IDs) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 33139, 9120, 360514, 209685, 6513, 5867, 1187, 580, 121802, 6176, 264, 474504, 84721, 2518, 1435, 269, 4, 454] === فك الترميز رمزا برمز === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 33139 -> ' اك' 9120 -> 'تب' 360514 -> ' دالة' 209685 -> ' فرز' 6513 -> ' بال' 5867 -> 'دم' 1187 -> 'ج' 580 -> ' (' 121802 -> 'merge' 6176 -> ' sort' 264 -> ')' 474504 -> ' بلغة' 84721 -> ' باي' 2518 -> 'ث' 1435 -> 'ون' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' === Текст после ChatML шаблона === <|im_start|>system You are a precise router model.<|im_end|> <|im_start|>user __CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|> === Токены (ID) === [5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 35013, 91654, 492868, 430356, 174599, 302515, 467815, 874, 66017, 269, 4, 454] === Декодирование по токенам === 5 -> '<|im_start|>system' 454 -> ' ' 3285 -> 'You' 934 -> ' are' 522 -> ' a' 21133 -> ' precise' 112585 -> ' router' 6457 -> ' model' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' 6 -> '<|im_start|>user' 454 -> ' ' 73 -> '__CODING__' 476 -> ' ' 88 -> '__PYTHON__' 35013 -> ' Нап' 91654 -> 'иши' 492868 -> ' функцию' 430356 -> ' сорти' 174599 -> 'ровки' 302515 -> ' слия' 467815 -> 'нием' 874 -> ' на' 66017 -> ' python' 269 -> '.' 4 -> '<|im_end|>' 454 -> ' ' ``` ## 📧 Contact & Licensing For joint ventures, hardware integration, or licensing inquiries: - **Email:** grabko@cmsmanhattan.com - **Phone:** +1 (516) 777-0945 - **Location:** New York, USA ## 📧 Copyright 2026 CMS Manhattan . All rights reserved