README: v2 layout (vocab 128256, tags in reserved slots, think/tool tokens)
Browse files
README.md
CHANGED
|
@@ -1,269 +1,296 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
language:
|
| 4 |
-
- multilingual
|
| 5 |
-
tags:
|
| 6 |
-
- tokenizer
|
| 7 |
-
- bpe
|
| 8 |
-
- byte-level-bpe
|
| 9 |
-
- chatml
|
| 10 |
-
- routing
|
| 11 |
-
- moe
|
| 12 |
-
- robotics
|
| 13 |
-
- jiarck
|
| 14 |
-
- llama
|
| 15 |
-
- llama3
|
| 16 |
-
- meta
|
| 17 |
-
---
|
| 18 |
-
|
| 19 |
-
# π Llama Robotics & Routing Tokenizer
|
| 20 |
-
- Llama Pro Tokenizer from JiRack team . Fully compatible with Llama3 models from META Inc.
|
| 21 |
-
- Size {
|
| 22 |
-
Vocab size:
|
| 23 |
-
|
| 24 |
-
eos_token_id:
|
| 25 |
-
}
|
| 26 |
-
- Compatible with Llama 3 β’ Optimized for Code .
|
| 27 |
-
- It needs 100k example to fully adapt routing features for RAG Routing model . So check llama modification tokenizer rules
|
| 28 |
-
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 29 |
-
<|fim_prefix|>,
|
| 30 |
-
<|fim_middle|>,
|
| 31 |
-
<f|im_suffix|>
|
| 32 |
-
)
|
| 33 |
-
- Added Robotics & Embodiment tags (
|
| 34 |
-
"<|action_start|>",
|
| 35 |
-
"<|action_end|>",
|
| 36 |
-
"<|trajectory_start|>",
|
| 37 |
-
"<|trajectory_end|>",
|
| 38 |
-
"<|joint_start|>",
|
| 39 |
-
"<|joint_end|>",
|
| 40 |
-
"<|sensor_start|>",
|
| 41 |
-
"<|sensor_end|>",
|
| 42 |
-
"<|command_start|>",
|
| 43 |
-
"<|command_end|>",
|
| 44 |
-
"<|state_start|>",
|
| 45 |
-
"<|state_end|>",
|
| 46 |
-
"<|pose|>",
|
| 47 |
-
"<|velocity|>",
|
| 48 |
-
"<|force|>",
|
| 49 |
-
"<|torque|>",
|
| 50 |
-
"<|gripper|>",
|
| 51 |
-
"<|navigation|>",
|
| 52 |
-
"<|obstacle|>",
|
| 53 |
-
"<|task_start|>",
|
| 54 |
-
"<|task_end|>",
|
| 55 |
-
"<|plan_start|>",
|
| 56 |
-
"<|plan_end|>",
|
| 57 |
-
"<|behavior_start|>",
|
| 58 |
-
"<|behavior_end|>",
|
| 59 |
-
"<|skill_start|>",
|
| 60 |
-
"<|skill_end|>",
|
| 61 |
-
"<|motor|>",
|
| 62 |
-
"<|servo|>",
|
| 63 |
-
"<|imu|>",
|
| 64 |
-
"<|lidar|>",
|
| 65 |
-
"<|camera|>",
|
| 66 |
-
"<|depth|>",
|
| 67 |
-
"<|waypoint|>",
|
| 68 |
-
"<|path|>",
|
| 69 |
-
"<|collision|>",
|
| 70 |
-
"<|grasp|>",
|
| 71 |
-
"<|release|>",
|
| 72 |
-
"<|homing|>",
|
| 73 |
-
"<|emergency_stop|>",
|
| 74 |
-
"<|calibration|>",
|
| 75 |
-
"<|manipulation|>",
|
| 76 |
-
"<|locomotion|>",
|
| 77 |
-
"<|feedback|>",
|
| 78 |
-
"<|control_loop|>",)
|
| 79 |
-
|
| 80 |
-
- Added Multi models support (
|
| 81 |
-
"<|image|>",
|
| 82 |
-
"<|video|>",
|
| 83 |
-
"<|sound|>",
|
| 84 |
-
"<|voice|>",
|
| 85 |
-
"<|listening|>",
|
| 86 |
-
"<|vision|>",)
|
| 87 |
-
|
| 88 |
-
- Added Human mood tags (
|
| 89 |
-
"<|mood_happy|>",
|
| 90 |
-
"<|mood_sad|>",
|
| 91 |
-
"<|mood_angry|>",
|
| 92 |
-
"<|mood_neutral|>",
|
| 93 |
-
)
|
| 94 |
-
|
| 95 |
-
- Added RAG routing tags for RAG MoE Systems (
|
| 96 |
-
"__SCIENCE__",
|
| 97 |
-
"__CODING__",
|
| 98 |
-
"__STOCK_EXCHANGE__",
|
| 99 |
-
"__MEDICINE__",
|
| 100 |
-
"__GOVERNMENT__",
|
| 101 |
-
"__NEWS__",
|
| 102 |
-
"__GENERAL__",
|
| 103 |
-
"__MATERIAL_SCIENCE__",
|
| 104 |
-
"__ELECTRONICS__",
|
| 105 |
-
"__MICROELECTRONICS__",
|
| 106 |
-
"__ENGINEERING__",
|
| 107 |
-
"__ROBOTICS__",
|
| 108 |
-
"__ENERGY__",
|
| 109 |
-
"__AUTOMOTIVE__",
|
| 110 |
-
"__AVIATION__",
|
| 111 |
-
"__MATH__",
|
| 112 |
-
"__PYTHON__",
|
| 113 |
-
"__C__",
|
| 114 |
-
"__CPP__",
|
| 115 |
-
"__C_SHARP__",
|
| 116 |
-
"__JAVA__",
|
| 117 |
-
"__JAVASCRIPT__",
|
| 118 |
-
"__TYPESCRIPT__",
|
| 119 |
-
"__RUST__",
|
| 120 |
-
"__GO__",
|
| 121 |
-
"__RUBY__",
|
| 122 |
-
"__PHP__",
|
| 123 |
-
"__SWIFT__",
|
| 124 |
-
"__KOTLIN__",
|
| 125 |
-
"__BASH__",
|
| 126 |
-
"__SQL__",
|
| 127 |
-
"__ASSEMBLY__",
|
| 128 |
-
"__PHILOSOPHY__",
|
| 129 |
-
"__LITERATURE__",
|
| 130 |
-
"__SOCIOLOGY__",
|
| 131 |
-
"__PSYCHOLOGY__",
|
| 132 |
-
"__POLITICAL_SCIENCE__",
|
| 133 |
-
"__CULTURAL_STUDIES__",
|
| 134 |
-
"__ETHNOGRAPHY__",
|
| 135 |
-
"__HUMAN_RIGHTS__",
|
| 136 |
-
"__COMPLIANCE__",
|
| 137 |
-
"__MILITARY__",
|
| 138 |
-
"__BANKING__",
|
| 139 |
-
"__OIL_INDUSTRY__",
|
| 140 |
-
"__LIGHT_INDUSTRY__",
|
| 141 |
-
"__NATURE__",
|
| 142 |
-
"__OCEAN__",
|
| 143 |
-
"__SPORT__",
|
| 144 |
-
"__CULINARY__",
|
| 145 |
-
"__TRAVEL__",
|
| 146 |
-
"__HOBBY__"
|
| 147 |
-
)
|
| 148 |
-
|
| 149 |
-
- Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
|
| 150 |
-
- Enables efficient training on large-scale coding data for superior code generation and understanding.
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
**Inventor:** Konstantin Vladimirovich Grabko
|
| 154 |
-
**Organization:** CMS Manhattan JiRack Technology
|
| 155 |
-
**Official Site:** [www.cmsmanhattan.com](http://www.cmsmanhattan.com)
|
| 156 |
-
Designed for Banking and Fintech Institutions
|
| 157 |
-
|
| 158 |
-
**Banks and Fintech** JiRack Architecture: Build Sovereign Financial Models from Scratch
|
| 159 |
-
- Leveraging the JiRack Tokenizer and our Open Dataset, we enable financial institutions to develop secure, internal AI models from the ground up. This approach ensures maximum data privacy and model sovereignty for high-stakes banking operations.
|
| 160 |
-
- There is fix price for FinTech
|
| 161 |
-
- I recommend initializing the model with a 4K context window for initial stability, followed by scaling to 8K context using specialized JiRack 8K datasets. This two-stage approach ensures robust positional encoding before extending the model's long-range dependency.
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
**JiRack Corp Tokenizer solution**
|
| 165 |
-
- Use JiRack models with trusted, high-quality coding datasets while maintaining full control over your code and data privacy.
|
| 166 |
-
- Excellent fit for Banks, Fintech companies, and any organization that requires strict data confidentiality and security.
|
| 167 |
-
- Update JiRack model for corp privacy coding.
|
| 168 |
-
|
| 169 |
-
**JiRack Tokenizer Subcription**
|
| 170 |
-
- All subscribed members will receive regular tokenizer updates optimized for the latest high-quality coding datasets.
|
| 171 |
-
|
| 172 |
-
### Open Robot platform
|
| 173 |
-
|
| 174 |
-
- **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
|
| 175 |
-
- **Unitree g1** https://a.co/d/0e4A8YVc
|
| 176 |
-
- **LimX Oli** https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
|
| 177 |
-
- **ubtrobot** https://www.ubtrobot.com/en/
|
| 178 |
-
- **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
## ROBOS HUMANOID Project
|
| 182 |
-
|
| 183 |
-
- https://github.com/cmsmanhattan/ROBOS
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
### Robot dataset example
|
| 187 |
-
|
| 188 |
-
```bash
|
| 189 |
-
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: a table in front of me, with a red mug on it. Command: pick up the mug."},{"role":"assistant","content":"reach(x=0.4,y=0.0,z=0.3)"}]}
|
| 190 |
-
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and .ROBOTICS"},{"role":"user","content":"Scene: hand near the mug, distance 2 cm. Command: grasp."},{"role":"assistant","content":"grasp(force=60)"}]}
|
| 191 |
-
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: mug in hand, stable. Command: lift 20 cm."},{"role":"assistant","content":"lift(z=0.2)"}]}
|
| 192 |
-
|
| 193 |
-
```
|
| 194 |
-
|
| 195 |
-
```bash
|
| 196 |
-
{
|
| 197 |
-
"messages": [
|
| 198 |
-
{
|
| 199 |
-
"role": "system",
|
| 200 |
-
"content": "You are a robot. Use tags and . ROBOTICS"
|
| 201 |
-
},
|
| 202 |
-
{
|
| 203 |
-
"role": "user",
|
| 204 |
-
"content": "Scene: a table in front of me, with a red mug on it. Command: pick up the mug."
|
| 205 |
-
},
|
| 206 |
-
{
|
| 207 |
-
"role": "assistant",
|
| 208 |
-
"content": "reach(x=0.4,y=0.0,z=0.3)"
|
| 209 |
-
}
|
| 210 |
-
]
|
| 211 |
-
}
|
| 212 |
-
```
|
| 213 |
-
|
| 214 |
-
###
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
-
|
| 243 |
-
|
| 244 |
-
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
-
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
#
|
| 269 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- multilingual
|
| 5 |
+
tags:
|
| 6 |
+
- tokenizer
|
| 7 |
+
- bpe
|
| 8 |
+
- byte-level-bpe
|
| 9 |
+
- chatml
|
| 10 |
+
- routing
|
| 11 |
+
- moe
|
| 12 |
+
- robotics
|
| 13 |
+
- jiarck
|
| 14 |
+
- llama
|
| 15 |
+
- llama3
|
| 16 |
+
- meta
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# π Llama Robotics & Routing Tokenizer
|
| 20 |
+
- Llama Pro Tokenizer from JiRack team . Fully compatible with Llama3 models from META Inc.
|
| 21 |
+
- Size {
|
| 22 |
+
Vocab size: 128256 (same as Llama 3, no embedding resize needed)
|
| 23 |
+
bos_token_id: 128000 (<|begin_of_text|>)
|
| 24 |
+
eos_token_id: 128009 (<|eot_id|>)
|
| 25 |
+
}
|
| 26 |
+
- Compatible with Llama 3 β’ Optimized for Code . All tags live in Llama 3 reserved slots, so the vocabulary keeps the native 128256 size
|
| 27 |
+
- It needs 100k example to fully adapt routing features for RAG Routing model . So check llama modification tokenizer rules
|
| 28 |
+
- A Llama 3-based tokenizer enhanced with FIM markers from Microsoft datasets (
|
| 29 |
+
<|fim_prefix|>,
|
| 30 |
+
<|fim_middle|>,
|
| 31 |
+
<f|im_suffix|>
|
| 32 |
+
)
|
| 33 |
+
- Added Robotics & Embodiment tags (
|
| 34 |
+
"<|action_start|>",
|
| 35 |
+
"<|action_end|>",
|
| 36 |
+
"<|trajectory_start|>",
|
| 37 |
+
"<|trajectory_end|>",
|
| 38 |
+
"<|joint_start|>",
|
| 39 |
+
"<|joint_end|>",
|
| 40 |
+
"<|sensor_start|>",
|
| 41 |
+
"<|sensor_end|>",
|
| 42 |
+
"<|command_start|>",
|
| 43 |
+
"<|command_end|>",
|
| 44 |
+
"<|state_start|>",
|
| 45 |
+
"<|state_end|>",
|
| 46 |
+
"<|pose|>",
|
| 47 |
+
"<|velocity|>",
|
| 48 |
+
"<|force|>",
|
| 49 |
+
"<|torque|>",
|
| 50 |
+
"<|gripper|>",
|
| 51 |
+
"<|navigation|>",
|
| 52 |
+
"<|obstacle|>",
|
| 53 |
+
"<|task_start|>",
|
| 54 |
+
"<|task_end|>",
|
| 55 |
+
"<|plan_start|>",
|
| 56 |
+
"<|plan_end|>",
|
| 57 |
+
"<|behavior_start|>",
|
| 58 |
+
"<|behavior_end|>",
|
| 59 |
+
"<|skill_start|>",
|
| 60 |
+
"<|skill_end|>",
|
| 61 |
+
"<|motor|>",
|
| 62 |
+
"<|servo|>",
|
| 63 |
+
"<|imu|>",
|
| 64 |
+
"<|lidar|>",
|
| 65 |
+
"<|camera|>",
|
| 66 |
+
"<|depth|>",
|
| 67 |
+
"<|waypoint|>",
|
| 68 |
+
"<|path|>",
|
| 69 |
+
"<|collision|>",
|
| 70 |
+
"<|grasp|>",
|
| 71 |
+
"<|release|>",
|
| 72 |
+
"<|homing|>",
|
| 73 |
+
"<|emergency_stop|>",
|
| 74 |
+
"<|calibration|>",
|
| 75 |
+
"<|manipulation|>",
|
| 76 |
+
"<|locomotion|>",
|
| 77 |
+
"<|feedback|>",
|
| 78 |
+
"<|control_loop|>",)
|
| 79 |
+
|
| 80 |
+
- Added Multi models support (
|
| 81 |
+
"<|image|>",
|
| 82 |
+
"<|video|>",
|
| 83 |
+
"<|sound|>",
|
| 84 |
+
"<|voice|>",
|
| 85 |
+
"<|listening|>",
|
| 86 |
+
"<|vision|>",)
|
| 87 |
+
|
| 88 |
+
- Added Human mood tags (
|
| 89 |
+
"<|mood_happy|>",
|
| 90 |
+
"<|mood_sad|>",
|
| 91 |
+
"<|mood_angry|>",
|
| 92 |
+
"<|mood_neutral|>",
|
| 93 |
+
)
|
| 94 |
+
|
| 95 |
+
- Added RAG routing tags for RAG MoE Systems (
|
| 96 |
+
"__SCIENCE__",
|
| 97 |
+
"__CODING__",
|
| 98 |
+
"__STOCK_EXCHANGE__",
|
| 99 |
+
"__MEDICINE__",
|
| 100 |
+
"__GOVERNMENT__",
|
| 101 |
+
"__NEWS__",
|
| 102 |
+
"__GENERAL__",
|
| 103 |
+
"__MATERIAL_SCIENCE__",
|
| 104 |
+
"__ELECTRONICS__",
|
| 105 |
+
"__MICROELECTRONICS__",
|
| 106 |
+
"__ENGINEERING__",
|
| 107 |
+
"__ROBOTICS__",
|
| 108 |
+
"__ENERGY__",
|
| 109 |
+
"__AUTOMOTIVE__",
|
| 110 |
+
"__AVIATION__",
|
| 111 |
+
"__MATH__",
|
| 112 |
+
"__PYTHON__",
|
| 113 |
+
"__C__",
|
| 114 |
+
"__CPP__",
|
| 115 |
+
"__C_SHARP__",
|
| 116 |
+
"__JAVA__",
|
| 117 |
+
"__JAVASCRIPT__",
|
| 118 |
+
"__TYPESCRIPT__",
|
| 119 |
+
"__RUST__",
|
| 120 |
+
"__GO__",
|
| 121 |
+
"__RUBY__",
|
| 122 |
+
"__PHP__",
|
| 123 |
+
"__SWIFT__",
|
| 124 |
+
"__KOTLIN__",
|
| 125 |
+
"__BASH__",
|
| 126 |
+
"__SQL__",
|
| 127 |
+
"__ASSEMBLY__",
|
| 128 |
+
"__PHILOSOPHY__",
|
| 129 |
+
"__LITERATURE__",
|
| 130 |
+
"__SOCIOLOGY__",
|
| 131 |
+
"__PSYCHOLOGY__",
|
| 132 |
+
"__POLITICAL_SCIENCE__",
|
| 133 |
+
"__CULTURAL_STUDIES__",
|
| 134 |
+
"__ETHNOGRAPHY__",
|
| 135 |
+
"__HUMAN_RIGHTS__",
|
| 136 |
+
"__COMPLIANCE__",
|
| 137 |
+
"__MILITARY__",
|
| 138 |
+
"__BANKING__",
|
| 139 |
+
"__OIL_INDUSTRY__",
|
| 140 |
+
"__LIGHT_INDUSTRY__",
|
| 141 |
+
"__NATURE__",
|
| 142 |
+
"__OCEAN__",
|
| 143 |
+
"__SPORT__",
|
| 144 |
+
"__CULINARY__",
|
| 145 |
+
"__TRAVEL__",
|
| 146 |
+
"__HOBBY__"
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
+
- Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
|
| 150 |
+
- Enables efficient training on large-scale coding data for superior code generation and understanding.
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
**Inventor:** Konstantin Vladimirovich Grabko
|
| 154 |
+
**Organization:** CMS Manhattan JiRack Technology
|
| 155 |
+
**Official Site:** [www.cmsmanhattan.com](http://www.cmsmanhattan.com)
|
| 156 |
+
Designed for Banking and Fintech Institutions
|
| 157 |
+
|
| 158 |
+
**Banks and Fintech** JiRack Architecture: Build Sovereign Financial Models from Scratch
|
| 159 |
+
- Leveraging the JiRack Tokenizer and our Open Dataset, we enable financial institutions to develop secure, internal AI models from the ground up. This approach ensures maximum data privacy and model sovereignty for high-stakes banking operations.
|
| 160 |
+
- There is fix price for FinTech
|
| 161 |
+
- I recommend initializing the model with a 4K context window for initial stability, followed by scaling to 8K context using specialized JiRack 8K datasets. This two-stage approach ensures robust positional encoding before extending the model's long-range dependency.
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
**JiRack Corp Tokenizer solution**
|
| 165 |
+
- Use JiRack models with trusted, high-quality coding datasets while maintaining full control over your code and data privacy.
|
| 166 |
+
- Excellent fit for Banks, Fintech companies, and any organization that requires strict data confidentiality and security.
|
| 167 |
+
- Update JiRack model for corp privacy coding.
|
| 168 |
+
|
| 169 |
+
**JiRack Tokenizer Subcription**
|
| 170 |
+
- All subscribed members will receive regular tokenizer updates optimized for the latest high-quality coding datasets.
|
| 171 |
+
|
| 172 |
+
### Open Robot platform
|
| 173 |
+
|
| 174 |
+
- **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
|
| 175 |
+
- **Unitree g1** https://a.co/d/0e4A8YVc
|
| 176 |
+
- **LimX Oli** https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
|
| 177 |
+
- **ubtrobot** https://www.ubtrobot.com/en/
|
| 178 |
+
- **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
## ROBOS HUMANOID Project
|
| 182 |
+
|
| 183 |
+
- https://github.com/cmsmanhattan/ROBOS
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
### Robot dataset example
|
| 187 |
+
|
| 188 |
+
```bash
|
| 189 |
+
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: a table in front of me, with a red mug on it. Command: pick up the mug."},{"role":"assistant","content":"reach(x=0.4,y=0.0,z=0.3)"}]}
|
| 190 |
+
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and .ROBOTICS"},{"role":"user","content":"Scene: hand near the mug, distance 2 cm. Command: grasp."},{"role":"assistant","content":"grasp(force=60)"}]}
|
| 191 |
+
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: mug in hand, stable. Command: lift 20 cm."},{"role":"assistant","content":"lift(z=0.2)"}]}
|
| 192 |
+
|
| 193 |
+
```
|
| 194 |
+
|
| 195 |
+
```bash
|
| 196 |
+
{
|
| 197 |
+
"messages": [
|
| 198 |
+
{
|
| 199 |
+
"role": "system",
|
| 200 |
+
"content": "You are a robot. Use tags and . ROBOTICS"
|
| 201 |
+
},
|
| 202 |
+
{
|
| 203 |
+
"role": "user",
|
| 204 |
+
"content": "Scene: a table in front of me, with a red mug on it. Command: pick up the mug."
|
| 205 |
+
},
|
| 206 |
+
{
|
| 207 |
+
"role": "assistant",
|
| 208 |
+
"content": "reach(x=0.4,y=0.0,z=0.3)"
|
| 209 |
+
}
|
| 210 |
+
]
|
| 211 |
+
}
|
| 212 |
+
```
|
| 213 |
+
|
| 214 |
+
### Version 2 (current): tags in Llama 3 reserved slots
|
| 215 |
+
|
| 216 |
+
Since commit `dd9c19ba` all tags are placed into the Llama 3 reserved special-token slots instead of being appended
|
| 217 |
+
after the vocabulary. The vocabulary stays at **128,256** β exactly the Llama 3 embedding size, so any Llama 3 model
|
| 218 |
+
loads it without `resize_token_embeddings`.
|
| 219 |
+
|
| 220 |
+
| IDs | Tokens |
|
| 221 |
+
|---|---|
|
| 222 |
+
| 128141 β 128146 | multimodal: `<\|image\|>` β¦ `<\|vision\|>` |
|
| 223 |
+
| 128147 β 128150 | mood: `<\|mood_happy\|>` β¦ `<\|mood_neutral\|>` |
|
| 224 |
+
| 128151 β 128153 | FIM: `<fim_prefix>`, `<fim_middle>`, `<fim_suffix>` |
|
| 225 |
+
| 128154 β 128198 | robotics: `<\|action_start\|>` β¦ `<\|control_loop\|>` |
|
| 226 |
+
| 128199 β 128249 | RAG routing: `__SCIENCE__` β¦ `__HOBBY__` |
|
| 227 |
+
| 128250 / 128251 | `<think>` / `</think>` |
|
| 228 |
+
| 128252 / 128253 | `<tool_call>` / `</tool_call>` |
|
| 229 |
+
| 128254 / 128255 | `<tool_response>` / `</tool_response>` |
|
| 230 |
+
|
| 231 |
+
Used by **Llama3Ternary 2B** ([CMSManhattan/Llama3TernaryPro_2b](https://huggingface.co/CMSManhattan/Llama3TernaryPro_2b)).
|
| 232 |
+
|
| 233 |
+
The previous version (vocab 128,365, tags appended at IDs 128256+, needs `resize_token_embeddings`) is still
|
| 234 |
+
available as revision `703ce8f1`:
|
| 235 |
+
|
| 236 |
+
```python
|
| 237 |
+
tok = AutoTokenizer.from_pretrained("CMSManhattan/LlamaRoboticsTokenizer", revision="703ce8f1")
|
| 238 |
+
```
|
| 239 |
+
|
| 240 |
+
### Key Features
|
| 241 |
+
|
| 242 |
+
- **Algorithm**: Byte-Level BPE
|
| 243 |
+
- **Vocabulary Size**: **128,256** tokens β excellent balance between precision and efficiency
|
| 244 |
+
- **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation
|
| 245 |
+
- **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts
|
| 246 |
+
|
| 247 |
+
### Special Tokens Support
|
| 248 |
+
|
| 249 |
+
- Full **Llama3 compatible format** dialogue format
|
| 250 |
+
- FIM (Fill-in-the-Middle) support for code generation
|
| 251 |
+
- Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.)
|
| 252 |
+
- Extended robotics and control tokens
|
| 253 |
+
|
| 254 |
+
### CMS Manhattan Service & Support
|
| 255 |
+
|
| 256 |
+
- Jirack patent guards your technology for competitors
|
| 257 |
+
- Redesign Llama , Qwen , Gemma to Ternary model
|
| 258 |
+
- Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
|
| 259 |
+
- Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights
|
| 260 |
+
|
| 261 |
+
**Install for Llamma compatible models in your chat script**
|
| 262 |
+
|
| 263 |
+
- from transformers import AutoModelForCausalLM
|
| 264 |
+
|
| 265 |
+
- model = AutoModelForCausalLM.from_pretrained(your model)
|
| 266 |
+
|
| 267 |
+
- # Version 2: no resize needed for Llama 3 models (vocab 128256). Only for the old revision 703ce8f1:
|
| 268 |
+
- # model.resize_token_embeddings(len(tokenizer))
|
| 269 |
+
|
| 270 |
+
- print("Embedding size:", model.get_input_embeddings().weight.shape[0])
|
| 271 |
+
|
| 272 |
+
- # Tesr Tokenizer size !
|
| 273 |
+
(venv_ji) root@jirack2:# python -c '
|
| 274 |
+
from transformers import AutoTokenizer
|
| 275 |
+
tok = AutoTokenizer.from_pretrained("CMSManhattan/LlamaRoboticsTokenizer")
|
| 276 |
+
print("Vocab size:", len(tok))
|
| 277 |
+
print("bos_token_id:", tok.bos_token_id)
|
| 278 |
+
print("eos_token_id:", tok.eos_token_id)
|
| 279 |
+
print("<tool_call>:", tok.convert_tokens_to_ids("<tool_call>"))
|
| 280 |
+
'
|
| 281 |
+
|
| 282 |
+
Vocab size: 128256
|
| 283 |
+
bos_token_id: 128000
|
| 284 |
+
eos_token_id: 128009
|
| 285 |
+
<tool_call>: 128252
|
| 286 |
+
|
| 287 |
+
|
| 288 |
+
## π§ Contact & Licensing
|
| 289 |
+
|
| 290 |
+
For joint ventures, hardware integration, or licensing inquiries:
|
| 291 |
+
|
| 292 |
+
- **Email:** grabko@cmsmanhattan.com
|
| 293 |
+
- **Phone:** +1 (516) 777-0945
|
| 294 |
+
- **Location:** New York, USA
|
| 295 |
+
|
| 296 |
+
## π§ Copyright 2026 CMS Manhattan . All rights reserved
|