kgrabko commited on
Commit
57729cd
·
verified ·
1 Parent(s): 9fabd23

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +141 -0
README.md CHANGED
@@ -7,6 +7,122 @@ license_link: LICENSE
7
  # 💎 JiRack Tokenizer
8
  - Compatible with Llama 3 • Optimized for Code
9
  - A Llama 3-based tokenizer enhanced with FIM markers (<fim_prefix>, <fim_suffix>, <fim_middle>).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  - Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
11
  - Enables efficient training on large-scale coding data for superior code generation and understanding.
12
 
@@ -30,6 +146,31 @@ Designed for Banking and Fintech Institutions
30
  **JiRack Tokenizer Subcription**
31
  - All subscribed members will receive regular tokenizer updates optimized for the latest high-quality coding datasets.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  **Install for Llamma compatible models in your chat script**
35
 
 
7
  # 💎 JiRack Tokenizer
8
  - Compatible with Llama 3 • Optimized for Code
9
  - A Llama 3-based tokenizer enhanced with FIM markers (<fim_prefix>, <fim_suffix>, <fim_middle>).
10
+ - Added Robotics & Embodiment tags (
11
+ "<|action_start|>",
12
+ "<|action_end|>",
13
+ "<|trajectory_start|>",
14
+ "<|trajectory_end|>",
15
+ "<|joint_start|>",
16
+ "<|joint_end|>",
17
+ "<|sensor_start|>",
18
+ "<|sensor_end|>",
19
+ "<|command_start|>",
20
+ "<|command_end|>",
21
+ "<|state_start|>",
22
+ "<|state_end|>",
23
+ "<|pose|>",
24
+ "<|velocity|>",
25
+ "<|force|>",
26
+ "<|torque|>",
27
+ "<|gripper|>",
28
+ "<|navigation|>",
29
+ "<|obstacle|>",
30
+ "<|task_start|>",
31
+ "<|task_end|>",
32
+ "<|plan_start|>",
33
+ "<|plan_end|>",
34
+ "<|behavior_start|>",
35
+ "<|behavior_end|>",
36
+ "<|skill_start|>",
37
+ "<|skill_end|>",
38
+ "<|motor|>",
39
+ "<|servo|>",
40
+ "<|imu|>",
41
+ "<|lidar|>",
42
+ "<|camera|>",
43
+ "<|depth|>",
44
+ "<|waypoint|>",
45
+ "<|path|>",
46
+ "<|collision|>",
47
+ "<|grasp|>",
48
+ "<|release|>",
49
+ "<|homing|>",
50
+ "<|emergency_stop|>",
51
+ "<|calibration|>",
52
+ "<|manipulation|>",
53
+ "<|locomotion|>",
54
+ "<|feedback|>",
55
+ "<|control_loop|>",)
56
+
57
+ - Added Multi models support (
58
+ "<|image|>",
59
+ "<|video|>",
60
+ "<|sound|>",
61
+ "<|voice|>",
62
+ "<|listening|>",
63
+ "<|vision|>",)
64
+
65
+ - Added Human mood tags (
66
+ "<|mood_happy|>",
67
+ "<|mood_sad|>",
68
+ "<|mood_angry|>",
69
+ "<|mood_neutral|>",
70
+ )
71
+
72
+ - Added RAG routing tags for RAG MoE Systems (
73
+ "__SCIENCE__",
74
+ "__CODING__",
75
+ "__STOCK_EXCHANGE__",
76
+ "__MEDICINE__",
77
+ "__GOVERNMENT__",
78
+ "__NEWS__",
79
+ "__GENERAL__",
80
+ "__MATERIAL_SCIENCE__",
81
+ "__ELECTRONICS__",
82
+ "__MICROELECTRONICS__",
83
+ "__ENGINEERING__",
84
+ "__ROBOTICS__",
85
+ "__ENERGY__",
86
+ "__AUTOMOTIVE__",
87
+ "__AVIATION__",
88
+ "__MATH__",
89
+ "__PYTHON__",
90
+ "__C__",
91
+ "__CPP__",
92
+ "__C_SHARP__",
93
+ "__JAVA__",
94
+ "__JAVASCRIPT__",
95
+ "__TYPESCRIPT__",
96
+ "__RUST__",
97
+ "__GO__",
98
+ "__RUBY__",
99
+ "__PHP__",
100
+ "__SWIFT__",
101
+ "__KOTLIN__",
102
+ "__BASH__",
103
+ "__SQL__",
104
+ "__ASSEMBLY__",
105
+ "__PHILOSOPHY__",
106
+ "__LITERATURE__",
107
+ "__SOCIOLOGY__",
108
+ "__PSYCHOLOGY__",
109
+ "__POLITICAL_SCIENCE__",
110
+ "__CULTURAL_STUDIES__",
111
+ "__ETHNOGRAPHY__",
112
+ "__HUMAN_RIGHTS__",
113
+ "__COMPLIANCE__",
114
+ "__MILITARY__",
115
+ "__BANKING__",
116
+ "__OIL_INDUSTRY__",
117
+ "__LIGHT_INDUSTRY__",
118
+ "__NATURE__",
119
+ "__OCEAN__",
120
+ "__SPORT__",
121
+ "__CULINARY__",
122
+ "__TRAVEL__",
123
+ "__HOBBY__"
124
+ )
125
+
126
  - Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
127
  - Enables efficient training on large-scale coding data for superior code generation and understanding.
128
 
 
146
  **JiRack Tokenizer Subcription**
147
  - All subscribed members will receive regular tokenizer updates optimized for the latest high-quality coding datasets.
148
 
149
+ ### Open Robot platform
150
+
151
+ - **Tiangong** : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
152
+ - **Unitree g1** https://a.co/d/0e4A8YVc
153
+ - **LimX Oli** https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
154
+ - **ubtrobot** https://www.ubtrobot.com/en/
155
+ - **x-humanoid** https://www.x-humanoid.com/detail/hskw.html
156
+
157
+
158
+
159
+ ### Key Features
160
+
161
+ - **Algorithm**: Byte-Level BPE
162
+ - **Vocabulary Size**: **128,000** tokens — excellent balance between precision and efficiency
163
+ - **Multilingual & Technical Strength**: Optimized for English, Russian, code, scientific literature, and technical documentation
164
+ - **Domain Specialization**: Strong performance on programming languages, engineering, robotics, and scientific texts
165
+
166
+ ### Special Tokens Support
167
+
168
+ - Full **Llama3 compatible format** dialogue format
169
+ - FIM (Fill-in-the-Middle) support for code generation
170
+ - Rich set of domain routing tokens (`__CODING__`, `__PYTHON__`, `__ROBOTICS__`, `__SCIENCE__`, etc.)
171
+ - Extended robotics and control tokens
172
+
173
+
174
 
175
  **Install for Llamma compatible models in your chat script**
176