GRRNMAKER commited on
Commit
40276ca
·
verified ·
1 Parent(s): e4c8aed

Upload Symb tokenizer files (tokenizer.json, configs, README)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - multilingual
5
+ license: apache-2.0
6
+ tags:
7
+ - tokenizer
8
+ - fast-tokenizer
9
+ - symb
10
+ - synaxim
11
+ - chike
12
+ - dudu
13
+ ---
14
+
15
+ # SymTokinzer (SYNAXIM / .symb Fast Tokenizer)
16
+
17
+ This repository hosts the official fast tokenizer for models running on the **SYNAXIM** runtime and native `.symb` binary container formats (including **Chíìkè**, **Dúdú**, and related agentic reasoning models).
18
+
19
+ ## Specifications
20
+
21
+ - **Vocabulary Size:** 128,256
22
+ - **Architecture:** Byte-Pair Encoding (BPE)
23
+ - **Context Length:** Up to 131,072+ tokens
24
+ - **Special Tokens:**
25
+ - `<|begin_of_text|>` (ID: `128000`) — BOS token
26
+ - `<|end_of_text|>` (ID: `128001`) — PAD / Stream end token
27
+ - `<|start_header_id|>` (ID: `128006`) — Start of role header
28
+ - `<|end_header_id|>` (ID: `128007`) — End of role header
29
+ - `<|eom_id|>` (ID: `128008`) — End of message
30
+ - `<|eot_id|>` (ID: `128009`) — End of turn / EOS token
31
+
32
+ ## Quickstart
33
+
34
+ ### 1. Using Hugging Face `transformers`
35
+
36
+ ```python
37
+ from transformers import AutoTokenizer
38
+
39
+ tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymTokinzer")
40
+
41
+ # Tokenize text
42
+ inputs = tokenizer("Hello, world!", return_tensors="pt")
43
+ print(inputs)
44
+
45
+ # Chat template formatting
46
+ messages = [
47
+ {"role": "system", "content": "You are a helpful reasoning assistant."},
48
+ {"role": "user", "content": "Explain quantum entanglement in simple terms."}
49
+ ]
50
+ formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
51
+ print(formatted)
52
+ ```
53
+
54
+ ### 2. Using Hugging Face `tokenizers` (Fast Rust Backend)
55
+
56
+ ```python
57
+ from tokenizers import Tokenizer
58
+ from huggingface_hub import hf_hub_download
59
+
60
+ tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymTokinzer", filename="tokenizer.json")
61
+ tokenizer = Tokenizer.from_file(tokenizer_file)
62
+
63
+ encoded = tokenizer.encode("Running on SYNAXIM native runtime.")
64
+ print("Token IDs:", encoded.ids)
65
+ print("Decoded:", tokenizer.decode(encoded.ids))
66
+ ```
67
+
68
+ ### 3. Using Native SYNAXIM / Chíìkè Inference
69
+
70
+ ```python
71
+ from chike_symb_inference import CozetTokenizer
72
+
73
+ tokenizer = CozetTokenizer()
74
+ input_ids = tokenizer.encode("Task: optimize latency.")
75
+ print(input_ids)
76
+ ```
special_tokens_map.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<|begin_of_text|>",
3
+ "eos_token": "<|eot_id|>",
4
+ "pad_token": "<|end_of_text|>"
5
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:12487b766b0b1584dcc5311824df327d5ea154939524790c643cdf2a3f6adf9f
3
+ size 17209921
tokenizer_config.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "128000": {
4
+ "content": "<|begin_of_text|>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "128001": {
12
+ "content": "<|end_of_text|>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "128006": {
20
+ "content": "<|start_header_id|>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "128007": {
28
+ "content": "<|end_header_id|>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "128008": {
36
+ "content": "<|eom_id|>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "128009": {
44
+ "content": "<|eot_id|>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ }
51
+ },
52
+ "bos_token": "<|begin_of_text|>",
53
+ "clean_up_tokenization_spaces": false,
54
+ "eos_token": "<|eot_id|>",
55
+ "pad_token": "<|end_of_text|>",
56
+ "model_max_length": 131072,
57
+ "tokenizer_class": "PreTrainedTokenizerFast",
58
+ "chat_template": "{{ bos_token }}{% for message in messages %}{{'<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n' + message['content'] | trim + '<|eot_id|>'}}{% endfor %}{% if add_generation_prompt %}{{'<|start_header_id|>assistant<|end_header_id|>\\n\\n'}}{% endif %}"
59
+ }