|
Download README.md from GRRNMAKE/SymbTokenizer: direct link, hf CLI and curl.
- Browser
- Download file 2.43 kB
-
https://huggingface.co/GRRNMAKE/SymbTokenizer/resolve/main/README.md
- Command line
-
hf download hf://GRRNMAKE/SymbTokenizer/README.md
-
curl -L -o README.md https://huggingface.co/GRRNMAKE/SymbTokenizer/resolve/main/README.md
2.43 kB
| language: | |
| - en | |
| - multilingual | |
| license: other | |
| license_name: cpsl-1.0 | |
| license_link: LICENSE | |
| tags: | |
| - tokenizer | |
| - fast-tokenizer | |
| - symb | |
| - synaxim | |
| - chike | |
| - dudu | |
| # SymbTokenizer (SYNAXIM / .symb Fast Tokenizer) | |
| This repository hosts the official fast tokenizer for models running on the **SYNAXIM** runtime and native `.symb` binary container formats (including **Chíìkè**, **Dúdú**, and related agentic reasoning models). | |
| ## Specifications | |
| - **Vocabulary Size:** 128,256 | |
| - **Architecture:** Byte-Pair Encoding (BPE) | |
| - **Context Length:** Up to 131,072+ tokens | |
| - **Special Tokens:** | |
| - `<|begin_of_text|>` (ID: `128000`) — BOS token | |
| - `<|end_of_text|>` (ID: `128001`) — PAD / Stream end token | |
| - `<|start_header_id|>` (ID: `128006`) — Start of role header | |
| - `<|end_header_id|>` (ID: `128007`) — End of role header | |
| - `<|eom_id|>` (ID: `128008`) — End of message | |
| - `<|eot_id|>` (ID: `128009`) — End of turn / EOS token | |
| ## Quickstart | |
| ### 1. Using Hugging Face `transformers` | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymbTokenizer") | |
| # Tokenize text | |
| inputs = tokenizer("Hello, world!", return_tensors="pt") | |
| print(inputs) | |
| # Chat template formatting | |
| messages = [ | |
| {"role": "system", "content": "You are a helpful reasoning assistant."}, | |
| {"role": "user", "content": "Explain quantum entanglement in simple terms."} | |
| ] | |
| formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | |
| print(formatted) | |
| ``` | |
| ### 2. Using Hugging Face `tokenizers` (Fast Rust Backend) | |
| ```python | |
| from tokenizers import Tokenizer | |
| from huggingface_hub import hf_hub_download | |
| tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymbTokenizer", filename="tokenizer.json") | |
| tokenizer = Tokenizer.from_file(tokenizer_file) | |
| encoded = tokenizer.encode("Running on SYNAXIM native runtime.") | |
| print("Token IDs:", encoded.ids) | |
| print("Decoded:", tokenizer.decode(encoded.ids)) | |
| ``` | |
| ### 3. Using Native SYNAXIM / Chíìkè Inference | |
| ```python | |
| from chike_symb_inference import CozetTokenizer | |
| tokenizer = CozetTokenizer() | |
| input_ids = tokenizer.encode("Task: optimize latency.") | |
| print(input_ids) | |
| ``` | |
| --- | |
| ## License | |
| This project is licensed under the **COGNITUS PUBLIC SYMBOLIC LICENSE (CPSL) Version 1.0 — November 2026**. | |
| Copyright (c) 2026 Cognitus Lab. All Rights Reserved. | |
| See the full text in the [LICENSE](LICENSE) file. | |