--- language: - en - multilingual license: other license_name: cpsl-1.0 license_link: LICENSE tags: - tokenizer - fast-tokenizer - symb - synaxim - chike - dudu --- # SymbTokenizer (SYNAXIM / .symb Fast Tokenizer) This repository hosts the official fast tokenizer for models running on the **SYNAXIM** runtime and native `.symb` binary container formats (including **Chíìkè**, **Dúdú**, and related agentic reasoning models). ## Specifications - **Vocabulary Size:** 128,256 - **Architecture:** Byte-Pair Encoding (BPE) - **Context Length:** Up to 131,072+ tokens - **Special Tokens:** - `<|begin_of_text|>` (ID: `128000`) — BOS token - `<|end_of_text|>` (ID: `128001`) — PAD / Stream end token - `<|start_header_id|>` (ID: `128006`) — Start of role header - `<|end_header_id|>` (ID: `128007`) — End of role header - `<|eom_id|>` (ID: `128008`) — End of message - `<|eot_id|>` (ID: `128009`) — End of turn / EOS token ## Quickstart ### 1. Using Hugging Face `transformers` ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymbTokenizer") # Tokenize text inputs = tokenizer("Hello, world!", return_tensors="pt") print(inputs) # Chat template formatting messages = [ {"role": "system", "content": "You are a helpful reasoning assistant."}, {"role": "user", "content": "Explain quantum entanglement in simple terms."} ] formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) print(formatted) ``` ### 2. Using Hugging Face `tokenizers` (Fast Rust Backend) ```python from tokenizers import Tokenizer from huggingface_hub import hf_hub_download tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymbTokenizer", filename="tokenizer.json") tokenizer = Tokenizer.from_file(tokenizer_file) encoded = tokenizer.encode("Running on SYNAXIM native runtime.") print("Token IDs:", encoded.ids) print("Decoded:", tokenizer.decode(encoded.ids)) ``` ### 3. Using Native SYNAXIM / Chíìkè Inference ```python from chike_symb_inference import CozetTokenizer tokenizer = CozetTokenizer() input_ids = tokenizer.encode("Task: optimize latency.") print(input_ids) ``` --- ## License This project is licensed under the **COGNITUS PUBLIC SYMBOLIC LICENSE (CPSL) Version 1.0 — November 2026**. Copyright (c) 2026 Cognitus Lab. All Rights Reserved. See the full text in the [LICENSE](LICENSE) file.