SymbTokenizer (SYNAXIM / .symb Fast Tokenizer)
This repository hosts the official fast tokenizer for models running on the SYNAXIM runtime and native .symb binary container formats (including Chíìkè, Dúdú, and related agentic reasoning models).
Specifications
- Vocabulary Size: 128,256
- Architecture: Byte-Pair Encoding (BPE)
- Context Length: Up to 131,072+ tokens
- Special Tokens:
<|begin_of_text|>(ID:128000) — BOS token<|end_of_text|>(ID:128001) — PAD / Stream end token<|start_header_id|>(ID:128006) — Start of role header<|end_header_id|>(ID:128007) — End of role header<|eom_id|>(ID:128008) — End of message<|eot_id|>(ID:128009) — End of turn / EOS token
Quickstart
1. Using Hugging Face transformers
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymbTokenizer")
# Tokenize text
inputs = tokenizer("Hello, world!", return_tensors="pt")
print(inputs)
# Chat template formatting
messages = [
{"role": "system", "content": "You are a helpful reasoning assistant."},
{"role": "user", "content": "Explain quantum entanglement in simple terms."}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted)
2. Using Hugging Face tokenizers (Fast Rust Backend)
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymbTokenizer", filename="tokenizer.json")
tokenizer = Tokenizer.from_file(tokenizer_file)
encoded = tokenizer.encode("Running on SYNAXIM native runtime.")
print("Token IDs:", encoded.ids)
print("Decoded:", tokenizer.decode(encoded.ids))
3. Using Native SYNAXIM / Chíìkè Inference
from chike_symb_inference import CozetTokenizer
tokenizer = CozetTokenizer()
input_ids = tokenizer.encode("Task: optimize latency.")
print(input_ids)
License
This project is licensed under the COGNITUS PUBLIC SYMBOLIC LICENSE (CPSL) Version 1.0 — November 2026.
Copyright (c) 2026 Cognitus Lab. All Rights Reserved.
See the full text in the LICENSE file.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support