SymbTokenizer (SYNAXIM / .symb Fast Tokenizer)

This repository hosts the official fast tokenizer for models running on the SYNAXIM runtime and native .symb binary container formats (including Chíìkè, Dúdú, and related agentic reasoning models).

Specifications

  • Vocabulary Size: 128,256
  • Architecture: Byte-Pair Encoding (BPE)
  • Context Length: Up to 131,072+ tokens
  • Special Tokens:
    • <|begin_of_text|> (ID: 128000) — BOS token
    • <|end_of_text|> (ID: 128001) — PAD / Stream end token
    • <|start_header_id|> (ID: 128006) — Start of role header
    • <|end_header_id|> (ID: 128007) — End of role header
    • <|eom_id|> (ID: 128008) — End of message
    • <|eot_id|> (ID: 128009) — End of turn / EOS token

Quickstart

1. Using Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymbTokenizer")

# Tokenize text
inputs = tokenizer("Hello, world!", return_tensors="pt")
print(inputs)

# Chat template formatting
messages = [
    {"role": "system", "content": "You are a helpful reasoning assistant."},
    {"role": "user", "content": "Explain quantum entanglement in simple terms."}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted)

2. Using Hugging Face tokenizers (Fast Rust Backend)

from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymbTokenizer", filename="tokenizer.json")
tokenizer = Tokenizer.from_file(tokenizer_file)

encoded = tokenizer.encode("Running on SYNAXIM native runtime.")
print("Token IDs:", encoded.ids)
print("Decoded:", tokenizer.decode(encoded.ids))

3. Using Native SYNAXIM / Chíìkè Inference

from chike_symb_inference import CozetTokenizer

tokenizer = CozetTokenizer()
input_ids = tokenizer.encode("Task: optimize latency.")
print(input_ids)

License

This project is licensed under the COGNITUS PUBLIC SYMBOLIC LICENSE (CPSL) Version 1.0 — November 2026.
Copyright (c) 2026 Cognitus Lab. All Rights Reserved.

See the full text in the LICENSE file.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support