SymbTokenizer / README.md
GRRNMAKER's picture
Update repository name references to GRRNMAKE/SymbTokenizer
a97b9ca verified
|
Raw History Blame Contribute Delete
2.43 kB
---
language:
- en
- multilingual
license: other
license_name: cpsl-1.0
license_link: LICENSE
tags:
- tokenizer
- fast-tokenizer
- symb
- synaxim
- chike
- dudu
---
# SymbTokenizer (SYNAXIM / .symb Fast Tokenizer)
This repository hosts the official fast tokenizer for models running on the **SYNAXIM** runtime and native `.symb` binary container formats (including **Chíìkè**, **Dúdú**, and related agentic reasoning models).
## Specifications
- **Vocabulary Size:** 128,256
- **Architecture:** Byte-Pair Encoding (BPE)
- **Context Length:** Up to 131,072+ tokens
- **Special Tokens:**
- `<|begin_of_text|>` (ID: `128000`) — BOS token
- `<|end_of_text|>` (ID: `128001`) — PAD / Stream end token
- `<|start_header_id|>` (ID: `128006`) — Start of role header
- `<|end_header_id|>` (ID: `128007`) — End of role header
- `<|eom_id|>` (ID: `128008`) — End of message
- `<|eot_id|>` (ID: `128009`) — End of turn / EOS token
## Quickstart
### 1. Using Hugging Face `transformers`
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("GRRNMAKE/SymbTokenizer")
# Tokenize text
inputs = tokenizer("Hello, world!", return_tensors="pt")
print(inputs)
# Chat template formatting
messages = [
{"role": "system", "content": "You are a helpful reasoning assistant."},
{"role": "user", "content": "Explain quantum entanglement in simple terms."}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted)
```
### 2. Using Hugging Face `tokenizers` (Fast Rust Backend)
```python
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
tokenizer_file = hf_hub_download(repo_id="GRRNMAKE/SymbTokenizer", filename="tokenizer.json")
tokenizer = Tokenizer.from_file(tokenizer_file)
encoded = tokenizer.encode("Running on SYNAXIM native runtime.")
print("Token IDs:", encoded.ids)
print("Decoded:", tokenizer.decode(encoded.ids))
```
### 3. Using Native SYNAXIM / Chíìkè Inference
```python
from chike_symb_inference import CozetTokenizer
tokenizer = CozetTokenizer()
input_ids = tokenizer.encode("Task: optimize latency.")
print(input_ids)
```
---
## License
This project is licensed under the **COGNITUS PUBLIC SYMBOLIC LICENSE (CPSL) Version 1.0 — November 2026**.
Copyright (c) 2026 Cognitus Lab. All Rights Reserved.
See the full text in the [LICENSE](LICENSE) file.