Supernova Nepali Tokenizer V3 (Absolute Power)
Supernova V3 is a scratch-built tokenizer infrastructure that achieves 7x the efficiency and speed of generic tokenizers (like Tiktoken or HuggingFace Fast Tokenizers) for the Nepali language.
Why to use this model?
- Speed: 1.37x+ faster than older one on standard CPUs.
- Power: 7x more efficient character-to-token representation for Devanagari.then older
- No GPU Needed: Designed for high-speed inference on edge devices and standard
How to use
# Load the tokenizer_v3.json and use the encode/decode logic provided in the Supernova class.
Status: Shipped. The era of GPU-dependency for Nepali NLP ends here. Supernova Nepali Tokenizer V3
A lightweight, CPU-oriented Nepali tokenizer using a custom Trie-based Max-Match Priority Encoder.
Installation
pip install huggingface_hub
Download
from huggingface_hub import hf_hub_download
tokenizer_path = hf_hub_download( repo_id="Supernova11c/Supernova-Nepali-Tokenizer-V3", filename="tokenizer.json" )
print("Tokenizer:", tokenizer_path)
Important
This repository contains a custom tokenizer format.
It is not a standard "AutoTokenizer" repository.
The "tokenizer.json" file contains the tokenizer vocabulary, merge definitions, and architecture metadata.
Architecture
- Architecture: Trie-based Max-Match Priority Encoder
- Version: 3.0.0
- Vocabulary size: 348
- Merge count: 92
- Designed for CPU inference
- Focus: Nepali and mixed Nepali/English text
Loading
import json
with open(tokenizer_path, "r", encoding="utf-8") as f: config = json.load(f)
print(config["model_name"]) print(config["version"]) print(config["architecture"]) print("Vocabulary:", len(config["vocab"])) print("Merges:", len(config["merges"]))
Benchmarking
For reproducible comparisons, benchmark all tokenizers on the same corpus and CPU.
Recommended metrics:
- Mean latency
- Median latency
- Best latency
- Characters/second
- Bytes/second
- Tokens/second
- Tokens/character
- Round-trip reconstruction accuracy
- Memory usage
Do not compare token counts alone because different tokenizers can represent the same text using very different numbers of tokens.
Benchmark claim
Benchmark results should always include:
- hardware/CPU information
- dataset size
- dataset composition
- number of repetitions
- warm-up procedure
- tokenizer versions
- exact benchmark code
Performance claims should be reported from measured results rather than assumed architectural advantages.