NyayaLM Tokenizer

SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, a bilingual (Nepali/English) legal-domain model with a Qwen3-style architecture and ChatML-style special tokens.

Model type: BPE, byte-fallback enabled (never emits <unk>)
Character coverage: 0.9995
Curated special tokens: 17
Reserved tokens for future use: 32

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("REPO_ID")
tok.apply_chat_template(
    [{'role': "user", "content": "नमस्ते"}],
    tokenize=False, add_generation_prompt=True,
)

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support