NyayaLM Tokenizer

SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, bilingual (Nepali/English) legal-domain model with Qwen3-style ChatML special tokens.

BPE + byte_fallback
Character coverage: 0.9995
Curated specials at high ids (not colliding with BPE pieces)

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("REPO_ID")
print(tok.apply_chat_template(
    [{"role": "user", "content": "नमस्ते"}],
    tokenize=False, add_generation_prompt=True,
))

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support