NyayaLM 64K Tokenizer

Domain-specific bilingual (Nepali + English) SentencePiece-BPE tokenizer for the NyayaLM-200M legal-domain language model.

Vocab size: 64 000
Algorithm: SentencePiece BPE + byte_fallback
Special tokens: Qwen3 / ChatML style (high IDs)
EOS: <|im_end|>
PAD: <|endoftext|>
Chat template: Qwen3-compatible

Quick use

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("YOUR_HF_REPO")
print(tok.apply_chat_template(
    [{'role': "user", "content": "नेपालको संविधान के हो?"}],
    tokenize=False,
    add_generation_prompt=True,
))

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support