sayurio/bn-tokenizer-32k

Bengali / Bangla Tokenizer 32k

sayurio/bn-tokenizer-32k is a high-efficiency Byte-Pair Encoding (BPE) tokenizer with a 32,768 vocabulary size, engineered specifically for native Bengali Natural Language Processing, Large Language Models (LLMs), and Vision-Language Document OCR architectures.

Unlike multilingual tokenizers (such as LLaMA, Mistral, or Qwen) that split Bengali words and conjuncts into multiple sub-byte fragments, bn-tokenizer-32k treats compound grapheme clusters and common roots as unified tokens. This yields an average compression ratio of ~8.00 characters per token on formal Bengali text — at half the embedding-table footprint of a 64k vocabulary.

---

Key Features

• High Compression Rate: Compresses Bengali text down to ~8.00 characters per token, reducing sequence lengths by up to 60–70% compared to standard multilingual subword tokenizers.
• Compact Vocabulary (32,768): Halves the embedding and output-projection parameters relative to a 64k tokenizer, while remaining large enough to keep Bengali roots, suffixes, and conjuncts as single tokens.
• Compound Conjunct (যুক্তবর্ণ) Preservation: Accurately unifies complex ligatures (যেমন: ক্ষ, জ্ঞ, ঞ্চ, ণ্ড, ক্ত, ত্র, ঙ্গ) and multi-syllable roots (শিক্ষাক্রম, পাঠ্যপুস্তক, বুদ্ধিমত্তা) into single token representations.
• Canonical NFC Normalization: Strictly enforces Unicode Normalization Form C (NFC) to eliminate split vowel diacritics (e.g., decomposing ো into ে + া).
• Metaspace Word Boundary Tracking: Uses Metaspace (▁) pre-tokenization and decoding for 100% lossless text reconstruction with exact whitespace preservation.
• Invisible Character Handling: Retains Zero-Width Non-Joiner (\u200c) and Zero-Width Joiner (\u200d) to ensure correct rendering of Bengali spelling rules.

---

Tokenizer Specifications

Parameter	Value
Model Type	Byte-Pair Encoding (BPE)
Vocabulary Size	32,768
Pre-Tokenizer	Metaspace (Replacement: ▁, Prepend: always)
Decoder	Metaspace (Replacement: ▁, Prepend: always)
Normalizer	Unicode NFC + Strip
Training Corpus	~1.11 GB clean text (500,000+ deduplicated sentences)
Corpus Domains	Bengali Wikipedia, AI4Bharat Sangraha, National Journalism
Min Token Frequency	3

Special Tokens

Token	ID	Role
<pad>	0	Padding token
<s>	1	Beginning of sequence (BOS)
</s>	2	End of sequence (EOS)
<unk>	3	Unknown token

---

Compression & Tokenization Benchmarks

Example 1: Formal & Academic Text

• Input Text: স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড
• Tokens: ['<s>', ' স্বাধীন', ' বাংলাদেশের', ' জাতীয়', ' শিক্ষাক্রম', ' ও', ' পাঠ্যপুস্তক', ' বোর্ড', '</s>']
• Token IDs: [1, 1223, 830, 688, 12508, 156, 13480, 2455, 2]
• Efficiency: 56 characters $\rightarrow$ 7 content tokens (8.00 chars/token)

Example 2: Scientific & Technical Text

• Input Text: বিজ্ঞান ও প্রযুক্তির উৎকর্ষে কম্পিউটার প্রোগ্রামিং এবং কৃত্রিম বুদ্ধিমত্তা অত্যন্ত গুরুত্বপূর্ণ।
• Tokens: ['<s>', ' বিজ্ঞান', ' ও', ' প্রযুক্তির', ' উৎকর্ষ', 'ে', ' কম্পিউটার', ' প্রোগ্রামিং', ' এবং', ' কৃত্রিম', ' বুদ্ধিমত্তা', ' অত্যন্ত', ' গুরুত্বপূর্ণ।', '</s>']
• Token IDs: [1, 1959, 156, 7613, 10092, 71, 3209, 11838, 187, 7270, 13209, 2747, 8446, 2]
• Efficiency: 96 characters $\rightarrow$ 12 content tokens (8.00 chars/token)

---

Quickstart

Using Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sayurio/bn-tokenizer-32k")

text = "স্বাধীন বাংলাদেশের জাতীয় শিক্ষাক্রম ও পাঠ্যপুস্তক বোর্ড"

# Encode
encoded = tokenizer(text)
tokens = tokenizer.convert_ids_to_tokens(encoded["input_ids"])

print("Tokens    :", tokens)
print("Token IDs :", encoded["input_ids"])

# Decode
decoded = tokenizer.decode(encoded["input_ids"], skip_special_tokens=True)
print("Decoded   :", decoded)

Using Hugging Face tokenizers (Fast Rust Backend)

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("sayurio/bn-tokenizer-32k")

output = tokenizer.encode("স্বাধীন বাংলাদেশ")
print("Tokens :", output.tokens)
print("IDs    :", output.ids)

Training Methodology

The tokenizer was trained on a multi-domain corpus of clean Bengali text comprising:

• Academic & Encyclopedic Prose: Extracted from Bengali Wikipedia dump.
• Formal & Legal Records: Derived from AI4Bharat's curated Bengali resources.
• Contemporary Regional News: Modern journalistic and colloquial usage across politics, economics, and culture.

All text underwent strict pre-filtering:

• Removal of foreign scripts, stray HTML/Markdown tags, and corrupted byte markers.
• Retention strictly of the Bengali Unicode block (\u0980–\u09FF), Bengali numerals (০-৯), and standard punctuation.
• Unicode NFC canonical composition applied before pair ranking.

License This tokenizer is released under the Apache 2.0 License. You are free to use, modify, and distribute it for both academic research and commercial applications.