README.md

---
language:
- as
license: mit
library_name: transformers
tags:
- assamese
- tokenizer
- bpe
- llm
- nlp
pipeline_tag: text-generation
---

# Assamese Tokenizer
# Assamese Tokenizer

A high-performance custom BPE tokenizer built specifically for Assamese Large Language Models (LLMs).

This project was created as part of a larger effort to build a fully native Assamese AI ecosystem — including datasets, tokenization pipelines, and future GPT-style language models trained primarily on Assamese text.

---

# Why I Built This

Most existing multilingual tokenizers do not properly handle Assamese.

Assamese is usually grouped together with Bengali or other Indic languages inside multilingual vocabularies. While this works at a basic level, it creates several problems:

- Poor subword segmentation
- Fragmented Assamese words
- Unnatural token boundaries
- Inefficient token compression
- Reduced language modeling quality
- Weak handling of Assamese morphology and suffix structures

Generic multilingual tokenizers are optimized for many languages simultaneously.
This tokenizer was built specifically for Assamese.

The goal is to:

- Preserve Assamese linguistic structure
- Improve token efficiency
- Reduce fragmentation
- Support large-scale Assamese language model training
- Create a tokenizer optimized for GPT-style autoregressive transformers
- Build foundational infrastructure for future Assamese AI systems

---

# Key Features

## 1. Custom Assamese BPE Vocabulary

This tokenizer uses Byte Pair Encoding (BPE) trained directly on Assamese text.

Features:

- Learns Assamese subwords automatically
- Captures common suffixes and morphemes
- Handles compound Assamese words efficiently
- Reduces vocabulary redundancy
- Improves token compression ratio

Vocabulary size:

```python
VOCAB_SIZE = 50_000
```

---

## 2. SQLite Streaming Training Pipeline

One of the most important features of this project is the streaming training architecture.

Instead of:

- loading massive text files into RAM
- generating temporary files
- requiring huge memory usage

this tokenizer streams data directly from SQLite.

Benefits:

- Extremely memory efficient
- Scales to huge datasets
- Faster dataset management
- Easier preprocessing workflows
- Better handling of terabyte-scale corpora in the future

Streaming occurs in configurable batches:

```python
BATCH_SIZE = 50_000
```

This makes the tokenizer suitable for large Assamese corpus training.

---
## 2.1 Dataset used to build this Tokenizer:
| Topic / Dataset                              | Tokens            | Approx. Scale | Source |
|----------------------------------------------|------------------:|---------------|--------|
| Poems Dataset                                | 92.6K             | 0.0000926B    | Kaggle & Sosanko Sarmah (Contributor) |
| Song Lyrics Dataset                          | 4.5M              | 0.0045B       | Kaggle (Spotify API) |
| Story Dataset                                | 52.6B             | 52.6 Billion  | HuggingFace Dataset |
| Crawled Data                                 | 7T                | 7 Trillion    | Various Web Sources |
| CC-100 Dataset                               | 5.9M              | 0.0059B       | Common Crawl |
| Qwen3 Tokens                                 | 2B                | 2 Billion     | Kaggle |
| Kaggle News Articles Dataset                 | 49.6B             | 49.6 Billion  | Kaggle |
| IndicCorp v2 (AI4Bharat)                     | 37.8T             | 37.8 Trillion | AI4Bharat Dataset |
| Assamese Monolingual Corpus (MWire-Labs)     | 38.7B             | 38.7 Billion  | MWire-Labs |
| DailyHunt Dataset                            | 184.2B            | 184.2 Billion | Rahular Varta Dataset |
| Wikipedia Dump (2019–2025)                   | 0.2T              | 200 Billion   | Wikipedia |
|                                    |         || |
| **Total**                                    | **45.3T**         | **45.333 Trillion** | |

---
## 3. Unicode Normalization for Assamese

Indic scripts often contain visually identical Unicode sequences represented differently internally.

This tokenizer applies NFC normalization:

```python
normalizers.NFC()
```

Benefits:

- Prevents token fragmentation
- Standardizes Unicode representations
- Improves vocabulary consistency
- Handles Assamese/Bengali script more reliably

---

## 4. GPT-Style Special Tokens

The tokenizer includes:

- `[UNK]`
- `[PAD]`
- `[BOS]`
- `[EOS]`

These are integrated using Hugging Face post-processing templates.

This design makes the tokenizer compatible with:

- Decoder-only transformers
- GPT-style training
- Autoregressive generation
- Custom Assamese language models

Example formatting:

```text
[BOS] Assamese sentence here [EOS]
```

---

## 5. Hugging Face Compatibility

The tokenizer is exported using:

```python
PreTrainedTokenizerFast
```

This allows direct compatibility with:

- Hugging Face Transformers
- PyTorch training pipelines
- Custom dataloaders
- AutoTokenizer
- Future Assamese LLM checkpoints

Load example:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./assamese_tokenizer")
```

---

## 6. Built-in Sanity Testing

The project includes automatic validation tests.

The tokenizer:

- Encodes Assamese sentences
- Decodes them back
- Verifies reconstruction integrity
- Displays token breakdowns

This ensures:

- Stable encoding
- Proper decoding
- Reliable tokenizer behavior

---

# Technical Architecture

## Tokenizer Type

- Algorithm: BPE (Byte Pair Encoding)
- Model Style: GPT-style autoregressive LM
- Script: Assamese (Bengali-Assamese script)

---

## Pre-tokenization Strategy

This tokenizer intentionally uses simple whitespace pre-tokenization:

```python
pre_tokenizers.Whitespace()
```

The BPE model then learns subword merges automatically.

This avoids unnecessary early fragmentation while allowing the tokenizer to learn Assamese structure naturally from data.

---

# Training Pipeline Overview

```text
SQLite Database
      ↓
Streaming Generator
      ↓
Unicode Normalization
      ↓
Whitespace Pre-tokenization
      ↓
BPE Training
      ↓
Special Token Processing
      ↓
Hugging Face Export
```

---

# Project Structure

```text
assamese_tokenizer/
│
├── tokenizer.json
├── tokenizer_config.json
└── README.md
```

---

# Generated Files

## tokenizer.json

Contains:

- Vocabulary
- Merge rules
- Decoder rules
- Normalization configuration
- Pre-tokenizer configuration
- Post-processing templates
- Special token definitions

This is the core tokenizer model.

---

## tokenizer_config.json

Contains:

- Hugging Face metadata
- Special token configuration
- Model max length
- Tokenizer settings

This enables easy loading with:

```python
AutoTokenizer.from_pretrained()
```

---

# Example Usage

## Loading the Tokenizer

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./assamese_tokenizer")
```

---

## Encoding Assamese Text

```python
text = "অসমীয়া ভাষা এটি সুন্দৰ ভাষা।"

encoded = tokenizer(text)
print(encoded["input_ids"])
```

---

## Decoding

```python
decoded = tokenizer.decode(encoded["input_ids"])
print(decoded)
```

---

# Design Philosophy

This tokenizer was not built as a generic multilingual tokenizer.

It was designed specifically for Assamese language modeling.

The focus is:

- linguistic preservation
- scalable infrastructure
- efficient training
- Assamese-first optimization
- future LLM compatibility

The long-term vision is to help build fully native Assamese AI systems.

---

# Future Goals

Planned improvements include:

- Larger Assamese corpora training
- Improved punctuation handling
- Advanced normalization research
- Token compression benchmarking
- Comparison against multilingual tokenizers
- Native Assamese conversational models
- Public Hugging Face release
- Integration with custom Assamese GPT models

---

# Author

Ranjit Das

Developer focused on Assamese AI infrastructure, tokenization systems, large-scale datasets, and native language model development.

---

# License

This project is intended for research and educational purposes. And it is free to all user including commercial uses