README.md

---
language:
- asm  # Assamese ISO 639-1 code
license: apache-2.0
base_model: microsoft/Florence-2-large-ft
tags:
- vision
- ocr
- assamese
- northeast-india
- indic-languages
- character-recognition
- florence-2
- vision-language
datasets:
- darknight054/indic-mozhi-ocr
metrics:
- accuracy
- character_error_rate
library_name: transformers
pipeline_tag: image-to-text

model-index:
- name: AssameseOCR
  results:
  - task:
      type: image-to-text
      name: Optical Character Recognition
    dataset:
      name: Mozhi Indic OCR (Assamese)
      type: darknight054/indic-mozhi-ocr
      config: assamese
      split: test
    metrics:
    - type: accuracy
      value: 94.67
      name: Character Accuracy
      verified: false
    - type: character_error_rate
      value: 5.33
      name: Character Error Rate (CER)
      verified: false

---

# AssameseOCR

**AssameseOCR** is a vision-language model for Optical Character Recognition (OCR) of printed Assamese text. Built on Microsoft's Florence-2-large foundation model with a custom character-level decoder, it achieves 94.67% character accuracy on the Mozhi dataset.

## Model Details

### Model Description

- **Developed by:** MWire Labs
- **Model type:** Vision-Language OCR
- **Language:** Assamese (অসমীয়া)
- **License:** Apache 2.0
- **Base Model:** microsoft/Florence-2-large-ft
- **Architecture:** Florence-2 Vision Encoder + Custom Transformer Decoder

### Model Architecture

```
Image (768×768) 
  ↓
Florence-2 Vision Encoder (frozen, 360M params)
  ↓
Vision Projection (1024 → 512 dim)
  ↓
Transformer Decoder (4 layers, 8 heads)
  ↓
Character-level predictions (187 vocab)
```

**Key Components:**
- **Vision Encoder:** Florence-2-large DaViT architecture (frozen)
- **Decoder:** 4-layer Transformer with 512 hidden dimensions
- **Tokenizer:** Character-level with 187 tokens (Assamese chars + English + digits + symbols)
- **Total Parameters:** 378M (361M frozen, 17.5M trainable)

## Training Details

### Training Data

- **Dataset:** [Mozhi Indic OCR Dataset](https://huggingface.co/datasets/darknight054/indic-mozhi-ocr) (Assamese subset)
- **Training samples:** 79,697 word images
- **Validation samples:** 9,945 word images
- **Test samples:** 10,146 word images
- **Source:** IIT Hyderabad CVIT

### Training Procedure

**Hardware:**
- GPU: NVIDIA A40 (48GB VRAM)
- Training time: ~8 hours (3 epochs)

**Hyperparameters:**
- Epochs: 3
- Batch size: 16
- Learning rate: 3e-4
- Optimizer: AdamW (weight_decay=0.01)
- Scheduler: CosineAnnealingLR
- Max sequence length: 128 characters
- Gradient clipping: 1.0

**Training Strategy:**
- Froze Florence-2 vision encoder (leveraging pretrained visual features)
- Trained only the projection layer and transformer decoder
- Full fine-tuning (no LoRA) for maximum quality

## Performance

### Results

| Split | Character Accuracy | Loss |
|-------|-------------------|------|
| Epoch 1 (Val) | 91.61% | 0.2844 |
| Epoch 2 (Val) | 94.09% | 0.1548 |
| Epoch 3 (Val) | **94.67%** | **0.1221** |

**Character Error Rate (CER):** ~5.33%

### Comparison

The model achieves strong performance for a foundation model approach:
- Mozhi paper (CRNN+CTC specialist): ~99% accuracy
- AssameseOCR (Florence generalist): 94.67% accuracy

The 5% gap is expected when adapting a general vision-language model versus training a specialized OCR architecture. However, AssameseOCR offers:
- Extensibility to vision-language tasks (VQA, captioning, document understanding)
- Faster training (3 epochs vs typical 10-20 for CRNN)
- Foundation model benefits (transfer learning, robustness)

## Usage

### Installation

```bash
pip install torch torchvision transformers pillow
```

### Inference

```python
import torch
import torch.nn as nn
from PIL import Image
from transformers import AutoModelForCausalLM, CLIPImageProcessor
from huggingface_hub import hf_hub_download
import json

# CharTokenizer class
class CharTokenizer:
    def __init__(self, vocab):
        self.vocab = vocab
        self.char2id = {c: i for i, c in enumerate(vocab)}
        self.id2char = {i: c for i, c in enumerate(vocab)}
        self.pad_token_id = self.char2id["<pad>"]
        self.bos_token_id = self.char2id["<s>"]
        self.eos_token_id = self.char2id["</s>"]
        
    def encode(self, text, max_length=None, add_special_tokens=True):
        ids = [self.bos_token_id] if add_special_tokens else []
        for ch in text:
            ids.append(self.char2id.get(ch, self.char2id["<unk>"]))
        if add_special_tokens:
            ids.append(self.eos_token_id)
        if max_length:
            ids = ids[:max_length]
            if len(ids) < max_length:
                ids += [self.pad_token_id] * (max_length - len(ids))
        return ids
        
    def decode(self, ids, skip_special_tokens=True):
        chars = []
        for i in ids:
            ch = self.id2char.get(i, "")
            if skip_special_tokens and ch.startswith("<"):
                continue
            chars.append(ch)
        return "".join(chars)
    
    @classmethod
    def load(cls, path):
        with open(path, "r", encoding="utf-8") as f:
            vocab = json.load(f)
        return cls(vocab)

# FlorenceCharOCR model class
class FlorenceCharOCR(nn.Module):
    def __init__(self, florence_model, vocab_size, vision_hidden_dim, decoder_hidden_dim=512, num_layers=4):
        super().__init__()
        self.florence_model = florence_model
        
        for param in self.florence_model.parameters():
            param.requires_grad = False
        
        self.vision_proj = nn.Linear(vision_hidden_dim, decoder_hidden_dim)
        self.embedding = nn.Embedding(vocab_size, decoder_hidden_dim)
        decoder_layer = nn.TransformerDecoderLayer(
            d_model=decoder_hidden_dim, 
            nhead=8, 
            batch_first=True
        )
        self.decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
        self.fc_out = nn.Linear(decoder_hidden_dim, vocab_size)
        
    def forward(self, pixel_values, tgt_ids, tgt_mask=None):
        with torch.no_grad():
            vision_feats = self.florence_model._encode_image(pixel_values)
        
        vision_feats = self.vision_proj(vision_feats)
        tgt_emb = self.embedding(tgt_ids)
        decoder_out = self.decoder(tgt_emb, vision_feats, tgt_mask=tgt_mask)
        logits = self.fc_out(decoder_out)
        
        return logits

# Load components
device = "cuda" if torch.cuda.is_available() else "cpu"

# Download files from HuggingFace
tokenizer_path = hf_hub_download(repo_id="MWirelabs/assamese-ocr", filename="assamese_char_tokenizer.json")
model_path = hf_hub_download(repo_id="MWirelabs/assamese-ocr", filename="assamese_ocr_best.pt")

# Load tokenizer
char_tokenizer = CharTokenizer.load(tokenizer_path)

# Load Florence base model
florence_model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Florence-2-large-ft",
    trust_remote_code=True
).to(device)

# Load image processor
image_processor = CLIPImageProcessor.from_pretrained("microsoft/Florence-2-large-ft")

# Initialize OCR model
ocr_model = FlorenceCharOCR(
    florence_model=florence_model,
    vocab_size=len(char_tokenizer.vocab),
    vision_hidden_dim=1024,
    decoder_hidden_dim=512,
    num_layers=4
).to(device)

# Load trained weights
checkpoint = torch.load(model_path, map_location=device)
ocr_model.load_state_dict(checkpoint['model_state_dict'])
ocr_model.eval()

# Inference function
def recognize_text(image_path):
    # Load and process image
    image = Image.open(image_path).convert("RGB")
    pixel_values = image_processor(images=[image], return_tensors="pt")['pixel_values'].to(device)
    
    # Generate prediction
    with torch.no_grad():
        # Start with BOS token
        generated_ids = [char_tokenizer.bos_token_id]
        
        for _ in range(128):  # max length
            tgt_tensor = torch.tensor([generated_ids], device=device)
            logits = ocr_model(pixel_values, tgt_tensor)
            
            # Get next token
            next_token = logits[0, -1].argmax().item()
            generated_ids.append(next_token)
            
            # Stop if EOS
            if next_token == char_tokenizer.eos_token_id:
                break
    
    # Decode
    text = char_tokenizer.decode(generated_ids, skip_special_tokens=True)
    return text

# Example usage
result = recognize_text("assamese_text.jpg")
print(f"Recognized text: {result}")
```

## Vocabulary

The character-level tokenizer includes:
- **Assamese characters:** 119 unique chars (consonants, vowels, diacritics, conjuncts)
- **English:** 52 chars (a-z, A-Z)
- **Digits:** 30 chars (ASCII 0-9, Assamese ০-৯, Devanagari ०-९)
- **Symbols:** 33 chars (punctuation, special chars)
- **Special tokens:** 6 tokens (`<pad>`, `<s>`, `</s>`, `<unk>`, `<OCR>`, `<lang_as>`)
- **Total vocabulary:** 187 tokens

## Limitations

- Trained only on printed text (not handwritten)
- Word-level images from Mozhi dataset (may not generalize to full-page OCR without line segmentation)
- Character-level decoder may struggle with very long sequences (>128 chars)
- Does not handle layout analysis or reading order
- Performance on degraded/low-quality images not extensively tested

## Future Work

- Extend to **MeiteiOCR** for Meitei Mayek script
- Scale to **NE-OCR** covering all 9+ Northeast Indian languages
- Add document layout analysis and reading order detection
- Improve performance with synthetic data augmentation
- Fine-tune for handwritten text recognition
- Extend to multimodal tasks (image captioning, VQA for documents)

## Citation

If you use AssameseOCR in your research, please cite:

```bibtex
@software{assameseocr2026,
  author = {MWire Labs},
  title = {AssameseOCR: Vision-Language Model for Assamese Text Recognition},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/MWirelabs/assamese-ocr}
}
```

## Acknowledgments

- **Dataset:** Mozhi Indic OCR Dataset by IIT Hyderabad CVIT ([Mathew et al., 2022](https://arxiv.org/abs/2205.06740))
- **Base Model:** Florence-2 by Microsoft Research
- **Organization:** MWire Labs, Shillong, Meghalaya, India

## Contact

- **Organization:** [MWire Labs](https://huggingface.co/MWirelabs)
- **Location:** Shillong, Meghalaya, India
- **Focus:** Language technology for Northeast Indian languages

---

**Part of the MWire Labs NLP suite:**
- [KhasiBERT](https://huggingface.co/MWirelabs/KhasiBERT-110M) - Khasi language model
- [NE-BERT](https://huggingface.co/MWirelabs/NE-BERT) - 9 Northeast languages
- [Kren-M](https://huggingface.co/MWirelabs/Kren-M) - Khasi-English conversational AI
- **AssameseOCR** - Assamese text recognition