Indic-Document-VLM

Indic-Document-VLM is an experimental multilingual Vision-Language Model for structured information extraction from document images.

It combines:

• SigLIP Base Patch16-384 as a frozen vision encoder
• a trainable two-layer visual projector
• Qwen2-0.5B as the language-model backbone
• LoRA adapters for multilingual and document-domain adaptation

The model accepts a document image and an extraction instruction, then generates a structured JSON object containing the fields visible in the document.

Supported Languages

• English
• Hindi
• Bengali
• Tamil

Supported Document Types

• Identity card
• Resident card
• Ration card
• Voter card
• Certificate
• Official notice

Architecture

Document Image
      ↓
Frozen SigLIP Base Patch16-384
      ↓
576 visual tokens × 768 dimensions
      ↓
Two-layer MLP projector
768 → 1024 → 896
      ↓
Adaptive average pooling
576 → 384 visual tokens
      ↓
Qwen2-0.5B + LoRA
      ↓
Structured JSON
Training Curriculum
Stage 1 — Indic Language Adaptation

Qwen2-0.5B was adapted using LoRA on multilingual text to strengthen its capabilities in Hindi, Bengali and Tamil.

Trainable component: LoRA adapter
Vision encoder: Not used

Stage 2 — Vision Projector Alignment

A projector was trained to map frozen SigLIP Base visual representations into Qwen's embedding space using a multilingual image-caption dataset.

Trainable component: Vision projector
Frozen components: SigLIP Base, Qwen2 base weights and Stage-1 LoRA

Stage 2B joint projector-LoRA alignment was not performed.

Stage 3 — Document Understanding

The Stage-2 projector and Stage-1 LoRA were jointly fine-tuned on multilingual synthetic document images for structured JSON extraction.

Trainable components: Vision projector and LoRA adapter
Frozen components: SigLIP Base and Qwen2 base weights

Training Dataset

Stage 3 uses:

arikatokachi/indic-synthetic-documents

The dataset contains:

2,400 synthetic document images
4 languages
6 document types
100 samples for each language-document-type pair

The dataset was divided into:

1,920 training samples
240 validation samples
240 held-out test samples

The split was stratified by language and document type.

Model Files
File	Purpose
projector.safetensors	Stage-3 visual projector
projector_config.json	Projector dimensions and pooling configuration
lora_adapter/	Stage-3 LoRA adapter
tokenizer/	Tokenizer files used during training
training_config.json	Stage-3 training hyperparameters
evaluation_results.json	Evaluation summary
architecture.json	Architecture and training-stage metadata
sample_predictions.jsonl	Example held-out test predictions

The repository does not duplicate the Qwen2 or SigLIP base weights. They must be loaded from their original Hugging Face repositories.

Loading the Artifacts
from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="arikatokachi/Indic-Document-VLM"
)

Reconstruct the model using:

google/siglip-base-patch16-384
+
projector.safetensors
+
Qwen/Qwen2-0.5B
+
lora_adapter/

The project GitHub repository will provide the complete inference implementation.

Input Format

The model was trained with prompts in the following form:

Instruction:
Extract all visible fields from the document and return valid JSON.

Answer:

The visual embeddings are prepended to the text embeddings before being passed to Qwen.

Preliminary Evaluation

Qualitative evaluation on held-out synthetic test images shows:

Strengths
Consistent valid-JSON generation
Strong document-type and schema recognition
Good extraction of recurring semantic fields
Multilingual output in English, Hindi, Bengali and Tamil
Strong performance on departments, subjects, recipients, authorities and long notice text
Known Weaknesses

The model is less reliable on high-entropy fields requiring exact character-level recognition:

Dates
Dates of birth
Identification numbers
Reference numbers
Some names and demographic fields

The model may generate a plausible value matching the expected format rather than reading the exact characters in the image.

This behaviour reflects a limitation of using:

a generic SigLIP visual encoder rather than an OCR-specialized encoder
384 × 384 document resolution
adaptive pooling from 576 to 384 visual tokens
a relatively small 0.5B language-model backbone
Intended Use

This model is intended for:

multimodal-learning research
multilingual document-understanding experiments
synthetic document extraction
educational demonstrations
prototyping structured document pipelines
Out-of-Scope Use

Do not use this model for:

official identity verification
financial or legal decision-making
government eligibility decisions
production OCR without independent verification
processing sensitive personal documents without appropriate privacy controls
Limitations
Trained primarily on synthetic document templates
Limited visual and layout diversity
Not evaluated on production government documents
Exact OCR accuracy is weaker than semantic field extraction
May hallucinate plausible values for fields it cannot read
May not generalize to handwriting, blur, occlusion or unfamiliar layouts
Does not provide confidence scores or uncertainty calibration
Ethical Considerations

The training dataset is synthetic and was designed to avoid exposing real personal identity information. However, a deployed document-understanding system can process sensitive information.

Users should:

obtain appropriate consent
minimize collection and storage
encrypt uploaded documents
avoid logging personally identifiable information
independently verify generated values
provide human review for consequential workflows
Future Work
Evaluate all 240 held-out test samples with field-level metrics
Increase document-image resolution
Preserve more visual tokens
Add region-based image tiling
Introduce OCR transcription supervision
Explore OCR-aware document encoders
Evaluate on real-world documents with privacy safeguards
Add confidence estimation and field-level uncertainty
Deploy an interactive Hugging Face Space
Citation
@misc{walia2026indicdocumentvlm,
  author = {Sanat Walia},
  title = {Indic-Document-VLM: Multilingual Structured Document Understanding},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/arikatokachi/Indic-Document-VLM}}
}
Acknowledgements

This project builds upon:

Qwen2 by the Qwen team
SigLIP by Google
Hugging Face Transformers and PEFT
AI4Bharat resources used during multilingual data preparation
Disclaimer

This is a research prototype trained primarily on synthetic data. Generated outputs may be incomplete or incorrect and must be independently verified before use.