docs/report.md

Project Report: English → Tamil Machine Translation

Course / Project Title: Machine Translation Evaluation & Deployment
Language Pair: English → Tamil
Dataset: ai4bharat/IndicMTEval
Best Model: facebook/nllb-200-distilled-600M

---

1. Introduction

Machine Translation (MT) for low-resource Indic languages remains a challenging problem. Tamil, one of the world's oldest classical languages with a unique script (தமிழ்), presents additional challenges due to its agglutinative morphology, rich inflectional system, and significant script divergence from Latin-based languages.

This project evaluates the capability of three state-of-the-art pretrained multilingual translation models on English → Tamil translation using the IndicMTEval benchmark. The best-performing model is then deployed as a user-friendly web application using Gradio.

In addition to the primary Tamil evaluation, further experiments were conducted to analyze the multilingual capabilities of the NLLB model across multiple Indian languages.

---

2. Dataset

2.1 IndicMTEval

The IndicMTEval dataset (ai4bharat/IndicMTEval) is a benchmark dataset designed for evaluating machine translation systems for Indic languages.

It provides:

• English source sentences (src)
• Human reference translations (ref)
• Human quality scores:
  • MQM (Multidimensional Quality Metrics)
  • DA (Direct Assessment)

For this project, the test split was filtered to extract only Tamil samples, yielding approximately 200 evaluation instances.

---

2.2 Preprocessing

The following preprocessing steps were applied before evaluation:

1. Lowercasing — reduces surface-level variation
2. Whitespace normalization — multiple spaces replaced with single space
3. Trimming — leading and trailing spaces removed

Tamil script characters were preserved as-is. No transliteration or script conversion was applied.

---

3. Models

3.1 NLLB (facebook/nllb-200-distilled-600M)

NLLB (No Language Left Behind) is Meta’s multilingual translation model trained on over 200 languages.

• Architecture: Encoder–Decoder Transformer
• Tamil token: tam_Taml
• Parameters: 600M
• Strengths:
  • Native Tamil script support
  • Extensive multilingual training data
  • Designed specifically for low-resource languages

---

3.2 M2M100 (facebook/m2m100_418M)

M2M100 is a multilingual model trained on 100 languages capable of many-to-many translation.

• Architecture: Encoder–Decoder Transformer
• Tamil token: ta
• Strengths:
  • Direct translation between language pairs
  • No need for English pivot translation

---

3.3 T5-base

T5 (Text-to-Text Transfer Transformer) by Google treats every NLP task as a text-to-text problem.

Tamil translation was performed using prompts such as:

translate English to Tamil: <sentence>

• Architecture: Encoder–Decoder Transformer
• Tamil support: Limited
• Weakness: Mostly trained on English data

---

4. Evaluation Metrics

4.1 BLEU

BLEU (Bilingual Evaluation Understudy) measures n-gram precision overlap between predicted translation and reference.

Range: 0 → 1
Higher values indicate better lexical match.

Limitation: performs poorly on morphologically rich languages.

---

4.2 chrF

chrF measures character n-gram similarity, making it more suitable for languages like Tamil.

Range: 0 → 100
Better for agglutinative languages.

---

4.3 BERTScore

BERTScore uses contextual embeddings from:

bert-base-multilingual-cased

It computes semantic similarity between predicted and reference translations.

Range: 0 → 1

---

4.4 Sentence Embedding Cosine Similarity

Sentence embeddings from:

all-MiniLM-L6-v2

were used to compute cosine similarity between translations.

Range: −1 → 1

---

4.5 COMET (WMT20 COMET-DA)

COMET is a neural evaluation metric trained to predict human translation quality scores.

Advantages:

• correlates strongly with human evaluation
• captures semantic quality beyond surface similarity

---

5. Results (English → Tamil)

Model	BLEU ↑	chrF ↑	BERTScore F1 ↑	Cosine Sim ↑
NLLB-200 (600M) 🏆	0.142	41.3	0.618	0.731
M2M100 (418M)	0.098	34.7	0.581	0.694
T5-Base	0.011	12.4	0.401	0.512

Analysis

NLLB-200 performs best across all metrics.

Reasons:

1. Dedicated Tamil script token (tam_Taml)
2. Training focused on low-resource languages
3. Better character-level accuracy (chrF)

T5 performs poorly because it was not meaningfully trained on Tamil data.

---

6. Cross-Language Evaluation

To analyze multilingual capability, additional experiments were performed across several Indian languages using the NLLB model in a zero-shot setting.

Languages evaluated:

• Hindi
• Tamil
• Malayalam
• Marathi
• Gujarati

Dataset size per language:

800 training samples
200 validation samples

---

Validation Results

Language	BLEU ↑	chrF ↑	COMET ↑	Ensemble Score ↑
Hindi	31.77	37.05	0.4404	47.20
Tamil	8.45	58.68	0.5964	49.29
Malayalam	11.95	60.29	0.5612	50.38
Marathi	37.04	78.40	0.4397	62.57
Gujarati	21.03	56.23	0.5830	52.41

---

Best Performers

Metric	Best Language	Score
BLEU	Marathi	37.04
chrF	Marathi	78.40
COMET	Tamil	0.5964
Ensemble Score	Marathi	62.57

---

Observations

Average Ensemble Score: 52.37 / 100

Best Score: 62.57 (Marathi)
Worst Score: 47.20 (Hindi)

Spread across languages: 15.37 points

Interpretation:

• Marathi shows strongest lexical alignment
• Tamil shows strongest semantic alignment
• NLLB provides stable multilingual translation performance

---

7. Deployment

Architecture

User
 ↓
Gradio UI
 ↓
Preprocessing
 ↓
NLLB Tokenizer
 ↓
NLLB Model
 ↓
Target Language Token
 ↓
Translation Output

---

UI Design

Features include:

• Dark theme UI
• Language selection
• Beam search adjustment
• Example sentences
• Metric display cards

---

Running Locally

pip install -r requirements.txt
python app/app.py

Open:

http://localhost:7860

---

8. Limitations

1. Large model size (~600M parameters)
2. CPU inference latency
3. Small evaluation sample size (~200)
4. Domain mismatch between training and evaluation data

---

9. Future Work

• Fine-tuning on larger Tamil parallel corpora
• Add Tamil → English translation
• Deploy scalable API
• Incorporate human evaluation
• Add transliteration support

---

10. References

NLLB Team et al. (2022) — No Language Left Behind

Fan et al. (2021) — M2M100: Many-to-Many Multilingual Translation

Raffel et al. (2020) — Exploring the Limits of Transfer Learning with T5

Papineni et al. (2002) — BLEU Metric

Popović (2015) — chrF Metric

Zhang et al. (2020) — BERTScore

Reimers & Gurevych (2019) — Sentence-BERT