| # Project Report: English → Tamil Machine Translation | |
| **Course / Project Title:** Machine Translation Evaluation & Deployment | |
| **Language Pair:** English → Tamil | |
| **Dataset:** ai4bharat/IndicMTEval | |
| **Best Model:** facebook/nllb-200-distilled-600M | |
| --- | |
| # 1. Introduction | |
| Machine Translation (MT) for low-resource Indic languages remains a challenging problem. Tamil, one of the world's oldest classical languages with a unique script (`தமிழ்`), presents additional challenges due to its agglutinative morphology, rich inflectional system, and significant script divergence from Latin-based languages. | |
| This project evaluates the capability of three state-of-the-art pretrained multilingual translation models on English → Tamil translation using the IndicMTEval benchmark. The best-performing model is then deployed as a user-friendly web application using Gradio. | |
| In addition to the primary Tamil evaluation, further experiments were conducted to analyze the **multilingual capabilities of the NLLB model across multiple Indian languages**. | |
| --- | |
| # 2. Dataset | |
| ## 2.1 IndicMTEval | |
| The **IndicMTEval dataset (ai4bharat/IndicMTEval)** is a benchmark dataset designed for evaluating machine translation systems for Indic languages. | |
| It provides: | |
| - English source sentences (`src`) | |
| - Human reference translations (`ref`) | |
| - Human quality scores: | |
| - MQM (Multidimensional Quality Metrics) | |
| - DA (Direct Assessment) | |
| For this project, the **test split** was filtered to extract only Tamil samples, yielding approximately **200 evaluation instances**. | |
| --- | |
| ## 2.2 Preprocessing | |
| The following preprocessing steps were applied before evaluation: | |
| 1. **Lowercasing** — reduces surface-level variation | |
| 2. **Whitespace normalization** — multiple spaces replaced with single space | |
| 3. **Trimming** — leading and trailing spaces removed | |
| Tamil script characters were preserved as-is. No transliteration or script conversion was applied. | |
| --- | |
| # 3. Models | |
| ## 3.1 NLLB (facebook/nllb-200-distilled-600M) | |
| **NLLB (No Language Left Behind)** is Meta’s multilingual translation model trained on over **200 languages**. | |
| - Architecture: Encoder–Decoder Transformer | |
| - Tamil token: `tam_Taml` | |
| - Parameters: 600M | |
| - Strengths: | |
| - Native Tamil script support | |
| - Extensive multilingual training data | |
| - Designed specifically for low-resource languages | |
| --- | |
| ## 3.2 M2M100 (facebook/m2m100_418M) | |
| **M2M100** is a multilingual model trained on **100 languages** capable of many-to-many translation. | |
| - Architecture: Encoder–Decoder Transformer | |
| - Tamil token: `ta` | |
| - Strengths: | |
| - Direct translation between language pairs | |
| - No need for English pivot translation | |
| --- | |
| ## 3.3 T5-base | |
| **T5 (Text-to-Text Transfer Transformer)** by Google treats every NLP task as a text-to-text problem. | |
| Tamil translation was performed using prompts such as: | |
| ``` | |
| translate English to Tamil: <sentence> | |
| ``` | |
| - Architecture: Encoder–Decoder Transformer | |
| - Tamil support: Limited | |
| - Weakness: Mostly trained on English data | |
| --- | |
| # 4. Evaluation Metrics | |
| ## 4.1 BLEU | |
| BLEU (Bilingual Evaluation Understudy) measures **n-gram precision overlap** between predicted translation and reference. | |
| Range: 0 → 1 | |
| Higher values indicate better lexical match. | |
| Limitation: performs poorly on morphologically rich languages. | |
| --- | |
| ## 4.2 chrF | |
| chrF measures **character n-gram similarity**, making it more suitable for languages like Tamil. | |
| Range: 0 → 100 | |
| Better for agglutinative languages. | |
| --- | |
| ## 4.3 BERTScore | |
| BERTScore uses contextual embeddings from: | |
| ``` | |
| bert-base-multilingual-cased | |
| ``` | |
| It computes semantic similarity between predicted and reference translations. | |
| Range: 0 → 1 | |
| --- | |
| ## 4.4 Sentence Embedding Cosine Similarity | |
| Sentence embeddings from: | |
| ``` | |
| all-MiniLM-L6-v2 | |
| ``` | |
| were used to compute cosine similarity between translations. | |
| Range: −1 → 1 | |
| --- | |
| ## 4.5 COMET (WMT20 COMET-DA) | |
| COMET is a neural evaluation metric trained to predict human translation quality scores. | |
| Advantages: | |
| - correlates strongly with human evaluation | |
| - captures semantic quality beyond surface similarity | |
| --- | |
| # 5. Results (English → Tamil) | |
| | Model | BLEU ↑ | chrF ↑ | BERTScore F1 ↑ | Cosine Sim ↑ | | |
| |---|---|---|---|---| | |
| | **NLLB-200 (600M)** 🏆 | **0.142** | **41.3** | **0.618** | **0.731** | | |
| | M2M100 (418M) | 0.098 | 34.7 | 0.581 | 0.694 | | |
| | T5-Base | 0.011 | 12.4 | 0.401 | 0.512 | | |
| ## Analysis | |
| **NLLB-200 performs best across all metrics.** | |
| Reasons: | |
| 1. Dedicated Tamil script token (`tam_Taml`) | |
| 2. Training focused on low-resource languages | |
| 3. Better character-level accuracy (chrF) | |
| T5 performs poorly because it was not meaningfully trained on Tamil data. | |
| --- | |
| # 6. Cross-Language Evaluation | |
| To analyze multilingual capability, additional experiments were performed across several Indian languages using the **NLLB model in a zero-shot setting**. | |
| Languages evaluated: | |
| - Hindi | |
| - Tamil | |
| - Malayalam | |
| - Marathi | |
| - Gujarati | |
| Dataset size per language: | |
| ``` | |
| 800 training samples | |
| 200 validation samples | |
| ``` | |
| --- | |
| ## Validation Results | |
| | Language | BLEU ↑ | chrF ↑ | COMET ↑ | Ensemble Score ↑ | | |
| |---|---|---|---|---| | |
| | Hindi | 31.77 | 37.05 | 0.4404 | 47.20 | | |
| | Tamil | 8.45 | 58.68 | **0.5964** | 49.29 | | |
| | Malayalam | 11.95 | 60.29 | 0.5612 | 50.38 | | |
| | Marathi | **37.04** | **78.40** | 0.4397 | **62.57** | | |
| | Gujarati | 21.03 | 56.23 | 0.5830 | 52.41 | | |
| --- | |
| ## Best Performers | |
| | Metric | Best Language | Score | | |
| |---|---|---| | |
| | BLEU | Marathi | 37.04 | | |
| | chrF | Marathi | 78.40 | | |
| | COMET | Tamil | **0.5964** | | |
| | Ensemble Score | Marathi | **62.57** | | |
| --- | |
| ## Observations | |
| Average Ensemble Score: **52.37 / 100** | |
| Best Score: **62.57 (Marathi)** | |
| Worst Score: **47.20 (Hindi)** | |
| Spread across languages: **15.37 points** | |
| Interpretation: | |
| - Marathi shows strongest lexical alignment | |
| - Tamil shows strongest semantic alignment | |
| - NLLB provides stable multilingual translation performance | |
| --- | |
| # 7. Deployment | |
| ## Architecture | |
| ``` | |
| User | |
| ↓ | |
| Gradio UI | |
| ↓ | |
| Preprocessing | |
| ↓ | |
| NLLB Tokenizer | |
| ↓ | |
| NLLB Model | |
| ↓ | |
| Target Language Token | |
| ↓ | |
| Translation Output | |
| ``` | |
| --- | |
| ## UI Design | |
| Features include: | |
| - Dark theme UI | |
| - Language selection | |
| - Beam search adjustment | |
| - Example sentences | |
| - Metric display cards | |
| --- | |
| ## Running Locally | |
| ``` | |
| pip install -r requirements.txt | |
| python app/app.py | |
| ``` | |
| Open: | |
| ``` | |
| http://localhost:7860 | |
| ``` | |
| --- | |
| # 8. Limitations | |
| 1. Large model size (~600M parameters) | |
| 2. CPU inference latency | |
| 3. Small evaluation sample size (~200) | |
| 4. Domain mismatch between training and evaluation data | |
| --- | |
| # 9. Future Work | |
| - Fine-tuning on larger Tamil parallel corpora | |
| - Add Tamil → English translation | |
| - Deploy scalable API | |
| - Incorporate human evaluation | |
| - Add transliteration support | |
| --- | |
| # 10. References | |
| NLLB Team et al. (2022) — *No Language Left Behind* | |
| Fan et al. (2021) — *M2M100: Many-to-Many Multilingual Translation* | |
| Raffel et al. (2020) — *Exploring the Limits of Transfer Learning with T5* | |
| Papineni et al. (2002) — *BLEU Metric* | |
| Popović (2015) — *chrF Metric* | |
| Zhang et al. (2020) — *BERTScore* | |
| Reimers & Gurevych (2019) — *Sentence-BERT* |