| ---
|
| title: English → Indian Languages Machine Translation
|
| emoji: 🌐
|
| colorFrom: green
|
| colorTo: purple
|
| sdk: gradio
|
| sdk_version: 6.12.0
|
| app_file: app.py
|
| pinned: false
|
| ---
|
|
|
| # 🌐 English → Indian Languages Machine Translation
|
|
|
| A complete machine translation pipeline for **English → Indian languages** using the **NLLB-200 multilingual model**, evaluated with multiple automatic metrics and deployed via an interactive **Gradio web application**.
|
|
|
| ---
|
|
|
| # 📋 Table of Contents
|
|
|
| - Overview
|
| - Languages Supported
|
| - Dataset
|
| - Models Evaluated
|
| - Evaluation Results
|
| - Cross-Language Evaluation
|
| - Metric Descriptions
|
| - Multilingual Extension Task
|
| - Project Structure
|
| - Quick Start
|
| - Running the App
|
| - Running Evaluation
|
| - Deployment Architecture
|
| - Report
|
|
|
| ---
|
|
|
| # Overview
|
|
|
| This project evaluates multiple pretrained multilingual translation models on **English → Tamil** (primary evaluation) and demonstrates multilingual capability across multiple Indian languages using the best performing model.
|
|
|
| The best performing model was:
|
|
|
| ```
|
| facebook/nllb-200-distilled-600M
|
| ```
|
|
|
| This model supports **200+ languages**, allowing the application to translate English sentences into multiple Indian languages using a single model.
|
|
|
| Key observation:
|
|
|
| - NLLB-200 significantly outperforms **M2M100** and **T5-Base** for English → Tamil translation.
|
| - The same model can also perform **zero-shot translation** for other Indian languages.
|
|
|
| ---
|
|
|
| # Languages Supported
|
|
|
| | Language | Script | NLLB Token | BERTScore Lang |
|
| |--------|--------|------------|---------------|
|
| | Tamil | தமிழ் | `tam_Taml` | ta |
|
| | Hindi | हिन्दी | `hin_Deva` | hi |
|
| | Telugu | తెలుగు | `tel_Telu` | te |
|
| | Kannada | ಕನ್ನಡ | `kan_Knda` | kn |
|
| | Malayalam | മലയാളം | `mal_Mlym` | ml |
|
|
|
| Switching languages in the application **does not reload the model**.
|
| Only the **target language token (`forced_bos_token_id`)** changes.
|
|
|
| ---
|
|
|
| # Dataset
|
|
|
| | Property | Value |
|
| |---|---|
|
| | Dataset | ai4bharat/IndicMTEval |
|
| | Primary evaluation language | Tamil |
|
| | Additional demo languages | Hindi, Telugu, Kannada, Malayalam |
|
| | Split used | test |
|
| | Samples per language | up to 200 |
|
|
|
| Tamil was selected as the **primary benchmark language** because IndicMTEval provides **human evaluation scores (MQM / Direct Assessment)** for Tamil translations.
|
| ### URL:
|
| https://huggingface.co/datasets/ai4bharat/IndicMTEval
|
| ---
|
|
|
| # Preprocessing Applied
|
|
|
| - Lowercasing
|
| - Removing extra whitespace
|
| - Trimming leading and trailing spaces
|
|
|
| ---
|
|
|
| # Models Evaluated
|
|
|
| The following translation models were evaluated for **English → Tamil**:
|
|
|
| | Model | Parameters | Architecture | Tamil Token |
|
| |------|------------|-------------|------------|
|
| | `facebook/nllb-200-distilled-600M` | 600M | Encoder-Decoder (NLLB) | tam_Taml |
|
| | `facebook/m2m100_418M` | 418M | Encoder-Decoder (M2M100) | ta |
|
| | `t5-base` | 220M | Encoder-Decoder (T5) | prompt-based |
|
|
|
| Additional semantic evaluation was performed using:
|
|
|
| | Evaluation Model | Purpose |
|
| |---|---|
|
| | `WMT20 COMET-DA` | Neural MT evaluation metric |
|
| | `all-MiniLM-L6-v2` | Sentence embedding similarity |
|
|
|
| ---
|
|
|
| # Evaluation Results
|
|
|
| ### Primary Model Comparison (English → Tamil)
|
|
|
| | Model | BLEU ↑ | chrF ↑ | BERTScore F1 ↑ | Cosine Sim ↑ |
|
| |------|------|------|------|------|
|
| | **NLLB-200 (600M)** 🏆 | **0.142** | **41.3** | **0.618** | **0.731** |
|
| | M2M100 (418M) | 0.098 | 34.7 | 0.581 | 0.694 |
|
| | T5-Base | 0.011 | 12.4 | 0.401 | 0.512 |
|
|
|
| NLLB-200 clearly performs best across all evaluation metrics.
|
|
|
| ---
|
|
|
| # Cross-Language Evaluation (Indic MT Benchmark)
|
|
|
| To analyze multilingual performance, the **NLLB-200 model** was evaluated across several Indian languages using:
|
|
|
| - BLEU
|
| - chrF
|
| - COMET (WMT20 COMET-DA)
|
| - Sentence embedding cosine similarity
|
|
|
| Each language used:
|
|
|
| ```
|
| 800 training samples
|
| 200 validation samples
|
| ```
|
|
|
| ---
|
|
|
| ### Validation Results
|
|
|
| | Language | BLEU ↑ | chrF ↑ | COMET ↑ | Ensemble Score ↑ |
|
| |------|------|------|------|------|
|
| | Hindi | 31.77 | 37.05 | 0.4404 | 47.20 |
|
| | Tamil | 8.45 | 58.68 | **0.5964** | 49.29 |
|
| | Malayalam | 11.95 | 60.29 | 0.5612 | 50.38 |
|
| | Marathi | **37.04** | **78.40** | 0.4397 | **62.57** |
|
| | Gujarati | 21.03 | 56.23 | 0.5830 | 52.41 |
|
|
|
| ---
|
|
|
| ### Best Performers
|
|
|
| | Metric | Best Language | Score |
|
| |------|------|------|
|
| | BLEU | Marathi | 37.04 |
|
| | chrF | Marathi | 78.40 |
|
| | COMET | Tamil | **0.5964** |
|
| | Ensemble Score | Marathi | **62.57** |
|
|
|
| ---
|
|
|
| ### Evaluation Insights
|
|
|
| - Marathi shows strong lexical alignment with the reference translations.
|
| - Tamil achieves the **highest COMET score**, indicating strong semantic alignment.
|
| - Overall multilingual performance is stable across languages.
|
|
|
| ```
|
| Average Ensemble Score: 52.37 / 100
|
| Best Ensemble Score: 62.57 (Marathi)
|
| Worst Ensemble Score: 47.20 (Hindi)
|
| Score Spread: 15.37 points
|
| ```
|
|
|
| All experiments were performed using the **pretrained NLLB-200 model in a zero-shot translation setting**.
|
|
|
| ---
|
|
|
| # Metric Descriptions
|
|
|
| BLEU
|
| Measures n-gram overlap between predicted translation and reference translation.
|
|
|
| chrF
|
| Character-level F-score suited for morphologically rich languages.
|
|
|
| BERTScore
|
| Uses contextual embeddings to measure semantic similarity.
|
|
|
| COMET
|
| A neural metric trained to predict human translation quality judgments.
|
|
|
| Cosine Similarity
|
| Measures similarity between sentence embeddings of predicted and reference translations.
|
|
|
| ---
|
|
|
| # Multilingual Extension Task
|
|
|
| Since the NLLB model supports **200+ languages**, the application was extended to support translation into multiple Indian languages using the same model.
|
|
|
| Supported translations:
|
|
|
| - English → Tamil
|
| - English → Hindi
|
| - English → Telugu
|
| - English → Kannada
|
| - English → Malayalam
|
|
|
| Instead of loading multiple models, the system simply changes the **target language token**:
|
|
|
| ```
|
| model.generate(
|
| **inputs,
|
| forced_bos_token_id=tokenizer.convert_tokens_to_ids("tam_Taml")
|
| )
|
| ```
|
|
|
| This allows multilingual translation using a **single model instance**.
|
|
|
| ---
|
|
|
| # Project Structure
|
|
|
| ```
|
| en-tamil-mt/
|
| ├── app/
|
| │ └── app.py
|
| ├── evaluation/
|
| │ ├── evaluate_models.py
|
| │ └── evaluate_multilingual.py
|
| ├── notebooks/
|
| │ └── evaluation.ipynb
|
| ├── docs/
|
| │ └── report.md
|
| ├── requirements.txt
|
| ├── .gitignore
|
| └── README.md
|
| ```
|
|
|
| ---
|
|
|
| # Quick Start
|
|
|
| Clone the repository:
|
|
|
| ```
|
| git clone https://github.com/<your-username>/en-tamil-mt.git
|
| cd en-tamil-mt
|
| ```
|
|
|
| Create environment:
|
|
|
| ```
|
| python -m venv venv
|
| source venv/bin/activate
|
| ```
|
|
|
| Install dependencies:
|
|
|
| ```
|
| pip install -r requirements.txt
|
| ```
|
|
|
| ---
|
|
|
| # Running the App
|
|
|
| Run locally:
|
|
|
| ```
|
| python app/app.py
|
| ```
|
|
|
| Open:
|
|
|
| ```
|
| http://localhost:7860
|
| ```
|
|
|
| Live deployed application:
|
|
|
| ```
|
| https://huggingface.co/spaces/ashrithagowthami/Machine-Translation
|
| ```
|
|
|
| ---
|
|
|
| # Running Evaluation
|
|
|
| Evaluate model comparison:
|
|
|
| ```
|
| python evaluation/evaluate_models.py --model all --samples 200
|
| ```
|
|
|
| Multilingual evaluation:
|
|
|
| ```
|
| python evaluation/evaluate_multilingual.py --lang all --samples 200
|
| ```
|
|
|
| ---
|
|
|
| # Deployment Architecture
|
|
|
| ```
|
| User Input (English)
|
| │
|
| ▼
|
| Gradio UI
|
| │
|
| ▼
|
| Preprocessing
|
| │
|
| ▼
|
| NLLB Tokenizer
|
| │
|
| ▼
|
| NLLB Model.generate()
|
| │
|
| ▼
|
| Target language token
|
| │
|
| ▼
|
| Translated output
|
| ```
|
|
|
| ---
|
|
|
| # Report
|
|
|
| Full project report available in:
|
|
|
| ```
|
| docs/report.md
|
| ```
|
|
|
| The report includes:
|
|
|
| - Dataset analysis
|
| - Model comparison
|
| - Evaluation methodology
|
| - System architecture
|
| - Results and conclusions
|
|
|
| # Contributor
|
| Ashritha Gowthami Nelakurthi
|
| |
