krutrim-ai-labs/Chitranuvad

1. Introduction

Chitranuvad (चित्रानुवाद; Chitra = Image, Anuvad = Translation) is a multimodal translation model that leverages both images and text to perform translation from English to Indic languages. The model is designed to improve translation quality by grounding language understanding in visual context. Chitranuvad was developed for the English-to-Low Resource Multimodal Translation Task at WMT 2024, focusing on translation to Indic languages including Hindi, Bengali, and Malayalam and is a fine-tuned version of Chitrarth-1. The model integrates a vision encoder with a multilingual large language model, allowing it to resolve ambiguities in text using visual information.

Example of multimodal translation where the meaning of words like court and right depends on visual context. This demonstrates how visual input helps disambiguate translations.

Supported Tasks

• Text-only Translation: Translate English text into Indic languages without image input.
• Image Captioning: Generate captions from images in Indic languages.
• Multimodal Translation: Translate English text while conditioning on the corresponding image.

These tasks were evaluated as part of the WMT 2024 Multimodal Machine Translation shared task.

2. Model Architecture and Training

Chitranuvad is a multimodal translation model that integrates visual grounding with multilingual language modeling. The model processes the input image using a vision encoder to extract visual features, which are then projected onto the embedding space of the language model. These visual embeddings are fused with tokenized text embeddings and processed by the multilingual LLM to generate the translation.

Chitranuvad is trained in three stages to progressively align visual and textual representations and specialize the model for multimodal translation.

Stage 1 — Feature Alignment

The vision encoder and modality projector are trained to align image features with textual embeddings. This stage teaches the model to associate visual objects with their textual representations.

Stage 2 — Multimodal Instruction Tuning

The model is instruction-tuned on multimodal prompts involving both text and images. This enables the system to follow instructions such as captioning images or translating text while considering visual context.

Stage 3 — Task-Specific Finetuning

The final stage fine-tunes the model specifically for translation tasks using multimodal translation datasets containing images, English source sentences, and their translations in Indic languages.

3. Inference code

# Clone the Repository
git clone https://github.com/ola-krutrim/Chitranuvad.git
conda create --name chitranuvad python=3.10 -y
conda activate chitranuvad
cd Chitranuvad
pip install -r requirements.txt
pip install -e .
python chitranuvad/inference.py --model-path "krutrim-ai-labs/Chitranuvad" --image-file "assets/player_tennis_court.jpg" --query "Translate the caption below for the given image to Hindi language:\nA Tennis player in a court."

4. Example outputs

Example translations into Hindi, Bengali, and Malayalam conditioned on visual input. We enrich the dataset to include labels of all the identified objects

5. Results

Chitranuvad was the winning entry among participating submissions for the English to Low-Resource Multimodal Machine Translation task at the Workshop on Machine Translation, EMNLP 2024. The shared task comprised three tracks: Text-only Translation, Multimodal Translation, and Image Captioning, covering the languages Hindi, Bengali, and Malayalam. Model performance was evaluated using the BLEU and RIBES metrics.

6. License

This code repository and the model weights are licensed under the Krutrim Community License Agreement Version 1.0.

7. Citation

@inproceedings{khan-etal-2024-chitranuvad,
    title = "Chitranuvad: Adapting Multi-lingual {LLM}s for Multimodal Translation",
    author = "Khan, Shaharukh  and
      Tarun, Ayush  and
      Faraz, Ali  and
      Kamble, Palash  and
      Dahiya, Vivek  and
      Pokala, Praveen  and
      Kulkarni, Ashish  and
      Khatri, Chandra  and
      Ravi, Abhinav  and
      Agarwal, Shubham",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.wmt-1.80/",
    doi = "10.18653/v1/2024.wmt-1.80",
    pages = "839--851"
}

8. Contact

Contributions are welcome! If you have any improvements or suggestions, feel free to submit a pull request on GitHub.

9. Acknowledgements

Chitranuvad is built with reference to the code of the following projects:

• Chitrarth
• Transformers
• LLaVA-1.5.

We thank them for their great work!