README.md

---
language:
  - en
  - de
tags:
  - machine-translation
  - in-context-learning
  - quality-estimation
  - large-language-models
  - domain-adaptation
  - amta-2024
license: mit
papers:
  - https://aclanthology.org/2024.amta-research.9
---

# 🧠 ICLviaQE

This repository contains the **data, code, and models** required to replicate the experiments from our paper:

> **Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation**  
> *Javad Pourmostafa R. Sh., Dimitar Shterionov, Pieter Spronck*  
> Accepted at **AMTA 2024**  
> 📄 [Read the paper on ACL Anthology](https://aclanthology.org/2024.amta-research.9)

---

## 📘 Summary

The quality of large language model (LLM) outputs in machine translation (MT) strongly depends on the **in-context examples (ICEs)** provided along with the input text.  
Selecting the most effective examples typically requires reference translations or human judgment — both costly and impractical at scale.

Our method, **ICLviaQE**, introduces a **quality-estimation-guided in-context learning framework** that selects and orders examples based on predicted translation quality, without using reference translations.  
By leveraging **XGLM** as a QE estimator, the system automatically identifies the most beneficial examples, leading to consistent improvements across domains and outperforming fine-tuned **mBART-50** baselines.

---

## 🧩 Methodology Overview

Below is a high-level summary of our approach:

<p align="center">
  <img src="https://github.com/JoyeBright/ICLviaQE/blob/main/Overview.png?raw=true" width="700" alt="ICLviaQE Methodology Overview">
</p>

🎥 [**Watch our 5-minute video presentation**](https://www.youtube.com/watch?v=CkVs-XV0LW0&ab_channel=JavadPourmostafa)

---

## ⚙️ Methodology Breakdown

For full implementation details and code for all stages and baselines, please refer to our GitHub repository:

👉 **[ICLviaQE on GitHub](https://github.com/JoyeBright/ICLviaQE)**

---

## 🧪 Baselines

| Baseline | Description | Code |
|-----------|--------------|------|
| **Random** | Randomly selects examples for ICL. | [`random_file.py`](random_file.py) → [`run_generation.py`](run_generation.py) |
| **Task-level** | Uses task-level contextual examples. | [`create_task_file.py`](create_task_file.py) → [`run_generation.py`](run_generation.py) |
| **BM25** | Retrieves similar pairs using BM25. | [`create_BM25_file.py`](create_BM25_file.py) |
| **R-BM25** | Enhanced BM25 version (external). | [R-BM25 Repository](https://github.com/sweta20/inContextMT) |
| **mBART-50** | Fine-tuned reference model. | [mBART-50 Repository](https://github.com/JoyeBright/MT-HF) |

---

## 🧾 Citation

If you use this work, please cite:

```bibtex
@inproceedings{pourmostafa-roshan-sharami-etal-2024-guiding,
  title     = "Guiding In-Context Learning of {LLM}s through Quality Estimation for Machine Translation",
  author    = "Pourmostafa Roshan Sharami, Javad and Shterionov, Dimitar and Spronck, Pieter",
  booktitle = "Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)",
  month     = sep,
  year      = "2024",
  address   = "Chicago, USA",
  publisher = "Association for Machine Translation in the Americas",
  url       = "https://aclanthology.org/2024.amta-research.9",
  pages     = "88--101"
}