metadata
title: README
emoji: 🐠
colorFrom: pink
colorTo: blue
sdk: static
pinned: false
license: cc-by-sa-4.0
Welcome to RLHN (EMNLP 2025 Findings)
RLHN (ReLabeing Hard Negatives) uses a cascading LLM framework to identify and relabel false negatives in IR training datasets.
This repository contains training datasets curated by RLHN & models fine-tuned on these curated datasets.
List of Contributors:
- Nandan Thakur*
- Crystina Zhang*
- Xueguang Ma
- Jimmy Lin
Paper URL: https://aclanthology.org/2025.findings-emnlp.481/
Citation
@inproceedings{thakur-etal-2025-hard,
title = "Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with {LLM}s",
author = "Thakur, Nandan and
Zhang, Crystina and
Ma, Xueguang and
Lin, Jimmy",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.481/",
doi = "10.18653/v1/2025.findings-emnlp.481",
pages = "9064--9083",
ISBN = "979-8-89176-335-7",
abstract = "Training robust retrieval and reranker models typically relies on large-scale retrieval datasets; for example, the BGE collection contains 1.6 million query-passage pairs sourced from various data sources. However, we find that certain datasets can negatively impact model effectiveness {---} pruning 8 out of 15 datasets from the BGE collection, reduces the training set size by 2.35{\texttimes}, surprisingly increases nDCG@10 on BEIR by 1.0 point. This motivates a deeper examination of training data quality, with a particular focus on ``false negatives'', where relevant passages are incorrectly labeled as irrelevant. We utilize LLMs as a simple, cost-effective approach to \textit{identify} and \textit{relabel} false negatives in training datasets. Experimental results show that relabeling false negatives as true positives improves both E5 (base) and Qwen2.5-7B retrieval models by 0.7-1.4 points on BEIR and by 1.7-1.8 points at nDCG@10 on zero-shot AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of LLMs to identify false negatives is supported by human annotation results. Our training dataset and code are publicly available."
}