README.md

---
license: mit
language:
  - en
library_name: transformers
pipeline_tag: text-classification
base_model: google-bert/bert-base-uncased
datasets:
  - egcortes/qa-completeness-relevance
tags:
  - answer-evaluation
  - evaluation-metrics
  - completeness
  - long-form-qa
  - question-answering
  - regression
metrics:
  - spearmanr
---

# QA Completeness Regressor

**Give it a question and a long answer. It tells you how complete the answer is.**

[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-qa--completeness--relevance-yellow)](https://huggingface.co/datasets/egcortes/qa-completeness-relevance)
[![GitHub](https://img.shields.io/badge/GitHub-QA--CompletenessRelevance-black)](https://github.com/eduardogc8/QA-CompletenessRelevance)
[![Paper](https://img.shields.io/badge/Paper-Lang%20Resources%20%26%20Evaluation-blue)](https://doi.org/10.1007/s10579-026-09936-6)

A complete answer covers everything the question asks for. An incomplete one leaves things out.
This model reads the question and the answer together and returns one number for that.

You do not need a reference answer. That is the point of it.

It is `bert-base-uncased` with a small regression head, trained on synthetic data. The paper is
*Beyond accuracy: completeness and relevance metrics for evaluating the quality of long answers*,
**Language Resources and Evaluation** 60(3), article 58 (2026). It is open access:
[doi.org/10.1007/s10579-026-09936-6](https://doi.org/10.1007/s10579-026-09936-6).

---

## Quick start

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "egcortes/qa-completeness-regressor"
model = AutoModelForSequenceClassification.from_pretrained(name, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(name)

question = "How does RAM memory work?"
answer = "RAM stores data your computer is using right now. It is fast but it forgets everything when you turn the power off."

text = f"Question: {question}\n\nfAnswer: {answer}\n\nHow complete is this answer?"
inputs = tokenizer(text, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    score = model(**inputs).logits.item()

print(round(score, 3))   # 0 = nothing is covered, 1 = everything is covered
```

You can also use a pipeline:

```python
from transformers import pipeline

pipe = pipeline("text-classification", model="egcortes/qa-completeness-regressor",
                trust_remote_code=True, function_to_apply="none")
pipe(text)   # [{'label': 'completeness', 'score': 0.26}]
```

Two things to know:

- `trust_remote_code=True` is needed. The head is a plain linear layer on the CLS token, which is not
  one of the standard transformers heads, so the model class ships with the model.
- **The input format matters.** Build the text exactly as shown, including the odd `f` before
  `Answer:`. That typo was in the training code, so the model expects it. Without it the scores drift.
- **The output is a score, not a class.** The model is tagged as text classification because Hugging
  Face has no tag for regression, but it has a single output and no labels. In a pipeline the number
  comes back in a field called `score`, which normally means a probability. It is not one here, so
  always pass `function_to_apply="none"`.

---

## Results

Tested on the 212 human-scored answers in the
[companion dataset](https://huggingface.co/datasets/egcortes/qa-completeness-relevance), against the
average of four annotators:

| | Score |
|---|---|
| Spearman | 0.67 |
| Kendall | 0.47 |
| Pearson | 0.68 |

That beats every reference-based metric in the paper except ROUGE, and unlike ROUGE it does not need
a reference answer.

It also behaves sensibly when you take an answer apart. Cutting an expert answer down to fewer and
fewer sentences lowers the score every time:

| Sentences kept | Score |
|---|---|
| all | 0.59 |
| 80% | 0.53 |
| 60% | 0.44 |
| 40% | 0.34 |
| 20% | 0.24 |

The score went down at every step, for all 8 answers tested.

---

## Limitations

**It does not check facts.** An answer can be complete and still be wrong.

**It was trained on one domain.** The training data is "How to..." questions, mostly about computer
science, from ELI5 and WebGPT. It may behave differently elsewhere.

**The output is not calibrated.** Treat it as a ranking signal, not an absolute percentage. Across the 212
answers in the dataset it ranged from 0.08 to 0.76. Multiply by 100 to compare it with the
dataset scores.

---

## Training

Long answers were taken from ELI5 and WebGPT, then broken on purpose: sentences were removed to make
them less complete, and unrelated sentences were added to make them less relevant. Each damaged
answer got a score from how much of the original survived. That gave about 100,000 training examples.

The model then learned to predict that score from the question and the answer.

- Base model: `bert-base-uncased`
- Head: one linear layer, 768 to 1, on the CLS token
- Loss: MSE, 5 epochs, batch size 8, Adam, learning rate 1e-5
- Max length: 512 tokens

Because the training answers were damaged in a mechanical way, the model is good at spotting missing
content and less tested against the messier ways real answers go wrong.

---

## Links

- **Dataset**: [egcortes/qa-completeness-relevance](https://huggingface.co/datasets/egcortes/qa-completeness-relevance), 106 questions and 212 answers scored by four annotators
- **Code and data files**: [github.com/eduardogc8/QA-CompletenessRelevance](https://github.com/eduardogc8/QA-CompletenessRelevance)
- **Paper**: [doi.org/10.1007/s10579-026-09936-6](https://doi.org/10.1007/s10579-026-09936-6)

---

## Citation

Please cite the paper:

```bibtex
@article{cortes2026beyond,
  title   = {Beyond accuracy: completeness and relevance metrics for
             evaluating the quality of long answers},
  author  = {Cortes, Eduardo G. and Vieira, Renata and Barone, Dante A. C.},
  journal = {Language Resources and Evaluation},
  volume  = {60},
  number  = {3},
  pages   = {58},
  year    = {2026},
  doi     = {10.1007/s10579-026-09936-6}
}
```

---

## Contact

eduardogcortes8@gmail.com