SRK-CobuildX's picture
Release v0.1
041950a verified
|
Raw History Blame Contribute Delete
7.3 kB
---
license: apache-2.0
base_model: jhu-clsp/ettin-encoder-68m
pipeline_tag: text-classification
language:
- en
tags:
- rag
- rag-evaluation
- context-utilization
---
# cbx-rag-context-utilization-68M
Judges which **context sentences the answer actually used**. Give it the question, the passages and the answer. It
returns:
- a score and flag for every context sentence;
- the **utilization fraction**: the share of context sentences the answer used, with the RAGBench convention.
It is one of four CBX RAG evaluator models.
## How to use
```bash
pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-context-utilization-68M --revision v0.1 --local-dir cbx-rag-context-utilization-68M
cd cbx-rag-context-utilization-68M
pip install -r requirements.txt
python example.py
```
```python
from cbx_rag_eval import ContextUtilization
judge = ContextUtilization(".")
out = judge(question, passages, answer)
out["utilization"]
[s for s in out["sentences"] if s["utilized"]]
```
**How the fraction is computed:** each sentence probability is calibrated (isotonic, fitted on validation), and the
fraction is their average.
**Prefilter (`contract` profile):** only the 64 sentences closest to the question and answer (BM25), plus their
neighbours, are scored; the rest count as unused. On validation this kept 98.8% of used sentences while scoring
about 39% of all sentences, with the same RMSE. Pass `prefilter=False` to score everything. The test numbers
below are without the prefilter.
**Speed** (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):
| Input | GPU | CPU |
|---|---|---|
| one contract (about 414 sentences) | 0.61 s | 23.43 s |
| several short passages (about 29 sentences) | 0.12 s | 3.04 s |
## The four CBX RAG evaluator models
| Model | Judges | Output |
|---|---|---|
| [cbx-rag-faithfulness-68M](https://huggingface.co/CobuildX-AI/cbx-rag-faithfulness-68M) | each answer sentence against the passages | support score per sentence, answer score, hallucination flag |
| [cbx-rag-response-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-response-relevance-68M) | the answer against the question | direct / partial / irrelevant |
| [cbx-rag-context-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-relevance-68M) | each context sentence against the question | relevance per sentence, relevance fraction, context precision |
| [cbx-rag-context-utilization-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M) | each context sentence against the question and answer | use per sentence, utilization fraction |
All four share the wrapper `cbx_rag_eval.py`. Every model is fine-tuned from
[jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) (68M parameters). Current version:
**v0.1** (use `--revision v0.1` to pin it).
## Sentences and profiles
- **Sentences:** the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper
splits raw text by profile (`split_sentences`; chosen on validation):
- `contract`: line breaks, then NLTK punkt;
- `financial`: blank lines, then pysbd.
On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.990 → 0.981 (CUAD); sentence AUROC 0.996 → 0.995 (FinQA); answer RMSE 0.040 → 0.043 (CUAD); answer RMSE 0.036 → 0.040 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and `answer_sentences=`
for the answer); the splitter is then skipped.
- **Profiles:** the settings were fixed on two kinds of validation data, so every call takes a `profile`.
- `"contract"`: one document, such as a contract (CUAD-like).
- `"financial"`: several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check
here.
- `"auto"` (default): `"contract"` for a single passage, `"financial"` for two or more.
For other kinds of text, choose the profile whose shape is closest and check the results on your own data.
## Training data and labels
- **Source:** the CUAD (contracts) and FinQA (financial reports) subsets of
[RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench) (CC BY 4.0). Its answers were written by
claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
- Train: 1,530 CUAD and 12,502 FinQA answers.
- Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
- **Labels were reviewed by LLMs, not humans:**
- **Faithfulness:** majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku
passes for CUAD train.
- **CUAD context relevance / utilization:** majority vote of RAGBench and two Claude Haiku passes.
- **FinQA context labels:** kept as RAGBench made them.
- **Audits:** samples were checked blind by Claude Opus and Claude Sonnet.
- **Response relevance:** uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions
and labelled them.
- **Test sets:** the RAGBench test splits, with the same label views.
- **Baselines:** RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
- **Rule:** every threshold and setting was fixed on validation.
## Licence
Apache-2.0 for the model weights and code.
**Data attribution:**
- RAGBench (Friel et al., 2024), CC BY 4.0;
- CUAD (The Atticus Project), CC BY 4.0;
- FinQA (Chen et al., 2021), MIT.
Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.
**Disclaimer:** the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an
answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential
decisions.
## Results (test sets)
- **Labels:** CUAD uses reviewed labels (majority of RAGBench and two Claude Haiku passes); FinQA uses RAGBench
labels.
- **Constant:** predicts the validation average.
**Sentence level:**
| Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold |
|---|---|---|---|
| CUAD contracts | 0.981 [0.973, 0.987] | 0.737 [0.697, 0.774] | 0.78 / 0.60 |
| FinQA reports | 0.995 [0.994, 0.996] | 0.945 [0.933, 0.956] | 0.90 / 0.88 |
**Answer-level utilization fraction:** RMSE, lower is better. GPT-3.5 is RAGBench's stored score; RAGBench has no
other utilization baseline.
| Test set | This model | GPT-3.5 | Constant |
|---|---|---|---|
| CUAD contracts (415 answers with a baseline score) | **0.039** | 0.059 | 0.063 |
| FinQA reports (2,294) | **0.033** | 0.134 | 0.067 |
Both differences against GPT-3.5 and the constant are significant (context-clustered bootstrap). On RAGBench's own
CUAD labels the model ties GPT-3.5 (0.108 vs 0.105).
## Limitations
- **CUAD recall:** at the default threshold it misses about 40% of used sentences (recall 0.60). When it flags a
sentence, it is usually right (precision 0.78).
- **Label noise:** in a blind Claude Sonnet audit, 6 of 10 sentences the review added and 4 of 10 model misses were
confirmed as used.
- **Not reported:** evidence completeness (relevant sentences the answer used). Combined with
cbx-rag-context-relevance it did not beat a constant.
- **Coverage:** two domains (contracts, financial reports), English, LLM-made labels.