cbx-rag-context-relevance-68M

Judges which retrieved context sentences are relevant to the question. Give it the question and the passages. It returns:

  • a relevance score and flag for every context sentence (it reads each sentence with two neighbours on each side);
  • the relevance fraction: the share of context sentences that are relevant;
  • context precision, when there are two or more passages: are the relevant passages ranked first?

It is one of four CBX RAG evaluator models.

How to use

pip install -U huggingface_hub
hf download CobuildX-AI/cbx-rag-context-relevance-68M --revision v0.1 --local-dir cbx-rag-context-relevance-68M
cd cbx-rag-context-relevance-68M
pip install -r requirements.txt
python example.py
from cbx_rag_eval import ContextRelevance

judge = ContextRelevance(".")
out = judge(question, passages)         # passages in retrieval order
out["relevance"], out.get("context_precision")
[s for s in out["sentences"] if s["relevant"]]

Outputs:

  • relevance: the answer-level fraction, with the same convention as RAGBench (relevant sentences / all sentences).
  • context_precision: average precision over the passages in the order given. A passage counts as relevant when any of its sentences is flagged.

On long single documents (contract profile), use the per-sentence scores. There the answer-level fraction is close to a constant: few sentences are relevant to any one question, so a fixed guess is hard to beat.

Speed (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32):

Input GPU CPU
one contract (about 414 sentences) 1.02 s 43.87 s
several short passages (about 29 sentences) 0.10 s 1.71 s

The four CBX RAG evaluator models

Model Judges Output
cbx-rag-faithfulness-68M each answer sentence against the passages support score per sentence, answer score, hallucination flag
cbx-rag-response-relevance-68M the answer against the question direct / partial / irrelevant
cbx-rag-context-relevance-68M each context sentence against the question relevance per sentence, relevance fraction, context precision
cbx-rag-context-utilization-68M each context sentence against the question and answer use per sentence, utilization fraction

All four share the wrapper cbx_rag_eval.py. Every model is fine-tuned from jhu-clsp/ettin-encoder-68m (68M parameters). Current version: v0.1 (use --revision v0.1 to pin it).

Sentences and profiles

  • Sentences: the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper splits raw text by profile (split_sentences; chosen on validation):

    • contract: line breaks, then NLTK punkt;
    • financial: blank lines, then pysbd.

    On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.942 β†’ 0.920 (CUAD); sentence AUROC 0.965 β†’ 0.962 (FinQA); answer RMSE 0.034 β†’ 0.034 (CUAD); answer RMSE 0.066 β†’ 0.070 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and answer_sentences= for the answer); the splitter is then skipped.

  • Profiles: the settings were fixed on two kinds of validation data, so every call takes a profile.

    • "contract": one document, such as a contract (CUAD-like).
    • "financial": several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check here.
    • "auto" (default): "contract" for a single passage, "financial" for two or more.

    For other kinds of text, choose the profile whose shape is closest and check the results on your own data.

Training data and labels

  • Source: the CUAD (contracts) and FinQA (financial reports) subsets of RAGBench (CC BY 4.0). Its answers were written by claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo.
    • Train: 1,530 CUAD and 12,502 FinQA answers.
    • Test: 510 CUAD (102 contracts) and 2,294 FinQA answers.
  • Labels were reviewed by LLMs, not humans:
    • Faithfulness: majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku passes for CUAD train.
    • CUAD context relevance / utilization: majority vote of RAGBench and two Claude Haiku passes.
    • FinQA context labels: kept as RAGBench made them.
    • Audits: samples were checked blind by Claude Opus and Claude Sonnet.
  • Response relevance: uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions and labelled them.
  • Test sets: the RAGBench test splits, with the same label views.
  • Baselines: RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs.
  • Rule: every threshold and setting was fixed on validation.

Licence

Apache-2.0 for the model weights and code.

Data attribution:

  • RAGBench (Friel et al., 2024), CC BY 4.0;
  • CUAD (The Atticus Project), CC BY 4.0;
  • FinQA (Chen et al., 2021), MIT.

Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models.

Disclaimer: the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential decisions.

Results (test sets)

  • Labels: CUAD uses reviewed labels: RAGBench's long clause-block labels were rejected by audit in most cases and replaced by the majority of RAGBench and two Claude Haiku passes. FinQA uses RAGBench labels.
  • Constant: predicts the validation average.

Sentence level:

Test set AUROC [95% CI] PR-AUC [95% CI] Precision / recall at the threshold Positives
CUAD contracts 0.933 [0.920, 0.945] 0.372 [0.321, 0.421] 0.40 / 0.42 1.5% of 104,725 sentences
FinQA reports 0.977 [0.972, 0.982] 0.858 [0.839, 0.877] 0.81 / 0.79 6.1% of 53,640 sentences

Answer-level relevance fraction: RMSE, lower is better. Baselines are RAGBench's stored scores, on the answers they cover.

Test set This model RAGAS GPT-3.5 TruLens Constant
CUAD contracts 0.047 0.057 0.118 0.666 0.050
FinQA reports 0.054 0.106 0.103 0.793 0.081

On CUAD the model beats RAGAS and GPT-3.5 significantly, but not the constant (0.044 vs 0.047 over all answers, a difference within noise).

Context precision (FinQA, 2,281 answers with a relevant passage): RMSE 0.164 [0.143, 0.184], against 0.209 for a constant. RAGBench stores no context-precision baselines. It is not available for single-document contexts.

Limitations

  • CUAD sentence precision and recall are only about 0.4. A blind Claude Sonnet audit found that about 3 in 4 of the model's misses, and 1 in 4 of its false alarms, were label errors. So the true quality is probably higher than measured, but this is not quantified.
  • It disagrees with RAGBench's CUAD labels on purpose. On those block labels its AUROC is 0.72.
  • Not reported: evidence completeness (relevant sentences the answer used). Combined with cbx-rag-context-utilization it did not beat a constant.
  • Cost: every context sentence is scored, about 0.5 s per 280-sentence contract on a GPU.
  • Coverage: two domains (contracts, financial reports), English, LLM-made labels.
Downloads last month
18
Safetensors
Model size
68.1M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for CobuildX-AI/cbx-rag-context-relevance-68M

Finetuned
(47)
this model

Collection including CobuildX-AI/cbx-rag-context-relevance-68M