|
Download README.md from CobuildX-AI/cbx-rag-context-utilization-68M: direct link, hf CLI and curl.
- Browser
- Download file 7.3 kB
-
https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M/resolve/main/README.md
- Command line
-
hf download hf://CobuildX-AI/cbx-rag-context-utilization-68M/README.md
-
curl -L -o README.md https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M/resolve/main/README.md
7.3 kB
| license: apache-2.0 | |
| base_model: jhu-clsp/ettin-encoder-68m | |
| pipeline_tag: text-classification | |
| language: | |
| - en | |
| tags: | |
| - rag | |
| - rag-evaluation | |
| - context-utilization | |
| # cbx-rag-context-utilization-68M | |
| Judges which **context sentences the answer actually used**. Give it the question, the passages and the answer. It | |
| returns: | |
| - a score and flag for every context sentence; | |
| - the **utilization fraction**: the share of context sentences the answer used, with the RAGBench convention. | |
| It is one of four CBX RAG evaluator models. | |
| ## How to use | |
| ```bash | |
| pip install -U huggingface_hub | |
| hf download CobuildX-AI/cbx-rag-context-utilization-68M --revision v0.1 --local-dir cbx-rag-context-utilization-68M | |
| cd cbx-rag-context-utilization-68M | |
| pip install -r requirements.txt | |
| python example.py | |
| ``` | |
| ```python | |
| from cbx_rag_eval import ContextUtilization | |
| judge = ContextUtilization(".") | |
| out = judge(question, passages, answer) | |
| out["utilization"] | |
| [s for s in out["sentences"] if s["utilized"]] | |
| ``` | |
| **How the fraction is computed:** each sentence probability is calibrated (isotonic, fitted on validation), and the | |
| fraction is their average. | |
| **Prefilter (`contract` profile):** only the 64 sentences closest to the question and answer (BM25), plus their | |
| neighbours, are scored; the rest count as unused. On validation this kept 98.8% of used sentences while scoring | |
| about 39% of all sentences, with the same RMSE. Pass `prefilter=False` to score everything. The test numbers | |
| below are without the prefilter. | |
| **Speed** (seconds per call, model loaded; GPU NVIDIA TITAN RTX fp16, CPU 8 threads fp32): | |
| | Input | GPU | CPU | | |
| |---|---|---| | |
| | one contract (about 414 sentences) | 0.61 s | 23.43 s | | |
| | several short passages (about 29 sentences) | 0.12 s | 3.04 s | | |
| ## The four CBX RAG evaluator models | |
| | Model | Judges | Output | | |
| |---|---|---| | |
| | [cbx-rag-faithfulness-68M](https://huggingface.co/CobuildX-AI/cbx-rag-faithfulness-68M) | each answer sentence against the passages | support score per sentence, answer score, hallucination flag | | |
| | [cbx-rag-response-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-response-relevance-68M) | the answer against the question | direct / partial / irrelevant | | |
| | [cbx-rag-context-relevance-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-relevance-68M) | each context sentence against the question | relevance per sentence, relevance fraction, context precision | | |
| | [cbx-rag-context-utilization-68M](https://huggingface.co/CobuildX-AI/cbx-rag-context-utilization-68M) | each context sentence against the question and answer | use per sentence, utilization fraction | | |
| All four share the wrapper `cbx_rag_eval.py`. Every model is fine-tuned from | |
| [jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) (68M parameters). Current version: | |
| **v0.1** (use `--revision v0.1` to pin it). | |
| ## Sentences and profiles | |
| - **Sentences:** the models judge sentences, and they were trained on RAGBench's sentence splits. The wrapper | |
| splits raw text by profile (`split_sentences`; chosen on validation): | |
| - `contract`: line breaks, then NLTK punkt; | |
| - `financial`: blank lines, then pysbd. | |
| On validation, the wrapper's splitting instead of RAGBench's own sentence splits changed sentence AUROC 0.990 → 0.981 (CUAD); sentence AUROC 0.996 → 0.995 (FinQA); answer RMSE 0.040 → 0.043 (CUAD); answer RMSE 0.036 → 0.040 (FinQA). For the best accuracy, pass your own sentences (a list of strings per passage, and `answer_sentences=` | |
| for the answer); the splitter is then skipped. | |
| - **Profiles:** the settings were fixed on two kinds of validation data, so every call takes a `profile`. | |
| - `"contract"`: one document, such as a contract (CUAD-like). | |
| - `"financial"`: several short passages with numbers and tables (FinQA-like). Faithfulness adds a numeric check | |
| here. | |
| - `"auto"` (default): `"contract"` for a single passage, `"financial"` for two or more. | |
| For other kinds of text, choose the profile whose shape is closest and check the results on your own data. | |
| ## Training data and labels | |
| - **Source:** the CUAD (contracts) and FinQA (financial reports) subsets of | |
| [RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench) (CC BY 4.0). Its answers were written by | |
| claude-3-haiku and gpt-3.5-turbo; its labels came from gpt-4-turbo. | |
| - Train: 1,530 CUAD and 12,502 FinQA answers. | |
| - Test: 510 CUAD (102 contracts) and 2,294 FinQA answers. | |
| - **Labels were reviewed by LLMs, not humans:** | |
| - **Faithfulness:** majority vote of RAGBench and two LLM reviewers: Claude Haiku and gpt-5.4-mini, or two Haiku | |
| passes for CUAD train. | |
| - **CUAD context relevance / utilization:** majority vote of RAGBench and two Claude Haiku passes. | |
| - **FinQA context labels:** kept as RAGBench made them. | |
| - **Audits:** samples were checked blind by Claude Opus and Claude Sonnet. | |
| - **Response relevance:** uses its own set (rr-v0.1). Claude Haiku wrote varied answers to the RAGBench questions | |
| and labelled them. | |
| - **Test sets:** the RAGBench test splits, with the same label views. | |
| - **Baselines:** RAGAS, GPT-3.5 and TruLens scores are the ones stored in RAGBench, not re-runs. | |
| - **Rule:** every threshold and setting was fixed on validation. | |
| ## Licence | |
| Apache-2.0 for the model weights and code. | |
| **Data attribution:** | |
| - RAGBench (Friel et al., 2024), CC BY 4.0; | |
| - CUAD (The Atticus Project), CC BY 4.0; | |
| - FinQA (Chen et al., 2021), MIT. | |
| Labels were produced with Claude (Anthropic) and gpt-5.4-mini (OpenAI) models. | |
| **Disclaimer:** the scores are statistical estimates for evaluating RAG systems. They are not a guarantee that an | |
| answer is correct, complete or safe. Do not use them as the sole basis for legal, financial or other consequential | |
| decisions. | |
| ## Results (test sets) | |
| - **Labels:** CUAD uses reviewed labels (majority of RAGBench and two Claude Haiku passes); FinQA uses RAGBench | |
| labels. | |
| - **Constant:** predicts the validation average. | |
| **Sentence level:** | |
| | Test set | AUROC [95% CI] | PR-AUC [95% CI] | Precision / recall at the threshold | | |
| |---|---|---|---| | |
| | CUAD contracts | 0.981 [0.973, 0.987] | 0.737 [0.697, 0.774] | 0.78 / 0.60 | | |
| | FinQA reports | 0.995 [0.994, 0.996] | 0.945 [0.933, 0.956] | 0.90 / 0.88 | | |
| **Answer-level utilization fraction:** RMSE, lower is better. GPT-3.5 is RAGBench's stored score; RAGBench has no | |
| other utilization baseline. | |
| | Test set | This model | GPT-3.5 | Constant | | |
| |---|---|---|---| | |
| | CUAD contracts (415 answers with a baseline score) | **0.039** | 0.059 | 0.063 | | |
| | FinQA reports (2,294) | **0.033** | 0.134 | 0.067 | | |
| Both differences against GPT-3.5 and the constant are significant (context-clustered bootstrap). On RAGBench's own | |
| CUAD labels the model ties GPT-3.5 (0.108 vs 0.105). | |
| ## Limitations | |
| - **CUAD recall:** at the default threshold it misses about 40% of used sentences (recall 0.60). When it flags a | |
| sentence, it is usually right (precision 0.78). | |
| - **Label noise:** in a blind Claude Sonnet audit, 6 of 10 sentences the review added and 4 of 10 model misses were | |
| confirmed as used. | |
| - **Not reported:** evidence completeness (relevant sentences the answer used). Combined with | |
| cbx-rag-context-relevance it did not beat a constant. | |
| - **Coverage:** two domains (contracts, financial reports), English, LLM-made labels. | |