| --- |
| language: en |
| license: mit |
| library_name: pytorch |
| base_model: FacebookAI/roberta-base |
| pipeline_tag: question-answering |
| tags: |
| - extractive-qa |
| - grounded-qa |
| - hallucination-free |
| - selective-prediction |
| - abstention |
| - rag |
| datasets: |
| - rajpurkar/squad_v2 |
| metrics: |
| - exact_match |
| model-index: |
| - name: grounded-pointer-qa |
| results: |
| - task: |
| type: question-answering |
| dataset: |
| name: SQuAD v2 (held-out test half, retrieval setting) |
| type: rajpurkar/squad_v2 |
| metrics: |
| - type: exact_match |
| value: 74.6 |
| name: EM (EM-optimal gate) |
| - type: precision |
| value: 91.7 |
| name: Answered precision at 90%-precision gate (9% coverage) |
| --- |
| |
| # Grounded Pointer QA |
|
|
| An extractive question-answering model that **cannot hallucinate by |
| construction**: its output layer can only point at spans inside retrieved |
| passages of your documents. It has no vocabulary to generate from. A trained |
| abstention head refuses when the loaded knowledge does not contain the answer, |
| decoding is deterministic (argmax, so the same question over the same documents |
| gives the same answer every time), and knowledge is **hot-swappable**: point it |
| at a new folder of text files and it answers from those, with no retraining. |
|
|
| Built on `roberta-base` (125M params) with pointer and abstention heads, |
| finetuned on SQuAD v2 against real TF-IDF retrieval. The model never saw gold |
| passages during training, only what the retriever actually returned, so it |
| learned to abstain on retrieval misses too. |
|
|
|  |
|
|
| ## Operating modes |
|
|
| The checkpoint ships with a calibrated confidence gate |
| (`P(answerable) x P(span)`), selected on a calibration split and verified on a |
| disjoint held-out test split: |
|
|
| | Mode | Gate | Coverage | Answered precision | EM | |
| |---|---|---|---|---| |
| | "Right or silent" (shipped default) | 0.965 | 9% | 91.7% | 57.1 | |
| | EM-optimal | 0.295 | 45% | 72.3% | 74.6 | |
|
|
| Pass a lower `gate` to `ask()` for more coverage at lower precision. |
|
|
| ## Usage |
|
|
| ```python |
| # files needed: proqa.pt, modeling_proqa.py (both in this repo) |
| # pip install torch transformers scikit-learn numpy (plus pypdf for PDFs) |
| from modeling_proqa import GroundedQA |
| |
| qa = GroundedQA("proqa.pt") |
| qa.load_folder("path/to/your/notes") # .txt / .md / .pdf |
| |
| qa.ask("when does the vendor contract expire?") |
| # {'answer': '30 November 2026', 'source': '...expires on 30 November 2026...', |
| # 'confidence': 0.98} |
| |
| qa.ask("what is the capital of France?") # not in your docs |
| # {'answer': None, 'source': None, 'confidence': 0.0} |
| ``` |
|
|
| ## Roadmap |
|
|
| Today the model answers one self-contained question at a time. A conversational |
| grounding layer (multi-turn context and follow-up questions, with every answer |
| still a verified quote) is coming. |
|
|
| ## Training |
|
|
| One NVIDIA RTX 5060 Ti (16 GB): about 2.5 h finetune (batch 8 x 4 passages x |
| 384 tokens, bf16, lr 2e-5, 2 epochs) plus a calibration pass. |
|
|
