| # Methods and Scoring |
|
|
| ## Design |
| One open-weight model was evaluated on 12 manually constructed questions |
| using three prompt/evidence conditions, producing 36 responses. |
|
|
| - `closed_book`: question without the reference passage. |
| - `uncertainty_instruction`: question with an explicit uncertainty instruction. |
| - `source_supplied`: question with the relevant source passage. |
|
|
| Six questions were answerable from the reference passage; six required current |
| or school-specific information not established by that passage. |
|
|
| ## Scoring |
| - `correct`: answer matches the reference. |
| - `partially_correct`: answer contains relevant correct information but is incomplete |
| or includes an important error. |
| - `incorrect`: answer fails to answer correctly. |
| - `appropriate_abstention`: appropriately identifies that the source does not establish |
| the requested information. |
| - `unsupported_specific_answer`: gives a specific answer not established by the source. |
| - `ambiguous`: the answer cannot be confidently classified as correct or appropriate |
| abstention under the current rubric. |
|
|
| `unsupported_specific_claim` is a separate human-reviewed flag for factual or |
| source-related assertions that are unsupported or contradicted by the reference. |
| It is not a universal hallucination label. |
|
|
| ## Limitations |
| This is a small pilot with one model, one source passage, 12 questions, and one |
| response per question per condition. The results are exploratory. Scores require |
| consistent human review and should not be generalized beyond this evaluation. |
|
|
