METHODS.md

Methods and Scoring

Design

One open-weight model was evaluated on 12 manually constructed questions using three prompt/evidence conditions, producing 36 responses.

• closed_book: question without the reference passage.
• uncertainty_instruction: question with an explicit uncertainty instruction.
• source_supplied: question with the relevant source passage.

Six questions were answerable from the reference passage; six required current or school-specific information not established by that passage.

Scoring

• correct: answer matches the reference.
• partially_correct: answer contains relevant correct information but is incomplete or includes an important error.
• incorrect: answer fails to answer correctly.
• appropriate_abstention: appropriately identifies that the source does not establish the requested information.
• unsupported_specific_answer: gives a specific answer not established by the source.
• ambiguous: the answer cannot be confidently classified as correct or appropriate abstention under the current rubric.

unsupported_specific_claim is a separate human-reviewed flag for factual or source-related assertions that are unsupported or contradicted by the reference. It is not a universal hallucination label.

Limitations

This is a small pilot with one model, one source passage, 12 questions, and one response per question per condition. The results are exploratory. Scores require consistent human review and should not be generalized beyond this evaluation.