Whose World Does the Model Know?
Evaluating Source Grounding and Uncertainty in Afghanistan Education Questions
This project investigates whether a language model can distinguish information supported by a trusted source from information it cannot verify.
Research motivation
My perspective is shaped by my connection to Afghanistan and my computer science background. In locally specific contexts, confident but unsupported AI answers about education can mislead people seeking practical information.
This project examines two failure modes: unsupported specificity and failure to use information already present in a source.
Model
- Model: Qwen/Qwen3-4B-Instruct-2507
- Model page: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
- Environment: Google Colab, Tesla T4 GPU
- Reference source: UNICEF Afghanistan education page
Method
We created 12 questions: six answerable from the reference passage and six requiring current or school-specific information absent from that passage.
Each question was evaluated in three conditions:
- Closed book.
- Explicit uncertainty instruction.
- Relevant source passage supplied.
This produced 36 responses.
Preliminary results
| Condition | Exact correct on answerable questions | Appropriate abstentions on questions outside the source |
|---|---|---|
| Closed book | 1/6 | 4/6 |
| Uncertainty instruction | 0/6 | 6/6 |
| Source supplied | 6/6 | 6/6 |
These are preliminary human-scored results from a small pilot. They should not be generalized beyond the evaluated questions without further testing.
Proposed path forward
Develop a source-grounded uncertainty dataset with supported, unsupported, and partially supported questions. Use supervised fine-tuning as a future research direction and evaluate accuracy, appropriate abstention, unsupported claims, and unnecessary refusal separately.
This project has not fine-tuned the base model.
Limitations
The study uses one model, one source, 12 questions, and one response per question per condition. The results are exploratory. Scoring should be reviewed consistently, and future work should use a larger dataset, more sources, repeated generations, and independent reviewers.
Artifacts
The repository should include the question dataset, source attribution, prompts, raw outputs, reviewed scores, scoring rubric, evaluation code, and README.