| --- |
| tags: |
| - evaluation |
| - uncertainty |
| - question-answering |
| - source-grounding |
| - afghanistan |
| - responsible-ai |
| language: |
| - en |
| --- |
| |
| # Whose World Does the Model Know? |
| ## Evaluating Source Grounding and Uncertainty in Afghanistan Education Questions |
|
|
| This project investigates whether a language model can distinguish information supported by a trusted source from information it cannot verify. |
|
|
| ### Research motivation |
|
|
| My perspective is shaped by my connection to Afghanistan and my computer science background. In locally specific contexts, confident but unsupported AI answers about education can mislead people seeking practical information. |
|
|
| This project examines two failure modes: unsupported specificity and failure to use information already present in a source. |
|
|
| ### Model |
|
|
| - Model: Qwen/Qwen3-4B-Instruct-2507 |
| - Model page: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 |
| - Environment: Google Colab, Tesla T4 GPU |
| - Reference source: [UNICEF Afghanistan education page](https://www.unicef.org/afghanistan/education) |
|
|
|
|
| ### Method |
|
|
| We created 12 questions: six answerable from the reference passage and six requiring current or school-specific information absent from that passage. |
|
|
| Each question was evaluated in three conditions: |
| 1. Closed book. |
| 2. Explicit uncertainty instruction. |
| 3. Relevant source passage supplied. |
|
|
| This produced 36 responses. |
|
|
| ### Preliminary results |
|
|
| | Condition | Exact correct on answerable questions | Appropriate abstentions on questions outside the source | |
| |---|---:|---:| |
| | Closed book | 1/6 | 4/6 | |
| | Uncertainty instruction | 0/6 | 6/6 | |
| | Source supplied | 6/6 | 6/6 | |
|
|
| These are preliminary human-scored results from a small pilot. They should not be generalized beyond the evaluated questions without further testing. |
|
|
| ### Proposed path forward |
|
|
| Develop a source-grounded uncertainty dataset with supported, unsupported, and partially supported questions. Use supervised fine-tuning as a future research direction and evaluate accuracy, appropriate abstention, unsupported claims, and unnecessary refusal separately. |
|
|
| This project has not fine-tuned the base model. |
|
|
| ### Limitations |
|
|
| The study uses one model, one source, 12 questions, and one response per question per condition. The results are exploratory. Scoring should be reviewed consistently, and future work should use a larger dataset, more sources, repeated generations, and independent reviewers. |
|
|
| ### Artifacts |
|
|
| The repository should include the question dataset, source attribution, prompts, raw outputs, reviewed scores, scoring rubric, evaluation code, and README. |
|
|
| Model: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 |
