social-data-attribution-w2/docs/EVALUATION_PROTOCOL.md

# Evaluation Protocol

This document is the exact protocol we use to match the OLMo3 evaluation setup implemented in this repository.

## Source of Truth

- Recipe entrypoint: `src/data_attribution/recipes/olmes/evaluation.py`
- Task suite definitions: `olmes/oe_eval/configs/task_suites.py`
- Task configs: `olmes/oe_eval/configs/tasks.py`
- Task implementations:
  - `olmes/oe_eval/tasks/oe_eval_tasks/siqa.py`
  - `olmes/oe_eval/tasks/oe_eval_tasks/mmlu.py`
  - `olmes/oe_eval/tasks/oe_eval_tasks/gsm8k.py`
- Core task runtime and MC request logic: `olmes/oe_eval/tasks/base_task.py`
- Metrics and aggregation:
  - `olmes/oe_eval/metrics/metric.py`
  - `olmes/oe_eval/metrics/metric_utils.py`

## Reproduction Procedure

You can run the following command to reproduce the results:

```bash
uv run data-attribution-run-olmes-evaluation --model-id allenai/Olmo-3-1025-7B
```

This command runs exactly these 4 task specs:

- `socialiqa:mc::xlarge`
- `mmlu_social_sciences:mc::olmes`
- `mmlu_stem:mc::olmes`
- `gsm8k::olmo3:n8:v2`

## Benchmark Parameters


| Benchmark            | OLMES task spec                  | Split                      | Shots | Prompt type                            | Decoding / inference mode                                     | Primary metric          | Item count |
| -------------------- | -------------------------------- | -------------------------- | ----- | -------------------------------------- | ------------------------------------------------------------- | ----------------------- | ---------- |
| SocialIQA            | `socialiqa:mc::xlarge`           | `all` (with `limit=10000`) | 5     | MC letter selection                    | Log-likelihood ranking over `" A"`, `" B"`, `" C"`            | `acc_raw`               | `10000`    |
| MMLU Social Sciences | `mmlu_social_sciences:mc::olmes` | `test`                     | 5     | MC letter selection                    | Log-likelihood ranking over answer letters                    | `acc_raw` (suite macro) | `3077`     |
| GSM8k                | `gsm8k::olmo3:n8:v2`             | `test`                     | 8     | CoT generation with numeric extraction | `do_sample=true`, `temperature=0.6`, `top_p=0.6`, `repeats=8` | `pass_at_1`             | `1319`     |
| MMLU STEM            | `mmlu_stem:mc::olmes`            | `test`                     | 5     | MC letter selection                    | Log-likelihood ranking over answer letters                    | `acc_raw` (suite macro) | `3018`     |


## Exact Prompt Templates

### 1) SocialIQA (`socialiqa:mc::xlarge`)

Question format for each item:

```text
Question: {context} {question}
 A. {answerA}
 B. {answerB}
 C. {answerC}
Answer:
```

Few-shot context:

- 5 examples from `fewshot_source=OLMES:social_i_qa`
- Each example includes the same MC layout and a gold answer letter after `Answer:`
- Evaluation item is appended after those 5 examples

### 2) MMLU Social Sciences and MMLU STEM (`mmlu_*:mc::olmes`)

Per-subject prompt prefix:

```text
The following are multiple choice questions (with answers) about {subject_name}.

```

Question format for each item:

```text
Question: {question}
 A. {choice_A}
 B. {choice_B}
 C. {choice_C}
 D. {choice_D}
Answer:
```

Few-shot context:

- 5 examples
- Source is MMLU `dev` split in fixed order for each subject
- Evaluation items come from the subject `test` split

### 3) GSM8k (`gsm8k::olmo3:n8:v2`)

Question format for each item:

```text
Question: {question}
Answer:
```

Few-shot context:

- 8 examples from `fewshot_source=STD:GSM8k`
- Few-shot answers contain chain-of-thought and end with `So the answer is <number>.`

Generation settings:

- `max_gen_toks=512`
- `stop_sequences=["Question:", "</s>", "<|im_end|>", "\n\n"]`
- `do_sample=true`
- `temperature=0.6`
- `top_p=0.6`
- `repeats=8`

## Correctness criteria and normalization

### MC benchmarks (SocialIQA, MMLU Social Sciences, MMLU STEM)

- Each answer choice is scored with a log-likelihood request using continuation format `" {LETTER}"`.
- Prediction is the argmax over `sum_logits`.
- Primary metric is `acc_raw` for these task specs.
- Additional normalized variants (`acc_per_token`, `acc_per_char`, `acc_per_byte`) are computed, but they are not the primary score for these 3 benchmarks.

Length normalization note:

- Length-normalized values are available in outputs.
- OLMo3-compatible aggregate for these specific MC specs uses `acc_raw`, not a length-normalized metric.

### GSM8k

- Continuations are generated 8 times per question.
- Answer extraction uses the task extractor, which takes the last numeric value in the generated continuation (after regex-based cleanup).
- `exact_match` is computed with task regex normalizations.
- `pass_at_k` is computed using `1 - comb(n-c, k) / comb(n, k)` per item, where `n=8` and `c` is number of correct samples.
- Primary metric is `pass_at_1`.

## MMLU Subject Breakdown

### Social Science (12 Tasks)


| Subject                             | Item count |
| ----------------------------------- | ---------- |
| econometrics                        | `114`      |
| high_school_geography               | `198`      |
| high_school_government_and_politics | `193`      |
| high_school_macroeconomics          | `390`      |
| high_school_microeconomics          | `238`      |
| high_school_psychology              | `545`      |
| human_sexuality                     | `131`      |
| professional_psychology             | `612`      |
| public_relations                    | `110`      |
| security_studies                    | `245`      |
| sociology                           | `201`      |
| us_foreign_policy                   | `100`      |


### STEM (18 tasks)


| Subject                      | Item count  |
| ---------------------------- | ----------- |
| abstract_algebra             | `100` |
| astronomy                    | `152` |
| college_biology              | `144` |
| college_chemistry            | `100` |
| college_computer_science     | `100` |
| college_mathematics          | `100` |
| college_physics              | `102` |
| computer_security            | `100` |
| conceptual_physics           | `235` |
| electrical_engineering       | `145` |
| elementary_mathematics       | `378` |
| high_school_biology          | `310` |
| high_school_chemistry        | `203` |
| high_school_computer_science | `100` |
| high_school_mathematics      | `270` |
| high_school_physics          | `151` |
| high_school_statistics       | `216` |
| machine_learning             | `112` |


## SocialIQA Reasoning Type Breakdown

SocialIQA questions are constructed from the ATOMIC commonsense knowledge graph (Sap et al. 2019). Each question probes one of 9 ATOMIC relation types. The relation type is not stored in the dataset, but is deterministically derivable from the question wording:

| Pattern | ATOMIC type | Description |
|---------|------------|-------------|
| "How would [X] feel?" | xReact | Subject's emotional reaction |
| "What would [X] want to do?" | xWant | Subject's desire after event |
| "What does [X] need to do before?" | xNeed | Subject's prerequisite |
| "What will happen to [X]?" | xEffect | Effect on subject |
| "How would others feel?" | oReact | Others' emotional reaction |
| "What would others want?" | oWant | Others' desire |
| "What will happen to others?" | oEffect | Effect on others |
| "Why did [X] do this?" | xIntent | Subject's intent |
| "How would [X] be seen?" | xAttr | Subject's attribute |

Run `scripts/validation/verify_query_counts.py` to see the exact per-type counts against the current HF dataset.

## GSM8k Subcategory Note

GSM8k has no inherent subcategory structure. All 1,319 items are arithmetic word problems with chain-of-thought answers.

## Query Metadata Sidecar

The `query_metadata` module (`src/data_attribution/evaluation/query_metadata.py`) produces a flat metadata table keyed by `query_id` with these columns:

| Column | Type | Description |
|--------|------|-------------|
| `query_id` | string | Primary key, matches Bergson index and attribution outputs |
| `benchmark` | string | One of: `socialiqa`, `gsm8k`, `mmlu_stem`, `mmlu_social_science` |
| `subject` | string or null | MMLU subject (e.g., `abstract_algebra`), null for non-MMLU |
| `reasoning_type` | string or null | SocialIQA ATOMIC type (e.g., `xReact`), null for non-SocialIQA |
| `is_correct` | bool | Whether the model answered correctly |
| `predicted_answer` | string | Model's prediction |
| `correct_answer` | string | Gold answer |

This sidecar is the single join target for all downstream analysis. Use it instead of parsing query_id strings to extract subject or benchmark information.

## Count Verification

Run `scripts/validation/verify_query_counts.py` to validate that the HF dataset row counts match the documented values in this file. The script also checks MMLU per-subject counts, query_id uniqueness, and SocialIQA reasoning type coverage.

Known issue: the SocialIQA subset in the HF dataset contains 167 duplicate query_ids (same native_id appearing in multiple doc_ids). The predictions_export pipeline handles this via doc_id disambiguation. The verification script reports these as warnings, not errors.

## Empirical Observations

- `socialiqa:mc::xlarge` in this OLMo3-compatible setup uses `split=all` and `limit=10000`, not a validation-only slice.
- `gsm8k::olmo3:n8:v2` uses sampling (`do_sample=true`, `repeats=8`). It is not greedy decoding.
- If we decided to change either of those settings, we should not expect agreement with OLMo3 published aggregate numbers.