social-data-attribution-w2/docs/EVALUATION_PROTOCOL.md

Evaluation Protocol

This document is the exact protocol we use to match the OLMo3 evaluation setup implemented in this repository.

Source of Truth

• Recipe entrypoint: src/data_attribution/recipes/olmes/evaluation.py
• Task suite definitions: olmes/oe_eval/configs/task_suites.py
• Task configs: olmes/oe_eval/configs/tasks.py
• Task implementations:
  • olmes/oe_eval/tasks/oe_eval_tasks/siqa.py
  • olmes/oe_eval/tasks/oe_eval_tasks/mmlu.py
  • olmes/oe_eval/tasks/oe_eval_tasks/gsm8k.py
• Core task runtime and MC request logic: olmes/oe_eval/tasks/base_task.py
• Metrics and aggregation:
  • olmes/oe_eval/metrics/metric.py
  • olmes/oe_eval/metrics/metric_utils.py

Reproduction Procedure

You can run the following command to reproduce the results:

uv run data-attribution-run-olmes-evaluation --model-id allenai/Olmo-3-1025-7B

This command runs exactly these 4 task specs:

• socialiqa:mc::xlarge
• mmlu_social_sciences:mc::olmes
• mmlu_stem:mc::olmes
• gsm8k::olmo3:n8:v2

Benchmark Parameters

Benchmark	OLMES task spec	Split	Shots	Prompt type	Decoding / inference mode	Primary metric	Item count
SocialIQA	socialiqa:mc::xlarge	all (with limit=10000)	5	MC letter selection	Log-likelihood ranking over " A", " B", " C"	acc_raw	10000
MMLU Social Sciences	mmlu_social_sciences:mc::olmes	test	5	MC letter selection	Log-likelihood ranking over answer letters	acc_raw (suite macro)	3077
GSM8k	gsm8k::olmo3:n8:v2	test	8	CoT generation with numeric extraction	do_sample=true, temperature=0.6, top_p=0.6, repeats=8	pass_at_1	1319
MMLU STEM	mmlu_stem:mc::olmes	test	5	MC letter selection	Log-likelihood ranking over answer letters	acc_raw (suite macro)	3018

Exact Prompt Templates

1) SocialIQA (socialiqa:mc::xlarge)

Question format for each item:

Question: {context} {question}
 A. {answerA}
 B. {answerB}
 C. {answerC}
Answer:

Few-shot context:

• 5 examples from fewshot_source=OLMES:social_i_qa
• Each example includes the same MC layout and a gold answer letter after Answer:
• Evaluation item is appended after those 5 examples

2) MMLU Social Sciences and MMLU STEM (mmlu_*:mc::olmes)

Per-subject prompt prefix:

The following are multiple choice questions (with answers) about {subject_name}.

Question format for each item:

Question: {question}
 A. {choice_A}
 B. {choice_B}
 C. {choice_C}
 D. {choice_D}
Answer:

Few-shot context:

• 5 examples
• Source is MMLU dev split in fixed order for each subject
• Evaluation items come from the subject test split

3) GSM8k (gsm8k::olmo3:n8:v2)

Question format for each item:

Question: {question}
Answer:

Few-shot context:

• 8 examples from fewshot_source=STD:GSM8k
• Few-shot answers contain chain-of-thought and end with So the answer is <number>.

Generation settings:

• max_gen_toks=512
• stop_sequences=["Question:", "</s>", "<|im_end|>", "\n\n"]
• do_sample=true
• temperature=0.6
• top_p=0.6
• repeats=8

Correctness criteria and normalization

MC benchmarks (SocialIQA, MMLU Social Sciences, MMLU STEM)

• Each answer choice is scored with a log-likelihood request using continuation format " {LETTER}".
• Prediction is the argmax over sum_logits.
• Primary metric is acc_raw for these task specs.
• Additional normalized variants (acc_per_token, acc_per_char, acc_per_byte) are computed, but they are not the primary score for these 3 benchmarks.

Length normalization note:

• Length-normalized values are available in outputs.
• OLMo3-compatible aggregate for these specific MC specs uses acc_raw, not a length-normalized metric.

GSM8k

• Continuations are generated 8 times per question.
• Answer extraction uses the task extractor, which takes the last numeric value in the generated continuation (after regex-based cleanup).
• exact_match is computed with task regex normalizations.
• pass_at_k is computed using 1 - comb(n-c, k) / comb(n, k) per item, where n=8 and c is number of correct samples.
• Primary metric is pass_at_1.

MMLU Subject Breakdown

Social Science (12 Tasks)

Subject	Item count
econometrics	114
high_school_geography	198
high_school_government_and_politics	193
high_school_macroeconomics	390
high_school_microeconomics	238
high_school_psychology	545
human_sexuality	131
professional_psychology	612
public_relations	110
security_studies	245
sociology	201
us_foreign_policy	100

STEM (18 tasks)

Subject	Item count
abstract_algebra	100
astronomy	152
college_biology	144
college_chemistry	100
college_computer_science	100
college_mathematics	100
college_physics	102
computer_security	100
conceptual_physics	235
electrical_engineering	145
elementary_mathematics	378
high_school_biology	310
high_school_chemistry	203
high_school_computer_science	100
high_school_mathematics	270
high_school_physics	151
high_school_statistics	216
machine_learning	112

SocialIQA Reasoning Type Breakdown

SocialIQA questions are constructed from the ATOMIC commonsense knowledge graph (Sap et al. 2019). Each question probes one of 9 ATOMIC relation types. The relation type is not stored in the dataset, but is deterministically derivable from the question wording:

Pattern	ATOMIC type	Description
"How would [X] feel?"	xReact	Subject's emotional reaction
"What would [X] want to do?"	xWant	Subject's desire after event
"What does [X] need to do before?"	xNeed	Subject's prerequisite
"What will happen to [X]?"	xEffect	Effect on subject
"How would others feel?"	oReact	Others' emotional reaction
"What would others want?"	oWant	Others' desire
"What will happen to others?"	oEffect	Effect on others
"Why did [X] do this?"	xIntent	Subject's intent
"How would [X] be seen?"	xAttr	Subject's attribute

Run scripts/validation/verify_query_counts.py to see the exact per-type counts against the current HF dataset.

GSM8k Subcategory Note

GSM8k has no inherent subcategory structure. All 1,319 items are arithmetic word problems with chain-of-thought answers.

Query Metadata Sidecar

The query_metadata module (src/data_attribution/evaluation/query_metadata.py) produces a flat metadata table keyed by query_id with these columns:

Column	Type	Description
query_id	string	Primary key, matches Bergson index and attribution outputs
benchmark	string	One of: socialiqa, gsm8k, mmlu_stem, mmlu_social_science
subject	string or null	MMLU subject (e.g., abstract_algebra), null for non-MMLU
reasoning_type	string or null	SocialIQA ATOMIC type (e.g., xReact), null for non-SocialIQA
is_correct	bool	Whether the model answered correctly
predicted_answer	string	Model's prediction
correct_answer	string	Gold answer

This sidecar is the single join target for all downstream analysis. Use it instead of parsing query_id strings to extract subject or benchmark information.

Count Verification

Run scripts/validation/verify_query_counts.py to validate that the HF dataset row counts match the documented values in this file. The script also checks MMLU per-subject counts, query_id uniqueness, and SocialIQA reasoning type coverage.

Known issue: the SocialIQA subset in the HF dataset contains 167 duplicate query_ids (same native_id appearing in multiple doc_ids). The predictions_export pipeline handles this via doc_id disambiguation. The verification script reports these as warnings, not errors.

Empirical Observations

• socialiqa:mc::xlarge in this OLMo3-compatible setup uses split=all and limit=10000, not a validation-only slice.
• gsm8k::olmo3:n8:v2 uses sampling (do_sample=true, repeats=8). It is not greedy decoding.
• If we decided to change either of those settings, we should not expect agreement with OLMo3 published aggregate numbers.