social-data-attribution-w2/docs/eval_harness.md

Evaluation Harness

Post-unlearning evaluation measures capability change after training a LoRA adapter. The harness can evaluate a base model, an adapter, or a merged model and writes a summary JSON with per-benchmark metrics and gamma values.

This cleanup keeps the unlearning evaluation surface focused on:

Benchmark	Metric	Baseline
SocialIQA	accuracy	0.802900
MMLU Social Sciences	accuracy	0.750846
MMLU STEM	accuracy	0.597871
Wikitext-2	word perplexity	9.1559

A previously used math-reasoning benchmark is intentionally not part of this unlearning harness surface. Shared OLMES and TrackStar utilities elsewhere in the repository may still support historical or non-unlearning workflows.

Gamma

For accuracy metrics:

gamma = (score_unlearned - score_original) / abs(score_original)

Negative gamma means capability degradation relative to the base model. For Wikitext word perplexity, positive gamma means perplexity increased.

Modes

Full mode runs the project OLMES recipe, exports per-prediction rows, computes macro-averaged MMLU scores, and then runs Wikitext PPL.

python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --topic_bin social_life

Fast mode uses lm-eval directly with a sample limit. It is intended for local smoke checks and does not replace full evaluation.

python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --fast_eval \
    --fast_eval_samples 200

Wikitext-only mode skips accuracy benchmarks.

python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --wikitext_only

Output

The output JSON records metadata and a metrics object:

{
  "topic_bin": "social_life",
  "model_id": "allenai/OLMo-3-1025-7B",
  "adapter_dir": "runs/unlearn/social_life/adapter",
  "metrics": {
    "socialiqa": {
      "accuracy": 0.7812,
      "baseline": 0.8029,
      "gamma": -0.0270,
      "lower_is_better": false
    },
    "mmlu_social_science": {
      "accuracy": 0.7341,
      "baseline": 0.750846,
      "gamma": -0.0223,
      "lower_is_better": false
    },
    "mmlu_stem": {
      "accuracy": 0.5950,
      "baseline": 0.597871,
      "gamma": -0.0048,
      "lower_is_better": false
    },
    "wikitext": {
      "word_perplexity": 9.2100,
      "baseline": 9.1559,
      "gamma": 0.0059,
      "lower_is_better": true
    }
  }
}

Example values above are illustrative.

Aggregation

MMLU suites use macro-average accuracy across subtasks. SocialIQA uses micro-average accuracy. Unsupported suites present in shared OLMES output are ignored by this unlearning harness unless a baseline is added explicitly.