social-data-attribution-w2/docs/eval_harness.md

# Evaluation Harness

Post-unlearning evaluation measures capability change after training a LoRA
adapter. The harness can evaluate a base model, an adapter, or a merged model
and writes a summary JSON with per-benchmark metrics and gamma values.

This cleanup keeps the unlearning evaluation surface focused on:

| Benchmark | Metric | Baseline |
|---|---|---:|
| SocialIQA | accuracy | 0.802900 |
| MMLU Social Sciences | accuracy | 0.750846 |
| MMLU STEM | accuracy | 0.597871 |
| Wikitext-2 | word perplexity | 9.1559 |

A previously used math-reasoning benchmark is intentionally not part of this
unlearning harness surface. Shared OLMES and TrackStar utilities elsewhere in
the repository may still support historical or non-unlearning workflows.

## Gamma

For accuracy metrics:

```text
gamma = (score_unlearned - score_original) / abs(score_original)
```

Negative gamma means capability degradation relative to the base model. For
Wikitext word perplexity, positive gamma means perplexity increased.

## Modes

Full mode runs the project OLMES recipe, exports per-prediction rows, computes
macro-averaged MMLU scores, and then runs Wikitext PPL.

```bash
python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --topic_bin social_life
```

Fast mode uses `lm-eval` directly with a sample limit. It is intended for local
smoke checks and does not replace full evaluation.

```bash
python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --fast_eval \
    --fast_eval_samples 200
```

Wikitext-only mode skips accuracy benchmarks.

```bash
python -m unlearning.eval_harness \
    --model_id allenai/OLMo-3-1025-7B \
    --adapter_dir runs/unlearn/social_life/adapter \
    --output_json runs/unlearn/social_life/eval_results.json \
    --wikitext_only
```

## Output

The output JSON records metadata and a `metrics` object:

```json
{
  "topic_bin": "social_life",
  "model_id": "allenai/OLMo-3-1025-7B",
  "adapter_dir": "runs/unlearn/social_life/adapter",
  "metrics": {
    "socialiqa": {
      "accuracy": 0.7812,
      "baseline": 0.8029,
      "gamma": -0.0270,
      "lower_is_better": false
    },
    "mmlu_social_science": {
      "accuracy": 0.7341,
      "baseline": 0.750846,
      "gamma": -0.0223,
      "lower_is_better": false
    },
    "mmlu_stem": {
      "accuracy": 0.5950,
      "baseline": 0.597871,
      "gamma": -0.0048,
      "lower_is_better": false
    },
    "wikitext": {
      "word_perplexity": 9.2100,
      "baseline": 9.1559,
      "gamma": 0.0059,
      "lower_is_better": true
    }
  }
}
```

Example values above are illustrative.

## Aggregation

MMLU suites use macro-average accuracy across subtasks. SocialIQA uses
micro-average accuracy. Unsupported suites present in shared OLMES output are
ignored by this unlearning harness unless a baseline is added explicitly.