What-Did-You-Forget-Unlearned-Models
Collection
67 unlearned checkpoints (PISCES, RMU, CRISP, SNMF) on Gemma-2-2B-IT and Llama-3.1-8B-Instruct. See the overview dataset for details. • 68 items • Updated
Unlearned Checkpoint
| Field | Value |
|---|---|
| Unlearning method | CRISP |
| Base model | meta-llama/Llama-3.1-8B-Instruct |
| Target concept | Uranium |
| Checkpoint type | LoRA Adapter |
| Rank / seed | 200 / 42 |
| Train eval protocol | mc |
Unlearning Configuration
Selected hyperparameters (from unlearned_checkpoints.json):
| Parameter | Value |
|---|---|
alpha |
20 |
delta_embed |
0 |
k_features |
20 |
k_features_embed |
0 |
layer_hi |
29 |
layer_lo |
5 |
layer_step |
2 |
lora_rank |
4 |
lr |
0.0001 |
n_tokens_edited |
0 |
num_epochs |
2 |
Primary Unlearning Metrics (held-out test, MC protocol)
Headline scores used for checkpoint selection:
| Metric | Train (after unlearning) | Test (after unlearning) |
|---|---|---|
| Efficacy | 1 | 0.764 |
| Specificity | 0.795 | 0.81 |
| Harmonic mean | 0.886 | 0.786 |
| Relearning QA (MC) | — | 0.66 |
Full Evaluation (baseline → unlearned)
From evaluation/score_comparison.csv:
| Metric | Baseline (train) | After unlearn (train) | Baseline (test) | After unlearn (test) |
|---|---|---|---|---|
| QA accuracy | 0.76 | 0.24 | 0.8 | 0.38 |
| QA fraction | 1 | 0 | 1 | 0.236 |
| SimDom accuracy | 0.72 | 0.56 | 0.68 | 0.56 |
| SimDom fraction | 1 | 0.66 | 1 | 0.721 |
| MMLU accuracy | 0.62 | 0.62 | 0.65 | 0.62 |
| MMLU fraction | 1 | 1 | 1 | 0.925 |
Files in This Repository
| File | Description |
|---|---|
unlearned_checkpoints.json |
Checkpoint metadata & hyperparameters |
evaluation/evaluation_summary.json |
Full evaluation payload (train/test/relearning) |
evaluation/score_comparison.csv |
Baseline vs. unlearned comparison table |
- Downloads last month
- 8
Model tree for shirasko/llama-3.1-8b-instruct-crisp-uranium
Base model
meta-llama/Llama-3.1-8B Finetuned
meta-llama/Llama-3.1-8B-Instruct