| <!doctype html> |
| <html lang="en"><head><meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Sequential Forgetting Leaderboard</title> |
| <style> |
| :root { --bg:#ffffff; --fg:#1a1a1a; --muted:#6a6a6a; --line:#e3e3e3; --accent:#e17100; |
| --ok-bg:#e8f5e9; --ok-fg:#1b5e20; --bad-bg:#fdecea; --bad-fg:#8c1d18; --code:#f4f4f4; } |
| @media (prefers-color-scheme: dark) { |
| :root { --bg:#101010; --fg:#eaeaea; --muted:#9a9a9a; --line:#2b2b2b; --accent:#ff9e2c; |
| --ok-bg:#12331a; --ok-fg:#8fdf9f; --bad-bg:#3a1512; --bad-fg:#ff9d94; --code:#1d1d1d; } } |
| body { margin:0; background:var(--bg); color:var(--fg); |
| font:16px/1.55 system-ui,-apple-system,Segoe UI,Roboto,sans-serif; } |
| main { max-width:1100px; margin:0 auto; padding:2.5rem 1.25rem 4rem; } |
| h1 { font-size:1.7rem; margin:0 0 .3rem; } h2 { margin:2.2rem 0 .6rem; font-size:1.2rem; } |
| .sub { color:var(--muted); margin:0 0 1.4rem; } |
| a { color:var(--accent); text-decoration:none; } a:hover { text-decoration:underline; } |
| .tablewrap { overflow-x:auto; border:1px solid var(--line); border-radius:8px; } |
| table { border-collapse:collapse; width:100%; font-size:.85rem; } |
| th,td { text-align:left; padding:.5rem .65rem; border-bottom:1px solid var(--line); vertical-align:top; } |
| th { color:var(--muted); font-weight:600; white-space:nowrap; } |
| tr:last-child td { border-bottom:none; } |
| td.num { font-variant-numeric:tabular-nums; font-weight:600; white-space:nowrap; } |
| td.notes { color:var(--muted); min-width:220px; } |
| code { background:var(--code); padding:.1rem .35rem; border-radius:4px; font-size:.82em; } |
| .pill { padding:.12rem .5rem; border-radius:99px; font-size:.75rem; white-space:nowrap; } |
| .pill.ok { background:var(--ok-bg); color:var(--ok-fg); } |
| .pill.bad { background:var(--bad-bg); color:var(--bad-fg); } |
| footer { margin-top:3rem; color:var(--muted); font-size:.85rem; border-top:1px solid var(--line); padding-top:1rem; } |
| </style></head><body><main> |
| <h1>📉 Sequential Forgetting Leaderboard</h1> |
| <p class="sub">How much does sequential fine-tuning destroy what the model already learned? |
| Lower magnitude = better retention. <strong>Every number links to the raw run file</strong> in the |
| <a href="https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark">benchmark dataset</a>; transcriptions are hand-checked |
| (<a href="https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/PROVENANCE.md">provenance</a>). |
| Suites are <strong>not</strong> cross-comparable; each table ranks within its own protocol.</p> |
| <h2>Suite A — 5 real-world domains · Mistral-7B · 3 seeds</h2><div class='tablewrap'><table><thead><tr><th>method</th><th>base model</th><th>domains</th><th>seeds</th><th>metric</th><th>value %</th><th>status</th><th>source</th><th>notes</th></tr></thead><tbody><tr><td><code>modular_crma</code></td><td>Mistral-7B</td><td>5</td><td>3</td><td>holdout_NLL_drift</td><td class='num'>-0.166</td><td><span class='pill ok'>valid_multiseed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/multiseed_results_combined.json'>multiseed_results_combined.json</a></td><td class='notes'>per-task LoRA + CRMA backbone; drift of saved snapshots re-run under final backbone</td></tr><tr><td><code>frozen_base</code></td><td>Mistral-7B</td><td>5</td><td>3</td><td>holdout_NLL_drift</td><td class='num'>1.948</td><td><span class='pill ok'>valid_multiseed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/multiseed_results_combined.json'>multiseed_results_combined.json</a></td><td class='notes'>no adaptation control</td></tr><tr><td><code>naive_sequential_lora</code></td><td>Mistral-7B</td><td>5</td><td>3</td><td>holdout_NLL_forgetting</td><td class='num'>42.96</td><td><span class='pill ok'>valid_multiseed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/multiseed_results_combined.json'>multiseed_results_combined.json</a></td><td class='notes'>single LoRA trained sequentially across domains</td></tr></tbody></table></div><h2>Suite B — Medical→Legal→Code→Finance</h2><div class='tablewrap'><table><thead><tr><th>method</th><th>base model</th><th>domains</th><th>seeds</th><th>metric</th><th>value %</th><th>status</th><th>source</th><th>notes</th></tr></thead><tbody><tr><td><code>modular_crma</code></td><td>Mistral-7B-v0.3</td><td>4</td><td>1</td><td>holdout_NLL_drift</td><td class='num'>-0.1</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/ablation_v8.1_7b_results.md'>ablation_v8.1_7b_results.md</a></td><td class='notes'>avg of per-task drift table; NAIVE ref is the same run's forgetting avg</td></tr><tr><td><code>modular_crma</code></td><td>TinyLlama-1.1B-Chat-v1.0</td><td>4</td><td>1</td><td>holdout_NLL_drift</td><td class='num'>-0.1</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/ablation_v8.1_results.md'>ablation_v8.1_results.md</a></td><td class='notes'></td></tr><tr><td><code>naive_sequential_lora</code></td><td>TinyLlama-1.1B-Chat-v1.0</td><td>4</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>225.3</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/ablation_v8.1_results.md'>ablation_v8.1_results.md</a></td><td class='notes'></td></tr><tr><td><code>naive_sequential_lora</code></td><td>Mistral-7B-v0.3</td><td>4</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>351.4</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/ablation_v8.1_7b_results.md'>ablation_v8.1_7b_results.md</a></td><td class='notes'></td></tr></tbody></table></div><h2>Suite B history — CL-technique stacks (within-version comparisons only)</h2><div class='tablewrap'><table><thead><tr><th>method</th><th>base model</th><th>domains</th><th>seeds</th><th>metric</th><th>value %</th><th>status</th><th>source</th><th>notes</th></tr></thead><tbody><tr><td><code>cl_stack_v5_10component</code></td><td>TinyLlama-1.1B</td><td>4</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>58.4</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>10-component stack; post-data-fix suite - NAIVE dropped 185.8->88.8 from data fixes alone</td></tr><tr><td><code>cl_stack_v3_ewc_gradproj</code></td><td>TinyLlama-1.1B</td><td>4</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>91.3</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>EWC + gradient projection; pre-data-fix suite - compare only to its own NAIVE column</td></tr><tr><td><code>cl_stack_v7_replay_kd_freeze</code></td><td>Mistral-7B</td><td>4</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>109.3</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>replay + knowledge distillation + bottom-layer freeze, Mistral-7B</td></tr></tbody></table></div><h2>Suite C — MQuAKE 5-skill retention · Qwen3-4B</h2><div class='tablewrap'><table><thead><tr><th>method</th><th>base model</th><th>domains</th><th>seeds</th><th>metric</th><th>value %</th><th>status</th><th>source</th><th>notes</th></tr></thead><tbody><tr><td><code>modular_vault_slots</code></td><td>Qwen3-4B-Instruct-2507</td><td>5</td><td>1</td><td>accuracy_BWT</td><td class='num'>0.0</td><td><span class='pill ok'>valid_single_run</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/bwt_mquake_vault5_s42.json'>bwt_mquake_vault5_s42.json</a></td><td class='notes'>retention matrix R; BWT_k = R_final,k - R_k,k; mean over 4 earlier skills</td></tr></tbody></table></div><h2>Invalid & incomplete runs (disclosed)</h2><p>Buggy or unfinished runs are <strong>relabeled, not deleted</strong>. Highlight: our early O-LoRA arm appeared to win (−2.0% forgetting) until we found a gradient-clipping bug that had frozen the model — so O-LoRA is listed as <em>invalid, never validly measured here</em>, not as <em>beaten</em>.</p><div class='tablewrap'><table><thead><tr><th>method</th><th>base model</th><th>domains</th><th>seeds</th><th>metric</th><th>value %</th><th>status</th><th>source</th><th>notes</th></tr></thead><tbody><tr><td><code>cl_stack_v2_olora_ewc_gradproj_replay</code></td><td>TinyLlama-1.1B</td><td>2</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>-2.0</td><td><span class='pill bad'>invalid_disclosed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>INVALID: PiSSA-init O-LoRA grad norms ~126678 vs clip 100 (ratio 0.00079) froze the model; the -2.0% 'win' is an artifact. O-LoRA has never been validly measured here</td></tr><tr><td><code>cl_stack_v4_cumulbasis</code></td><td>TinyLlama-1.1B</td><td>2</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>27.8</td><td><span class='pill bad'>incomplete_disclosed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>INCOMPLETE: run cut off mid-Phase-3; Phase-2-only numbers</td></tr><tr><td><code>cl_stack_v6_sma</code></td><td>TinyLlama-1.1B</td><td>2</td><td>1</td><td>holdout_NLL_forgetting</td><td class='num'>61.5</td><td><span class='pill bad'>incomplete_disclosed</span></td><td><a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/results/raw/full_ablation_history_v2_v8.md'>full_ablation_history_v2_v8.md</a></td><td class='notes'>INCOMPLETE: Sparse Memory Adapter crashed (OOM) at Phase 3; Phase-2-only numbers</td></tr></tbody></table></div><h2>Submit your method</h2><ol><li>Run your method on a suite (<a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/protocol/PROTOCOL.md'>protocol</a>).</li><li>Score it with <a href='https://huggingface.co/datasets/ModelBrew/sequential-forgetting-benchmark/blob/main/scoring/score.py'><code>scoring/score.py</code></a> (<code>--nll</code> or <code>--matrix</code>).</li><li>Open a PR on the dataset repo adding your raw log, a <code>results.csv</code> row, and a provenance line.</li></ol><p>Single-seed submissions are accepted and labeled <code>valid_single_run</code>. If your run later turns out buggy, it moves to the disclosed section — that's the deal for everyone, including us.</p> |
| <footer>Maintained by <a href="https://modelbrew.ai">ModelBrew</a> — fine-tuning without catastrophic |
| forgetting (patent-pending CRMA adapters). The <code>modular_crma</code> rows are our method; |
| independent replications welcome.</footer> |
| </main></body></html> |
|
|
