Publish full native-grouped MMLU results and raise overview table
Browse filesMove the results-at-a-glance table directly below the opening pitch and above the caveat. Fill the completed native-grouped 10% and 20% full-MMLU values, add ratio-labelled downstream rows, sync the compact summary, and publish the latest report.
- README.md +18 -17
- reports/pruning_eval_progress.md +9 -0
- results/summary.json +8 -1
README.md
CHANGED
|
@@ -25,6 +25,20 @@ identifies **10% of Laguna XS.2's routed MoE parameters** for removal while pres
|
|
| 25 |
structure designed for efficient deployment kernels. That is **3.14B removable parameters**, or
|
| 26 |
**9.39% of the full 33.44B-parameter model**.
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
> [!IMPORTANT]
|
| 29 |
> The current artifact measures pruning quality, not deployment speed. It applies the structured
|
| 30 |
> pruning mask by zeroing blocks inside the original tensors, so the released Laguna kernel still
|
|
@@ -38,21 +52,6 @@ structured removal: `3,140,616,192` parameters, or `9.39%` of the full model. Fu
|
|
| 38 |
perplexity moves from `10.455647` to `10.629271` (`+1.66%`), while strict GSM8K-CoT exact match on
|
| 39 |
a paired first-10% subset moves from `0.931818` to `0.909091`.
|
| 40 |
|
| 41 |
-
## Results at a glance
|
| 42 |
-
|
| 43 |
-
| Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
|
| 44 |
-
| --- | ---: | ---: | ---: | ---: |
|
| 45 |
-
| BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
|
| 46 |
-
| 10% pruning | 2.363612 | not run | 0.693942 | 0.909091 |
|
| 47 |
-
| 20% pruning | 2.416442 | not run | 0.679036 | 0.780303 |
|
| 48 |
-
|
| 49 |
-
Loss uses the full held-out 4k cache. GSM8K-CoT uses the paired first-10% subset (`132` examples).
|
| 50 |
-
Native-grouped full MMLU 0-shot was run for the baseline and the exploratory 25% stress point.
|
| 51 |
-
Native-grouped MMLU-STEM 5-shot was also run for the 10% and 20% masks. Accuracy changes from
|
| 52 |
-
`0.690771` to `0.693942` (`+0.003172`) at 10%, then to `0.679036` (`-0.011735`) at 20%. We treat
|
| 53 |
-
the small positive 10% delta as retained quality, not evidence that pruning improves the model.
|
| 54 |
-
The separate atomic-pruning MMLU sweep is not substituted here.
|
| 55 |
-
|
| 56 |
## What structured pruning means here
|
| 57 |
|
| 58 |
Laguna routes tokens to parent experts. Inside each parent expert, the computation can be decomposed
|
|
@@ -215,7 +214,9 @@ more sensitive:
|
|
| 215 |
|
| 216 |
| Benchmark | BF16 baseline | Structured-pruning variant | Delta |
|
| 217 |
| --- | ---: | ---: | ---: |
|
| 218 |
-
| Full MMLU 0-shot accuracy | 0.733514 | 0.
|
|
|
|
|
|
|
| 219 |
| MMLU-STEM 5-shot accuracy, 10% pruning | 0.690771 | 0.693942 | +0.003172 |
|
| 220 |
| MMLU-STEM 5-shot accuracy, 20% pruning | 0.690771 | 0.679036 | -0.011735 |
|
| 221 |
| MMLU-STEM 5-shot accuracy, 25% pruning | 0.690771 | 0.666350 | -0.024421 |
|
|
@@ -305,7 +306,7 @@ uv run --extra dev python scripts/plot_pruning_insights.py
|
|
| 305 |
## Status
|
| 306 |
|
| 307 |
- Completed: native grouped loss sweep at 10%, 20%, 25%, and 40%.
|
| 308 |
-
- Completed: full baseline and native-grouped 25% MMLU.
|
| 309 |
- Completed: exploratory repacked child-routing loss sweep.
|
| 310 |
- Completed: paired first-10% GSM8K-CoT baseline and native-grouped 10%, 20%, and 25% curve.
|
| 311 |
- Stopped after smoke validation: paired CRUXEval-O CoT comparison, due to the remaining time box.
|
|
|
|
| 25 |
structure designed for efficient deployment kernels. That is **3.14B removable parameters**, or
|
| 26 |
**9.39% of the full 33.44B-parameter model**.
|
| 27 |
|
| 28 |
+
## Results at a glance
|
| 29 |
+
|
| 30 |
+
| Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
|
| 31 |
+
| --- | ---: | ---: | ---: | ---: |
|
| 32 |
+
| BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
|
| 33 |
+
| 10% pruning | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
|
| 34 |
+
| 20% pruning | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
|
| 35 |
+
|
| 36 |
+
Loss uses the full held-out 4k cache. GSM8K-CoT uses the paired first-10% subset (`132` examples).
|
| 37 |
+
Full MMLU 0-shot remains effectively flat at 10% and 20%. Native-grouped MMLU-STEM 5-shot changes
|
| 38 |
+
from `0.690771` to `0.693942` (`+0.003172`) at 10%, then to `0.679036` (`-0.011735`) at 20%.
|
| 39 |
+
We treat the small positive deltas as retained quality, not evidence that pruning improves the
|
| 40 |
+
model. The separate atomic-pruning MMLU sweep is not substituted here.
|
| 41 |
+
|
| 42 |
> [!IMPORTANT]
|
| 43 |
> The current artifact measures pruning quality, not deployment speed. It applies the structured
|
| 44 |
> pruning mask by zeroing blocks inside the original tensors, so the released Laguna kernel still
|
|
|
|
| 52 |
perplexity moves from `10.455647` to `10.629271` (`+1.66%`), while strict GSM8K-CoT exact match on
|
| 53 |
a paired first-10% subset moves from `0.931818` to `0.909091`.
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## What structured pruning means here
|
| 56 |
|
| 57 |
Laguna routes tokens to parent experts. Inside each parent expert, the computation can be decomposed
|
|
|
|
| 214 |
|
| 215 |
| Benchmark | BF16 baseline | Structured-pruning variant | Delta |
|
| 216 |
| --- | ---: | ---: | ---: |
|
| 217 |
+
| Full MMLU 0-shot accuracy, 10% pruning | 0.733514 | 0.735437 | +0.001923 |
|
| 218 |
+
| Full MMLU 0-shot accuracy, 20% pruning | 0.733514 | 0.734012 | +0.000499 |
|
| 219 |
+
| Full MMLU 0-shot accuracy, 25% pruning | 0.733514 | 0.725965 | -0.007549 |
|
| 220 |
| MMLU-STEM 5-shot accuracy, 10% pruning | 0.690771 | 0.693942 | +0.003172 |
|
| 221 |
| MMLU-STEM 5-shot accuracy, 20% pruning | 0.690771 | 0.679036 | -0.011735 |
|
| 222 |
| MMLU-STEM 5-shot accuracy, 25% pruning | 0.690771 | 0.666350 | -0.024421 |
|
|
|
|
| 306 |
## Status
|
| 307 |
|
| 308 |
- Completed: native grouped loss sweep at 10%, 20%, 25%, and 40%.
|
| 309 |
+
- Completed: full baseline and native-grouped 10%, 20%, and 25% MMLU.
|
| 310 |
- Completed: exploratory repacked child-routing loss sweep.
|
| 311 |
- Completed: paired first-10% GSM8K-CoT baseline and native-grouped 10%, 20%, and 25% curve.
|
| 312 |
- Stopped after smoke validation: paired CRUXEval-O CoT comparison, due to the remaining time box.
|
reports/pruning_eval_progress.md
CHANGED
|
@@ -26,6 +26,13 @@ router. It zeroes importance-ranked 64-atom blocks only inside routed expert ten
|
|
| 26 |
GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
|
| 27 |
decoding for every row.
|
| 28 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
## Loss / Perplexity
|
| 30 |
|
| 31 |
| Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
|
|
@@ -78,6 +85,8 @@ Per-layer 25% mask statistics: `artifacts/reports/native_group_pruning_stats.md`
|
|
| 78 |
| Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
|
| 79 |
| Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
|
| 80 |
| Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
|
|
|
|
|
|
|
| 81 |
| Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
|
| 82 |
| MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
|
| 83 |
| MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
|
|
|
|
| 26 |
GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
|
| 27 |
decoding for every row.
|
| 28 |
|
| 29 |
+
| Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
|
| 30 |
+
| --- | ---: | ---: | ---: | ---: |
|
| 31 |
+
| BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
|
| 32 |
+
| 10% | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
|
| 33 |
+
| 20% | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
|
| 34 |
+
| 25% | 2.466706 | 0.725965 | 0.666350 | 0.681818 |
|
| 35 |
+
|
| 36 |
## Loss / Perplexity
|
| 37 |
|
| 38 |
| Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
|
|
|
|
| 85 |
| Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
|
| 86 |
| Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
|
| 87 |
| Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
|
| 88 |
+
| Full MMLU 0-shot native grouped 10% | full 14,042 samples | accuracy 0.735437; delta +0.001923 | `artifacts/runs/downstream/mmlu_native_group_10pct_full.json` |
|
| 89 |
+
| Full MMLU 0-shot native grouped 20% | full 14,042 samples | accuracy 0.734012; delta +0.000499 | `artifacts/runs/downstream/mmlu_native_group_20pct_full.json` |
|
| 90 |
| Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
|
| 91 |
| MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
|
| 92 |
| MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
|
results/summary.json
CHANGED
|
@@ -172,7 +172,14 @@
|
|
| 172 |
},
|
| 173 |
"mmlu": {
|
| 174 |
"num_examples": 14042,
|
| 175 |
-
"num_fewshot": 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
},
|
| 177 |
"mmlu_stem_5shot": {
|
| 178 |
"num_examples": 3153,
|
|
|
|
| 172 |
},
|
| 173 |
"mmlu": {
|
| 174 |
"num_examples": 14042,
|
| 175 |
+
"num_fewshot": 0,
|
| 176 |
+
"baseline_accuracy": 0.733514,
|
| 177 |
+
"native_group_10pct_accuracy": 0.735437,
|
| 178 |
+
"native_group_10pct_delta": 0.001923,
|
| 179 |
+
"native_group_20pct_accuracy": 0.734012,
|
| 180 |
+
"native_group_20pct_delta": 0.000499,
|
| 181 |
+
"native_group_25pct_accuracy": 0.725965,
|
| 182 |
+
"native_group_25pct_delta": -0.007549
|
| 183 |
},
|
| 184 |
"mmlu_stem_5shot": {
|
| 185 |
"num_examples": 3153,
|