nikgeo commited on
Commit
39f8d71
·
verified ·
1 Parent(s): 541d5f8

Publish full native-grouped MMLU results and raise overview table

Browse files

Move the results-at-a-glance table directly below the opening pitch and above the caveat. Fill the completed native-grouped 10% and 20% full-MMLU values, add ratio-labelled downstream rows, sync the compact summary, and publish the latest report.

README.md CHANGED
@@ -25,6 +25,20 @@ identifies **10% of Laguna XS.2's routed MoE parameters** for removal while pres
25
  structure designed for efficient deployment kernels. That is **3.14B removable parameters**, or
26
  **9.39% of the full 33.44B-parameter model**.
27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  > [!IMPORTANT]
29
  > The current artifact measures pruning quality, not deployment speed. It applies the structured
30
  > pruning mask by zeroing blocks inside the original tensors, so the released Laguna kernel still
@@ -38,21 +52,6 @@ structured removal: `3,140,616,192` parameters, or `9.39%` of the full model. Fu
38
  perplexity moves from `10.455647` to `10.629271` (`+1.66%`), while strict GSM8K-CoT exact match on
39
  a paired first-10% subset moves from `0.931818` to `0.909091`.
40
 
41
- ## Results at a glance
42
-
43
- | Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
44
- | --- | ---: | ---: | ---: | ---: |
45
- | BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
46
- | 10% pruning | 2.363612 | not run | 0.693942 | 0.909091 |
47
- | 20% pruning | 2.416442 | not run | 0.679036 | 0.780303 |
48
-
49
- Loss uses the full held-out 4k cache. GSM8K-CoT uses the paired first-10% subset (`132` examples).
50
- Native-grouped full MMLU 0-shot was run for the baseline and the exploratory 25% stress point.
51
- Native-grouped MMLU-STEM 5-shot was also run for the 10% and 20% masks. Accuracy changes from
52
- `0.690771` to `0.693942` (`+0.003172`) at 10%, then to `0.679036` (`-0.011735`) at 20%. We treat
53
- the small positive 10% delta as retained quality, not evidence that pruning improves the model.
54
- The separate atomic-pruning MMLU sweep is not substituted here.
55
-
56
  ## What structured pruning means here
57
 
58
  Laguna routes tokens to parent experts. Inside each parent expert, the computation can be decomposed
@@ -215,7 +214,9 @@ more sensitive:
215
 
216
  | Benchmark | BF16 baseline | Structured-pruning variant | Delta |
217
  | --- | ---: | ---: | ---: |
218
- | Full MMLU 0-shot accuracy | 0.733514 | 0.725965 | -0.007549 |
 
 
219
  | MMLU-STEM 5-shot accuracy, 10% pruning | 0.690771 | 0.693942 | +0.003172 |
220
  | MMLU-STEM 5-shot accuracy, 20% pruning | 0.690771 | 0.679036 | -0.011735 |
221
  | MMLU-STEM 5-shot accuracy, 25% pruning | 0.690771 | 0.666350 | -0.024421 |
@@ -305,7 +306,7 @@ uv run --extra dev python scripts/plot_pruning_insights.py
305
  ## Status
306
 
307
  - Completed: native grouped loss sweep at 10%, 20%, 25%, and 40%.
308
- - Completed: full baseline and native-grouped 25% MMLU.
309
  - Completed: exploratory repacked child-routing loss sweep.
310
  - Completed: paired first-10% GSM8K-CoT baseline and native-grouped 10%, 20%, and 25% curve.
311
  - Stopped after smoke validation: paired CRUXEval-O CoT comparison, due to the remaining time box.
 
25
  structure designed for efficient deployment kernels. That is **3.14B removable parameters**, or
26
  **9.39% of the full 33.44B-parameter model**.
27
 
28
+ ## Results at a glance
29
+
30
+ | Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
31
+ | --- | ---: | ---: | ---: | ---: |
32
+ | BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
33
+ | 10% pruning | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
34
+ | 20% pruning | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
35
+
36
+ Loss uses the full held-out 4k cache. GSM8K-CoT uses the paired first-10% subset (`132` examples).
37
+ Full MMLU 0-shot remains effectively flat at 10% and 20%. Native-grouped MMLU-STEM 5-shot changes
38
+ from `0.690771` to `0.693942` (`+0.003172`) at 10%, then to `0.679036` (`-0.011735`) at 20%.
39
+ We treat the small positive deltas as retained quality, not evidence that pruning improves the
40
+ model. The separate atomic-pruning MMLU sweep is not substituted here.
41
+
42
  > [!IMPORTANT]
43
  > The current artifact measures pruning quality, not deployment speed. It applies the structured
44
  > pruning mask by zeroing blocks inside the original tensors, so the released Laguna kernel still
 
52
  perplexity moves from `10.455647` to `10.629271` (`+1.66%`), while strict GSM8K-CoT exact match on
53
  a paired first-10% subset moves from `0.931818` to `0.909091`.
54
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  ## What structured pruning means here
56
 
57
  Laguna routes tokens to parent experts. Inside each parent expert, the computation can be decomposed
 
214
 
215
  | Benchmark | BF16 baseline | Structured-pruning variant | Delta |
216
  | --- | ---: | ---: | ---: |
217
+ | Full MMLU 0-shot accuracy, 10% pruning | 0.733514 | 0.735437 | +0.001923 |
218
+ | Full MMLU 0-shot accuracy, 20% pruning | 0.733514 | 0.734012 | +0.000499 |
219
+ | Full MMLU 0-shot accuracy, 25% pruning | 0.733514 | 0.725965 | -0.007549 |
220
  | MMLU-STEM 5-shot accuracy, 10% pruning | 0.690771 | 0.693942 | +0.003172 |
221
  | MMLU-STEM 5-shot accuracy, 20% pruning | 0.690771 | 0.679036 | -0.011735 |
222
  | MMLU-STEM 5-shot accuracy, 25% pruning | 0.690771 | 0.666350 | -0.024421 |
 
306
  ## Status
307
 
308
  - Completed: native grouped loss sweep at 10%, 20%, 25%, and 40%.
309
+ - Completed: full baseline and native-grouped 10%, 20%, and 25% MMLU.
310
  - Completed: exploratory repacked child-routing loss sweep.
311
  - Completed: paired first-10% GSM8K-CoT baseline and native-grouped 10%, 20%, and 25% curve.
312
  - Stopped after smoke validation: paired CRUXEval-O CoT comparison, due to the remaining time box.
reports/pruning_eval_progress.md CHANGED
@@ -26,6 +26,13 @@ router. It zeroes importance-ranked 64-atom blocks only inside routed expert ten
26
  GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
27
  decoding for every row.
28
 
 
 
 
 
 
 
 
29
  ## Loss / Perplexity
30
 
31
  | Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
@@ -78,6 +85,8 @@ Per-layer 25% mask statistics: `artifacts/reports/native_group_pruning_stats.md`
78
  | Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
79
  | Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
80
  | Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
 
 
81
  | Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
82
  | MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
83
  | MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
 
26
  GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
27
  decoding for every row.
28
 
29
+ | Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
30
+ | --- | ---: | ---: | ---: | ---: |
31
+ | BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
32
+ | 10% | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
33
+ | 20% | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
34
+ | 25% | 2.466706 | 0.725965 | 0.666350 | 0.681818 |
35
+
36
  ## Loss / Perplexity
37
 
38
  | Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
 
85
  | Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
86
  | Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
87
  | Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
88
+ | Full MMLU 0-shot native grouped 10% | full 14,042 samples | accuracy 0.735437; delta +0.001923 | `artifacts/runs/downstream/mmlu_native_group_10pct_full.json` |
89
+ | Full MMLU 0-shot native grouped 20% | full 14,042 samples | accuracy 0.734012; delta +0.000499 | `artifacts/runs/downstream/mmlu_native_group_20pct_full.json` |
90
  | Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
91
  | MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
92
  | MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
results/summary.json CHANGED
@@ -172,7 +172,14 @@
172
  },
173
  "mmlu": {
174
  "num_examples": 14042,
175
- "num_fewshot": 0
 
 
 
 
 
 
 
176
  },
177
  "mmlu_stem_5shot": {
178
  "num_examples": 3153,
 
172
  },
173
  "mmlu": {
174
  "num_examples": 14042,
175
+ "num_fewshot": 0,
176
+ "baseline_accuracy": 0.733514,
177
+ "native_group_10pct_accuracy": 0.735437,
178
+ "native_group_10pct_delta": 0.001923,
179
+ "native_group_20pct_accuracy": 0.734012,
180
+ "native_group_20pct_delta": 0.000499,
181
+ "native_group_25pct_accuracy": 0.725965,
182
+ "native_group_25pct_delta": -0.007549
183
  },
184
  "mmlu_stem_5shot": {
185
  "num_examples": 3153,