reports/pruning_eval_progress.md

# Pruning Evaluation Progress

Last updated: 2026-05-30 UTC.

## Scope

Evaluation cache: `artifacts/data/calibration/c0543d66aa2199f7/chunks.npy`

Baseline: BF16 `poolside/Laguna-XS.2`, full 4k cache, 1024 chunks.

Importance artifact: `artifacts/runs/heapr_paper_128x2048_4xl40s_alllayers_fused_device_b4/atomic_scores.npy`

## Decision Summary

The primary quality experiment preserves Laguna's static/shared expert and original top-8 routed-expert
router. It zeroes importance-ranked 64-atom blocks only inside routed expert tensors.

| Native grouped pruning | Full-cache loss | PPL delta | GSM8K-CoT strict EM | Interpretation |
| --- | ---: | ---: | ---: | --- |
| BF16 baseline | 2.347142 | 0.00% | 0.931818 | unpruned anchor |
| 10% | 2.363612 | +1.66% | 0.909091 | strongest deployment candidate |
| 20% | 2.416442 | +7.18% | 0.780303 | upper-bound tradeoff point |
| 25% | 2.466706 | +12.70% | 0.681818 | stress point; quality loss is substantial |
| 40% | 2.683520 | +39.99% | not run | loss indicates excessive degradation |

GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
decoding for every row.

| Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
| --- | ---: | ---: | ---: | ---: |
| BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
| 10% | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
| 20% | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
| 25% | 2.466706 | 0.725965 | 0.666350 | 0.681818 |

## Loss / Perplexity

| Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| BF16 baseline | unpruned | 2.347443 | 0.000000 | 10.458797 | 0.00% | `artifacts/runs/baselines/bf16_4k_full_4xl40s_loss.json` |
| BF16 baseline, explicit static cache | unpruned | 2.347142 | 0.000000 | 10.455647 | 0.00% | `artifacts/runs/baselines/bf16_4k_full_4xl40s_static_cache_b4_loss.json` |
| Atomic 10% | lowest atomic scores zeroed globally | 2.362229 | +0.014786 | 10.614584 | +1.49% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_10pct_loss.json` |
| Atomic 20% | lowest atomic scores zeroed globally | 2.412423 | +0.064979 | 11.160969 | +6.71% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_20pct_loss.json` |
| Atomic 40% | lowest atomic scores zeroed globally | 2.705508 | +0.358065 | 14.961917 | +43.06% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_40pct_loss.json` |
| Group 10% diagnostic | importance-sorted 64-atom groups zeroed inside original parent routing | 2.362365 | +0.014922 | 10.616028 | +1.50% | `artifacts/runs/pruned_loss_full_4k_4xl40s/group_pruned_10pct_loss.json` |
| Repacked group 10% | independently routed 64-atom child groups, fixed top-64 backfill | 2.404768 | +0.057626 | 11.075862 | +5.93% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss.json` |
| Repacked group 20% | independently routed 64-atom child groups, fixed top-64 backfill | 2.517318 | +0.170176 | 12.395307 | +18.55% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_20pct_loss.json` |
| Repacked group 40% | independently routed 64-atom child groups, fixed top-64 backfill | 2.928546 | +0.581404 | 18.700425 | +78.85% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_40pct_loss.json` |
| Native grouped 10% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.363612 | +0.016469 | 10.629271 | +1.66% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_10pct_loss.json` |
| Random native grouped 10% | random seed 20260530; original top-8 parent routing; same minimum-retention rule | 2.480595 | +0.133453 | 11.948376 | +14.28% | `artifacts/runs/random_group_10pct/group_pruned_10pct_loss.json` |
| Native grouped 20% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.416442 | +0.069300 | 11.205921 | +7.18% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_20pct_loss.json` |
| Native grouped 25% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.466706 | +0.119564 | 11.783569 | +12.70% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_25pct_loss.json` |
| Random native grouped 25% | random seed 20260530; original top-8 parent routing; same minimum-retention rule | 2.561328 | +0.214186 | 12.953011 | +23.89% | `artifacts/runs/random_group_25pct/group_pruned_25pct_loss.json` |
| Native grouped 40% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.683520 | +0.336378 | 14.636528 | +39.99% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_40pct_loss.json` |

## Expanded Grouped Router Validation

| Run | Chunks | Mean loss | Perplexity | Artifact |
| --- | ---: | ---: | ---: | --- |
| BF16 baseline, first full-cache 16 chunks | 16 | 2.475593 | 11.888757 | `artifacts/runs/baselines/bf16_4k_full_first16_4xl40s_loss.json` |
| Repacked grouped, 0% pruned, first full-cache 16 chunks | 16 | 2.477158 | 11.907380 | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_00pct_loss_pilot16.json` |
| Repacked grouped, 10% pruned, first full-cache 16 chunks | 16 | 2.515994 | 12.378907 | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss_pilot16.json` |

## Native Grouped Pruning Pilot

The primary grouped experiment now preserves Laguna's original top-8 parent-expert router and native
forward. Pruned 64-atom groups are zeroed inside each original expert tensor. This measures grouped
structural-pruning quality soundly, although the current native kernel does not realize the potential
deployment speedup.

Per-layer 25% mask statistics: `artifacts/reports/native_group_pruning_stats.md`

| Run | Chunks | Mean loss | Perplexity | Artifact |
| --- | ---: | ---: | ---: | --- |
| Native grouped 10% | 16 | 2.482592 | 11.972254 | `artifacts/runs/native_group_loss_pilot16/group_pruned_10pct_loss.json` |
| Native grouped 20% | 16 | 2.527117 | 12.517363 | `artifacts/runs/native_group_loss_pilot16/group_pruned_20pct_loss.json` |
| Native grouped 40% | 16 | 2.765421 | 15.885724 | `artifacts/runs/native_group_loss_pilot16/group_pruned_40pct_loss.json` |

## Downstream Evaluation

| Run | Scope | Result | Artifact |
| --- | --- | --- | --- |
| Full MMLU 0-shot baseline smoke | 0.1% limit; 58 subject samples | harness completed; accuracy 0.879310 | `artifacts/runs/downstream/mmlu_baseline_smoke.json` |
| Full MMLU 0-shot baseline | full 14,042 samples | accuracy 0.733514; STEM slice 0.702188 | `artifacts/runs/downstream/mmlu_baseline_full.json` |
| Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
| Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
| Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
| Full MMLU 0-shot native grouped 10% | full 14,042 samples | accuracy 0.735437; delta +0.001923 | `artifacts/runs/downstream/mmlu_native_group_10pct_full.json` |
| Full MMLU 0-shot native grouped 20% | full 14,042 samples | accuracy 0.734012; delta +0.000499 | `artifacts/runs/downstream/mmlu_native_group_20pct_full.json` |
| Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
| MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
| MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
| MMLU-STEM 5-shot native grouped 20% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.679036; delta -0.011735 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_20pct_full.json` |
| MMLU-STEM 5-shot native grouped 25% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.666350; delta -0.024421 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_25pct_full.json` |
| GSM8K-CoT 8-shot baseline | first 10%; 132 paired samples | strict EM 0.931818; flexible EM 0.886364 | `artifacts/runs/downstream/gsm8k_cot_baseline_10pct.json` |
| GSM8K-CoT 8-shot native grouped 10% | first 10%; 132 paired samples | strict EM 0.909091; delta -0.022727; flexible EM 0.863636; delta -0.022727 | `artifacts/runs/downstream/gsm8k_cot_native_group_10pct_10pct.json` |
| GSM8K-CoT 8-shot native grouped 20% | first 10%; 132 paired samples | strict EM 0.780303; delta -0.151515; flexible EM 0.742424; delta -0.143939 | `artifacts/runs/downstream/gsm8k_cot_native_group_20pct_10pct.json` |
| GSM8K-CoT 8-shot native grouped 25% | first 10%; 132 paired samples | strict EM 0.681818; delta -0.250000; flexible EM 0.689394; delta -0.196970 | `artifacts/runs/downstream/gsm8k_cot_native_group_25pct_10pct.json` |
| CRUXEval-O CoT baseline smoke | one function; 10 sampled generations; executable pass@1 | harness completed; pass@1 0.0 | `artifacts/runs/downstream/cruxeval_output_cot_baseline_smoke.json` |
| SWE-bench Lite baseline smoke | one oracle-prepared instance: `astropy__astropy-12907` | official grader completed; generated patch failed to apply | `artifacts/runs/downstream/swebench_baseline_smoke_predictions.jsonl` |

The SWE-bench smoke validates generation, prediction serialization, Docker availability, and official harness invocation. It is not a quality estimate: the single generated continuation produced a malformed patch and the grader recorded one patch-apply error. Further SWE-bench work is paused while the basic native-grouped benchmarks run.

Additional small benchmark selected from the Laguna technical-report table: `gsm8k_cot`. The installed
`lm-eval` task is configured for the report-aligned 8-shot chain-of-thought exact-match setup. Run it
for the baseline and native-grouped 25% variant after MMLU. Because a full run takes more than one hour
per model, use the same first 10% subset for both variants. Runs explicitly cap generation at 256 new
tokens because Laguna's shipped generation config otherwise requests up to 4096.

The report-metric-aligned local MMLU-STEM 5-shot baseline is `0.690771`, below the Laguna technical
report's `0.781`. The local `lm-eval` comparison remains useful because the unpruned and pruned models
run through the same harness, but it does not exactly reproduce the report's evaluation stack.

The GSM8K-CoT comparison is likewise a matched local-harness comparison rather than an exact
reproduction of Poolside's internal framework. The local task uses eight fixed chain-of-thought
demonstrations and greedy decoding (`do_sample=False`). An audit of the paired 10% run confirmed
identical prompts and test examples. The 256-token cap did not bind: generated responses were short,
and the grouped-model errors were predominantly incorrect arithmetic answers rather than truncations.

CRUXEval-O CoT executable grading was smoke-tested successfully. The paired coding subset was stopped
after `84 / 400` baseline generations when the remaining session was time-boxed. The smoke validates
the local generation and executable-grader path but is not a coding-quality estimate.

## Grouped Compute Note

For grouped pruning, the simple deployment interpretation is the pruned group fraction. The prioritized
25% experiment removes exactly 19,968 of 79,872 64-wide groups. That is the meaningful reduction if a
grouped MoE kernel only materializes retained child experts.

The primary native grouped evaluator preserves the original top-8 parent router and zeroes pruned
groups in place. Its current kernel still computes those zeroed blocks, so observed runtime is a
quality-measurement harness rather than a realized speedup. The separately reported repacked-child
exploration uses fixed top-64 backfill and is not the primary result.

## Method Notes

- Atomic pruning was evaluated in memory by loading the BF16 model, zeroing pruned atom rows/columns in the packed expert tensors, and running the native Laguna forward on the full 4k cache.
- Current fixed-shape loss runs pass an explicit Hugging Face `StaticCache` into each direct forward call with `use_cache=True`, avoiding Laguna's internal dynamic-cache construction path.
- Atomic and diagnostic-group deltas above use the earlier BF16 baseline. Repacked-group deltas use the explicit-static-cache BF16 baseline, matching those runs' cache path.
- The grouped 10% diagnostic is not the full grouped mini-expert routing experiment. It keeps the original parent-expert router/top-8 behavior and only zeroes 64-wide groups inside selected parent experts.
- That original-router grouped path is now the primary experimental quality measurement: it directly applies 64-atom structural pruning while preserving the source model's routing semantics.
- For grouped mini-experts as independently routable child experts, the evaluator needs to route over child groups. A plausible inherited-router test is: expand each parent expert score to its 8 child groups, mask pruned child groups, select top 64 child groups per token, and evaluate those group contributions. Under no pruning this collapses to the original top-8 parents; under group pruning it can select child groups from more than 8 parent experts.
- The repacked grouped runtime now implements that inherited-router test by repacking each sparse layer from 256 parent experts x 512 atoms into 2048 child experts x 64 atoms, expanding router rows to child groups, selecting top 64 child groups, and assigning each selected group its parent expert's normalized router weight. The 0% pruned first-16 validation above checks that this preserves the unpruned model.

## Current Status

- Completed: BF16 full-cache baseline.
- Completed: atomic 10%, 20%, and 40% full-cache loss.
- Completed but diagnostic-only: constrained grouped 10% full-cache loss.
- Stopped: constrained grouped 20%/40%, because it does not answer the independently-routable grouped mini-expert question.
- Completed: repacked grouped 10% full-cache loss with explicit static cache.
- Completed: repacked grouped 20% full-cache loss with explicit static cache.
- Completed: repacked grouped 40% full-cache loss with explicit static cache.
- Completed: unpruned full-cache loss with explicit static cache and directly comparable grouped deltas.
- Completed: baseline MMLU and SWE-bench Lite smoke paths.
- Completed: full baseline and atomic 10%/20%/40% MMLU comparison runs.
- Stopped: full repacked-group MMLU. Repacked child routing over-selects groups for the primary experiment and remains exploratory only.
- Completed: native original-router grouped 10%/20%/40% loss pilot on 16 chunks.
- Completed: native original-router grouped 10%/20%/40% full-cache loss.
- Priority changed by request: evaluate baseline versus native original-router grouped 25% only.
- Completed: native original-router grouped 25% full-cache loss.
- Completed: full-MMLU 0-shot native grouped 25%. This is useful auxiliary evidence but does not match the tech-report MMLU-STEM 5-shot row.
- Completed: report-metric-aligned local MMLU-STEM 5-shot baseline and native grouped 25%.
- Stopped: full GSM8K-CoT baseline after runtime projection exceeded one hour.
- Completed: paired first-10% GSM8K-CoT baseline and native grouped 25%.
- Completed: paired first-10% GSM8K-CoT native grouped 10% and 20% curve points.
- Stopped by request: CRUXEval-O CoT paired subset after executable-grader smoke validation, due to the remaining time box.
- Completed: random native grouped 25% full-cache loss control with seed 20260530. It is +0.094622 loss worse than importance-ranked 25% pruning.
- Completed: random native grouped 10% full-cache loss control with seed 20260530. It is +0.116984 loss worse than importance-ranked 10% pruning.
- Skipped from the grouped MMLU downstream matrix: native grouped 10%, 20%, and 40%.
- Paused by request: further SWE-bench evaluation.
- Validated: full test suite `28 passed`; Python compilation and `git diff --check` passed.