Hugging Face's logo

poolside-laguna-hackathon
/
laguna-martini

Text Generation
laguna
laguna-xs.2
poolside
Mixture of Experts
pruning
heapr
inference
Model card Files
xet
 Community
laguna-martini / reports /pruning_eval_progress.md
nikgeo's picture
nikgeo
Publish full native-grouped MMLU results and raise overview table
39f8d71 verified
preview code
|
Raw
History Blame Contribute Delete
16.4 kB

Pruning Evaluation Progress

Last updated: 2026-05-30 UTC.

Scope

Evaluation cache: artifacts/data/calibration/c0543d66aa2199f7/chunks.npy

Baseline: BF16 poolside/Laguna-XS.2, full 4k cache, 1024 chunks.

Importance artifact: artifacts/runs/heapr_paper_128x2048_4xl40s_alllayers_fused_device_b4/atomic_scores.npy

Decision Summary

The primary quality experiment preserves Laguna's static/shared expert and original top-8 routed-expert router. It zeroes importance-ranked 64-atom blocks only inside routed expert tensors.

Native grouped pruning Full-cache loss PPL delta GSM8K-CoT strict EM Interpretation
BF16 baseline 2.347142 0.00% 0.931818 unpruned anchor
10% 2.363612 +1.66% 0.909091 strongest deployment candidate
20% 2.416442 +7.18% 0.780303 upper-bound tradeoff point
25% 2.466706 +12.70% 0.681818 stress point; quality loss is substantial
40% 2.683520 +39.99% not run loss indicates excessive degradation

GSM8K-CoT values use the same first 10% subset (132 examples), fixed eight-shot prompt, and greedy decoding for every row.

Native grouped pruning Full-cache loss Full MMLU 0-shot accuracy MMLU-STEM 5-shot accuracy GSM8K-CoT 8-shot strict EM
BF16 baseline 2.347142 0.733514 0.690771 0.931818
10% 2.363612 0.735437 0.693942 0.909091
20% 2.416442 0.734012 0.679036 0.780303
25% 2.466706 0.725965 0.666350 0.681818

Loss / Perplexity

Run Pruning interpretation Mean loss Delta loss Perplexity PPL delta Artifact
BF16 baseline unpruned 2.347443 0.000000 10.458797 0.00% artifacts/runs/baselines/bf16_4k_full_4xl40s_loss.json
BF16 baseline, explicit static cache unpruned 2.347142 0.000000 10.455647 0.00% artifacts/runs/baselines/bf16_4k_full_4xl40s_static_cache_b4_loss.json
Atomic 10% lowest atomic scores zeroed globally 2.362229 +0.014786 10.614584 +1.49% artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_10pct_loss.json
Atomic 20% lowest atomic scores zeroed globally 2.412423 +0.064979 11.160969 +6.71% artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_20pct_loss.json
Atomic 40% lowest atomic scores zeroed globally 2.705508 +0.358065 14.961917 +43.06% artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_40pct_loss.json
Group 10% diagnostic importance-sorted 64-atom groups zeroed inside original parent routing 2.362365 +0.014922 10.616028 +1.50% artifacts/runs/pruned_loss_full_4k_4xl40s/group_pruned_10pct_loss.json
Repacked group 10% independently routed 64-atom child groups, fixed top-64 backfill 2.404768 +0.057626 11.075862 +5.93% artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss.json
Repacked group 20% independently routed 64-atom child groups, fixed top-64 backfill 2.517318 +0.170176 12.395307 +18.55% artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_20pct_loss.json
Repacked group 40% independently routed 64-atom child groups, fixed top-64 backfill 2.928546 +0.581404 18.700425 +78.85% artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_40pct_loss.json
Native grouped 10% original top-8 parent routing; pruned 64-atom groups zeroed in place 2.363612 +0.016469 10.629271 +1.66% artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_10pct_loss.json
Random native grouped 10% random seed 20260530; original top-8 parent routing; same minimum-retention rule 2.480595 +0.133453 11.948376 +14.28% artifacts/runs/random_group_10pct/group_pruned_10pct_loss.json
Native grouped 20% original top-8 parent routing; pruned 64-atom groups zeroed in place 2.416442 +0.069300 11.205921 +7.18% artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_20pct_loss.json
Native grouped 25% original top-8 parent routing; pruned 64-atom groups zeroed in place 2.466706 +0.119564 11.783569 +12.70% artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_25pct_loss.json
Random native grouped 25% random seed 20260530; original top-8 parent routing; same minimum-retention rule 2.561328 +0.214186 12.953011 +23.89% artifacts/runs/random_group_25pct/group_pruned_25pct_loss.json
Native grouped 40% original top-8 parent routing; pruned 64-atom groups zeroed in place 2.683520 +0.336378 14.636528 +39.99% artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_40pct_loss.json

Expanded Grouped Router Validation

Run Chunks Mean loss Perplexity Artifact
BF16 baseline, first full-cache 16 chunks 16 2.475593 11.888757 artifacts/runs/baselines/bf16_4k_full_first16_4xl40s_loss.json
Repacked grouped, 0% pruned, first full-cache 16 chunks 16 2.477158 11.907380 artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_00pct_loss_pilot16.json
Repacked grouped, 10% pruned, first full-cache 16 chunks 16 2.515994 12.378907 artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss_pilot16.json

Native Grouped Pruning Pilot

The primary grouped experiment now preserves Laguna's original top-8 parent-expert router and native forward. Pruned 64-atom groups are zeroed inside each original expert tensor. This measures grouped structural-pruning quality soundly, although the current native kernel does not realize the potential deployment speedup.

Per-layer 25% mask statistics: artifacts/reports/native_group_pruning_stats.md

Run Chunks Mean loss Perplexity Artifact
Native grouped 10% 16 2.482592 11.972254 artifacts/runs/native_group_loss_pilot16/group_pruned_10pct_loss.json
Native grouped 20% 16 2.527117 12.517363 artifacts/runs/native_group_loss_pilot16/group_pruned_20pct_loss.json
Native grouped 40% 16 2.765421 15.885724 artifacts/runs/native_group_loss_pilot16/group_pruned_40pct_loss.json

Downstream Evaluation

Run Scope Result Artifact
Full MMLU 0-shot baseline smoke 0.1% limit; 58 subject samples harness completed; accuracy 0.879310 artifacts/runs/downstream/mmlu_baseline_smoke.json
Full MMLU 0-shot baseline full 14,042 samples accuracy 0.733514; STEM slice 0.702188 artifacts/runs/downstream/mmlu_baseline_full.json
Full MMLU 0-shot atomic 10% full 14,042 samples accuracy 0.737715; delta +0.004202 artifacts/runs/downstream/mmlu_atomic_10pct_full.json
Full MMLU 0-shot atomic 20% full 14,042 samples accuracy 0.734511; delta +0.000997 artifacts/runs/downstream/mmlu_atomic_20pct_full.json
Full MMLU 0-shot atomic 40% full 14,042 samples accuracy 0.681028; delta -0.052485 artifacts/runs/downstream/mmlu_atomic_40pct_full.json
Full MMLU 0-shot native grouped 10% full 14,042 samples accuracy 0.735437; delta +0.001923 artifacts/runs/downstream/mmlu_native_group_10pct_full.json
Full MMLU 0-shot native grouped 20% full 14,042 samples accuracy 0.734012; delta +0.000499 artifacts/runs/downstream/mmlu_native_group_20pct_full.json
Full MMLU 0-shot native grouped 25% full 14,042 samples accuracy 0.725965; delta -0.007549; STEM slice 0.688233 artifacts/runs/downstream/mmlu_native_group_25pct_full.json
MMLU-STEM 5-shot baseline report-metric-aligned local lm-eval; 3,153 samples accuracy 0.690771 artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json
MMLU-STEM 5-shot native grouped 10% report-metric-aligned local lm-eval; 3,153 samples accuracy 0.693942; delta +0.003172 artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json
MMLU-STEM 5-shot native grouped 20% report-metric-aligned local lm-eval; 3,153 samples accuracy 0.679036; delta -0.011735 artifacts/runs/downstream/mmlu_stem_5shot_native_group_20pct_full.json
MMLU-STEM 5-shot native grouped 25% report-metric-aligned local lm-eval; 3,153 samples accuracy 0.666350; delta -0.024421 artifacts/runs/downstream/mmlu_stem_5shot_native_group_25pct_full.json
GSM8K-CoT 8-shot baseline first 10%; 132 paired samples strict EM 0.931818; flexible EM 0.886364 artifacts/runs/downstream/gsm8k_cot_baseline_10pct.json
GSM8K-CoT 8-shot native grouped 10% first 10%; 132 paired samples strict EM 0.909091; delta -0.022727; flexible EM 0.863636; delta -0.022727 artifacts/runs/downstream/gsm8k_cot_native_group_10pct_10pct.json
GSM8K-CoT 8-shot native grouped 20% first 10%; 132 paired samples strict EM 0.780303; delta -0.151515; flexible EM 0.742424; delta -0.143939 artifacts/runs/downstream/gsm8k_cot_native_group_20pct_10pct.json
GSM8K-CoT 8-shot native grouped 25% first 10%; 132 paired samples strict EM 0.681818; delta -0.250000; flexible EM 0.689394; delta -0.196970 artifacts/runs/downstream/gsm8k_cot_native_group_25pct_10pct.json
CRUXEval-O CoT baseline smoke one function; 10 sampled generations; executable pass@1 harness completed; pass@1 0.0 artifacts/runs/downstream/cruxeval_output_cot_baseline_smoke.json
SWE-bench Lite baseline smoke one oracle-prepared instance: astropy__astropy-12907 official grader completed; generated patch failed to apply artifacts/runs/downstream/swebench_baseline_smoke_predictions.jsonl

The SWE-bench smoke validates generation, prediction serialization, Docker availability, and official harness invocation. It is not a quality estimate: the single generated continuation produced a malformed patch and the grader recorded one patch-apply error. Further SWE-bench work is paused while the basic native-grouped benchmarks run.

Additional small benchmark selected from the Laguna technical-report table: gsm8k_cot. The installed lm-eval task is configured for the report-aligned 8-shot chain-of-thought exact-match setup. Run it for the baseline and native-grouped 25% variant after MMLU. Because a full run takes more than one hour per model, use the same first 10% subset for both variants. Runs explicitly cap generation at 256 new tokens because Laguna's shipped generation config otherwise requests up to 4096.

The report-metric-aligned local MMLU-STEM 5-shot baseline is 0.690771, below the Laguna technical report's 0.781. The local lm-eval comparison remains useful because the unpruned and pruned models run through the same harness, but it does not exactly reproduce the report's evaluation stack.

The GSM8K-CoT comparison is likewise a matched local-harness comparison rather than an exact reproduction of Poolside's internal framework. The local task uses eight fixed chain-of-thought demonstrations and greedy decoding (do_sample=False). An audit of the paired 10% run confirmed identical prompts and test examples. The 256-token cap did not bind: generated responses were short, and the grouped-model errors were predominantly incorrect arithmetic answers rather than truncations.

CRUXEval-O CoT executable grading was smoke-tested successfully. The paired coding subset was stopped after 84 / 400 baseline generations when the remaining session was time-boxed. The smoke validates the local generation and executable-grader path but is not a coding-quality estimate.

Grouped Compute Note

For grouped pruning, the simple deployment interpretation is the pruned group fraction. The prioritized 25% experiment removes exactly 19,968 of 79,872 64-wide groups. That is the meaningful reduction if a grouped MoE kernel only materializes retained child experts.

The primary native grouped evaluator preserves the original top-8 parent router and zeroes pruned groups in place. Its current kernel still computes those zeroed blocks, so observed runtime is a quality-measurement harness rather than a realized speedup. The separately reported repacked-child exploration uses fixed top-64 backfill and is not the primary result.

Method Notes

  • Atomic pruning was evaluated in memory by loading the BF16 model, zeroing pruned atom rows/columns in the packed expert tensors, and running the native Laguna forward on the full 4k cache.
  • Current fixed-shape loss runs pass an explicit Hugging Face StaticCache into each direct forward call with use_cache=True, avoiding Laguna's internal dynamic-cache construction path.
  • Atomic and diagnostic-group deltas above use the earlier BF16 baseline. Repacked-group deltas use the explicit-static-cache BF16 baseline, matching those runs' cache path.
  • The grouped 10% diagnostic is not the full grouped mini-expert routing experiment. It keeps the original parent-expert router/top-8 behavior and only zeroes 64-wide groups inside selected parent experts.
  • That original-router grouped path is now the primary experimental quality measurement: it directly applies 64-atom structural pruning while preserving the source model's routing semantics.
  • For grouped mini-experts as independently routable child experts, the evaluator needs to route over child groups. A plausible inherited-router test is: expand each parent expert score to its 8 child groups, mask pruned child groups, select top 64 child groups per token, and evaluate those group contributions. Under no pruning this collapses to the original top-8 parents; under group pruning it can select child groups from more than 8 parent experts.
  • The repacked grouped runtime now implements that inherited-router test by repacking each sparse layer from 256 parent experts x 512 atoms into 2048 child experts x 64 atoms, expanding router rows to child groups, selecting top 64 child groups, and assigning each selected group its parent expert's normalized router weight. The 0% pruned first-16 validation above checks that this preserves the unpruned model.

Current Status

  • Completed: BF16 full-cache baseline.
  • Completed: atomic 10%, 20%, and 40% full-cache loss.
  • Completed but diagnostic-only: constrained grouped 10% full-cache loss.
  • Stopped: constrained grouped 20%/40%, because it does not answer the independently-routable grouped mini-expert question.
  • Completed: repacked grouped 10% full-cache loss with explicit static cache.
  • Completed: repacked grouped 20% full-cache loss with explicit static cache.
  • Completed: repacked grouped 40% full-cache loss with explicit static cache.
  • Completed: unpruned full-cache loss with explicit static cache and directly comparable grouped deltas.
  • Completed: baseline MMLU and SWE-bench Lite smoke paths.
  • Completed: full baseline and atomic 10%/20%/40% MMLU comparison runs.
  • Stopped: full repacked-group MMLU. Repacked child routing over-selects groups for the primary experiment and remains exploratory only.
  • Completed: native original-router grouped 10%/20%/40% loss pilot on 16 chunks.
  • Completed: native original-router grouped 10%/20%/40% full-cache loss.
  • Priority changed by request: evaluate baseline versus native original-router grouped 25% only.
  • Completed: native original-router grouped 25% full-cache loss.
  • Completed: full-MMLU 0-shot native grouped 25%. This is useful auxiliary evidence but does not match the tech-report MMLU-STEM 5-shot row.
  • Completed: report-metric-aligned local MMLU-STEM 5-shot baseline and native grouped 25%.
  • Stopped: full GSM8K-CoT baseline after runtime projection exceeded one hour.
  • Completed: paired first-10% GSM8K-CoT baseline and native grouped 25%.
  • Completed: paired first-10% GSM8K-CoT native grouped 10% and 20% curve points.
  • Stopped by request: CRUXEval-O CoT paired subset after executable-grader smoke validation, due to the remaining time box.
  • Completed: random native grouped 25% full-cache loss control with seed 20260530. It is +0.094622 loss worse than importance-ranked 25% pruning.
  • Completed: random native grouped 10% full-cache loss control with seed 20260530. It is +0.116984 loss worse than importance-ranked 10% pruning.
  • Skipped from the grouped MMLU downstream matrix: native grouped 10%, 20%, and 40%.
  • Paused by request: further SWE-bench evaluation.
  • Validated: full test suite 28 passed; Python compilation and git diff --check passed.