--- tags: - mixture-of-experts - expert-pruning --- # APEX-MoE calibration priors Calibration files for APEX-MoE, prompt-based expert pruning for mixture-of-experts language models: at the end of prefill, each request keeps a fixed share of the routed experts in every MoE layer, and decode routes among those only. These files hold the per-expert statistics the selection score needs, fitted once per model on a calibration set. | folder | model | MoE layers x experts | |---|---|---| | `qwen3.6-35b-a3b` | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 40 x 256 | | `qwen3.8-flash-next-fp8` | [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) | 48 x 512 | | `deepseek-v4.1-flash` | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) | 40 x 384 | Each folder holds, for MoE layer `l` and routed expert `e`: - `gain.npy` `[L, E]`: the expert's gain, its router-weighted mean output norm on the calibration set, `sum g * ||f(x)|| / sum g`; - `prior.npy` `[L, E]`: the expert's share of the summed REAP saliency `g * ||f(x)||` on the calibration set (a static prior); - `transfer.npz`: the transfer prior, a per-layer ridge regression of a request's decode profile on its prompt's routing: `matrix` `[L, E, E]`, `offset` `[L, E]`, `lam` (ridge strength), `power` (2: fitted on the square root of the decode share); - `meta.json`: model revision, calibration runs, fit settings. ## Selection score Per request and layer, with `m` the prompt's summed router weight per expert, `c` its route count per expert, `n` its token count and `share(v) = v / sum(v)`: ``` prior = share(max(0, share(c) @ matrix[l] + offset[l]) ** power) score = n * share(gain[l] ** 2 * m) + 256 * prior ``` The `round(keep * E)` highest-scoring experts are kept (ties: lowest expert id); decode masks the other router logits before the top-k. Settings used with these files: gain power 2, prior strength 256, keep 25% and 50%. ## Calibration 1024 prompts (`calmix1024`, seed 0): 256 each from Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1 and FineWeb-Edu passages, decontaminated against the evaluated benchmarks. Each model answered every prompt itself with thinking on; a teacher-forced replay of prompt and answer recorded the routed experts' router weights and output norms. The Qwen3.6 files were fitted on the BF16 checkpoint. ## v3 priors (`v3/`) Refit priors for Qwen3.6-35B-A3B (`v3/qwen3.6-35b-a3b`) and Qwen3.8-Flash-Next-FP8 (`v3/qwen3.8-flash-next-fp8`) on a larger calibration set, used **ridge only**: the transfer prediction alone ranks the experts (prior strength 1e9; the prompt term only breaks ties). Besides `gain.npy`, `prior.npy` and `transfer.npz` (as above) each folder holds: - `transfer_window.npz`: the window ridge for dynamic re-selection during decode, a per-layer ridge regression of the square root of the next 256 decode tokens' saliency share on the last window's unmasked route-count share and the prompt's: `matrix` `[L, 2E, E]`, `offset` `[L, E]`, `window` (256), `lam`, `power` (2), `pairs` (calibration window pairs); - `meta.json`: model revision, calibration runs and excluded domains, fit settings (including `ridge_intercept`: Qwen3.6 without, `offset` 0; Qwen3.8 with). Static selection, per request and layer (`c`: the prompt's route counts per expert): ``` keep top round(keep * E) of share(c) @ matrix[l] + offset[l] ``` Dynamic selection: every 256 decode tokens the set is chosen again as the top of `[share(w) | share(c)] @ window.matrix[l] + window.offset[l]`, `w` the unmasked top-k route counts of the last 256 decode tokens, plus 0.5 standard deviations of that prediction for the experts kept now. Calibration (2,304 requests, each answered by the model itself): `calmix1024` as above; `calmcq512`, 256 MMLU dev and 256 ARC-Challenge questions, answered once with thinking off (16k) and once with thinking on; `calv3-1024` without its Tulu-3 rows, 256 each from the MATH train split, the MBPP train split and GPQA-style questions, three shards with thinking on and one off. The Qwen3.6 files were fitted on the BF16 checkpoint. ## v4 priors (`v4/`) The final calibration mix, fitted for Qwen3.8-Flash-Next-FP8 (`v4/qwen3.8-flash-next-fp8`) and Qwen3.6-35B-A3B (`v4/qwen3.6-35b-a3b`, fitted on the BF16 checkpoint). Ridge only: each folder holds `transfer.npz` (`matrix` `[L, E, E]`, `offset` `[L, E]` all zero, `lam`, `power` 2) and `meta.json` (model revision, fit settings, calibration sets and runs). There is no `gain.npy` or `prior.npy`: the selection uses T alone. Fit, per MoE layer `l`: `sqrt(s) ~ r @ matrix[l]` by ridge regression without intercept over the calibration requests (`lambda = 1`, scaled by `tr(R R^T) / requests`), with `r` the prompt's route-count share per expert and `s` the response's REAP saliency share (router weight times expert output norm), measured by teacher-forcing the model's own response. Selection, per request and layer (`c`: the prompt's route counts per expert): ``` keep top round(keep * E) of share(c) @ matrix[l] ``` Calibration (3,584 requests, each answered by the model itself at temperature 1.0): | set | requests | data | |---|---|---| | calmix1024 | 1,024 | Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1, FineWeb-Edu | | callong-ot384 | 384 | OpenThoughts-114k problems with the longest reference answers | | callong-otcode384 | 384 | OpenThoughts-114k code problems with the longest reference answers | | calagent-swe256 | 256 | SWE-smith trajectories cut before an agent turn, SWE-agent's tools | | calagent-xlam256 | 256 | xLAM function-calling queries with their tools | | calown-wp256 | 256 | WritingPrompts curated | | calown-personas256 | 256 | tulu-3-sft-personas-math | | calown-cfcots384 | 384 | codeforces-cots (Python) problems with the longest R1 traces | | calotcode-nothink384 | 384 | further OpenThoughts-114k code problems, answered with thinking off | All sets are answered with thinking on except `calotcode-nothink384`. No prompt contains a question of MATH-500, GPQA-Diamond, MMLU-Pro, MBPP+ or HumanEval+ (substring or shared word 13-gram); the thinking-off code set is also checked against LiveCodeBench v6 (2025-02 to 2025-05). No data is drawn from LiveCodeBench, LeetCode or AtCoder. ## Use With [apex-engine](https://github.com/IST-DASLab/apex-engine) (FreeToken with APEX): ```bash ft serve --model Qwen/Qwen3.6-35B-A3B --moe-strategy offload \ --apex-dir /qwen3.6-35b-a3b --apex-keep 0.25 ```