ManniX-ITA

/

Qwen3.6-27B-A3B-Coder

like 6

Thanks — some context first on why the maps look the way they do.

I tried a lot of selection methods on Gemma-4 A4B, and plain activation-frequency targeting is what gave me the best results. The awkward part with both Gemma-4 A4B and Qwen3.6 A3B is that a lot of capability is spread thin across many routed experts rather than concentrated — multilingual and the agentic end-of-turn behaviour bit me hardest. Miss a few of those and the model starts looping or misbehaving even when the aggregate metrics still look fine. (Both models also have an always-on path alongside the routed mixture — a gated shared expert in Qwen3.6, a parallel dense MLP in Gemma-4 — but that isn't where the damage was.) Every time I mixed additional corpus channels into the selection signal, the intended targeting — coding — came out measurably weaker, so I kept it narrow.

I also tried folding the pruned experts back into the survivors and it was catastrophic; I'm not claiming that's a verdict on the idea, my implementation was probably too naive. Next I want to try a mixed recipe: activation targeting plus Samsung's REAM (https://github.com/SamsungSAILMontreal/ream), which is roughly the principled version of what I attempted by hand.

On your actual question — not an easy one. I had Claude run the analysis below; it looks right to me and the scripts are in the repo so you can re-run them yourself, but I haven't reviewed every cell by hand. Note it partly cuts against my own experience above: frequency is what won empirically for me, while the analysis says the sensitivity signal is the one that genuinely diverges. I read that as "worth testing", not "the recipe was wrong" — nobody has cut a model by rnorm and evaluated it.

It isn't documented anywhere because it had never actually been measured. Now it has.

First, the honest part: on the maps the released model was cut from, this can't be answered.

The three *_coder* maps (competence_qwen35b_coder.json, _coder_lcbmpe.json, _coder_lcbmpeife.json) were profiled with --tc-only. wnorm/rnorm/wsum are present in the schema but identically zero on every cell:

So the shipped cut was a pure routing-frequency decision — the sensitivity channel was plumbed but never populated on those runs. Answering your question properly on the released artifact needs a re-profile without --tc-only, which I haven't done yet.

What I could measure is competence_qwen35b.json, the balanced 8-bench baseline map, which does carry rnorm and does carry corpus_humaneval + corpus_mbpp as their own categories. That gives a code-vs-non-code frequency split on the same cells as rnorm. It's a proxy — it lacks the corpus_targeted_lcb/_mpe channels the shipped weighting actually keyed on — so treat everything below as directionally informative about the method, not as a measurement of the released cut.

Everything below is reproducible from the repo — the three scripts are in recipes/qwen3_6_35b_a3b_prune/, run them against the maps in results/:

• qwen_map_fields.py — the field-population check that produced the table above.
• qwen_rnorm_freq_corr.py — the correlation + bottom-K selection-overlap tables.
• qwen_wmax_addendum.py — the sum-vs-wmax aggregator comparison.

Each takes an optional map path as argv[1] and defaults to results/competence_qwen35b.json, so if you re-profile your own maps you can point them straight at those.

Method notes, since they change the numbers: rnorm is per (category, layer, expert) and is an RMS over routed tokens (sqrt(rnsq/tc)), not a sum, so I aggregate it across categories as a tc-weighted mean. Cells with tc == 0 are excluded — they're never routed, can have no rnorm, and including them inflates every coefficient. 10,199 / 10,240 (layer, expert) pairs survive. Correlations are Spearman first, because the drop map is a per-layer bottom-K rank selection — rank agreement is the quantity that decides which experts die. And I report per-layer, not just pooled: pooling 40 layers hides real per-layer variation, and the selection is per-layer.

rnorm vs the frequency signals

So: they track, but not tightly — per-layer ρ ≈ 0.82 against unweighted frequency. (The wnorm row is worth noting on its own: ρ = 0.67 with r = 0.21 means the relationship is monotone but strongly non-linear — heavy-tailed, as you'd expect from an output-norm.)

The thing I didn't expect

The premise of your question is that "code-weighted" and "unweighted" are two meaningfully different signals to correlate against. Under sum aggregation, they're almost the same vector:

Up-weighting 2 of 8 categories by 1.5–2× inside a sum is arithmetically close to a no-op — the other six categories carry enough mass to hold the ranking in place. That's not a defect of the recipe, it's why the shipped cut used --agg wmax --cat-weight corpus_targeted_lcb=2.0 and not --agg sum. Under wmax (max over w_c · tc_c) a weighted category can own an expert's score, and the divergence appears:

And rnorm's agreement falls as you move toward the shipped configuration: +0.823 (sum unweighted) → +0.810 (sum 1.5×) → +0.809 (wmax unweighted) → +0.711 (wmax 2.0×). The sensitivity signal diverges most from exactly the aggregation the release used.

Where it actually matters — the tail

A ρ of 0.82 still leaves a materially different cut, because only the bottom-K matters. Ranking by each signal and taking the bottom-72 per layer (the shipped 256 → 184 cut):

So a sensitivity-ranked cut would drop ~21 of 72 experts per layer that a frequency-ranked cut keeps, and vice versa. The frequency-weighting knob moves ~3–4 experts per layer; rnorm moves ~21. Which is to say: your instinct is right that these are separable signals, and the separation is bigger than the one the recipe currently exposes as a tuning knob.

One more datapoint on why: code-only vs non-code frequency correlate at only ρ = +0.431 pooled / +0.413 per-layer median. The code channel genuinely does carry distinct routing information — it just gets diluted to near-nothing once it's one of eight summands.

Caveats, restated because they matter: this is the balanced map, not the shipped one; humaneval+mbpp is a stand-in for the targeted LCB/MPE corpora; and none of it is an ablation — I have not cut a model by rnorm and evaluated it, so "different cut" here means different, not better. A proper answer wants a non---tc-only re-profile on the coder corpus plus at least one paired eval. Happy to run that if there's interest; it's a few GPU-hours, not a research program.

Thanks for the prod — this was a real gap.

That's a good insight. Combining frequency and sensitivity as metrics for selection might make it more complex but most likely gives a more rounded view of the trade-offs. It's a positive thing to have refined the work based on frequency before adding this complexity

P.s. Impressive pace that you're working at there!

Working hard, only limited by the inference I have access to 😂

Did a check on the old tasks with Claude on my ML session (+2GB transcript, it's a monster) and indeed we tried it but it was a code based on the paper:

● On the REAM question — I checked the code rather than trusting the task title, and your recollection is the accurate one. What T193 ran was our own code, not Samsung's, and quite possibly not their method either.

omnimergekit/scripts/redist.py:447 is the whole of it:

  class REAM(HCSMoE):
      name = "ream"
      blurb = "router-weighted expert activation merging (arXiv:2604.04356, REAP successor)"

      def _saliency(self, data, li, freq, mean_out, E):
          gate_mass = torch.zeros(E).scatter_add_(0, ti, tw)
          return gate_mass * mean_out.float().norm(dim=-1)

24 lines subclassing our own HC-SMoE fold, overriding only the saliency term (router gate-mass × mean SwiGLU output norm). It cites the arXiv number in a docstring blurb; no Samsung source is vendored anywhere in the
repo, no upstream dependency, no third-party licence file. So "REAM" in T193 means "HC-SMoE with a saliency formula borrowed from the REAM paper" — it sat in the registry next to hcsmoe and mergemoe as a third
closed-form fold variant.

Two things make it likely it wasn't their algorithm at all:

• We used it as a post-prune fold — dropped experts folded into a fixed survivor topology, router untouched. Samsung's entrypoint is a standalone merge.py --model --merge_size --save_path with no prune arguments, and
  the paper is titled "Merging Improves Pruning of Experts" — merging as a preprocessing step whose output is then pruned. Different pipelines. Their README doesn't state the algorithm, so I've written this into task 762 as
  something to confirm, not as established.
• We already caught our own version's premise being wrong: experiments/gemma4_62e_redist.md:187 records that the redist.py REAM docstring's claim to "sidestep REAP Theorem 1" is conceptually incorrect — changing the
  merge weighting doesn't help when a fixed router still can't gate the merged function.

So T193's negative result (HE+164 −1.2pp, MPE-100 −3.3pp, bf16 loop_screen 19.0% vs A2's 15.5%, unconfounded) falsifies our saliency-weighted fold. It does not falsify the paper. I've rewritten task 762 to say exactly
that, with step 1 now being "read merge.py, diff it against redist.py's REAM, and close this task only if they turn out to be the same thing." I also added a note that the competence map has to be re-profiled on the
merged model if REAM goes before the map — reusing a pre-merge map would be profiling a model that no longer exists.

Both things I owed you are done, plus the REAM comparison. Same basis throughout: same base, same 184/256 budget, same calibration data, Q6_K + imatrix, greedy, one host, one binary. Jitter band ±1.0 pp on MPE, 0.0 pp on LCB (measured from two bit-identical builds, not assumed).

1. rnorm vs the code-weighted signal — answered as a build, not a correlation

Rather than report a correlation coefficient, I re-profiled and built the arm: same recipe, but the map scored by plain row-norm instead of the task-attributed competence score.

The two signals mostly agree — row-norm moves only 16.7 experts per layer off the shipped map, ~9% of the keeps. But that 9% is not free:

Identical pass rate on hard LCB, 5 pp of MPE gone, and nearly triple the degenerate-loop rate. So they track closely as rankings and diverge where it counts — and the divergence shows up on the generation-behaviour axis far more than on the score axis. That's the part I would have missed if I'd only looked at pass rates, which is I think the useful half of your question.

For scale: the REAP-saliency cut disagrees with ours on 38.5 experts/layer — more than twice the row-norm disagreement — and scores the same as ours everywhere. Direction matters; distance doesn't.

2. Mixing activation-targeting with REAM

Short version: on this model REAM's merge helps only when it is nearly doing nothing, and injecting our saliency into it is what makes it start doing something — badly.

Stock REAM (REAP saliency, group_size=16) reaches MPE 0.720 / HE+ 0.957, within a band of our cut. But its grouping puts ~64% of each merged group's weight on the centroid, so it is close to plain selection. Substituting our competence saliency into the same machinery drops the centroid share to ~42% — a real average — and:

Monotonic in the dilution dose. The two low arms are also the only ones that truncate: at group_size=16 the median HumanEval+ completion is 16,382 tokens against a 16,384 ceiling (56% of items capped), against 0/164 for every selection-only arm.

So the naive mix isn't a hybrid, it's a dilution. I don't think that's a bug in REAM — it's that its grouping is tuned to an ordering whose top is dominant, and ours isn't shaped that way. Worth noting this is now the second time folding pruned experts back into survivors has failed here, with a real implementation rather than my earlier naive one.

3. What I'm trying instead

Rather than merging, use REAP as a constraint on our ranking: keep our selection, but forbid dropping the experts REAP ranks highest, evicting our own lowest-ranked keeps to pay for it. Two doses (12 and 24 experts/layer) are built and running.

First cell is in and it's not encouraging — MPE 0.680 at the low dose, below both parents (ours 0.730, REAP-select 0.717) and equal to the row-norm control. My working hypothesis is that the eviction leg does the damage: the experts our map ranks lowest are still load-bearing, and the REAP-rescued ones don't replace them. The higher dose should be worse if that's right. I'm finishing both regardless — MPE was never the axis these arms were built for; the claim is about the loop rate.

4. One caveat on the numbers above

Stock REAM's hard-LCB score (0.468) is truncation-taxed — 55 of 77 completions hit the length ceiling, median 32,166 tokens against 32,768. A ceiling that binds 9× harder on one arm than another is part of what's being measured, so I'm re-running all arms on the same 77 problems at 24k thinking / 48k total. That's in flight now.

Full assessment — including the loop/rumination breakdown, not just pass rates — once it lands.

Follow-up as promised. The 24k/48k cap-validation pass finished — 9 arms, one basis, greedy. Four things I said in my last reply need correcting. Two favour your suggestion, one was a measurement error of mine that made your suggestion look worse than it is, and the last one takes back a number I was about to brag about.

Hard LiveCodeBench v6, 77 hard problems, 24k thinking / 48k total

Control for the basis change: every arm that wasn't truncating at 32k reproduced its 32k solve set exactly, and two cells with bit-identical weights agreed on all 77 problems. Only genuinely clipped arms moved.

Correction 1 — the row-norm control is not behind

I reported rnorm as the weakest signal. On hard LCB it is 48/77 vs our 47 — ahead of our activation-targeted cut and level with the unpruned base. So the case for activation targeting cannot be made on hard-LCB score. It does hold on MultiPL-E, where rnorm is the weakest arm in the program at 0.680 against our 0.740 (see Correction 3 for that table and why I trust it now). But "rnorm is dominated" was too strong, and on the benchmark this thread is about, it isn't.

Correction 2 — I called the mixed-saliency hybrid "not encouraging". It is the best thing in the program.

Recipe, concretely: keep our competence-map ranking, then protect the p experts REAP ranks highest among our drop set, evicting the p lowest-ranked of our keeps. Two doses:

• p=12 → 57/77 (+10 over our cut)
• p=24 → 56/77 (+9)

The reason I now believe this rather than treating it as a lucky draw: the gain set is shared. Against our cut, p=12 gains 18 problems and p=24 gains 21 — and all 18 of p=12's gains are a subset of p=24's. Jaccard between the two gain sets is 0.86, and the two hybrids are each other's nearest neighbour in the full pairwise matrix (0.74, against 0.45–0.61 for every other pair). Doubling the dose does not find a different set of problems; it finds the same ones plus three. That is the signature of a systematic property of the recipe, not two coincidences landing on similar counts.

So your intuition — that our saliency and REAP's saliency are measuring different things and the intersection is where the useful experts are — is the strongest result here.

Correction 3 — I reported a MultiPL-E cost. There is no cost. The cost was my measurement.

In my last reply I said the hybrid pays on MultiPL-E, and I spent a day explaining why. The explanation was wrong, and then the number was wrong too. Both are worth writing up, because the second one is a reproducibility trap anyone benchmarking this family can fall into.

The mechanism I proposed does not exist. p=12 was returning a bare newline on 18/300 MPE problems, and I called it premature end-of-generation. To test that I built a terminator emit-map — hook the router, accumulate routing mass at positions that predict <|im_end|>, normalise against all other positions so a globally-hot expert can't top the list by being hot everywhere. The 12 experts per layer that p=24 rescues and p=12 leaves dropped came back less terminator-preferential than the pool they're drawn from (rank 0.588 vs 0.637; permutation p = 1.00 against enrichment). So I stopped proxying and measured P(EOG) directly at the first generated position on all 300 prompts: no arm separates (all 3–13 × 10⁻⁵), and on p=12 the empty problems carried lower EOG probability than the ones it answered. A greedy rollout on each arm's own empty problems kept writing code, 18/18 and 9/9 and 4/4. The weights were telling me there was no defect. I assumed my probe was broken.

It wasn't. The serving geometry was unmatched. Our chain script pinned the parallelism but not the KV budget, so llama.cpp's context was auto-planned from free VRAM and silently drifted between chains: the two hybrid arms were served 4096 tokens per slot while all eleven arms they're compared against ran at 12288. Re-running both hybrids at the comparators' geometry, changing nothing else:

p=12 goes 0.680 → 0.790 and 18 empties → 0. p=24 goes 0.710 → 0.823, also to zero. For scale: our published cut and a bit-identical rebuild of it score 0.740 and 0.730 with 1 and 4 empties, so the run-to-run band on identical weights is about ±0.01 — the hybrid is eight times that above the best comparator, on both doses.

So the hybrid does not trade LCB for MultiPL-E. It wins both, and the higher dose wins by more on both.

The trap is worth spelling out, because "it fits in the context" is not the safety condition you'd assume. MPE requests only 1024 generation tokens and the longest prompt here is 778, so 4096 per slot is ample and the harness correctly reported zero truncations. Nothing was cut off. What actually happened, comparing the two runs completion-by-completion: 279 of the 300 completions changed, only 21 are byte-identical, and the output is deranged in both directions at the smaller context (mean 501 chars and a 3612-char max at 4096, versus 311 and 1771 at 12288). The 18 "empties" are strings like '\n' and '\n \n' on problems where the same model at 12288 writes the correct one-liner. They're spread evenly across the task order, so it isn't slot exhaustion building up over the run.

So this is the familiar reason greedy decoding isn't bit-reproducible under batching: change the KV cache geometry and you change floating-point reduction order, and greedy turns one flipped logit into a different program. Usually that's a ±0.01 nuisance — here it happened to be worth 11 points. Concretely: pin the context explicitly and read the per-slot value back out of the server log, and note that llama.cpp divides -c by --parallel, so the number you pin is not the number each request gets. Zero truncations is not evidence that the geometry was adequate.

The LCB table above is unaffected — I checked all nine of those cells and every one ran at 65536 tokens per slot across 8 slots, comfortably above the 49152 generation cap, because that template pins its context explicitly instead of letting it be planned. That's why I'm happy to put the headline in front of you.

Correction 4 — and the real cost, now that I can see it

HumanEval+ had the same drift, in the more dangerous direction: the hybrids were served 16384 per slot against a template asking for 16384 generation tokens, so prompt plus generation could not fit. Re-run at the comparators' 24576:

This bench is near-saturated and very tight — a straight repeat of our cut reproduced 0.9695 exactly, as did two other arms — so a 3–4 problem gap is outside the noise even though it looks small. Note this also retires the "p=24 scores 0.9817, best in the program" claim I was about to make: that number came from the broken geometry and it is gone.

So the honest summary is a trade, not a free win. Against our published cut, the hybrid gains ~10 problems on hard LCB and 0.05–0.08 on MultiPL-E, and gives back 3–4 problems on HumanEval+ — landing there at about the level of the unpruned base. Given p=24 wins MultiPL-E (0.823 vs 0.790), loses only one LCB problem to p=12 (56 vs 57), and gives back one fewer on HE+, p=24 is the arm I'd recommend.

What the emit-map did find is about the published cut, not the hybrid. Our shipped 184e drop set is significantly enriched in terminator-carrying experts: mean rank 0.625 against a random-72 null of 0.500 ± 0.005 (p < 0.0001), and 35% of each layer's top-8 EOG experts are dropped versus 28% by chance. The sign is the interesting part — removing terminator machinery should suppress stopping, and that is what we see: the pruned cut returns fewer premature empties than the unpruned base (1–4 vs 9 on the table above) and is biased toward running long, consistent with the length behaviour reported on this family. That's a concrete --force-keep target for the next cut. It's a separate finding from the hybrid question and it does not explain p=12.

Net: the hybrid's gain is real on hard LCB and on MultiPL-E, it costs a few problems on the near-saturated HumanEval+, and the specific weakness I reported last time was mine rather than the model's.

One caveat on my own numbers

Our 184e cut and the unpruned 256e both solve 47 — but only 33 in common, 14 each-only. "The prune is free on hard LCB" holds only in aggregate; the pruned model is a different solver that breaks even. Worth knowing before reading any of the deltas above too literally.

Thanks for the push toward mixed targeting — it's the only thing that has moved this benchmark by double digits.

This is an promising set of findings. The gains on hard LCB are substantial and the reduction in HumanEval+ back to unpruned levels is an acceptable trade-off.

If I interpret it correctly then the median token comparison for hard LBC (first table) is another significant trade-off – a 6-7x jump. Maybe that jump is inflated because of expert uncertainty and persistence on the lost problem sets. Regardless of the source of the increase, it doesn't look so appealing on an 'accuracy per token' basis.

Others have worked on fine-tunes for token length but those efforts are very limited. For example kai-os's Grug-35B-A3B concise reasoning finetune states a limitation of over-compression on reasoning intensive tasks. So that doesn't seem to be a viable approach to compensate. Any fine-tuning budget would be better spent elsewhere.

Thank you for the thorough investigation and detailed summary.