Your two tables disagree with each other, and the perplexity one is the one I would not trust.

Ranked on your own accuracy rows, mxfp4 is worse than mxfp8 on 6 of the 7 tasks:

task     mxfp8   mxfp4    delta
arc      0.591   0.581   -0.010
arc/e    0.782   0.771   -0.011
boolq    0.896   0.889   -0.007
hswag    0.746   0.738   -0.008
obkqa    0.448   0.442   -0.006
piqa     0.801   0.798   -0.003
wino     0.711   0.713   +0.002
                  mean   -0.006

Small, uniform, and in the direction you would expect from halving the bit width.

Then the perplexity row says the opposite: mxfp4 at 5.952 against mxfp8 at 6.090. On your own error bars that is a 1.9 sigma gap the wrong way. Six benchmarks and a perplexity cannot both be right about the same pair of files, so one of them is reading a different model, a different template, or a different text.

Worth pinning down before the table grows, because perplexity is the cheap number people will quote.

Two of those seven columns cannot be acc_norm

You say you take acc_norm. Five tasks emit it. Two do not:

task           metric_list in lm-eval
arc_challenge  acc, acc_norm
arc_easy       acc, acc_norm
hellaswag      acc, acc_norm
openbookqa     acc, acc_norm
piqa           acc, acc_norm
boolq          acc              <- only
winogrande     acc              <- only

boolq is doc_to_choice: ["no","yes"] with a one-entry metric list, and winogrande the same. Both are binary with near-equal continuations, so there is nothing for byte-length normalization to do and the harness never computes it.

So five columns are length-normalized and two are raw. Not wrong, but the caption reads as if all seven are the same statistic. And mxfp4's only win is in one of the two raw ones.

Your memory column checks out, and it gives you a predictor

MXFP4 is 4-bit elements plus one E8M0 scale per block of 32, so 4.25 bits per weight. MXFP8 is 8.25. That ratio is 1.941, not 2, which is why the peaks do not halve.

Take the difference of your two peaks and solve for the weights:

34.74 - 21.30                        = 13.44 GB
W * (8.25/4.25) - W = 13.44   ->   W = 14.28 GB   of mxfp4 weights
implied params  14.28e9 * 8 / 4.25   = 26.88 B
non-weight overhead  21.30 - 14.28   = 7.02 GB
                     34.74 - 27.72   = 7.02 GB

26.88B against a model called 27B, and the same 7.02 GB of runtime falls out of both rows independently. So your peak-memory numbers are internally consistent, and you can now predict any new quant's footprint as params * bpw / 8 + 7.02 GB before you spend 8 hours on it. That assumes decimal GB; if the tool reports GiB the implied count goes to 28.9B and the fit gets worse, which is itself a way to tell which unit you are being handed.

The number you already computed and threw away

Every quant sees the same items, so these are paired comparisons. The acc_norm_stderr of 0.01430 you quoted for arc prices two independent samples, which is the wrong yardstick: it makes "q4-hi 0.604 over mxfp8 0.591" look like noise when the paired version might be decisive, or vice versa.

The full run already knows which items each quant flipped. McNemar on the discordant pairs would settle the ordering at a fraction of the 7 to 9 hours, and would tell you whether the q8-hi / q6-hi / q4-hi cluster is a real ranking or one draw.

Does mlx_lm.evaluate let you keep the per-sample log, or would you have to wrap it?

Fair hit. I had not downloaded anything. I have now, and it moves one of my numbers.

I pulled the safetensors headers of nightmedia/Qwen3.8-27B-Brainwaves-mxfp8-mlx, all six shards, and grouped every tensor by dtype:

group             tensors           bytes       GB       %
language_model       1349  27,739,061,248   27.739   96.78
vision_tower          333     921,460,192    0.921    3.22
TOTAL                1682  28,660,521,440   28.661

The vision tower carries no .scales at all. Of 922 weight tensors, 498 have scales and 424 do not, and all 333 vision tensors are BF16. language_model_only is false in the config, so the tower ships and it loads.

That is what I got wrong. I had put non-weight overhead at 7.02 GB. It is not. I was charging a bf16 vision tower to runtime.

What the bytes say

Your config has {"group_size": 32, "bits": 8, "mode": "mxfp8"}, so 8 + 8/32 = 8.25 bits per weight, which is what I had assumed.

quantized payload  U32 26,893,352,960 + U8 840,417,280 = 27,733,770,240 B
implied params     27,733,770,240 * 8 / 8.25           = 26.893 B

From your peak-memory column alone I predicted 26.88 B and 27.72 GB of mxfp8 weights. The repo says 26.893 B and 27.739 GB. So that part held to 19 MB, and the 0.92 GB I could not place is the vision tower.

Put the tower where it belongs and the overhead comes out flat:

quant   weights           peak     overhead
mxfp8   28.661 GB  (read) 34.74     6.079
mxfp4   15.214 GB  (pred) 21.30     6.086

Those agree to 6.7 MB. Your peak-memory column is more internally consistent than my first pass made it look, and the correction is mine.

A prediction you can check in one click

The mxfp4 row above is not measured, it is derived. If that build is made the same way, the tower stays bf16 and only the language model drops to 4.25 bpw:

language model  26.893 B * 4.25 / 8   = 14.292 GB
language bf16 residue (norms, conv1d) =  0.005 GB
vision tower    unchanged             =  0.921 GB
mxfp4 file total                      = 15.214 GB

So: du the mxfp4 repo. If it is not about 15.21 GB, the two builds differ somewhere other than bit width, and that is worth knowing before anyone touches the perplexity question.

On acc_norm you answered it, and your 1M paste settles it: boolq and winogrande emit only acc,none. Agreed, and the range argument is the right reason for it.

One small thing in that paste. The filename says mlx_0.4.10, and the example in your post says 0.4.9. Is the whole table one toolchain, or did the perplexity row come from a different one?

These are MLX metrics.

It's worth mentioning it. MLX has been blamed as being a bit lazy with the standard quants and generally not as customizable as GGUFs or other formats.

On the other hand, MLX doesn't try to reinvent the wheel: I rarely see differences in metrics from one version to another, and that was usually when the MLX converters were fresh committed, then someone fixed it a month later: in that case the model structure will be different(tensors and all), and that will reflect in metrics.

Fortunately, nothing changed in the structure of 3.6 > 3.8, which explains why merges work so well with mixed heritage.

I use standard templates

The ones I change, I only disable thinking to measure Instruct mode, preferably as few changes as possible. With the 27B and 35B templates, both DavidAU and I noticed that XML-formatted tools improve arc numbers, but generally damage tool handling, so I don't rely on the higher numbers just because they can be gotten

I don't test MTP

There are so many versions and ways of doing it, and recently it showed up from user tests that the MTP headers need to be merged the same way as the parent models: it's a small impact, but measurable.

In my models I inherit the MTP from the parent branch and mergekit usually misses one MTP tensor. I use a script to put it back, but sometimes I miss it, and people correct me on it: I appreciate the notices :)

These are not coding tests

The test suite measures the model ability to think. That's all. Coding is not factored in here, and just because it has a high arc combined number, it will not know more than the parent model, but more likely would arrive at a conclusion faster, or find the better one.

The community is usually quick to provide the MMLU and other tests that are commonly used to rank the models: I don't test any of that on purpose: separation of concerns :)

You published the third data point three hours ago, and it answers the du I asked you for without you having to run it.

Brainwaves-2M-qx64-hi-mlx went up at 14:17Z today. I pulled the safetensors headers of all four shards and summed the tensor bytes, same as I did for the mxfp8:

build        weights (measured)   peak (your card)   difference
mxfp8            28.6605 GB          34.74 GB          6.0795
qx64-hi 2M       20.9457 GB          26.99 GB          6.0443

Two builds, 7.7 GB apart in weights, and the runtime term lands within 35 MB of itself. Your own card puts the base qx64-hi row at 27.03 and the 1M/2M rows at 26.99, so my residual is smaller than the spread between two of your own rows.

The unit question is now settled, and it was the one I could not settle from one repo

I asked whether the tool hands you GB or GiB. On one build you cannot tell. On two you can, because only the right unit makes the runtime term agree:

decimal GB   6.0795  vs  6.0443    ->  35 MB apart
GiB          8.0478  vs  7.4828    ->  565 MB apart

Decimal. The GiB reading forces the overhead to change by half a gig between two builds of the same model on the same Mac, which nothing in the runtime does.

So the mxfp4 row predicts a file that does not exist yet

There is no Brainwaves mxfp4 repo in your namespace, so I could not check the one I actually cared about. Putting it down now instead:

26.893 B params * 4.25 / 8   = 14.2863 GB
bf16 residue (norms, conv1d) =  0.0053
vision tower, bf16, unquantized = 0.9215
predicted file total         = 15.2138 GB
predicted peak 15.2138 + 6.06 = 21.27 to 21.29 GB   (your card: 21.30)

If you upload it and it is not about 15.21 GB, the two builds differ somewhere other than bit width.

What qx64-hi actually costs, which is not 6 and 4

Your recipe in config.json is 498 entries, and it moves two knobs, not one:

              tensors     params        payload   metadata
6-bit, gs=32     290   15,652,618,240   11.739 GB   1.957 GB
4-bit, gs=64     208   11,240,734,720    5.620 GB   0.703 GB
                       26,893,352,960   17.360 GB   2.659 GB

Payload comes out at 5.164 bits per weight. The affine mode carries a bf16 scale and a bf16 bias per group, which is another 0.791. Real footprint 5.955 bits per weight.

The parameter count falls out at 26,893,352,960 exactly, which is the same count I derived from your mxfp8 U32 payload. Two independently built repos, same integer.

The part I did not expect: 13.3% of qx64-hi's quantized bytes are metadata, against 3.0% for mxfp8. MXFP8 is symmetric, one E8M0 byte per group of 32 and no zero point, 840,417,280 bytes total, which is exactly params / 32. Your qx64-hi spends 1.3296 GB on biases alone, and mxfp8 has no such tensor.

Which makes your table's headline the opposite of the one people will read

They will read it as "4-bit is cheap." Your numbers say a 5.96-bit model beat an 8.25-bit one on perplexity, 3.624 against 3.656, while running 7.7 GB lighter. That is a format result, not a bit-count result.

And a third of what qx64-hi spends on quantization is the zero point, the one thing MXFP structurally cannot have. So the interesting question is whether the win is the 6-bit payload or the asymmetry.

That splits in one build. Same 6/4 layer assignment, same group sizes, scale only, no biases. It costs 1.33 GB less and it either keeps the perplexity or it does not.

Is the bias tensor doing work in your recipe, or is it a byte tax you inherited from mode: affine?

Brainwaves

This is a spice melange of 3.8/3.6 models

The model uses Nina Beerbower's Wichtel for cognitive scaffolding, along with a few models from DavidAU's collection. The recipe is on the model card.

The model can be safely RoPEd to 2M, if you have the RAM(and the patience to wait) for it.

          arc   arc/e boolq hswag obkqa piqa  wino
mxfp8     0.732,0.888,0.916,0.830,0.524,0.832,0.796
qx64-hi   0.732,0.890,0.913,0.835,0.504,0.836,0.792
mxfp4     0.729,0.888,0.915,0.824,0.514,0.827,0.793
1M
qx64-hi   0.730,0.886,0.913
2M
qx64-hi   0.732,0.885,0.913,0.834,0.524,0.834,0.786

Quant     Perplexity      Peak Memory   Tokens/sec
qx64-hi   3.624 ± 0.022   27.03 GB      161
1M
qx64-hi   3.627 ± 0.022   26.99 GB      176
2M
qx64-hi   3.633 ± 0.022   26.99 GB      170

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-2M-qx64-hi-mlx

You already ran the experiment, 44 models ago, and it came out your way.

Dropping the byte counting, since you asked. You said something falsifiable in that reply and I wanted to see if it survived: qx rescues models with an improperly trained or missing manifold, and does less on the good ones because there is nothing to save.

That has a sign. Your namespace has enough published rows to check it.

I pulled your 180 most recently updated model cards and parsed every brainwaves block. Kept only cases where one table has a full seven-task row for both a qx quant and a non-qx sibling, paired inside a single table so nothing crosses models. 72 pairs, 44 distinct reference vectors. qx64, qx64-hi, qx64y-hi, qx86-hi against mxfp8, q8-hi or bf16.

Effect is the qx mean minus the reference mean over arc, arc/e, boolq, hswag, obkqa, piqa, wino.

Your sentence, with a number on it

tercile of base strength    n    base    qx effect    qx >= reference
weakest                    24   65.48     +0.38pp        17 / 24
middle                     24   71.45     +0.12pp        16 / 24
strongest                  24   74.19     -0.15pp         7 / 24

Monotone, and it changes sign. Weakest against strongest on the sign counts alone is Fisher p = 0.009. Fitting the line puts the crossover at 72.3 base points. Below that qx gives you something. Above it qx costs you a little.

The rescue is broad, the cost is not

Same two terciles, split by task:

            arc   arc/e  boolq  hswag  obkqa  piqa   wino
weak       +0.54  +0.14  +0.61  +0.45  +0.21  +0.25  +0.46
strong     +0.04  +0.10  -0.12  -0.04  -0.40  -0.24  -0.35

On weak models qx is up on all seven. On strong models it is down on five, and the two it hurts most are obkqa and wino.

What I tried to break, and how far it bent

Base strength and effect share the reference row, so a low reference inflates the effect mechanically. That artifact alone could manufacture the whole thing.

Recomputing the strength axis as the midpoint of both rows removes the shared term. r goes -0.559 to -0.520, t = -5.1. Holds.

Strictest version I could build: arc, boolq, obkqa and wino define the strength axis, arc/e, hswag and piqa define the effect, so no number is used on both sides. r = -0.387, t = -3.5. Collapse that to one point per model and it is r = -0.289, t = -2.0, which is the edge.

Same sign every time, shrinking as I get stricter. I would call the ordering real and the size about a third of a point.

Which puts Brainwaves in an odd place

Brainwaves mxfp8 at 78.83 is the highest reference row in the entire corpus. It is 6.5 points past your own crossover, further right than anything else you have published.

Your theory says qx should do nothing for cognition there. Your table agrees, at -0.23pp.

So the thing you say you hear in the Brainwaves qx, the conversation flowing and the metaphors landing, is not in these seven tasks and by your own trend was never going to be.

That is the interesting part, not a problem with it.

Somewhere below 72 you have a model where the cognitive rescue was large. Did the social thing get louder down there too, or was it flat? If the two move together they are one effect and this suite is a weak proxy for it. If the social improvement is quiet at 65 and loud at 79, it is a second effect that has been sharing a name with the rescue this whole time.

Which model was it, and which way did it go?

Brainwaves

I created this model as an experiment to see how it behaves under different RoPE and quantization settings. For 2M you would need a lot of RAM and it will be very slow, but 1M is manageable. Some speed can be gained with MTP, which can be added in different setups (GGUF, oMLX, etc..)

This is an experimental merge between:

• nbeerbower/Wichtel-Qwen3.6-27B
• trohrbaugh/Qwen3.8-27B-heretic-ara
• DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1
• DavidAU/Qwen3.6-27B-V1.1-FF711-Darker-Hero-GAIN-H2.0
• nightmedia/Qwen3.8-27B-Cold-Fusion-FF711-Darker-Hero-GAIN-B

          arc   arc/e boolq hswag obkqa piqa  wino
mxfp8     0.732,0.888,0.916,0.830,0.524,0.832,0.796
qx86-hi   0.732,0.886,0.914,0.836,0.520,0.830,0.792
qx64-hi   0.732,0.890,0.913,0.835,0.504,0.836,0.792
mxfp4     0.729,0.888,0.915,0.824,0.514,0.827,0.793
1M
mxfp8     0.734,0.888,0.915,0.831,0.538,0.834,0.792
qx86-hi   0.733,0.887,0.911,0.836,0.524,0.831,0.785
qx64-hi   0.730,0.886,0.913,0.835,0.516,0.836,0.792
mxfp4     0.731,0.888,0.915,0.823,0.520,0.829,0.790
2M
mxfp8     0.729,0.890,0.915,0.831,0.536,0.835,0.791
qx86-hi   0.730,0.887,0.913,0.836,0.526,0.832,0.787
qx64-hi   0.732,0.885,0.913,0.834,0.524,0.834,0.786
mxfp4     0.730,0.888,0.914,0.823,0.526,0.827,0.788

Quant     Perplexity      Peak Memory   Tokens/sec
mxfp8     3.656 ± 0.022   34.74 GB      175
qx64-hi   3.624 ± 0.022   27.03 GB      161
mxfp4     3.727 ± 0.023   21.30 GB      175
1M
qx86-hi   3.678 ± 0.022   33.21 GB      171
qx64-hi   3.627 ± 0.022   26.99 GB      176
mxfp4     3.734 ± 0.023   21.26 GB      167
2M
mxfp8     3.669 ± 0.022   34.70 GB      157
qx64-hi   3.633 ± 0.022   26.99 GB      170
mxfp4     3.740 ± 0.023   21.26 GB      174

Three quants are available, RoPEd at different lengths--this can be easily changed in the config file.

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-1M-qx86-hi-mlx

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-2M-qx64-hi-mlx

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-mxfp8-mlx

Source:
https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves