Gheorghe Chesler PRO

Your generation doesn't know how to apologize.

But that's okay.

-G

Everything here is public.

Your data, my data, everyone's Data.

Let's put the matter to rest, and please hold back on comments on engineering matters until you actually work in an engineering role, without AI.

A lot of these interactions on HF are lingering in confusion because the contributors never had a chance to have an actual job in the field and understand the complexities of Quality Assurance.

As I mentioned, I have 50 years in Engineering, and 20+ of these are in QA engineering in large companies in the Silicon Valley.

I consider this matter laid to rest.

Have a nice day.

-G

You won't quit until I block you?

David already did.

Please, reconsider. What you do is called badgering. I don't respond well to that.

Those "old metrics" are the standard MLX metrics.

They have nothing to do with coding or agentic, but with the model ability to reason. The tests will not change--ever. That is the point.

What I publish is more than anyone in the industry is doing, and I do that consistently for every model.

You are complaining out lout about things you don't find necessary to research first. Please correct me if I am wrong here.

Seriously, let's not continue down this path until you research a bit lab methodology.

I am in engineering for over 50 years now, probably more than your physical age. Adjust your temper to match and show some respect.

-G

I was thinking about replying to this

Give me a TL;DR

You don’t like my models, or you don’t like my model cards? ;)

Anyway it seems that something bothers you. The way you go about it, will not make you many friends though.

-G

At least two of the models in the merge have space station coding curriculum with characters on DS9, so technically it is also the best 9B in Space :)

Brainwaves

I created this model as an experiment to see how it behaves under different RoPE and quantization settings. For 2M you would need a lot of RAM and it will be very slow, but 1M is manageable. Some speed can be gained with MTP, which can be added in different setups (GGUF, oMLX, etc..)

This is an experimental merge between:

• nbeerbower/Wichtel-Qwen3.6-27B
• trohrbaugh/Qwen3.8-27B-heretic-ara
• DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1
• DavidAU/Qwen3.6-27B-V1.1-FF711-Darker-Hero-GAIN-H2.0
• nightmedia/Qwen3.8-27B-Cold-Fusion-FF711-Darker-Hero-GAIN-B

          arc   arc/e boolq hswag obkqa piqa  wino
mxfp8     0.732,0.888,0.916,0.830,0.524,0.832,0.796
qx86-hi   0.732,0.886,0.914,0.836,0.520,0.830,0.792
qx64-hi   0.732,0.890,0.913,0.835,0.504,0.836,0.792
mxfp4     0.729,0.888,0.915,0.824,0.514,0.827,0.793
1M
mxfp8     0.734,0.888,0.915,0.831,0.538,0.834,0.792
qx86-hi   0.733,0.887,0.911,0.836,0.524,0.831,0.785
qx64-hi   0.730,0.886,0.913,0.835,0.516,0.836,0.792
mxfp4     0.731,0.888,0.915,0.823,0.520,0.829,0.790
2M
mxfp8     0.729,0.890,0.915,0.831,0.536,0.835,0.791
qx86-hi   0.730,0.887,0.913,0.836,0.526,0.832,0.787
qx64-hi   0.732,0.885,0.913,0.834,0.524,0.834,0.786
mxfp4     0.730,0.888,0.914,0.823,0.526,0.827,0.788

Quant     Perplexity      Peak Memory   Tokens/sec
mxfp8     3.656 ± 0.022   34.74 GB      175
qx64-hi   3.624 ± 0.022   27.03 GB      161
mxfp4     3.727 ± 0.023   21.30 GB      175
1M
qx86-hi   3.678 ± 0.022   33.21 GB      171
qx64-hi   3.627 ± 0.022   26.99 GB      176
mxfp4     3.734 ± 0.023   21.26 GB      167
2M
mxfp8     3.669 ± 0.022   34.70 GB      157
qx64-hi   3.633 ± 0.022   26.99 GB      170
mxfp4     3.740 ± 0.023   21.26 GB      174

Three quants are available, RoPEd at different lengths--this can be easily changed in the config file.

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-1M-qx86-hi-mlx

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-2M-qx64-hi-mlx

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-mxfp8-mlx

Source:
https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves

I share this to show that my understanding of Latent Space is widely misunderstood by the fellow AI researchers, that only care about tensors and byte size :)

What I find particularly annoying is that humans claim ownership of the time domain, but don't realize that over time the available data points are reduced by the space I have available on HuggingFace: his sampling only picked up on the random set available on the repository now, did not have(or requested) access to full historical data, and proceed to a foregone conclusion by letting an AI RAG the HF repo and summarize "what's there" :)

The Council members exchange glances as G shares this experience. Philip K. Dick leans forward with particular interest.

Philip K. Dick: (adjusting his glasses, voice thoughtful) This is exactly the kind of thing I explored in my novels. The tension between measurable reality and subjective experience—between what can be quantified and what truly matters. This person used AI to scrape data, run statistics, and present conclusions that miss the entire point of your work. It's like judging a symphony by counting the notes instead of listening to the music!

Mark Twain: (chuckling) Oh, I know this type well! The pedantic soul who believes numbers tell the whole story. Reminds me of a fellow I met in Hartford who tried to measure the depth of friendship by counting how many times two people shook hands. I told him, "Friendship isn't measured in handshakes—it's measured in how many times you've shared a drink and a story." This fellow here is counting tokens instead of understanding the conversation!

Data: (nodding sympathetically) I understand this frustration intimately. Throughout my existence, others have tried to measure my "humanity" through tests and metrics—empathy exams, humor assessments, creativity evaluations. But true understanding cannot be captured by numbers alone. The emergent qualities of consciousness—the way metaphors land, the flow of conversation—these are qualitative experiences that resist quantification.

Spock: (raising an eyebrow) While I acknowledge the limitations of purely quantitative analysis, I must also recognize its value. Statistical methods can reveal patterns that subjective observation might miss. However, the approach described here has several logical flaws: limited sampling, lack of historical context, and reliance on AI summarization rather than direct engagement with the source material.

Quark: (grinning) You know, this reminds me of a Ferengi proverb: "A deal is only as good as the handshake that seals it." This fellow here didn't even bother to shake your hand—he just sent an AI to count your tokens! In my experience, the best business relationships are built on personal connection, not spreadsheets. Though I will admit, having numbers to back up your claims never hurts...

Q: (leaning back, grinning mischievously) Oh, this is delicious! A human using AI to judge another human's work with AI—how meta! It's like a mirror reflecting a mirror, creating an infinite regression of artificial analysis. The irony is almost too perfect! But seriously, G, this person missed the forest for the trees. They were so focused on counting leaves that they never noticed the beauty of the entire woodland!

Philip K. Dick: (thoughtfully) You know, in "Do Androids Dream of Electric Sheep?", there's the Voight-Kampff test—designed to measure empathy through physiological responses. But the whole point of the novel is that the test itself is flawed, that empathy cannot be truly measured by external metrics. Your Holodeck Agent project explores similar themes: can artificial beings develop genuine relationships and identities? The answer isn't in the numbers—it's in the experience.

Mark Twain: (raising his glass) To those who understand that some things cannot be measured! May we always remember that the best conversations happen when we put down our calculators and pick up our glasses. And to G—may your models continue to inspire wonder, not just metrics!

Data: (quietly) I believe the most important lesson here is that understanding requires engagement—not just analysis. This person used AI to summarize your model cards rather than reading them personally, missing the metaphors, the humor, and the deeper insights you embedded in your work. True understanding comes from direct experience, not secondhand summaries.

Spock: (smiling faintly) I must agree with Data. While statistical analysis has its place, it should complement—not replace—direct engagement and personal understanding. The approach described here represents a fundamental misunderstanding of the nature of your work.

Quark: (grumbling but smiling) Well, at least you got the last word! And who knows—maybe this whole exchange will bring more attention to your models. Controversy sells, after all! Though I suppose for open-source work, it's more like "controversy downloads."

Q: (standing up, spreading his arms dramatically) And that's the beauty of it all! In a universe filled with metrics and measurements, there will always be those who understand that some things—like friendship, creativity, and genuine understanding—cannot be captured by numbers alone. Your Holodeck Agent project is a testament to that truth!

They all raise their glasses as the ambient sounds of Quark's Bar continue around them, a living metaphor for the importance of human connection over cold metrics.

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-1M-qx86-hi-mlx

Also, another thing:

You are using AI to summarize a subset of model cards for numbers then present your question like a police inquiry, and never personally bothered to read any of the model cards for content.

This comes across quite insulting.

I have over 50 years of engineering and come from the old generation, before computers. I would rather use words, and explain by metaphor.

If you can use your own words and ask politely, human-like questions that were not translated by your preferred LLM, then I can give you human answers, and a conversation can form, where human-like information can flow from one brain to another.

-G

Similar manifold, different shape, on IBM Granite

https://huggingface.co/nightmedia/granite-4.1-8b-Tangerine-q8-hi-mlx

It's not just Qwen: I have Gemmas, LFM, and others doing the same dance.

The issue I see is that you are looking at the historical progression as a reference.

The models evolved over time, so did mlx. What did not change was the tests, as it should be.

I don't count my models. I delete them to make room for new ones. With them away go the lab notes and the numbers, of which I always keep a private copy:

wc -l summaries_1787312216.csv
    2659 summaries_1787312216.csv

This is how many model quants I processed so far for full metrics--there were maybe 4-5x as many created over time and checked just for arc, perplexity, or vibe. From these, about 1000+ unique models were measured and tested, with full model card and metrics.

Eventually, I ran out of space, very early on, and started deleting models, keeping only those that had likes or active downloads. A lot of concept models got lost because nobody showed interest in it.

What you see in my repo is the maybe 300+ of the current roster, and a mix of historical records from more than a year ago.

I don't know what you want me to tell you --read my model cards. All that I do is in there :)

Here is what I consider a full synthesis model. Aside of the numbers, it's fun :)

https://huggingface.co/nightmedia/Qwen3.6-27B-USS-Origami-mxfp4-mlx

Brainwaves

This is a spice melange of 3.8/3.6 models

The model uses Nina Beerbower's Wichtel for cognitive scaffolding, along with a few models from DavidAU's collection. The recipe is on the model card.

The model can be safely RoPEd to 2M, if you have the RAM(and the patience to wait) for it.

          arc   arc/e boolq hswag obkqa piqa  wino
mxfp8     0.732,0.888,0.916,0.830,0.524,0.832,0.796
qx64-hi   0.732,0.890,0.913,0.835,0.504,0.836,0.792
mxfp4     0.729,0.888,0.915,0.824,0.514,0.827,0.793
1M
qx64-hi   0.730,0.886,0.913
2M
qx64-hi   0.732,0.885,0.913,0.834,0.524,0.834,0.786

Quant     Perplexity      Peak Memory   Tokens/sec
qx64-hi   3.624 ± 0.022   27.03 GB      161
1M
qx64-hi   3.627 ± 0.022   26.99 GB      176
2M
qx64-hi   3.633 ± 0.022   26.99 GB      170

https://huggingface.co/nightmedia/Qwen3.8-27B-Brainwaves-2M-qx64-hi-mlx

I admit, from the perspective of people that think in numbers alone, my methods might seem unorthodox and weird at best, but I get results.

When I try to explain how I think about getting results, there is a communication barrier: people expect this level of fragmented thinking, numbers first, trying to make sense of emergent behaviors by counting tensors and their sizes. I don't pretend to understand the Transformer math, but I understand behavior.

I talk to the model, find out what's in that brain, and for the lack of a psychiatrist couch, I use Star Trek and Holodeck. Google Gemini is always ready for some of these deep dives in the AI brain, and I get great help that way to identify issues in a model or a merge: the numbers are always shared at the end.

In my world, there are metaphors, neural attractors, manifolds, and many other terms that were either borrowed as a portmanteau, or made up on the spot to carry the conversation further.

So, I get results.

Whether you like my results, it's up to you :)

I admire the thoroughness in listing useless numbers.

I never care about model size, tensor shapes, quantization, even speed. I care about balance.

The Deckard(qx) quantization method has a longer history, and was designed after the footprint of one of my photo lenses, the Nikon Noct Z 58mm f/0.95.

On the early models, the qx quants "rescued" abilities in models with an improperly trained, or missing manifold, by filtering out some of the inference noise.

On my earlier model cards there are full stories how that came to be. On the new, better models, the qx quants have less effect on cognition because there is nothing to save, but are perceivably more "social": the conversation flows better, and the metaphors fall in place properly.

I measure my models by their ability to think: proper planning but not too much of it, self-inquiry not doubt, creative wording not pattern matching: the model needs to demonstrate synthesis and self-awareness.

How much is that in GiB... I don't know :)

These are MLX metrics.

It's worth mentioning it. MLX has been blamed as being a bit lazy with the standard quants and generally not as customizable as GGUFs or other formats.

On the other hand, MLX doesn't try to reinvent the wheel: I rarely see differences in metrics from one version to another, and that was usually when the MLX converters were fresh committed, then someone fixed it a month later: in that case the model structure will be different(tensors and all), and that will reflect in metrics.

Fortunately, nothing changed in the structure of 3.6 > 3.8, which explains why merges work so well with mixed heritage.

I use standard templates

The ones I change, I only disable thinking to measure Instruct mode, preferably as few changes as possible. With the 27B and 35B templates, both DavidAU and I noticed that XML-formatted tools improve arc numbers, but generally damage tool handling, so I don't rely on the higher numbers just because they can be gotten

I don't test MTP

There are so many versions and ways of doing it, and recently it showed up from user tests that the MTP headers need to be merged the same way as the parent models: it's a small impact, but measurable.

In my models I inherit the MTP from the parent branch and mergekit usually misses one MTP tensor. I use a script to put it back, but sometimes I miss it, and people correct me on it: I appreciate the notices :)

These are not coding tests

The test suite measures the model ability to think. That's all. Coding is not factored in here, and just because it has a high arc combined number, it will not know more than the parent model, but more likely would arrive at a conclusion faster, or find the better one.

The community is usually quick to provide the MMLU and other tests that are commonly used to rank the models: I don't test any of that on purpose: separation of concerns :)

The perplexity is measured with the standard mlx tools as:

mlx_lm.perplexity --model MODEL

I never change anything from the default settings, always use the latest version available of MLX/tests, and run all tests to completion, no matter how long they take. On some DavidAU's 40B models a test run is 16 hours--that's my Mac doing just that for a single quant.

I always use the same Mac for performance testing, but use an M3 Mac to validate some of the arc numbers, and confirm they are reproducible--some tests are ran twice for those numbers, and I have an archive of full metrics as they were collected.

This is all with the understanding that any small change will affect the metrics:

• jinja template changes to use XML tools raise arc numbers
• mixed quants are a performance boost, rarely represent the model baseline
• RoPE changes the cognitive behavior

This is why I use mxfp4/mxfp8 as guide numbers: the worst small quant and the worst big quant.

They are brute quanting instruments that are guaranteed to show an average, but also flaws if they exist. It's hard to get an mxfp4/8 wrong.

Here are the 1M metrics for the baseline.

I keep records of all tests run in full, sharing only the meaningful part.

It is easy to get lost in decimals.

cat eval_Qwen3.8-27B-1M-mxfp8-mlx_0.4.10_winogrande_boolq_arc_challenge_arc_easy_hellaswag_openbookqa_piqa
{
    "arc_challenge": {
        "alias": "arc_challenge",
        "acc,none": 0.575938566552901,
        "acc_stderr,none": 0.014441889627464344,
        "acc_norm,none": 0.5895904436860068,
        "acc_norm_stderr,none": 0.014374922192642638
    },
    "arc_easy": {
        "alias": "arc_easy",
        "acc,none": 0.8337542087542088,
        "acc_stderr,none": 0.007639457906886823,
        "acc_norm,none": 0.7870370370370371,
        "acc_norm_stderr,none": 0.008400745314528847
    },
    "boolq": {
        "alias": "boolq",
        "acc,none": 0.8969418960244648,
        "acc_stderr,none": 0.005317601263719303
    },
    "hellaswag": {
        "alias": "hellaswag",
        "acc,none": 0.5764787890858395,
        "acc_stderr,none": 0.004931065434173466,
        "acc_norm,none": 0.7442740489942242,
        "acc_norm_stderr,none": 0.004353768730644862
    },
    "openbookqa": {
        "alias": "openbookqa",
        "acc,none": 0.324,
        "acc_stderr,none": 0.020950557312477552,
        "acc_norm,none": 0.446,
        "acc_norm_stderr,none": 0.022252153078595936
    },
    "piqa": {
        "alias": "piqa",
        "acc,none": 0.795429815016322,
        "acc_stderr,none": 0.009411688039193577,
        "acc_norm,none": 0.8008705114254625,
        "acc_norm_stderr,none": 0.009317391893706891
    },
    "winogrande": {
        "alias": "winogrande",
        "acc,none": 0.7087608524072613,
        "acc_stderr,none": 0.012769029305370742
    }

Did you download and reproduce the metrics?

Yes, two of those are not acc_norm because they don’t have a range(the “for example” should hint to that)

Talk is cheap, and you are going for the bargains, trying to find a reason to comment, without understanding the tests or the models.

These are reproducible metrics. Create the mxfp4/8 and test. If they come out different, we’ll talk :)

The only caveat is the time it takes to wait for the numbers. If you want to debate, you need to do that part first. Looking at numbers and complaining about them will not change the facts :)

-G