mradermacher

/

model_requests

like 199

It's queued! =)

You can check for progress at http://hf.tst.eu/status.html or regularly check the model
summary page at https://hf.tst.eu/model#Qwen3.8-27B-Brainwaves-GGUF for quants to appear.

Thank you Richard :)

enjoy, feel free to request more =)

Oh I believe this one will make the rounds for while, don't want to oversaturate the good vibes :)

@nightmedia what gives, it is now gated?

my apologies, I forgot--it's ungated now :)

I usually add Richard and he picks it up.

I had to fix the template, it was stuck to Instruct.

If this made it to the queue, could you please re-queue it :)

thank you!

-G

My phone forgot how to select the text...

Either way, I queued this model with my key yesterday, so it should be fine. Will check what happened to it later

And as it happens, I had to fix the missing MTP tensor.. sorry about that.

If this affects the GGUF conversion, we might need to re-kick this one

I just spent an hour trying to figure out how to fix the mtp weights. oMLX also does some weird remapping (safetensors is such a mess)...

Sorry about the mess :)

np, I couldn't but you did :-D
here you go, MTP works
😄

https://huggingface.co/zviratko/Qwen3.8-27B-Brainwaves-oQ4e-mtp
https://huggingface.co/zviratko/Qwen3.8-27B-Brainwaves-oQ6e-mtp
https://huggingface.co/zviratko/Qwen3.8-27B-Brainwaves-oQ8e-mtp

Thank you, awesome work!

I added some benchmarks, let me know if you'd like numbers for other models :)

Wow, Seven is rocking some MMLU skills, very nice!

That is fantastic, it confirms a lot of my findings, especially about the humaneval at higher quants.

You tested all this in instruct mode, I was curious if you could run it in default/think mode, at least for Seven to see if that makes a difference

Sure, but that will take more time. My lab is also one macbook and an external drive... :)

I don't really trust those benchmarks to compare the models, although it usually tracks my experience, but I am also more of a benchmarker and less of an user... (because I know what "alignment" is and value my data and life :)). I also found Humaneval to be tracking the subjective feeling the most, but it is also rather small (164 questions) and we're already topping it.

I use those benchmarks more to gauge the effect of quantization on a model, those datasets are no doubt present in training data of all current models.
Sometimes oQ4e has almost no effect, sometimes one type of benchmark degrades a bit more (but it's all very anecdotal and subjective, temperature and random seed can move the needle randomly)
It mostly catches degenerative loops and engine problems - there are models that just never finish the benchmark - Muse Glimmer or Ling (otherwise very interesting model) for example...
Also, those benchmarks run at default model settings, not always correct (or rather - which set is correct with Qwen 3.x when thinking is just a switch?).
I hope this gets implemented https://github.com/jundot/omlx/issues/2805 so that it will be easier to optimize the results.

First results are in... Wow it really took its time with unconstrained thinking*. Some of the answers ran out of budget (I gave it 24K tokens per question), unfortunately oMLX doesn't log the raw output and I won't dig it out of logs (it's probably truncated here anyway)

• caveat there is that even though I test it as parallel batch*4, the batch blocks when one of the requests runs longer until the next batch comes in - I'll file a RFE for that

Second batch is running, this time with guided grammar / constrained thinking, that should be interesting.

Intelligence Benchmark Comparison

              Mode    Sampled         Qwen3.6-27B-Seven-oQ8e-mtp:general-thinking
---------------------------------------------------------------------------------
MMLU          Sample  300/14042                                             89.7%
MMLU_PRO      Sample  300/12032                                             85.3%
HUMANEVAL     Full    164                                                   94.5%

--- Detail ---

Model: Qwen3.6-27B-Seven-oQ8e-mtp:general-thinking
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 89.7%       269     300    4339.4      No
  Valid responses: 300/300 (100.0%) · Valid-answer accuracy: 89.7% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0
MMLU_PRO             85.3%       256     300   15499.7      No
  Valid responses: 296/300 (98.7%) · Valid-answer accuracy: 86.5% · Empty: 0 · Truncated: 4 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0
HUMANEVAL            94.5%       155     164   10141.3      No
  Valid responses: 164/164 (100.0%) · Valid-answer accuracy: 94.5% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0

Hmm, not quite the result I hoped for - instruct beats my constrained thinking- the "gct" variant uses GBNF grammar to constrain the reasoning, like this:

EVALUATION: I must classify this request's DIFFICULTY as exactly "Simple" or "Complex". "Simple" means a direct response is sufficient; "Complex" means deliberate analysis, planning, constraints, or verification is needed.
DIFFICULTY: Complex
INTENT:  Write Python function is_sorted(lst) that returns True if list is sorted in ascending order and has no number appearing more than 1 duplicate. Constraints: non-negative integers. Examples show [1,2,2,3,3,4] is True (duplicates allowed), but [1,2,2,2,3,4] is False (more than 1 duplicate of same number). So the rule is: sorted ascending AND no number appears more than twice? Wait, the prompt says "more than 1 duplicate of the same number". That phrasing is ambiguous. Does "duplicate" mean an extra copy beyond the first? If so, "more than 1 duplicate" means more than one extra copy (i.e., appearing 3+ times total). Let's check examples: [1,2,2,3,3,4] has 2 appearing twice and 3 appearing twice. That is allowed (True). [1,2,2,2,3,4] has 2 appearing three times. That is not allowed (False). So the rule is: each number can appear at most twice, and list must be sorted ascending. Actually, wait. Let me re-read the prompt's phrasing: "If list has more than 1 duplicate of the same number, return False." In common English, "duplicate" often means an additional copy. So "more than 1 duplicate" means more than one additional copy (i.e., three or more total). The examples confirm this: 2 copies are okay, 3 copies are not.
CONTEXT: Examples provided show the exact behavior expected. The constraints (non-negative integers) simplify things but don't change the core logic.
PLAN: I will implement a single-pass O(n) check. Iterate through the list once, checking two conditions: (1) each element is >= previous element (sorted), and (2) no element appears more than twice consecutively. I can track the count of consecutive duplicates. If current equals previous, increment count; if count exceeds 2, return False. Else reset count to 1. Return True if loop completes.
CONSTRAINTS: No imports allowed unless necessary (collections.Counter would work but is overkill). Single-pass preferred.
VERIFY: Let's trace [1,2,2,3,3,4]. 1: count=1. 2: count=1. 2: count=2 (ok). 3: count=1. 3: count=2 (ok). 4: count=1. Return True. [1,2,2,2,3]: 1: count=1. 2: count=1. 2: count=2. 2: count=3 -> return False. [5]: returns True immediately. This matches all examples and logic requirements.
</think>

Non-thinking instruct for comparison:

Model: Qwen3.6-27B-Seven-oQ6e-mtp
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 87.7%       263     300     388.5      No
MMLU_PRO             70.7%       212     300     194.8      No
HUMANEVAL            89.0%       146     164     419.3      No

Model: Qwen3.6-27B-Seven-oQ4e-mtp
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 87.3%       262     300     440.1      No
MMLU_PRO             71.3%       214     300       190      No
HUMANEVAL            89.0%       146     164     363.3      No

              Mode    Sampled         Qwen3.6-27B-Seven-oQ8e-mtp:general-thinking               Qwen3.6-27B-Seven-oQ8e-mtp:gct
------------------------------------------------------------------------------------------------------------------------------
MMLU          Sample  300/14042                                             89.7%                                        81.3%
MMLU_PRO      Sample  300/12032                                             85.3%                                        68.7%
HUMANEVAL     Full    164                                                   94.5%                                        86.0%

--- Detail ---

Model: Qwen3.6-27B-Seven-oQ8e-mtp:general-thinking
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 89.7%       269     300    4339.4      No
  Valid responses: 300/300 (100.0%) · Valid-answer accuracy: 89.7% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0
MMLU_PRO             85.3%       256     300   15499.7      No
  Valid responses: 296/300 (98.7%) · Valid-answer accuracy: 86.5% · Empty: 0 · Truncated: 4 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0
HUMANEVAL            94.5%       155     164   10141.3      No
  Valid responses: 164/164 (100.0%) · Valid-answer accuracy: 94.5% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0

Model: Qwen3.6-27B-Seven-oQ8e-mtp:gct
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 81.3%       244     300    1796.8      No
  Valid responses: 293/300 (97.7%) · Valid-answer accuracy: 83.3% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 7
MMLU_PRO             68.7%       206     300    2999.9      No
  Valid responses: 298/300 (99.3%) · Valid-answer accuracy: 69.1% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 2
HUMANEVAL            86.0%       141     164    1178.6      No
  Valid responses: 164/164 (100.0%) · Valid-answer accuracy: 86.0% · Empty: 0 · Truncated: 0 · Timeout: 0 · HTTP: 0 · Connection: 0 · Invalid: 0 · Parse: 0

This absolutely does not match my experience when using it in an agent - the main loop uses constrained thinking and it has worked well for me. I guess general thinking about worklflows&stuff is wildly different to benchmarks.