ManniX PRO

@mlabonne

Really nice client hope you will release it!

Any suggestion for replay datasets to use with a fine-tuning (if you know)?
I'm fine-tuning Gemma-4 E2B for AN (Autonomus Networks), so netconf/5g/a2a/a2a-t drivers.
Wondering if you can recommend something and what to be careful about formatting, tools usage, thinking format.
I had to normalize all to Gemma-4 format for E2B and it was a quite painful iterative process as I didn't anticipate many issues; right now I settled for Hermes Function Calling and a mix of reasoning Math.

I'm very curious to see how they fare head to head but it's a pity there's no MTP drafter, Gemma will probably win on the performance comparison.

Need to finish a fine-tune on the RTX6000s, when one will be free I will re-run with the new version all of them.
Tested Q4_K_M on the 3090 where the concurrency stressed the system much sooner and was scoring 85%.
Now it scores 100% so the F16 quant will for sure score perfectly as before.

Sorted out and it was not a quant problem, quite unexpected instead.
There was a bug in the prefill code which was invisible in normal usage.
It was triggered by the high number of requests in parallel, some sessions were simply getting stuck.

A4B is the fastest model in prefill, really fast, and the Q4_K_M quant enough faster than the F16 to trigger it.
Plus on top some issues with the test harness.

So I had a review of the API, the Langgraf extension and the test harness with a consultants council using ollama models and implemented a lot of fixes and improvements overall.
Next release will carry them.

There's always dequant-requant hop and llama.cpp has already native low-bit kernels, opencoti-llamafile has more.

The matter is that shouldn't impact quality on stress.
I'm re-testing everything as those were done with an older version of opencoti-llamafile and I have updated the test-harness.
I've added some specific fixes for Gemma-4 since then and fixed the quants (new template from Google and a bug in the GGUF converter, but they should not have an impact on this harness).

I've also updated the summary in the README.md, it was completely wrong.
Added some information on cache re-use as well.

It's a performance gate more than correctness; the resulting wall time depends on the gen tps of the backend on that quant.
The operator needs to characterize and find the exact sweet spot for his workload; does it really needs a lot of agents in parallel? Is it a requirement that they complete their job quickly? or maybe they will heavily depend on async calls and stay most of the time idle and/or quit early so it's better that they pickup more items as possible from the queue?

The best way to calibrate would be to adapt the test harness and replicate synthetically the real agents workload but it may be just an over-achievement that doesn't bring much value.
Once validated that the model can use tools properly and what's the floor_tps, I think it can be transferred with high confidence on almost anything else.

Testing with an RTX6000, which is the best Pro GPU you can get, it's pretty evident that you need to use small models or MoE if you really want run an agentic fleet. The right floor_tps is generally between 15 and 25 tps.
Using dense models like Gemma-4 31B and Qwen 27B, needs a more careful tuning than faster ones; it's a floor value anyway, higher is the aggregate tps and easier it gets.
Things gets probably more interesting with datacenter class GPUs or running SLMs on CPU, but I didn't have time to test it yet.

About the correctness gate, on top of the basic tools usage check, there's also the quantization impact on quality that comes into play.
Some models just drops packages always, like OmniMerge v4, others are perfect like Qwen 35b A3B.
I have tested only Gemma-4 A4B quantized vs F16 and indeed under massive stress, at floor_tps 5 and 1, the Q4 quant starts dropping packages while the F16 doesn't.
This is another useful metric and more than a model limitation is probably an engine limitation, likely inherited from llama.cpp.
Gemma-4 is very peculiar and running any model quantized means that in many inference steps it has to be lifted to F16 and quantized back down. Running it directly at F16 is way much easier for the engine and the GPU, the conversions are always lossy.

In any case, the test harness gives you a critical overview of your chosen setup. It's a must-run.

The package_courier test harness does it; it's the only thing an operator has to find, the right floor_tps for that model & quantization.
Looping the test harness at different floor_tps, find the sweet spot.

Fair question — I ran the cells to answer it directly, including the overload run you suggested.

Same fixed workload (10 packages × 55 steps, fixed seed, same model/GPU), both gates 100%: the pre-P7 gate finished with 9 workers admitted, P7 with 8 — essentially equal concurrency — yet P7 delivered the identical workload in 1919 s vs 2187 s (−12% wall) with p50 delivery 764 s vs 1350 s (−43%). If the gain were stricter throttling, equal-or-fewer workers would mean a longer wall; it's shorter because admitted workers spent far less time degraded (time-under-floor −52%, deep sub-floor −83%). Less thrash, not less work.

Overload cell (16 packages, up to 16 workers, ~2× past this GPU's knee): neither gate drops tasks — both score 100% (legacy: wall 3029 s, p50 769 s, workers →10, 68% of samples under the 15 tok/s floor; P7: wall 2999 s, p50 737 s, workers →11, 71% under floor). At full saturation an admission policy can't manufacture throughput — the GPU is the ceiling and both gates pin it; the enforced-admission 429 backstop keeps sessions alive either way, so work gets slow, not lost. I didn't rig a hard deadline to force drops — degradation is graceful by design on both sides. P7's value is the mid-load regime: honoring a tps floor while scheduling headroom exists (that's the −43% p50), plus the idle-gap estimate that treats tool-call idle as free capacity in /capacity projections (projected_idle_estimate: true) — the opposite of counting it as busy.

Raw work-reports and logs available if you want to dig.

https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/bench/courier-admission/

I use HE+ mostly to verify if the quant quality, if it doesn't collapse the quant is good, or to evaluate the loss at 2/3-bit.
Added MPE cause it's another canary and it has a bit more discriminative signal (depends on the mode capabilities with other coding languages than Python).
I don't use LCB because it takes hours to run; with 20-27 quants per model it's a NO GO, while usually HE+/MPE are taking 3-7 minutes each.

I cannot say:

• there's not much overlap at ISO disk size, I should compare Qwen3.5-9B F16 vs v7-coder Q6_K: not really much use for it, nobody would use it at F16
• I didn't run the evaluations for Qwen3.5-9B other than Q6_K and there's no matching quant even at Q8_0 with v7-coder
• they serve different domains; Qwen3.5-9B is an excellent Coder but the thinking is awful. Different tasks. Qwen3.5-9B can be used for simple tasks, completion or as coding assistant, not for agentic coding or any too complex thinking. It has a surprisingly good knowledge of Science but I think the high GPQA score comes more from a training bias than real reasoning capabilities. It scores very good on HE/HE+/MPE, but the LiveCodeBench score is revealing: only 58% vs v7-coder 95-98% and low AIME. Still, it's a 9B model and in that class you can hardly do more.

Yes indeed, I have my canonical OmniMergeKit evaluations.
If you check the v7-coder GGUF repo you can see the scores of Gemma-4 A4B which it's a 26B.
The balanced 184e version was awful compared to that and even miserably beaten in more than a few by the (excellent) Qwen3.5-9B. The speed advantage of A3B wasn't worth at all the 3x times VRAM usage of 27B vs 9B.

Both A4B and A3B untargeted prunes were quite disappointing; not excelling in anything, dipping low in some.
In general scoring at the same level or worse than an unpruned model with the same amount of parameters.
Not worth it.

I can track the context size and context / concurrent agents ratio and plot over time the with/without PolyKV ratio and context size with and without.

Right now I'm validating the fine-tuned model with the harness, once done will make some runs comparing all the uses cases with and without polykv, pooled and static workers.

The drawback of the pruning is clearly state in the verbosity/rumination chapter:

https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder#verbosity--rumination-length-breakdown-per-eval

What you can see in the table is that the routing, despite the pruning, is not really impacting the target usage.
The biggest issue was the collapse of IFEval, which means generically a loss of instruction following capabilities.
But the top-10 recovered it and at the end it works even better than the full 256e on AIME and MATH.
AIME and GSM8K even are scoring slightly higher with the pruned version.

It's a very different behavior from the Gemma-4 A4B pruning; it had much more less runaway both in the base and pruned but the length and rumination ballooned for almost everything.
But the cost is really low, despite that it's still very fast.
The issue with A4B is that it ends up looping which is a base behavior which is inherited and augmented by the pruning.

I will soon try with both REAM instead of REAP as suggested.

Yes, I had a look at the overlap of both; there's not much because the knowledge is very very spread over the experts.
Gemma-4 A4B has only one expert dedicated to coding fully, all the others are scattered over 128e.
Adding additional targeting ends up reducing a lot the good experts selection.
There's a huge base of mandatory experts that must be included for specific reasons and excluding them destroys the model; like those for multi-lingual and agentic end of turn.
For Qwen A3B is a bit different because there are 256 experts instead of 128e.
There's more 'choice' but the pruning hits really hard everything, scores goes down all-around: targeting with a lower weight also MPE partially recovered.
But despite that there was no way to increase MPE score without scarifying too much everything else.

I'm very busy with many other things right now but I'm considering it.
Working on the same type of harness to excercise Agent2Agent protocol that simulates a more "real" usage.
That does have 2 modes with an heavy template and a light one.

Do you mean as context of the agents increases a projection of what would be the context without PolyKV and track the ratio?

I really can't measure all use cases. The pruning is targeted for a good reason!
I have done the 184e pruning non targeted: it was terrible on almost everything.
The coding eval coverage is pretty good and in my experience covers pretty well the domains; python with HE+, multiple languages with MPE and code reasoning with LCB.

As of the targeting, it pays off to target LCB. Adding more specific targeting it's only hurting.
Once he prune is targeted for LCB it covers already almost all the experts for coding.
Adding more coding targeting is going to include low-value experts that only contains residuals in spite of the strong ones already selected by LCB.

The scores with top-8 are much worse, that's why it's needed top-10.
I tried to fix the routing with all other solutions I have, it gets worse in a way or another.

You need to think more "out of the box" and have a boarder vision.
The evaluation harness is limited and exercise a single use case.

The only number that matters is the reduction at P, Step 0.
If you consider also S per step, it's not going to tell you anything really valuable.

This is a fixed, mocked eval harness that runs a static number of steps.
It has its own purpose which is exercising and validating the API and the engine.
Doesn't really reflect a real use case; maybe there are similar ones but it's an edge case.

Every agent in the harness have a very small prefix (P=1216) which is most often not true.
Unless you are using agents to do some very basic and trivial things, hardly a real use case (most probably you don't need an agent for this kind of stuff).

In a real use case the P is what matters most of the time; in my claude-hooks consultants skill the P is gigantic: 60-200K to describe the project, provide the tools, the AST, list of the files to inspect, what was done, test results, etc etc.

Then the agent will start its work and you have no idea how many steps will take, how long will be the output, how much thinking will be needed. Ever spawned agent will have, in non synthetic tasks, most likely a very different behavior. So you can't compute and forecast a ratio of the savings due to the shared P and involve S and the steps.

The capacity lever is at P: let's frame a realistic scenario, 128k of "P" plus max 128k for "S".
If you have space for 1M tokens on your GPU and you need to run agents without PolyKV you can run 4 of them.
With PolyKV you can run 7 agents and they will be faster overall plus the context compaction gotcha.

The context compaction is another big win: without PolyKV every agent needs to be instructed to preserve the P and they all need to process its own P. They could easily mess it up and remove critical information or just make mistakes.
With PolyKV the P is static and it's not reprocessed by the agents, doesn't have a chance to get messed up.

You are really wrong when you say that the fleet size is not a lever: it's exactly the opposite.
The floor tps scheduler is a killer feature just like the shared P.
That's why I uploaded so many reports with different models and different floor tps.
Each combo model/GPU needs the right floor tps otherwise it starts crapping out and the wall time will increase, the quality will decrease. That's the value of the test harness as it is configured now.

You can't quite precisely predict the load of every agent, if you choose a static value that's it. You are bound to it.
If you are running multiples instances of any other inference engine without the floor tps scheduler you have either to under or over subscribe the number of agents.
Unless you put on top a complex custom orchestrator which needs to be managed.

Since opencoti-llamafile with the PolyKV API knows the actual load on the GPU it can auto-balance with just the right tps floor value for the model, independently of any other instance running or not running in parallel.
If there's no other load can scale up the workers up or above the available slots, your decision.
In case there's load the floor tps scheduler will restrict the amount of agents that can be spawned automatically.
I still need to improve concurrency and avoid over subscription at start but the logic is simple and should be clear.

Abou the chart: I will probably replace the graph of tps at step as it's redundant to the tps with context and less granular.
But as I said a "cell reduction" graph as you describe I don't think that really make sense.
Have to think a bit more about it on how to frame it.