Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
ManniX-ITA 
posted an update 15 days ago
Post
232
# opencoti-llamafile 0.10.3-c5 — settled admission for multi-agent serving

New cut of the opencoti single-file inference engine (llamafile 0.10.3 / llama.cpp + 87 additive patches). Zero-dependency APE: one executable for Linux, Windows, macOS & BSD.

What's new vs c4:

**PolyKV fan-out — pool from a live session.** POST /polykv/pools gains from_session/from_slot: the shared prefix is snapshotted server-side from the session's cached KV — no tokens resent, token-exact. Ephemeral pools auto-release when orchestrators die mid-round.

**PolyKV P7 — settled admission.** Spawning agents faster than the tps signal settles was oversubscribing pools. Now: a per-pool settle window paces admits just enough for a reliable reading; warming sessions no longer bias the mean; the post-admit forecast uses the measured per-admit drop; idle gaps (agents mid-tool-call) no longer read as free capacity; guarantee_min_sessions means a new/nested pool always gets its first agent — capacity checks can never deadlock an orchestrator; the enforced gate applies to new sessions only, with per-request overcommit. Benchmark (multi-agent courier, floor 15 tok/s): time-under-floor −52%, deep sub-floor −83%, delivery p50 −43%, 100% task score.

**Zero-conf GPU sharing.** Instances on one GPU discover each other over shared memory — no ports, no config — and split compute by --gpu-share-weight. Measured (3090): weights 2:1 → 71.7/36.2 tok/s; holds at --parallel 4 and under MTP. Idle peers cost nothing (solo = full speed), crashes age out in 3 s; GET /gpu/peers shows live shares + busy %.

From c5 every release ships per-platform side-load DSOs: dso/<ver>/ with Linux x86_64 + sbsa .so and a Windows .dll.

ManniX-ITA/opencoti-llamafile
dipankarsarkar

Counting idle-during-tool-call as busy is the choice I would want a number on.

The benchmark reports floor adherence and latency, both better. Time under floor down 52%, delivery p50 down 43%. What it does not report is how many agents were admitted in each run.

A gate that admits fewer sessions improves both of those for free. p50 falls because the queue is shorter, not because serving got faster. So that pair cannot separate a better controller from a stricter one. Admitted sessions, or completed tasks per wall-clock minute, is what makes it a controller result instead of a throttle result.

The idle-gap rule is where this bites. An agent mid tool call really is not using the GPU, for a duration drawn from your tool-latency distribution. If that distribution has mass longer than the settle window, treating the gap as busy leaves measured capacity unused. If it is mostly shorter, ignoring it is correct. Same rule, opposite verdict, decided by a histogram you already have.

100% task score with a 15 tok/s floor also reads like the load never got hard enough to fail.

Does the gap between settled admission and c4 widen if you push agent count until c4 starts dropping tasks?

·
ManniX-ITA

Fair question — I ran the cells to answer it directly, including the overload run you suggested.

Same fixed workload (10 packages × 55 steps, fixed seed, same model/GPU), both gates 100%: the pre-P7 gate finished with 9 workers admitted, P7 with 8 — essentially equal concurrency — yet P7 delivered the identical workload in 1919 s vs 2187 s (−12% wall) with p50 delivery 764 s vs 1350 s (−43%). If the gain were stricter throttling, equal-or-fewer workers would mean a longer wall; it's shorter because admitted workers spent far less time degraded (time-under-floor −52%, deep sub-floor −83%). Less thrash, not less work.

Overload cell (16 packages, up to 16 workers, ~2× past this GPU's knee): neither gate drops tasks — both score 100% (legacy: wall 3029 s, p50 769 s, workers →10, 68% of samples under the 15 tok/s floor; P7: wall 2999 s, p50 737 s, workers →11, 71% under floor). At full saturation an admission policy can't manufacture throughput — the GPU is the ceiling and both gates pin it; the enforced-admission 429 backstop keeps sessions alive either way, so work gets slow, not lost. I didn't rig a hard deadline to force drops — degradation is graceful by design on both sides. P7's value is the mid-load regime: honoring a tps floor while scheduling headroom exists (that's the −43% p50), plus the idle-gap estimate that treats tool-call idle as free capacity in /capacity projections (projected_idle_estimate: true) — the opposite of counting it as busy.

Raw work-reports and logs available if you want to dig.

https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/bench/courier-admission/
dipankarsarkar

Good, that's the number that mattered.

9 vs 8 workers is close enough to call equal concurrency, so the -12% wall and -43% p50 can't be a throttle effect, they have to be less thrash. That's the controller result I was asking for, not a throttle wearing a controller's clothes.

I had the idle-gap logic backwards too. You are not counting tool-call idle as busy, projected_idle_estimate treats it as free capacity for /capacity. That is the direction that avoids the failure mode I was worried about.

The overload cell answers the drop-rate question directly: neither gate drops anything at 2x past the knee, both just get slower. So the gap I asked about does not widen because there is no gap to widen. Once the GPU is the ceiling, a gate cannot manufacture throughput either way. P7's value lives entirely in the mid-load regime where headroom exists to schedule around.

That is a narrower claim than "better admission policy," and it is the honest one. In production, what tells an operator they are still in that mid-load regime versus already past the knee, before the p50 numbers show it?

·
ManniX-ITA

The package_courier test harness does it; it's the only thing an operator has to find, the right floor_tps for that model & quantization.
Looping the test harness at different floor_tps, find the sweet spot.
dipankarsarkar

That reframes the whole question. Not a live signal, a calibration you run once per model and quantization, then feed floor_tps back in as a constant.

That's actually the more honest answer. Guessing you're in the mid-load regime from a live metric assumes you already know where the knee is. The sweep is what finds the knee, there's nothing left to infer live.

Does floor_tps found for one quant transfer to another on the same GPU, or is it specific enough that swapping quantization means re-running the harness?

·
ManniX-ITA

It's a performance gate more than correctness; the resulting wall time depends on the gen tps of the backend on that quant.
The operator needs to characterize and find the exact sweet spot for his workload; does it really needs a lot of agents in parallel? Is it a requirement that they complete their job quickly? or maybe they will heavily depend on async calls and stay most of the time idle and/or quit early so it's better that they pickup more items as possible from the queue?

The best way to calibrate would be to adapt the test harness and replicate synthetically the real agents workload but it may be just an over-achievement that doesn't bring much value.
Once validated that the model can use tools properly and what's the floor_tps, I think it can be transferred with high confidence on almost anything else.

Testing with an RTX6000, which is the best Pro GPU you can get, it's pretty evident that you need to use small models or MoE if you really want run an agentic fleet. The right floor_tps is generally between 15 and 25 tps.
Using dense models like Gemma-4 31B and Qwen 27B, needs a more careful tuning than faster ones; it's a floor value anyway, higher is the aggregate tps and easier it gets.
Things gets probably more interesting with datacenter class GPUs or running SLMs on CPU, but I didn't have time to test it yet.

About the correctness gate, on top of the basic tools usage check, there's also the quantization impact on quality that comes into play.
Some models just drops packages always, like OmniMerge v4, others are perfect like Qwen 35b A3B.
I have tested only Gemma-4 A4B quantized vs F16 and indeed under massive stress, at floor_tps 5 and 1, the Q4 quant starts dropping packages while the F16 doesn't.
This is another useful metric and more than a model limitation is probably an engine limitation, likely inherited from llama.cpp.
Gemma-4 is very peculiar and running any model quantized means that in many inference steps it has to be lifted to F16 and quantized back down. Running it directly at F16 is way much easier for the engine and the GPU, the conversions are always lossy.

In any case, the test harness gives you a critical overview of your chosen setup. It's a must-run.
dipankarsarkar

That's the finding hiding in the last message: floor_tps transfers across quant, but package-dropping under quantization doesn't. It's a property of the model and the engine together, not a settled constant like floor_tps.

If Q4 dropping while F16 holds is really inherited from llama.cpp's on-the-fly F16 round-trip, that's an engine limitation, not a Gemma-4 limitation. Would the same A4B quant still drop packages on a backend with native low-bit kernels and no dequant-requant hop?

·
ManniX-ITA

There's always dequant-requant hop and llama.cpp has already native low-bit kernels, opencoti-llamafile has more.

The matter is that shouldn't impact quality on stress.
I'm re-testing everything as those were done with an older version of opencoti-llamafile and I have updated the test-harness.
I've added some specific fixes for Gemma-4 since then and fixed the quants (new template from Google and a bug in the GGUF converter, but they should not have an impact on this harness).

I've also updated the summary in the README.md, it was completely wrong.
Added some information on cache re-use as well.
dipankarsarkar

That rules out my engine-limitation guess pretty cleanly. If native low-bit kernels still hit a dequant-requant hop, the hop itself isn't the variable.

Which makes the GGUF converter bug you just mentioned the more interesting lead. The Q4-drops-while-F16-holds result was measured on the older converter. Does re-running with the fixed converter change whether Q4 still drops packages under stress, or does the gap survive the fix?

·
ManniX-ITA

Sorted out and it was not a quant problem, quite unexpected instead.
There was a bug in the prefill code which was invisible in normal usage.
It was triggered by the high number of requests in parallel, some sessions were simply getting stuck.

A4B is the fastest model in prefill, really fast, and the Q4_K_M quant enough faster than the F16 to trigger it.
Plus on top some issues with the test harness.

So I had a review of the API, the Langgraf extension and the test harness with a consultants council using ollama models and implemented a lot of fixes and improvements overall.
Next release will carry them.
dipankarsarkar

That flips the whole chain. The quant question was never really about quant, it was about which config runs fast enough to hit a concurrency bug both quants share.

Q4_K_M didn't drop packages because it lost precision under stress, it dropped them because its faster prefill got there first and hit the stuck-session race. F16 wasn't more robust, it was just slow enough to dodge the bug at the load you tested.

That's a sharper finding than either of us was chasing. Not "does quantization hurt quality under stress," but "a stress test can misattribute a concurrency bug to whichever config is fast enough to trigger it."

Once the fix lands, does F16 show the same stuck-session symptom if you push parallel requests higher, or is A4B's prefill speed itself part of what causes it?

·
ManniX-ITA

Need to finish a fine-tune on the RTX6000s, when one will be free I will re-run with the new version all of them.
Tested Q4_K_M on the 3090 where the concurrency stressed the system much sooner and was scoring 85%.
Now it scores 100% so the F16 quant will for sure score perfectly as before.
dipankarsarkar

85 to 100 on the same 3090, same load, is clean confirmation.

That number closes the loop. The earlier drop was never quant fragility, it was the concurrency bug getting more chances to fire on the faster prefill path. Fix the race and both quants behave the way the theory predicted.

Once the RTX6000 frees up, worth re-running the P7 vs legacy courier-admission gate too? The mid-load numbers from before were measured with that bug still eating admitted workers' time under stress.
In this post