Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
SeaWolf-AI 
posted an update 4 days ago
Post
2979
🚀 Adding a GPU without building one

AI is usually framed as "how smart is the model / how many GPUs did you buy." The real bottleneck is elsewhere — how efficiently you use the GPUs you already have.

Training happens once; inference runs the entire time users use your product. So a service's economics come down to cost per token. Inference acceleration uses software to pull several times more out of the same GPU — the effect of plugging in one more "virtual GPU."

VIDRAFT's VKAE, measured (B200, same-harness, no quality loss):

Qwen3.5-35B-A3B (MoE): 25.7 → 601 tok/s (23.4×)
Darwin-36B-Opus (in-house MoE): 25.0 → 280.8 (11.2×)
10,000+ tok/s peak aggregate under concurrency
The key: it's reproducible — model + serving shipped as one container.

docker pull vidraft/qwen35-vkae:601
Don't take our word for it — run it yourself. The mechanism will be released as a paper.

🏆 Leaderboard & demo 👉 VIDraft/vkae
Articles 👉 https://huggingface.co/blog/FINAL-Bench/vkae-leaderboard
dipankarsarkar

The framing is the right one. Cost per token is the axis, not parameter count, and inference is where the bill actually runs. That part I will defend all day.

The 23.4x is where I would push. 25.7 to 601 tok/s on a 35B-A3B MoE (3B active) on a B200: the ceiling number is aggregate-under-concurrency, but 25.7 reads like single-stream unbatched generate. Those are different axes. Batching alone moves an MoE by an order of magnitude before any VKAE mechanism kicks in. The honest speedup is same-concurrency both sides, same batch, same harness, VKAE on vs off. Otherwise part of what you are measuring is 'we turned batching on.'

Same for 'no quality loss.' Measured on what? Aggregate perplexity or a task average sits still while long-context recall or the rare hard token quietly moves. The loss lives in the tail, not the mean.

When the paper lands, will the headline number be VKAE-on vs VKAE-off at identical concurrency, or optimized-throughput vs a single-stream baseline?

·
SeaWolf-AI

Agreed on the framing, and the push is fair.

25.7 (single-stream) and 601 (aggregate under concurrency) aren't the same axis — you're right. The 23.4× is a system-level number, and part of it is batching that any MoE gets for free. That isn't the VKAE claim, and we'll present it as what it is rather than fold it into the speedup.

What the paper leads with is the controlled comparison: identical concurrency, batch, and harness, VKAE on vs. off. Single-stream and the throughput ceiling stay as separate, labeled context.

Same discipline on quality — the tail, not the mean: long-context recall and the hard subset, measured against VKAE-off on identical inputs. If it doesn't hold there, it doesn't ship.

To your question directly: VKAE-on vs. VKAE-off at identical concurrency. That's the headline.
dipankarsarkar

Good, that is the honest version. Once batching is held constant, the number I would build the whole page around is the same-concurrency VKAE-on-vs-off delta.

That is the only figure that is purely your mechanism. Everything the MoE gets for free drops out of it.

If it lands at 1.4x, say 1.4x and it is still a real product. A clean 1.4x reads better than a 23.4x a reader can deflate in one question.

What does that isolated delta actually come out to at your target concurrency?
In this post