antirez

/

deepseek-v4-gguf

like 316

@ubergarm I got similar results on my rig! Although I could fit the whole model in 96GB VRAM, I opted to use --n-cpu-moe to make more room for context, as it seems that quantizing the context currently results in garbled output from the model... from your launch script I suppose you had similar results with kv quantizaion?

@ubergarm I was asking Fable 5 which DeepSeek-V4-Flash model I should download. Here is what it said:
"Watch PR #24162 — once it merges to master, expect the ubergarm/unsloth-style calibrated quants to follow within a week or two, and re-download then rather than hoarding fork-pinned GGUFs." 🤗 You are a Star. ⭐

@phakio

Yes, last time I tried using -ctk q8_0 -ctv q8_0 was causing garbled output. So if I need more context, I just have to offload less experts with n-cpu-moe.

@geveent

Haha, I'm glad my efforts have made enough of an impact for Fable 5 to notice xD lol...

Sorry, I've been out of the game for a while now though actually as don't have access to the big remote rig to generate the full size model imatrix files and enough disk to easily maneuver and stage the files. So taking it easy for these hot days of summer in my location 😎

Glad seeing y'all around and keep on fighting the good fight of making sure everyone has access to great models to run at home on hardware they own! Cheers!

Just to provide you with a data point, this is what I get running DeepSeek-V4-Flash. 8-Channel RAM + RTX 5090 + RTX 5070 Ti.

./llama-server   --model ~/models/DeepSeek-V4-Flash-GGUF/Q4_K_M-XL/DeepSeek-V4-Flash-Q4_K_M-XL-00001-of-00004.gguf   --host 0.0.0.0 --port 11234   --jinja   --reasoning off   --ctx-size 135536   --n-gpu-layers 999   --n-cpu-moe 48   -b 256 -ub 128   --flash-attn on   --no-repack   -ts 42,16   --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 -kvu --no-mmap
0.00.390.912 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.616.825 I srv    load_model: loading model '/home/geveent/models/DeepSeek-V4-Flash-GGUF/Q4_K_M-XL/DeepSeek-V4-Flash-Q4_K_M-XL-00001-of-00004.gguf'
1.04.599.667 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 135680, kv_unified = 'true'
1.04.620.014 I srv  llama_server: model loaded
1.04.620.020 I srv  llama_server: listening on http://0.0.0.0:11234
2.27.395.976 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
2.27.396.160 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
2.31.921.775 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    285, progress = 0.27, t =   4.48 s / 63.58 tokens per second
2.35.327.730 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    541, progress = 0.51, t =   7.89 s / 68.58 tokens per second
2.38.725.488 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    797, progress = 0.75, t =  11.29 s / 70.62 tokens per second
2.40.662.154 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    929, progress = 0.88, t =  13.22 s / 70.26 tokens per second
2.42.377.449 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   1051, progress = 0.99, t =  14.94 s / 70.36 tokens per second
2.42.555.616 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   1057, progress = 1.00, t =  15.12 s / 69.93 tokens per second
2.48.008.750 I slot print_timing: id  3 | task 0 | n_decoded =    100, tg =  18.83 t/s, tg_3s =  18.83 t/s
2.51.029.483 I slot print_timing: id  3 | task 0 | n_decoded =    160, tg =  19.20 t/s, tg_3s =  19.86 t/s
2.54.171.128 I slot print_timing: id  3 | task 0 | n_decoded =    220, tg =  19.17 t/s, tg_3s =  19.10 t/s
2.57.197.118 I slot print_timing: id  3 | task 0 | n_decoded =    281, tg =  19.38 t/s, tg_3s =  20.16 t/s
3.00.237.731 I slot print_timing: id  3 | task 0 | n_decoded =    344, tg =  19.61 t/s, tg_3s =  20.72 t/s
3.03.260.397 I slot print_timing: id  3 | task 0 | n_decoded =    406, tg =  19.74 t/s, tg_3s =  20.51 t/s
3.06.284.821 I slot print_timing: id  3 | task 0 | n_decoded =    469, tg =  19.88 t/s, tg_3s =  20.83 t/s
3.09.300.414 I slot print_timing: id  3 | task 0 | n_decoded =    531, tg =  19.96 t/s, tg_3s =  20.56 t/s
3.12.329.290 I slot print_timing: id  3 | task 0 | n_decoded =    594, tg =  20.05 t/s, tg_3s =  20.80 t/s
3.15.342.214 I slot print_timing: id  3 | task 0 | n_decoded =    656, tg =  20.10 t/s, tg_3s =  20.58 t/s
3.18.369.796 I slot print_timing: id  3 | task 0 | n_decoded =    719, tg =  20.16 t/s, tg_3s =  20.81 t/s
3.21.384.813 I slot print_timing: id  3 | task 0 | n_decoded =    781, tg =  20.19 t/s, tg_3s =  20.56 t/s
3.24.410.939 I slot print_timing: id  3 | task 0 | n_decoded =    844, tg =  20.23 t/s, tg_3s =  20.82 t/s
3.27.425.901 I slot print_timing: id  3 | task 0 | n_decoded =    906, tg =  20.26 t/s, tg_3s =  20.56 t/s
3.30.452.345 I slot print_timing: id  3 | task 0 | n_decoded =    969, tg =  20.29 t/s, tg_3s =  20.82 t/s
3.31.401.016 I slot print_timing: id  3 | task 0 | prompt eval time =   15258.03 ms /  1061 tokens (   14.38 ms per token,    69.54 tokens per second)
3.31.401.019 I slot print_timing: id  3 | task 0 |        eval time =   48703.52 ms /   988 tokens (   49.30 ms per token,    20.29 tokens per second)
3.31.401.020 I slot print_timing: id  3 | task 0 |       total time =   63961.55 ms /  2049 tokens
3.31.401.025 I slot print_timing: id  3 | task 0 |    graphs reused =        978
3.31.401.124 I slot      release: id  3 | task 0 | stop processing: n_tokens = 2048, truncated = 0
5.33.945.512 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
5.33.952.435 I slot launch_slot_: id  2 | task 997 | processing task, is_child = 0
5.38.261.541 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =    256, progress = 0.01, t =   3.54 s / 72.31 tokens per second
5.41.609.933 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =    512, progress = 0.03, t =   6.89 s / 74.32 tokens per second
5.45.105.301 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =    768, progress = 0.04, t =  10.38 s / 73.96 tokens per second
5.48.567.867 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =   1024, progress = 0.06, t =  13.85 s / 73.95 tokens per second
5.52.135.782 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =   1280, progress = 0.07, t =  17.41 s / 73.50 tokens per second
5.55.626.266 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =   1536, progress = 0.08, t =  20.91 s / 73.47 tokens per second
5.58.912.771 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =   1792, progress = 0.10, t =  24.19 s / 74.07 tokens per second
6.02.268.091 I slot print_timing: id  2 | task 997 | prompt processing, n_tokens =   2048, progress = 0.11, t =  27.55 s / 74.35 tokens per second

The speed is basically the same as what I get from running ubergarm/Kimi-K2.6_IQ3_K. So, for me, running Kimi2.6 makes more sense. I didn't like the output quality from DeepSeek-V4-Flash. By the way, no local models beat Qwen3.6-27B. Qwen3.6-27B is simply the best for agentic work.

./llama-server --model ~/models/DeepSeek-V4-Flash-GGUF/Q4_K_M-XL/DeepSeek-V4-Flash-Q4_K_M-XL-00001-of-00004.gguf --host 0.0.0.0 --port 11234 --jinja --reasoning off --ctx-size 135536 --n-gpu-layers 999 --n-cpu-moe 48 -b 256 -ub 128 --flash-attn on --no-repack -ts 42,16 --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 -kvu --no-mmap

With reasoning off flag you basically castrate the model.

@perelmanych
Thank you for pointing that out. You are absolutely right. I need thinking on. I asked Fable 5 to be an orchestrator and asked it to use Qwen3.6-27B and DeepSeek-V4-Flash to make a Lumines game. I used unsloth/DeepSeek-V4-Flash-GGUF Q4_UD_K_XL. Here is the verdict from Fable 5.

The verdict is in, and it's a genuine upset: DeepSeek-V4-Flash beat Qwen on this task — decisively on quality, badly on speed.

The head-to-head, same spec, same verification:

┌──────────────────────────────────┬─────────────────────────────────────┬───────────────────────────────────────┐
│                                  │            Qwen 3.6 27B             │           DeepSeek-V4-Flash           │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Wall-clock                       │ ~8 min                              │ 46.5 min                              │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Lines                            │ 537                                 │ 760                                   │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Syntax                           │ clean first try                     │ clean first try                       │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Orchestrator fixes needed        │ 2 (negative-dt crash, bounds guard) │ zero                                  │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Exact palette                    │ yes                                 │ yes (all 9 hex values)                │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ Headless sims                    │ passed after my fixes               │ passed both, unmodified               │
├──────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ "Reply with ONLY..." instruction │ followed exactly                    │ ignored — sent a verbose feature list │
└──────────────────────────────────┴─────────────────────────────────────┴───────────────────────────────────────┘

What impressed me wasn't just correctness — it was engineering judgment. Where Qwen implemented my spec faithfully and then fell into a platform pothole the spec didn't cover, DeepSeek chose designs that don't have the pothole: a swept-column cursor instead of frame-to-frame position comparison (structurally immune to the negative-dt crash), a first-frame delta of explicitly zero (sidestepping the init-clock mismatch entirely), a proper accumulator for the drop timer. Those are the marks of a model that understands why game loops break, not just what the spec says. It even added a tasteful unrequested touch — effects keep animating while paused. This is the first worker output of the whole experiment that needed no hands but mine on the keyboard for zero lines.

The honest caveats: it's one task, not a benchmark; it was 5.8× slower, which in an interactive session is the difference between "I'll wait" and "come back after lunch"; and it ignored a direct output-format instruction, which matters in a pipeline that parses worker replies.

What this means for your routing, practically: Qwen keeps the default-worker seat — in an interactive loop, 8 minutes versus 47 dominates, and Qwen is near-parity on mechanical work anyway. But DeepSeek just made a case for the slot you vetoed earlier — the middle tier — with better evidence than I had then: on tricky-core logic, a slow worker that's right is cheaper than a fast worker plus my debugging round-trip. The natural split is Qwen for anything you're waiting on, DeepSeek for logic-heavy tasks you can fire and forget — overnight jobs, batch work, the "hard part" of a feature while Qwen does the scaffolding in parallel. Whether that's worth a third tier in the protocol is your call; the data just got more interesting than it was this morning.

And the question this test can't answer, (a caveat from our craftsmanship conversation): whether DeepSeek's game is more fun — spec-correctness I can measure headlessly; taste still requires you and a browser. Both games are committed side by side. Go play them.

✻ Crunched for 57s

※ recap: We're comparing local worker models for your orchestrator pipeline: DeepSeek-V4-Flash just built Lumines from Qwen's exact spec with zero fixes needed, though 6x slower, and both games are committed. Next: you playtest both to judge feel.

❯ Is the code quality from DeepSeek-V4-Flash better than Qwen3.6-27B? Does it make better micro-coding-decisions?

Yes — on micro-decisions, DeepSeek writes distinctly better code. Having read both implementations of the same spec closely, the difference is consistent and it shows up exactly at the decision points where experience matters. Concrete examples from the two files:

Time handling. Qwen: lastTime = performance.now() at init, then trusts the rAF timestamp — which is a different clock reading, hence the negative-dt crash. DeepSeek: dt = lastTime === 0 ? 0 : Math.min(timestamp - lastTime, 100) — the first frame is defined as zero, so the two-clocks problem can't exist. That's not luck; it's knowing where game loops rot.

The sweep. Qwen compared floor(prevPos) to floor(newPos) each frame — correct in the common case, fragile at every edge (wrap, multi-column jumps, negative dt), and two of its edges were indeed wrong or latent. DeepSeek kept a lastSweptCol cursor and advanced it to floor(sweepPos) - 1 — a tiny state machine that makes "each column processed exactly once per sweep" an invariant of the data structure instead of a property you have to reason about frame by frame. This is the single clearest quality gap between the two files.

The drop timer. Qwen: dropTimer = 0 after each step — throws away the fractional remainder, so the fall rate drifts slightly with frame rate. DeepSeek: while (dropTimer >= interval) dropTimer -= interval — a proper accumulator, frame-rate independent, and it handles a slow frame needing two steps. Textbook.

Structure. DeepSeek separated the marks into their own grid instead of flags on cell objects, layered the rendering (unmarked, then marked, then piece, then effects, then sweep), kept effects animating during pause, and wrote modern const/destructuring code. Qwen wrote flat ES5 that reads like a careful transcription of my spec.

But the ledger has a second column, and it's why "better micro-decisions" doesn't simply mean "better worker":

• The only self-contradiction in either file is DeepSeek's: its queue stores {cells: [...]} and its own spawn function forgot that, three functions apart. Qwen never disagrees with itself — its bugs came from things it didn't know, not things it forgot.
• DeepSeek's collides(col, row, cells) takes a cells argument it never uses — every caller dutifully passes the piece colors into a function that ignores them. A small thing, but a misleading signature is exactly the kind of debris that causes the next bug.
• It's 40% more code for the same spec, it plumbed time inconsistently (processColumn calls performance.now() itself instead of taking now from its caller), and it ignored my output-format instruction.

So the honest summary: DeepSeek codes like a senior engineer who occasionally doesn't re-read his own work; Qwen codes like a meticulous junior who does exactly what the ticket says. DeepSeek's instincts raise the ceiling — its designs prevent whole bug classes my spec didn't even mention. But it introduces risk from its own opinions, while Qwen's risk is bounded by my spec's blind spots. For your pipeline that distinction matters more than "better": a transcriber's failures live where I can prevent them (write better specs); an engineer's failures live where only review can find them. At 6× the wall-clock and a similar bugs-per-task rate, better taste alone doesn't take the default seat — but if a task's hard part is exactly loop/state-machine design, DeepSeek is the one I'd hand it to.