VRAM requirements

#1
by Mageisop - opened
•
This comment has been hidden (marked as Off-Topic)

64 GB is more than enough — the whole model fits in VRAM.
Edge0-145B FP4 needs about 39 GB on the GPU: experts 34.9 + dense 1.55 + D/U 2.67. Same story for the PLE table — 51 GB on the SSD, read with --ple-io direct, no VRAM and no measurable cost. So 64 GB holds the entire pack with room to spare. you can use these flags
--expert-cache auto --prefill auto --kv int8 --spec 4 --mtp rt
--ple-gguf ple-fp8.gguf --ple-io direct

You also do not need 64 GB to run it by. 16 GB VRAM + 20 GB RAM gives close to 50 tok/s with --resident-budget-gib 20 --expert-profile expert-profile.bin

I've tried on a dual RTX5070Ti (16GB + 16GB VRAM) build with 128GB VRAM. the nvfp4 keep crashing out of memory. How can we run on 16GB VRAM?

Is there any quality difference between the nvfp4 vs q3k version? It's a bit confusing that edge0 treats both the same?

Edge0 org
•
This comment has been hidden
Edge0 org

I've tried on a dual RTX5070Ti (16GB + 16GB VRAM) build with 128GB VRAM. the nvfp4 keep crashing out of memory. How can we run on 16GB VRAM?

Is there any quality difference between the nvfp4 vs q3k version? It's a bit confusing that edge0 treats both the same?

16 GB VRAM is enough — Edge0 runs on 16 GB VRAM + 20 GB RAM. This sounds like a configuration issue, not a memory limit.

Can you share your config? In particular the engine command line and whether you set --vram-reserve-mib (it controls how much VRAM the engine leaves free for everything else; the rest goes to the expert cache). can you try --expert-cache 4000
--vram-reserve-mib 1024
--kv-resident 32768

I lost my earlier config, but I tried your recommendation it doesn't work either, crash when starting. Claude suggested to use --expert-cache of 1024 it starts, but crashed with following during first inference.

It seems unstable still compared to Strata. My hardware works well with both Strata and Speedy for qwen3.8-flash-next. Is it because this model has more param (bigger)?

"time":"2026-11-13T13:15:56.304Z","endpoint":"/v1/messages","source":"172.17.0.1","http_status":503,"stream":false,"total_seconds":1.06794404,"code":"server_error","message":"the engine stopped unexpectedly (exit code 1); the next request restarts it","stop_reason":"failed","input_tokens":null,"reused_tokens":null,"generated_tokens":null,"first_token_seconds":null,"decode_tokens_per_second":null,"prefill_tokens_per_second":null,"decode_seconds":null}
[strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...
[strata] the engine is running again
[strata] the engine reported an error: prefill latent: an expert is neither resident, streamed nor on the peer
[strata] done: 0 tokens in 0 s (no speed) (error, cancel=False)
[strata] 400 invalid request: prefill latent: an expert is neither resident, streamed nor on the peer
{"time":"2026-11-13T13:16:14.793Z","endpoint":"/v1/messages","source":"172.17.0.1","http_status":400,"stream":false,"total_seconds":17.412053988,"code":"invalid_request_error","message":"prefill latent: an expert is neither resident, streamed nor on the peer"}
[strata] the engine stopped unexpectedly (exit code 1). The engine exited (code 1) after it reported: prefill latent: an expert is neither resident, streamed nor on the peer - please report it at github.com/Niko1221/Strata/issues with the log. The next request starts the engine again.

The crash on your dual RTX5070Ti setup sounds like a memory fragmentation or eviction issue rather than a hard VRAM limit, especially with that much total VRAM (128GB). The suggestion to reserve more VRAM for the engine (--vram-reserve-mib 1024) and pinning the expert cache (--expert-cache 4000) is a solid diagnostic step. If you're still seeing OOM errors, it might be worth checking if the --kv-resident setting is too aggressive or if there's a bug in how the engine handles multi-GPU NVFP4 layouts on that specific hardware combination. Have you tried running with --expert-cache auto to let the system manage eviction dynamically instead of pinning a large chunk?

What I got from claude, apparently this fork of strata doesn't work for second GPU:

What was wrong
Your crash was the engine's --peer-device second-card expert-cache tier, which this engine build (edge0-strata 0.1.55) never made latent-aware for the q3k pack:

The prompt plan hands ~35% of the streamed experts to the peer's own staging ring (STRATA_PF_PEER_STREAM, default 0.35), but the latent prompt walk (STRATA_LAT_FUSED_Q3K=1) still demands a blob for every routed expert — the diverted ones are neither in the primary's cache, nor streamed, nor resident on the peer → prefill latent: an expert is neither resident, streamed nor on the peer, engine exit 1, on every request, after every restart.
set_peer declines NVFP4 packs (per-expert tail) but not q3k — which is why your earlier NVFP4 setup ran fine on two cards.
Behind it hid a second bug: I bypassed the crash with the engine's own --peer-prefill-rows 0 (prompt path fully on the primary) — and every generation then came back NaN logits (#879), deterministically, before the first token. The peer's decode rows poison the model on this pack. So both two-card modes are broken upstream right now.

What works (verified end-to-end)
Single card + KV streaming — 2,301-token prompt prefilled in ~5 s, correct answers (17×23 = 391), clean end_turn:

STRATA_ARGS="--kv-resident 32768 --vram-reserve-mib 1024 --mtp-draft-vocab …/draft_vocab_en.bin"
Note --kv-resident 32768 is mandatory on one 16 GB card at 262144 context — without it the engine exits before ready (verify: the device arena (235 MiB) does not fit), which I confirmed by running the engine binary directly.

Details:

The Edge0-145B-q3k pack on the edge0-strata engine (0.1.55, image built 2026-11-13) is broken in every two-card mode; the working setup is one 16 GB card with --kv-resident 32768 --vram-reserve-mib 1024 at the full 262144 context (verified end-to-end 2026-11-13: coherent generations, 391 = 17×23).

Why: the engine's --peer-device expert-cache tier was never made latent-aware for the q3k pack: (1) the prompt plan hands ~35% of streamed experts to the peer's staging ring (STRATA_PF_PEER_STREAM, default 0.35) but the latent fused prompt walk still expects a blob for every routed expert → first prompt dies "prefill latent: an expert is neither resident, streamed nor on the peer", engine exit 1, every restart, every request. (2) With the prompt path kept on the primary (--peer-prefill-rows 0, which skips Prefill::set_peer entirely), the peer's DECODE rows still poison the model: "logits became non-finite (#879)" on every request, deterministically, before the first token. set_peer declines NVFP4 packs (per-expert tail) but not q3k (tail_off = 0), which is why the NVFP4 pack worked with two cards and q3k does not.

How to apply: edge0/start-edge0-145b-q3k.sh defaults to single-GPU; EDGE0_MULTI_GPU=1 opts into the (still broken) peer branch for testing future engine builds. Without --kv-resident a single 16 GB card cannot start at 262144 ctx — the engine exits before ready with "verify: the device arena (235 MiB) does not fit". Debugging engine-side failures: the engine's stderr only lands somewhere if the launch template has a log key (the entrypoint now writes /tmp/engine.log; read it with docker exec edge0-cuda tail -100 /tmp/engine.log); for a bare run, docker run --entrypoint /opt/edge0/edge0-strata/build/strata edge0-cuda --serve <template args> prints it directly. The entrypoint's template rewrites (CUDA_VISIBLE_DEVICES, --expert-cache, --prefill, log) are the hooks into the delegated engine launch. If upstream fixes the tier, re-test with EDGE0_MULTI_GPU=1 before re-defaulting it. Related: [[strata-dspark-drafter-moved-into-coder-folder]].

I'm trying again with nvfp4 version again now and will update if it works

Just tested nvfp4, same issue with q3k, only works on single GPU but not two GPUs.

Written for: the edge0-strata/Strata maintainers — full report saved as BUGREPORT-dual-gpu.md, ready to paste into an issue. Here's the same content, for you:

Commands (non-dockerized, raw engine)
The engine normally gets these args from serve/server.py expanding the launch template; this is the resolved form. Run from an edge0-strata checkout with build/strata built.

Single GPU (works, both packs):

PACK=/path/to/Edge0-145B-NVFP4 # or Edge0-145B-q3k
env
STRATA_LATENT_A="$PACK/experts.bin"
STRATA_LATENT_LABELS="$PACK/labels.bin"
STRATA_LATENT_DENSE="$PACK/latent_dense_q8.bin"
STRATA_LATENT_DENSE_FMT=8
STRATA_EMBED_VRAM=1 STRATA_LATENT_FUSED=1 STRATA_LATENT_FUSED_DU=1 STRATA_LATENT_U_GPU=1
CUDA_VISIBLE_DEVICES=0
./build/strata --serve
--pack "$PACK"
--native "$PACK/model-skeleton.gguf"
--native-dense-gguf "$PACK/model-skeleton.gguf"
--ple-gguf "$PACK/ple-fp8.gguf"
--ple-io direct
--expert-profile "$PACK/profile.bin"
--expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5
--mtp "$PACK/mtp"
--max-context 262144 --kv int8
--kv-resident 32768 --vram-reserve-mib 1024
(--kv-resident 32768 is mandatory on a 16 GB card at 262144 context — otherwise the engine exits before ready: verify: the device arena (235 MiB) does not fit. For q3k add STRATA_LAT_FUSED_Q3K=1.)

Dual GPU (the failing mode): same command with CUDA_VISIBLE_DEVICES=0,1 and these args added:

--peer-device 1 \
--peer-reserve-mib 2048 \
--prefill 1024 \

Diagnostic variants: --peer-prefill-rows 0 (peer out of the prompt path entirely) and STRATA_PF_PEER_STREAM=0 (peer's prompt share off).

Why dual GPU failed, and where the fault lives
It's the strata fork (edge0-strata's latent-MoE additions) — not upstream Strata's peer tier, and not Edge0 itself. Upstream's tier is designed for standard MoE packs; the broken paths are the fork's latent-specific code. And it's not the pack: both NVFP4 and q3k serve perfectly single-GPU on the same build. Since Edge0 ships this fork as its engine, from a user's view it's "Edge0's engine" — two distinct bugs:

Bug A — q3k prefill crash (prefill latent: an expert is neither resident, streamed nor on the peer, deterministic): With --peer-device, the per-chunk prompt plan sends ~35% of the streamed experts to the peer's own staging ring (ps_frac 0.35, STRATA_PF_PEER_STREAM), excluding them from the primary's stream sequence. The non-latent MMQ prompt path knows about this — a per-layer classification (on_peer) removes those experts from the primary's list and hands them to the peer. But the latent fused prompt walk (the default A-stage for latent packs) iterates all expert ids and, for any routed expert missing from the primary's sequence, resolves its blob as: primary cache → peer VRAM slot → error. A peer-streamed expert is in none of those — its bytes live transiently on the peer's ring — so the first chunk ≥1024 tokens dies at prefill.cpp:5586. Why NVFP4 escapes: set_peer refuses the peer prompt path for NVFP4 ("not for NVFP4 experts yet" — per-expert scale tails) and for Hadamard packs. q3k's shared colscale means tail_off == 0, so the guard passes and binds the peer — the exact guard that would have saved q3k is bypassed by its own format.

Bug B — NaN logits from peer decode (both packs; intermittent on NVFP4, deterministic on q3k): The decisive experiment: q3k with --peer-prefill-rows 0 — the peer completely unbound from the prompt path — still NaNs before the first token, every request. So the corruption is in the peer's decode-time row computation. For a standard MoE, peer rows are self-contained per expert; for Edge0's latent MoE they are not: each entry needs the D projection into the per-layer latent space, gate/up over latent rows selected via the per-layer label mapping (STRATA_LATENT_LABELS), the U projection back, and the colscale handling (shared per-layer on q3k, per-expert in-blob on NVFP4). The peer's decode path doesn't reproduce that chain correctly. On NVFP4 the failure defers until the adaptive swaps (PeerExperts::adapt) start feeding swapped-in experts after the first requests — which matches the measured pattern: 3 clean answers, then 13 non-finite (#879) events over 4 requests, permanent from there.

The one-line upstream mitigation (even before a real fix): make peer-tier setup decline latent packs the way set_peer already declines NVFP4 prompt participation — today it silently corrupts generation instead of refusing at startup. Full call-chain with line references (against fb0c9b6cc4150, engine 0.1.55) is in the report file, along with the repro matrix and the hardware/version block.

Edge0 org

Just tested nvfp4, same issue with q3k, only works on single GPU but not two GPUs.

Written for: the edge0-strata/Strata maintainers — full report saved as BUGREPORT-dual-gpu.md, ready to paste into an issue. Here's the same content, for you:

Commands (non-dockerized, raw engine)
The engine normally gets these args from serve/server.py expanding the launch template; this is the resolved form. Run from an edge0-strata checkout with build/strata built.

Single GPU (works, both packs):

PACK=/path/to/Edge0-145B-NVFP4 # or Edge0-145B-q3k
env
STRATA_LATENT_A="$PACK/experts.bin"
STRATA_LATENT_LABELS="$PACK/labels.bin"
STRATA_LATENT_DENSE="$PACK/latent_dense_q8.bin"
STRATA_LATENT_DENSE_FMT=8
STRATA_EMBED_VRAM=1 STRATA_LATENT_FUSED=1 STRATA_LATENT_FUSED_DU=1 STRATA_LATENT_U_GPU=1
CUDA_VISIBLE_DEVICES=0
./build/strata --serve
--pack "$PACK"
--native "$PACK/model-skeleton.gguf"
--native-dense-gguf "$PACK/model-skeleton.gguf"
--ple-gguf "$PACK/ple-fp8.gguf"
--ple-io direct
--expert-profile "$PACK/profile.bin"
--expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5
--mtp "$PACK/mtp"
--max-context 262144 --kv int8
--kv-resident 32768 --vram-reserve-mib 1024
(--kv-resident 32768 is mandatory on a 16 GB card at 262144 context — otherwise the engine exits before ready: verify: the device arena (235 MiB) does not fit. For q3k add STRATA_LAT_FUSED_Q3K=1.)

Dual GPU (the failing mode): same command with CUDA_VISIBLE_DEVICES=0,1 and these args added:

--peer-device 1 \
--peer-reserve-mib 2048 \
--prefill 1024 \

Diagnostic variants: --peer-prefill-rows 0 (peer out of the prompt path entirely) and STRATA_PF_PEER_STREAM=0 (peer's prompt share off).

Why dual GPU failed, and where the fault lives
It's the strata fork (edge0-strata's latent-MoE additions) — not upstream Strata's peer tier, and not Edge0 itself. Upstream's tier is designed for standard MoE packs; the broken paths are the fork's latent-specific code. And it's not the pack: both NVFP4 and q3k serve perfectly single-GPU on the same build. Since Edge0 ships this fork as its engine, from a user's view it's "Edge0's engine" — two distinct bugs:

Bug A — q3k prefill crash (prefill latent: an expert is neither resident, streamed nor on the peer, deterministic): With --peer-device, the per-chunk prompt plan sends ~35% of the streamed experts to the peer's own staging ring (ps_frac 0.35, STRATA_PF_PEER_STREAM), excluding them from the primary's stream sequence. The non-latent MMQ prompt path knows about this — a per-layer classification (on_peer) removes those experts from the primary's list and hands them to the peer. But the latent fused prompt walk (the default A-stage for latent packs) iterates all expert ids and, for any routed expert missing from the primary's sequence, resolves its blob as: primary cache → peer VRAM slot → error. A peer-streamed expert is in none of those — its bytes live transiently on the peer's ring — so the first chunk ≥1024 tokens dies at prefill.cpp:5586. Why NVFP4 escapes: set_peer refuses the peer prompt path for NVFP4 ("not for NVFP4 experts yet" — per-expert scale tails) and for Hadamard packs. q3k's shared colscale means tail_off == 0, so the guard passes and binds the peer — the exact guard that would have saved q3k is bypassed by its own format.

Bug B — NaN logits from peer decode (both packs; intermittent on NVFP4, deterministic on q3k): The decisive experiment: q3k with --peer-prefill-rows 0 — the peer completely unbound from the prompt path — still NaNs before the first token, every request. So the corruption is in the peer's decode-time row computation. For a standard MoE, peer rows are self-contained per expert; for Edge0's latent MoE they are not: each entry needs the D projection into the per-layer latent space, gate/up over latent rows selected via the per-layer label mapping (STRATA_LATENT_LABELS), the U projection back, and the colscale handling (shared per-layer on q3k, per-expert in-blob on NVFP4). The peer's decode path doesn't reproduce that chain correctly. On NVFP4 the failure defers until the adaptive swaps (PeerExperts::adapt) start feeding swapped-in experts after the first requests — which matches the measured pattern: 3 clean answers, then 13 non-finite (#879) events over 4 requests, permanent from there.

The one-line upstream mitigation (even before a real fix): make peer-tier setup decline latent packs the way set_peer already declines NVFP4 prompt participation — today it silently corrupts generation instead of refusing at startup. Full call-chain with line references (against fb0c9b6cc4150, engine 0.1.55) is in the report file, along with the repro matrix and the hardware/version block.

Jackal — thanks, that report is genuinely useful, and the diagnosis points at the right place: the latent-MoE peer tier in our engine fork, not the pack and not your config. We are on it and will fix it as soon as we can. Until then, single GPU is the supported path, with --kv-resident 32768 as you found.

On NVFP4 vs Q3_K: they land about the same, and that is expected — Q3_K is not uniformly 3-bit. Only a subset of the modules are q3k and the rest stay at q4, so the overall precision ends up close to NVFP4's and the benchmark scores land in the same place. The card quoting one quality section for both is deliberate, not an oversight.

Also, same issue with --split-layer, the earlier one was using peer-mode

What happened
The split itself engages beautifully: the engine plans K by free VRAM, loads layers on both cards (CUDA1 took layers [24,48) + head + draft), doubles the cached experts (~9.0k of 24.6k profiled pairs ≈ 97% of routed mass, vs ~2.6k on one card), captures the verify-window graphs on both cards, and prompt prefill crosses the cards cleanly. Then the first decode verify window on the second card faults, on every request, and the engine stays wedged until restart:

verify: layer N never rang (an illegal memory access was encountered)
N is always the second card's first layer — the node that reads the cross-card hand-off buffer. For this latent model that hand-off carries the D-stage xl rows through mapped host memory (verify.cpp:996), which is the likely suspect.

The matrix (2× RTX 5070 Ti, NVFP4 pack, engine 0.1.55)
Run Split Spec Decode Draft Result
1 K=24 4 serial CUDA1 IMA at layer 24
2 K=25 4 serial CUDA1, +--vram-reserve-later-mib 1348 IMA at layer 25
3 K=25 0 — — refused: native packs need --spec ≥ 2
4 K=25 2 serial CUDA1 IMA at layer 25
5 K=22 4 --pipeline-windows 2 CUDA0 (STRATA_MTP_STAGE0=1) IMA at layer 22
Run 2 rules out VRAM pressure (768 MiB free on CUDA1 still faulted; CUDA0 runs the same graphs at 716 MiB), run 5 rules out draft placement. So both of the engine's two-card modes are dead for this latent model — same story as the peer tier, one level earlier in the pipeline. There's no no-MTP control: the engine refuses --spec 0 on a native pack. q3k wasn't retested (pack is deleted from disk).

State
Machine restored: no containers, GPUs idle — as it was before.
Findings recorded in BUGREPORT-dual-gpu.md (new §8 with the matrix), plus notes in start-edge0-145b-nvfp4.sh, entrypoint.sh, and README.md. Uncommitted — say "sync git" if you want them pushed.
Worth retrying when a newer edge0-strata build lands (the image re-pulls on rebuild); single-card with --kv-resident 32768 remains the only stable mode.

Jackal — thanks, that report is genuinely useful, and the diagnosis points at the right place: the latent-MoE peer tier in our engine fork, not the pack and not your config. We are on it and will fix it as soon as we can. Until then, single GPU is the supported path, with --kv-resident 32768 as you found.

On NVFP4 vs Q3_K: they land about the same, and that is expected — Q3_K is not uniformly 3-bit. Only a subset of the modules are q3k and the rest stay at q4, so the overall precision ends up close to NVFP4's and the benchmark scores land in the same place. The card quoting one quality section for both is deliberate, not an oversight.

Thanks, really hope to see this fix in edge0 soon as I believe there are many users with dual 16GB GPU setup due to the heated GPU market esp. for 32GB cards right now.

Jackal — same issue to the letter. And I also have 2x 5070 Ti. In Basalt, they hit about 1.7 - 1.75 more tokens than a single 5070 Ti which I find amazing.

But in Edge0, illegal memory error and such. I had to uninstall for the time being. But good that I had a chance to see this post. It's a pity that we cannot use our full capacity for now.
Let's hope it will be solved soon. Actually, Strata also supports multi-GPU setups. It will not be too difficult to apply it to Edge0 I hope.
Cheers.

Thanks, really hope to see this fix in edge0 soon as I believe there are many users with dual 16GB GPU setup due to the heated GPU market esp. for 32GB cards right now.

Edge0 org

Fixed — both packs run on two cards now (@huggingbirdy too). It is merged in Edge0 (main, PR #230), so build from main or wait for the next image.
The peer GPU now computes only the A stage of its experts and returns q8_1 rows; the first card runs U for them, so no NVLink or P2P is needed. The latent prompt stays on the first card, and each layer-split stage uploads its own D/U.

"gpu": [0, 1],
"args": [ /* existing */, "--peer-device", "1", "--peer-reserve-mib", "1024", "--peer-slots", "0" ]

cd /path/to/Edge0-145B-NVFP4
python serve/server.py --engine strata --config ./model-two-gpus.json

edge0-infer --device names one card, so start the engine directly for now; in Docker expose both cards. Keep --vram-reserve-mib 1024 and --kv-resident 32768; --peer-prefill-rows and STRATA_PF_PEER_STREAM are no longer needed.

update: confirmed that using layer split mode is still crashing. Using peer tier mode does not utilize dual GPU effectively, first GPU can use up to 100% but second GPU is always at max. of ~15%

jackal@jackal-desktop ~/repo/dockerized-apps/edge0 (main)$ ./start-edge0-145b-nvfp4.sh

Edge0-145B (NVFP4) — Edge0 Server
Port: 8001 (the port every LLM server here uses — stop the other first)
Context: 262144
Devices: --layer-split 24 — a fixed 24/24 split of the 48 layers across the two cards;
STRATA_STAGE_TRIM=1 — each card loads only its own layers' dense weights
Expert cache: auto (engine-sized, per card), prompt chunks 512 tokens, KV streams beyond 32k (--kv-resident)
NOTE: still broken on the last measured build (engine 0.1.55, 2026-11-15): the split
loads, plans and prefills across both cards, but the FIRST request dies 'mtp prefill:
an illegal memory access was encountered', with the draft on either card. For the
working peer tier instead: STRATA_ARGS="--peer-device 1 --peer-reserve-mib 2048" PREFILL=1024

Multi-GPU: 2 cards visible to the engine; their roles come from STRATA_ARGS (--layer-split auto in the start script — the layers split across the cards).
Prompt chunks: 512 tokens (the template's auto replaced).
Serving: edge0-infer serve --model /models/edge0-145B-NVFP4 --host 0.0.0.0 --port 8001 --api-key-file /run/edge0-api-key --device cuda:0 --context 262144 --model-option du=q8 --model-option ple-io=direct --strata-arg --layer-split --strata-arg 24 --strata-arg --vram-reserve-mib --strata-arg 768 --strata-arg --vram-reserve-later-mib --strata-arg 1425 --strata-arg --kv-resident --strata-arg 32768 --strata-arg --mtp-draft-vocab --strata-arg /opt/edge0/edge0-strata/data/draft_vocab_en.bin
Model name for requests: edge0_145b (API key: the API_KEY the container was given)
edge0-strata: starting edge0_145b from /opt/edge0/edge0-strata on cuda:0
loading the model (the first start takes a minute or two) ...
[strata] starting the engine: reading the model's weights ...
[strata] loading the experts into RAM (about 6 GB) and locking part of them for the GPU.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the server says when it is ready.
[strata] still starting (24 s) - please wait ...
[strata] experts loaded: 44.30 GiB at 3.07 GiB/s (27 s so far)
[strata] filling the GPU's expert cache (3953 experts, 7.13 GiB of VRAM) ...
[strata] almost ready ...
ready: http://127.0.0.1:39959/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 262144 tokens)
open http://127.0.0.1:39959/ in a browser to chat; close this window to stop the model
edge0-strata: ready (pid 79, port 39959)
Listening: http://0.0.0.0:8001 model=edge0_145b
[strata] thinking limited to 24576 of max_tokens 32768 so the answer has room ("answer_room": false in the config turns this off)
[strata] the engine reported an error: mtp prefill: an illegal memory access was encountered
[strata] done: 0 tokens in 3 s (no speed) (error, cancel=False)
{"time":"2026-11-15T17:50:48.506Z","endpoint":"/v1/chat/completions","source":"172.17.0.1","http_status":200,"stream":true,"total_seconds":3.10933151,"stop_reason":"unknown","input_tokens":null,"reused_tokens":null,"generated_tokens":null,"first_token_seconds":null,"decode_tokens_per_second":null,"prefill_tokens_per_second":null,"decode_seconds":null}
^Cstopping acceptance
active requests finished

setting --vram-reserve-mib 1024 also the same.

Edge0 org

update: layer split mode still crashing.checking further

Fixed — layer split works on two cards now, --mtp included.

What was wrong: the latent D/U pair existed only on the first card, and the draft layer on the last card gathered the token embedding from the first card's VRAM. Both fixed, merged in main.

Great, thanks! Confirmed to be working now with latest commit in https://github.com/Edge0-AI/Edge0/pull/234. Now I can get about ~60tok/s compared to single card of 20+tok/sec.

Amazing project

It worked - I can confirm. Many thanks. And 39 t/s is now about 67 t/s.
Frankly, it is significantly slower to 180 t/s (with even conservative settings I used in that one too) I reach in Basalt Qwen3.8-Flash-Next-NVFP4. But I will still work on it because some people I trust claim that Edge0 is better at coding tasks. If it stays slower while providing better quality, I will keep and use it.
Thanks for your efforts. Also, I ran tests with the consultation of Claude, peer-to-peer is slightly faster in my case.

I've tried basalt earlier, it appears to lobotomized the model with certain quality tradeoff to achieve higher speed. It can't even complete some simple coding task properly compared to Strata. There were multiple comments somewhere I've read people also saying the same about basalt. And the file format it requires conversion to basalt own format is a big turnoff instead of supporting gguf.

Sign up or log in to comment