Multi-GPU tested: Edge0-145B-NVFP4 across 2× RTX 5060 Ti 16 GB (38.9 tok/s decode @ 122k; prefill is RAM/NVMe-bound)

#9
by path-dev - opened

Tested and fully measured Nov 15, 2026 on: Ryzen 5 7500F (6C/12T), 30 GiB DDR5, 2 TB NVMe, 2× RTX 5060 Ti 16 GB (sm_120, PCIe 5.0 x8, PHB topology, no P2P). Engine: edge0-strata @ f833336e + the layer-split patch set from Edge0-AI/Edge0#205. Layer split 24 (CUDA0: L0–23, CUDA1: L24–47), MTP drafter on card 2.

TL;DR — the 145B boots and runs cleanly on the exact same dual-GPU stack as the 120B (= model-agnostic image, only .env changes): 18/18 bench runs fault-free, zero OOM under the heaviest stress mix, quality probes green.

prompt tokens prefill tok/s decode tok/s draft acceptance
443 57.0 32.4 74.7%
3,034 163.2 37.1 68.8%
11,995 258.4 37.8 71.0%
23,965 371.2 40.1 74.6%
47,835 486.7 38.9 69.1%
89,696 490.9 38.9 71.6%

(medians of 3, temp 0, cached_tokens=0 verified; 122k-target prefill: 183 s)

Decode target met: 38.9 tok/s median @ 122k, MTP acceptance 68–75% — with one mandatory config flip vs the 145B defaults:

EMBED_VRAM=0   # the 145B model.json defaults to 1, which keeps the
               # native embedding table in card-0 VRAM — the MTP drafter
               # on card 2 is its only cross-device consumer and faults
               # (IMA in mtp.cpp gather_dev). Mapped host memory = clean.

Rest of the production config (identical to the twin-GPU 120B stack): --layer-split 24, --vram-reserve-mib 768 --vram-reserve-later-mib 1425, STRATA_STAGE_TRIM=1, fused latent kernels on, RESIDENT_BUDGET_GIB=16, KV_RESIDENT=32768, PLE_IO=direct, cgroup mem_limit: 24g. Boot: ready in ~50 s.

Finding worth knowing before you size hardware (prefill): at 43.7% + 16 GiB RAM the expert pool covers ~80% of the 24,576 expert pairs (10,727 in VRAM + 8,876 pinned). The remaining 9 GiB live on the NVMe file tier and bench filler prompts route uniformly → the misses amplify to **251 GB of SSD reads per 122k request** (engine-reported), making worst-case prefill NVMe-bound (~491 tok/s) instead of compute-bound. On higher-RAM machines this is where the 145B's prefill should improve disproportionately; decode in the 35–40 tok/s corridor is exactly what a 30 GiB host can expect.

Real-workload note: agent-style prompts route profile-like rather than uniformly (our 120B real operation showed 100% expert-cache hits), so real prefill sits between the bench worst case and the 120B's numbers.

Speed vs quality on this specific box: the Edge0-120B-NVFP4 twin-GPU config prefills ~3.5–4× faster (1,930 vs 490 tok/s at 122k) while the 145B is the qualitative step up (+3 terminal-bench / +6 deepswe per the model card) — a usage decision; both are one .env swap apart (never run both simultaneously on this hardware class).

Switch between them:

# 145B -> MODEL_DIR=/models/Edge0-145B-NVFP4, ALIAS=edge0_145b
# 120B -> MODEL_DIR= (empty = pack default), ALIAS=edge0_120b, DU_FILE= / DU_FMT= empty

Full write-up — instructions, boot capacity table, benchmark matrix, go/no-go analysis, and raw result JSONs — in a public repo:

GitHub: accelerando-developers/edge0-nvfp4-multi-gpu — includes the issue-#205 patch bundle, a reproducible Docker build, and the 120B twin-GPU thread on the model page. Both model profiles are also on Open LLM Radar:

Happy to share logs/details on request.

Edge0 org

Great report, and your diagnosis is the bug we found the same day: the embedding table lived in card-0 VRAM and the MTP drafter on card 2 was its only cross-device consumer. Fixed and merged (main, #234): a split keeps that table in mapped host memory, so EMBED_VRAM=0 is no longer needed — and with #231, the #205 patch set is not needed either.

Re-run your bench on current main with the defaults; that is the cleanest sm_120 confirmation without P2P.

Your prefill finding (~80% of pairs in the pool, the rest on NVMe → ~251 GB of reads per 122k request) is the honest sizing guidance for a 30 GiB host.

Sign up or log in to comment