Multi-GPU tested: Edge0-120B-NVFP4 across 2× RTX 5060 Ti 16 GB — layer-split works after small patches (62–67 tok/s decode)

#2
by path-dev - opened

Tested and fully measured Nov 14–15, 2026 on: Ryzen 5 7500F (6C/12T), 30 GiB DDR5, 2 TB NVMe, 2× RTX 5060 Ti 16 GB (sm_120, PCIe 5.0 x8, PHB topology, no P2P). Engine: edge0-strata @ f833336e + the layer-split patch set from Edge0-AI/Edge0#205.

TL;DR — once the unpatched layer-split crash is fixed, the 120B runs across both cards and delivers (medians, temp 0, cached_tokens=0):

prompt tokens prefill tok/s decode tok/s draft acceptance
443 611.8 66.7 73.2%
3,034 1,215.5 62.7 75.1%
11,995 1,226.9 61.9 75.9%
23,965 1,606.1 66.8 80.4%
47,835 1,930.1 63.0 76.4%
89,696 1,930.0 64.3 79.2%
  • vs single-GPU on the same box: decode 2.0–2.3× (was 27–31 tok/s), prefill 3.0–6.8× (was 90–550); 122k prefill 46 s (was 163 s)
  • output quality token-identical to the single-GPU reference answers; 18/18 bench runs fault-free, no NaN, no IMAs
  • MTP drafting works on the split (acceptance 73–80%, +5–15 tok/s vs the suffix drafter) — after one mandatory config change:
EMBED_VRAM=0   # embedding table must NOT sit only in card-0 VRAM:
               # the MTP drafter on card 2 is its only cross-device consumer
               # → IMA in mtp.cpp gather_dev at the first request otherwise

Key production config (30 GiB host):

STRATA_EXTRA=--layer-split 24 --vram-reserve-mib 768 --vram-reserve-later-mib 1425
STRATA_STAGE_TRIM=1
MTP=1
EMBED_VRAM=0            # mandatory, see above
LATENT_FUSED=1          # fused latent kernels are correct after the patches
RESIDENT_BUDGET_GIB=16  # host-RAM expert complement (20 OOM-killed on a 30 GiB host under real load)
KV_RESIDENT=32768
PLE_IO=direct
SPEC=4
# start serve/server.py directly with a config WITHOUT a "gpu" key
# (the edge0-infer wrapper hard-sets CUDA_VISIBLE_DEVICES=0 → second card invisible)

What made it work — the patch bundle ported from issue #205 (applies cleanly on f833336e):

  • per-device latent D/U instead of a process-global pointer (root cause of the deterministic verify: layer N never rang (illegal memory access) crash on every unpatched split attempt)
  • thread-local hand-off slots + per-stage D/U trim (each card's dense weights only cover its own layers)
  • cuBLAS warm-up on CUDA (fixes the cuBLAS status 14 class)

Dead ends on these NVFP4 latent-MoE packs (documented with repro steps in the repo): the helper-tier route --expert-cache-device1 and the peer route produce divergent output (finite but wrong logits) on this build — don't use them.

Everything is written up with full instructions, benchmark matrix, debug logs and raw result JSONs in a public repo:

GitHub: accelerando-developers/edge0-nvfp4-multi-gpu — includes a reproducible Docker build (Dockerfile, compose, entrypoint, .env.example).

The 145B sibling runs on the exact same stack unchanged (model swap via .env): 38.9 tok/s decode @ 122k on this setup, ~80% expert coverage from 32 GiB combined memory → worst-case prefill ~491 tok/s (NVMe-bound on 30 GiB RAM); see its model card thread: Edge0-145B-NVFP4.

Model profiles on Open LLM Radar, if you want quick specs:

Happy to share any log details if you hit something similar — the NaN/diagnosis ladder is written up step by step in the repo.

Sign up or log in to comment