Instructions to use Edge0/Edge0-145B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Edge0/Edge0-145B-NVFP4 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Llama new
- LM Studio
- Jan
- vLLM
How to use Edge0/Edge0-145B-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Edge0/Edge0-145B-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Edge0/Edge0-145B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Ollama
How to use Edge0/Edge0-145B-NVFP4 with Ollama:
ollama run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Unsloth Desktop
- Pi
How to use Edge0/Edge0-145B-NVFP4 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Edge0/Edge0-145B-NVFP4:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Edge0/Edge0-145B-NVFP4 with Docker Model Runner:
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Lemonade
How to use Edge0/Edge0-145B-NVFP4 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Edge0/Edge0-145B-NVFP4:BF16
Run and chat with the model
lemonade run user.Edge0-145B-NVFP4-BF16
List all available models
lemonade list
- Hermes Agent
How to use Edge0/Edge0-145B-NVFP4 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Edge0/Edge0-145B-NVFP4:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Edge0/Edge0-145B-NVFP4 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Edge0/Edge0-145B-NVFP4:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Multi-GPU tested: Edge0-145B-NVFP4 across 2× RTX 5060 Ti 16 GB (38.9 tok/s decode @ 122k; prefill is RAM/NVMe-bound)
Tested and fully measured Nov 15, 2026 on: Ryzen 5 7500F (6C/12T), 30 GiB DDR5, 2 TB NVMe, 2× RTX 5060 Ti 16 GB (sm_120, PCIe 5.0 x8, PHB topology, no P2P). Engine: edge0-strata @ f833336e + the layer-split patch set from Edge0-AI/Edge0#205. Layer split 24 (CUDA0: L0–23, CUDA1: L24–47), MTP drafter on card 2.
TL;DR — the 145B boots and runs cleanly on the exact same dual-GPU stack as the 120B (= model-agnostic image, only .env changes): 18/18 bench runs fault-free, zero OOM under the heaviest stress mix, quality probes green.
| prompt tokens | prefill tok/s | decode tok/s | draft acceptance |
|---|---|---|---|
| 443 | 57.0 | 32.4 | 74.7% |
| 3,034 | 163.2 | 37.1 | 68.8% |
| 11,995 | 258.4 | 37.8 | 71.0% |
| 23,965 | 371.2 | 40.1 | 74.6% |
| 47,835 | 486.7 | 38.9 | 69.1% |
| 89,696 | 490.9 | 38.9 | 71.6% |
(medians of 3, temp 0, cached_tokens=0 verified; 122k-target prefill: 183 s)
Decode target met: 38.9 tok/s median @ 122k, MTP acceptance 68–75% — with one mandatory config flip vs the 145B defaults:
EMBED_VRAM=0 # the 145B model.json defaults to 1, which keeps the
# native embedding table in card-0 VRAM — the MTP drafter
# on card 2 is its only cross-device consumer and faults
# (IMA in mtp.cpp gather_dev). Mapped host memory = clean.
Rest of the production config (identical to the twin-GPU 120B stack): --layer-split 24, --vram-reserve-mib 768 --vram-reserve-later-mib 1425, STRATA_STAGE_TRIM=1, fused latent kernels on, RESIDENT_BUDGET_GIB=16, KV_RESIDENT=32768, PLE_IO=direct, cgroup mem_limit: 24g. Boot: ready in ~50 s.
Finding worth knowing before you size hardware (prefill): at 43.7% + 16 GiB RAM the expert pool covers ~80% of the 24,576 expert pairs (10,727 in VRAM + 8,876 pinned). The remaining 9 GiB live on the NVMe file tier and bench filler prompts route uniformly → the misses amplify to **251 GB of SSD reads per 122k request** (engine-reported), making worst-case prefill NVMe-bound (~491 tok/s) instead of compute-bound. On higher-RAM machines this is where the 145B's prefill should improve disproportionately; decode in the 35–40 tok/s corridor is exactly what a 30 GiB host can expect.
Real-workload note: agent-style prompts route profile-like rather than uniformly (our 120B real operation showed 100% expert-cache hits), so real prefill sits between the bench worst case and the 120B's numbers.
Speed vs quality on this specific box: the Edge0-120B-NVFP4 twin-GPU config prefills ~3.5–4× faster (1,930 vs 490 tok/s at 122k) while the 145B is the qualitative step up (+3 terminal-bench / +6 deepswe per the model card) — a usage decision; both are one .env swap apart (never run both simultaneously on this hardware class).
Switch between them:
# 145B -> MODEL_DIR=/models/Edge0-145B-NVFP4, ALIAS=edge0_145b
# 120B -> MODEL_DIR= (empty = pack default), ALIAS=edge0_120b, DU_FILE= / DU_FMT= empty
Full write-up — instructions, boot capacity table, benchmark matrix, go/no-go analysis, and raw result JSONs — in a public repo:
GitHub: accelerando-developers/edge0-nvfp4-multi-gpu — includes the issue-#205 patch bundle, a reproducible Docker build, and the 120B twin-GPU thread on the model page. Both model profiles are also on Open LLM Radar:
Happy to share logs/details on request.
Great report, and your diagnosis is the bug we found the same day: the embedding table lived in card-0 VRAM and the MTP drafter on card 2 was its only cross-device consumer. Fixed and merged (main, #234): a split keeps that table in mapped host memory, so EMBED_VRAM=0 is no longer needed — and with #231, the #205 patch set is not needed either.
Re-run your bench on current main with the defaults; that is the cleanest sm_120 confirmation without P2P.
Your prefill finding (~80% of pairs in the pool, the rest on NVMe → ~251 GB of reads per 122k request) is the honest sizing guidance for a 30 GiB host.