Instructions to use Edge0/Edge0-145B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Edge0/Edge0-145B-NVFP4 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Llama new
- LM Studio
- Jan
- vLLM
How to use Edge0/Edge0-145B-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Edge0/Edge0-145B-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Edge0/Edge0-145B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Ollama
How to use Edge0/Edge0-145B-NVFP4 with Ollama:
ollama run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Unsloth Desktop
- Pi
How to use Edge0/Edge0-145B-NVFP4 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Edge0/Edge0-145B-NVFP4:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Edge0/Edge0-145B-NVFP4 with Docker Model Runner:
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Lemonade
How to use Edge0/Edge0-145B-NVFP4 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Edge0/Edge0-145B-NVFP4:BF16
Run and chat with the model
lemonade run user.Edge0-145B-NVFP4-BF16
List all available models
lemonade list
- Hermes Agent
How to use Edge0/Edge0-145B-NVFP4 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Edge0/Edge0-145B-NVFP4:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Edge0/Edge0-145B-NVFP4 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Edge0/Edge0-145B-NVFP4:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
epic findings on this model
All non expert parameters are identical to base qwen3.8 flash so they were not trained.
Experts have less params - qwen experts are 2560Γ640 and Edge0 experts are 1792Γ640. To get the output back to 2560-dimensional space, there are two additional up and down 2560Γ1792 projections in each layer and they are shared between experts. Presumably the experts were retrained or post trained which would be required to explain the benchmarks in the readme.
The information above was produced by qwen3.8 flash next but it might still be true
Your reading is correct, and none of it was hidden β the card names Qwen3.8-Flash-Next as base_model.
Yes, the non-expert tensors are the base model's. We did not retrain them. It is not that we could not: changing weights is cheap, a few hundred SFT samples would move them. We left them alone because there was no need to.
What we did is the architecture. We replaced the original large experts with a latent MoE, so the experts run in a smaller latent space and each expert carries far fewer parameters. The D/U projections you spotted are new β they exist only because of that change, and they are trained along with the experts.
That is the point for us: the expert pool is what has to fit in VRAM and RAM, and shrinking it is what makes a model this size usable on low-resource edge devices.
Thank you for your epic findings which help clarify our efforts to bring LLM to consumer desktop.