Instructions to use nace-ai/Hy4-flash-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use nace-ai/Hy4-flash-MLX-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("nace-ai/Hy4-flash-MLX-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use nace-ai/Hy4-flash-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nace-ai/Hy4-flash-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "nace-ai/Hy4-flash-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use nace-ai/Hy4-flash-MLX-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "nace-ai/Hy4-flash-MLX-4bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "nace-ai/Hy4-flash-MLX-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nace-ai/Hy4-flash-MLX-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use nace-ai/Hy4-flash-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nace-ai/Hy4-flash-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default nace-ai/Hy4-flash-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use nace-ai/Hy4-flash-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nace-ai/Hy4-flash-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "nace-ai/Hy4-flash-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
We introduce Hy4 flash MLX 4-bit from Nace.AI: Tencent's Hy4-flash for mlx-lm, with its routed experts in 4 bits.
Overview
Hy4 flash MLX 4-bit is a Nace general-purpose quant of Hy4-flash. Each class of weights gets its own precision, and every score is measured against the native model.
- 131.20 GB of complete text-model weight files, 4.8 bits per weight
Results
37 measured Hy4-flash variants on the same 11 texts, 5,621 scored tokens. Chart sizes use text-weight payload, including required Engram tables, excluding vision/MTP. Complete weight-file sizes and exact scores are in the comparison data.
Scores use the same 11 comparison texts, 511 next-token positions each, KL against the native model.
- 131.20 GB of complete text-model weight files: 0.113 KL to native Hy4-flash.
- 90.5% top-1 agreement: the most likely next token matches the native model's.
- Lower KL than Xylolabs MLX 4-bit: 0.113 against 0.116. The complete weight files are 131.20 GB and 160.18 GB, respectively.
KL divergence measures how far the next-token probabilities are from native Hy4-flash. Lower is closer. The 11 texts cover prose, code, math, multilingual and instruction writing. Exact scores are in the comparison data above.
Nace Weight Space Quantization
Precision goes where Hy4-flash needs it.
Nace Weight Space Quantization sets the precision of each class of weights on its own. The routed experts are recompressed to 4 bits, and the Engram n-gram tables are stored in 4 bits with finer groups. Attention, shared experts, embeddings and the output head are stored in 8 bits. The router, norms and other small sensitive weights keep their source precision.
Run it
mlx-lm 0.32.2 or newer runs the model with trust_remote_code, which loads hy_v4_flash.py from this repo. We ran it with stock mlx-lm 0.32.2 and mlx 0.32.4 on an NVIDIA B200, and ran the same model file on an M5 Pro with the weights memory-mapped from the SSD.
pip install "mlx-lm>=0.32.2"
hf download nace-ai/Hy4-flash-MLX-4bit --local-dir Hy4-flash-MLX-4bit
python Hy4-flash-MLX-4bit/run_mlx.py Hy4-flash-MLX-4bit "Name three primary colors." 128
python Hy4-flash-MLX-4bit/run_mlx.py Hy4-flash-MLX-4bit "Name three primary colors." 128 --dfly
--dfly drafts tokens with the DFly2 drafter in this repo. On a B200 it decoded 2.1x faster than plain greedy decoding on average, and 3.4x faster on code. Its output can differ from plain greedy decoding where two tokens are nearly tied. The vision tower also ships in the repo, but this model file does not run it.
License
Tencent Hy License, Version 1.0, including its Section 10 commercial terms for large Model-as-a-Service operators. See also MODIFICATIONS.md.
- Downloads last month
- 10
4-bit
Model tree for nace-ai/Hy4-flash-MLX-4bit
Base model
tencent/Hy4-flash