Instructions to use AtomicChat/Hy4-flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AtomicChat/Hy4-flash-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS # Run inference directly in the terminal: llama cli -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS # Run inference directly in the terminal: llama cli -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS # Run inference directly in the terminal: ./build/bin/llama cli -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Use Docker
docker model run hf.co/AtomicChat/Hy4-flash-GGUF:IQ2_XXS
- Llama new
- LM Studio
- Jan
- vLLM
How to use AtomicChat/Hy4-flash-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AtomicChat/Hy4-flash-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtomicChat/Hy4-flash-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/AtomicChat/Hy4-flash-GGUF:IQ2_XXS
- Ollama
How to use AtomicChat/Hy4-flash-GGUF with Ollama:
ollama run hf.co/AtomicChat/Hy4-flash-GGUF:IQ2_XXS
- Unsloth Desktop
- Pi
How to use AtomicChat/Hy4-flash-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AtomicChat/Hy4-flash-GGUF:IQ2_XXS" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AtomicChat/Hy4-flash-GGUF with Docker Model Runner:
docker model run hf.co/AtomicChat/Hy4-flash-GGUF:IQ2_XXS
- Lemonade
How to use AtomicChat/Hy4-flash-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Run and chat with the model
lemonade run user.Hy4-flash-GGUF-IQ2_XXS
List all available models
lemonade list
- Hermes Agent
How to use AtomicChat/Hy4-flash-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AtomicChat/Hy4-flash-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Hy4-flash-GGUF:IQ2_XXS
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AtomicChat/Hy4-flash-GGUF:IQ2_XXS" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Hy4-flash for 64 GB machines
Tencent's 220B Hy4-flash with 49 GB of weights in memory: the experts at ~2.3 bits, everything else at Q8_0, the Engram n-gram table in a file of its own that stays on the SSD.
- Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
- Needs llama.cpp with Hy4-flash support: the open PR ggml-org/llama.cpp#32517, tested at
0a054d0b. Stock llama.cpp, LM Studio and Ollama do not load these files yet.
Hy4-flash is Tencent's Mixture-of-Experts model with 220B parameters: 155.5B in the network itself (512 routed experts, 16 active per token, 11.3B active parameters in all) and 63.5B in Engram, a table of learned bigram and trigram embeddings looked up by hash. It reads text and images and has a 1M-token context.
The Engram table is read row by row, a few kilobytes per token, so it can stay on disk. What has to fit in memory is the rest. Tencent's own GGUF builds (AngelSlim/Hy4-flash-GGUF) keep 72–105 GB of it in memory and are the better choice on 96 GB and 128 GB machines. This repo has one file built for 64 GB: 49.1 GB of weights, the table in a second file.
Files
| file | what is in it | GB on disk |
|---|---|---|
Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf |
everything but the Engram table: routed experts with down projections at IQ2_S and gate/up at IQ2_XXS, everything else at Q8_0 (2.52 bits per weight) | 49.05 |
Hy4-flash-AD-IQ2_S-IQ2_XXS-00002-of-00002.gguf |
the Engram table, Q4_0 (660M rows of 96) | 35.64 |
mmproj-Hy4-flash-F16.gguf |
the vision projector, for image input | 0.97 |
Point llama.cpp at the first part; it loads the second by itself. With the vision projector and a 32K context, llama-server held 51.4 GB of GPU memory on our CUDA box. We have not measured it on a Mac.
Quality
Measured against our BF16 conversion of the original weights with llama-perplexity on our 30-language neutral corpus (104 chunks of 4096 tokens), llama.cpp PR #32517 at 0a054d0b, 8x A100 80 GB. "Same top token" is how often the file picks the same next token as BF16; "mean KLD" is the KL divergence of its next-token distribution from BF16's.
| file | in memory | GB on disk | mean KLD | 99th pct KLD | same top token |
|---|---|---|---|---|---|
| Q8_0 (ours, not published) | 165.5 | 232.8 | 0.136 | 2.92 | 91.8% |
Tencent Q4_K_M |
105.0 | 168.4 | 0.198 | 4.18 | 89.3% |
Tencent IQ2-AH |
71.6 | 135.0 | 0.651 | 8.73 | 77.9% |
AD-IQ2_S-IQ2_XXS (this repo) |
49.1 | 84.7 | 0.725 | 9.08 | 76.4% |
"In memory" is everything but the Engram table. Tencent's files also carry a multi-token-prediction block from their internal export, which the public checkpoint does not have.
This model loses more to quantization than most. Even Q8_0 picks a different top token than BF16 on 8.2% of tokens, where most models stay above 98% agreement. The losses are concentrated: Tencent's Q4_K_M has a median KLD of 0.008, so most tokens come out nearly exact and a minority diverge a lot. Our reading is that small rounding differences change which 16 of the 512 experts the router picks. Read the table as a ranking.
On that scale, this file needs 22.5 GB less memory than Tencent's smallest file and gives up 1.4 points of top-token agreement. Tencent reports its IQ2-AH between 2 and 6.5 points below BF16 on its own agent and coding benchmarks; we have not run task benchmarks on this file.
Treat it as an experimental build: it brings the model to 64 GB machines, with the losses above.
What we checked by hand
llama-server --jinja with this file and the vision projector, temperature 0 unless noted:
- English: short factual answers and arithmetic were correct.
- Reasoning on (the default): a reasoning puzzle, answered correctly.
- Chinese: a two-sentence answer in Chinese.
- Code: a correct Python function.
- Images: a test image described correctly: shapes, colours, their positions, and the text on it.
- Tool calls: with reasoning on, the model called the provided tool. With
reasoning_effortset tono_thinkit answered in prose instead of calling the tool. Our Q8_0 called it in both modes. For agents, keep reasoning on.
Speed on 4x A100 80 GB (layers split across the cards, the Engram table on the host): about 23 to 26 tokens/s of generation. This is not a Mac number; Tencent reports 48 to 50 tokens/s for its Q4_K_M and Q3_K on an M5 Max 128 GB.
How to run
Build llama.cpp from the PR:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/32517/head:hy4-flash && git checkout 0a054d0b
cmake -B build && cmake --build build --config Release -j
Serve it, with the Engram table kept on the host side:
./build/bin/llama-server -m Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf --mmproj mmproj-Hy4-flash-F16.gguf \
-ngl 99 -ot 'per_layer_token_embd\.weight=CPU' -c 32768 --jinja --temp 0.9 --top-p 1.0
Reasoning is on by default; for direct answers add --chat-template-kwargs '{"reasoning_effort":"no_think"}'. Temperature 0.9 and top-p 1.0 are Tencent's recommended sampling.
On a 64 GB Mac, let the GPU use more of the memory before starting: sudo sysctl iogpu.wired_limit_mb=57344 (resets on reboot).
The PR is not merged yet. If its format changes before the merge, we will update these files.
How it was made
- Conversion. From the BF16 safetensors of
tencent/Hy4-flash(revisiond88c9dfd) with the PR's converter. The public checkpoint names a few things differently from what that converter expects: the architecture names, the sparse-attention layer type, the Engram token map shipped as a tensor, and noengram_enableflag. Without changes the converter refuses the checkpoint, or drops the Engram layers if only the names are fixed.conversion/prep.pyandconversion/hy4-flash-public-ckpt.patchin this repo make the mapping. - Check against Tencent's GGUF. The Engram token map and hash parameters are identical, and all 1,171 tensors that Tencent's
IQ2-AHkeeps unquantized match ours byte for byte. - Importance matrix. Computed on the BF16 weights over 1,114 chunks of 512 tokens (about 570k tokens). That is an even eighth of our 4.56M-token calibration build, rendered with Hy4-flash's chat template; the long-context part was not used. Every routed expert received data: at least 69 tokens, median about 11,000.
- Layout.
- Routed experts: gate/up at IQ2_XXS, down at IQ2_S.
- At Q8_0: attention, the shared expert, the latent projections around the experts, the sparse-attention indexer, the Engram projections, the embeddings and the output head.
- At F32: the hyper-connection matrices.
- The Engram table at Q4_0.
- Table precision barely matters here. With the same weights and the table at Q4_0, F8_E4M3 or Q8_0, mean KLD is 0.7255, 0.7203 and 0.7204.
- Other 64 GB layouts we measured:
- down at IQ2_XS (47.5 GB in memory): 0.754 mean KLD, 75.7% same top token;
- gate/up at IQ1_M with down at IQ3_XXS (48.3 GB): 0.919 and 72.9%.
- Not included. The DFly2 speculative drafter, which llama.cpp does not support.
All logs, the importance matrix and the hand checks are in AtomicChat/Hy4-flash-GGUF-metrics.
License and credits
Hy4-flash is made by Tencent and released under the Tencent Hy License (LICENSE in this repo). It is Apache 2.0 with additional commercial terms: a separate license from Tencent is required for model-as-a-service businesses above US$50M annual revenue or for products above 100M monthly active users. These GGUF files are derived from it and carry the same license. Quantized by AtomicChat.
- Downloads last month
- -
2-bit
Model tree for AtomicChat/Hy4-flash-GGUF
Base model
tencent/Hy4-flash


