Hy4-flash for 64 GB machines

Tencent's 220B Hy4-flash with 49 GB of weights in memory: the experts at ~2.3 bits, everything else at Q8_0, the Engram n-gram table in a file of its own that stays on the SSD.

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
  • Needs llama.cpp with Hy4-flash support: the open PR ggml-org/llama.cpp#32517, tested at 0a054d0b. Stock llama.cpp, LM Studio and Ollama do not load these files yet.

Hy4-flash is Tencent's Mixture-of-Experts model with 220B parameters: 155.5B in the network itself (512 routed experts, 16 active per token, 11.3B active parameters in all) and 63.5B in Engram, a table of learned bigram and trigram embeddings looked up by hash. It reads text and images and has a 1M-token context.

The Engram table is read row by row, a few kilobytes per token, so it can stay on disk. What has to fit in memory is the rest. Tencent's own GGUF builds (AngelSlim/Hy4-flash-GGUF) keep 72–105 GB of it in memory and are the better choice on 96 GB and 128 GB machines. This repo has one file built for 64 GB: 49.1 GB of weights, the table in a second file.

Files

file what is in it GB on disk
Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf everything but the Engram table: routed experts with down projections at IQ2_S and gate/up at IQ2_XXS, everything else at Q8_0 (2.52 bits per weight) 49.05
Hy4-flash-AD-IQ2_S-IQ2_XXS-00002-of-00002.gguf the Engram table, Q4_0 (660M rows of 96) 35.64
mmproj-Hy4-flash-F16.gguf the vision projector, for image input 0.97

Point llama.cpp at the first part; it loads the second by itself. With the vision projector and a 32K context, llama-server held 51.4 GB of GPU memory on our CUDA box. We have not measured it on a Mac.

Quality

Measured against our BF16 conversion of the original weights with llama-perplexity on our 30-language neutral corpus (104 chunks of 4096 tokens), llama.cpp PR #32517 at 0a054d0b, 8x A100 80 GB. "Same top token" is how often the file picks the same next token as BF16; "mean KLD" is the KL divergence of its next-token distribution from BF16's.

file in memory GB on disk mean KLD 99th pct KLD same top token
Q8_0 (ours, not published) 165.5 232.8 0.136 2.92 91.8%
Tencent Q4_K_M 105.0 168.4 0.198 4.18 89.3%
Tencent IQ2-AH 71.6 135.0 0.651 8.73 77.9%
AD-IQ2_S-IQ2_XXS (this repo) 49.1 84.7 0.725 9.08 76.4%

"In memory" is everything but the Engram table. Tencent's files also carry a multi-token-prediction block from their internal export, which the public checkpoint does not have.

This model loses more to quantization than most. Even Q8_0 picks a different top token than BF16 on 8.2% of tokens, where most models stay above 98% agreement. The losses are concentrated: Tencent's Q4_K_M has a median KLD of 0.008, so most tokens come out nearly exact and a minority diverge a lot. Our reading is that small rounding differences change which 16 of the 512 experts the router picks. Read the table as a ranking.

On that scale, this file needs 22.5 GB less memory than Tencent's smallest file and gives up 1.4 points of top-token agreement. Tencent reports its IQ2-AH between 2 and 6.5 points below BF16 on its own agent and coding benchmarks; we have not run task benchmarks on this file.

Treat it as an experimental build: it brings the model to 64 GB machines, with the losses above.

What we checked by hand

llama-server --jinja with this file and the vision projector, temperature 0 unless noted:

  • English: short factual answers and arithmetic were correct.
  • Reasoning on (the default): a reasoning puzzle, answered correctly.
  • Chinese: a two-sentence answer in Chinese.
  • Code: a correct Python function.
  • Images: a test image described correctly: shapes, colours, their positions, and the text on it.
  • Tool calls: with reasoning on, the model called the provided tool. With reasoning_effort set to no_think it answered in prose instead of calling the tool. Our Q8_0 called it in both modes. For agents, keep reasoning on.

Speed on 4x A100 80 GB (layers split across the cards, the Engram table on the host): about 23 to 26 tokens/s of generation. This is not a Mac number; Tencent reports 48 to 50 tokens/s for its Q4_K_M and Q3_K on an M5 Max 128 GB.

How to run

Build llama.cpp from the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/32517/head:hy4-flash && git checkout 0a054d0b
cmake -B build && cmake --build build --config Release -j

Serve it, with the Engram table kept on the host side:

./build/bin/llama-server -m Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf --mmproj mmproj-Hy4-flash-F16.gguf \
  -ngl 99 -ot 'per_layer_token_embd\.weight=CPU' -c 32768 --jinja --temp 0.9 --top-p 1.0

Reasoning is on by default; for direct answers add --chat-template-kwargs '{"reasoning_effort":"no_think"}'. Temperature 0.9 and top-p 1.0 are Tencent's recommended sampling.

On a 64 GB Mac, let the GPU use more of the memory before starting: sudo sysctl iogpu.wired_limit_mb=57344 (resets on reboot).

The PR is not merged yet. If its format changes before the merge, we will update these files.

How it was made

  • Conversion. From the BF16 safetensors of tencent/Hy4-flash (revision d88c9dfd) with the PR's converter. The public checkpoint names a few things differently from what that converter expects: the architecture names, the sparse-attention layer type, the Engram token map shipped as a tensor, and no engram_enable flag. Without changes the converter refuses the checkpoint, or drops the Engram layers if only the names are fixed. conversion/prep.py and conversion/hy4-flash-public-ckpt.patch in this repo make the mapping.
  • Check against Tencent's GGUF. The Engram token map and hash parameters are identical, and all 1,171 tensors that Tencent's IQ2-AH keeps unquantized match ours byte for byte.
  • Importance matrix. Computed on the BF16 weights over 1,114 chunks of 512 tokens (about 570k tokens). That is an even eighth of our 4.56M-token calibration build, rendered with Hy4-flash's chat template; the long-context part was not used. Every routed expert received data: at least 69 tokens, median about 11,000.
  • Layout.
    • Routed experts: gate/up at IQ2_XXS, down at IQ2_S.
    • At Q8_0: attention, the shared expert, the latent projections around the experts, the sparse-attention indexer, the Engram projections, the embeddings and the output head.
    • At F32: the hyper-connection matrices.
    • The Engram table at Q4_0.
  • Table precision barely matters here. With the same weights and the table at Q4_0, F8_E4M3 or Q8_0, mean KLD is 0.7255, 0.7203 and 0.7204.
  • Other 64 GB layouts we measured:
    • down at IQ2_XS (47.5 GB in memory): 0.754 mean KLD, 75.7% same top token;
    • gate/up at IQ1_M with down at IQ3_XXS (48.3 GB): 0.919 and 72.9%.
  • Not included. The DFly2 speculative drafter, which llama.cpp does not support.

All logs, the importance matrix and the hand checks are in AtomicChat/Hy4-flash-GGUF-metrics.

License and credits

Hy4-flash is made by Tencent and released under the Tencent Hy License (LICENSE in this repo). It is Apache 2.0 with additional commercial terms: a separate license from Tencent is required for model-as-a-service businesses above US$50M annual revenue or for products above 100M monthly active users. These GGUF files are derived from it and carry the same license. Quantized by AtomicChat.

Downloads last month
-
GGUF
Model size
219B params
Architecture
hy_v4_flash
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AtomicChat/Hy4-flash-GGUF

Quantized
(19)
this model

Collection including AtomicChat/Hy4-flash-GGUF