We introduce Hy4 flash MLX 4-bit from Nace.AI: Tencent's Hy4-flash for mlx-lm, with its routed experts in 4 bits.

Overview

Hy4 flash MLX 4-bit is a Nace general-purpose quant of Hy4-flash. Each class of weights gets its own precision, and every score is measured against the native model.

  • 131.20 GB of complete text-model weight files, 4.8 bits per weight

Results

Hy4-flash quantization quality across 37 measured variants

37 measured Hy4-flash variants on the same 11 texts, 5,621 scored tokens. Chart sizes use text-weight payload, including required Engram tables, excluding vision/MTP. Complete weight-file sizes and exact scores are in the comparison data.

Scores use the same 11 comparison texts, 511 next-token positions each, KL against the native model.

  • 131.20 GB of complete text-model weight files: 0.113 KL to native Hy4-flash.
  • 90.5% top-1 agreement: the most likely next token matches the native model's.
  • Lower KL than Xylolabs MLX 4-bit: 0.113 against 0.116. The complete weight files are 131.20 GB and 160.18 GB, respectively.

KL divergence measures how far the next-token probabilities are from native Hy4-flash. Lower is closer. The 11 texts cover prose, code, math, multilingual and instruction writing. Exact scores are in the comparison data above.

Nace Weight Space Quantization

Precision goes where Hy4-flash needs it.

Nace Weight Space Quantization sets the precision of each class of weights on its own. The routed experts are recompressed to 4 bits, and the Engram n-gram tables are stored in 4 bits with finer groups. Attention, shared experts, embeddings and the output head are stored in 8 bits. The router, norms and other small sensitive weights keep their source precision.

Run it

mlx-lm 0.32.2 or newer runs the model with trust_remote_code, which loads hy_v4_flash.py from this repo. We ran it with stock mlx-lm 0.32.2 and mlx 0.32.4 on an NVIDIA B200, and ran the same model file on an M5 Pro with the weights memory-mapped from the SSD.

pip install "mlx-lm>=0.32.2"
hf download nace-ai/Hy4-flash-MLX-4bit --local-dir Hy4-flash-MLX-4bit
python Hy4-flash-MLX-4bit/run_mlx.py Hy4-flash-MLX-4bit "Name three primary colors." 128
python Hy4-flash-MLX-4bit/run_mlx.py Hy4-flash-MLX-4bit "Name three primary colors." 128 --dfly

--dfly drafts tokens with the DFly2 drafter in this repo. On a B200 it decoded 2.1x faster than plain greedy decoding on average, and 3.4x faster on code. Its output can differ from plain greedy decoding where two tokens are nearly tied. The vision tower also ships in the repo, but this model file does not run it.

License

Tencent Hy License, Version 1.0, including its Section 10 commercial terms for large Model-as-a-Service operators. See also MODIFICATIONS.md.

Downloads last month
10
Safetensors
Model size
220B params
Tensor type
U32
·
F32
·
BF16
·
I32
·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nace-ai/Hy4-flash-MLX-4bit

Quantized
(40)
this model

Collection including nace-ai/Hy4-flash-MLX-4bit