EXL3 quants of Hy4-flash
โ ๏ธ Requires ExLlamaV3 dev at 627c8710 with the Hy4-flash patch in exllamav3-hy4/. Text only.
SC quants:
Uniform quants:
No AIME-25 / GPQA-D table yet: these need long generations, and the patch caches at most 2,048 keys on the indexed attention layers.
Details
| quant | recipe | expert bpw | trunk/head K | size GB | size GB, no Engram | KL | top-1 | ppl | KL -gp |
ppl -gp |
predicted KL |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2.25 bpw | SC | 2.10 | 6 | 114.8 | 48.8 | 0.8418 | 0.7027 | 48.6734 | 0.5041 | 12.1408 | 0.461 |
| 3.08 bpw | uniform | 3.00 | 5 | 130.8 | 64.8 | 0.4335 | 0.7917 | 39.8938 | 0.1504 | 9.3330 | 0.491 |
| source | 0 | 1 | 37.5530 | 0 | 8.8528 |
- The bpw in the branch name counts every quantized tensor (routed experts, trunk and head) and leaves out the Engram tables. Sizes include the Engram file (66.0 GB) and the copied vision tower and dfly draft layers.
- KL, top-1 agreement and perplexity:
eval/model_diff.py -ma <source> -mb <quant>at its defaults (wiki2, 100 rows of 2,048 tokens), and again with-gp(chat-template generation prompt on every row). The charts use the default run. - Predicted KL: sc_optimize's own per-tensor table (
-t) from the same measurement, summed over the final recipe. - This model is sensitive to small numerical changes: the bf16 transformers reference differs from an fp32 run by KL 0.40 and 0.57 on prompts of 1,836 and 3,236 tokens. The source's own wiki2 perplexity is 37.5530.
eval/ppl.py -m <quant>on 2.25 bpw: perplexity 48.673418 (100 rows of 2,048 tokens, wiki2).eval/ppl.py -m <quant>on 3.08 bpw: perplexity 39.893814 (100 rows of 2,048 tokens, wiki2).
Method
- SC quants, exllamav3 self-calibration:
sc_trace.pywithtencent/hy4-flashon OpenRouter as the teacher (Tencent provider, fp8), 250 rows of up to 2,048 tokens;sc_measure.py --streaming --shapedon the whole model;sc_optimize.py -b <b> -t <table>at its defaults, which sets one K per layer for that layer's 512 routed experts. - Trunk rule: every tensor that is not a routed expert, and the head, get K = highest expert K + 2.
- Uniform quants (3.08 bpw): every routed expert at one K, trunk and head at that K + 2 (3.08: K3 / K5). No sc_optimize allocation; same trace and convert flags.
- All quants are converted against the same calibration trace:
convert.py -rcp <recipe> -cd <trace> -cb mcg -mae 0.01. - The Engram tables (4 layers, 63.4B parameters) are stored once as fp8 rows with one fp32 scale per row
(
ngram_embedding.safetensors). The model itself rounds every row it reads through fp8 in the same way, so the file matches the reference value for value. Every branch carries the same file. exllamav3'sutil/convert_ngram.py(trellis tables) does not support this layout.
Routed-expert K per layer (layer 0 is dense)
- 2.25 bpw: 1 K1, 2-15 K2, 16 K3, 17 K2, 18-20 K3, 21-23 K2, 24 K3, 25-26 K2, 27 K3, 28 K4, 29 K3, 30-31 K2, 32-33 K3, 34 K2, 35 K1, 36 K2, 37-41 K1.
- 3.08 bpw: 1-41 K3 (uniform).
Loader limits
- Stock exllamav3 cannot load these quants; apply the patch (base
dev627c8710ab69bd8a581018489a177c64746b2688). - With a cache, the 12 indexed attention layers support at most 2,048 keys. Without a cache the indexer also runs past 2,048 keys (checked against the reference up to 3,236 tokens).
- The KV cache keeps the expanded per-head keys and values, so it uses more memory than the reference.
- The port was checked against the transformers reference (PR #50903) on all 42 layers: on every layer it is closer to an fp32 run than the bf16 reference is.
Hy4-flash is released under the Tencent Hy License, Version 1.0 (LICENSE), including its Section 10 on large
Model-as-a-Service businesses. These quants are a modified form of the model and carry the same license.
Model tree for OpensourceWTF/Hy4-flash-exl3
Base model
tencent/Hy4-flash
