EXL3 quants of Hy4-flash

โš ๏ธ Requires ExLlamaV3 dev at 627c8710 with the Hy4-flash patch in exllamav3-hy4/. Text only.

SC quants:

2.25 bits per weight

Uniform quants:

3.08 bits per weight

qb_kld qb_ppl_vram

No AIME-25 / GPQA-D table yet: these need long generations, and the patch caches at most 2,048 keys on the indexed attention layers.

Details

quant recipe expert bpw trunk/head K size GB size GB, no Engram KL top-1 ppl KL -gp ppl -gp predicted KL
2.25 bpw SC 2.10 6 114.8 48.8 0.8418 0.7027 48.6734 0.5041 12.1408 0.461
3.08 bpw uniform 3.00 5 130.8 64.8 0.4335 0.7917 39.8938 0.1504 9.3330 0.491
source 0 1 37.5530 0 8.8528
  • The bpw in the branch name counts every quantized tensor (routed experts, trunk and head) and leaves out the Engram tables. Sizes include the Engram file (66.0 GB) and the copied vision tower and dfly draft layers.
  • KL, top-1 agreement and perplexity: eval/model_diff.py -ma <source> -mb <quant> at its defaults (wiki2, 100 rows of 2,048 tokens), and again with -gp (chat-template generation prompt on every row). The charts use the default run.
  • Predicted KL: sc_optimize's own per-tensor table (-t) from the same measurement, summed over the final recipe.
  • This model is sensitive to small numerical changes: the bf16 transformers reference differs from an fp32 run by KL 0.40 and 0.57 on prompts of 1,836 and 3,236 tokens. The source's own wiki2 perplexity is 37.5530.
  • eval/ppl.py -m <quant> on 2.25 bpw: perplexity 48.673418 (100 rows of 2,048 tokens, wiki2).
  • eval/ppl.py -m <quant> on 3.08 bpw: perplexity 39.893814 (100 rows of 2,048 tokens, wiki2).

Method

  • SC quants, exllamav3 self-calibration: sc_trace.py with tencent/hy4-flash on OpenRouter as the teacher (Tencent provider, fp8), 250 rows of up to 2,048 tokens; sc_measure.py --streaming --shaped on the whole model; sc_optimize.py -b <b> -t <table> at its defaults, which sets one K per layer for that layer's 512 routed experts.
  • Trunk rule: every tensor that is not a routed expert, and the head, get K = highest expert K + 2.
  • Uniform quants (3.08 bpw): every routed expert at one K, trunk and head at that K + 2 (3.08: K3 / K5). No sc_optimize allocation; same trace and convert flags.
  • All quants are converted against the same calibration trace: convert.py -rcp <recipe> -cd <trace> -cb mcg -mae 0.01.
  • The Engram tables (4 layers, 63.4B parameters) are stored once as fp8 rows with one fp32 scale per row (ngram_embedding.safetensors). The model itself rounds every row it reads through fp8 in the same way, so the file matches the reference value for value. Every branch carries the same file. exllamav3's util/convert_ngram.py (trellis tables) does not support this layout.

Routed-expert K per layer (layer 0 is dense)

  • 2.25 bpw: 1 K1, 2-15 K2, 16 K3, 17 K2, 18-20 K3, 21-23 K2, 24 K3, 25-26 K2, 27 K3, 28 K4, 29 K3, 30-31 K2, 32-33 K3, 34 K2, 35 K1, 36 K2, 37-41 K1.
  • 3.08 bpw: 1-41 K3 (uniform).

Loader limits

  • Stock exllamav3 cannot load these quants; apply the patch (base dev 627c8710ab69bd8a581018489a177c64746b2688).
  • With a cache, the 12 indexed attention layers support at most 2,048 keys. Without a cache the indexer also runs past 2,048 keys (checked against the reference up to 3,236 tokens).
  • The KV cache keeps the expanded per-head keys and values, so it uses more memory than the reference.
  • The port was checked against the transformers reference (PR #50903) on all 42 layers: on every layer it is closer to an fp32 run than the bf16 reference is.

Hy4-flash is released under the Tencent Hy License, Version 1.0 (LICENSE), including its Section 10 on large Model-as-a-Service businesses. These quants are a modified form of the model and carry the same license.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OpensourceWTF/Hy4-flash-exl3

Quantized
(38)
this model