Hy4-flash-NVFP4

Tencent Hy4-flash (220B MoE, vision, 1M context) in two-level NVFP4: 161 GB instead of 440 GB, at 0.66% weight error, which is 1.7x less error than the vendor's 4-bit Spark build. Made for the DGX Spark (GB10), where the vendor's own FP8 release does not fit.

Status: weights only. No inference runtime has been verified to load this checkpoint yet, and no whole-model evaluation has been run. The numbers on this page are weight-reconstruction error against BF16. Support in memra is planned. If you need a model that runs today, see Which build to use.

Base model tencent/Hy4-flash, revision d88c9dfd96ba
Size 161.1 GB in 50 safetensors files (BF16 original: 440.5 GB)
Experts NVFP4 weight-only, E4M3 block scale per 16 values + FP32 scale per expert
Attention projections, engram tables FP8 E4M3
Everything else unchanged BF16 / F32
Built-in DFly2 drafter quantized with the same recipe per tensor class
Calibration none (data-free)
License Tencent Hy License 1.0, as the base model. This is not an official Tencent release.

Weight error

Measured on every quantized tensor, not a sample, as rel = sum((W - Wq)^2) / sum(W^2) against the BF16 source. The per-tensor tables are in quant-stats/, and the method is in QUALITY.md.

tensors format count mean error worst
routed experts, layers 1-41 NVFP4, this checkpoint 62,976 expert matrices 0.660% 0.664%
same matrices MXFP4 (E8M0 per 32), measured, not shipped 62,976 1.258% 1.316%
vendor SPARK-4Bit-Fast GGUF, NVFP4 decoded from the file 18 sampled expert slices 1.14% 1.30%
DFly2 MLPs + hidden_correction NVFP4 18 0.660% 0.661%
q_b_proj, o_proj, linear_gate FP8, 128x128 block scales 138 0.070% 0.070%
engram n-gram tables FP8, one scale per row 4 0.067% 0.068%
  • Against the vendor GGUF: on the same 18 slices this checkpoint has 0.66%, 1.7x less squared error.
  • Against MXFP4: MXFP4 has 1.91x the error of NVFP4 on average, and is worse on every one of the 62,976 matrices.
  • No outliers: per-layer means stay between 0.642% and 0.661%.

Where the gain comes from

Hy4-flash's expert weights are small. With no per-tensor scale, 99.8% of the 16-value blocks get a subnormal E4M3 scale and lose precision. That is the case in the GGUF NVFP4 block, which has no per-tensor scale. On 140 expert slices from 7 layers:

recipe error
absmax block scale, no per-tensor scale 1.03%
MSE search over E4M3 block-scale codes, no per-tensor scale 0.77%
MSE search + per-expert power-of-two FP32 scale (this checkpoint) 0.66%

Format

The experts use the two-level ModelOpt NVFP4 layout. Each quantized weight has three tensors:

  • weight: E2M1 codes, two per byte (U8);
  • weight_scale: one E4M3 scale per 16 input values;
  • weight_scale_2: one FP32 scale per expert (or per tensor).
W[o, i] = e2m1(weight[o, i]) * weight_scale[o, i // 16] * weight_scale_2[e]
  • Block scales are chosen by an MSE search over E4M3 codes around the absmax code.
  • weight_scale_2 is a power of two: the smallest lift that leaves at most 0.1% of blocks with a subnormal scale, capped so that no block scale exceeds 448.
  • No input_scale: activations are left to the runtime.
  • Names: tensors keep their source names. Gate and up are separate tensors (experts.gate_proj, experts.up_proj, each [512, 1536, 1536]); fuse them at load time if your kernel wants gate_up.

config.json carries a quantization_config block and hf_quant_config.json the same description: quant_algo MIXED_PRECISION, with W4A16_NVFP4 (group 16) for the NVFP4 modules and FP8_PB_WO (128x128) for the FP8 projections. The config follows the ModelOpt schema; no runtime has been verified to load it. Two details differ from a ModelOpt export: the block-FP8 scale is named weight_scale_inv (the tencent/Hy4-flash-FP8 name), and the FP8 per-row engram tables have no ModelOpt algo name, so a comment field describes them. The full tensor-by-tensor layout is in LAYOUT.md; quant_config.json lists every quantized module.

What stays unchanged:

  • layer 0 (dense);
  • q_a_proj, kv_a_proj_with_mqa, kv_b_proj;
  • the shared experts and the expert latent projections;
  • the router and the sparse-attention indexer;
  • the engram key/value projections;
  • norms, hyper-connection parameters and attention sinks;
  • embed_tokens, lm_head and the 27-layer vision tower.

The model, briefly

From the base model's config and card:

  • 42 layers with hidden size 3072. Layer 0 is dense; layers 1-41 are MoE with 512 routed experts, 16 active per token plus one shared expert, sigmoid routing.
  • MLA-style attention, mixing DeepSeek sparse-attention layers (a compressed indexer picks up to 2,048 positions) with 512-token sliding-window layers. Context is 1,048,576 tokens.
  • Engram n-gram embedding tables at four layers (63.4B of the parameters).
  • A vision encoder.
  • DFly2, a built-in 5-layer speculative drafter (0.85B). It drafts a block of tokens in one pass and reads the main model's own KV cache and indexer selections, so it adds no KV memory. Its acceptance with these quantized weights has not been measured.

Which build to use

  • To run Hy4-flash today on data-center GPUs: use Tencent's Hy4-flash-FP8 with their vLLM or SGLang images.
  • To run it today on a DGX Spark: use AngelSlim's Hy4-flash-GGUF with the vendor's llama.cpp build. It is the only runtime we know of that serves this model on GB10 now.
  • Use this checkpoint if you are building or testing an NVFP4 runtime for Hy4-flash, or want the most accurate 4-bit weights to start from.

On GB10 the native NVFP4 tensor-core instruction needs FP4 activations as well (W4A4). Using these weights with BF16 or FP8 activations needs a kernel that dequantizes in registers.

Limits

  • No calibration: no GPTQ or AWQ. Those need a forward pass of the 440 GB BF16 model.
  • No whole-model evaluation yet: perplexity and task scores were not measured.
  • Drafter acceptance not measured.
  • No verified loader for this checkpoint yet.

Modification notice

Per Section 4(b) of the Tencent Hy License, these files were changed from the original tencent/Hy4-flash release:

  • model-experts-L01..L41.safetensors: new files. Each layer's routed-expert gate_up_proj and down_proj weights are quantized to NVFP4. gate_up_proj is split into gate_proj and up_proj.
  • model-nonexpert-01..09.safetensors: new files holding all other tensors.
    • The tensors listed above as FP8 are quantized to FP8 E4M3 with scales.
    • The DFly2 MLP and hidden_correction weights are quantized to NVFP4.
    • All other tensors are copied unchanged.
  • model.safetensors.index.json: a new index for the new shard layout.
  • config.json: modified. A quantization_config block is added; every other field is unchanged.
  • New files: README.md, LAYOUT.md, QUALITY.md, hf_quant_config.json, quant_config.json, quant-stats/.
  • Unchanged copies: LICENSE, generation_config.json, processor_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja.

The safetensors metadata of every shard carries "note": "modified, see README".

License

Released under the Tencent Hy License, Version 1.0, the same license as the base model, with a copy in LICENSE. Section 10 (additional commercial terms for large Model-as-a-Service businesses) applies to this derivative as it does to the original.

Hy4 flash is developed by the Tencent Hy Team; see the base model card for the model itself.

Downloads last month
-
Safetensors
Model size
220B params
Tensor type
BF16
·
U8
·
F8_E4M3
·
F32
·
I32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Avifenesh/Hy4-flash-NVFP4

Quantized
(19)
this model