Instructions to use Avifenesh/Hy4-flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Avifenesh/Hy4-flash-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Avifenesh/Hy4-flash-NVFP4") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("Avifenesh/Hy4-flash-NVFP4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Avifenesh/Hy4-flash-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Avifenesh/Hy4-flash-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Avifenesh/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Avifenesh/Hy4-flash-NVFP4
- SGLang
How to use Avifenesh/Hy4-flash-NVFP4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Avifenesh/Hy4-flash-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Avifenesh/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Avifenesh/Hy4-flash-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Avifenesh/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Avifenesh/Hy4-flash-NVFP4 with Docker Model Runner:
docker model run hf.co/Avifenesh/Hy4-flash-NVFP4
Hy4-flash-NVFP4
Tencent Hy4-flash (220B MoE, vision, 1M context) in two-level NVFP4: 161 GB instead of 440 GB, at 0.66% weight error, which is 1.7x less error than the vendor's 4-bit Spark build. Made for the DGX Spark (GB10), where the vendor's own FP8 release does not fit.
Status: weights only. No inference runtime has been verified to load this checkpoint yet, and no whole-model evaluation has been run. The numbers on this page are weight-reconstruction error against BF16. Support in memra is planned. If you need a model that runs today, see Which build to use.
| Base model | tencent/Hy4-flash, revision d88c9dfd96ba |
| Size | 161.1 GB in 50 safetensors files (BF16 original: 440.5 GB) |
| Experts | NVFP4 weight-only, E4M3 block scale per 16 values + FP32 scale per expert |
| Attention projections, engram tables | FP8 E4M3 |
| Everything else | unchanged BF16 / F32 |
| Built-in DFly2 drafter | quantized with the same recipe per tensor class |
| Calibration | none (data-free) |
| License | Tencent Hy License 1.0, as the base model. This is not an official Tencent release. |
Weight error
Measured on every quantized tensor, not a sample, as rel = sum((W - Wq)^2) / sum(W^2) against the BF16 source. The
per-tensor tables are in quant-stats/, and the method is in QUALITY.md.
| tensors | format | count | mean error | worst |
|---|---|---|---|---|
| routed experts, layers 1-41 | NVFP4, this checkpoint | 62,976 expert matrices | 0.660% | 0.664% |
| same matrices | MXFP4 (E8M0 per 32), measured, not shipped | 62,976 | 1.258% | 1.316% |
vendor SPARK-4Bit-Fast GGUF, NVFP4 |
decoded from the file | 18 sampled expert slices | 1.14% | 1.30% |
DFly2 MLPs + hidden_correction |
NVFP4 | 18 | 0.660% | 0.661% |
q_b_proj, o_proj, linear_gate |
FP8, 128x128 block scales | 138 | 0.070% | 0.070% |
| engram n-gram tables | FP8, one scale per row | 4 | 0.067% | 0.068% |
- Against the vendor GGUF: on the same 18 slices this checkpoint has 0.66%, 1.7x less squared error.
- Against MXFP4: MXFP4 has 1.91x the error of NVFP4 on average, and is worse on every one of the 62,976 matrices.
- No outliers: per-layer means stay between 0.642% and 0.661%.
Where the gain comes from
Hy4-flash's expert weights are small. With no per-tensor scale, 99.8% of the 16-value blocks get a subnormal E4M3 scale and lose precision. That is the case in the GGUF NVFP4 block, which has no per-tensor scale. On 140 expert slices from 7 layers:
| recipe | error |
|---|---|
| absmax block scale, no per-tensor scale | 1.03% |
| MSE search over E4M3 block-scale codes, no per-tensor scale | 0.77% |
| MSE search + per-expert power-of-two FP32 scale (this checkpoint) | 0.66% |
Format
The experts use the two-level ModelOpt NVFP4 layout. Each quantized weight has three tensors:
weight: E2M1 codes, two per byte (U8);weight_scale: one E4M3 scale per 16 input values;weight_scale_2: one FP32 scale per expert (or per tensor).
W[o, i] = e2m1(weight[o, i]) * weight_scale[o, i // 16] * weight_scale_2[e]
- Block scales are chosen by an MSE search over E4M3 codes around the absmax code.
weight_scale_2is a power of two: the smallest lift that leaves at most 0.1% of blocks with a subnormal scale, capped so that no block scale exceeds 448.- No
input_scale: activations are left to the runtime. - Names: tensors keep their source names. Gate and up are separate tensors (
experts.gate_proj,experts.up_proj, each[512, 1536, 1536]); fuse them at load time if your kernel wantsgate_up.
config.json carries a quantization_config block and hf_quant_config.json the same description: quant_algo
MIXED_PRECISION, with W4A16_NVFP4 (group 16) for the NVFP4 modules and FP8_PB_WO (128x128) for the FP8
projections. The config follows the ModelOpt schema; no runtime has been verified to load it. Two details differ from a
ModelOpt export: the block-FP8 scale is named weight_scale_inv (the tencent/Hy4-flash-FP8 name), and the FP8
per-row engram tables have no ModelOpt algo name, so a comment field describes them. The full tensor-by-tensor layout
is in LAYOUT.md; quant_config.json lists every quantized module.
What stays unchanged:
- layer 0 (dense);
q_a_proj,kv_a_proj_with_mqa,kv_b_proj;- the shared experts and the expert latent projections;
- the router and the sparse-attention indexer;
- the engram key/value projections;
- norms, hyper-connection parameters and attention sinks;
embed_tokens,lm_headand the 27-layer vision tower.
The model, briefly
From the base model's config and card:
- 42 layers with hidden size 3072. Layer 0 is dense; layers 1-41 are MoE with 512 routed experts, 16 active per token plus one shared expert, sigmoid routing.
- MLA-style attention, mixing DeepSeek sparse-attention layers (a compressed indexer picks up to 2,048 positions) with 512-token sliding-window layers. Context is 1,048,576 tokens.
- Engram n-gram embedding tables at four layers (63.4B of the parameters).
- A vision encoder.
- DFly2, a built-in 5-layer speculative drafter (0.85B). It drafts a block of tokens in one pass and reads the main model's own KV cache and indexer selections, so it adds no KV memory. Its acceptance with these quantized weights has not been measured.
Which build to use
- To run Hy4-flash today on data-center GPUs: use Tencent's Hy4-flash-FP8 with their vLLM or SGLang images.
- To run it today on a DGX Spark: use AngelSlim's Hy4-flash-GGUF with the vendor's llama.cpp build. It is the only runtime we know of that serves this model on GB10 now.
- Use this checkpoint if you are building or testing an NVFP4 runtime for Hy4-flash, or want the most accurate 4-bit weights to start from.
On GB10 the native NVFP4 tensor-core instruction needs FP4 activations as well (W4A4). Using these weights with BF16 or FP8 activations needs a kernel that dequantizes in registers.
Limits
- No calibration: no GPTQ or AWQ. Those need a forward pass of the 440 GB BF16 model.
- No whole-model evaluation yet: perplexity and task scores were not measured.
- Drafter acceptance not measured.
- No verified loader for this checkpoint yet.
Modification notice
Per Section 4(b) of the Tencent Hy License, these files were changed from the original tencent/Hy4-flash release:
model-experts-L01..L41.safetensors: new files. Each layer's routed-expertgate_up_projanddown_projweights are quantized to NVFP4.gate_up_projis split intogate_projandup_proj.model-nonexpert-01..09.safetensors: new files holding all other tensors.- The tensors listed above as FP8 are quantized to FP8 E4M3 with scales.
- The DFly2 MLP and
hidden_correctionweights are quantized to NVFP4. - All other tensors are copied unchanged.
model.safetensors.index.json: a new index for the new shard layout.config.json: modified. Aquantization_configblock is added; every other field is unchanged.- New files:
README.md,LAYOUT.md,QUALITY.md,hf_quant_config.json,quant_config.json,quant-stats/. - Unchanged copies:
LICENSE,generation_config.json,processor_config.json,tokenizer.json,tokenizer_config.json,chat_template.jinja.
The safetensors metadata of every shard carries "note": "modified, see README".
License
Released under the Tencent Hy License, Version 1.0, the same license as the base model, with a copy in
LICENSE. Section 10 (additional commercial terms for large Model-as-a-Service businesses) applies to this derivative
as it does to the original.
Hy4 flash is developed by the Tencent Hy Team; see the base model card for the model itself.
- Downloads last month
- -
Model tree for Avifenesh/Hy4-flash-NVFP4
Base model
tencent/Hy4-flash