Hy4-flash for Basalt
Website: basalt.jgulfo.com
Model file for Basalt, a local inference engine for NVIDIA Blackwell GPUs (sm_120) with an OpenAI- and Anthropic-compatible API. Basalt runs this model on one Blackwell card, optionally a second card as a peer expert tier, and system RAM for the experts that do not fit in VRAM.
One self-contained file: weights, the Engram table, the vision encoder, the tokenizer and the DFly2 drafter, built from Tencent's BF16 checkpoint (tencent/Hy4-flash). Needs Basalt 0.5.0 or later.
| file | size | experts |
|---|---|---|
Hy4-flash-NVFP4.basalt |
159 GB | NVFP4, rounded from BF16 |
- Vision: images work with
--vision dynamic, encoded inside the engine from the file's SigLIP2 tower. - Context: up to 1,048,576 tokens (
--max-context). The cache grows on demand from the expert cache.--kv fp8halves it (8.4 GiB at 1M instead of 14.3 GiB); the default is FP16. - Drafter: DFly2 drafts by default. Drafted tokens are verified, so the output is the model's own.
--spec 0turns it off.
What is in the file
Every part is rounded once from Tencent's BF16 weights; nothing is calibrated on data.
| part | format | size |
|---|---|---|
| Routed experts (41 MoE layers × 512, gate/up/down) | NVFP4: E2M1 codes, a UE4M3 scale per 16 values picked for least squared error, an FP32 scale per projection | 83.6 GB |
| Engram table (4 layers, 660M rows of 96) | E4M3, one rounding of the BF16 values | 63.4 GB |
| Attention (MLA q/kv projections, output), latent MoE projections, shared expert, layer 0's dense MLP, indexer and Engram projections, token embedding | Q8_0 | 8.3 GB |
| Output head | BF16 | 0.74 GB |
| Router, norms, hyper-connections, biases | F32 | small |
| Vision encoder (SigLIP2) and projector | BF16 | 1.0 GB |
| DFly2 drafter | BF16 (the engine rounds its projections to Q8_0 at load) | 1.7 GB |
Speed
Single stream, RTX 5090 at 400 W + RTX 5060 Ti at 180 W as a peer, 121 GiB of RAM, tokens/s. Numbers move with every release.
| prose | structured | chat | prefill, 14K prompt |
|---|---|---|---|
| 84 | 172 | 92 | 3,454 |
DFly2 drafting on (the default). Long prompts read at about 3,500 tok/s at 128K and 2,090 tok/s at 1M; decode after a 116K prompt runs at about 70 tok/s.
Quality
Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path
against a llama.cpp BF16 reference of the original model (reference PPL 18.572 on the scored rows; the reference is in
basalt-kld-bases).
| pack | PPL | KLD mean | same top-1 |
|---|---|---|---|
| NVFP4 | 17.55 | 0.201 | 86.09% |
| Q4_K_M (AngelSlim's GGUF, for comparison; not shipped) | 19.82 | 0.176 | 86.96% |
Both measured the same way on the same machine.
Needle-in-a-haystack retrieval passes at 256K (10%, 50% and 90% depth, FP16 and FP8 cache) and at 1M (50% depth, FP8 cache).
Run
Build the engine as described in the Basalt README, then:
hf download jesdga/Hy4-flash-Basalt Hy4-flash-NVFP4.basalt --local-dir models
bin/basalt serve models/Hy4-flash-NVFP4.basalt --vision dynamic
Add --gpu 0,1 to use a second card and --max-context N for a longer context. The engine reads the model's chat
format, tool-call syntax and thinking switch from the file. More in
Hy4-flash on Basalt and
models and the router.
Sources
- Model: tencent/Hy4-flash, Tencent.
- This pack: built from the BF16 checkpoint; what each part is stored as is below. Tokenizer and chat template from the original release.
- Basalt is a fork of Strata (MIT).
License
The model is released by Tencent under the Tencent Hy License, Version 1.0, included as LICENSE. This file is a modified (quantized and repacked) form of the original weights. This is not an official Tencent release. The Basalt engine is MIT.
Model tree for jesdga/Hy4-flash-Basalt
Base model
tencent/Hy4-flash