Basalt

Hy4-flash for Basalt

Website: basalt.jgulfo.com

Model file for Basalt, a local inference engine for NVIDIA Blackwell GPUs (sm_120) with an OpenAI- and Anthropic-compatible API. Basalt runs this model on one Blackwell card, optionally a second card as a peer expert tier, and system RAM for the experts that do not fit in VRAM.

One self-contained file: weights, the Engram table, the vision encoder, the tokenizer and the DFly2 drafter, built from Tencent's BF16 checkpoint (tencent/Hy4-flash). Needs Basalt 0.5.0 or later.

file size experts
Hy4-flash-NVFP4.basalt 159 GB NVFP4, rounded from BF16
  • Vision: images work with --vision dynamic, encoded inside the engine from the file's SigLIP2 tower.
  • Context: up to 1,048,576 tokens (--max-context). The cache grows on demand from the expert cache. --kv fp8 halves it (8.4 GiB at 1M instead of 14.3 GiB); the default is FP16.
  • Drafter: DFly2 drafts by default. Drafted tokens are verified, so the output is the model's own. --spec 0 turns it off.

What is in the file

Every part is rounded once from Tencent's BF16 weights; nothing is calibrated on data.

part format size
Routed experts (41 MoE layers × 512, gate/up/down) NVFP4: E2M1 codes, a UE4M3 scale per 16 values picked for least squared error, an FP32 scale per projection 83.6 GB
Engram table (4 layers, 660M rows of 96) E4M3, one rounding of the BF16 values 63.4 GB
Attention (MLA q/kv projections, output), latent MoE projections, shared expert, layer 0's dense MLP, indexer and Engram projections, token embedding Q8_0 8.3 GB
Output head BF16 0.74 GB
Router, norms, hyper-connections, biases F32 small
Vision encoder (SigLIP2) and projector BF16 1.0 GB
DFly2 drafter BF16 (the engine rounds its projections to Q8_0 at load) 1.7 GB

Speed

Single stream, RTX 5090 at 400 W + RTX 5060 Ti at 180 W as a peer, 121 GiB of RAM, tokens/s. Numbers move with every release.

prose structured chat prefill, 14K prompt
84 172 92 3,454

DFly2 drafting on (the default). Long prompts read at about 3,500 tok/s at 128K and 2,090 tok/s at 1M; decode after a 116K prompt runs at about 70 tok/s.

Quality

Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path against a llama.cpp BF16 reference of the original model (reference PPL 18.572 on the scored rows; the reference is in basalt-kld-bases).

pack PPL KLD mean same top-1
NVFP4 17.55 0.201 86.09%
Q4_K_M (AngelSlim's GGUF, for comparison; not shipped) 19.82 0.176 86.96%

Both measured the same way on the same machine.

Needle-in-a-haystack retrieval passes at 256K (10%, 50% and 90% depth, FP16 and FP8 cache) and at 1M (50% depth, FP8 cache).

Run

Build the engine as described in the Basalt README, then:

hf download jesdga/Hy4-flash-Basalt Hy4-flash-NVFP4.basalt --local-dir models
bin/basalt serve models/Hy4-flash-NVFP4.basalt --vision dynamic

Add --gpu 0,1 to use a second card and --max-context N for a longer context. The engine reads the model's chat format, tool-call syntax and thinking switch from the file. More in Hy4-flash on Basalt and models and the router.

Sources

  • Model: tencent/Hy4-flash, Tencent.
  • This pack: built from the BF16 checkpoint; what each part is stored as is below. Tokenizer and chat template from the original release.
  • Basalt is a fork of Strata (MIT).

License

The model is released by Tencent under the Tencent Hy License, Version 1.0, included as LICENSE. This file is a modified (quantized and repacked) form of the original weights. This is not an official Tencent release. The Basalt engine is MIT.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jesdga/Hy4-flash-Basalt

Finetuned
(15)
this model