Hy4-flash in CMF for cortiq: CUDA and Vulkan, from 12 GB cards

#4
by infosave - opened

Hi Hy4 team, thank you for releasing Hy4-flash!

We added Hy4-flash support to cortiq, an open-source Rust inference engine (no Python, no ML framework), and published the model as single CMF files: https://huggingface.co/infosave/Hy4-flash-cmf

file routed experts the rest size
hy4-flash-q4tp.cmf 4-bit 8-bit, Engram as released 150.5 GB
hy4-flash-q2tp.cmf gate/up 2-bit, down 4-bit 4-bit, Engram as released 122.8 GB
hy4-flash-q2tp-sfq.cmf as q2tp, experts calibrated (better on code) 4-bit 122.8 GB

The engine runs the whole text stack: the four iHC streams, gated MLA with sinks, the 30 sliding-window and 12 sparse layers with the compressed indexer, the latent MoE (16 of 512 experts on the 1536-wide latent, plus the shared expert) and Engram. The files are converted from the FP8 release (block scales decoded exactly) with the bf16 attention shards on top.

Measured on one RTX PRO 6000 (96 GB) with CUDA, 256 greedy tokens after a 1 000-token prompt:

whole card within 24 GB within 12 GB
q2tp 65.1 tok/s 37.4 tok/s 24.0 tok/s
q4tp 55.3 tok/s 25.8 tok/s —

On smaller cards the routed experts live in host RAM and the card keeps the hot ones. The same files also run through Vulkan / DX12 / Metal (wgpu): 45.1 tok/s (q2tp) and 37.4 tok/s (q4tp) on the same card. Perplexity (512-token windows, WikiText-2 / Chinese docs / Rust): q4tp 35.80 / 39.13 / 3.92.

Quick start:

cargo install cortiq-cli --features cuda
hf download infosave/Hy4-flash-cmf hy4-flash-q4tp.cmf --local-dir .
CMF_GPU=cuda cortiq run hy4-flash-q4tp.cmf --prompt "Explain in three sentences why the sky is blue." --greedy --no-think

cortiq serve exposes an OpenAI-compatible API. cortiq reads text only for now (the vision tower is kept in the files). Feedback is welcome!

Sign up or log in to comment