Hy4 MLX

We introduce Hy4 MLX from Nace.AI, Tencent's Hy4 in 447.3 GB of text weights at 4.6 bits per weight. Generate text with the included Linux CUDA runner.

Results

Selected Hy4 quantizations: KL divergence versus download size

Selected public quantizations, scored against native BF16 on the same 11 contexts and 5,621 positions. Download sizes include vision weights, with the shared BF16 projector added to every GGUF file. Bit labels show average text-model storage precision, excluding vision. MLX and GGUF use different inference backends, so measured divergence can include backend effects.

Overview

The full model retains every routed expert, with precision assigned by weight class.

  • 78 decoder layers, with all 256 routed experts retained in every MoE layer.
  • 456.3 GB complete weight download, including the retained nontext weights.

Precision assignment, native-reference comparison and file verification

Nace Weight Space Quantization

Four-bit experts, higher precision for the rest.

Routed expert weights use 4-bit storage. Eligible trunk matrices use 8-bit storage, while routers, KV expansion and sensitive parameters keep higher precision. The text model averages 4.6 bits per weight.

Storage precision by weight class

Run it

Use Linux with CUDA 13, as tested on an NVIDIA A40.

python3 -m venv .venv-hy4
. .venv-hy4/bin/activate
pip install huggingface_hub
hf download nace-ai/Hy4-MLX-4.6bit --local-dir ./Hy4-MLX-4.6bit
pip install -r ./Hy4-MLX-4.6bit/requirements-cuda13.txt
CUDA_VISIBLE_DEVICES=0 python ./Hy4-MLX-4.6bit/run.py \
  --model ./Hy4-MLX-4.6bit --output ./hy4-answer \
  --device gpu --prompt 'What is the capital of France? Answer one short sentence.' \
  --max-tokens 16

Choose a new output directory for each prompt.

License

Tencent Hy License, Version 1.0, including its Section 10 commercial terms for large Model-as-a-Service operators. See NOTICE and modifications.

Downloads last month
55
Safetensors
Model size
774B params
Tensor type
U32
·
F32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nace-ai/Hy4-MLX-4.6bit

Base model

tencent/Hy4
Quantized
(9)
this model

Collection including nace-ai/Hy4-MLX-4.6bit