Instructions to use kamiyugi/Hy3-NVFP4-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kamiyugi/Hy3-NVFP4-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kamiyugi/Hy3-NVFP4-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kamiyugi/Hy3-NVFP4-W4A16")
model = AutoModelForCausalLM.from_pretrained("kamiyugi/Hy3-NVFP4-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kamiyugi/Hy3-NVFP4-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kamiyugi/Hy3-NVFP4-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kamiyugi/Hy3-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kamiyugi/Hy3-NVFP4-W4A16

SGLang

How to use kamiyugi/Hy3-NVFP4-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kamiyugi/Hy3-NVFP4-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kamiyugi/Hy3-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kamiyugi/Hy3-NVFP4-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kamiyugi/Hy3-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use kamiyugi/Hy3-NVFP4-W4A16 with Docker Model Runner:
```
docker model run hf.co/kamiyugi/Hy3-NVFP4-W4A16
```

Hy3 — NVFP4 (W4A16, routed experts only)

A weight-only 4-bit NVFP4 quantization of tencent/Hy3, a 295B-parameter MoE (21B active). The routed MoE experts — the overwhelming majority of the weights — are quantized to 4-bit; every quality-sensitive component stays in BF16.

TL;DR: 295B → 174 GB (~30% of the ~590 GB BF16 source). Weight-only, so vLLM serves it on the MARLIN NvFp4 kernel. Quantized on a single NVIDIA DGX Spark (128 GB unified memory) via disk-offloaded, data-free RTN.

What is quantized to what

Component	Precision	Rationale
Routed experts (`…mlp.experts.{0..191}.{gate,up,down}_proj`)	NVFP4 (4-bit)	~95% of the weights — the only place the size win lives
Shared expert (`…mlp.shared_mlp.*`)	BF16	always-on path; quantizing it spreads error to every token
Attention, MoE router, dense-layer (layer 0) MLP	BF16	sensitive / small — kept native
Embeddings, `lm_head`, all norms	BF16 / F32	unchanged
MTP draft layer (layer 80)	BF16 (full precision)	copied verbatim from the source checkpoint

The ignore list keeps attention, the router, the shared expert, embeddings, lm_head, and the entire MTP layer out of quantization. Only the routed FFN experts (79 MoE layers × 192 experts × 3 projections = 45,507 linear modules) are converted to FP4.

Format & footprint


Size	174 GB (167 GB quantized body + 7.5 GB BF16 MTP layer)
Format	`compressed-tensors` `nvfp4-pack-quantized`, weight-only (W4A16)
Structure	E2M1 4-bit weights + FP8-E4M3 group-16 scales + FP32 per-tensor global scale
Base	tencent/Hy3 — 295B MoE (21B active, 3.8B MTP), 80 layers + 1 MTP layer, 192 experts top-8 + 1 shared, 256K context

Note on the reported parameter count. Hugging Face reports fewer "params" than the base's 295B because it counts packed 4-bit storage elements (each U8 byte holds two FP4 weights) plus the FP8/FP32 scales — not logical parameters. The logical model is unchanged: 295B total, 21B active.

Weight-reconstruction fidelity (SQNR)

Measured over 60 sampled expert projections (5 layers × 4 experts × 3 projections), comparing the original BF16 weights against the dequantized NVFP4 weights:

Metric	Value
Mean SQNR	20.44 dB (relative error ≈ 9.5%)
Spread	20.42–20.47 dB (σ ≈ 0.02 dB)

The tight spread across all sampled layers and experts confirms the quantization was applied uniformly — no layer or expert was skipped or corrupted. The mean is ~0.9 dB below an independent MSE-scale build (≈21.3 dB), exactly as expected for round-to-nearest (RTN) vs MSE-optimal scale selection. SQNR measures weight fidelity only; task-level accuracy is measured below.

Evaluation

Benchmark	This build (RTN)	Reference (MSE-scale build)
GSM8K (full 1319, 8-shot CoT)	93.93% (1239/1319)	~95.8%
Weight SQNR (sampled experts)	20.4 dB	~21.3 dB

The ~2-point GSM8K gap tracks the ~0.9 dB weight-fidelity gap between RTN and MSE-optimal scale selection — a consistent, modest cost for the simpler data-free RTN path. Evaluate on your own workload before deciding whether the extra fidelity is worth it.

Strict-match and flexible-extract agree exactly (both 93.93%, ±0.66), which indicates the quantized model still emits answers in the expected format — format-following is preserved, not just raw accuracy.

How it was made

Tool: llm-compressor (QuantizationModifier, scheme NVFP4A16), data-free RTN.
Hardware: a single DGX Spark (GB10, 128 GB unified memory). The 590 GB BF16 source does not fit in memory, so it was loaded with disk offloading (device_map="auto_offload") and quantized layer by layer. Total quantization time ≈ 40 min (load 1 min + compress 18 min + save 5.5 min).
hy_v3 support: the Hy3 architecture (hy_v3) was not registered in llm-compressor. Because HYV3Experts is implemented identically to Qwen2MoeExperts/Qwen3MoeExperts and the checkpoint uses the same per-expert 2D naming, the quantization was enabled by registering hy_v3 to reuse the qwen2_moe linearization mapping.
MTP: the source's MTP layer (layer 80) is not loaded by the transformers Hy3 class, so it was copied verbatim (BF16) into the output and added to the ignore list so serving engines treat it as non-quantized.

Serving with vLLM

HYV3ForCausalLM is natively supported by vLLM (recipe).

This is a weight-only (W4A16) build, so vLLM serves it on the MARLIN NvFp4 kernel. The FlashInfer fp4 tensor-core backends (flashinfer_trtllm / cutlass / cutedsl) are W4A4 and reject a weight-only scheme (kernel does not support QuantKey(u8, scale(f8e4m3))), so vLLM auto-selects MARLIN. Do not pass --moe-backend — let it auto-select.

Single large-VRAM GPU:

vllm serve kamiyugi/Hy3-NVFP4-W4A16 \
  --served-model-name hy3 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Multi-GPU / multi-node tensor parallelism (e.g. 2× smaller cards):

vllm serve kamiyugi/Hy3-NVFP4-W4A16 \
  --served-model-name hy3 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Note: Hy3 has 8 KV heads, so tensor-parallel size must divide 8 (1/2/4/8); TP=3 is not valid. Start at --max-model-len 4096 and raise once loading is confirmed.

Design notes & limitations

Data-free RTN. Weight-only NVFP4 scales are derived from the weights themselves, so no calibration data was used. An AWQ or MSE-optimal-scale variant would likely improve fidelity at the cost of build time; not applied here.
Evaluated. GSM8K (full 1319-task, 8-shot CoT): 93.93% (1239/1319). Weight fidelity: SQNR ≈ 20.4 dB (above). For reference, an independent NVFP4 W4A16 build of the same base model using MSE-optimal scale selection reports GSM8K ≈ 96% — about 2 points higher, consistent with the ~0.9 dB higher weight SQNR of MSE vs the round-to-nearest (RTN) scales used here. In other words, RTN keeps math/reasoning largely intact; MSE-scale or AWQ would recover roughly the remaining ~2 points at extra build cost.
Quantization was verified structurally: 45,507 expert projections carry the weight_packed / weight_scale / weight_global_scale triplet, and all ignored components remain BF16.

License

Apache License 2.0, inherited from the base model tencent/Hy3. See the base model license.

Acknowledgements

Built with llm-compressor and compressed-tensors. Base model by Tencent.

Downloads last month: 233

Safetensors

Model size

170B params

Tensor type

F32

BF16

F8_E4M3

Model tree for kamiyugi/Hy3-NVFP4-W4A16

Base model

tencent/Hy3

Quantized

(43)

this model