HY3-Abliterated-NVFP4

NVFP4 (4-bit) quantization of the abliterated Tencent Hy3 295B-A21B Mixture-of-Experts model.

~167 GB · fits a single 8×96 GB (or 4× at reduced context) Blackwell node · ~1.8× faster than BF16 · full 256K context.

What this is

A 4-bit NVFP4 quantization of trohrbaugh/Hy3-heretic — an abliterated ("decensored") version of tencent/Hy3, a 295B-parameter / 21B-active MoE reasoning-and-agent model. Abliteration removes refusal behavior (base Hy3 refuses ~99/100 harmful prompts; Hy3-heretic ~1/100). This quantization shrinks the model from ~590 GB (BF16) to ~167 GB while preserving that behavior and near-full quality.

Quantization details

Method	NVFP4 (E2M1 4-bit + FP8-E4M3 block scales + FP32 global scale), weight-only (W4A16)
Format	NVIDIA TensorRT Model Optimizer 0.45.0 (quant_method: modelopt, quant_algo: NVFP4)
Group size	16
Quantized	routed MoE experts only — model.layers.*.mlp.experts.*.{gate,up,down}_proj (45,504 tensors, ~95% of weights)
Kept in BF16	all attention (incl. o_proj), shared expert, router + expert bias, dense layer-0 MLP, embeddings, lm_head, all norms
Calibration	none — weight-only; scales derived from the weights
Fidelity	per-expert round-trip cosine ≈ 0.995 vs BF16
Size	590 GB (BF16) → 167 GB (NVFP4)

Recipe follows the community NVFP4 Hy3 work by LibertAIDAI/Hy3-NVFP4 and kodelow/Hy3-NVFP4-W4A16.

Serving (vLLM on Blackwell)

Serves on the NvFp4 MARLIN kernel; requires a vLLM build with hy_v3 / HYV3ForCausalLM support.

vllm serve Blackfrost-AI/Hy3-heretic-NVFP4 \
  --served-model-name hy3 --tensor-parallel-size 8 \
  --quantization modelopt --moe-backend marlin \
  --trust-remote-code --max-model-len 262144

For 4× Blackwell 96 GB, use --tensor-parallel-size 4. Note: this abliterated variant does not carry the base model's MTP (multi-token-prediction) head, so MTP speculative decoding is unavailable.

Credits

Full credit to everyone whose work this builds on:

• Tencent Hunyuan — the base Hy3 295B MoE model.
• trohrbaugh — the Heretic abliteration (Hy3-heretic).
• p-e-w / Heretic — the automatic-abliteration tool.
• LibertAIDAI and kodelow — prior NVFP4 Hy3 recipes this follows.
• NVIDIA TensorRT Model Optimizer — the NVFP4 quantization format.
• Blackfrost-AI — this NVFP4 conversion.

License

Apache 2.0, inherited from the base model.

Responsible use

This is an uncensored (abliterated) model: refusal behavior has been removed, and it will follow instructions the base model would decline. Use it lawfully and responsibly — you are accountable for what you generate.