Hy4-flash AWQ G32

This checkpoint is a quantized derivative of Tencent Hy4-flash, prepared by leoncca. The base revision is d88c9dfd96ba.

This initial AWQ release was prepared to quickly verify that ReVolta can load and serve Hy4-flash on NVIDIA V100 GPUs. If we produce a higher-fidelity quantization, we will publish it in this repository and update this model card.

Quantization method

The routed experts in 41 layers use asymmetric W4A16 with group size 32, integer zero points and FP16 scales in AWQ GEMM format. Calibration captured routed-expert inputs from 100,513 teacher-forced token positions in the BF16 model. We used those activations to select per-channel scales for the gate and up projections, then applied damped GPTQ error compensation during weight quantization.

The four Engram tables use E4M3 codes with BF16 per-row scales. The checkpoint comprises 50 safetensors shards (167.24 GB).

Validation

ReVolta is the tested runtime for text serving on four NVIDIA V100 GPUs with tensor parallelism. A development build passed text serving checks for reasoning, direct answers, tool calls and prefix reuse.

The table measures exact full-vocabulary KL(candidate || BF16) and top-1 agreement on 8,675 teacher-forced positions from 28 held-out text records. The BF16 baseline and eager AWQ reference ran on an RTX 3090.

Checkpoint and runtime Mean KL (nats) Median KL (nats) p95 KL (nats) Top-1 agreement
RTN G32 on ReVolta 0.1571 0.0069 0.5160 90.37%
AWQ G32 on ReVolta 0.1222 0.0041 0.3097 92.31%
AWQ G32 in the eager reference 0.1204 0.0042 0.3418 92.41%

In a separate 3,025-token diagnostic prompt, top-1 agreement with BF16 was 64.40% for eager AWQ and 63.60% for ReVolta AWQ. The held-out table covers the 28 shorter text records.

License and modifications

Tencent's model and this derivative are distributed under the Tencent Hy License, Version 1.0. The original model is by the Tencent Hy Team. See NOTICE for the changed files and SHA256SUMS for payload integrity.

Downloads last month
36
Safetensors
Model size
220B params
Tensor type
BF16
·
I32
·
F32
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for leoncca/Hy4-flash-AWQ-g32

Quantized
(23)
this model

Collection including leoncca/Hy4-flash-AWQ-g32