Instructions to use leoncca/Hy4-flash-AWQ-g32 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps Settings
- vLLM
How to use leoncca/Hy4-flash-AWQ-g32 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "leoncca/Hy4-flash-AWQ-g32" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "leoncca/Hy4-flash-AWQ-g32", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/leoncca/Hy4-flash-AWQ-g32
- SGLang
How to use leoncca/Hy4-flash-AWQ-g32 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "leoncca/Hy4-flash-AWQ-g32" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "leoncca/Hy4-flash-AWQ-g32", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "leoncca/Hy4-flash-AWQ-g32" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "leoncca/Hy4-flash-AWQ-g32", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use leoncca/Hy4-flash-AWQ-g32 with Docker Model Runner:
docker model run hf.co/leoncca/Hy4-flash-AWQ-g32
Hy4-flash AWQ G32
This checkpoint is a quantized derivative of Tencent Hy4-flash, prepared by leoncca. The base revision is d88c9dfd96ba.
This initial AWQ release was prepared to quickly verify that ReVolta can load and serve Hy4-flash on NVIDIA V100 GPUs. If we produce a higher-fidelity quantization, we will publish it in this repository and update this model card.
Quantization method
The routed experts in 41 layers use asymmetric W4A16 with group size 32, integer zero points and FP16 scales in AWQ GEMM format. Calibration captured routed-expert inputs from 100,513 teacher-forced token positions in the BF16 model. We used those activations to select per-channel scales for the gate and up projections, then applied damped GPTQ error compensation during weight quantization.
The four Engram tables use E4M3 codes with BF16 per-row scales. The checkpoint comprises 50 safetensors shards (167.24 GB).
Validation
ReVolta is the tested runtime for text serving on four NVIDIA V100 GPUs with tensor parallelism. A development build passed text serving checks for reasoning, direct answers, tool calls and prefix reuse.
The table measures exact full-vocabulary KL(candidate || BF16) and top-1 agreement on 8,675 teacher-forced positions from 28 held-out text records. The BF16 baseline and eager AWQ reference ran on an RTX 3090.
| Checkpoint and runtime | Mean KL (nats) | Median KL (nats) | p95 KL (nats) | Top-1 agreement |
|---|---|---|---|---|
| RTN G32 on ReVolta | 0.1571 | 0.0069 | 0.5160 | 90.37% |
| AWQ G32 on ReVolta | 0.1222 | 0.0041 | 0.3097 | 92.31% |
| AWQ G32 in the eager reference | 0.1204 | 0.0042 | 0.3418 | 92.41% |
In a separate 3,025-token diagnostic prompt, top-1 agreement with BF16 was 64.40% for eager AWQ and 63.60% for ReVolta AWQ. The held-out table covers the 28 shorter text records.
License and modifications
Tencent's model and this derivative are distributed under the Tencent Hy License, Version 1.0. The original model is by the Tencent Hy Team. See NOTICE for the changed files and SHA256SUMS for payload integrity.
- Downloads last month
- 36
Model tree for leoncca/Hy4-flash-AWQ-g32
Base model
tencent/Hy4-flash