---
base_model:
- tencent/Hy3
pipeline_tag: text-generation
license: apache-2.0
---
Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.
📖 Documentation | 🤗 Hugging Face | 🤖 ModelScope | 💬 WeChat
# Hy3 GPTQ-Int4 Quantization
We use GPTQ 4-bit quantization to compress Hy3 to ~1/4 size with minimal accuracy loss. See the benchmark below:
## Quickstart
### vLLM
Build vLLM from source:
```bash
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto
```
Start the vLLM server:
```bash
# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve AngelSlim/Hy3-GPTQ-Int4 \
--tensor-parallel-size 8 \
--tool-call-parser hy_v3 \
--reasoning-parser hy_v3 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy3-gptq-int4
```
