ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ

Qwen3.8-27B — 3-bit GSQ

Low-bit quantization of Qwen/Qwen3.8-27B produced with GSQ (Gumbel-Softmax Quantization).

This checkpoint applies GSQ-based post-training quantization to the model weights, reducing precision while preserving the original model's reasoning, coding, multilingual, long-context, and agentic capabilities.

The transformer weights are quantized to 3-bit GSQ with group size 128. The embedding layer and LM head are quantized separately to 4-bit RTN with group size 64 to preserve output quality and embedding fidelity.

• Paper: GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv:2604.18556)
• Paper page on HF: https://huggingface.co/papers/2604.18556
• Code: https://github.com/IST-DASLab/GSQ
• Collection: https://huggingface.co/collections/ISTA-DASLab/gsq

Evaluation Results

We evaluate the quantized checkpoint against the original Qwen/Qwen3.8-27B. Both models were evaluated with xhigh thinking enabled.

Accuracy

Benchmark	Base Model	3-bit GSQ
AIME 2025	100.00	100.00
GPQA Diamond	89.90	91.41

Generation Statistics

Benchmark	Base Model	3-bit GSQ
AIME 2025	0.603M	0.615M
GPQA Diamond	3.721M	3.705M

Note: Due to the stochastic nature of these tasks, benchmark results can exhibit significant variance across runs. The reported scores correspond to a single evaluation run and should not be interpreted as definitive estimates of model performance.

Calibration dataset

The GSQ quantization calibration dataset was constructed to represent a broad range of LLM workloads, including reasoning, coding, scientific tasks, multilingual understanding, long-context processing, and agentic behaviour.

The calibration mixture consists of:

Category	Percentage
Math	13.5%
Code	17.5%
Science	20.0%
General	12.5%
Multilingual	12.5%
Long context	14.0%
Agentic trajectories	10.0%

Serving with vLLM

Serving this checkpoint requires a patched vLLM installation.

Requirements:

• vLLM 0.27.1
• The included patch:

patch_vllm_qwen35_embedding.py

This patch enables vLLM support for quantized embedding weights.

Install vLLM:

pip install vllm==0.27.1

Apply the patch in the same Python environment:

python patch_vllm_qwen35_embedding.py

Then serve the model:

vllm serve ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \

Important: If vLLM is reinstalled or the environment is recreated, run the patch again before serving the checkpoint.

Model size / text-only usage

The full checkpoint size is approximately 11.83 GB when deployed with vision capabilities enabled.

The quantization calibration dataset used for this release did not include vision samples. Therefore, while the vision components are preserved in the checkpoint and can be loaded, they were not calibrated using multimodal calibration data.

For text-only deployment, the vision components are not required. The model can be loaded with the --language-model-only option:

vllm serve ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --language-model-only

This removes the vision-related components from the loaded model and reduces the checkpoint size to approximately 10.90 GB.

This release does not currently support speculative decoding. The MTP (Multi-Token Prediction) components have been removed from the published checkpoint and are not available for MTP-based inference.

Actual VRAM usage during serving will be higher than the raw checkpoint size and depends on:

• KV-cache allocation
• context length
• batch size
• vLLM runtime overhead
• GPU architecture

Citation

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026},
  url    = {https://arxiv.org/abs/2604.18556}
}