devan-carlin/Qwen3.8-27B-int4-AutoRound

Qwen3.8-27B INT4 (AutoRound)

INT4 quantization of Qwen/Qwen3.8-27B, produced with Intel AutoRound.

• Base model: Qwen/Qwen3.8-27B (Apache-2.0)
• Quantization: AutoRound, INT4, group size 128 (w4g128), mixed symmetric/asymmetric
• Size: ~18 GB (vs ~52 GB for BF16)
• Context: 256K
• Architecture: qwen3_5 (Qwen3_5ForConditionalGeneration), 64 layers, 3:1 linear:full attention

Why

Runs a 27B dense model on 4x Intel Arc Pro B70 (32 GB each) with vLLM tensor parallelism. INT4 fits comfortably and runs faster than BF16 on this hardware.

Load with vLLM

vllm serve devan-carlin/Qwen3.8-27B-int4-AutoRound \
  --quantization auto-round \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code

Note: loading mixed symmetric/asymmetric INT4 checkpoints on the XPU/ARK path requires the qzeros guard fix (see the vLLM PR referenced below). Without it, symmetric layers crash with a copy_() shape mismatch during weight loading.

Benchmark (4x Intel Arc Pro B70, TP=4, max_tokens=16384)

Model	Gen speed	Notes
INT4 (this repo)	47.8 tok/s	18 GB
Qwen3.8-27B BF16	30.2 tok/s	~52 GB

• 58% faster than BF16
• Quality parity on math, logic, algorithm, and code prompts (identical final answers)
• INT4 reasons more verbosely (more completion tokens), so on very long outputs it can hit a fixed max_tokens cap earlier — raise the cap if needed

Quantization details

• Method: AutoRound (Intel)
• Bits: 4, group size: 128, symmetric packing
• mtp.fc kept in BF16; MTP layers quantized to INT4 (model stays MTP-capable)

License

Apache-2.0 (same as the base model).

---

Quantized and benchmarked by devan-carlin on 4x Intel Arc Pro B70 (XPU).