evilbb2/Qwen3-30B-A3B-Instruct-2507-W4A16-g128-RDNA4

Qwen3-30B-A3B-Instruct-2507 — W4A16 (INT4, group 128) for AMD RDNA4 / ROCm

The first 4-bit MoE quant that actually loads on vLLM + AMD RDNA4 consumer/prosumer GPUs (Radeon AI PRO R9700, RX 9070/9070 XT — gfx1201), as of 2026-08-14.

Why this exists

Existing community 4-bit quants of this MoE use group_size: 32, which has no compatible kernel on vLLM ROCm (ConchLinearKernel supports group sizes [-1, 128]; Exllama needs uint4b8). They fail at load time with Failed to find a kernel that can implement the WNA16 linear layer.

This quant uses group_size 128, symmetric INT4 (compressed-tensors pack-quantized format), which loads and runs out of the box on the official AMD vLLM image.

Verified on

• Radeon AI PRO R9700 32 GB (gfx1201), ROCm 7.2.1 host driver
• Image: rocm/vllm:rocm7.13.0_gfx120X-all_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1 (vLLM 0.19.1)
• Loads with --max-model-len 16384 → 123,440 tokens of KV cache free on a single 32 GB card
• Smoke test: coherent multilingual output, correct arithmetic

docker run -d --name vllm-moe \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --shm-size=16g --security-opt seccomp=unconfined \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 \
  rocm/vllm:rocm7.13.0_gfx120X-all_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1 \
  vllm serve evilbb2/Qwen3-30B-A3B-Instruct-2507-W4A16-g128-RDNA4 --max-model-len 16384

How it was made

Shard-by-shard data-free RTN (round-to-nearest) symmetric quantization, scale = absmax/7, groups of 128 along the input dimension, packed with compressed_tensors.pack_to_int32. Quantized: all q/k/v/o_proj, gate/up/down_proj (experts included). Kept in original precision: MoE router gates, lm_head, embeddings, norms.

Honest caveats

• No calibration data was used (pure RTN). Expect some quality loss versus a calibrated AWQ/GPTQ quant. Structured quality benchmarks are in progress and this card will be updated.
• Experimental community quant — not affiliated with Qwen/Alibaba.