Qwen3.8-27B-AWQ-INT4

AWQ W4A16 quantization of Qwen/Qwen3.8-27B, produced with llm-compressor in compressed-tensors pack-quantized format.

Built and qualified on Intel Arc Pro GPUs under SGLang.

Speculative decoding is enabled and qualified. The checkpoint's multi-token-prediction head is excluded from quantization, so SGLang can serve it as a draft head. On four Arc Pro B70 at TP4 this takes decode from 59.1 to 102.6 tok/s at 8K context and 60.2 to 136.7 tok/s at 256, with no measured quality regression across 421 executed HumanEval and MBPP problems. See Speculative decoding.

1,048,576 token context

This repository ships a 1,048,576 token context revision. Same weights, byte identical to main; only config.json differs.

from huggingface_hub import snapshot_download
snapshot_download("ulkaa/Qwen3.8-27B-AWQ-INT4", revision="1m")

Serve it with:

--context-length 1048576 --max-total-tokens 1048576 --max-mamba-cache-size 48

--max-mamba-cache-size 48 is required. At the default the Gated DeltaNet recurrent state pool takes 10.34 GB per rank and the KV pool comes up at 769,920 tokens while the server still reports context_len=1048576. Check that max_total_num_tokens equals context_len at startup.

Four Arc Pro B70 at TP4 hold the full 1,048,576 with 9.34 GB per card free. Decode at 8K/1K, c=1 is 16.70 to 17.08 ms TPOT, 58.5 to 59.9 tok/s, measured across two server starts on byte-identical inputs. main measures 16.81 ms at the same shape, inside that spread. At 262,144 resident tokens, 24.02 ms and 41.6 tok/s. Retrieval verified by needle probe at 109,057, 300,076 and 500,093 prompt tokens, exact at each.

main remains at the native 262,144 for anyone who wants Qwen's shipped rope settings.

Size	18.2 GiB (from 55.6 GiB BF16)
Format	compressed-tensors / pack-quantized
Scheme	W4A16 asymmetric, group size 128
Vision tower	preserved, BF16
MTP head	preserved, BF16
Gated DeltaNet projections	quantized, see below
Architecture	Qwen3_5ForConditionalGeneration
Context	262,144 on main, 1,048,576 on the 1m revision

What is quantized, and what is not

Qwen3.8-27B is a hybrid: 48 of its 64 decoder layers use Gated DeltaNet (linear_attn.*) and 16 use full attention.

Those DeltaNet layers hold three large projections per layer, in_proj_qkv at 100 MiB, in_proj_z at 60 MiB and out_proj at 60 MiB, which together are 10.36 GiB in BF16, 47 percent of everything read on a decode step. Some 4-bit builds leave them at full precision. This one quantizes them, which is what the model authors do in their own FP8 release: the official Qwen3.8-27B-FP8 checkpoint carries weight_scale_inv for exactly those three tensors and excludes only the small ones.

Left at BF16, matching the FP8 release's modules_to_not_convert: the vision tower, the MTP head, lm_head, embed_tokens, all norms, and the DeltaNet scalar gates in_proj_a and in_proj_b. Those gates are 96 wide before sharding, so a 32-element group would span a third of a row.

Result: Gated DeltaNet falls from 10.36 GiB to 2.73 GiB, and the whole artifact reads 14.19 GiB per decode step instead of 21.82.

How much of this is actually 4 bit

"INT4" covers a wide range of formats and this one sits at the conservative end.

stored at 4 bits	24.33 B parameters, 4.16 bits each once per group scales and zero points are counted
kept at BF16	3.45 B parameters, 12.4 percent of the model
overall	5.63 bits per parameter, 18.2 GiB

What stays at BF16: the embedding table, lm_head, every norm, the Gated DeltaNet scalar gates, the vision tower and the MTP head. Activations are BF16 throughout; only the weights are quantized, hence W4A16.

This is deliberately larger than a maximally compressed 4 bit build of the same model. The choices that make it larger, per group asymmetric scales at group 128, full precision embeddings and output head, and an untouched vision tower, are the ones that keep it close to the original in the checks above.

Footprint and throughput are therefore not comparable across quantization formats. A build that packs more of the model into fewer bits will be smaller and will usually decode faster on the same hardware, because decode at batch one is bound by how many bytes move per token. Which model is better for your use is answered by measuring quality on your workload, not by comparing file sizes or tokens per second between formats.

Quantization details

• Source: Qwen/Qwen3.8-27B BF16, not re-quantized from the FP8 release.
• Method: AWQ via llm-compressor, W4A16 asymmetric, group size 128.
• Calibration: 512 sequences of 1024 tokens, 60 percent code instructions from codeparrot/self-instruct-starcoder and 40 percent general instructions from HuggingFaceH4/ultrachat_200k, chat-template formatted.
• Pipeline: sequential, one decoder layer at a time (sequential_targets=["Qwen3_5DecoderLayer"]), so each layer is calibrated against the quantized output of the layers before it.

Group size 128 rather than 32 was a measured choice. Finer groups quadruple the scale and zero-point metadata that is read alongside the weights on every step, and the int4 matmul is slower on them. A group-32 build of this same model measured slower end to end despite holding fewer weight bytes.

Quality

Checked against the BF16 original on prompts spanning code, reasoning, factual recall and summarization. Where the two are comparable the quantized model agrees with BF16 on the most likely token at every step, and the BF16 token never falls outside the top eight candidates. Prefill perplexity moves from 8.87 to 9.33.

Greedy text still diverges from BF16, as it will for any 4 bit build. Treat this as a 4 bit model: good for interactive use, coding assistance and long context work, and not a drop in replacement where exact BF16 outputs matter.

Usage

SGLang on Intel Arc

docker run --rm -d --name qwen38 \
  --device=/dev/dri -v /dev/dri:/dev/dri \
  --group-add video --group-add "$(getent group render | cut -d: -f3)" \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --ipc=host --shm-size=64g --ulimit memlock=-1 \
  -p 30000:30000 -v /path/to/Qwen3.8-27B-AWQ-INT4:/model:ro \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  rahulunair/sglang-xpu:qwen3.8-27b-20260816 \
  python -m sglang.launch_server --model-path /model \
    --device xpu --tp-size 4 --host 0.0.0.0 --port 30000 \
    --trust-remote-code --attention-backend intel_xpu --page-size 64 \
    --context-length 32768 --max-total-tokens 32768 \
    # 1m revision: --context-length 1048576 --max-total-tokens 1048576 \
    #              --max-mamba-cache-size 48 \
    --chunked-prefill-size 4096 --mem-fraction-static 0.85 \
    --cuda-graph-config '{"decode":{"backend":"full","bs":[1,2,4,8]},"prefill":{"backend":"disabled"}}' \
    --skip-server-warmup

SYS_PTRACE and seccomp=unconfined are required. Without them the symmetric-memory all-reduce declines silently and the slower stock collective runs, with nothing in the log to say so.

Speculative decoding with the MTP head

The multi-token-prediction head is kept at BF16 and excluded from quantization, so it loads as plain BF16 linears. It is off unless asked for, and costs nothing while off. Add:

... --speculative-algorithm EAGLE \
    --speculative-num-steps 7 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 8

Use a draft window of 8. The commonly quoted setting is 4 and it leaves most of the gain unclaimed on this checkpoint.

Four Arc Pro B70, TP4, graph capture on, concurrency 1, second run:

prompt / output	without	with MTP
256 / 1,024	60.2 tok/s	136.7 tok/s
8,192 / 1,024	59.1 tok/s	102.6 tok/s

Three things to know:

• It is a single-stream gain. At concurrency 8 it is a wash. Use it for interactive sessions and coding agents, not for a saturated server.
• It is a short-context gain. The verify step grows with context while a plain decode step does not, so the advantage narrows quickly: measured 2.3x at 256 tokens, 1.7x at 8K, and a net loss by 16K. Leave it off above about 8K.
• Output is not bit-identical to non-speculative decoding. Across 421 executed coding problems there were zero regressions and 379 of 421 completions were byte-identical, with pass@1 0.7927 to 0.8171 on HumanEval and 0.7198 to 0.7276 on MBPP. If you need runs reproducible against a non-speculative baseline, leave it off.

Limitations

• Multimodal is preserved but lightly exercised. The vision tower and processor configs are present and the model loads as a conditional generation model; image inputs beyond basic checks have not been qualified.
• Calibration is text only. The vision tower is not quantized so it is unaffected, but no image data was used.
• Long-context quality is not graded. Retrieval was verified by needle probe past the native context, which shows extension works; a graded evaluation across depths has not been run.

License

Apache 2.0, inherited from the base model.