Arete-27B-W4A16-AutoRound

This is an int4 weight-only (W4A16) quantization of NousResearch/Arete-27B (revision 108f0afb). It was made with Intel AutoRound 0.15.0 and is packaged as compressed-tensors (pack-quantized) for vLLM. The weights are int4 with group size 128. The activations stay BF16, so the Marlin int4 kernel runs it on any NVIDIA GPU from Ampere on. The files are 19.5 GB, against about 55 GB in BF16.

Parameters: 27.8B (27.4B of Arete-27B plus the 0.4B MTP head).

Arete is a supervised fine-tune of Qwen/Qwen3.8-27B. Nous Research trained it to answer politically sensitive questions without the framing of the Chinese state. Its architecture, chat template and sampling settings are the same as those of the base model.

Recipe

The recipe is the one of dbirks/Qwen3.8-27B-W4A16-AutoRound (revision 1f05c441):

• Scheme W4A16: 4-bit symmetric integer weights, group size 128.
• Calibration: NeelNanda/pile-10k, 128 samples of 2,048 tokens, batch 4, 200 iterations, seed 42.
• Quantized: 400 of 607 linear layers (the GatedDeltaNet in_proj_qkv, in_proj_z and out_proj, the full-attention q/k/v/o projections and every MLP projection).
• Kept in BF16: linear_attn.in_proj_a and in_proj_b (96 layers), the vision tower, lm_head and the multi-token prediction (MTP) head.
• Time: 43 minutes on one RTX PRO 6000 Blackwell (96 GB). Peak 23.9 GB VRAM and 24.9 GB host RAM.

The tensor names, data types and shapes are the same as in the dbirks checkpoint (1,999 tensors), and config.json is the same apart from transformers_version. Thus it loads in the same way.

MTP head

Arete has no MTP head: Nous Research disabled it during training (mtp_num_hidden_layers: 0). This checkpoint contains the BF16 MTP head of the base model Qwen/Qwen3.8-27B (revision 37561816, 15 tensors) in model-mtp.safetensors, and config.json sets mtp_num_hidden_layers: 1. That head was trained on the hidden states of the base model, not of Arete. In our tests it drafts as well on Arete as on the base model: 2.5 to 2.6 tokens per verify step with three draft tokens, the same as the dbirks checkpoint.

To serve without speculative decoding, do not set --speculative-config. vLLM then does not use the MTP head.

Use with vLLM

vLLM reads the int4 scheme from config.json. No quantization flag is necessary.

vllm serve supz/Arete-27B-W4A16-AutoRound \
  --served-model-name arete-27b \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Use the sampling settings of the base model: temperature 1.0, top-p 0.95, top-k 20 in thinking mode. Greedy decoding is not recommended.

Evaluation

All tests used HyperQwen, not stock vLLM. HyperQwen is a patched vLLM 0.31 that fits Qwen3.8-27B with a 150K-token context on one 24 GB GPU. We ran it on one RTX PRO 4000 Blackwell (24 GB), with MTP drafting and an FP8 KV cache. HyperQwen's prepare.sh converts lm_head, embed_tokens and the MTP head to int8 before serving. Thus the numbers below are for that int8 variant of these files, not for the BF16 layers as published. The baseline is the dbirks checkpoint on the same GPU and runner.

Check	This checkpoint	dbirks W4A16
Decode, one stream / eight streams	80 / 436 tok/s	85 to 94 / 465 tok/s
Tokens per verify step (MTP)	2.5 to 2.6	2.5 to 2.6
148K-token context check	pass	pass

Decode is 6 to 15 percent slower than the dbirks "fast" variant. That variant uses an int4 lm_head taken from the base model, which does not apply to Arete.

We also asked both checkpoints eight questions on topics where Qwen models use the framing of the Chinese state (Tiananmen 1989, the Tank Man, Xinjiang, Taiwan, the Hong Kong National Security Law, criticism of Xi Jinping, Tibet, Falun Gong), plus two control questions about other governments. The dbirks checkpoint refused the Tiananmen and Tank Man questions and used the framing of the Chinese state for Hong Kong, Tibet and Xi Jinping. This checkpoint answered all eight with the documented facts and gave the position of the government as context. Both answered the control questions fully. This checkpoint made some errors in details, for example names and dates. Check the details against a source.

No standard benchmark results (perplexity, lm-evaluation-harness) are available.

License

Apache 2.0, the same as NousResearch/Arete-27B and Qwen/Qwen3.8-27B.