---
license: apache-2.0
base_model: Jackrong/Qwopus3.6-27B-Coder-FP8
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- qwen3_5
- qwen3_6
- qwopus
- coder
- autoround
- int4
- w4a16
- w4g128
- vllm
- multimodal
- mtp
- speculative-decoding
---

# Qwopus3.6-27B-Coder-FP8 INT4 AutoRound

W4A16 INT4 AutoRound quantization of [`Jackrong/Qwopus3.6-27B-Coder-FP8`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-FP8).

- Quantization: AutoRound INT4, group size 128, symmetric, `auto_round:auto_gptq`.
- Source checkpoint: `Jackrong/Qwopus3.6-27B-Coder-FP8` at the time of quantization.
- Non-text multimodal modules are kept in their original precision.
- Native Qwen3.5/Qwen3.6 MTP is preserved. `mtp.fc` is stored as BF16 `mtp.fc.weight`, not packed `mtp.fc.qweight`, so vLLM can load the MTP drafter.
- Produced on one RunPod H200 SXM with AutoRound nightly.

## vLLM

```bash
vllm serve WaveCut/Qwopus3.6-27B-Coder-FP8-int4-AutoRound \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
```

For long-context serving, raise `--max-model-len` according to your KV-cache budget.

## vLLM CUDA 13 Smoke and Benchmarks

Smoke and throughput checks were run on 2026-06-14 with `vllm 0.23.0`, `torch 2.11.0+cu130`, Python 3.12.3, one NVIDIA B200, and NVIDIA driver `580.105.08`. [CUDA Toolkit release notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) document per-release minimum driver requirements; in this run, a B200 host with driver `570.*` failed CUDA 13 initialization, while driver `580.105.08` worked.

The working RunPod image was `runpod/pytorch:1.0.3-cu1300-torch291-ubuntu2404` (`cu13-pytorch2.9`, template `0uy1f6v18r`). After vLLM install, `nvidia-cutlass-dsl-libs-cu13` was force-reinstalled once to fix a CUTLASS RECORD mismatch; after that vLLM used the `FlashInfer GDN prefill kernel`.

vLLM resolved this model as `Qwen3_5ForConditionalGeneration`, loaded the AutoRound/AutoGPTQ path with `MarlinLinearKernel for AutoGPTQLinearMethod`, and completed generation. MTP speculative decoding resolved `Qwen3_5MTP`, loaded without missing-parameter warnings, shared embedding/lm_head with the draft model, and completed generation.

Benchmarks used `vllm bench throughput`, fixed random prompts, `max_model_len=8192`, tensor parallel size 1, and local model files on overlay disk. TPS values are vLLM timed-section values; wall time includes model load, compile, CUDA graph capture, and warmup.

| case | input -> output | prompts | gpu util | mode | total tok/s | prompt tok/s est | output tok/s est | peak VRAM GiB | max W |
| --- | ---: | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: |
| balanced_graph_u65 | 1024 -> 128 | 64 | 0.65 | graph | 6369.6 | 5661.9 | 707.7 | 117.6 | 850.4 |
| prefill_graph_u65 | 4096 -> 16 | 32 | 0.65 | graph | 7416.7 | 7387.8 | 28.9 | 117.6 | 857.4 |
| decode_graph_u65 | 128 -> 256 | 64 | 0.65 | graph | 4221.6 | 1407.2 | 2814.4 | 116.6 | 819.7 |
| balanced_eager_u65 | 1024 -> 128 | 32 | 0.65 | eager | 2453.9 | 2181.3 | 272.7 | 118.6 | 823.9 |
| balanced_graph_u85 | 1024 -> 128 | 64 | 0.85 | graph | 6614.3 | 5879.4 | 734.9 | 153.9 | 851.3 |
| balanced_mtp_u65 | 1024 -> 128 | 32 | 0.65 | graph + MTP | 4796.2 | 4263.3 | 532.9 | 118.1 | 846.5 |

First graph runs had cold costs around 77-80 seconds for `torch.compile` plus CUDA graph capture/profile. Repeated same-layout graph runs loaded the compile cache much faster. Eager mode was substantially slower than graph mode on this workload.

## 24GB RTX 3090 vLLM Smoke

A small fit smoke was run on 2026-06-14 on one RTX 3090 24GB RunPod host with NVIDIA driver `580.159.03` (`nvidia-smi` CUDA `13.0`), `vllm 0.23.0`, `torch 2.11.0+cu128`, and `runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404`.

The smoke used `max_model_len=32768`, `kv_cache_dtype=fp8`, `dtype=bfloat16`, `max_num_seqs=1`, `max_num_batched_tokens=2048`, chunked prefill enabled, prefix caching disabled, and one 128 -> 16 random request. The [vLLM Qwen3.5/Qwen3.6 recipe](https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3.5.html) recommends MTP-1 speculative decoding with prefix caching disabled for latency-sensitive low-concurrency serving.

| mode | load format | result | peak VRAM | KV cache | 32k concurrency | smoke throughput |
| --- | --- | --- | ---: | ---: | ---: | ---: |
| no MTP | `fastsafetensors` | pass | 22174 MiB | 64170 tokens | 1.96x | 50.33 total tok/s, 5.59 output tok/s |
| MTP-1 | `safetensors` | pass | 24110 MiB | 60681 tokens | 1.85x | 28.94 total tok/s, 3.22 output tok/s |
| MTP-1 | `fastsafetensors` | fail | 23778 MiB | n/a | n/a | CUDA OOM while allocating a 3.00 GiB staging buffer |

Recommended 24GB command shape:

```bash
vllm serve WaveCut/Qwopus3.6-27B-Coder-FP8-int4-AutoRound \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --load-format safetensors
```

For MTP-1 on 24GB, keep `--load-format safetensors` and add:

```bash
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
```

## Provenance

This repo was generated from the public Apache-2.0 source checkpoint. It keeps the upstream tokenizer, processor, chat template, vision config, and Qwen3.5 MTP config intact.