--- base_model: Jackrong/Qwopus3.6-27B-Coder base_model_relation: quantized license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text language: - en - zh tags: - nvfp4 - fp4 - w4a4 - gptq - quantized - compressed-tensors - llm-compressor - vllm - qwen3_5 - vision-language - thinking - code - coder ---
โ—† Rogue Quants ยท NVFP4

๐Ÿช Qwopus3.6-27B-Coder ยท NVFP4

๐Ÿ’ป Coder

27B coder vision-language · agentic + tool-calling · thinking · GPTQ NVFP4 W4A4

โš™๏ธ NVFP4 · W4A4 ๐Ÿ’พ ~18 GB ๐Ÿ“‰ PPL 6.63 ๐Ÿ“ 256K context ๐Ÿš€ vLLM · Blackwell ๐Ÿ’ป Coder ๐Ÿ› ๏ธ Tool-calling
Size on disk 18 GB vs 55.6 GB bf16 (~33%)
wikitext-2 PPL 6.63 near-lossless vs bf16
Context 256K 262144 tokens
Scheme NVFP4 W4A4 · GPTQ + MSE
**TL;DR:** Qwopus3.6-27B-Coder, quantized to NVFP4 (W4A4) for vLLM on NVIDIA Blackwell. 18 GB, wikitext-2 PPL 6.63, 256K agentic coder. # Qwopus3.6-27B-Coder NVFP4 NVFP4 (W4A4) quantization of [Jackrong/Qwopus3.6-27B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder), packed in the `compressed-tensors` `nvfp4-pack-quantized` format with [llm-compressor](https://github.com/vllm-project/llm-compressor). Weights are quantized with GPTQ (error-compensated rounding) and an MSE observer, on a domain-matched calibration blend that includes code. **Near-lossless.** Fused layers (q/k/v, gate/up) share one NVFP4 global scale, so vLLM loads it cleanly with no per-layer-scale warning or fallback. wikitext-2 perplexity for this build: 6.63. - About 18 GB on disk versus about 55.6 GB for the bf16 source (about 33%). - Built for vLLM on NVIDIA Blackwell, where both the 4-bit weight and 4-bit activation paths are accelerated. On pre-Blackwell GPUs vLLM runs it weight-only. - Loading and generation verified in vLLM v0.23.0 on an NVIDIA GB10 (Blackwell, sm_121). ## Fidelity Near-lossless versus the bf16 source: wikitext-2 perplexity for this build is **6.63**. | Metric | Value | |---|---| | wikitext-2 PPL | 6.63 | | Weights | NVFP4 W4A4, group 16 | | Size | 18 GB vs 55.6 GB bf16 (~33%) | NVFP4 uses GPTQ error compensation, an MSE observer, and shared fused-layer scales, so the drop from bf16 is minimal. ## Quickstart NVFP4 is auto-detected from `config.json` (`compressed-tensors`); no quantization flag needed. `--reasoning-parser qwen3` splits the `` block into `reasoning_content`; `--tool-call-parser qwen3_coder` enables tool/function calling for agentic coding. ```bash vllm serve maci0/Qwopus3.6-27B-Coder-NVFP4 \ --served-model-name qwopus-27b-coder-nvfp4 \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` - Supports up to 262144 tokens; keep at least 128K to preserve thinking quality. `--max-model-len 131072` is a safe default; raise it if memory allows. - Add `--language-model-only` to skip the vision tower and free KV cache for text use. - The parser flags are not auto-detected; pass them explicitly. ### Python (OpenAI client) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="x") r = client.chat.completions.create( model="qwopus-27b-coder-nvfp4", messages=[{"role": "user", "content": "Write a Python function that merges two sorted lists."}], ) print(r.choices[0].message.content) ``` ### curl ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "qwopus-27b-coder-nvfp4", "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}] }' ``` ## About the base model A 27B Qwen3.5-family vision-language model specialized for code (Qwopus 3.6 Coder), with thinking-mode reasoning and a 256K context window. - 64 decoder layers: hybrid gated delta-net linear attention plus full attention, dense MLP, plus a vision tower for image and video input. - 256K context (`max_position_embeddings` 262144). - Thinking mode by default, with an instruct toggle. ## Quantization | | | |---|---| | Scheme | NVFP4, W4A4 | | Weight rounding | GPTQ (Hessian-based error compensation), MSE observer | | Weights | FP4 (E2M1), `group_size=16`, `tensor_group`, FP8 (E4M3) group scales, shared across fused layers | | Activations | FP4, dynamic per-group, FP8 (E4M3) scales | | Quantized | all language-model `Linear` layers | | Kept in bf16 | vision tower (`model.visual.*`), `lm_head`, MTP head | | Untouched | gated delta-net `Conv1d` and SSM params (`A_log`, `dt_bias`), never `Linear` | GPTQ is a quantization-time cost only; inference speed and format are identical to plain round-to-nearest NVFP4, but it chooses better 4-bit values. Calibration: 512 domain-matched samples (long reasoning + general chat + code), `max_seq_len=2048`, text-only path through the VL model. ## Recommended sampling Thinking mode is the default. - Thinking, precise coding: `temperature=0.6`, `top_p=0.95`, `top_k=20` - Thinking, general: `temperature=1.0`, `top_p=0.95`, `top_k=20` - Instruct / non-thinking: `temperature=0.7`, `top_p=0.80`, `top_k=20` - To run non-thinking, set `{%- set enable_thinking = false %}` in the chat template, or pass `extra_body={"chat_template_kwargs": {"enable_thinking": false}}`. ## Reproduction Toolchain: `llmcompressor==0.12.0`, `compressed-tensors==0.17.1`, `transformers==5.12.1`, `torch==2.11.0+cu130`, on an NVIDIA GB10 (Blackwell, sm_121). llm-compressor 0.12 shares the NVFP4 global scale across fused layers automatically (q/k/v, gate/up). ## Related - Base model: [Jackrong/Qwopus3.6-27B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder) - Space: [Rogue Quants](https://huggingface.co/spaces/maci0/rogue-quants) - Collection: [NVFP4 Quants](https://huggingface.co/collections/maci0/nvfp4-quants-gb10-blackwell-6a446fc03174db196e436339) - Sibling NVFP4 quants: - [Qwopus3.6-27B-Coder abliterated](https://huggingface.co/maci0/Qwopus3.6-27B-Coder-abliterated-NVFP4) (the abliterated version) - [Qwopus3.6-27B-v2 abliterated](https://huggingface.co/maci0/Qwopus3.6-27B-v2-abliterated-NVFP4) - [Ornith-1.0-35B MoE abliterated](https://huggingface.co/maci0/Ornith-1.0-35B-abliterated-NVFP4) - [Qwen3.6-40B Deckard](https://huggingface.co/maci0/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NVFP4) - [Huihui-Qwythos-9B Claude-Mythos](https://huggingface.co/maci0/Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NVFP4) - [Ornith-1.0-9B abliterated](https://huggingface.co/maci0/Ornith-1.0-9B-abliterated-NVFP4) ## Notes - Needs NVIDIA Blackwell (sm_121, e.g. GB10) for accelerated W4A4; pre-Blackwell GPUs run it weight-only. - `--reasoning-parser` and `--tool-call-parser` are not auto-detected; pass them explicitly. - Thinking mode is on by default; toggle it via the chat template or `chat_template_kwargs`. ## License Apache-2.0, following the base model. Intended use and all responsibility for use follow the base model. ## Credits - Base model: [Jackrong](https://huggingface.co/Jackrong) - Quantization tooling: [llm-compressor](https://github.com/vllm-project/llm-compressor) / [compressed-tensors](https://github.com/neuralmagic/compressed-tensors)
Part of ๐ŸŽฒ Rogue Quants, a set of NVFP4 (W4A4) quants for vLLM on Blackwell. See the full NVFP4 Quants collection.
Built on NVIDIA GB10 (Blackwell, sm_121) with llm-compressor ยท GPTQ + MSE ยท shared fused-layer scales.