File size: 3,774 Bytes
8899636 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
base_model: beyoru/Opera
license: apache-2.0
tags:
- nvfp4
- fp4
- compressed-tensors
- llm-compressor
- quantized
- qwen3_5
pipeline_tag: image-text-to-text
---
# Opera-NVFP4
NVFP4 (W4A4) quantization of [`beyoru/Opera`](https://huggingface.co/beyoru/Opera), a
finetune of Qwen3.6-27B. **55.6 GB → 22.6 GB** (2.5× smaller).
The recipe is reproduced field-for-field from
[`unsloth/Qwen3.6-27B-NVFP4`](https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4), read back
out of its `quantization_config`, so this checkpoint should behave like that one applied to
the Opera weights.
## Recipe
`compressed-tensors`, `format: mixed-precision`, two groups:
| group | weights | activations | targets |
|---|---|---|---|
| **group_0** — FP8 W8A8 | 8-bit float, symmetric, `channel`, static | 8-bit float, symmetric, `token`, **dynamic** | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56–63** `mlp.{gate,up,down}_proj` |
| **group_1** — NVFP4 W4A4 | 4-bit float, `group_size=16`, symmetric, `tensor_group`, static | 4-bit float, `group_size=16`, `tensor_group`, dynamic=`local` | all other `mlp.{gate,up,down}_proj` |
`ignore`: the entire vision tower (`model.visual.*`, 302 modules) stays **bf16**.
Three deliberate choices carried over from the reference recipe:
1. **The last 8 layers (56–63) keep FP8 instead of dropping to FP4** — final layers are the
most sensitive to quantization.
2. **All attention and GDN projections stay FP8**; only the mid-stack MLP goes to W4A4.
3. **Vision tower untouched.**
Both groups use *dynamic* activation scales and weight scales derived from the weights
themselves, so no calibration dataset was used.
## Reproducing
```bash
pip install llmcompressor==0.12.0 compressed-tensors==0.17.1
python quantize_opera_nvfp4.py --model beyoru/Opera --output ./Opera-NVFP4
```
Two things that will bite you:
- **`oneshot()` needs `pipeline="datafree"` explicitly.** The default (`independent`)
dispatches to `sequential`, which requires a dataloader to trace the model and dies with
`TypeError: 'NoneType' object is not iterable`. The recipe genuinely needs no calibration
data, but llmcompressor will not infer that.
- **Load with `AutoModelForImageTextToText`, not `AutoModelForCausalLM`.** Opera is
`Qwen3_5ForConditionalGeneration` and carries a 333-tensor vision tower.
llmcompressor emits `Calibration pipeline is set to 'datafree', but it is recommended to use
'sequential'` — the run completes, but that warning is the first place to look if output
quality disappoints.
## ⚠️ Serving support is unverified
This checkpoint has **not** been load-tested. It uses `compressed-tensors`, which is the same
format that **SGLang 0.5.12 refused** for `unsloth/Qwen3.6-27B-NVFP4`
(`NotImplementedError: No compressed-tensors compatible scheme was found`). A newer vLLM or
SGLang is likely required. NVFP4 W4A4 also needs FP4 tensor cores — Blackwell (sm_100 /
sm_120 / sm_121); it will not run on Hopper.
## Note on the base model
`beyoru/Opera` is missing the MTP head that `Qwen/Qwen3.6-27B` ships: 1184 tensors vs 1199,
and the 15 absent ones are exactly `mtp.*`. `linear_attn` (432) and `visual` (333) are
identical. So this checkpoint **cannot do NEXTN speculative decoding** — the finetune dropped
the draft head. Grafting the 15 tensors back from the base model would restore it, at the cost
of an MTP head trained against the base's hidden states rather than the finetuned ones.
## Architecture
`qwen3_5`, 64 layers, hidden 5120, 24 attention heads, `full_attention_interval=4` — so
**48 of 64 layers use GDN linear attention**, which makes the KV cache dramatically cheaper
than a comparable full-attention model at long context.
|