# Qwen3.5-27B Writer-V2-Heretic + coding_mix_8k LoRA — FP8 (servable)

Built 2026-06-22. This is the **corrected, servable** FP8 artifact.

## Provenance
- Base: `llmfan46/Qwen3.5-27B-Writer-V2-uncensored-heretic` (BF16, model_type qwen3_5, hybrid GDN+full attention, VLM)
- LoRA: `coding_mix_8k` **checkpoint-25** (r16, alpha32), preserved at
  `training/qwen35-writer-heretic-lora-r16/adapters/final_candidates/8k_checkpoint_25_candidate_20260622_072520`
- Merged BF16: `/home/op/models/qwen/qwen35-writer-v2-heretic-coding-merged-bf16`
- Quant: llmcompressor 0.12.0, `QuantizationModifier(targets=Linear, scheme=FP8_DYNAMIC)` (compressed-tensors, float-quantized; channel-wise FP8 weights, dynamic per-token activations). Calibration-free.

## What is quantized
- **FP8 (256 Linears):** the standard text LMs only — `q/k/v/o_proj` on the 16 full-attention layers + `gate/up/down_proj` on all 64 layers. All 64-aligned → load via vLLM **FP8 Marlin (W8A16)** on Ampere.
- **BF16 (kept):** `lm_head`, the entire **GDN linear-attention** subtree (`model...linear_attn.*`), and the entire **vision tower** (`model.visual.*`).

### Why (do not regress this)
The first quant used **plain-string** `ignore` (`"visual"`, `"vision"`…). llmcompressor matches exact names / `re:` regex, so those matched nothing → vision tower + GDN projections got FP8'd with Marlin-incompatible dims (2152, 48) → vLLM crashed on Ampere: `size_n = 2152 is not divisible by tile_n_size = 64`. Fixed `ignore` to `["lm_head", "re:.*linear_attn.*", "re:.*visual.*"]`.
Also: llmcompressor's `save_pretrained` calls `from_accelerate()` which grabs a GPU and OOMs *after* writing weights but *before* the tokenizer → run the quant with `CUDA_VISIBLE_DEVICES=""`. Tokenizer/processor/chat_template here were copied from the merged BF16 dir.

## Serving (validated)
- Backend launcher: `/home/op/ULTIMATE/NAMEOFMODEL/run_qwen35_27b_writer_heretic_coding_fp8.sh`
- vLLM 0.19.1 (conda env `vllm`), **TP=2 on GPUs 0,1**, dtype bfloat16, **`--max-model-len 200000`**, **`--kv-cache-dtype fp8`**, util 0.92, text-only, `--disable-custom-all-reduce`, `--reasoning-parser qwen3`. **No MTP** (this model has no MTP head — layers 0–63 only). No tool-calling.
- **TP must be 2** (or 4): `num_key_value_heads=4` is not divisible by 3, so TP=3 is INVALID; only 3 GPUs available, so TP=2.
- 200k fits via the hybrid arch (only 16/64 layers carry growing KV) + fp8 KV: KV pool 53,312 tokens, "max concurrency for 200,000 tokens/request: 1.04x" (≈1 full-length request at a time; shorter requests are more concurrent).
- Proxy family `writer-heretic-coding-fp8` on http://127.0.0.1:12434 (backend :12562), **5 variants**: `writer-heretic-coding-fp8-{instruct,general,coding,thinking,rp}` (rp = roleplay; added to the proxy's global presets).
- **Sampling = official Qwen3.5-27B recommended** (per-family `preset_overrides`, no presence_penalty): instruct 1.0/0.95, general 0.7/0.8, coding 0.6/0.95 (think), thinking 1.0/0.95 (think), rp 1.0/0.95 — all top_k 20, min_p 0.
- Smoke: direct (:12562) + proxy (:12434), all modes (non-thinking + thinking + rp) return correct, coherent output. `verify_fp8.py` PASS.

## Notes
- `../qwen35-writer-v2-heretic-coding-fp8-dynamic` is the FIRST (broken) quant — kept for forensics; safe to delete to reclaim ~28GB.
- Vision tower is present in BF16 but served text-only; enabling vision needs an image-path test first.