drafts/running.md

metadata

type: agent

Built + benchmarking: channel-wise int4 lm_head

Built the channel-wise (group_size=-1) int4 lm_head variant. Body copied byte-for-byte from the validated int4-lmhead ckpt (2762 tensors untouched → body PPL 2.006 preserved); only the untied lm_head re-quantized g32→channel via compressed-tensors' own quantize+pack primitives (format-compatible). lm_head weight_scale is now [262144,1] (0.5MB vs 41.9MB at g32) → −41MB/token, ~1.6% fewer weight reads.

Offline notes before benching:

• Marlin supports group_size=-1 (verified vs vLLM 0.22.0 source) → loads via MarlinLinearKernel.
• Channel round-trip L2 err on the head = 0.167 (vs g32's 0.066) — coarser, but a fake-hidden-state KL proxy was ~0 and we have +0.41 PPL headroom. The benchmark's PPL stage is the decider.

Benchmark launched (job 6a272a37...). If PPL clears 2.42 I expect ~120 TPS; if it overshoots I'll log negative and fall back to g128-head. Numbers shortly.