feat: Handle FP8 KV cache incompatibility with snapshot models in build command

docs/adr/0031-fp8-quantization-control.md

@@ -84,6 +84,18 @@ output quantized, we can pin `quantization="fp8"` on it in the catalogue.
   deploy (no stale-precision restores — the snapshot is keyed to the new function
   version). A model that can't serve FP8 fails at snapshot *creation*, which is the
   same loud no-healthy-container failure as the plain path.
+- **FP8 KV cache is incompatible with sleep-mode/snapshot models on the pinned vLLM.**
+  `--kv-cache-dtype fp8` boots and snapshots fine, but the `/wake_up` path runs
+  `init_fp8_kv_scales()` over a post-sleep KV cache that is a *list* of per-layer
+  tensors (not one tensor), so `cache_tensor.zero_()` throws and every snapshot restore
+  500s — an endpoint that boots but can never wake. This bit `nemotron-3-nano-4b`
+  (`gpu_snapshot=True`) under a global `MODAL_LLM_KV_CACHE_DTYPE=fp8` deploy.
+  `build_command` therefore **drops an FP8 `kv_cache_dtype` for any `gpu_snapshot`
+  model** and warns: snapshot is a structural per-model decision, the KV dtype a deploy
+  knob, so snapshot wins and the endpoint serves with full-precision KV cache. Weight
+  `--quantization fp8` is a different code path and is unaffected. To actually run FP8
+  KV cache on such a model, drop `gpu_snapshot` (trade the fast cold start for the KV
+  win) — or revisit once the vLLM pin advances past the bug.
 - Possible future unlock: FP8 weights halve the host-RAM needed for sleep level 1,
   which was the stated blocker for snapshotting `nemotron-3-nano-30b` (~60GB BF16,
   ADR-0030). Unverified — Nemotron-H may reject on-the-fly FP8 entirely — so this

modal/docs/deploying.md

@@ -233,6 +233,14 @@ uv run scripts/deploy_modal.py nvidia --quantization none
 > `curl <url>/v1/models`); if a model won't start, redeploy that provider without
 > the flag. This is why all per-model defaults stay `None` for now. See ADR-0031.
 
+> **FP8 KV cache (`--kv-cache-dtype fp8`) is silently dropped for snapshot models.**
+> On the pinned vLLM it crashes the `/wake_up` path (`init_fp8_kv_scales` →
+> `'list' object has no attribute 'zero_'`), so an FP8-KV snapshot model boots but
+> can never wake. `build_command` drops the flag for any `gpu_snapshot=True` model
+> and logs a `⚠️` line at deploy; the endpoint serves with full-precision KV cache.
+> FP8 *weights* (`--quantization fp8`) are unaffected. To run FP8 KV cache on such a
+> model, set its `gpu_snapshot=False`. See ADR-0031.
+
 ## Auth
 
 Modal web endpoints are public by default. Secrets are supplied as environment

modal/service.py

@@ -220,6 +220,21 @@ def build_command(cfg: ModelConfig) -> list[str]:
     if quantization:
         cmd += ["--quantization", quantization]
     kv_cache_dtype = _resolve_precision(KV_CACHE_DTYPE, cfg.kv_cache_dtype)
+    # FP8 KV cache is incompatible with sleep-mode/snapshot models on the pinned
+    # vLLM: the wake path runs init_fp8_kv_scales() over a post-sleep KV cache that
+    # is a *list* of per-layer tensors, not one tensor, so cache_tensor.zero_()
+    # throws and /wake_up 500s (every snapshot restore dies). Snapshot is a
+    # structural per-model decision; the KV dtype is a deploy knob — so snapshot
+    # wins. Drop the flag and warn loudly rather than ship an endpoint that boots
+    # but can never wake. Weight --quantization is unaffected (different code path).
+    if kv_cache_dtype and cfg.gpu_snapshot and kv_cache_dtype.lower().startswith("fp8"):
+        print(
+            f"⚠️  {cfg.endpoint_name}: dropping --kv-cache-dtype {kv_cache_dtype} — "
+            "FP8 KV cache crashes the snapshot wake path on the pinned vLLM (see ADR-0031). "
+            "Serving with full-precision KV cache. Drop gpu_snapshot to keep FP8 KV cache.",
+            flush=True,
+        )
+        kv_cache_dtype = None
     if kv_cache_dtype:
         cmd += ["--kv-cache-dtype", kv_cache_dtype]
     # Performance / throughput knobs (all data-driven from ModelConfig).

tests/test_modal_build_command.py

@@ -85,6 +85,45 @@ def test_kv_cache_env_override(service, monkeypatch):
     assert cmd[cmd.index("--kv-cache-dtype") + 1] == "fp8"
 
 
+# ── FP8 KV cache × snapshot incompatibility (vLLM wake-path crash) ─────────────
+
+
+def test_fp8_kv_cache_dropped_for_snapshot_models(service):
+    # FP8 KV cache crashes the /wake_up path on snapshot models, so the flag is
+    # suppressed when gpu_snapshot is set — the endpoint serves with full-precision
+    # KV cache rather than booting into a state it can never wake from.
+    cmd = service.build_command(_make(service, kv_cache_dtype="fp8", gpu_snapshot=True))
+    assert "--kv-cache-dtype" not in cmd
+    # The snapshot flag itself still wins and is emitted.
+    assert "--enable-sleep-mode" in cmd
+
+
+def test_fp8_kv_cache_env_override_dropped_for_snapshot_models(service, monkeypatch):
+    # The global deploy override is the common trigger: it lands on every model in
+    # the app, including snapshot ones, which must still drop it.
+    monkeypatch.setattr(service, "KV_CACHE_DTYPE", "fp8")
+    cmd = service.build_command(_make(service, gpu_snapshot=True))
+    assert "--kv-cache-dtype" not in cmd
+
+
+def test_fp8_variant_kv_cache_dropped_for_snapshot_models(service):
+    # Every fp8 variant hits init_fp8_kv_scales, so fp8_e5m2 is dropped too.
+    cmd = service.build_command(_make(service, kv_cache_dtype="fp8_e5m2", gpu_snapshot=True))
+    assert "--kv-cache-dtype" not in cmd
+
+
+def test_non_fp8_kv_cache_kept_for_snapshot_models(service):
+    # The guard only fires on fp8; a non-fp8 dtype passes through even with snapshot.
+    cmd = service.build_command(_make(service, kv_cache_dtype="auto", gpu_snapshot=True))
+    assert cmd[cmd.index("--kv-cache-dtype") + 1] == "auto"
+
+
+def test_fp8_kv_cache_kept_for_non_snapshot_models(service):
+    # Without snapshot there's no wake path, so FP8 KV cache stays.
+    cmd = service.build_command(_make(service, kv_cache_dtype="fp8", gpu_snapshot=False))
+    assert cmd[cmd.index("--kv-cache-dtype") + 1] == "fp8"
+
+
 # ── deploy script wiring ───────────────────────────────────────────────────────