feat: Replace llama.cpp backend with in-process transformers backend for local GPU inference

docs/adr/0032-llama-cpp-local-backend.md

@@ -2,8 +2,11 @@
 
 ## Status
 
-Accepted (extends ADR-0024 *second inference backend / unified registry*, amends
-ADR-0015 *LiteLLM gateway*, ADR-0022 *per-agent explicit model binding*)
+Superseded by ADR-0033. llama.cpp's persistent `llama-server` cannot hold a GPU under
+ZeroGPU's per-call grant model; replaced by an in-process `transformers` backend that
+works on any HF Space hardware (ADR-0024 *second inference backend / unified registry*,
+ADR-0015 *LiteLLM gateway*, ADR-0022 *per-agent explicit model binding* remain in
+force; this ADR is retained as a historical record only).
 
 ## Context
 

docs/adr/0033-local-inproc-transformers-backend.md

@@ -0,0 +1,132 @@
+# ADR-0033: Local In-Process `transformers` Backend (Supersedes ADR-0032)
+
+**Status:** Accepted
+**Date:** 2026-06-13
+**Deciders:** project maintainers
+
+## Context
+
+ADR-0032 added a llama.cpp backend: a persistent `llama-server` process that exposes
+an OpenAI-compatible HTTP endpoint and uses the operator's GPU for the lifetime of that
+process. This model is structurally incompatible with Hugging Face **ZeroGPU**.
+
+ZeroGPU grants a GPU *only for the duration of a `@spaces.GPU`-decorated function call*,
+then reclaims it. A long-lived HTTP server needs to hold the GPU between requests — there
+is no GPU to hold on ZeroGPU. The same mismatch rules out vLLM-as-a-server for the same
+reason. Per the ZeroGPU documentation: the runtime is Gradio-SDK only; the GPU is an
+NVIDIA RTX Pro 6000 Blackwell (48 GB `large` / 96 GB `xlarge`); anonymous users get ~2
+minutes of free GPU time per day, authenticated users ~5 minutes; and the canonical usage
+pattern is `transformers`/`diffusers` with model weights placed on `cuda` at module load
+and the forward pass wrapped in `@spaces.GPU`. CUDA is *emulated* (no-op) outside the
+decorated function and real inside it.
+
+A second goal existed alongside ZeroGPU compatibility: the backend must be
+**hardware-agnostic**. An HF Space can be assigned a **dedicated GPU** (T4, L4, L40S,
+A100, …) or run on a local CUDA box. The solution should serve equally well in all three
+environments — ZeroGPU, dedicated GPU, and local CUDA — without a code branch per
+environment.
+
+Replacing the llama.cpp GGUF path also means explicitly dropping the **Llama Champion**
+bonus-quest badge (which required a real llama.cpp runtime in the cast). This is a
+deliberate tradeoff: ZeroGPU compatibility and hardware-agnosticism are higher-value than
+one badge.
+
+No new Python dependencies are needed: `torch` and `transformers` already ship
+transitively via `sentence-transformers`.
+
+## Decision
+
+We will replace the llama.cpp backend with an in-process `transformers` backend that runs
+model inference inside a `@spaces.GPU`-decorated call, caches loaded weights at module
+level in the parent process so each ZeroGPU call inherits them without re-loading, and
+gates availability on operator capability rather than a URL environment variable.
+
+Four concrete sub-decisions:
+
+**1. Non-HTTP router seam (`ProfileSpec.kind`).** `ProfileSpec` gains a `kind` field
+(`"litellm"` | `"local"`). When `kind == "local"` the router dispatches to
+`LocalTransformersProvider` directly, bypassing LiteLLM and HTTP entirely. This is the
+first backend that does not go through the HTTP gateway — it is a clean extension of the
+router seam left open by ADR-0024, not a hack around it.
+
+**2. In-process forward pass with effect-free decorator.** `LocalTransformersProvider`
+wraps the `transformers` forward pass in a module-level `@spaces.GPU(duration=<dynamic>)`
+function. On ZeroGPU this decorator acquires and releases the GPU for that call's
+duration. On a dedicated GPU or local CUDA box the decorator is a no-op (effect-free
+passthrough), so the same code path is a persistent in-process provider on those
+environments. No environment-specific branching in the provider.
+
+**3. Parent-process model cache, lazy on first use.** The official ZeroGPU guidance is to
+load model weights at module level (import time) so forked `@spaces.GPU` calls inherit
+them via copy-on-write. We deviate deliberately: weights are loaded lazily on the first
+`complete()` call and cached in a module-level `_LOADED` dict. This avoids loading unused
+models at app boot and keeps the no-API-key stub fast. `torch` and `transformers` imports
+are also lazy (never at module import) to prevent CUDA initialisation before the fork and
+to avoid tripping the PyTorch multiprocessing fork guard. The tradeoff is that the *first*
+call to a model incurs load time; all subsequent calls within the process inherit the
+cache for free, matching the per-call efficiency of the module-level-load pattern.
+
+**4. Capability gate and per-run opt-in.** `local_catalogue.has_credentials()` gates on
+one of three signals: `SPACES_ZERO_GPU` is set in env (HF ZeroGPU Space), or
+`LOCAL_INFERENCE=1` is set (explicit operator opt-in on a dedicated GPU or local CUDA
+box), or a cached `torch.cuda.is_available()` probe is true — but the auto-probe runs
+only when the env argument is `None` or `os.environ` itself, so tests passing an explicit
+env dict never import torch and stay deterministic without a GPU. Picking "Local GPU" in
+the Lab backend radio is the per-run opt-in; when none of the three signals is present the
+backend stays inactive and the deterministic stub owns the no-config demo path.
+
+## Alternatives Considered
+
+| Option | Pros | Cons |
+|--------|------|------|
+| Keep llama.cpp + server-per-request workaround | Preserves Llama Champion badge; GGUF models need no full-precision weights | Structurally incompatible with ZeroGPU; server startup latency per request; high complexity |
+| vLLM-as-a-server on ZeroGPU | High throughput batching | Same persistent-process / per-call-GPU mismatch as llama.cpp |
+| `llama-cpp-python` (in-process, no server) | Llama Champion badge preserved; GGUF quantisation | Additional heavy binary dep; GGUF format separate from HF model hub; weaker `transformers` ecosystem integration |
+| In-process `transformers` (chosen) | Works on ZeroGPU, dedicated GPU, and local CUDA without branching; no new deps; full HF Hub model catalogue; keeps OpenBMB/MiniCPM and Tiny-Titan lanes | Drops Llama Champion badge; full-precision weights (larger VRAM footprint than GGUF quants) |
+
+## Consequences
+
+**Positive:**
+- The Space runs on ZeroGPU free tier with no code changes — every forward pass
+  naturally fits the per-call GPU grant model.
+- The same binary runs on a dedicated GPU (T4/L4/L40S/A100) or local CUDA box with no
+  env-branch; the decorator is transparent.
+- No new Python dependencies — `torch` and `transformers` are already transitive deps.
+- The parent-process cache means each ZeroGPU call after the first is weight-load-free
+  within a session.
+- **Prize-lane impact:** keeps the **OpenBMB / MiniCPM** track (MiniCPM4.1-8B is in the
+  local catalogue), the **Tiny Titan** lane (Qwen2.5-3B-Instruct at the `tiny` tier is
+  ≤4B), and the **Community Choice** on-device-inference story for the HF Space demo.
+
+**Negative / Risks:**
+- **Llama Champion badge is explicitly dropped.** No llama.cpp runtime in the cast means
+  this submission does not qualify for that bonus quest. This is a deliberate, accepted
+  tradeoff.
+- First call to a model in a fresh process incurs full weight-load latency. On ZeroGPU
+  with the daily quota (~2–5 min GPU/day) this is a real cost on cold sessions.
+- Full-precision (BF16/FP16) weights require more VRAM than GGUF quantised equivalents.
+  The ZeroGPU `large` tier (48 GB) is the practical ceiling; models above ~28B BF16 will
+  not fit without external quantisation.
+- The ZeroGPU free-tier quota (≈5 min GPU/day authenticated) limits live demo length.
+  A dedicated GPU Space eliminates this limit but costs credits.
+- Lazy torch import means the first call also pays Python import overhead for
+  `torch` + `transformers`. Acceptable for interactive demo pacing; unacceptable for
+  low-latency production workloads.
+
+**Neutral / Notes:**
+- `llamacpp_catalogue.py` and `llamacpp_server.py` are deleted; `app.py`'s
+  `gpu_selftest` `@spaces.GPU` guard is retained — it detects ZeroGPU availability at
+  startup and is unrelated to the inference path.
+- The single catalogue entry tagged as a tier default is `Qwen/Qwen2.5-3B-Instruct`
+  (`tiny`). Additional models (MiniCPM4.1-8B, Qwen2.5-7B-Instruct) are in the catalogue
+  but untagged; ZeroGPU quota pressure justifies keeping the default footprint minimal.
+- Tests live in `tests/test_local_backend.py`. All 676 tests pass; the capability-gate
+  logic is fully covered without a GPU or torch import in test processes.
+
+## Related ADRs
+
+- ADR-0032: llama.cpp local backend — superseded by this ADR; retained as historical record.
+- ADR-0024: Hugging Face inference backend / unified registry — the `ProfileSpec.kind` seam this ADR extends.
+- ADR-0015: LiteLLM gateway — bypassed by `kind="local"`; all other backends still go through it.
+- ADR-0022: Per-agent explicit model binding — unchanged; `local:<repo_id>` qualified keys follow the same binding contract.
+- ADR-0019: Single model catalogue — local catalogue follows the same `binding_for` / `has_credentials` interface.

docs/architecture/model-routing.md

@@ -77,7 +77,7 @@ Profiles map to the OpenAI-compatible vLLM endpoints served on Modal
 The LiteLLM model string for an OpenAI-compatible custom endpoint is
 `openai/<served_model_id>` with `api_base` pointing at the endpoint's `/v1` URL.
 
-### Backends: Modal · Hugging Face · llama.cpp
+### Backends: Modal · Hugging Face · Local GPU
 
 A *backend* is just a catalogue + a binding rule, unified behind one registry
 (`src/models/inference.py`, ADR-0024). Models are named by a **backend-qualified
@@ -88,25 +88,37 @@ working. Three backends ship today:
 |---|---|---|---|
 | Modal | *(bare)* | vLLM endpoints you deploy on Modal GPUs | `MODAL_WORKSPACE` / `MODAL_LLM_BASE_URL` |
 | Hugging Face | `hf:` | serverless Inference Providers router | `HF_TOKEN` |
-| llama.cpp | `llamacpp:` | **local** GGUF via `llama-server`, GPU when present | `LLAMACPP_BASE_URL` |
+| Local GPU | `local:` | in-process `transformers` model on the host's own GPU | `SPACES_ZERO_GPU` / `LOCAL_INFERENCE=1` / CUDA auto-detected |
 
-The llama.cpp backend (ADR-0032) runs a cast fully on your own machine. Launch a
-model and export the URL it prints:
+The local backend (ADR-0033, supersedes ADR-0032) runs a cast fully in-process —
+no HTTP server, no extra process, no token. It loads a small instruct model via
+`transformers` inside a `@spaces.GPU` function (`LocalTransformersProvider`,
+`src/models/local_provider.py`). The hardware path is transparent:
 
-```bash
-uv run python -m src.models.llamacpp_server nemotron-3-nano-4b   # GPU auto-detected
-export LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1
-```
+- **ZeroGPU Space** — `@spaces.GPU` allocates a GPU per call from the shared pool
+  (~5 min/day free quota). Enabled when `SPACES_ZERO_GPU` is set.
+- **Dedicated-GPU Space** (T4 / L4 / L40S / A100) — persistent GPU, no per-call
+  quota. Enabled with `LOCAL_INFERENCE=1`.
+- **Local CUDA box** — the same `LOCAL_INFERENCE=1` flag, or CUDA auto-detected at
+  startup.
+
+Off ZeroGPU and without `LOCAL_INFERENCE=1`, `@spaces.GPU` is a no-op and the
+engine falls back to the deterministic stub — so the demo is always reproducible on
+CPU-only hosts. Pick "Local GPU" in the Lab's backend radio to opt in per run.
+
+Available models (all ≤32B; select via `local:<repo_id>`):
+
+| Key | Model | Notes |
+|---|---|---|
+| `local:Qwen/Qwen2.5-3B-Instruct` | Qwen 2.5 3B | **tiny default** — latency + quota guardrail |
+| `local:openbmb/MiniCPM4.1-8B` | MiniCPM 4.1 8B | alternate; OpenBMB prize lane |
+| `local:Qwen/Qwen2.5-7B-Instruct` | Qwen 2.5 7B | alternate |
 
-The launcher detects an accelerator — Apple Metal on macOS, NVIDIA CUDA via
-`nvidia-smi`, else CPU — and offloads every layer to the GPU (`-ngl 999`) when one
-is present. `llama-server` downloads the GGUF on first run (`-hf`) and serves it
-under `--alias <key>`, so the engine binds to a stable id (`openai/<key>`) through
-the same LiteLLM transport. Bind a tier to a local model with a qualified key:
+Bind a tier to a local model with a qualified key:
 
 ```yaml
 profiles:
-  tiny: { endpoint: "llamacpp:nemotron-3-nano-4b", temperature: 0.7, max_tokens: 192 }
+  tiny: { endpoint: "local:Qwen/Qwen2.5-3B-Instruct", temperature: 0.7, max_tokens: 192 }
 ```
 
 ### Real cost → Governor
@@ -182,9 +194,9 @@ everything on the big model.
 - `src/core/registry.py` — `Registry.from_world()` (a UI/LLM-composed run on the same path)
 - `src/models/litellm_provider.py` — `LiteLLMProvider` (live transport, real cost)
 - `src/models/modal_catalogue.py` — engine view of the catalogue (key → binding)
-- `src/models/inference.py` — unified backend registry (Modal · HF · llama.cpp); qualified keys
-- `src/models/llamacpp_catalogue.py` — local GGUF catalogue (key → binding)
-- `src/models/llamacpp_server.py` — `llama-server` launcher: GPU detection + command building
+- `src/models/inference.py` — unified backend registry (Modal · HF · Local GPU); qualified keys
+- `src/models/local_catalogue.py` — local model catalogue + capability gate (`has_credentials`)
+- `src/models/local_provider.py` — `LocalTransformersProvider`: in-process `@spaces.GPU` inference
 - `src/core/manifest.py` — `resolve_model()` (env → catalogue default)
 - `src/core/registry.py` — `build_router()`, `_resolve_model_endpoints()`, `_expand_env()`
 - `src/models/provider.py` — `ModelProvider.last_usage`, `estimate_tokens()`

docs/architecture/next-steps/arena-roadmap.md

@@ -24,7 +24,7 @@ drift from the log.
 | Competitive scenarios | ◐ only The Steeped is a real game; Open Table and Oracle Grove have **no judge at all** | `config/scenarios/*.yaml` |
 | Leaderboard / history UI | ❌ none | `src/ui/fishbowl/` |
 | Commentator | ❌ none (narrator feed is a transcript, not commentary) | `src/ui/fishbowl/show.py` |
-| Models | ◐ Modal vLLM (Nemotron 4B/30B/14B, MiniCPM 8B + o-4.5, Gemma 4 12B/26B), HF serverless (Arch-Router-1.5B), stub. No llama.cpp, no fine-tune, no gpt-oss | `modal/catalogue.py:144-296`, `src/models/hf_catalogue.py:71-85` |
+| Models | ◐ Modal vLLM (Nemotron 4B/30B/14B, MiniCPM 8B + o-4.5, Gemma 4 12B/26B), HF serverless (Arch-Router-1.5B), in-process Local GPU (Qwen2.5-3B/7B, MiniCPM4.1-8B — ADR-0033), stub. No fine-tune, no gpt-oss | `modal/catalogue.py:144-296`, `src/models/hf_catalogue.py:71-85`, `src/models/local_catalogue.py` |
 
 ---
 
@@ -177,10 +177,13 @@ engine change:
    OpenAI open model is in the catalogue. Add `gpt-oss-20b` (vLLM serves it) and
    make it the `strong`-tier default for the live demo path. Highest strategic
    priority in this workstream.
-2. **llama.cpp backend** (🦙 *Llama Champion*) — third backend in
-   `src/models/inference.py`'s registry: `LLAMA_CPP_BASE_URL` env var, a tiny
-   static catalogue (`llamacpp:` key prefix), OpenAI-compatible transport
-   already handled by LiteLLM. ~1 day, unlocks a badge.
+2. **Local GPU backend** (✅ *shipped* — ADR-0033) — third backend in
+   `src/models/inference.py`'s registry: in-process `transformers` via
+   `@spaces.GPU` (`local:` key prefix, `LOCAL_INFERENCE=1` opt-in). Works on
+   ZeroGPU, dedicated-GPU Spaces, and local CUDA boxes. Supports the Community-
+   Choice / Tiny-Titan / OpenBMB lanes. **Note:** this replaces the earlier
+   llama.cpp design (ADR-0032) — the 🦙 Llama Champion runtime badge is not
+   pursued; on-device inference ships instead as the in-process GPU path.
 3. **Broaden the Modal catalogue** with strong ≤32B chat models so arena seats
    differ meaningfully: Qwen3 (4B/8B/14B/32B), Llama-3.1-8B-Instruct,
    Phi-4-14B, Mistral-Small-24B. Prefer models vLLM serves without
@@ -305,7 +308,8 @@ Phase D (reach)       W4 models (gpt-oss → llama.cpp → catalogue → fine-tu
   no-API-key stub path kept fully working.
 - Prize coverage per phase: A unlocks 📡 trace export; B unlocks the Best
   Agent/Best Demo arena story; C is the delight criterion + 🐜 Tiny Titan; D is
-  OpenAI track + 🦙 + 🎯.
+  OpenAI track + 🎯 (Local GPU / on-device inference already ships for Community-
+  Choice, OpenBMB, and Tiny-Titan lanes — 🦙 Llama Champion is not pursued).
 
 ## Beyond (post-arena ideas, in rough value order)
 

docs/runbook-live-mode.md

@@ -74,13 +74,33 @@ catalogue key in `config/models.yaml` (source of truth: `modal/catalogue.py`).
 ### Option B — one explicit endpoint
 
 Point every profile at a single OpenAI-compatible base URL (one Modal-served
-model, or a local llama.cpp / vLLM box):
+model, or any other OpenAI-compatible endpoint):
 
 ```ini
 # .env
 MODAL_LLM_BASE_URL=https://your-workspace--google-llms-gemma-4-12b.modal.run/v1
 ```
 
+### Option C — Local GPU (in-process transformers)
+
+Run inference in-process on the host's own GPU — no server to launch, no token.
+The engine uses `LocalTransformersProvider` behind a `@spaces.GPU` function,
+which works on ZeroGPU Spaces, dedicated-GPU Spaces (T4/L4/L40S/A100), and local
+CUDA boxes. On a CPU-only host the call is a no-op and the stub remains active.
+
+**On a CUDA box or dedicated-GPU Space:**
+
+```ini
+# .env
+LOCAL_INFERENCE=1
+```
+
+Then pick **"Local GPU"** in the Lab's backend radio. On a ZeroGPU Space,
+`SPACES_ZERO_GPU` is set automatically — no `.env` change needed, just select the
+backend in the UI. See
+[`docs/architecture/model-routing.md`](architecture/model-routing.md) for the
+full model list and per-tier config syntax (`local:<repo_id>`).
+
 ### Per-profile overrides
 
 Highest priority. Override the model string bound to any profile — the cheapest

src/models/local_provider.py

@@ -14,14 +14,16 @@ one decorated ``_generate`` covers every flavour:
   * **Dedicated GPU / local CUDA** — the decorator is a passthrough; the model runs on
     the persistent GPU.
 
-**Why the model loads in the parent, not inside ``@spaces.GPU``.** On ZeroGPU each call
-forks a GPU worker that inherits the parent's already-loaded model (CUDA is *emulated* in
-the parent, materialised on the real GPU inside the call). Loading inside the decorated
-function would reload the weights on every call — the HF docs call this out as
-"significantly less efficient". So :meth:`complete` warms a module-level cache in the
-parent first (lazily, on first use — never at app boot), and the decorated ``_generate``
-only runs the forward pass. On a dedicated GPU the same cache simply keeps the model
-resident across calls.
+**Where the weights load vs. where CUDA is touched.** ZeroGPU grants a real GPU *only*
+for the duration of a ``@spaces.GPU`` call (each call forks a worker); the parent process
+never gets one, and any low-level CUDA init outside such a call — including a lazy
+``.to("cuda")`` at request time, which ZeroGPU's startup hook does not capture — trips the
+fork guard and kills the process. So the split is: :meth:`complete` warms a module-level
+**CPU** cache in the parent first (lazily, on first use — never at app boot, never on
+CUDA), and the decorated ``_generate`` moves those weights onto the GPU **inside the
+granted window** and runs the forward pass. The forked worker inherits the parent's CPU
+weights, so it pays a host→device copy, not a disk reload; on a dedicated GPU the cached
+module simply stays resident across calls (the move is a no-op after the first).
 
 Heavy imports (``torch`` / ``transformers``) are lazy — confined to the functions that
 need them — so importing this module never initialises CUDA (which would trip ZeroGPU's
@@ -49,18 +51,20 @@ _LOADED: dict[str, tuple] = {}
 
 
 def _ensure_loaded(repo_id: str, trust_remote_code: bool) -> tuple:
-    """Load (once, cached) the tokenizer + model for *repo_id*, placed on CUDA.
-
-    Called from :meth:`LocalTransformersProvider.complete` in the parent process so the
-    placement happens under ZeroGPU's CUDA emulation (or directly on a dedicated GPU) and
-    every later ``@spaces.GPU`` call inherits it. ``dtype="auto"`` lets transformers pick
-    the weights' native precision; we fall back to the legacy ``torch_dtype`` kwarg name
-    for older transformers, and to CPU when no CUDA is present (the gate normally prevents
-    that, but it keeps the path honest).
+    """Load (once, cached) the tokenizer + model for *repo_id* **on CPU**.
+
+    Called from :meth:`LocalTransformersProvider.complete` in the parent process to warm
+    the weights in host RAM. It deliberately **never touches CUDA**: under ZeroGPU the
+    parent process gets no GPU, and any low-level CUDA init outside a ``@spaces.GPU`` call
+    (a lazy ``.to("cuda")`` at request time is not captured by ZeroGPU's startup hook)
+    trips the fork guard and kills the process. The CPU→GPU transfer happens inside the
+    decorated :func:`_generate`, where a real GPU is granted; the forked worker inherits
+    these CPU weights, so it pays a device copy, not a disk reload. ``dtype="auto"`` lets
+    transformers pick the weights' native precision (fallback to the legacy ``torch_dtype``
+    kwarg name on older transformers).
     """
     if repo_id in _LOADED:
         return _LOADED[repo_id]
-    import torch
     from transformers import AutoModelForCausalLM, AutoTokenizer
 
     tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=trust_remote_code)
@@ -68,8 +72,6 @@ def _ensure_loaded(repo_id: str, trust_remote_code: bool) -> tuple:
         model = AutoModelForCausalLM.from_pretrained(repo_id, dtype="auto", trust_remote_code=trust_remote_code)
     except TypeError:  # pragma: no cover - older transformers use the torch_dtype kwarg name
         model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto", trust_remote_code=trust_remote_code)
-    if torch.cuda.is_available():
-        model = model.to("cuda")
     model.eval()
     _LOADED[repo_id] = (tokenizer, model)
     return _LOADED[repo_id]
@@ -90,13 +92,19 @@ def _generate(repo_id, trust_remote_code, system, prompt, max_new_tokens, temper
     """Run one chat completion on the GPU; return ``(text, prompt_tokens, completion_tokens)``.
 
     Module-level and decorated so ZeroGPU registers it and grants a GPU for the call. The
-    model is fetched from the parent-warmed cache (a hit — never a reload here); only the
-    input tensors are built and moved to the device inside the GPU window.
+    model weights are fetched from the parent-warmed CPU cache (a hit — never a disk reload
+    here) and moved onto the device **inside this GPU window** — the only place ZeroGPU
+    permits CUDA init. On ZeroGPU the forked worker inherits CPU weights and pays one
+    device copy per call; on a dedicated GPU the cached module is already resident after
+    the first call, so the move is a no-op. Input tensors are built and placed on the same
+    device.
     """
     import torch
 
     tokenizer, model = _ensure_loaded(repo_id, trust_remote_code)
-    device = "cuda" if torch.cuda.is_available() else "cpu"
+    if torch.cuda.is_available():
+        model = model.to("cuda")
+    device = next(model.parameters()).device
     messages = [{"role": "system", "content": system}, {"role": "user", "content": prompt}]
     inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(device)
     do_sample = temperature and float(temperature) > 0

src/ui/fishbowl/assets/styles.css

@@ -939,7 +939,7 @@ footer { display: none !important; }
 .fishbowl.fb-split .splitview { padding: 6px 0 0; max-width: 100%; }
 
 /* ---- FEED pane: read like a live transcript, comfortable scroll ---- */
-.fishbowl.fb-feed .feed { padding: 4px 4px 8px; max-height: 70vh; }
+.fishbowl.fb-feed .feed { padding: 4px 4px 8px; max-height: 50vh; }
 .fishbowl .fe.narr p { font-size: 13px; }
 
 /* ---- NARRATOR feed entrance is already animated (feIn); add a hover tint ---- */

tests/test_local_backend.py

@@ -155,3 +155,45 @@ def test_provider_resolves_trust_remote_code_from_catalogue():
     assert LocalTransformersProvider(model="Qwen/Qwen2.5-3B-Instruct")._trust_remote_code() is False
     # An off-catalogue repo defaults to the safe choice.
     assert LocalTransformersProvider(model="some/random-repo")._trust_remote_code() is False
+
+
+# ── ZeroGPU contract: CUDA only inside @spaces.GPU, never in the parent ───────────────
+# Regression guard for the production crash "Low-level CUDA init (torch._C._cuda_init)
+# reached … ZeroGPU's emulation did not intercept": the parent process gets no GPU, so any
+# CUDA placement outside the @spaces.GPU window (a lazy .to("cuda") at request time) kills
+# the worker. The forward pass can only be exercised with a GPU + weights (integration),
+# so we pin the *structural* invariant — where CUDA may be touched — by source contract.
+
+
+def test_parent_loader_never_initialises_cuda():
+    import ast
+    import inspect
+
+    from src.models import local_provider
+
+    # _ensure_loaded runs in the parent (warm CPU cache, inherited by the fork). It must
+    # not perform any CUDA operation — placement happens later, inside the decorated
+    # function. Check the executable body with the docstring stripped (the docstring
+    # explains the invariant in prose, so it legitimately mentions CUDA); the dangerous
+    # ops are the device move and any torch.cuda.* call.
+    fn = ast.parse(inspect.getsource(local_provider._ensure_loaded)).body[0]
+    if ast.get_docstring(fn):
+        fn.body = fn.body[1:]
+    code = ast.unparse(fn)
+    assert 'to("cuda")' not in code
+    assert "torch.cuda" not in code
+    assert ".cuda(" not in code
+
+
+def test_gpu_transfer_lives_inside_the_spaces_gpu_function():
+    from pathlib import Path
+
+    from src.models import local_provider
+
+    # _generate is wrapped by @spaces.GPU, so read the module source and isolate its block.
+    module_src = Path(local_provider.__file__).read_text()
+    gen_block = module_src.split("def _generate(", 1)[1].split("\ndef ", 1)[0]
+    # The CPU→GPU move is here (the one place ZeroGPU grants a device)…
+    assert '.to("cuda")' in gen_block
+    # …and the function carries the decorator the platform registers.
+    assert "@spaces.GPU" in module_src.split("def _generate(", 1)[0].rsplit("\n\n", 1)[-1]