Revise ollama & hf deployment to llama.cpp

README.md

@@ -10,6 +10,14 @@ app_file: app.py
 pinned: false
 short_description: Tell a story. Watch its feelings take shape.
 license: mit
+tags:
+  - track:wood
+  - sponsor:openbmb
+  - sponsor:modal
+  - achievement:offgrid
+  - achievement:offbrand
+  - achievement:llama
+  - achievement:fieldnotes
 ---
 
 # The Shape of Words
@@ -33,25 +41,23 @@ custom HTML/JS frontend gets queuing, streaming, and Hugging Face Spaces hosting
 
 ```
 ┌─────────────────────────────────────────────────────────────┐
-│  Browser — static/index.html (custom HTML/JS, no build step) │
-│   • renderer · mappings · layout · sound · share card · UI   │
-│   • all rendering is client-side (JS port of engine/)        │
-│   • calls the backend for MODEL JUDGMENT only                │
+│ Browser — static/index.html (custom HTML/JS, no build step) │
+│  • renderer · mappings · layout · sound · share card · UI   │
+│  • all rendering is client-side (JS port of engine/)        │
+│  • calls the backend for MODEL JUDGMENT only                │
 └───────────────┬─────────────────────────────────────────────┘
                 │  /gradio_api/call/<name>   (queued, SSE)
 ┌───────────────▼─────────────────────────────────────────────┐
-│  app.py — gradio.Server                                      │
-│   @app.api  judge_beat · judge_beat_segmented ·              │
-│             continue_story · reveal · title_story · paint    │
-│   @app.get "/"  serves the frontend                          │
+│ app.py — gradio.Server                                      │
+│  @app.api  judge_beat · judge_beat_segmented ·              │
+│            continue_story · reveal · title_story · paint    │
+│  @app.get "/"  serves the frontend                          │
 └───────┬──────────────────────────────────────┬──────────────┘
-        │ model/backend.py                      │ model/painter.py
-        │  STORY_SHAPES_BACKEND                 │  STORY_SHAPES_PAINT_BACKEND
-        ▼                                       ▼
+        │ model/backend.py                     │ model/painter.py
+        │  STORY_SHAPES_BACKEND                │  STORY_SHAPES_PAINT_BACKEND
+        ▼                                      ▼
   llamacpp  GGUF in-process (llama.cpp)   modal       Modal endpoint (FLUX.2 Klein)
   modal_llm Modal endpoint (transformers) flux_local  in-process diffusers
-  ollama    local Ollama daemon
-  hf        transformers in-process (Qwen3)
 ```
 
 The **model only judges affect** (valence / arousal / dominance + a few flags +
@@ -61,42 +67,32 @@ in perception research (see the spec's references).
 
 ---
 
-## Run locally (Ollama + Qwen3-8B)
+## Run locally (llama.cpp + MiniCPM4.1-8B)
 
-**1. Install Ollama** — https://ollama.com/download
+The default backend runs the LLM in-process via `llama-cpp-python` — no daemon,
+no separate model server.
 
-**2. Pull the model** (once):
-```bash
-ollama pull qwen3:8b
-```
-
-**3. Make sure Ollama is running** (it usually runs as a background service; if
-not: `ollama serve`).
-
-**4. Install Python deps and launch:**
 ```bash
 pip install -r requirements.txt
-python app.py
+python app.py            # serves http://localhost:7860
 ```
 
-**5. Open** http://localhost:7860
+On first run it pulls a quantized GGUF (default `openbmb/MiniCPM4.1-8B-GGUF`,
+Q4_K_M, ~5 GB). With a GPU it offloads all layers automatically; **CPU-only?**
+set `STORY_SHAPES_LLAMACPP_GPU_LAYERS=0`. Then open http://localhost:7860.
 
 Config via env vars (all optional):
 | var | default | meaning |
 |-----|---------|---------|
-| `STORY_SHAPES_BACKEND` | `ollama` | `llamacpp` · `modal_llm` · `ollama` · `hf` |
-| `STORY_SHAPES_MODEL`   | `qwen3:8b` | Ollama model tag (`ollama`) or HF repo id (`hf`) |
-| `OLLAMA_URL`           | `http://localhost:11434` | Ollama host |
-| `PORT`                 | `7860` | server port |
+| `STORY_SHAPES_BACKEND` | `llamacpp` | `llamacpp` (in-process) or `modal_llm` (Modal endpoint) |
+| `STORY_SHAPES_LLAMACPP_REPO` | `openbmb/MiniCPM4.1-8B-GGUF` | GGUF Hub repo |
+| `STORY_SHAPES_LLAMACPP_FILE` | `*Q4_K_M.gguf` | GGUF filename glob |
+| `STORY_SHAPES_LLAMACPP_GPU_LAYERS` | `-1` | layers on GPU (`-1` all, `0` CPU-only) |
+| `PORT` | `7860` | server port |
 
 > The frontend falls back to a built-in keyword **stub** if the backend is
 > unreachable, so `static/index.html` also opens standalone for quick UI testing.
 
-Prefer the **llama.cpp** runtime locally too? Set `STORY_SHAPES_BACKEND=llamacpp`
-— it pulls a quantized GGUF (default `openbmb/MiniCPM4.1-8B-GGUF`, Q4_K_M) and
-runs in-process via `llama-cpp-python`, no Ollama daemon needed. Set
-`STORY_SHAPES_LLAMACPP_GPU_LAYERS=0` for CPU-only.
-
 ---
 
 ## Deploy to a Hugging Face Space
@@ -126,12 +122,11 @@ ZeroGPU's per-request quota attribution.
 | `STORY_SHAPES_PAINT_BACKEND` | `modal` | painter on a Modal GPU (`modal deploy modal_painter.py`) |
 | `STORY_SHAPES_PAINT_MODAL_URL` | *(from `modal deploy`)* | the printed `modal_painter` URL |
 
-> **Why not run MiniCPM4.1 in-process under the `hf` backend?** gradio 6 forces
-> `transformers ≥ 5.0`, and MiniCPM4.1's `trust_remote_code` modeling code is
-> incompatible with transformers 5.x (it loads with a shim but crashes during
-> generation). So MiniCPM4.1 runs via **llama.cpp** (no transformers) or on
-> **Modal** with transformers pinned to 4.x. The `hf` backend still works for
-> Qwen3, which is transformers-5-native.
+> **Why not run MiniCPM4.1 in-process under plain `transformers`?** gradio 6
+> forces `transformers ≥ 5.0`, and MiniCPM4.1's `trust_remote_code` modeling
+> code is incompatible with transformers 5.x (it loads with a shim but crashes
+> during generation). That's exactly why the LLM runs via **llama.cpp** (no
+> transformers at all) or on **Modal** with transformers pinned to 4.x.
 
 See `requirements.txt` for the llama.cpp install (a prebuilt CUDA wheel, with a
 one-comment CPU-fallback toggle) and the diffusers-from-source line FLUX needs.
@@ -148,7 +143,7 @@ engine/                deterministic core (also mirrored in the frontend JS)
   renderer.py          geometry -> shape points
   scorer.py            zone-based puzzle scoring (+ band + hint)
 model/
-  backend.py           LLM abstraction: llamacpp / modal_llm / ollama / hf
+  backend.py           LLM abstraction: llamacpp / modal_llm
   painter.py           painting abstraction: modal / flux_local
 modal_llm.py           Modal GPU endpoint for the LLM (deploy once)
 modal_painter.py       Modal GPU endpoint for FLUX.2 Klein (deploy once)
@@ -173,8 +168,8 @@ docs/
 - ✅ **Origin tooltip**: hover/tap a shape or layer row to see its source beat.
 - ✅ **Painting step (FLUX.2 Klein img2img)**: "Paint this"; free style field +
   preset styles + 🎲 Surprise me; randomized seed; `modal` or `flux_local`.
-- ✅ **LLM backends**: `llamacpp` (GGUF, Llama Champion), `modal_llm`, `ollama`,
-  `hf` (Qwen3). All produce schema-constrained JSON.
+- ✅ **LLM backends**: `llamacpp` (GGUF, Llama Champion) and `modal_llm`. Both
+  produce schema-constrained JSON.
 - ✅ **Felt-quality eval** (`eval_felt_quality.py`): A/B models on a fixed set.
 - ✅ **HF Space deployment** (gradio 6.18, `gr.Server`).
 - ⬜ Puzzle mode UI (engine + scorer exist).

app.py

@@ -3,15 +3,17 @@ app.py — Story → Shapes backend, built on gradio.Server (FastAPI + Gradio en
 
 Serves the custom HTML/JS frontend at "/" and exposes JSON API endpoints the
 frontend calls. The model judgment (affect) comes from the model backend
-(Ollama locally / HF on a Space); the deterministic scoring lives here; geometry
-+ color rendering happen client-side (the frontend has the ported renderer).
+(llama.cpp in-process, or a Modal GPU endpoint); the deterministic scoring lives
+here; geometry + color rendering happen client-side (the frontend has the
+ported renderer).
 
-Run locally:
-    ollama serve           # in one terminal (if not already running)
-    ollama pull qwen3:8b   # once
-    python app.py          # serves http://localhost:7860
+Run locally (llama.cpp, the default backend):
+    pip install -r requirements.txt
+    python app.py          # serves http://localhost:7860 (pulls the GGUF on first run)
+    # CPU-only? set STORY_SHAPES_LLAMACPP_GPU_LAYERS=0
 
-On a HF Space: set STORY_SHAPES_BACKEND=hf and wire model/backend.py:_hf_chat.
+On a HF Space: keep STORY_SHAPES_BACKEND=llamacpp (in-process GGUF) or set
+modal_llm + STORY_SHAPES_LLM_MODAL_URL to offload the LLM to Modal.
 """
 import os
 from gradio import Server

app_spec.md

@@ -332,14 +332,11 @@ app.py — gradio.Server  (satisfies Off-Brand badge; gradio 6.x)
             · title_story · paint   (each concurrency_limit=1)
          │
   model/backend.py  ← STORY_SHAPES_BACKEND
-    "llamacpp"  →  GGUF in-process via llama-cpp-python  ← recommended on-Space
+    "llamacpp"  →  GGUF in-process via llama-cpp-python  ← default; local & on-Space
                   GBNF grammar from JSON schema → guaranteed-valid JSON.
                   No transformers; ~5GB at Q4_K_M. Earns Llama Champion.
     "modal_llm" →  HTTP POST to a Modal GPU endpoint (modal_llm.py)
                   transformers pinned to 4.x there, where MiniCPM4.1 works.
-    "ollama"    →  Ollama HTTP (local dev); JSON-schema `format`.
-    "hf"        →  transformers in-process. WORKS for Qwen3; does NOT work for
-                  MiniCPM4.1 under transformers 5.x (see §20). Kept for Qwen3.
          │
   model/painter.py  ← STORY_SHAPES_PAINT_BACKEND
     "modal"      →  Modal web endpoint (A10G GPU, FLUX.2 Klein 4B)
@@ -358,7 +355,7 @@ All engine math is mirrored in `static/index.html` (JS port, verified identical
 **Model judgment:**
 - Full story context with per-beat status flags on every call — globally aware pacing.
 - Pacing note branches: "no shape yet → be generous" vs "settled beats → skip."
-- Judgment model: MiniCPM4.1-8B (OpenBMB prize) or Qwen3-8B — swappable via `STORY_SHAPES_MODEL`. Structured output is enforced by the backend's native grammar facility (Ollama `format`, llama.cpp GBNF-from-schema) or, for the prompt-based backends, `_extract_json` + a 3-attempt parse/retry. `<think>` blocks are stripped (grammar-constrained paths suppress them outright).
+- Judgment model: MiniCPM4.1-8B (OpenBMB prize), run via `llamacpp` (GGUF) or `modal_llm`. Structured output is enforced by llama.cpp's GBNF-from-schema grammar (guaranteed-valid JSON) or, for the Modal path, `_extract_json` + a 3-attempt parse/retry. `<think>` blocks are stripped (the grammar path suppresses them outright).
 
 ---
 
@@ -366,8 +363,7 @@ All engine math is mirrored in `static/index.html` (JS port, verified identical
 
 | Variable | Default | Purpose |
 |---|---|---|
-| `STORY_SHAPES_BACKEND` | `ollama` | LLM backend: `llamacpp` · `modal_llm` · `ollama` · `hf` |
-| `STORY_SHAPES_MODEL` | `qwen3:8b` / `Qwen/Qwen3-8B` | model tag or HF repo id (`ollama`/`hf` backends) |
+| `STORY_SHAPES_BACKEND` | `llamacpp` | LLM backend: `llamacpp` or `modal_llm` |
 | `STORY_SHAPES_LLM_MODAL_URL` | *(required for `modal_llm`)* | URL from `modal deploy modal_llm.py` |
 | `STORY_SHAPES_LLAMACPP_REPO` | `openbmb/MiniCPM4.1-8B-GGUF` | GGUF Hub repo (`llamacpp` backend) |
 | `STORY_SHAPES_LLAMACPP_FILE` | `*Q4_K_M.gguf` | GGUF filename glob |
@@ -376,7 +372,6 @@ All engine math is mirrored in `static/index.html` (JS port, verified identical
 | `STORY_SHAPES_PAINT_BACKEND` | `modal` | `modal` or `flux_local` |
 | `STORY_SHAPES_PAINT_MODAL_URL` | *(required for modal paint)* | URL from `modal deploy modal_painter.py` |
 | `STORY_SHAPES_FLUX_MODEL` | `black-forest-labs/FLUX.2-klein-4B` | FLUX repo (`flux_local`) |
-| `OLLAMA_URL` | `http://localhost:11434` | Ollama host (local only) |
 | `PORT` | `7860` | server port |
 
 ---
@@ -402,10 +397,8 @@ All engine math is mirrored in `static/index.html` (JS port, verified identical
 | Share card (canvas + painting, popup) | ✅ | AI title (`title_story`, cached); transparent shapes card |
 | Reveal sound (chimes / tick / pad) + replay | ✅ | Web Audio; pentatonic; Settings toggle |
 | Origin tooltip (shape + layer, hover/touch) | ✅ | shows verbatim source beat |
-| llama.cpp backend (`llamacpp`) | ✅ | GGUF in-process; GBNF from schema; **Llama Champion** |
+| llama.cpp backend (`llamacpp`) | ✅ | default; GGUF in-process; GBNF from schema; **Llama Champion** |
 | Modal LLM backend (`modal_llm`) | ✅ | transformers 4.x on Modal; MiniCPM4.1 verified |
-| Ollama backend (JSON-schema constrained) | ✅ | local dev |
-| HF transformers backend (`hf`) | ✅ | Qwen3 only; MiniCPM4.1 broken on transformers 5.x (§20) |
 | Felt-quality eval harness | ✅ | `eval_felt_quality.py` |
 | `devReveal()` console shortcut | ✅ | dev/testing only |
 | **HF Space deployment** | ✅ | gradio 6.18 (`gr.Server`); LLM via llama.cpp/Modal, painter via Modal |

field_notes.md

@@ -232,14 +232,12 @@ app.py — gradio.Server  (satisfies the Off-Brand badge; gradio 6.x)
   @app.api  judge_beat · judge_beat_segmented · continue_story · reveal
             · title_story · paint   (each concurrency_limit=1)
          │
-  model/backend.py
-    "llamacpp"  →  GGUF in-process via llama-cpp-python  ← recommended on-Space
+  model/backend.py   ← STORY_SHAPES_BACKEND
+    "llamacpp"  →  GGUF in-process via llama-cpp-python  ← default; local & on-Space
                   GBNF grammar from JSON schema → guaranteed-valid JSON.
                   No transformers; ~5GB at Q4_K_M. Earns Llama Champion.
     "modal_llm" →  HTTP POST to a Modal GPU endpoint (modal_llm.py),
                   with transformers pinned to 4.x, where MiniCPM4.1 works.
-    "ollama"    →  Ollama HTTP (local dev); JSON-schema `format`.
-    "hf"        →  transformers in-process. Works for Qwen3; kept for it.
          │
   model/painter.py
     "modal"      →  Modal web endpoint (A10G GPU, FLUX.2 Klein 4B)
@@ -248,9 +246,9 @@ app.py — gradio.Server  (satisfies the Off-Brand badge; gradio 6.x)
 
 Both Modal services (`modal_llm.py`, `modal_painter.py`) are deployed once with `modal deploy`; they keep one container warm (`scaledown_window`) and lazy-load weights on the first call.
 
-**Why four LLM backends?** This was the hardest part of shipping. `gr.Server` (for the custom frontend) needs gradio 6.x, which forces `huggingface-hub ≥ 1.2`, which forces `transformers ≥ 5.0`. But MiniCPM4.1-8B loads via `trust_remote_code`, and its remote code is incompatible with transformers 5.x: loading needs a shim, and generation then still crashes deep in attention. The two clean escapes are to run it on **Modal** with transformers pinned to 4.x, or — better — to run it through **llama.cpp** as a quantized GGUF, which uses no `transformers` at all and is small enough (~5 GB) to sit beside FLUX on a single 24 GB GPU. The llama.cpp route also earns the Llama Champion badge.
+**Why llama.cpp, not in-process `transformers`?** This was the hardest part of shipping. `gr.Server` (for the custom frontend) needs gradio 6.x, which forces `huggingface-hub ≥ 1.2`, which forces `transformers ≥ 5.0`. But MiniCPM4.1-8B loads via `trust_remote_code`, and its remote code is incompatible with transformers 5.x: loading needs a shim, and generation then still crashes deep in attention. The two clean escapes are to run it on **Modal** with transformers pinned to 4.x, or — better — to run it through **llama.cpp** as a quantized GGUF, which uses no `transformers` at all and is small enough (~5 GB) to sit beside FLUX on a single 24 GB GPU. The llama.cpp route also earns the Llama Champion badge.
 
-**The deployment journey.** The four backends weren't designed up front — they're the fossil record of getting this to run on the hardware I actually had (a laptop with **16 GB RAM / 8 GB VRAM**, nowhere near enough for an 8B LLM *and* a 4B diffusion model at once):
+**The deployment journey.** Those two backends weren't the first attempt — they're what survived after a path of dead ends, driven by the hardware I actually had (a laptop with **16 GB RAM / 8 GB VRAM**, nowhere near enough for an 8B LLM *and* a 4B diffusion model at once):
 
 1. **Qwen3-8B on Ollama (local) + FLUX.2 Klein on Modal.** The first working setup. The LLM ran locally through Ollama (which is itself llama.cpp under the hood); the painter — far too heavy for 8 GB — was pushed to a Modal GPU endpoint. This split kept the loop fast to iterate on while the expensive image step lived in the cloud.
 2. **MiniCPM4.1-8B via llama.cpp + FLUX.2 Klein on Modal.** To target the OpenBMB prize I swapped the judge to MiniCPM4.1-8B, and moved to running it directly on **llama.cpp** (a quantized GGUF via `llama-cpp-python`) rather than Ollama — which also claims the Llama Champion badge. The painter is still on Modal due to previous RAM constraints.

modal_llm.py

@@ -57,10 +57,7 @@ _NO_THINK_SUFFIX = [{"role": "assistant", "content": "<think>\n\n</think>\n\n"}]
 class LLM:
     @modal.enter()
     def load(self):
-        # import transformers.utils.import_utils
-        # setattr(transformers.utils.import_utils, 'is_torch_fx_available', lambda: True)
-
-        from transformers import pipeline
+         from transformers import pipeline
         import torch
         print(f"Loading {MODEL_ID}…")
         self.pipe = pipeline(

model/backend.py

@@ -1,31 +1,28 @@
 """
-...
-Backends (STORY_SHAPES_BACKEND env var, default "ollama"):
-  - "ollama"  : local dev. HTTP to a running Ollama daemon. Uses Ollama's
-                structured-output `format` for guaranteed-valid JSON.
-  - "hf"      : HF Space / any machine with a GPU + transformers. Loads the
-                model in-process. Uses prompt-based JSON + parse+retry for
-                structured output. Thinking disabled via the Qwen3 assistant
-                prefix trick (<think>\\n\\n</think>).
-
-Qwen3 note: thinking is disabled in both backends (Ollama: /no_think header;
-HF: empty-think assistant prefix). This keeps latency low and JSON clean.
+model/backend.py — language-model abstraction for the affect judgments.
+
+Backends (STORY_SHAPES_BACKEND env var, default "llamacpp"):
+  - "llamacpp"  : recommended; runs locally and on a GPU Space. A quantized GGUF
+                  run in-process via llama-cpp-python (the llama.cpp runtime) —
+                  no `transformers`. Structured output is enforced by a GBNF
+                  grammar built from the JSON schema, so JSON is guaranteed
+                  valid. ~5 GB at Q4_K_M, so it fits beside the FLUX painter on
+                  one 24 GB GPU.
+  - "modal_llm" : HTTP POST to a deployed Modal GPU endpoint (modal_llm.py),
+                  where `transformers` is pinned to 4.x (the version MiniCPM4.1
+                  needs). Use when the Space itself has no GPU.
+
+(The painter is independent — see model/painter.py / STORY_SHAPES_PAINT_BACKEND.)
+
+MiniCPM4.1-8B uses the <think> tag convention; thinking is suppressed by the
+grammar (JSON paths) or a /no_think system line (prose paths), and stripped
+defensively either way.
 """
 import os, json, re, urllib.request
 
-# ZeroGPU: decorate GPU-using functions with @_GPU so HF Spaces can allocate
-# the GPU on demand. Falls back to a no-op when running locally (no `spaces` pkg).
+# ZeroGPU: @spaces.GPU lets a GPU Space allocate the GPU on demand for the
+# in-process llama.cpp calls. On dedicated/local hardware it's a passthrough.
 import spaces
-# try:
-#     import spaces
-#     # _GPU = spaces.GPU
-# except ImportError:
-#     def _GPU(fn=None, *, duration=60):
-#         return fn if fn is not None else (lambda f: f)
-
-# is_torch_fx_available fix
-import transformers.utils.import_utils
-setattr(transformers.utils.import_utils, 'is_torch_fx_available', lambda: True)
 
 import logging
 logging.basicConfig(
@@ -35,34 +32,23 @@ logging.basicConfig(
 )
 log = logging.getLogger("story_shapes")
 
-# BACKEND = os.environ.get("STORY_SHAPES_BACKEND", "ollama")
-BACKEND = os.environ.get("STORY_SHAPES_BACKEND", "modal_llm")
-OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
+BACKEND = os.environ.get("STORY_SHAPES_BACKEND", "llamacpp")
 LLM_URL = os.environ.get("STORY_SHAPES_LLM_MODAL_URL", "")   # Modal LLM endpoint URL
-# Default: Qwen3-8B. Swap to MiniCPM4.1-8B for the OpenBMB prize by setting
-#   STORY_SHAPES_MODEL=openbmb/MiniCPM4.1-8B  (HF Space, BACKEND=hf)
-#   STORY_SHAPES_MODEL=openbmb/minicpm4.1      (Ollama local)
-# MiniCPM4.1-8B uses the same <think> tag convention as Qwen3, so the existing
-# empty-think assistant prefix (see _NO_THINK_SUFFIX) suppresses thinking for both.
-# Requires transformers>=4.56 (already pinned in requirements.txt).
-MODEL = os.environ.get(
-    "STORY_SHAPES_MODEL",
-    "qwen3:8b" if BACKEND == "ollama" else "Qwen/Qwen3-8B"
-)
 
-# llama.cpp backend (STORY_SHAPES_BACKEND=llamacpp): runs a quantized GGUF
-# in-process via llama-cpp-python. Q4_K_M of MiniCPM4.1-8B is ~5 GB VRAM, so it
-# coexists with the FLUX painter on a single 24 GB L4. Does NOT use transformers
-# (so it sidesteps the transformers-5 / MiniCPM remote-code incompatibility).
+# llama.cpp backend: a quantized GGUF run in-process via llama-cpp-python.
 LLAMACPP_REPO = os.environ.get("STORY_SHAPES_LLAMACPP_REPO", "openbmb/MiniCPM4.1-8B-GGUF")
 LLAMACPP_FILE = os.environ.get("STORY_SHAPES_LLAMACPP_FILE", "*Q4_K_M.gguf")
 LLAMACPP_CTX  = int(os.environ.get("STORY_SHAPES_LLAMACPP_CTX", "4096"))
 # -1 offloads all layers to GPU; set 0 for CPU-only.
 LLAMACPP_GPU_LAYERS = int(os.environ.get("STORY_SHAPES_LLAMACPP_GPU_LAYERS", "-1"))
 
+# Active model identifier, for /health and the eval harness (informational —
+# the Modal endpoint pins its own model id internally).
+MODEL = os.environ.get("STORY_SHAPES_MODEL", LLAMACPP_REPO)
+
 PROMPT_MARKDOWN_DIVIDER = "================================================================================"
 
-# ---- JSON schemas (mirror the GBNF grammars; used as Ollama `format`) ----
+# ---- JSON schemas (source for the llama.cpp GBNF / schema-constrained output) ----
 UNIT = {"type": "number", "minimum": 0, "maximum": 1}
 MATERIAL_ENUM = {"type": "string", "enum": ["paper", "ink", "glass", "enamel", "chalk", "metal"]}
 CORE_SCHEMA = {
@@ -139,102 +125,19 @@ def _load_prompt(name):
     with open(os.path.join(here, "prompts", name), encoding="utf-8") as f:
         return f.read()
 
-# ---------------------------------------------------------------------------
-# Ollama backend
-# ---------------------------------------------------------------------------
-def _ollama_chat(system, user, schema):
-    body = {
-        "model": MODEL,
-        "messages": [
-            {"role": "system", "content": system + "\n/no_think"},  # Qwen3: thinking off
-            {"role": "user", "content": user},
-        ],
-        "stream": False,
-        "format": schema,            # structured output -> guaranteed-valid JSON
-        "options": {"temperature": 0.6, "top_p": 0.9},
-    }
-    req = urllib.request.Request(
-        OLLAMA_URL + "/api/chat",
-        data=json.dumps(body).encode(),
-        headers={"Content-Type": "application/json"},
-    )
-    with urllib.request.urlopen(req, timeout=120) as r:
-        resp = json.loads(r.read())
-    content = resp["message"]["content"].strip()
-    log.info("OLLAMA raw  <- %s", content[:500])
-    # strip any stray <think></think> if the model emitted one despite /no_think
-    if "</think>" in content:
-        content = content.split("</think>", 1)[1].strip()
-    parsed = json.loads(content)
-    log.info("OLLAMA json -> %s", parsed)
-    return parsed
-
 # extract just the system prompt block from a prompt .md (between the SYSTEM markers)
 def _system_block(md, marker="SYSTEM PROMPT"):
     if marker in md:
         after = md.split(marker, 1)[1]
-        # cut at the next "====" divider
-        # return after.split("====", 1)[0].strip().lstrip("=").strip()
         return after.split(PROMPT_MARKDOWN_DIVIDER, 2)[1].strip()
     return md
 
 # ---------------------------------------------------------------------------
-# HF backend — transformers in-process (HF Space / any GPU machine)
+# Shared JSON helpers (used by both backends)
 # ---------------------------------------------------------------------------
-_hf_pipe = None   # lazy-loaded pipeline
-
-def _get_hf_pipe():
-    global _hf_pipe
-    if _hf_pipe is not None:
-        return _hf_pipe
-    from transformers import pipeline, GenerationConfig
-    import torch
-    log.info("loading HF model %s …", MODEL)
-    _hf_pipe = pipeline(
-        "text-generation",
-        model=MODEL,
-        dtype=torch.bfloat16,       # transformers 5.x: dtype, not torch_dtype
-        device_map="auto",
-        trust_remote_code=True,
-    )
-    # Clear the model's generation_config so our per-call GenerationConfig is
-    # the sole source of truth. MiniCPM4.1 ships max_length=20 in its
-    # generation_config.json; if left in place it conflicts with max_new_tokens
-    # and causes a Key/Value sequence-length mismatch in SDPA.
-    _hf_pipe.model.generation_config = GenerationConfig()
-    log.info("HF model loaded.")
-    return _hf_pipe
-
-# Qwen3 thinking-disable: append an assistant message with an empty <think> block.
-# Stateless (one-turn only), strictly prevents thinking tokens per Qwen3 docs.
-_NO_THINK_SUFFIX = [{"role": "assistant", "content": "<think>\n\n</think>\n\n"}]
-
-# @_GPU(duration=120)
-@spaces.GPU(duration=60)
-def _hf_generate(messages, max_new_tokens=512, temperature=0.6) -> str:
-    from transformers import GenerationConfig
-    pipe = _get_hf_pipe()
-    full_messages = messages + _NO_THINK_SUFFIX
-    # Pass a single GenerationConfig rather than mixing kwargs + model's config.
-    # Transformers 5.x deprecates passing both simultaneously.
-    gen_cfg = GenerationConfig(
-        max_new_tokens=max_new_tokens,
-        temperature=temperature,
-        do_sample=temperature > 0,
-    )
-    out = pipe(
-        full_messages,
-        generation_config=gen_cfg,
-        return_full_text=False,
-    )
-    text = out[0]["generated_text"]
-    if "</think>" in text:
-        text = text.split("</think>", 1)[1]
-    return text.strip()
-
 def _extract_json(text: str) -> dict:
     """Extract the first {...} JSON object, handling markdown fences and
-    trailing commas — the three most common model formatting mistakes."""
+    trailing commas — the most common model formatting mistakes."""
     text = re.sub(r"```(?:json)?", "", text).strip()
     m = re.search(r"\{.*\}", text, re.DOTALL)
     if not m:
@@ -268,45 +171,8 @@ def _apply_defaults(parsed: dict, schema: dict) -> dict:
             parsed[k] = prop["enum"][0]   # coerce invalid enum value to first valid
     return parsed
 
-def _hf_chat(system: str, user: str, schema: dict) -> dict:
-    """Call the HF pipeline, parse JSON, validate, retry up to 3 times."""
-    hint = _schema_hint(schema)
-    messages = [
-        {"role": "system", "content": system},
-        {"role": "user",   "content": f"{user}\n\nReturn ONLY a JSON object with keys: {hint}"},
-    ]
-    for attempt in range(3):
-        raw = _hf_generate(messages)
-        log.info("HF raw (attempt %d) <- %s", attempt + 1, raw[:400])
-        try:
-            parsed = _apply_defaults(_extract_json(raw), schema)
-            log.info("HF json -> %s", parsed)
-            return parsed
-        except (ValueError, json.JSONDecodeError, KeyError) as e:
-            log.warning("HF parse attempt %d failed: %s", attempt + 1, e)
-            if attempt == 2:
-                raise RuntimeError(
-                    f"HF backend failed to produce valid JSON after 3 attempts. "
-                    f"Last output: {raw!r}") from e
-
-def _hf_continue_story(story: str) -> str:
-    if story == "":
-        user_prompt = f"Story so far:\n{story}\nStart with 1-3 sentences."
-    else:
-        user_prompt = f"Story so far:\n{story}\nContinue with 1-3 sentences."
-    messages = [
-        {"role": "system", "content": (
-            "You are co-writing a story one story beat at a time. "
-            "Continue with exactly 1-3 sentences that follow naturally. "
-            "If the story is empty, start it by any means; up to you."
-            "Output only the sentence(s), no quotes, no preamble."
-        )},
-        {"role": "user", "content": user_prompt},
-    ]
-    return _hf_generate(messages, max_new_tokens=200, temperature=0.85)
-
 # ---------------------------------------------------------------------------
-# Modal LLM backend — HTTP POST to deployed modal_llm.py endpoint
+# Modal LLM backend — HTTP POST to the deployed modal_llm.py endpoint
 # ---------------------------------------------------------------------------
 def _modal_generate(messages, max_new_tokens=1024, temperature=0.6) -> str:
     if not LLM_URL:
@@ -329,7 +195,7 @@ def _modal_generate(messages, max_new_tokens=1024, temperature=0.6) -> str:
     return resp["text"]
 
 def _modal_llm_chat(system: str, user: str, schema: dict) -> dict:
-    """Like _hf_chat but generates via the Modal LLM endpoint instead of in-process."""
+    """Generate JSON via the Modal LLM endpoint; parse, validate, retry up to 3×."""
     hint = _schema_hint(schema)
     messages = [
         {"role": "system", "content": system},
@@ -358,6 +224,13 @@ def _get_llama():
     global _llama
     if _llama is not None:
         return _llama
+    # Import torch first: it loads the CUDA runtime libraries (libcudart, etc.)
+    # into the process, which the prebuilt llama-cpp-python CUDA wheel's
+    # libllama.so needs to resolve when it loads. Harmless if torch is absent.
+    try:
+        import torch  # noqa: F401
+    except Exception:
+        pass
     from llama_cpp import Llama
     log.info("loading llama.cpp model %s / %s …", LLAMACPP_REPO, LLAMACPP_FILE)
     _llama = Llama.from_pretrained(
@@ -415,13 +288,17 @@ def _chat(system, user, schema):
     log.info("system prompt: %s", system)
     log.info("user prompt: %s", user)
     log.info("output schema: %s", schema)
-    if BACKEND == "ollama":
-        return _ollama_chat(system, user, schema)
     if BACKEND == "modal_llm":
         return _modal_llm_chat(system, user, schema)
-    if BACKEND == "llamacpp":
-        return _llamacpp_chat(system, user, schema)
-    return _hf_chat(system, user, schema)
+    return _llamacpp_chat(system, user, schema)
+
+# Co-writing system prompt, shared by the pass-the-pen continuation.
+_CONTINUE_SYSTEM = (
+    "You are co-writing a story one story beat at a time. "
+    "Continue with exactly 1-3 sentences that follow naturally. "
+    "If the story is empty, start it; up to you. "
+    "Output only the sentence(s), no quotes, no preamble."
+)
 
 # ---------------------------------------------------------------------------
 # Public interface
@@ -487,91 +364,27 @@ def judge_attempt(target_affect, shape_sentence):
 
 def continue_story(story):
     log.info("continue_story (story %d chars)", len(story))
-
-    if BACKEND == "hf":
-        out = _hf_continue_story(story)
-        log.info("continue_story -> %r", out)
-        return out
-
+    prompt = f"Story so far:\n{story}\n{'Continue' if story else 'Start'} with 1-3 sentences."
+    messages = [
+        {"role": "system", "content": _CONTINUE_SYSTEM},
+        {"role": "user", "content": prompt},
+    ]
     if BACKEND == "modal_llm":
-        prompt = f"Story so far:\n{story}\n{'Continue' if story else 'Start'} with 1-3 sentences."
-        messages = [
-            {"role": "system", "content": (
-                "You are co-writing a story one story beat at a time. "
-                "Continue with exactly 1-3 sentences that follow naturally. "
-                "If the story is empty, start it; up to you. "
-                "Output only the sentence(s), no quotes, no preamble."
-            )},
-            {"role": "user", "content": prompt},
-        ]
         out = _modal_generate(messages, max_new_tokens=200, temperature=0.85)
-        if "</think>" in out:
-            out = out.split("</think>", 1)[1].strip()
-        log.info("continue_story (modal_llm) -> %r", out)
-        return out.strip().strip('"')
-
-    if BACKEND == "llamacpp":
-        prompt = f"Story so far:\n{story}\n{'Continue' if story else 'Start'} with 1-3 sentences."
-        messages = [
-            {"role": "system", "content": (
-                "You are co-writing a story one story beat at a time. "
-                "Continue with exactly 1-3 sentences that follow naturally. "
-                "If the story is empty, start it; up to you. "
-                "Output only the sentence(s), no quotes, no preamble."
-            )},
-            {"role": "user", "content": prompt},
-        ]
+    else:  # llamacpp
         out = _llamacpp_generate(messages, max_new_tokens=200, temperature=0.85)
-        log.info("continue_story (llamacpp) -> %r", out)
-        return out.strip().strip('"')
-
-    md = _load_prompt("prompt_passpen.md")
-    system = _system_block(md)
-    if story == "":
-        user = f"Story so far:\n{story}\nStart with 1-3 sentences."
-    else:
-        user = f"Story so far:\n{story}\nContinue with 1-3 sentences."
-    log.info("system prompt: %s", system)
-    log.info("user prompt: %s", user)
-    # prose, not JSON — call without a schema
-    body = {"model": MODEL,
-            "messages": [{"role": "system", "content": system + "\n/no_think"},
-                         {"role": "user", "content": user}],
-            "stream": False, "options": {"temperature": 0.85}}
-    req = urllib.request.Request(OLLAMA_URL + "/api/chat",
-        data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
-    with urllib.request.urlopen(req, timeout=120) as r:
-        resp = json.loads(r.read())
-    out = resp["message"]["content"].strip()
     if "</think>" in out:
         out = out.split("</think>", 1)[1].strip()
-    log.info("continue_story -> %r", out.strip().strip('"'))
+    log.info("continue_story -> %r", out)
     return out.strip().strip('"')
 
 def _generate_prose(system: str, user: str, temperature=0.8, max_new_tokens=200) -> str:
     """Backend-agnostic prose (non-JSON) generation, used by title_story."""
-    if BACKEND == "hf":
-        out = _hf_generate(
-            [{"role": "system", "content": system}, {"role": "user", "content": user}],
-            max_new_tokens=max_new_tokens, temperature=temperature)
-    elif BACKEND == "modal_llm":
-        out = _modal_generate(
-            [{"role": "system", "content": system}, {"role": "user", "content": user}],
-            max_new_tokens=max_new_tokens, temperature=temperature)
-    elif BACKEND == "llamacpp":
-        out = _llamacpp_generate(
-            [{"role": "system", "content": system}, {"role": "user", "content": user}],
-            max_new_tokens=max_new_tokens, temperature=temperature)
-    else:  # ollama
-        body = {"model": MODEL,
-                "messages": [{"role": "system", "content": system + "\n/no_think"},
-                             {"role": "user", "content": user}],
-                "stream": False, "options": {"temperature": temperature}}
-        req = urllib.request.Request(OLLAMA_URL + "/api/chat",
-            data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
-        with urllib.request.urlopen(req, timeout=120) as r:
-            resp = json.loads(r.read())
-        out = resp["message"]["content"].strip()
+    messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
+    if BACKEND == "modal_llm":
+        out = _modal_generate(messages, max_new_tokens=max_new_tokens, temperature=temperature)
+    else:  # llamacpp
+        out = _llamacpp_generate(messages, max_new_tokens=max_new_tokens, temperature=temperature)
     if "</think>" in out:
         out = out.split("</think>", 1)[1].strip()
     return out.strip()

requirements.txt

@@ -32,25 +32,8 @@ diffusers @ git+https://github.com/huggingface/diffusers.git
 # Runs a quantized GGUF (default openbmb/MiniCPM4.1-8B-GGUF Q4_K_M, ~5GB) in
 # process — no transformers, so it avoids the transformers-5/MiniCPM crash and
 # leaves VRAM for the FLUX painter on one 24GB L4.
-#
-# ┌─ PICK EXACTLY ONE BLOCK ────────────────────────────────────────────────┐
-#
-# [A] GPU (default) — prebuilt CUDA-12 wheel + runtime libs. The nvidia-*-cu12
-#     packages provide libcudart.so.12 / libcublas*.so.12, which
-#     model/backend.py preloads before importing llama_cpp (the Space image
-#     doesn't expose CUDA-12 runtime libs on the loader path otherwise).
-#     Env: STORY_SHAPES_LLAMACPP_GPU_LAYERS=-1  (or just leave it unset).
 --extra-index-url https://download.pytorch.org/whl/cu128
 torch==2.8.0
 hf_transfer
 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
 llama-cpp-python>=0.3.0
-
-#
-# [B] CPU FALLBACK — if the CUDA build keeps failing. Comment out the four [A]
-#     lines above, uncomment the one line below, and set the Space variable
-#     STORY_SHAPES_LLAMACPP_GPU_LAYERS=0. Plain CPU wheel, installs everywhere;
-#     the LLM runs on CPU (~10-15s per short JSON judgment), painter still GPU.
-# llama-cpp-python>=0.3.0
-#
-# └─────────────────────────────────────────────────────────────────────────┘