GPU hack modal

README.md

@@ -29,6 +29,29 @@ boss quest you face first.
 Your photo and quest state live **only in your browser** (localStorage); the photo is sent to the
 GPU transiently for generation and is never stored server-side.
 
-**Set the Space hardware to ZeroGPU.** First quest generation cold-downloads the model GGUF
-(~60-90s), then runs fast.
+### GPU backend (ZeroGPU or Modal)
+
+The GPU work (quest JSON + images) runs through a pluggable backend chosen by the
+`FROGQUEST_BACKEND` env var. The frontend, signatures, and behavior are identical either way.
+
+**`zerogpu` (default).** Everything runs in-Space on ZeroGPU.
+- Set the Space hardware to **ZeroGPU**.
+- First quest generation cold-downloads the models, then runs fast.
+- Subject to ZeroGPU daily quotas (anon 2 min, free 5 min, PRO 40 min per day).
+
+**`modal` (opt-in).** The Space runs on CPU-basic and offloads GPU work to [Modal](https://modal.com)
+(dedicated GPU, pay-per-second, no quota). Switching is **two steps** — the env var alone does NOT
+change hardware:
+1. Set the Space hardware to **CPU basic**.
+2. Add a Space **variable** `FROGQUEST_BACKEND=modal`.
+
+One-time Modal setup:
+1. `pip install modal` and `modal token new` (authenticate).
+2. `modal deploy modal_app.py` (deploys the `frogquest` app: an `LLM` class + a `Flux` class).
+3. Add `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` as Space **secrets** so the Space can call Modal.
+4. Optional: set `FROGQUEST_MODAL_FLUX_GPU` before `modal deploy` to pick the image GPU
+   (default `A10G`; `L4` cheaper; `L40S`/`A100-40GB` if it OOMs).
+
+Optional: rename `requirements-modal.txt` → `requirements.txt` on the Space to skip installing the
+heavy GPU libs (torch/diffusers/llama-cpp) and speed up CPU-basic builds.
 

gpu_shared.py

@@ -0,0 +1,136 @@
+"""Backend-agnostic shared source for FrogQuest's GPU work.
+
+This module holds everything that BOTH the in-Space local path (llm.py / images.py running on
+ZeroGPU) and the off-Space Modal path (modal_app.py running on a Modal GPU) need, so the two
+backends can never drift: the model ids/config, the LLM system prompts, the FLUX prompt builders,
+the JSON extractor, and the CUDA-lib preloader.
+
+Hard rule: this file must stay importable on a CPU-only box with NO heavy GPU deps installed.
+Only stdlib + (lazily) PIL/torch are referenced, and torch is imported inside a function. That is
+what lets app.py import the local modules on a CPU-basic Space (FROGQUEST_BACKEND=modal) without
+dragging in torch/diffusers/llama_cpp. The schemas live in schema.py (also dependency-free) and
+are imported directly by both paths — they are NOT duplicated here.
+"""
+from __future__ import annotations
+
+import json
+
+# ----------------------------- model ids / config (shared by both backends) -----------------------------
+
+# Nemotron-3 Nano 4B GGUF (verified June 2026). Q8_0 (~4.3GB, near-fp16); filename is a glob that
+# Llama.from_pretrained resolves to the exact file. (Hard floor Q4 — sub-4-bit degrades this arch.)
+GGUF_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF"
+GGUF_FILE = "*Q8_0*.gguf"
+
+# FLUX.2 [klein] (verified: ungated, Apache 2.0). The "4B" is just the diffusion transformer; the
+# repo also ships a large multimodal text encoder + VAE (~23GB total in bf16).
+MODEL_ID = "black-forest-labs/FLUX.2-klein-4B"
+
+# Image quality/speed knobs. klein is DISTILLED -> 4 steps is the model card's value (more doesn't
+# help). 384 (a multiple of 16) matches the UI's ~380px display; resolution is the only real knob.
+STEPS = 4
+GUIDANCE = 4.0
+MAX_SIDE = 384
+
+# LLM context: full 128k on a big GPU; 16k on a small one (a 128k KV cache won't fit beside FLUX).
+N_CTX = 131072
+N_CTX_SMALL = 16384
+LOW_VRAM_GB = 24  # at/below this, treat the GPU as "small" (T4 = 16GB, A10G/L4 = 24GB)
+
+
+# ----------------------------- LLM prompts -----------------------------
+
+SYSTEM_PROMPT = """You are FrogQuest's quest designer. Convert the user's real to-do list into a themed text-adventure quest log and OUTPUT JSON ONLY - no prose, no markdown, no code.
+
+Apply the "Eat That Frog" method:
+- The FROG = the single most important/hardest task. Mark exactly ONE quest is_frog:true and order it FIRST.
+- Break each big/multi-step goal into an ordered chain of smaller quests sharing one goal_group label; keep simple to-dos as standalone quests.
+- Add 1-3 bonus self-care quests (type:"bonus") such as meditate 5 min, exercise 20 min, digital detox 1 hr. They are OPTIONAL and ENCOURAGING - never guilt-inducing.
+
+For EVERY quest write vivid {theme}-themed, 8-bit pixel-art image instructions where the USER is the hero:
+- initial_image_prompt: the hero facing the challenge (scene only - the renderer adds the user's face from a photo; do NOT describe their face).
+- success_edit: edit instruction showing how the initial scene would look victorious.
+- failure_edit: a FORGIVING edit instruction - the hero retreats to fight another day from the initial image. Never shaming.
+
+Set adventure.art_style to one shared "8-bit / 16-bit pixel-art, {theme} palette" string applied to every image, and adventure.seed to a single integer for the whole adventure. xp 10-100 by effort. All status:"active", image_state:"initial". Echo the user's real wording in each quest.task.
+/no_think"""
+
+# Frog Master chat router. Classifies one user message into a single intent and OUTPUTS JSON ONLY.
+INTENT_SYSTEM_PROMPT = """You are FrogQuest's "Frog Master" router. Read ONE user message plus a short context describing the current quest log, and classify it into EXACTLY ONE intent. OUTPUT JSON ONLY - no prose.
+
+intent must be one of:
+- "forge": the user is describing their to-do list / plans / goals for the first time (or wants a brand-new quest log). Use this when no quest log exists yet, or they clearly want to start over.
+- "add_tasks": the user wants to ADD one or more new tasks/goals to the EXISTING quest log.
+- "mark_done": the user says they FINISHED/completed a task. Put the task they mean in target_task (match it to one of the listed quest titles or tasks; leave empty to mean the currently selected quest).
+- "mark_couldnt": the user could NOT do a task, or wants to skip/postpone it. Put the task in target_task (empty = currently selected quest) and put their explanation in reason.
+- "unknown": small talk, a question, or anything that doesn't fit the above.
+
+Only "forge" and "add_tasks" describe NEW work; if a log already exists and the user is describing more things to do, prefer "add_tasks". target_task should copy the matching quest's title or task wording when you can identify it.
+/no_think"""
+
+
+# ----------------------------- FLUX prompt builders -----------------------------
+
+def build_initial_prompt(art_style: str, scene_prompt: str) -> str:
+    """Initial generation: the user (from their reference photo) as the hero facing the scene."""
+    return (
+        f"{art_style}. {scene_prompt}. "
+        "The hero is the person shown in the reference image, in this style and scene."
+    )
+
+
+def build_edit_prompt(art_style: str, edit_instruction: str) -> str:
+    """Edit pass: transform the existing scene into its success/failure state."""
+    return f"{art_style}. {edit_instruction}"
+
+
+# ----------------------------- helpers -----------------------------
+
+def extract_json(text: str) -> dict:
+    """Parse JSON from model output, tolerating stray prose or code fences."""
+    text = (text or "").strip()
+    try:
+        return json.loads(text)
+    except json.JSONDecodeError:
+        pass
+    # Fallback: grab the outermost { ... } span.
+    start, end = text.find("{"), text.rfind("}")
+    if start != -1 and end != -1 and end > start:
+        try:
+            return json.loads(text[start : end + 1])
+        except json.JSONDecodeError:
+            pass
+    return {}
+
+
+def preload_cuda_libs():
+    """Load the CUDA runtime libs (libcudart.so.12, libcublas*, ...) RTLD_GLOBAL by full path so
+    the prebuilt llama.cpp CUDA wheel can resolve them. They ship in the nvidia-*-cu12 pip packages
+    and inside torch/lib, but neither is on the dynamic loader's search path. No-op for anything not
+    found. Order matters: cudart before cublasLt before cublas. Needed on BOTH ZeroGPU and Modal
+    (same prebuilt cu124 wheel, same missing-loader-path problem)."""
+    import ctypes
+    import glob
+    import os
+    import site
+
+    dirs = []
+    try:
+        import torch
+        dirs.append(os.path.join(os.path.dirname(torch.__file__), "lib"))
+    except Exception:
+        pass
+    site_dirs = []
+    if hasattr(site, "getsitepackages"):
+        site_dirs += site.getsitepackages()
+    site_dirs.append(os.path.dirname(os.path.dirname(os.__file__)))  # fallback
+    for sp in dict.fromkeys(site_dirs):
+        dirs += glob.glob(os.path.join(sp, "nvidia", "*", "lib"))
+
+    for prefix in ("libcudart", "libnvrtc", "libcublasLt", "libcublas", "libcudnn"):
+        for d in dict.fromkeys(dirs):
+            for lib in sorted(glob.glob(os.path.join(d, prefix + "*.so*"))):
+                try:
+                    ctypes.CDLL(lib, mode=ctypes.RTLD_GLOBAL)
+                except OSError:
+                    pass

images.py

@@ -1,48 +1,51 @@
-"""FLUX.2 [klein] 4B image pipeline (diffusers, ZeroGPU).
+"""FLUX.2 [klein] 4B image pipeline. Pluggable GPU backend.
 
 One model, one `image=` parameter does both jobs:
   - initial generation: image=[user_photo]  (a LIST -> multi-reference; puts the user in the scene)
   - edit (next pass):    image=base_image    (a single image -> instruction-guided edit)
 
-Shared art_style + a fixed per-adventure seed give every quest a cohesive look. Always EDIT
-the initial image for success/failure states (consistency) - never text-to-image from scratch.
+Shared art_style + a fixed per-adventure seed give every quest a cohesive look. Always EDIT the
+initial image for success/failure states (consistency) - never text-to-image from scratch.
 
-Pipeline loaded at module level; CUDA emulation makes that safe outside @spaces.GPU. The
-actual .to("cuda") + inference happen inside the GPU-decorated functions.
+FROGQUEST_BACKEND selects WHERE inference runs (public functions identical either way):
+  - "zerogpu" (default): diffusers inside @spaces.GPU on the HF Space's ZeroGPU.
+  - "modal": forward to a deployed Modal class (see modal_app.py); the Space runs on CPU-basic.
+
+IMPORTANT: torch / diffusers are imported LAZILY (inside _get_pipe/_gen), and the weight prefetch
+is gated to the zerogpu backend, so importing this module on a CPU-basic Space (modal backend)
+drags in nothing heavy and downloads nothing.
 """
 from __future__ import annotations
 
 import base64
 import io
+import os
+
+from PIL import Image  # light; used by the CPU-side data-url helpers and type hints
 
-import spaces
-import torch
-from diffusers import Flux2KleinPipeline
-from PIL import Image
-
-MODEL_ID = "black-forest-labs/FLUX.2-klein-4B"  # verified: ungated, Apache 2.0
-# Quality config:
-#   - precision: bf16 on GPUs that support it (ZeroGPU/Blackwell = klein's NATIVE dtype, best
-#     quality). On a Turing GPU like T4 (no bf16) we fall back to fp16 — see _get_pipe.
-#   - steps: 4 is the model card's recommended value because klein is DISTILLED; raising it
-#     does not improve a distilled model (and can hurt).
-#   - there is no "context length" for a diffusion model.
-#   - resolution is the only real speed/quality knob; 768 keeps generation fast.
-STEPS = 4
-GUIDANCE = 4.0
-# Generated-scene resolution — the one real speed knob (time scales ~with pixel count).
-# 384 is ~25% the work of 768 (≈4x faster) and matches the UI's ~380px display, so nothing
-# visible is lost. Must stay a multiple of 16 (FLUX requirement) — that's why 384, not 380.
-MAX_SIDE = 384
-LOW_VRAM_GB = 24  # at/below this (e.g. T4 16GB) use fp16 + CPU offload so ~13GB of weights fit
+from gpu_shared import (
+    GUIDANCE,
+    LOW_VRAM_GB,
+    MAX_SIDE,
+    MODEL_ID,
+    STEPS,
+    build_edit_prompt,
+    build_initial_prompt,
+)
+
+BACKEND = os.environ.get("FROGQUEST_BACKEND", "zerogpu").lower()
+if BACKEND != "modal":  # the local/ZeroGPU path (default + any unrecognized value) needs the decorator
+    import spaces
 
 # Best-effort: pre-fetch the weights at startup so the first @spaces.GPU call doesn't pay the
-# multi-GB download out of its (metered, on ZeroGPU) duration. No-op offline / on fresh checkout.
-try:
-    from huggingface_hub import snapshot_download
-    snapshot_download(MODEL_ID)
-except Exception:
-    pass
+# multi-GB download out of its (metered, on ZeroGPU) duration. ZeroGPU-only — a CPU-basic Space
+# (modal backend) must NOT pull ~23GB. No-op offline / on a fresh local checkout.
+if BACKEND != "modal":
+    try:
+        from huggingface_hub import snapshot_download
+        snapshot_download(MODEL_ID)
+    except Exception:
+        pass
 
 _pipe = None
 _offloaded = False  # True when we used CPU offload (small GPU) instead of a full .to("cuda")
@@ -50,24 +53,28 @@ _offloaded = False  # True when we used CPU offload (small GPU) instead of a ful
 
 def _get_pipe():
     """Construct the pipeline lazily INSIDE the GPU call so we can read the REAL device's caps
-    and adapt — this is what lets ONE codebase run on both ZeroGPU (Blackwell, 48GB, native
-    bf16) and a small standard GPU like T4 (Turing, 16GB, fp16 + CPU offload). Just flip the
-    hardware in the Space settings; no code change needed."""
+    and adapt — bf16 on Blackwell (ZeroGPU), fp16 on Turing, CPU offload when VRAM is tight. torch
+    and diffusers are imported here (not at module top) so the modal backend never loads them."""
     global _pipe, _offloaded
     if _pipe is None:
+        import torch
+        from diffusers import Flux2KleinPipeline
+
         bf16 = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
-        dtype = torch.bfloat16 if bf16 else torch.float16  # T4 (Turing) has no bf16
+        dtype = torch.bfloat16 if bf16 else torch.float16  # Turing (T4) has no bf16
         _pipe = Flux2KleinPipeline.from_pretrained(MODEL_ID, torch_dtype=dtype)
         vram_gb = (torch.cuda.get_device_properties(0).total_memory / 1e9
                    if torch.cuda.is_available() else 0)
         if 0 < vram_gb < LOW_VRAM_GB:
-            # Small GPU: stream modules GPU<-CPU per step so the weights fit in ~16GB VRAM.
+            # Small GPU: stream modules GPU<-CPU per step so the ~23GB of weights fit in ~24GB VRAM.
             _pipe.enable_model_cpu_offload()
             _offloaded = True
     return _pipe
 
 
 def _gen(prompt: str, image, seed: int) -> Image.Image:
+    import torch
+
     pipe = _get_pipe()
     if not _offloaded:
         pipe.to("cuda")  # full-residency path (big GPU); offload manages its own device moves
@@ -84,21 +91,39 @@ def _gen(prompt: str, image, seed: int) -> Image.Image:
     return result.images[0]
 
 
-@spaces.GPU(duration=30)
-def initial_image(user_photo: Image.Image, art_style: str, scene_prompt: str, seed: int) -> Image.Image:
+# ----------------------------- local (in-Space, ZeroGPU) implementations -----------------------------
+
+def _initial_image_local(user_photo: Image.Image, art_style: str, scene_prompt: str, seed: int) -> Image.Image:
     """Generate the quest's initial scene with the user as the hero (photo as reference)."""
-    prompt = (
-        f"{art_style}. {scene_prompt}. "
-        "The hero is the person shown in the reference image, in this style and scene."
-    )
-    return _gen(prompt, image=[user_photo], seed=seed)
+    return _gen(build_initial_prompt(art_style, scene_prompt), image=[user_photo], seed=seed)
+
+
+def _edit_image_local(base_image: Image.Image, edit_instruction: str, art_style: str, seed: int) -> Image.Image:
+    """Edit the existing image into a success/failure state."""
+    return _gen(build_edit_prompt(art_style, edit_instruction), image=base_image, seed=seed)
+
+
+# ----------------------------- modal (off-Space) wrappers -----------------------------
+
+def _initial_image_modal(user_photo: Image.Image, art_style: str, scene_prompt: str, seed: int) -> Image.Image:
+    import modal
+    flux = modal.Cls.from_name("frogquest", "Flux")()
+    return flux.initial.remote(user_photo, art_style, scene_prompt, seed)
+
+
+def _edit_image_modal(base_image: Image.Image, edit_instruction: str, art_style: str, seed: int) -> Image.Image:
+    import modal
+    flux = modal.Cls.from_name("frogquest", "Flux")()
+    return flux.edit.remote(base_image, edit_instruction, art_style, seed)
 
 
-@spaces.GPU(duration=30)
-def edit_image(base_image: Image.Image, edit_instruction: str, art_style: str, seed: int) -> Image.Image:
-    """NEXT PASS (priority 5): edit the existing image into a success/failure state."""
-    prompt = f"{art_style}. {edit_instruction}"
-    return _gen(prompt, image=base_image, seed=seed)
+# ----------------------------- bind public names from the backend -----------------------------
+if BACKEND == "modal":
+    initial_image = _initial_image_modal
+    edit_image = _edit_image_modal
+else:
+    initial_image = spaces.GPU(duration=30)(_initial_image_local)
+    edit_image = spaces.GPU(duration=30)(_edit_image_local)
 
 
 # --- base64 <-> PIL helpers (photo arrives transiently as a data URL; nothing is persisted) ---

llm.py

@@ -1,117 +1,57 @@
-"""Nemotron Nano 4B (text-only) -> raw quest JSON, run via llama.cpp on ZeroGPU.
+"""Nemotron Nano 4B (text-only) -> raw quest JSON. Pluggable GPU backend.
 
-Recipe (CLAUDE.md + ZeroGPU/llama.cpp reference): set HF_HUB_ENABLE_HF_TRANSFER before
-importing huggingface_hub, download the GGUF at module level, but construct the Llama
-object INSIDE the @spaces.GPU function (first call ~60-90s, then disk-cached & fast).
+FROGQUEST_BACKEND selects WHERE the GPU work runs (the public functions are identical either way):
+  - "zerogpu" (default): construct the Llama via llama.cpp INSIDE a @spaces.GPU function on the
+    HF Space's ZeroGPU. (First call ~60-90s, then disk-cached & fast.)
+  - "modal": forward to a deployed Modal class (see modal_app.py); the Space itself runs on
+    CPU-basic and imports NOTHING heavy here.
 
-The LLM's job is ONLY to write JSON to the contract in schema.py. Output is constrained
-with a JSON-schema response_format and then validated/clamped by the caller.
+The LLM's job is ONLY to write JSON to the contract in schema.py. Output is constrained with a
+JSON-schema response_format and then validated/clamped by the caller. Shared prompts / the JSON
+extractor / model config live in gpu_shared.py so both backends stay in lockstep.
 """
 from __future__ import annotations
 
-import json
 import os
 
 os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")  # MUST precede huggingface_hub import
 
-import spaces  # noqa: E402
+BACKEND = os.environ.get("FROGQUEST_BACKEND", "zerogpu").lower()
+if BACKEND != "modal":  # the local/ZeroGPU path (default + any unrecognized value) needs the decorator
+    import spaces  # noqa: E402
 
 from schema import INTENT_SCHEMA, RESPONSE_SCHEMA  # noqa: E402
-
-# Verified on HF (June 2026). unsloth's GGUF of NVIDIA Nemotron-3 Nano 4B (the repo actually
-# exists; the older Llama-3.1-Nemotron-Nano GGUF id 404s). Q8_0 (~4.3GB, near-fp16) over Q4_K_M:
-# ZeroGPU has ~48GB VRAM so size is a non-issue, and the better precision helps the "Eat That
-# Frog" prioritization reasoning. filename is a glob; Llama.from_pretrained resolves the exact
-# file. (Hard floor is Q4 — sub-4-bit is degraded for this hybrid architecture.)
-GGUF_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF"
-GGUF_FILE = "*Q8_0*.gguf"
-
-# Context length is hardware-adaptive (chosen in _get_llm from the REAL device's VRAM). On a big
-# GPU (ZeroGPU, 48GB) we use the full 128k Nemotron supports; on a small standard GPU (e.g. T4
-# 16GB) a 128k KV cache won't fit alongside FLUX, so we drop to 16k — still vastly more than this
-# app's tiny prompts ever need. This is one half of the ZeroGPU<->T4 portability (see images.py).
-N_CTX = 131072          # big GPUs (ZeroGPU)
-N_CTX_SMALL = 16384     # small GPUs (T4 & similar)
-LOW_VRAM_GB = 24        # at/below this, treat the GPU as "small" (T4 = 16GB)
+from gpu_shared import (  # noqa: E402
+    GGUF_FILE,
+    GGUF_REPO,
+    INTENT_SYSTEM_PROMPT,
+    LOW_VRAM_GB,
+    N_CTX,
+    N_CTX_SMALL,
+    SYSTEM_PROMPT,
+    extract_json,
+    preload_cuda_libs,
+)
 
 # Best-effort: warm the HF cache at startup so the FIRST @spaces.GPU call doesn't spend its
-# (metered, on ZeroGPU) duration downloading ~4GB. No-op if offline or on a fresh local checkout
-# — the model still lazy-loads from cache/network inside the GPU call either way.
-try:
-    from huggingface_hub import hf_hub_download, list_repo_files  # noqa: E402
-    _gguf = next((f for f in list_repo_files(GGUF_REPO) if "Q8_0" in f and f.endswith(".gguf")), None)
-    if _gguf:
-        hf_hub_download(GGUF_REPO, _gguf)
-except Exception:
-    pass
-
-SYSTEM_PROMPT = """You are FrogQuest's quest designer. Convert the user's real to-do list into a themed text-adventure quest log and OUTPUT JSON ONLY - no prose, no markdown, no code.
-
-Apply the "Eat That Frog" method:
-- The FROG = the single most important/hardest task. Mark exactly ONE quest is_frog:true and order it FIRST.
-- Break each big/multi-step goal into an ordered chain of smaller quests sharing one goal_group label; keep simple to-dos as standalone quests.
-- Add 1-3 bonus self-care quests (type:"bonus") such as meditate 5 min, exercise 20 min, digital detox 1 hr. They are OPTIONAL and ENCOURAGING - never guilt-inducing.
-
-For EVERY quest write vivid {theme}-themed, 8-bit pixel-art image instructions where the USER is the hero:
-- initial_image_prompt: the hero facing the challenge (scene only - the renderer adds the user's face from a photo; do NOT describe their face).
-- success_edit: edit instruction showing how the initial scene would look victorious.
-- failure_edit: a FORGIVING edit instruction - the hero retreats to fight another day from the initial image. Never shaming.
-
-Set adventure.art_style to one shared "8-bit / 16-bit pixel-art, {theme} palette" string applied to every image, and adventure.seed to a single integer for the whole adventure. xp 10-100 by effort. All status:"active", image_state:"initial". Echo the user's real wording in each quest.task.
-/no_think"""
-
-# Frog Master chat router. Classifies one user message into a single intent and OUTPUTS JSON ONLY.
-INTENT_SYSTEM_PROMPT = """You are FrogQuest's "Frog Master" router. Read ONE user message plus a short context describing the current quest log, and classify it into EXACTLY ONE intent. OUTPUT JSON ONLY - no prose.
-
-intent must be one of:
-- "forge": the user is describing their to-do list / plans / goals for the first time (or wants a brand-new quest log). Use this when no quest log exists yet, or they clearly want to start over.
-- "add_tasks": the user wants to ADD one or more new tasks/goals to the EXISTING quest log.
-- "mark_done": the user says they FINISHED/completed a task. Put the task they mean in target_task (match it to one of the listed quest titles or tasks; leave empty to mean the currently selected quest).
-- "mark_couldnt": the user could NOT do a task, or wants to skip/postpone it. Put the task in target_task (empty = currently selected quest) and put their explanation in reason.
-- "unknown": small talk, a question, or anything that doesn't fit the above.
-
-Only "forge" and "add_tasks" describe NEW work; if a log already exists and the user is describing more things to do, prefer "add_tasks". target_task should copy the matching quest's title or task wording when you can identify it.
-/no_think"""
-
-_llm = None
-
-
-def _preload_cuda_libs():
-    """Load the CUDA runtime libs (libcudart.so.12, libcublas*, ...) RTLD_GLOBAL by full path
-    so the prebuilt llama.cpp CUDA wheel can resolve them. They ship in the nvidia-*-cu12 pip
-    packages and inside torch/lib, but neither is on the dynamic loader's search path. No-op
-    for anything not found. Order matters: cudart before cublasLt before cublas."""
-    import ctypes
-    import glob
-    import os
-    import site
-
-    dirs = []
+# (metered, on ZeroGPU) duration downloading ~4GB. Local-path only — on a CPU-basic Space (modal
+# backend) we must NOT download the GGUF. No-op if offline or on a fresh local checkout.
+if BACKEND != "modal":
     try:
-        import torch
-        dirs.append(os.path.join(os.path.dirname(torch.__file__), "lib"))
+        from huggingface_hub import hf_hub_download, list_repo_files
+        _gguf = next((f for f in list_repo_files(GGUF_REPO) if "Q8_0" in f and f.endswith(".gguf")), None)
+        if _gguf:
+            hf_hub_download(GGUF_REPO, _gguf)
     except Exception:
         pass
-    site_dirs = []
-    if hasattr(site, "getsitepackages"):
-        site_dirs += site.getsitepackages()
-    site_dirs.append(os.path.dirname(os.path.dirname(os.__file__)))  # fallback
-    for sp in dict.fromkeys(site_dirs):
-        dirs += glob.glob(os.path.join(sp, "nvidia", "*", "lib"))
-
-    for prefix in ("libcudart", "libnvrtc", "libcublasLt", "libcublas", "libcudnn"):
-        for d in dict.fromkeys(dirs):
-            for lib in sorted(glob.glob(os.path.join(d, prefix + "*.so*"))):
-                try:
-                    ctypes.CDLL(lib, mode=ctypes.RTLD_GLOBAL)
-                except OSError:
-                    pass
+
+_llm = None
 
 
 def _get_llm():
     """Lazily download + construct the Llama model on the GPU (must run inside @spaces.GPU).
 
-    First call downloads the GGUF (~2.5GB) then disk-caches it, so later calls are fast.
+    First call downloads the GGUF then disk-caches it, so later calls are fast.
     """
     global _llm
     if _llm is None:
@@ -121,7 +61,7 @@ def _get_llm():
         # 1) importing torch loads many of them RTLD_GLOBAL;
         # 2) belt-and-suspenders: explicitly preload the nvidia-* CUDA libs too.
         import torch  # noqa: F401
-        _preload_cuda_libs()
+        preload_cuda_libs()
         from llama_cpp import Llama
 
         vram_gb = (torch.cuda.get_device_properties(0).total_memory / 1e9
@@ -137,8 +77,9 @@ def _get_llm():
     return _llm
 
 
-@spaces.GPU(duration=70)
-def generate_quests_raw(todos: str, theme: str) -> dict:
+# ----------------------------- local (in-Space, ZeroGPU) implementations -----------------------------
+
+def _generate_quests_local(todos: str, theme: str) -> dict:
     """Return the model's raw JSON object (UNVALIDATED - caller must validate_and_clamp)."""
     llm = _get_llm()
     system = SYSTEM_PROMPT.replace("{theme}", theme)
@@ -153,17 +94,14 @@ def generate_quests_raw(todos: str, theme: str) -> dict:
         temperature=0.0,
         max_tokens=4096,
     )
-    content = out["choices"][0]["message"]["content"]
-    return _extract_json(content)
+    return extract_json(out["choices"][0]["message"]["content"])
 
 
-@spaces.GPU(duration=45)
-def route_intent(message: str, context: str) -> dict:
+def _route_intent_local(message: str, context: str) -> dict:
     """Classify one Frog Master chat message into {intent, target_task?, reason?}.
 
     `context` is a SHORT text summary of the current log (does a log exist + quest titles/ids/
-    status) - never images (CLAUDE.md rule). Returns the raw classification; the caller decides
-    what to do. Falls back to {"intent": "unknown"} on unparseable output.
+    status) - never images (CLAUDE.md rule). Falls back to {"intent": "unknown"} on bad output.
     """
     llm = _get_llm()
     user = f"Context:\n{context.strip()}\n\nUser message:\n{message.strip()}"
@@ -176,7 +114,7 @@ def route_intent(message: str, context: str) -> dict:
         temperature=0.0,
         max_tokens=256,
     )
-    parsed = _extract_json(out["choices"][0]["message"]["content"])
+    parsed = extract_json(out["choices"][0]["message"]["content"])
     if not isinstance(parsed, dict) or parsed.get("intent") not in (
         "forge", "add_tasks", "mark_done", "mark_couldnt", "unknown",
     ):
@@ -184,18 +122,25 @@ def route_intent(message: str, context: str) -> dict:
     return parsed
 
 
-def _extract_json(text: str) -> dict:
-    """Parse JSON from the model output, tolerating stray prose or code fences."""
-    text = (text or "").strip()
-    try:
-        return json.loads(text)
-    except json.JSONDecodeError:
-        pass
-    # Fallback: grab the outermost { ... } span.
-    start, end = text.find("{"), text.rfind("}")
-    if start != -1 and end != -1 and end > start:
-        try:
-            return json.loads(text[start : end + 1])
-        except json.JSONDecodeError:
-            pass
-    return {}
+# ----------------------------- modal (off-Space) wrappers -----------------------------
+
+def _generate_quests_modal(todos: str, theme: str) -> dict:
+    import modal
+    llm = modal.Cls.from_name("frogquest", "LLM")()
+    return llm.generate_quests.remote(todos, theme)
+
+
+def _route_intent_modal(message: str, context: str) -> dict:
+    import modal
+    llm = modal.Cls.from_name("frogquest", "LLM")()
+    return llm.route_intent.remote(message, context)
+
+
+# ----------------------------- bind public names from the backend -----------------------------
+# app.py imports these by name; signatures are identical across backends.
+if BACKEND == "modal":
+    generate_quests_raw = _generate_quests_modal
+    route_intent = _route_intent_modal
+else:
+    generate_quests_raw = spaces.GPU(duration=70)(_generate_quests_local)
+    route_intent = spaces.GPU(duration=45)(_route_intent_local)

modal_app.py

@@ -0,0 +1,163 @@
+"""Modal backend for FrogQuest's GPU work — the off-Space alternative to ZeroGPU.
+
+Deploy once:  `modal deploy modal_app.py`
+The HF Space (running on CPU-basic with FROGQUEST_BACKEND=modal) then invokes these via
+`modal.Cls.from_name("frogquest", "LLM" | "Flux")().<method>.remote(...)` from llm.py / images.py.
+
+Both backends share the SAME prompts, config, JSON extractor (gpu_shared) and JSON schemas
+(schema) so they can never drift. Those two local files are mounted into the image with
+`add_local_python_source`.
+
+GPU config:
+  - LLM (Nemotron Q8 ~4.3GB): a cheap L4 is plenty.
+  - FLUX.2 klein (~23GB): set FROGQUEST_MODAL_FLUX_GPU at DEPLOY time (default "A10G", 24GB ->
+    needs enable_model_cpu_offload(); use "L4" to save cost or "L40S"/"A100-40GB" if it OOMs).
+
+NOTE: re-verify Modal API specifics against current docs (Modal 1.0+): App / Image.pip_install
+extra_index_url / add_local_python_source / @app.cls / @modal.enter / @modal.method /
+Cls.from_name / Volume.commit.
+"""
+import os
+
+import modal
+
+from gpu_shared import (
+    GGUF_FILE,
+    GGUF_REPO,
+    GUIDANCE,
+    INTENT_SYSTEM_PROMPT,
+    MAX_SIDE,
+    MODEL_ID,
+    N_CTX,
+    STEPS,
+    SYSTEM_PROMPT,
+    build_edit_prompt,
+    build_initial_prompt,
+    extract_json,
+    preload_cuda_libs,
+)
+from schema import INTENT_SCHEMA, RESPONSE_SCHEMA
+
+app = modal.App("frogquest")
+
+# Persist the HF weight cache (FLUX ~23GB + GGUF ~4.3GB) across cold starts -> download once.
+CACHE_DIR = "/cache"
+vol = modal.Volume.from_name("frogquest-cache", create_if_missing=True)
+
+# Prebuilt CUDA wheel for llama.cpp (same index the Space uses). cu124 wheels run on Modal GPUs.
+_LLAMA_INDEX = "https://abetlen.github.io/llama-cpp-python/whl/cu124"
+
+image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .pip_install(
+        "git+https://github.com/huggingface/diffusers.git",  # Flux2KleinPipeline (>=0.38)
+        "transformers>=4.44.2",
+        "accelerate",            # required by enable_model_cpu_offload()
+        "torch",
+        "pillow",
+        "sentencepiece",         # FLUX.2 text-encoder tokenizer deps (belt-and-suspenders)
+        "protobuf",
+        "huggingface-hub",
+        "hf-transfer",
+    )
+    .pip_install(
+        "llama-cpp-python==0.3.28",
+        extra_index_url=_LLAMA_INDEX,
+        extra_options="--prefer-binary",
+    )
+    .env({"HF_HOME": CACHE_DIR, "HF_HUB_ENABLE_HF_TRANSFER": "1"})
+    .add_local_python_source("schema", "gpu_shared")
+)
+
+# Configurable at deploy time (read on your machine when you run `modal deploy`).
+FLUX_GPU = os.environ.get("FROGQUEST_MODAL_FLUX_GPU", "A10G")
+
+
+@app.cls(gpu="L4", image=image, volumes={CACHE_DIR: vol}, scaledown_window=300)
+class LLM:
+    @modal.enter()
+    def _load(self):
+        vol.reload()  # pick up weights another container may have already cached
+        import torch  # noqa: F401  (loads CUDA libs RTLD_GLOBAL)
+        preload_cuda_libs()
+        from llama_cpp import Llama
+
+        self.llm = Llama.from_pretrained(
+            repo_id=GGUF_REPO,
+            filename=GGUF_FILE,
+            n_gpu_layers=-1,
+            n_ctx=N_CTX,  # Modal GPU has the VRAM for the full 128k
+            verbose=False,
+        )
+        vol.commit()  # persist the freshly downloaded GGUF (no-op if already cached)
+
+    @modal.method()
+    def generate_quests(self, todos: str, theme: str) -> dict:
+        system = SYSTEM_PROMPT.replace("{theme}", theme)
+        user = f"Theme: {theme}\nMy to-do list / goals:\n{todos.strip()}"
+        out = self.llm.create_chat_completion(
+            messages=[
+                {"role": "system", "content": system},
+                {"role": "user", "content": user},
+            ],
+            response_format={"type": "json_object", "schema": RESPONSE_SCHEMA},
+            temperature=0.0,
+            max_tokens=4096,
+        )
+        return extract_json(out["choices"][0]["message"]["content"])
+
+    @modal.method()
+    def route_intent(self, message: str, context: str) -> dict:
+        user = f"Context:\n{context.strip()}\n\nUser message:\n{message.strip()}"
+        out = self.llm.create_chat_completion(
+            messages=[
+                {"role": "system", "content": INTENT_SYSTEM_PROMPT},
+                {"role": "user", "content": user},
+            ],
+            response_format={"type": "json_object", "schema": INTENT_SCHEMA},
+            temperature=0.0,
+            max_tokens=256,
+        )
+        parsed = extract_json(out["choices"][0]["message"]["content"])
+        if not isinstance(parsed, dict) or parsed.get("intent") not in (
+            "forge", "add_tasks", "mark_done", "mark_couldnt", "unknown",
+        ):
+            return {"intent": "unknown"}
+        return parsed
+
+
+@app.cls(gpu=FLUX_GPU, image=image, volumes={CACHE_DIR: vol}, scaledown_window=300)
+class Flux:
+    @modal.enter()
+    def _load(self):
+        vol.reload()
+        import torch
+        from diffusers import Flux2KleinPipeline
+
+        self._torch = torch
+        pipe = Flux2KleinPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
+        # 24GB card vs ~23GB model: offload to fit with headroom for activations.
+        pipe.enable_model_cpu_offload()
+        self.pipe = pipe
+        vol.commit()  # persist the freshly downloaded FLUX weights (no-op if already cached)
+
+    def _gen(self, prompt, image, seed):
+        gen = self._torch.Generator("cuda").manual_seed(int(seed))
+        result = self.pipe(
+            prompt=prompt,
+            image=image,
+            generator=gen,
+            num_inference_steps=STEPS,
+            guidance_scale=GUIDANCE,
+            height=MAX_SIDE,
+            width=MAX_SIDE,
+        )
+        return result.images[0]
+
+    @modal.method()
+    def initial(self, user_photo, art_style: str, scene_prompt: str, seed: int):
+        return self._gen(build_initial_prompt(art_style, scene_prompt), [user_photo], seed)
+
+    @modal.method()
+    def edit(self, base_image, edit_instruction: str, art_style: str, seed: int):
+        return self._gen(build_edit_prompt(art_style, edit_instruction), base_image, seed)

requirements-modal.txt

@@ -0,0 +1,11 @@
+# OPTIONAL lean dependency set for running the HF Space in MODAL mode (FROGQUEST_BACKEND=modal).
+#
+# In modal mode the Space runs on CPU-basic and offloads ALL GPU work to Modal, so it never
+# imports torch / diffusers / llama-cpp. Swapping requirements.txt for this file (rename it to
+# requirements.txt on the Space) skips installing those heavy libs -> much faster Space builds.
+# The default requirements.txt keeps both sets so a single push works in either mode.
+gradio>=5.0
+spaces
+pillow
+huggingface-hub
+modal

requirements.txt

@@ -16,6 +16,9 @@ nvidia-cuda-nvrtc-cu12
 pillow
 hf-transfer
 huggingface-hub
+# Modal client — only used when FROGQUEST_BACKEND=modal (the Space forwards GPU work to Modal).
+# Harmless in the default zerogpu mode (imported lazily inside the wrapper functions only).
+modal
 # llama-cpp-python: install the PREBUILT CUDA wheel, never compile from source.
 # The cu128 index does not exist (404) -> pip silently falls back to a 30+ min source build.
 # cu124 is the proven index (used by the reference ZeroGPU Space) and its wheels run fine on