CLAUDE.md

# CLAUDE.md

## Writing conventions
- Always use regular hyphens (`-`) instead of em dashes (`—`) in all generated text and documentation.

Working guidance for AI pair-programming (Claude Code) on **hackathon-ia-visualnovel** — an
AI-improvised anime visual novel for the Build Small Hackathon. Read this first, then
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for depth and [`docs/PROMPTS.md`](docs/PROMPTS.md)
for the exact prompts + grammar.

## What this project is (1 paragraph)

A voice-or-text VN where story, NPC dialogue, and art are **generated live** by small local models.
Three model roles: **The Weaver** (director — emits structured JSON directives), **The Voices** (NPC
actor — in-character dialogue), and **The Painter** (SDXL-Turbo — backdrops + sprites). The Weaver and
Voices share **one** LLM (two system prompts). Input is transcribed by **Whisper** (The Ear). Hard
constraint: **≤ 32B total params**, must stay a **Gradio app on a HF Space**.

## The golden rule

**The model proposes, code disposes.** The LLM returns a typed `DirectorOutput`; `state.apply_directives`
is the *only* function that mutates `GameState`. The `.md` files are a *derived view* for humans and for
re-injecting context — never the source of truth. Don't let the LLM hand-edit prose state.

## Commands (uv)

```bash
uv sync                                  # light: gradio + pydantic + pillow (mock mode)
uv run python -m visualnovel.smoke       # full loop, no UI, no models
uv run python app.py                     # custom VN UI → http://localhost:7860
GRADIO_MVP_UI=1 uv run python app.py     # gr.Blocks fallback

uv run ruff check . && uv run ruff format .
uv run pytest -q                         # state round-trip · directives · memory budget

# real backends (install the relevant extras first — see README for ROCm/Metal flags)
VN_MOCK=0 VN_LLM_BACKEND=transformers uv run python app.py
VN_TRACE=runs/$(date +%s).jsonl uv run python app.py   # Open-Trace bonus
```

## ⭐ The mock → real migration path (the whole point of this scaffold)

Everything runs **mock-first** (`VN_MOCK=1`). Replace one mock at a time; keep `pytest` + `smoke` green
after each. Work in this order (lowest risk → highest):

1. **LLM — director call.** Implement `llm.LlamaCppLLM` (or `TransformersLLM`), then **delete the
   `if config.USE_MOCK:` branches** in `orchestrator.direct_turn` / `init_world` / `compact_memory`
   so the real `complete_json(schema=…)` path runs. The JSON schema is already derived from
   `schemas.DirectorOutput` via `prompts.directive_schema()`. Tune sampling per `docs/PROMPTS.md §6`.
2. **Painter.** Implement `painter.SdxlTurboPainter._render` (stub already sketches diffusers +
   seed + LoRA). Caching, prompt composition, and seed-pinning are done. Set `VN_IMAGE_LORA` to your
   fine-tuned anime LoRA when ready.
3. **STT.** `stt.WhisperSTT` is sketched (faster-whisper). The frontend already records audio and
   calls `/transcribe`.
4. **Polish.** Sprite moods (pre-gen set, or FLUX.2-Klein conditioning), BiRefNet transparent sprites,
   speculative backdrop paint, real `compact_memory` summaries.

Acceptance after each step: `uv run python -m visualnovel.smoke` still completes and `uv run pytest -q`
is green. The mock branches are clearly marked so they're easy to find and remove.

## Architecture at a glance

```
player (text/voice)
  │ voice → stt.py (Whisper)
  ▼
engine.play_turn ─▶ orchestrator.direct_turn ─ one grammar-constrained LLM call ─▶ DirectorOutput
  │                      (memory.assemble_context builds the bounded prompt)
  ▼
state.apply_directives (deterministic; the ONLY mutator) ─▶ state.save_memory (.md views)
  │
  ├─ painter.backdrop / painter.sprite  (cached by entity·mood·seed)
  ▼
engine._view ─▶ ViewState ─▶ frontend/index.html  (backdrop + sprite + dialogue)
```

## File responsibilities

| File | Owns | Don't put here |
|---|---|---|
| `visualnovel/schemas.py` | all Pydantic contracts (`GameState`, `DirectorOutput`, `ViewState`) | logic |
| `visualnovel/state.py` | `apply_directives` (sole mutator) + `.md` render | model calls, UI |
| `visualnovel/llm.py` | one LLM wrapper: `complete` / `complete_json(schema)` | game logic |
| `visualnovel/orchestrator.py` | the Weaver: init / direct_turn / compact (+ mock branches) | raw prompt strings (import from `prompts.py`) |
| `visualnovel/characters.py` | the Voices: present-character context | image prompts |
| `visualnovel/painter.py` | prompt compose, cache, seeds, render | story decisions |
| `visualnovel/stt.py` | Whisper transcribe | anything else |
| `visualnovel/memory.py` | bounded context + compaction trigger | persistence format (that's state.py) |
| `visualnovel/engine.py` | façade tying it together → `ViewState` | model internals |
| `app.py` | `gradio.Server` routes, `@app.api`, `@spaces.GPU` | business logic (keep thin) |
| `frontend/index.html` | the entire VN UI; talks via the Gradio JS client | secrets, model logic |

## Conventions

- **Python 3.11+ (3.12 pinned), type hints everywhere, Pydantic for all IO contracts.** `ruff` formats/lints.
- **Keep `app.py` thin.** Logic lives in `visualnovel/` so it's unit-testable without a server.
- **One LLM call per turn by default** (`complete_json` with the directive schema). Extra calls (init,
  compaction) are allowed but rare — latency matters.
- **Every Painter call is cached** by `(kind, prompt, seed)`. Generate a sprite **once**; never re-paint
  to "refresh" (kills consistency). Seeds are explicit and stored in state.
- **Prompts are versioned in `docs/PROMPTS.md`** (mirrored in `prompts.py`). Don't scatter literals.
- **Heavy imports stay lazy** (inside the real backend classes) so a mock checkout needs nothing.
- **Stream text first, image second.** Dialogue appears before the (slower) image.

## Platform gotchas (read before you fight these)

1. **Python 3.12 is pinned** (`.python-version`) because torch / llama-cpp-python / ctranslate2 ship
   wheels for it. Don't bump to a bleeding-edge interpreter unless every dependency has wheels →
   otherwise `uv python pin 3.12 && uv sync`.
2. **No CUDA locally — ROCm + Metal.** llama.cpp build flags differ: `-DGGML_METAL=on` (Mac, default),
   `-DGGML_HIPBLAS=on` / `-DGGML_HIP=on` (AMD). torch reports **ROCm as "cuda"** → `detect_device()`
   returns `"cuda"` on the AMD box; `"mps"` on Mac.
3. **CTranslate2 has no Metal/ROCm** → `faster-whisper` runs on CPU on both your machines (fine for
   short clips). For GPU STT use `mlx-whisper` (Mac) or `whisper.cpp`.
4. **llama.cpp on hosted ZeroGPU is finicky.** Claim the llama.cpp badge **locally**; keep the
   `transformers` backend for the Space (`VN_LLM_BACKEND=transformers`).
5. **Image latency is the #1 UX risk.** 1–4-step SDXL-Turbo, 512–768px, text-before-image, cache hard.
6. **Context window is finite** ("Thousand Token Wood"). Feed summary + present sheets + last k turns —
   never the whole log. `memory.py` enforces it.
7. **Spaces filesystem is ephemeral.** `.md` memory is per-session unless you wire HF persistent storage.
8. **Qwen3 think-mode on ZeroGPU.** Qwen3-14B via `transformers` sometimes puts the entire JSON answer
   inside its `<think>` block and emits nothing after `</think>`. `TransformersLLM.complete_json` handles
   this by calling `apply_chat_template(..., enable_thinking=False)` to suppress thinking for structured
   output, and falls back to searching inside the think block if no JSON appears after it. Do NOT use
   `_quiet_stderr()` around transformers loads — it swallows loading errors that are essential for
   ZeroGPU debugging.

## Definition of done (MVP)

A player types (then speaks) a line → an NPC replies in character → the Weaver decides whether the
scene/character changed → the screen shows a coherent backdrop + sprite + dialogue, looping
indefinitely without crashing, on the laptop-safe config, inside the custom VN UI.

## Out of scope (resist these)

Multiplayer; save/load beyond one session; combat/stats UIs; more than ~3 NPCs on screen;
photorealism (anime is cheaper, faster, prettier here); a second LLM just to look "multi-agent".
Ship the loop.