CLAUDE.md

CLAUDE.md

Writing conventions

• Always use regular hyphens (-) instead of em dashes (—) in all generated text and documentation.

Working guidance for AI pair-programming (Claude Code) on hackathon-ia-visualnovel — an AI-improvised anime visual novel for the Build Small Hackathon. Read this first, then docs/ARCHITECTURE.md for depth and docs/PROMPTS.md for the exact prompts + grammar.

What this project is (1 paragraph)

A voice-or-text VN where story, NPC dialogue, and art are generated live by small local models. Three model roles: The Weaver (director — emits structured JSON directives), The Voices (NPC actor — in-character dialogue), and The Painter (SDXL-Turbo — backdrops + sprites). The Weaver and Voices share one LLM (two system prompts). Input is transcribed by Whisper (The Ear). Hard constraint: ≤ 32B total params, must stay a Gradio app on a HF Space.

The golden rule

The model proposes, code disposes. The LLM returns a typed DirectorOutput; state.apply_directives is the only function that mutates GameState. The .md files are a derived view for humans and for re-injecting context — never the source of truth. Don't let the LLM hand-edit prose state.

Commands (uv)

uv sync                                  # light: gradio + pydantic + pillow (mock mode)
uv run python -m visualnovel.smoke       # full loop, no UI, no models
uv run python app.py                     # custom VN UI → http://localhost:7860
GRADIO_MVP_UI=1 uv run python app.py     # gr.Blocks fallback

uv run ruff check . && uv run ruff format .
uv run pytest -q                         # state round-trip · directives · memory budget

# real backends (install the relevant extras first — see README for ROCm/Metal flags)
VN_MOCK=0 VN_LLM_BACKEND=transformers uv run python app.py
VN_TRACE=runs/$(date +%s).jsonl uv run python app.py   # Open-Trace bonus

⭐ The mock → real migration path (the whole point of this scaffold)

Everything runs mock-first (VN_MOCK=1). Replace one mock at a time; keep pytest + smoke green after each. Work in this order (lowest risk → highest):

1. LLM — director call. Implement llm.LlamaCppLLM (or TransformersLLM), then delete the if config.USE_MOCK: branches in orchestrator.direct_turn / init_world / compact_memory so the real complete_json(schema=…) path runs. The JSON schema is already derived from schemas.DirectorOutput via prompts.directive_schema(). Tune sampling per docs/PROMPTS.md §6.
2. Painter. Implement painter.SdxlTurboPainter._render (stub already sketches diffusers + seed + LoRA). Caching, prompt composition, and seed-pinning are done. Set VN_IMAGE_LORA to your fine-tuned anime LoRA when ready.
3. STT. stt.WhisperSTT is sketched (faster-whisper). The frontend already records audio and calls /transcribe.
4. Polish. Sprite moods (pre-gen set, or FLUX.2-Klein conditioning), BiRefNet transparent sprites, speculative backdrop paint, real compact_memory summaries.

Acceptance after each step: uv run python -m visualnovel.smoke still completes and uv run pytest -q is green. The mock branches are clearly marked so they're easy to find and remove.

Architecture at a glance

player (text/voice)
  │ voice → stt.py (Whisper)
  ▼
engine.play_turn ─▶ orchestrator.direct_turn ─ one grammar-constrained LLM call ─▶ DirectorOutput
  │                      (memory.assemble_context builds the bounded prompt)
  ▼
state.apply_directives (deterministic; the ONLY mutator) ─▶ state.save_memory (.md views)
  │
  ├─ painter.backdrop / painter.sprite  (cached by entity·mood·seed)
  ▼
engine._view ─▶ ViewState ─▶ frontend/index.html  (backdrop + sprite + dialogue)

File responsibilities

File	Owns	Don't put here
visualnovel/schemas.py	all Pydantic contracts (GameState, DirectorOutput, ViewState)	logic
visualnovel/state.py	apply_directives (sole mutator) + .md render	model calls, UI
visualnovel/llm.py	one LLM wrapper: complete / complete_json(schema)	game logic
visualnovel/orchestrator.py	the Weaver: init / direct_turn / compact (+ mock branches)	raw prompt strings (import from prompts.py)
visualnovel/characters.py	the Voices: present-character context	image prompts
visualnovel/painter.py	prompt compose, cache, seeds, render	story decisions
visualnovel/stt.py	Whisper transcribe	anything else
visualnovel/memory.py	bounded context + compaction trigger	persistence format (that's state.py)
visualnovel/engine.py	façade tying it together → ViewState	model internals
app.py	gradio.Server routes, @app.api, @spaces.GPU	business logic (keep thin)
frontend/index.html	the entire VN UI; talks via the Gradio JS client	secrets, model logic

Conventions

• Python 3.11+ (3.12 pinned), type hints everywhere, Pydantic for all IO contracts. ruff formats/lints.
• Keep app.py thin. Logic lives in visualnovel/ so it's unit-testable without a server.
• One LLM call per turn by default (complete_json with the directive schema). Extra calls (init, compaction) are allowed but rare — latency matters.
• Every Painter call is cached by (kind, prompt, seed). Generate a sprite once; never re-paint to "refresh" (kills consistency). Seeds are explicit and stored in state.
• Prompts are versioned in docs/PROMPTS.md (mirrored in prompts.py). Don't scatter literals.
• Heavy imports stay lazy (inside the real backend classes) so a mock checkout needs nothing.
• Stream text first, image second. Dialogue appears before the (slower) image.

Platform gotchas (read before you fight these)

1. Python 3.12 is pinned (.python-version) because torch / llama-cpp-python / ctranslate2 ship wheels for it. Don't bump to a bleeding-edge interpreter unless every dependency has wheels → otherwise uv python pin 3.12 && uv sync.
2. No CUDA locally — ROCm + Metal. llama.cpp build flags differ: -DGGML_METAL=on (Mac, default), -DGGML_HIPBLAS=on / -DGGML_HIP=on (AMD). torch reports ROCm as "cuda" → detect_device() returns "cuda" on the AMD box; "mps" on Mac.
3. CTranslate2 has no Metal/ROCm → faster-whisper runs on CPU on both your machines (fine for short clips). For GPU STT use mlx-whisper (Mac) or whisper.cpp.
4. llama.cpp on hosted ZeroGPU is finicky. Claim the llama.cpp badge locally; keep the transformers backend for the Space (VN_LLM_BACKEND=transformers).
5. Image latency is the #1 UX risk. 1–4-step SDXL-Turbo, 512–768px, text-before-image, cache hard.
6. Context window is finite ("Thousand Token Wood"). Feed summary + present sheets + last k turns — never the whole log. memory.py enforces it.
7. Spaces filesystem is ephemeral. .md memory is per-session unless you wire HF persistent storage.
8. Qwen3 think-mode on ZeroGPU. Qwen3-14B via transformers sometimes puts the entire JSON answer inside its <think> block and emits nothing after </think>. TransformersLLM.complete_json handles this by calling apply_chat_template(..., enable_thinking=False) to suppress thinking for structured output, and falls back to searching inside the think block if no JSON appears after it. Do NOT use _quiet_stderr() around transformers loads — it swallows loading errors that are essential for ZeroGPU debugging.

Definition of done (MVP)

A player types (then speaks) a line → an NPC replies in character → the Weaver decides whether the scene/character changed → the screen shows a coherent backdrop + sprite + dialogue, looping indefinitely without crashing, on the laptop-safe config, inside the custom VN UI.

Out of scope (resist these)

Multiplayer; save/load beyond one session; combat/stats UIs; more than ~3 NPCs on screen; photorealism (anime is cheaper, faster, prettier here); a second LLM just to look "multi-agent". Ship the loop.