--- title: Ephemeral Hearts emoji: 🐠 colorFrom: green colorTo: indigo sdk: gradio sdk_version: 6.18.0 python_version: '3.12' app_file: app.py pinned: false tags: - track:wood - sponsor:modal - achievement:offgrid - achievement:offbrand - achievement:llama - achievement:sharing - achievement:fieldnotes license: apache-2.0 thumbnail: >- https://cdn-uploads.huggingface.co/production/uploads/65c10674c4dcb96c31dd84c9/aWVwpGKH963R2e81blgxO.png --- ## Team - WillHbx β€” https://huggingface.co/WillHbx - SuperPoivre13 β€” https://huggingface.co/SuperPoivre13 # hackathon-ia-visualnovel πŸ„ > **Thousand Token Wood** β€” an AI-improvised, voice-or-text **anime visual novel**. > Build Small Hackathon (Chapter Two). Story, characters, and art are generated **live** by > small local models β€” nothing is pre-scripted. You step into a wood that is being *dreamed into existence* around you. Every spirit you meet is conjured on the spot; every backdrop is painted the moment you arrive. Because the wood is dreamed by a small, slightly forgetful mind, it is whimsical and never the same twice β€” **the model's quirks aren't bugs, they're the wood dreaming.** Take the models out and there's no game, only an empty Gradio shell. That's what makes the AI *load-bearing*. --- ## πŸ† Build Small Hackathon submission **Ephemeral Hearts** is an AI-improvised anime dating visual novel: the story, every character, every line of dialogue, every backdrop and sprite, and even the character voices are generated **live** by small local models. Nothing is pre-scripted. Five AI roles collaborate in **under 18B parameters total** behind one golden rule, *the model proposes, code disposes*: the LLM returns a typed `DirectorOutput` (one grammar-constrained call per turn) and deterministic code is the **only** thing that mutates game state. The same `visualnovel/` package runs in three places unchanged, selected by env vars: fully offline on a laptop (`llama.cpp` + `diffusers`), on on-demand Modal GPUs, or on a ZeroGPU Space. **Tech:** Qwen3-14B (the Weaver + the Voices, shared weights) Β· SDXL + a fine-tuned anime LoRA (the Painter) Β· Whisper (the Ear) Β· Kokoro-82M (the Voice) Β· Gradio `gradio.Server` with a bespoke HTML/JS frontend Β· Pydantic-derived JSON grammar Β· `uv`. - πŸŽ₯ **Demo video:** https://youtu.be/NVDTg_oMUY4 - πŸ“ **Blog / field notes:** https://huggingface.co/blog/WillHbx/ephemeral-hearts - πŸ’¬ **Social post:** https://www.linkedin.com/posts/lorenzo-lepoivre_ai-generativeai-llm-share-7471625802258939904-Pjx-/ **Track:** Thousand Token Wood **Sub-tracks:** Off-Brand Β· Best Demo Β· Best Agent Β· Bonus Quest Champion Β· Judges' Wildcard **Badges:** πŸ”Œ Off the Grid Β· 🎨 Off-Brand (Custom UI) Β· πŸ¦™ Llama Champion (llama.cpp) Β· πŸ“‘ Sharing is Caring (Open-Trace) Β· πŸ““ Field Notes --- ## ✨ This repo runs on a fresh checkout β€” with **zero models** It ships in **MOCK mode** (`VN_MOCK=1`, the default): deterministic fake LLM / painter / STT so the **entire loop works offline** with placeholder art. Open it, watch a turn happen, then have Claude Code replace each mock with a real model, one module at a time. See [`CLAUDE.md`](CLAUDE.md). ```bash uv sync # installs only gradio + pydantic + pillow (seconds) uv run python -m visualnovel.smoke # full loop in the terminal, no UI, no models uv run python app.py # custom VN UI β†’ http://localhost:7860 GRADIO_MVP_UI=1 uv run python app.py # plain gr.Blocks fallback (Phase 0/1) ``` --- ## The cast of models (the β€œseveral AIs”) Four *roles*, four *real* model families (text, image, speech-in, speech-out). The two text roles **share one LLM** (same weights, two system prompts) to respect the parameter budget. | In-world name | Role | Backed by | Output | |---|---|---|---| | 🧡 **The Weaver** | director / GM β€” builds the world, keeps canon, decides what changes | LLM (shared) | structured **directives** (JSON) | | 🎭 **The Voices** | actor β€” speaks as the present spirits, in voice | LLM (shared) | in-character **dialogue** + emotion | | 🎨 **The Painter** | renders backdrops + character sprites | **SDXL-Turbo** (+ your anime-style LoRA) | **images** | | πŸ‘‚ **The Ear** | turns spoken words into text | Whisper | transcribed input | | πŸ—£οΈ **The Voice** | speaks the spirits' dialogue aloud (per-character voice) | **Kokoro-82M** (ONNX) | synthesized **audio** | The Weaver and Voices are one **grammar-constrained LLM call per turn** β†’ `{ speaker, dialogue, emotion, directives }`. Code applies the directives deterministically; **the LLM never edits state directly.** --- ## Parameter budget β€” the β€œbeefy” config (≀ 32B total) | Component | Model | ~Params | |---|---|---| | LLM (shared) | **Qwen3-14B** | ~14B | | Image | **SDXL-Turbo** (+ fine-tuned anime LoRA) | ~3.5B | | STT | **Whisper-large-v3-turbo** | ~0.8B | | TTS | **Kokoro-82M** (ONNX) | ~0.08B | | Matting (optional) | BiRefNet | ~0.2B | | **Total** | | **β‰ˆ 18.6B** βœ… | MoE models count by **total** params. Everything is hosted on HF (you have credits), so the only hard limit is ≀ 32B. Override any model via env vars in [`visualnovel/config.py`](visualnovel/config.py). --- ## Setup & install (uv) > **Python 3.12** is pinned in `.python-version` β€” the ML stack (`torch` / > `llama-cpp-python` / `ctranslate2`) ships wheels for it. The package code is 3.11+ clean, > so a newer interpreter works too if every dependency publishes wheels for it. Your hardware is **AMD ROCm + Apple Metal (no local CUDA)**, which changes the GPU build flags: ```bash # ---- Apple Silicon (M3 Max) β€” Metal ---- uv sync --extra image --extra stt CMAKE_ARGS="-DGGML_METAL=on" uv pip install llama-cpp-python # Metal is default on macOS # torch uses the MPS backend automatically. faster-whisper runs on CPU (CTranslate2 has no # Metal); for GPU STT on Mac use mlx-whisper or whisper.cpp instead. # ---- AMD RX 7900 XTX β€” ROCm (Linux) ---- uv pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 # ROCm torch wheel CMAKE_ARGS="-DGGML_HIPBLAS=on" uv pip install llama-cpp-python # newer builds: -DGGML_HIP=on uv sync --extra image --extra stt # torch reports ROCm as "cuda" (so config.detect_device() returns "cuda" on this box). # CTranslate2 has no ROCm β†’ faster-whisper runs on CPU (fine for short clips), or use whisper.cpp. ``` Then download weights (or let them fetch lazily on first real use) and flip off mock mode: ```bash uv run python scripts/download_models.py VN_MOCK=0 uv run python app.py ``` --- ## How a turn works ``` START ─▢ Weaver: dream world + opening scene + first spirit ─▢ Painter: backdrop + sprite β”‚ └─▢ LOOP 1. player speaks/types ─▢ Ear (Whisper) ─▢ text 2. ONE grammar-constrained LLM call ─▢ { speaker, dialogue, emotion, directives } 3. state.apply_directives(...) deterministically (move scene, add/exit spirit, mood, relationship, flags, beat, ending) ─▢ .md dream-memory updated 4. if scene/character changed ─▢ Painter (cached, seed-pinned) 5. render: backdrop + sprite(mood) + dialogue (text streams first; image fills in) 6. The Voice (Kokoro) speaks the spirit's line in its per-character voice (cached) ``` Deep design in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md); prompts + JSON schema + GBNF grammar in [`docs/PROMPTS.md`](docs/PROMPTS.md). --- ## Project structure ``` hackathon-ia-visualnovel/ β”œβ”€β”€ app.py # gradio.Server entry + gr.Blocks MVP fallback (thin) β”œβ”€β”€ pyproject.toml # uv project; heavy ML libs are optional extras β”œβ”€β”€ .python-version # 3.12 (ML stack ships wheels for it) β”œβ”€β”€ frontend/index.html # custom layered VN UI (backdrop Β· sprite Β· dialogue Β· mic) β”œβ”€β”€ visualnovel/ β”‚ β”œβ”€β”€ schemas.py # Pydantic: GameState (truth) + DirectorOutput (LLM contract) β”‚ β”œβ”€β”€ config.py # paths, flags, model registry, THEMES, device detection β”‚ β”œβ”€β”€ prompts.py # system prompts + schema helpers (no scattered literals) β”‚ β”œβ”€β”€ llm.py # MockLLM Β· LlamaCppLLM Β· TransformersLLM behind one interface β”‚ β”œβ”€β”€ orchestrator.py # the Weaver: init_world / direct_turn / compact_memory β”‚ β”œβ”€β”€ characters.py # the Voices: present-character actor context β”‚ β”œβ”€β”€ painter.py # the Painter: prompt compose, cache, MockPainter + SDXL stub β”‚ β”œβ”€β”€ stt.py # the Ear: Whisper (mock + faster-whisper) β”‚ β”œβ”€β”€ tts.py # the Voice: per-character speech (mock + Kokoro-82M ONNX) β”‚ β”œβ”€β”€ state.py # apply_directives (the ONLY mutator) + .md render β”‚ β”œβ”€β”€ memory.py # bounded context assembly + compaction trigger β”‚ β”œβ”€β”€ engine.py # faΓ§ade: start_text/start_images Β· play_turn(_text/_images) β†’ ViewState β”‚ β”œβ”€β”€ utils.py # shared helpers (think-strip, JSON repair, quiet stderr) β”‚ β”œβ”€β”€ metrics.py # debug-mode metrics + live monitor (no-op in mock/prod) β”‚ β”œβ”€β”€ trace.py # per-turn JSONL trace (Open-Trace bonus) β”‚ └── smoke.py # `python -m visualnovel.smoke` β”œβ”€β”€ templates/ # the .md dream-memory views (rendered each turn) β”œβ”€β”€ tests/ # state round-trip Β· directive apply Β· memory budget (mock, green) β”œβ”€β”€ scripts/download_models.py └── docs/ # ARCHITECTURE.md Β· PROMPTS.md ``` --- ## Bonus quests targeted | Badge | How | |---|---| | 🎨 **Off-Brand** (custom UI) | bespoke VN frontend via `gradio.Server` (`frontend/index.html`) | | πŸ¦™ **Llama-Champion** | LLM through `llama-cpp-python` + JSON-schema/GBNF grammar (run **locally** for the video) | | πŸ”Œ **Off the Grid** | all weights local; demoed with the network off | | 🎯 **Well-Tuned** | the **fine-tuned SDXL-Turbo anime LoRA**, published on the Hub (`VN_IMAGE_LORA`) | | πŸ“‘ **Open-Trace** | `VN_TRACE=runs/run.jsonl` records every orchestration step β†’ publish as a dataset | | πŸ““ **Field-Notes** | write up the diegetic conceit + one-call director pattern | > ⚠️ **llama.cpp on the *hosted* ZeroGPU Space is the trickiest bit** β€” claim it **locally** in the > demo video and keep the `transformers` backend (`VN_LLM_BACKEND=transformers`) for the Space. > See [`docs/ARCHITECTURE.md` Β§7](docs/ARCHITECTURE.md). --- ## Quality ```bash uv run ruff check . && uv run ruff format . uv run pytest -q ``` ## Submission checklist - [ ] Registered on the Build-Small org by **June 3** + joined the Gradio Discord. - [ ] Space is live and loads without errors. - [ ] Total params **≀ 32B** (stated). - [ ] It's a Gradio app (`gradio.Server` counts). - [ ] Demo video + social post. Built for the **Build Small Hackathon** (Gradio Γ— Hugging Face, June 2026). Code Apache-2.0 β€” match the licenses of the weights you ship (SDXL-Turbo and Qwen3 are permissive; **check FLUX.2** if you swap the painter).