Spaces:
Running on Zero
Running on Zero
| title: Ephemeral Hearts | |
| emoji: 🐠 | |
| colorFrom: green | |
| colorTo: indigo | |
| sdk: gradio | |
| sdk_version: 6.18.0 | |
| python_version: '3.12' | |
| app_file: app.py | |
| pinned: false | |
| tags: | |
| - track:wood | |
| - sponsor:modal | |
| - achievement:offgrid | |
| - achievement:offbrand | |
| - achievement:llama | |
| - achievement:sharing | |
| - achievement:fieldnotes | |
| license: apache-2.0 | |
| thumbnail: >- | |
| https://cdn-uploads.huggingface.co/production/uploads/65c10674c4dcb96c31dd84c9/aWVwpGKH963R2e81blgxO.png | |
| ## Team | |
| - WillHbx — https://huggingface.co/WillHbx | |
| - SuperPoivre13 — https://huggingface.co/SuperPoivre13 | |
| # hackathon-ia-visualnovel 🍄 | |
| > **Thousand Token Wood** — an AI-improvised, voice-or-text **anime visual novel**. | |
| > Build Small Hackathon (Chapter Two). Story, characters, and art are generated **live** by | |
| > small local models — nothing is pre-scripted. | |
| You step into a wood that is being *dreamed into existence* around you. Every spirit you meet is | |
| conjured on the spot; every backdrop is painted the moment you arrive. Because the wood is dreamed | |
| by a small, slightly forgetful mind, it is whimsical and never the same twice — **the model's | |
| quirks aren't bugs, they're the wood dreaming.** Take the models out and there's no game, only an | |
| empty Gradio shell. That's what makes the AI *load-bearing*. | |
| --- | |
| ## 🏆 Build Small Hackathon submission | |
| **Ephemeral Hearts** is an AI-improvised anime dating visual novel: the story, every character, | |
| every line of dialogue, every backdrop and sprite, and even the character voices are generated | |
| **live** by small local models. Nothing is pre-scripted. Five AI roles collaborate in **under 18B | |
| parameters total** behind one golden rule, *the model proposes, code disposes*: the LLM returns a | |
| typed `DirectorOutput` (one grammar-constrained call per turn) and deterministic code is the **only** | |
| thing that mutates game state. The same `visualnovel/` package runs in three places unchanged, | |
| selected by env vars: fully offline on a laptop (`llama.cpp` + `diffusers`), on on-demand Modal | |
| GPUs, or on a ZeroGPU Space. | |
| **Tech:** Qwen3-14B (the Weaver + the Voices, shared weights) · SDXL + a fine-tuned anime LoRA (the | |
| Painter) · Whisper (the Ear) · Kokoro-82M (the Voice) · Gradio `gradio.Server` with a bespoke | |
| HTML/JS frontend · Pydantic-derived JSON grammar · `uv`. | |
| - 🎥 **Demo video:** https://youtu.be/NVDTg_oMUY4 | |
| - 📝 **Blog / field notes:** https://huggingface.co/blog/WillHbx/ephemeral-hearts | |
| - 💬 **Social post:** https://www.linkedin.com/posts/lorenzo-lepoivre_ai-generativeai-llm-share-7471625802258939904-Pjx-/ | |
| **Track:** Thousand Token Wood | |
| **Sub-tracks:** Off-Brand · Best Demo · Best Agent · Bonus Quest Champion · Judges' Wildcard | |
| **Badges:** 🔌 Off the Grid · 🎨 Off-Brand (Custom UI) · 🦙 Llama Champion (llama.cpp) · | |
| 📡 Sharing is Caring (Open-Trace) · 📓 Field Notes | |
| --- | |
| ## ✨ This repo runs on a fresh checkout — with **zero models** | |
| It ships in **MOCK mode** (`VN_MOCK=1`, the default): deterministic fake LLM / painter / STT so the | |
| **entire loop works offline** with placeholder art. Open it, watch a turn happen, then have Claude | |
| Code replace each mock with a real model, one module at a time. See [`CLAUDE.md`](CLAUDE.md). | |
| ```bash | |
| uv sync # installs only gradio + pydantic + pillow (seconds) | |
| uv run python -m visualnovel.smoke # full loop in the terminal, no UI, no models | |
| uv run python app.py # custom VN UI → http://localhost:7860 | |
| GRADIO_MVP_UI=1 uv run python app.py # plain gr.Blocks fallback (Phase 0/1) | |
| ``` | |
| --- | |
| ## The cast of models (the “several AIs”) | |
| Four *roles*, four *real* model families (text, image, speech-in, speech-out). The two text roles | |
| **share one LLM** (same weights, two system prompts) to respect the parameter budget. | |
| | In-world name | Role | Backed by | Output | | |
| |---|---|---|---| | |
| | 🧵 **The Weaver** | director / GM — builds the world, keeps canon, decides what changes | LLM (shared) | structured **directives** (JSON) | | |
| | 🎭 **The Voices** | actor — speaks as the present spirits, in voice | LLM (shared) | in-character **dialogue** + emotion | | |
| | 🎨 **The Painter** | renders backdrops + character sprites | **SDXL-Turbo** (+ your anime-style LoRA) | **images** | | |
| | 👂 **The Ear** | turns spoken words into text | Whisper | transcribed input | | |
| | 🗣️ **The Voice** | speaks the spirits' dialogue aloud (per-character voice) | **Kokoro-82M** (ONNX) | synthesized **audio** | | |
| The Weaver and Voices are one **grammar-constrained LLM call per turn** → `{ speaker, dialogue, | |
| emotion, directives }`. Code applies the directives deterministically; **the LLM never edits state | |
| directly.** | |
| --- | |
| ## Parameter budget — the “beefy” config (≤ 32B total) | |
| | Component | Model | ~Params | | |
| |---|---|---| | |
| | LLM (shared) | **Qwen3-14B** | ~14B | | |
| | Image | **SDXL-Turbo** (+ fine-tuned anime LoRA) | ~3.5B | | |
| | STT | **Whisper-large-v3-turbo** | ~0.8B | | |
| | TTS | **Kokoro-82M** (ONNX) | ~0.08B | | |
| | Matting (optional) | BiRefNet | ~0.2B | | |
| | **Total** | | **≈ 18.6B** ✅ | | |
| MoE models count by **total** params. Everything is hosted on HF (you have credits), so the only | |
| hard limit is ≤ 32B. Override any model via env vars in [`visualnovel/config.py`](visualnovel/config.py). | |
| --- | |
| ## Setup & install (uv) | |
| > **Python 3.12** is pinned in `.python-version` — the ML stack (`torch` / | |
| > `llama-cpp-python` / `ctranslate2`) ships wheels for it. The package code is 3.11+ clean, | |
| > so a newer interpreter works too if every dependency publishes wheels for it. | |
| Your hardware is **AMD ROCm + Apple Metal (no local CUDA)**, which changes the GPU build flags: | |
| ```bash | |
| # ---- Apple Silicon (M3 Max) — Metal ---- | |
| uv sync --extra image --extra stt | |
| CMAKE_ARGS="-DGGML_METAL=on" uv pip install llama-cpp-python # Metal is default on macOS | |
| # torch uses the MPS backend automatically. faster-whisper runs on CPU (CTranslate2 has no | |
| # Metal); for GPU STT on Mac use mlx-whisper or whisper.cpp instead. | |
| # ---- AMD RX 7900 XTX — ROCm (Linux) ---- | |
| uv pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 # ROCm torch wheel | |
| CMAKE_ARGS="-DGGML_HIPBLAS=on" uv pip install llama-cpp-python # newer builds: -DGGML_HIP=on | |
| uv sync --extra image --extra stt | |
| # torch reports ROCm as "cuda" (so config.detect_device() returns "cuda" on this box). | |
| # CTranslate2 has no ROCm → faster-whisper runs on CPU (fine for short clips), or use whisper.cpp. | |
| ``` | |
| Then download weights (or let them fetch lazily on first real use) and flip off mock mode: | |
| ```bash | |
| uv run python scripts/download_models.py | |
| VN_MOCK=0 uv run python app.py | |
| ``` | |
| --- | |
| ## How a turn works | |
| ``` | |
| START ─▶ Weaver: dream world + opening scene + first spirit ─▶ Painter: backdrop + sprite | |
| │ | |
| └─▶ LOOP | |
| 1. player speaks/types ─▶ Ear (Whisper) ─▶ text | |
| 2. ONE grammar-constrained LLM call ─▶ { speaker, dialogue, emotion, directives } | |
| 3. state.apply_directives(...) deterministically (move scene, add/exit spirit, mood, | |
| relationship, flags, beat, ending) ─▶ .md dream-memory updated | |
| 4. if scene/character changed ─▶ Painter (cached, seed-pinned) | |
| 5. render: backdrop + sprite(mood) + dialogue (text streams first; image fills in) | |
| 6. The Voice (Kokoro) speaks the spirit's line in its per-character voice (cached) | |
| ``` | |
| Deep design in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md); prompts + JSON schema + GBNF grammar | |
| in [`docs/PROMPTS.md`](docs/PROMPTS.md). | |
| --- | |
| ## Project structure | |
| ``` | |
| hackathon-ia-visualnovel/ | |
| ├── app.py # gradio.Server entry + gr.Blocks MVP fallback (thin) | |
| ├── pyproject.toml # uv project; heavy ML libs are optional extras | |
| ├── .python-version # 3.12 (ML stack ships wheels for it) | |
| ├── frontend/index.html # custom layered VN UI (backdrop · sprite · dialogue · mic) | |
| ├── visualnovel/ | |
| │ ├── schemas.py # Pydantic: GameState (truth) + DirectorOutput (LLM contract) | |
| │ ├── config.py # paths, flags, model registry, THEMES, device detection | |
| │ ├── prompts.py # system prompts + schema helpers (no scattered literals) | |
| │ ├── llm.py # MockLLM · LlamaCppLLM · TransformersLLM behind one interface | |
| │ ├── orchestrator.py # the Weaver: init_world / direct_turn / compact_memory | |
| │ ├── characters.py # the Voices: present-character actor context | |
| │ ├── painter.py # the Painter: prompt compose, cache, MockPainter + SDXL stub | |
| │ ├── stt.py # the Ear: Whisper (mock + faster-whisper) | |
| │ ├── tts.py # the Voice: per-character speech (mock + Kokoro-82M ONNX) | |
| │ ├── state.py # apply_directives (the ONLY mutator) + .md render | |
| │ ├── memory.py # bounded context assembly + compaction trigger | |
| │ ├── engine.py # façade: start_text/start_images · play_turn(_text/_images) → ViewState | |
| │ ├── utils.py # shared helpers (think-strip, JSON repair, quiet stderr) | |
| │ ├── metrics.py # debug-mode metrics + live monitor (no-op in mock/prod) | |
| │ ├── trace.py # per-turn JSONL trace (Open-Trace bonus) | |
| │ └── smoke.py # `python -m visualnovel.smoke` | |
| ├── templates/ # the .md dream-memory views (rendered each turn) | |
| ├── tests/ # state round-trip · directive apply · memory budget (mock, green) | |
| ├── scripts/download_models.py | |
| └── docs/ # ARCHITECTURE.md · PROMPTS.md | |
| ``` | |
| --- | |
| ## Bonus quests targeted | |
| | Badge | How | | |
| |---|---| | |
| | 🎨 **Off-Brand** (custom UI) | bespoke VN frontend via `gradio.Server` (`frontend/index.html`) | | |
| | 🦙 **Llama-Champion** | LLM through `llama-cpp-python` + JSON-schema/GBNF grammar (run **locally** for the video) | | |
| | 🔌 **Off the Grid** | all weights local; demoed with the network off | | |
| | 🎯 **Well-Tuned** | the **fine-tuned SDXL-Turbo anime LoRA**, published on the Hub (`VN_IMAGE_LORA`) | | |
| | 📡 **Open-Trace** | `VN_TRACE=runs/run.jsonl` records every orchestration step → publish as a dataset | | |
| | 📓 **Field-Notes** | write up the diegetic conceit + one-call director pattern | | |
| > ⚠️ **llama.cpp on the *hosted* ZeroGPU Space is the trickiest bit** — claim it **locally** in the | |
| > demo video and keep the `transformers` backend (`VN_LLM_BACKEND=transformers`) for the Space. | |
| > See [`docs/ARCHITECTURE.md` §7](docs/ARCHITECTURE.md). | |
| --- | |
| ## Quality | |
| ```bash | |
| uv run ruff check . && uv run ruff format . | |
| uv run pytest -q | |
| ``` | |
| ## Submission checklist | |
| - [ ] Registered on the Build-Small org by **June 3** + joined the Gradio Discord. | |
| - [ ] Space is live and loads without errors. | |
| - [ ] Total params **≤ 32B** (stated). | |
| - [ ] It's a Gradio app (`gradio.Server` counts). | |
| - [ ] Demo video + social post. | |
| Built for the **Build Small Hackathon** (Gradio × Hugging Face, June 2026). Code Apache-2.0 — match | |
| the licenses of the weights you ship (SDXL-Turbo and Qwen3 are permissive; **check FLUX.2** if you | |
| swap the painter). |