Spaces:
Running on Zero
Running on Zero
| title: "How We Built Ephemeral Hearts - a Live-Improvised Anime Visual Novel with Several Small AIs" | |
| thumbnail: /blog/assets/ephemeral-hearts/thumbnail.png | |
| authors: | |
| - William Hubaux ; Github, [WillHCode](https://github.com/WillHCode) | |
| - Lorenzo Lepoivre ; Github, [SupPepper](https://github.com/LorenzoLepoivre) | |
| date: "2026-06-09" | |
| tags: | |
| - hackathon | |
| - visual-novel | |
| - multi-agent | |
| - diffusion | |
| - gradio | |
| - modal | |
| - lora | |
| # How We Built Ephemeral Hearts - a Live-Improvised Anime VN with Several Small AIs | |
| *Build Small Hackathon - June 2026* | |
| --- | |
| ## Table of contents | |
| 1. [Who we are and why we built this](#who-we-are) | |
| 2. [The multi-agent setup: three roles, three model families](#multi-agent) | |
| 3. [What we discovered along the way](#discoveries) | |
| 4. [The "model proposes, code disposes" pattern](#pattern) | |
| 5. [Features that emerged naturally](#features) | |
| 6. [The journey from local to Modal](#local-to-modal) | |
| 7. [One codebase, three runtimes: local, Modal, Space](#three-runtimes) | |
| 8. [llama.cpp vs transformers vs ZeroGPU: what actually changes](#backends) | |
| 9. [Architecture, in depth](#architecture) | |
| 10. [Field notes: the problems we hit and how we fixed them](#field-notes) | |
| 11. [Stack](#stack) | |
| 12. [What this could become](#vision) | |
| --- | |
| ## Who we are and why we built this <a name="who-we-are"></a> | |
| We are two developers, and we both already work on a production AI project with RAG pipeline, local models via HuggingFace Transformers, multi-agent LLM system - all aimed at real business use cases. So we were not complete beginners when it came to language models. But this hackathon was something different: how do you put all of this together in a **fun** way, with creative constraints, and explore territories we had not seriously touched yet - image generation, voice models, fine-tuning diffusion models with LoRA, and Modal for on-demand GPU deployment. | |
| The base idea: an anime visual novel where *everything* is generated live. No pre-written script, no fixed assets. Every character, every backdrop, every line of dialogue - conjured on the fly by small models. The hackathon imposed ≤ 32B total parameters and a Gradio app on a HF Space. Perfect. We figured that if the constraints forced us to be creative about the architecture, we might as well build one we would be proud to explain. | |
| The result: **Ephemeral Hearts** - an anime dating VN where the story, dialogue, characters, and art are all generated in real time. Nothing is pre-scripted. | |
| --- | |
| ## The multi-agent setup: three roles, three model families <a name="multi-agent"></a> | |
| The first real design question was: how many models, which ones, and how do you make them collaborate? | |
| We defined three distinct **roles**, each handled by a different model family: | |
| > **🧵 The Weaver** - the game director. It reads the current scene, the character sheets, the recent conversation, and the player's input. It decides what changes structurally this turn: does the scene shift? Does a new character arrive? What is the relationship delta? It outputs a structured JSON block of *directives*. | |
| > **🎭 The Voices** - the actor. It speaks *as* the active character - their speech quirks, their current mood, their secrets. One to three sentences, anime style, never breaking the fourth wall. | |
| > **🎨 The Painter** - it generates backdrops and character sprites. Default backend: SDXL-base-1.0 distilled down to 4 inference steps with the ByteDance SDXL-Lightning LoRA (an SDXL-Turbo backend exists as an alternative, and `VN_IMAGE_LORA` accepts a custom style LoRA on top). | |
| > **👂 The Ear** - voice transcription. Whisper (large-v3-turbo by default), CPU via faster-whisper, good enough for short clips. | |
| > **🔊 The Larynx** - speech synthesis. Kokoro-82M via ONNX gives every character a voice, picked once at creation from their written voice description and frozen forever (same character = same voice). | |
| The key subtlety: **The Weaver and The Voices share a single LLM** (Qwen3-14B). Same weights, two system prompts, one grammar-constrained call per turn. They write the stage directions and the dialogue lines simultaneously. A second LLM call per turn would have doubled perceived latency - and the two roles are so tightly coupled that sharing context is actually an advantage. | |
| ### Architecture diagram | |
| ``` | |
| Player input (text or voice) | |
| | | |
| v | |
| +-------------------+ | |
| | THE EAR 👂 | Whisper-small | |
| | voice -> text | (CPU - faster-whisper) | |
| +--------+----------+ | |
| | | |
| v | |
| memory.assemble_context() | |
| +---------------------------------------------+ | |
| | summary + current scene + character sheets | | |
| | + last k exchanges (fixed token budget) | | |
| +------------------+--------------------------+ | |
| | | |
| v | |
| +------------------------------------------------------+ | |
| | ONE grammar-constrained call | | |
| | | | |
| | THE WEAVER 🧵 (director) | | |
| | + THE VOICES 🎭 (actor) | | |
| | | | |
| | Qwen3-14B | | |
| | +-- local : llama.cpp (Metal / ROCm) | | |
| | +-- cloud : Modal A10G (VN_LLM_BACKEND=modal) | | |
| +----------------------+-------------------------------+ | |
| | | |
| v | |
| DirectorOutput | |
| +---------------------------------+ | |
| | dialogue . emotion | | |
| | scene_change | | |
| | relationship_delta | | |
| | new_character / exit_char | | |
| | set_music . ending | | |
| | npc_relation_deltas | | |
| +--------------+-----------------+ | |
| | | |
| v | |
| state.apply_directives() | |
| <- THE ONLY MUTATOR | |
| GameState updated | |
| .md dream-memory written | |
| | | |
| +------------+-------------+ | |
| v v | |
| +-----------------+ +-----------------+ | |
| | THE PAINTER 🎨 | | ViewState | | |
| | | | -> frontend | | |
| | SDXL-Turbo | | (Gradio Server) | | |
| | + anime LoRA | +-----------------+ | |
| | | | |
| | +- local : | | |
| | | diffusers | | |
| | | (Metal/ROCm)| | |
| | +- cloud : | | |
| | Modal A10G | | |
| +--------+--------+ | |
| | | |
| v | |
| backdrop.png (cached by seed) | |
| sprite.png (cached by character + mood) | |
| ``` | |
| **Total: ≈18B parameters. Well under the 32B budget.** | |
| | Role | Model | Params | | |
| |---|---|---| | |
| | The Weaver + The Voices (shared) | Qwen3-14B | ~14B | | |
| | The Painter | SDXL-base-1.0 + Lightning LoRA | ~3.5B | | |
| | The Ear | Whisper large-v3-turbo | ~0.8B | | |
| | The Larynx | Kokoro-82M (TTS) | ~0.08B | | |
| | **Total** | | **≈18B** | | |
| --- | |
| ## What we discovered along the way <a name="discoveries"></a> | |
| We already had experience with LLMs and HuggingFace. But this project pushed us into three territories we had not seriously explored before. | |
| ### Image models - SDXL-Turbo and LoRA | |
| This is clearly what surprised us the most. We knew image generation existed, we had played with online demos, but actually integrating it into a real pipeline - managing prompts, seeds, caching, negative prompts, and visual consistency - is a genuine craft in itself. | |
| Choosing a few-step distilled model over a full diffusion pipeline was deliberate: 1 to 4 inference steps instead of 50. Slightly lower quality, but in a VN where the image arrives *while* the player is reading the dialogue, "fast and impressionistic" beats "slow and polished" every single time. We started with SDXL-Turbo and eventually settled on SDXL-base-1.0 + the ByteDance Lightning LoRA as the default - same philosophy, noticeably better quality at 4 steps (both backends remain in the code). | |
| What we had never done before: **LoRA fine-tuning**. We trained an anime LoRA on SDXL-Turbo to lock in the visual style - color palette, character rendering, line quality. The result: even as prompts vary wildly from scene to scene, the style stays consistent. That is exactly what a VN needs. The LoRA is published on the Hub (`VN_IMAGE_LORA`) and loads automatically at startup. | |
| Another revelation: **seed pinning**. Every character sprite is generated **once** per mood, with a fixed seed, and cached forever. Same character, same scene, same expression = same image, always. The visual consistency this gives does more for immersion than a higher-quality model would. | |
| ### Voice models - Whisper | |
| Whisper-small for voice input was the easiest onboarding. CTranslate2 via faster-whisper, CPU, works on both our machines without a dedicated GPU. The only gotcha: CTranslate2 has no ROCm backend, so on our AMD box it runs on CPU regardless. For 5-10 second clips it is more than sufficient. | |
| ### Modal - GPU on demand | |
| This was the real infrastructure discovery of this hackathon. | |
| We were used to deploying on HF Spaces or fixed GPU servers. Modal is a completely different mental model: you write a Python function, decorate it with `@app.function(gpu="A10G")`, and Modal handles everything else - containerization, scaling, cold starts, per-second billing. For a hackathon where you have no idea what load you will get, it is perfect. | |
| The Gradio Space stays lightweight (CPU only), model calls go out to Modal A10G containers. The `VN_LLM_BACKEND=modal` environment variable switches the backend without touching any game logic. Same interface, different implementations. | |
| --- | |
| ## The "model proposes, code disposes" pattern <a name="pattern"></a> | |
| The naive approach: let the LLM generate free prose and parse the result. This breaks immediately with small models - they drift, contradict themselves, forget who is on stage. | |
| Our approach: every turn is one grammar-constrained call that returns a typed Pydantic `DirectorOutput`: | |
| ```python | |
| # Schema is derived directly from the Pydantic model - can never drift | |
| out = llm.complete_json(schema=DirectorOutput.model_json_schema(), prompt=context) | |
| # The only place in the entire codebase that touches GameState | |
| effects = state.apply_directives(game_state, out) | |
| ``` | |
| `apply_directives` clamps values, validates references, and silently ignores impossible requests (like removing a character who is not on stage). The LLM never touches state directly - it proposes, the code decides. | |
| Direct consequence: the entire game loop was testable from day one with `VN_MOCK=1` - deterministic fake LLM, fake painter, zero real models. We built the NPC bond graph, the music system, the ending screens, and the relationship milestones entirely in mock mode before a single real model call ever ran. | |
| --- | |
| ## Features that emerged naturally <a name="features"></a> | |
| Once the directive pattern was solid, features fell out almost on their own. | |
| **NPC-to-NPC relationship graph.** The same `npc_relation_deltas` directive that tracks player-NPC affection also tracks relationships *between* NPCs. Jealousy, rivalry, camaraderie - all of this emerges naturally from the model's tendency to give characters an inner life. We expose it as a directed graph in the Relations tab, with one-word labels ("jealousy", "rivalry", "fond of") coming directly from the model. | |
| **Dynamic music system.** Six tracks (calm, romantic, dramatic, mystery, sad, joyful), crossfade on change, all controlled by a `set_music` directive. The model knows the available tracks and only switches for genuine tonal ruptures - not every mood shift. In practice it is surprisingly restrained. | |
| **Generated ending screens.** When a relationship hits +100 (romantic confession) or -100 (a falling-out that empties the stage), the model emits an `ending` directive with a poetic 2-4 sentence epilogue. The Painter generates a dedicated illustration - cherry blossoms at golden hour for the warm ending, a rain-soaked empty park bench at night for defeat. No pre-authored cutscenes. The ending art is improvised just like everything else. | |
| **Characters that remember you.** A `remember_fact` directive lets the speaker store one short note when the player reveals something personal ("the player plays violin", "the player is named Lorenzo"). Facts are capped at 8 per character and re-injected into their sheet every turn they are on stage - so a character you met ten scenes ago still knows your name. The same progression system unlocks personality traits at affection thresholds (20/40/60), reveals the secret goal at 80, and stages a one-shot "growing close" moment at 50 - all deterministic, all driven by the relationship value the model proposes. | |
| **Quality-of-life.** The player can set their name in the setup form (characters address them by it), and a session persisted server-side after every turn powers a "Continue last story" button - including a rebuilt journal with a condensed recap of everything that got compacted away. | |
| --- | |
| ## The journey from local to Modal <a name="local-to-modal"></a> | |
| We started on our two machines: an Apple M3 Max (Metal backend) and an AMD RX 7900 XTX (ROCm). First surprising fact: ROCm reports itself as `"cuda"` to PyTorch - so our device detection works on both machines without any special-casing. Second surprise: CTranslate2 has no ROCm backend, so Whisper runs on CPU on the AMD machine regardless. | |
| For the HF Space we first needed GPU inference without going through ZeroGPU (llama.cpp + ZeroGPU is notoriously unreliable). The solution: Modal for heavy model calls, Gradio for the UI. The Space stays CPU, GPU compute goes to on-demand Modal containers. We later added a third path - running the models directly on ZeroGPU with `transformers` - which turned out to have its own fascinating constraints (see the next two sections). | |
| One unexpected practical problem: HF Spaces now uses Xet storage and refuses binary files pushed through git. We had to soft-reset the commit that included the music MP3 files, add them to `.gitignore`, push the clean code, then upload the audio files via `HfApi().upload_file()`. Lesson learned the hard way: code goes through git, binary assets go through the Hub API. | |
| --- | |
| ## One codebase, three runtimes: local, Modal, Space <a name="three-runtimes"></a> | |
| The same `visualnovel/` package runs in three very different places. Nothing in the game logic knows which one it is in - the backends are selected by `config.py` from environment variables, and every backend implements the same two-method interface (`complete` / `complete_json` for the LLM, `_render` for the Painter). | |
| | | **local** | **modal** | **space (ZeroGPU)** | | |
| |---|---|---|---| | |
| | Where the app runs | your machine | your machine (UI) + Modal (GPU) | HF Space | | |
| | LLM engine | llama.cpp, GGUF Q4_K_M, Metal/ROCm | llama.cpp, GGUF Q8_0, A10G container | transformers, full bf16 weights | | |
| | Painter | diffusers in-process | A10G container (RPC) | diffusers in-process | | |
| | Selected by | default | `VN_LLM_BACKEND=modal` | `SPACE_ID` env var (auto-detected) | | |
| | Game state | in-process | in-process | `/tmp` JSON file (workers are stateless) | | |
| | Cold start cost | model load at launch | ~15-30 s container spin-up, then kept warm 10 min | per-worker lazy load inside the first GPU call | | |
| A few design decisions make this work: | |
| - **Auto-detection over configuration.** HF injects `SPACE_ID` into every Space, so `config.py` flips the LLM backend to `transformers` automatically when deployed - a fresh clone needs zero setup in any environment. Setting `VN_LLM_BACKEND=modal` also chains the Painter to Modal by default, because if you don't have a local GPU for the LLM you don't have one for SDXL either. | |
| - **Modal calls are plain RPC.** `ModalLLM` is a ~20-line proxy: it looks up the deployed class with `modal.Cls.from_name(...)` and calls `.remote()`. The JSON schema travels with every call, which has a lovely property: changing prompts, schemas, or game logic never requires a redeploy - only changes to the container code itself do. | |
| - **Keep-warm matters more than raw speed.** Reloading a 15 GB GGUF mid-conversation is a worse experience than any per-token slowness, so the Modal LLM container keeps a 10-minute `scaledown_window` and the app fires a fire-and-forget warmup ping at server start. | |
| --- | |
| ## llama.cpp vs transformers vs ZeroGPU: what actually changes <a name="backends"></a> | |
| These three names live at different levels - two are inference engines, one is a runtime - and each one reshaped a different part of the code. | |
| ### llama.cpp: the grammar is the contract | |
| The killer feature for this project: `llama-cpp-python` compiles a JSON schema into a **GBNF grammar** that constrains decoding token by token. The model *physically cannot* emit malformed JSON, invent keys, or put a string where an integer belongs. Our whole "model proposes, code disposes" pattern leans on this. | |
| It also exposes full sampling control (`temperature`, `top_p`, `presence_penalty` - the latter became our anti-repetition weapon). Two sharp edges, though: | |
| - The grammar cannot *finish* a document when `max_tokens` runs out mid-string. You get perfectly-shaped-but-truncated JSON, and `json.loads` explodes. We wrote a small `close_truncated_json()` repair (close the open string, drop the dangling comma, balance the brackets) so a runaway generation costs a few default-valued fields instead of a crashed turn. | |
| - Qwen3's thinking mode is baked into the GGUF chat template and there is no API switch - the only control is the `/no_think` soft switch appended to the user message. | |
| ### transformers: no grammar, so trust but verify | |
| On the Space there is no grammar constraint. Instead we derive a **JSON skeleton from the same Pydantic schema** and inject it into the system prompt ("Respond with ONLY a valid JSON object. Required structure: ..."), then parse with up to 3 retries and a regex extraction that tolerates markdown fences and stray prose. | |
| Two gotchas that cost us real debugging time: | |
| - `model.generate()` **silently ignores** `temperature` and `top_p` unless you also pass `do_sample=True`. For weeks the Space was running greedy decoding while we thought we were sampling at 0.7 - and the 3 retries were deterministic, so they retried into the exact same failure. | |
| - Qwen3 sometimes puts its entire answer *inside* the `<think>` block and emits nothing after it. `apply_chat_template(..., enable_thinking=False)` suppresses thinking for structured output, with a fallback that searches inside the think block if nothing comes after it. | |
| ### ZeroGPU: a runtime, not a backend | |
| ZeroGPU is the part that changes your *architecture* rather than your inference code. Each `@spaces.GPU` call can be dispatched to a **different worker subprocess** - in-memory state simply does not survive between two HTTP calls. Three consequences: | |
| 1. **State lives on disk.** After every mutation, `GameState` is serialized to `/tmp/vn_game_state.json` with an atomic write-then-rename. Every endpoint starts by rehydrating from that file if its own memory is empty. (Bonus: this is exactly the mechanism that later powered the "Continue last story" button.) | |
| 2. **Models load lazily.** Loading a 14B model at import time would happen on a CPU-only web worker; instead every backend loads inside the first GPU-decorated call. | |
| 3. **Turns are split in two phases** (`/turn_text` then `/turn_images`) so the dialogue can be displayed while the slower image phase runs - and the pending `DirectorOutput` rides along in the state file, because phase 2 might run on a different worker than phase 1. | |
| --- | |
| ## Architecture, in depth <a name="architecture"></a> | |
| The package is a strict one-way dependency graph - schemas at the bottom, the engine façade at the top, `app.py` as a thin HTTP shell. The golden rule sits in the middle: the LLM returns a typed `DirectorOutput`, and `state.apply_directives` is the **only** function in the codebase allowed to mutate `GameState`. | |
| ```mermaid | |
| flowchart TD | |
| UI["frontend/index.html<br/>(custom VN UI, Gradio JS client)"] | |
| APP["app.py - gradio.Server<br/>thin endpoints: /start_text /start_images<br/>/turn_text /turn_images /transcribe /resume"] | |
| ENG["engine.py<br/>the façade - owns the session"] | |
| subgraph CORE["visualnovel/ - testable without a server"] | |
| ORC["orchestrator.py<br/>the Weaver: init / direct_turn / compact"] | |
| MEM["memory.py<br/>context budget + trimming"] | |
| CH["characters.py<br/>present-character sheets"] | |
| PRM["prompts.py<br/>all prompts + JSON schemas"] | |
| LLM["llm.py<br/>Mock / LlamaCpp / Transformers / Modal"] | |
| ST["state.py<br/>apply_directives - THE ONLY MUTATOR"] | |
| PAINT["painter.py<br/>prompt compose + cache + render"] | |
| STT["stt.py - Whisper"] | |
| TTS["tts.py - Kokoro"] | |
| end | |
| UI -->|HTTP| APP --> ENG | |
| ENG --> ORC | |
| ORC --> MEM --> CH | |
| ORC --> PRM | |
| ORC --> LLM | |
| ENG --> ST | |
| ENG --> PAINT | |
| ENG --> STT | |
| ENG --> TTS | |
| ``` | |
| A full turn, with the two-phase split that keeps the dialogue snappy: | |
| ```mermaid | |
| sequenceDiagram | |
| autonumber | |
| participant P as Player (browser) | |
| participant A as app.py | |
| participant E as Engine | |
| participant W as Weaver (one LLM call) | |
| participant S as apply_directives | |
| participant PA as Painter | |
| participant T as Kokoro TTS | |
| P->>A: POST /turn_text ("Hello, you are so pretty today") | |
| A->>E: play_turn_text() | |
| Note over E: optional: Whisper if voice input | |
| E->>W: complete_json(context, directive schema) | |
| W-->>E: DirectorOutput (dialogue, emotion, directives) | |
| E->>S: apply_directives - clamps, validates, mutates GameState | |
| E->>E: save .md views + /tmp state (ZeroGPU workers) | |
| E-->>P: text-only ViewState - dialogue appears NOW | |
| P->>A: POST /turn_images | |
| A->>E: play_turn_images() | |
| E->>PA: backdrop(scene) + sprite(char, mood) | |
| Note over PA: cache hit by (kind, prompt, seed)?<br/>then 0 ms, else 4-step SDXL render | |
| E->>T: synthesize(dialogue, frozen voice) | |
| E-->>P: full ViewState - backdrop, sprites, audio | |
| ``` | |
| And the deployment picture - one engine, three homes: | |
| ```mermaid | |
| flowchart LR | |
| APP["app.py + visualnovel/<br/>(identical code everywhere)"] | |
| subgraph L["LOCAL - Mac M3 Max / AMD RX 7900 XTX"] | |
| L1["llama.cpp - Qwen3-14B Q4_K_M<br/>Metal / ROCm, grammar-constrained"] | |
| L2["diffusers - SDXL + Lightning LoRA"] | |
| end | |
| subgraph M["MODAL - on-demand A10G"] | |
| M1["ModalLLMBackend<br/>llama.cpp - Q8_0, kept warm 10 min"] | |
| M2["ModalPainterBackend<br/>SDXL + Lightning, rembg"] | |
| end | |
| subgraph Z["HF SPACE - ZeroGPU"] | |
| Z1["TransformersLLM - Qwen3-14B bf16<br/>JSON skeleton + retries"] | |
| Z2["diffusers in-process"] | |
| Z3[("/tmp state file<br/>(stateless workers)")] | |
| end | |
| APP -->|"default"| L | |
| APP -->|"VN_LLM_BACKEND=modal<br/>.remote() RPC"| M | |
| APP -->|"SPACE_ID detected<br/>@spaces.GPU"| Z | |
| ``` | |
| Two invariants hold everywhere: | |
| - **One LLM call per turn** (a guarded retry is allowed for repetition or a missed mandatory directive - never a loop). | |
| - **Every Painter render is cached** by `(kind, prompt, seed)`. A character's sprite is generated once per mood and reused forever; seeds are pinned per entity so the same character always has the same face. | |
| --- | |
| ## Field notes: the problems we hit and how we fixed them <a name="field-notes"></a> | |
| This is the section we wish we had read before starting. Almost every bug below was invisible in the code and obvious in a **saved game file** - our best debugging tool turned out to be exporting the save JSON after each playtest and actually reading it. | |
| ### 1. Anything optional in the grammar is something the model will skip | |
| The single biggest lesson of the project. Our `DirectorOutput` schema had `emotion`, `relationship_delta`, and `new_character` as optional fields with defaults - and the grammar therefore allowed the model to omit them. So it did. Systematically. The symptoms looked like three unrelated gameplay bugs: | |
| - every character stayed `"neutral"` forever (so the mood-keyed sprite system never showed a second expression), | |
| - affection stayed at 0 no matter how hard the player flirted (or insulted), | |
| - the "look around" action politely refused to introduce anyone. | |
| We tried prompt rules ("you MUST emit new_character"), then an explicit retry quoting the instruction. The model ignored both often enough to ruin the demo. The fix that actually worked: **promote the fields to `required` in the JSON schema** handed to the grammar. For "look around" we went further and built a dedicated schema variant where `new_character` is required *and non-nullable* - introduction guaranteed by construction. Prompt engineering is a suggestion; grammar is a constraint. | |
| ### 2. Instructions inside quoted player speech get read as... player speech | |
| Our action hints ("*the wanderer looks around...*") were concatenated with the player's text and injected as `THE WANDERER NOW SAYS: "<hint> <text>"`. Inside the quotes, the model treated our stage directions as something the player said out loud - and ignored them. Moving the hints to a separate `DIRECTOR NOTE (mandatory):` block outside the quotes changed compliance dramatically. Placement in the context is as important as wording. | |
| ### 3. Qwen3's thinking mode ate our memory summarizer | |
| The rolling summary ("THE TALE SO FAR") is rewritten every ~12 turns by a small free-text LLM call with a 320-token budget. On llama.cpp, Qwen3's chat template enables thinking by default - and the model happily spent the *entire* budget inside an unclosed `<think>` block. Our summary was an empty string for whole sessions, silently replaced by a raw-dialogue fallback, and we only noticed by reading a save file where the "summary" was a concatenation cut mid-sentence. Fix: the `/no_think` soft switch appended to the message, `strip_think()` on every free-text output, and a fallback that never degrades the existing summary. | |
| ### 4. Truncated JSON should cost fields, not turns | |
| One playtest produced a character whose `goals` field was a 5,000-character runaway sentence - which blew past `max_tokens` and crashed the turn with `json.decoder.JSONDecodeError: Unterminated string`. Three layers later: generated character fields are word-capped at creation *and* at context-injection time, `close_truncated_json()` repairs cut-off output, and `direct_turn` has a last-resort fallback line ("Hm? Sorry - I lost my train of thought...") so the worst possible outcome of a bad sample is one bland reply. | |
| ### 5. Small models loop, and they loop harder once a loop is in the context | |
| A character repeated the same line verbatim three turns in a row - despite a prompt rule forbidding exactly that. Once one repeat lands in the RECENT EXCHANGE window, it biases the next generation toward repeating again. Code-side guard: normalize and compare the new dialogue against the last 3 turns (`difflib` ratio >= 0.95), and on a hit, retry once at higher temperature with the forbidden line quoted back and `presence_penalty=0.8`. | |
| ### 6. Pacing has to be enforced in both directions | |
| Story beats (`opening -> rising -> turn -> resolution`) failed both ways. First the beat never advanced - because the prompt never mentioned the `advance_beat` field existed. After we documented it, the model sprinted from opening to resolution in 8 turns. Final design: the prompt explains the arc, a deterministic nudge fires after 7 stagnant turns ("consider advance_beat"), and `apply_directives` rate-limits beats to a minimum of 4 turns each. Suggest with the prompt, pace with the code. | |
| ### 7. The performance pass: death by a thousand reloads | |
| Profiling a real session surfaced waste that no benchmark would show: | |
| - **rembg reloaded a ~170 MB ONNX model for every sprite.** `rembg.remove(img)` without an explicit session creates one each call. One reused session per painter: 1-3 s saved per generated sprite. | |
| - **SDXL's stock VAE forced an fp32 upcast on every render** (plus a deprecation warning per image). Swapping in the `sdxl-vae-fp16-fix` VAE removed the per-render upcast on every backend. | |
| - **The turn blocked on image generation.** Splitting `/turn` into a text phase and an image phase made the perceived latency drop from "the whole pipeline" to "one LLM call" - the player reads the reply while SDXL paints. | |
| - Smaller wins: Pydantic JSON schemas cached instead of regenerated per turn, memory compaction retuned from every ~4 turns to every ~12 (one LLM call saved per cycle), context trimming fixed to drop the *oldest* exchanges instead of accidentally beheading the style guide and character sheets. | |
| ### 8. Test the loop, not the models | |
| None of the above needed a GPU to verify. The mock-first design (`VN_MOCK=1`) plus a test suite that grew from 6 to 51 tests during the optimization pass meant every fix shipped with a regression test that runs in seconds - including simulated ZeroGPU worker restarts (build a fresh `Engine`, rehydrate from the state file, finish the turn). | |
| --- | |
| ## Stack <a name="stack"></a> | |
| | Layer | Tool | | |
| |---|---| | |
| | App framework | Gradio (`gradio.Server` + custom HTML/JS frontend) | | |
| | LLM | Qwen3-14B - `llama-cpp-python` (local + Modal A10G) or `transformers` (ZeroGPU Space) | | |
| | Image generation | SDXL-base-1.0 + SDXL-Lightning LoRA (4 steps) via `diffusers`, fp16-fix VAE, `rembg` sprites | | |
| | Voice input | Whisper large-v3-turbo via `faster-whisper` (CPU) | | |
| | Voice output | Kokoro-82M via `kokoro-onnx` | | |
| | Data contracts | Pydantic v2 everywhere (schemas drive the LLM grammar) | | |
| | Package management | `uv` | | |
| | Deployment | local, Modal (on-demand A10G), HF Spaces (ZeroGPU) | | |
| --- | |
| ## What this could become - a vision for AI-native VNs <a name="vision"></a> | |
| This hackathon was a proof of concept, but building it made us realize something bigger: this architecture could fundamentally change how visual novels are designed and experienced. | |
| Traditional VNs are authored upfront - writers script every line, every branch, every possible outcome. That takes years and still results in a finite tree of possibilities. Players feel it. You replay a route and you already know what the character will say. The "living" feeling is missing. | |
| What we built points toward a different model. Imagine a VN where: | |
| **The story has a skeleton, not a script.** Writers define the main characters with deep, consistent personalities, a narrative arc with key story beats and milestones, and the emotional tone they want to hit. But the actual dialogue? Generated fresh for every player. Two people playing the same game would reach the same major plot moments through completely different conversations - because their choices, their phrasing, their relationship dynamics got there differently. | |
| **Unexpected characters can emerge organically.** In our current build, new NPCs appear because the Weaver decides to introduce them. In a real production game, a writer could define a cast of named characters with full backstories - but also allow the AI to populate the world with one-off characters you meet passing through a market, or a stranger who overhears a conversation and joins in. Characters you will never meet again but who felt real in that moment. That kind of living texture is impossible to hand-author. | |
| **A writer AI trained for this specific craft.** Right now we are using a general-purpose LLM. The real version of this would have a model fine-tuned on great visual novel writing - pacing, character voice consistency, slow emotional escalation, the art of the well-timed silence. An AI that understands that a confession scene needs three scenes of buildup, that a rival character needs to be sympathetic before they can be satisfying to defeat. Not just "generate dialogue" but "author a story." | |
| **Art defined by real artists, adapted by AI.** This is the part that excites us most. Today we use a LoRA to lock in a consistent style. But imagine a studio hiring a concept artist to define the visual language of the world - their lighting, their color palettes, their character design principles - and then using that as the style constraint for every generated scene. The artist sets the aesthetic DNA. The AI adapts it to each specific situation: a rainy reunion scene gets desaturated with soft rim lighting, a festival scene bursts with warm orange and lantern glow. The artist's vision stays intact but the world breathes. | |
| The technology to do all of this exists today, mostly at small scale. What is missing is the production pipeline - the tools for writers to define narrative skeletons, the fine-tuning infrastructure for domain-specific storytelling models, the artist workflows for style injection. That is what the next version of this looks like. | |
| Ephemeral Hearts is a rough demo of where this goes. But the direction feels right. | |
| --- | |
| *Code Apache-2.0. Check the licences of the weights you ship - Qwen3 and SDXL-Turbo are permissive.* | |
| *Build Small Hackathon - Gradio x Hugging Face - June 2026* | |