BLOG.md

metadata

title: >-
  How We Built Ephemeral Hearts - a Live-Improvised Anime Visual Novel with
  Several Small AIs
thumbnail: /blog/assets/ephemeral-hearts/thumbnail.png
authors:
  - William Hubaux ; Github, [WillHCode](https://github.com/WillHCode)
  - Lorenzo Lepoivre ; Github, [SupPepper](https://github.com/LorenzoLepoivre)
date: '2026-06-09'
tags:
  - hackathon
  - visual-novel
  - multi-agent
  - diffusion
  - gradio
  - modal
  - lora

How We Built Ephemeral Hearts - a Live-Improvised Anime VN with Several Small AIs

Build Small Hackathon - June 2026

---

Table of contents

1. Who we are and why we built this
2. The multi-agent setup: three roles, three model families
3. What we discovered along the way
4. The "model proposes, code disposes" pattern
5. Features that emerged naturally
6. The journey from local to Modal
7. One codebase, three runtimes: local, Modal, Space
8. llama.cpp vs transformers vs ZeroGPU: what actually changes
9. Architecture, in depth
10. Field notes: the problems we hit and how we fixed them
11. Stack
12. What this could become

---

Who we are and why we built this

We are two developers, and we both already work on a production AI project with RAG pipeline, local models via HuggingFace Transformers, multi-agent LLM system - all aimed at real business use cases. So we were not complete beginners when it came to language models. But this hackathon was something different: how do you put all of this together in a fun way, with creative constraints, and explore territories we had not seriously touched yet - image generation, voice models, fine-tuning diffusion models with LoRA, and Modal for on-demand GPU deployment.

The base idea: an anime visual novel where everything is generated live. No pre-written script, no fixed assets. Every character, every backdrop, every line of dialogue - conjured on the fly by small models. The hackathon imposed ≤ 32B total parameters and a Gradio app on a HF Space. Perfect. We figured that if the constraints forced us to be creative about the architecture, we might as well build one we would be proud to explain.

The result: Ephemeral Hearts - an anime dating VN where the story, dialogue, characters, and art are all generated in real time. Nothing is pre-scripted.

---

The multi-agent setup: three roles, three model families

The first real design question was: how many models, which ones, and how do you make them collaborate?

We defined three distinct roles, each handled by a different model family:

🧵 The Weaver - the game director. It reads the current scene, the character sheets, the recent conversation, and the player's input. It decides what changes structurally this turn: does the scene shift? Does a new character arrive? What is the relationship delta? It outputs a structured JSON block of directives.

🎭 The Voices - the actor. It speaks as the active character - their speech quirks, their current mood, their secrets. One to three sentences, anime style, never breaking the fourth wall.

🎨 The Painter - it generates backdrops and character sprites. Default backend: SDXL-base-1.0 distilled down to 4 inference steps with the ByteDance SDXL-Lightning LoRA (an SDXL-Turbo backend exists as an alternative, and VN_IMAGE_LORA accepts a custom style LoRA on top).

👂 The Ear - voice transcription. Whisper (large-v3-turbo by default), CPU via faster-whisper, good enough for short clips.

🔊 The Larynx - speech synthesis. Kokoro-82M via ONNX gives every character a voice, picked once at creation from their written voice description and frozen forever (same character = same voice).

The key subtlety: The Weaver and The Voices share a single LLM (Qwen3-14B). Same weights, two system prompts, one grammar-constrained call per turn. They write the stage directions and the dialogue lines simultaneously. A second LLM call per turn would have doubled perceived latency - and the two roles are so tightly coupled that sharing context is actually an advantage.

Architecture diagram

 Player input (text or voice)
         |
         v
 +-------------------+
 |   THE EAR  👂     |  Whisper-small
 |  voice -> text    |  (CPU - faster-whisper)
 +--------+----------+
          |
          v
 memory.assemble_context()
 +---------------------------------------------+
 |  summary + current scene + character sheets |
 |  + last k exchanges  (fixed token budget)   |
 +------------------+--------------------------+
                    |
                    v
 +------------------------------------------------------+
 |          ONE grammar-constrained call                |
 |                                                      |
 |   THE WEAVER  🧵  (director)                         |
 |   + THE VOICES 🎭  (actor)                           |
 |                                                      |
 |   Qwen3-14B                                          |
 |   +-- local : llama.cpp  (Metal / ROCm)              |
 |   +-- cloud : Modal A10G  (VN_LLM_BACKEND=modal)     |
 +----------------------+-------------------------------+
                        |
                        v
                  DirectorOutput
        +---------------------------------+
        |  dialogue  .  emotion          |
        |  scene_change                  |
        |  relationship_delta            |
        |  new_character / exit_char     |
        |  set_music  .  ending          |
        |  npc_relation_deltas           |
        +--------------+-----------------+
                       |
                       v
          state.apply_directives()
          <- THE ONLY MUTATOR
          GameState updated
          .md dream-memory written
                       |
          +------------+-------------+
          v                          v
 +-----------------+        +-----------------+
 |  THE PAINTER 🎨 |        |    ViewState     |
 |                 |        |  -> frontend     |
 |  SDXL-Turbo     |        |  (Gradio Server) |
 |  + anime LoRA   |        +-----------------+
 |                 |
 |  +- local :     |
 |  |  diffusers   |
 |  |  (Metal/ROCm)|
 |  +- cloud :     |
 |     Modal A10G  |
 +--------+--------+
          |
          v
   backdrop.png  (cached by seed)
   sprite.png    (cached by character + mood)

Total: ≈18B parameters. Well under the 32B budget.

Role	Model	Params
The Weaver + The Voices (shared)	Qwen3-14B	~14B
The Painter	SDXL-base-1.0 + Lightning LoRA	~3.5B
The Ear	Whisper large-v3-turbo	~0.8B
The Larynx	Kokoro-82M (TTS)	~0.08B
Total		≈18B

---

What we discovered along the way

We already had experience with LLMs and HuggingFace. But this project pushed us into three territories we had not seriously explored before.

Image models - SDXL-Turbo and LoRA

This is clearly what surprised us the most. We knew image generation existed, we had played with online demos, but actually integrating it into a real pipeline - managing prompts, seeds, caching, negative prompts, and visual consistency - is a genuine craft in itself.

Choosing a few-step distilled model over a full diffusion pipeline was deliberate: 1 to 4 inference steps instead of 50. Slightly lower quality, but in a VN where the image arrives while the player is reading the dialogue, "fast and impressionistic" beats "slow and polished" every single time. We started with SDXL-Turbo and eventually settled on SDXL-base-1.0 + the ByteDance Lightning LoRA as the default - same philosophy, noticeably better quality at 4 steps (both backends remain in the code).

What we had never done before: LoRA fine-tuning. We trained an anime LoRA on SDXL-Turbo to lock in the visual style - color palette, character rendering, line quality. The result: even as prompts vary wildly from scene to scene, the style stays consistent. That is exactly what a VN needs. The LoRA is published on the Hub (VN_IMAGE_LORA) and loads automatically at startup.

Another revelation: seed pinning. Every character sprite is generated once per mood, with a fixed seed, and cached forever. Same character, same scene, same expression = same image, always. The visual consistency this gives does more for immersion than a higher-quality model would.

Voice models - Whisper

Whisper-small for voice input was the easiest onboarding. CTranslate2 via faster-whisper, CPU, works on both our machines without a dedicated GPU. The only gotcha: CTranslate2 has no ROCm backend, so on our AMD box it runs on CPU regardless. For 5-10 second clips it is more than sufficient.

Modal - GPU on demand

This was the real infrastructure discovery of this hackathon.

We were used to deploying on HF Spaces or fixed GPU servers. Modal is a completely different mental model: you write a Python function, decorate it with @app.function(gpu="A10G"), and Modal handles everything else - containerization, scaling, cold starts, per-second billing. For a hackathon where you have no idea what load you will get, it is perfect.

The Gradio Space stays lightweight (CPU only), model calls go out to Modal A10G containers. The VN_LLM_BACKEND=modal environment variable switches the backend without touching any game logic. Same interface, different implementations.

---

The "model proposes, code disposes" pattern

The naive approach: let the LLM generate free prose and parse the result. This breaks immediately with small models - they drift, contradict themselves, forget who is on stage.

Our approach: every turn is one grammar-constrained call that returns a typed Pydantic DirectorOutput:

# Schema is derived directly from the Pydantic model - can never drift
out = llm.complete_json(schema=DirectorOutput.model_json_schema(), prompt=context)

# The only place in the entire codebase that touches GameState
effects = state.apply_directives(game_state, out)

apply_directives clamps values, validates references, and silently ignores impossible requests (like removing a character who is not on stage). The LLM never touches state directly - it proposes, the code decides.

Direct consequence: the entire game loop was testable from day one with VN_MOCK=1 - deterministic fake LLM, fake painter, zero real models. We built the NPC bond graph, the music system, the ending screens, and the relationship milestones entirely in mock mode before a single real model call ever ran.

---

Features that emerged naturally

Once the directive pattern was solid, features fell out almost on their own.

NPC-to-NPC relationship graph. The same npc_relation_deltas directive that tracks player-NPC affection also tracks relationships between NPCs. Jealousy, rivalry, camaraderie - all of this emerges naturally from the model's tendency to give characters an inner life. We expose it as a directed graph in the Relations tab, with one-word labels ("jealousy", "rivalry", "fond of") coming directly from the model.

Dynamic music system. Six tracks (calm, romantic, dramatic, mystery, sad, joyful), crossfade on change, all controlled by a set_music directive. The model knows the available tracks and only switches for genuine tonal ruptures - not every mood shift. In practice it is surprisingly restrained.

Generated ending screens. When a relationship hits +100 (romantic confession) or -100 (a falling-out that empties the stage), the model emits an ending directive with a poetic 2-4 sentence epilogue. The Painter generates a dedicated illustration - cherry blossoms at golden hour for the warm ending, a rain-soaked empty park bench at night for defeat. No pre-authored cutscenes. The ending art is improvised just like everything else.

Characters that remember you. A remember_fact directive lets the speaker store one short note when the player reveals something personal ("the player plays violin", "the player is named Lorenzo"). Facts are capped at 8 per character and re-injected into their sheet every turn they are on stage - so a character you met ten scenes ago still knows your name. The same progression system unlocks personality traits at affection thresholds (20/40/60), reveals the secret goal at 80, and stages a one-shot "growing close" moment at 50 - all deterministic, all driven by the relationship value the model proposes.

Quality-of-life. The player can set their name in the setup form (characters address them by it), and a session persisted server-side after every turn powers a "Continue last story" button - including a rebuilt journal with a condensed recap of everything that got compacted away.

---

The journey from local to Modal

We started on our two machines: an Apple M3 Max (Metal backend) and an AMD RX 7900 XTX (ROCm). First surprising fact: ROCm reports itself as "cuda" to PyTorch - so our device detection works on both machines without any special-casing. Second surprise: CTranslate2 has no ROCm backend, so Whisper runs on CPU on the AMD machine regardless.

For the HF Space we first needed GPU inference without going through ZeroGPU (llama.cpp + ZeroGPU is notoriously unreliable). The solution: Modal for heavy model calls, Gradio for the UI. The Space stays CPU, GPU compute goes to on-demand Modal containers. We later added a third path - running the models directly on ZeroGPU with transformers - which turned out to have its own fascinating constraints (see the next two sections).

One unexpected practical problem: HF Spaces now uses Xet storage and refuses binary files pushed through git. We had to soft-reset the commit that included the music MP3 files, add them to .gitignore, push the clean code, then upload the audio files via HfApi().upload_file(). Lesson learned the hard way: code goes through git, binary assets go through the Hub API.

---

One codebase, three runtimes: local, Modal, Space

The same visualnovel/ package runs in three very different places. Nothing in the game logic knows which one it is in - the backends are selected by config.py from environment variables, and every backend implements the same two-method interface (complete / complete_json for the LLM, _render for the Painter).

	local	modal	space (ZeroGPU)
Where the app runs	your machine	your machine (UI) + Modal (GPU)	HF Space
LLM engine	llama.cpp, GGUF Q4_K_M, Metal/ROCm	llama.cpp, GGUF Q8_0, A10G container	transformers, full bf16 weights
Painter	diffusers in-process	A10G container (RPC)	diffusers in-process
Selected by	default	VN_LLM_BACKEND=modal	SPACE_ID env var (auto-detected)
Game state	in-process	in-process	/tmp JSON file (workers are stateless)
Cold start cost	model load at launch	~15-30 s container spin-up, then kept warm 10 min	per-worker lazy load inside the first GPU call

A few design decisions make this work:

• Auto-detection over configuration. HF injects SPACE_ID into every Space, so config.py flips the LLM backend to transformers automatically when deployed - a fresh clone needs zero setup in any environment. Setting VN_LLM_BACKEND=modal also chains the Painter to Modal by default, because if you don't have a local GPU for the LLM you don't have one for SDXL either.
• Modal calls are plain RPC. ModalLLM is a ~20-line proxy: it looks up the deployed class with modal.Cls.from_name(...) and calls .remote(). The JSON schema travels with every call, which has a lovely property: changing prompts, schemas, or game logic never requires a redeploy - only changes to the container code itself do.
• Keep-warm matters more than raw speed. Reloading a 15 GB GGUF mid-conversation is a worse experience than any per-token slowness, so the Modal LLM container keeps a 10-minute scaledown_window and the app fires a fire-and-forget warmup ping at server start.

---

llama.cpp vs transformers vs ZeroGPU: what actually changes

These three names live at different levels - two are inference engines, one is a runtime - and each one reshaped a different part of the code.

llama.cpp: the grammar is the contract

The killer feature for this project: llama-cpp-python compiles a JSON schema into a GBNF grammar that constrains decoding token by token. The model physically cannot emit malformed JSON, invent keys, or put a string where an integer belongs. Our whole "model proposes, code disposes" pattern leans on this.

It also exposes full sampling control (temperature, top_p, presence_penalty - the latter became our anti-repetition weapon). Two sharp edges, though:

• The grammar cannot finish a document when max_tokens runs out mid-string. You get perfectly-shaped-but-truncated JSON, and json.loads explodes. We wrote a small close_truncated_json() repair (close the open string, drop the dangling comma, balance the brackets) so a runaway generation costs a few default-valued fields instead of a crashed turn.
• Qwen3's thinking mode is baked into the GGUF chat template and there is no API switch - the only control is the /no_think soft switch appended to the user message.

transformers: no grammar, so trust but verify

On the Space there is no grammar constraint. Instead we derive a JSON skeleton from the same Pydantic schema and inject it into the system prompt ("Respond with ONLY a valid JSON object. Required structure: ..."), then parse with up to 3 retries and a regex extraction that tolerates markdown fences and stray prose.

Two gotchas that cost us real debugging time:

• model.generate() silently ignores temperature and top_p unless you also pass do_sample=True. For weeks the Space was running greedy decoding while we thought we were sampling at 0.7 - and the 3 retries were deterministic, so they retried into the exact same failure.
• Qwen3 sometimes puts its entire answer inside the <think> block and emits nothing after it. apply_chat_template(..., enable_thinking=False) suppresses thinking for structured output, with a fallback that searches inside the think block if nothing comes after it.

ZeroGPU: a runtime, not a backend

ZeroGPU is the part that changes your architecture rather than your inference code. Each @spaces.GPU call can be dispatched to a different worker subprocess - in-memory state simply does not survive between two HTTP calls. Three consequences:

1. State lives on disk. After every mutation, GameState is serialized to /tmp/vn_game_state.json with an atomic write-then-rename. Every endpoint starts by rehydrating from that file if its own memory is empty. (Bonus: this is exactly the mechanism that later powered the "Continue last story" button.)
2. Models load lazily. Loading a 14B model at import time would happen on a CPU-only web worker; instead every backend loads inside the first GPU-decorated call.
3. Turns are split in two phases (/turn_text then /turn_images) so the dialogue can be displayed while the slower image phase runs - and the pending DirectorOutput rides along in the state file, because phase 2 might run on a different worker than phase 1.

---

Architecture, in depth

The package is a strict one-way dependency graph - schemas at the bottom, the engine façade at the top, app.py as a thin HTTP shell. The golden rule sits in the middle: the LLM returns a typed DirectorOutput, and state.apply_directives is the only function in the codebase allowed to mutate GameState.

flowchart TD
    UI["frontend/index.html<br/>(custom VN UI, Gradio JS client)"]
    APP["app.py - gradio.Server<br/>thin endpoints: /start_text /start_images<br/>/turn_text /turn_images /transcribe /resume"]
    ENG["engine.py<br/>the façade - owns the session"]

    subgraph CORE["visualnovel/ - testable without a server"]
        ORC["orchestrator.py<br/>the Weaver: init / direct_turn / compact"]
        MEM["memory.py<br/>context budget + trimming"]
        CH["characters.py<br/>present-character sheets"]
        PRM["prompts.py<br/>all prompts + JSON schemas"]
        LLM["llm.py<br/>Mock / LlamaCpp / Transformers / Modal"]
        ST["state.py<br/>apply_directives - THE ONLY MUTATOR"]
        PAINT["painter.py<br/>prompt compose + cache + render"]
        STT["stt.py - Whisper"]
        TTS["tts.py - Kokoro"]
    end

    UI -->|HTTP| APP --> ENG
    ENG --> ORC
    ORC --> MEM --> CH
    ORC --> PRM
    ORC --> LLM
    ENG --> ST
    ENG --> PAINT
    ENG --> STT
    ENG --> TTS

A full turn, with the two-phase split that keeps the dialogue snappy:

sequenceDiagram
    autonumber
    participant P as Player (browser)
    participant A as app.py
    participant E as Engine
    participant W as Weaver (one LLM call)
    participant S as apply_directives
    participant PA as Painter
    participant T as Kokoro TTS

    P->>A: POST /turn_text ("Hello, you are so pretty today")
    A->>E: play_turn_text()
    Note over E: optional: Whisper if voice input
    E->>W: complete_json(context, directive schema)
    W-->>E: DirectorOutput (dialogue, emotion, directives)
    E->>S: apply_directives - clamps, validates, mutates GameState
    E->>E: save .md views + /tmp state (ZeroGPU workers)
    E-->>P: text-only ViewState - dialogue appears NOW

    P->>A: POST /turn_images
    A->>E: play_turn_images()
    E->>PA: backdrop(scene) + sprite(char, mood)
    Note over PA: cache hit by (kind, prompt, seed)?<br/>then 0 ms, else 4-step SDXL render
    E->>T: synthesize(dialogue, frozen voice)
    E-->>P: full ViewState - backdrop, sprites, audio

And the deployment picture - one engine, three homes:

flowchart LR
    APP["app.py + visualnovel/<br/>(identical code everywhere)"]

    subgraph L["LOCAL - Mac M3 Max / AMD RX 7900 XTX"]
        L1["llama.cpp - Qwen3-14B Q4_K_M<br/>Metal / ROCm, grammar-constrained"]
        L2["diffusers - SDXL + Lightning LoRA"]
    end

    subgraph M["MODAL - on-demand A10G"]
        M1["ModalLLMBackend<br/>llama.cpp - Q8_0, kept warm 10 min"]
        M2["ModalPainterBackend<br/>SDXL + Lightning, rembg"]
    end

    subgraph Z["HF SPACE - ZeroGPU"]
        Z1["TransformersLLM - Qwen3-14B bf16<br/>JSON skeleton + retries"]
        Z2["diffusers in-process"]
        Z3[("/tmp state file<br/>(stateless workers)")]
    end

    APP -->|"default"| L
    APP -->|"VN_LLM_BACKEND=modal<br/>.remote() RPC"| M
    APP -->|"SPACE_ID detected<br/>@spaces.GPU"| Z

Two invariants hold everywhere:

• One LLM call per turn (a guarded retry is allowed for repetition or a missed mandatory directive - never a loop).
• Every Painter render is cached by (kind, prompt, seed). A character's sprite is generated once per mood and reused forever; seeds are pinned per entity so the same character always has the same face.

---

Field notes: the problems we hit and how we fixed them

This is the section we wish we had read before starting. Almost every bug below was invisible in the code and obvious in a saved game file - our best debugging tool turned out to be exporting the save JSON after each playtest and actually reading it.

1. Anything optional in the grammar is something the model will skip

The single biggest lesson of the project. Our DirectorOutput schema had emotion, relationship_delta, and new_character as optional fields with defaults - and the grammar therefore allowed the model to omit them. So it did. Systematically. The symptoms looked like three unrelated gameplay bugs:

• every character stayed "neutral" forever (so the mood-keyed sprite system never showed a second expression),
• affection stayed at 0 no matter how hard the player flirted (or insulted),
• the "look around" action politely refused to introduce anyone.

We tried prompt rules ("you MUST emit new_character"), then an explicit retry quoting the instruction. The model ignored both often enough to ruin the demo. The fix that actually worked: promote the fields to required in the JSON schema handed to the grammar. For "look around" we went further and built a dedicated schema variant where new_character is required and non-nullable - introduction guaranteed by construction. Prompt engineering is a suggestion; grammar is a constraint.

2. Instructions inside quoted player speech get read as... player speech

Our action hints ("the wanderer looks around...") were concatenated with the player's text and injected as THE WANDERER NOW SAYS: "<hint> <text>". Inside the quotes, the model treated our stage directions as something the player said out loud - and ignored them. Moving the hints to a separate DIRECTOR NOTE (mandatory): block outside the quotes changed compliance dramatically. Placement in the context is as important as wording.

3. Qwen3's thinking mode ate our memory summarizer

The rolling summary ("THE TALE SO FAR") is rewritten every ~12 turns by a small free-text LLM call with a 320-token budget. On llama.cpp, Qwen3's chat template enables thinking by default - and the model happily spent the entire budget inside an unclosed <think> block. Our summary was an empty string for whole sessions, silently replaced by a raw-dialogue fallback, and we only noticed by reading a save file where the "summary" was a concatenation cut mid-sentence. Fix: the /no_think soft switch appended to the message, strip_think() on every free-text output, and a fallback that never degrades the existing summary.

4. Truncated JSON should cost fields, not turns

One playtest produced a character whose goals field was a 5,000-character runaway sentence - which blew past max_tokens and crashed the turn with json.decoder.JSONDecodeError: Unterminated string. Three layers later: generated character fields are word-capped at creation and at context-injection time, close_truncated_json() repairs cut-off output, and direct_turn has a last-resort fallback line ("Hm? Sorry - I lost my train of thought...") so the worst possible outcome of a bad sample is one bland reply.

5. Small models loop, and they loop harder once a loop is in the context

A character repeated the same line verbatim three turns in a row - despite a prompt rule forbidding exactly that. Once one repeat lands in the RECENT EXCHANGE window, it biases the next generation toward repeating again. Code-side guard: normalize and compare the new dialogue against the last 3 turns (difflib ratio >= 0.95), and on a hit, retry once at higher temperature with the forbidden line quoted back and presence_penalty=0.8.

6. Pacing has to be enforced in both directions

Story beats (opening -> rising -> turn -> resolution) failed both ways. First the beat never advanced - because the prompt never mentioned the advance_beat field existed. After we documented it, the model sprinted from opening to resolution in 8 turns. Final design: the prompt explains the arc, a deterministic nudge fires after 7 stagnant turns ("consider advance_beat"), and apply_directives rate-limits beats to a minimum of 4 turns each. Suggest with the prompt, pace with the code.

7. The performance pass: death by a thousand reloads

Profiling a real session surfaced waste that no benchmark would show:

• rembg reloaded a ~170 MB ONNX model for every sprite. rembg.remove(img) without an explicit session creates one each call. One reused session per painter: 1-3 s saved per generated sprite.
• SDXL's stock VAE forced an fp32 upcast on every render (plus a deprecation warning per image). Swapping in the sdxl-vae-fp16-fix VAE removed the per-render upcast on every backend.
• The turn blocked on image generation. Splitting /turn into a text phase and an image phase made the perceived latency drop from "the whole pipeline" to "one LLM call" - the player reads the reply while SDXL paints.
• Smaller wins: Pydantic JSON schemas cached instead of regenerated per turn, memory compaction retuned from every ~4 turns to every ~12 (one LLM call saved per cycle), context trimming fixed to drop the oldest exchanges instead of accidentally beheading the style guide and character sheets.

8. Test the loop, not the models

None of the above needed a GPU to verify. The mock-first design (VN_MOCK=1) plus a test suite that grew from 6 to 51 tests during the optimization pass meant every fix shipped with a regression test that runs in seconds - including simulated ZeroGPU worker restarts (build a fresh Engine, rehydrate from the state file, finish the turn).

---

Stack

Layer	Tool
App framework	Gradio (gradio.Server + custom HTML/JS frontend)
LLM	Qwen3-14B - llama-cpp-python (local + Modal A10G) or transformers (ZeroGPU Space)
Image generation	SDXL-base-1.0 + SDXL-Lightning LoRA (4 steps) via diffusers, fp16-fix VAE, rembg sprites
Voice input	Whisper large-v3-turbo via faster-whisper (CPU)
Voice output	Kokoro-82M via kokoro-onnx
Data contracts	Pydantic v2 everywhere (schemas drive the LLM grammar)
Package management	uv
Deployment	local, Modal (on-demand A10G), HF Spaces (ZeroGPU)

---

What this could become - a vision for AI-native VNs

This hackathon was a proof of concept, but building it made us realize something bigger: this architecture could fundamentally change how visual novels are designed and experienced.

Traditional VNs are authored upfront - writers script every line, every branch, every possible outcome. That takes years and still results in a finite tree of possibilities. Players feel it. You replay a route and you already know what the character will say. The "living" feeling is missing.

What we built points toward a different model. Imagine a VN where:

The story has a skeleton, not a script. Writers define the main characters with deep, consistent personalities, a narrative arc with key story beats and milestones, and the emotional tone they want to hit. But the actual dialogue? Generated fresh for every player. Two people playing the same game would reach the same major plot moments through completely different conversations - because their choices, their phrasing, their relationship dynamics got there differently.

Unexpected characters can emerge organically. In our current build, new NPCs appear because the Weaver decides to introduce them. In a real production game, a writer could define a cast of named characters with full backstories - but also allow the AI to populate the world with one-off characters you meet passing through a market, or a stranger who overhears a conversation and joins in. Characters you will never meet again but who felt real in that moment. That kind of living texture is impossible to hand-author.

A writer AI trained for this specific craft. Right now we are using a general-purpose LLM. The real version of this would have a model fine-tuned on great visual novel writing - pacing, character voice consistency, slow emotional escalation, the art of the well-timed silence. An AI that understands that a confession scene needs three scenes of buildup, that a rival character needs to be sympathetic before they can be satisfying to defeat. Not just "generate dialogue" but "author a story."

Art defined by real artists, adapted by AI. This is the part that excites us most. Today we use a LoRA to lock in a consistent style. But imagine a studio hiring a concept artist to define the visual language of the world - their lighting, their color palettes, their character design principles - and then using that as the style constraint for every generated scene. The artist sets the aesthetic DNA. The AI adapts it to each specific situation: a rainy reunion scene gets desaturated with soft rim lighting, a festival scene bursts with warm orange and lantern glow. The artist's vision stays intact but the world breathes.

The technology to do all of this exists today, mostly at small scale. What is missing is the production pipeline - the tools for writers to define narrative skeletons, the fine-tuning infrastructure for domain-specific storytelling models, the artist workflows for style injection. That is what the next version of this looks like.

Ephemeral Hearts is a rough demo of where this goes. But the direction feels right.

---

Code Apache-2.0. Check the licences of the weights you ship - Qwen3 and SDXL-Turbo are permissive.

Build Small Hackathon - Gradio x Hugging Face - June 2026