README.md

metadata

title: Ephemeral Hearts
emoji: 🐠
colorFrom: green
colorTo: indigo
sdk: gradio
sdk_version: 6.18.0
python_version: '3.12'
app_file: app.py
pinned: false
tags:
  - track:wood
  - sponsor:modal
  - achievement:offgrid
  - achievement:offbrand
  - achievement:llama
  - achievement:sharing
  - achievement:fieldnotes
license: apache-2.0
thumbnail: >-
  https://cdn-uploads.huggingface.co/production/uploads/65c10674c4dcb96c31dd84c9/aWVwpGKH963R2e81blgxO.png

Team

• WillHbx — https://huggingface.co/WillHbx
• SuperPoivre13 — https://huggingface.co/SuperPoivre13

hackathon-ia-visualnovel 🍄

Thousand Token Wood — an AI-improvised, voice-or-text anime visual novel. Build Small Hackathon (Chapter Two). Story, characters, and art are generated live by small local models — nothing is pre-scripted.

You step into a wood that is being dreamed into existence around you. Every spirit you meet is conjured on the spot; every backdrop is painted the moment you arrive. Because the wood is dreamed by a small, slightly forgetful mind, it is whimsical and never the same twice — the model's quirks aren't bugs, they're the wood dreaming. Take the models out and there's no game, only an empty Gradio shell. That's what makes the AI load-bearing.

---

🏆 Build Small Hackathon submission

Ephemeral Hearts is an AI-improvised anime dating visual novel: the story, every character, every line of dialogue, every backdrop and sprite, and even the character voices are generated live by small local models. Nothing is pre-scripted. Five AI roles collaborate in under 18B parameters total behind one golden rule, the model proposes, code disposes: the LLM returns a typed DirectorOutput (one grammar-constrained call per turn) and deterministic code is the only thing that mutates game state. The same visualnovel/ package runs in three places unchanged, selected by env vars: fully offline on a laptop (llama.cpp + diffusers), on on-demand Modal GPUs, or on a ZeroGPU Space.

Tech: Qwen3-14B (the Weaver + the Voices, shared weights) · SDXL + a fine-tuned anime LoRA (the Painter) · Whisper (the Ear) · Kokoro-82M (the Voice) · Gradio gradio.Server with a bespoke HTML/JS frontend · Pydantic-derived JSON grammar · uv.

• 🎥 Demo video: https://youtu.be/NVDTg_oMUY4
• 📝 Blog / field notes: https://huggingface.co/blog/WillHbx/ephemeral-hearts
• 💬 Social post: https://www.linkedin.com/posts/lorenzo-lepoivre_ai-generativeai-llm-share-7471625802258939904-Pjx-/

Track: Thousand Token Wood Sub-tracks: Off-Brand · Best Demo · Best Agent · Bonus Quest Champion · Judges' Wildcard Badges: 🔌 Off the Grid · 🎨 Off-Brand (Custom UI) · 🦙 Llama Champion (llama.cpp) · 📡 Sharing is Caring (Open-Trace) · 📓 Field Notes

---

✨ This repo runs on a fresh checkout — with zero models

It ships in MOCK mode (VN_MOCK=1, the default): deterministic fake LLM / painter / STT so the entire loop works offline with placeholder art. Open it, watch a turn happen, then have Claude Code replace each mock with a real model, one module at a time. See CLAUDE.md.

uv sync                       # installs only gradio + pydantic + pillow (seconds)
uv run python -m visualnovel.smoke     # full loop in the terminal, no UI, no models
uv run python app.py                   # custom VN UI  →  http://localhost:7860
GRADIO_MVP_UI=1 uv run python app.py   # plain gr.Blocks fallback (Phase 0/1)

---

The cast of models (the “several AIs”)

Four roles, four real model families (text, image, speech-in, speech-out). The two text roles share one LLM (same weights, two system prompts) to respect the parameter budget.

In-world name	Role	Backed by	Output
🧵 The Weaver	director / GM — builds the world, keeps canon, decides what changes	LLM (shared)	structured directives (JSON)
🎭 The Voices	actor — speaks as the present spirits, in voice	LLM (shared)	in-character dialogue + emotion
🎨 The Painter	renders backdrops + character sprites	SDXL-Turbo (+ your anime-style LoRA)	images
👂 The Ear	turns spoken words into text	Whisper	transcribed input
🗣️ The Voice	speaks the spirits' dialogue aloud (per-character voice)	Kokoro-82M (ONNX)	synthesized audio

The Weaver and Voices are one grammar-constrained LLM call per turn → { speaker, dialogue, emotion, directives }. Code applies the directives deterministically; the LLM never edits state directly.

---

Parameter budget — the “beefy” config (≤ 32B total)

Component	Model	~Params
LLM (shared)	Qwen3-14B	~14B
Image	SDXL-Turbo (+ fine-tuned anime LoRA)	~3.5B
STT	Whisper-large-v3-turbo	~0.8B
TTS	Kokoro-82M (ONNX)	~0.08B
Matting (optional)	BiRefNet	~0.2B
Total		≈ 18.6B ✅

MoE models count by total params. Everything is hosted on HF (you have credits), so the only hard limit is ≤ 32B. Override any model via env vars in visualnovel/config.py.

---

Setup & install (uv)

Python 3.12 is pinned in .python-version — the ML stack (torch / llama-cpp-python / ctranslate2) ships wheels for it. The package code is 3.11+ clean, so a newer interpreter works too if every dependency publishes wheels for it.

Your hardware is AMD ROCm + Apple Metal (no local CUDA), which changes the GPU build flags:

# ---- Apple Silicon (M3 Max) — Metal ----
uv sync --extra image --extra stt
CMAKE_ARGS="-DGGML_METAL=on" uv pip install llama-cpp-python      # Metal is default on macOS
#   torch uses the MPS backend automatically. faster-whisper runs on CPU (CTranslate2 has no
#   Metal); for GPU STT on Mac use mlx-whisper or whisper.cpp instead.

# ---- AMD RX 7900 XTX — ROCm (Linux) ----
uv pip install torch --index-url https://download.pytorch.org/whl/rocm6.2   # ROCm torch wheel
CMAKE_ARGS="-DGGML_HIPBLAS=on" uv pip install llama-cpp-python    # newer builds: -DGGML_HIP=on
uv sync --extra image --extra stt
#   torch reports ROCm as "cuda" (so config.detect_device() returns "cuda" on this box).
#   CTranslate2 has no ROCm → faster-whisper runs on CPU (fine for short clips), or use whisper.cpp.

Then download weights (or let them fetch lazily on first real use) and flip off mock mode:

uv run python scripts/download_models.py
VN_MOCK=0 uv run python app.py

---

How a turn works

 START ─▶ Weaver: dream world + opening scene + first spirit ─▶ Painter: backdrop + sprite
   │
   └─▶ LOOP
        1. player speaks/types ─▶ Ear (Whisper) ─▶ text
        2. ONE grammar-constrained LLM call ─▶ { speaker, dialogue, emotion, directives }
        3. state.apply_directives(...) deterministically (move scene, add/exit spirit, mood,
           relationship, flags, beat, ending) ─▶ .md dream-memory updated
        4. if scene/character changed ─▶ Painter (cached, seed-pinned)
        5. render: backdrop + sprite(mood) + dialogue   (text streams first; image fills in)
        6. The Voice (Kokoro) speaks the spirit's line in its per-character voice (cached)

Deep design in docs/ARCHITECTURE.md; prompts + JSON schema + GBNF grammar in docs/PROMPTS.md.

---

Project structure

hackathon-ia-visualnovel/
├── app.py                     # gradio.Server entry + gr.Blocks MVP fallback (thin)
├── pyproject.toml             # uv project; heavy ML libs are optional extras
├── .python-version            # 3.12  (ML stack ships wheels for it)
├── frontend/index.html        # custom layered VN UI (backdrop · sprite · dialogue · mic)
├── visualnovel/
│   ├── schemas.py             # Pydantic: GameState (truth) + DirectorOutput (LLM contract)
│   ├── config.py              # paths, flags, model registry, THEMES, device detection
│   ├── prompts.py             # system prompts + schema helpers (no scattered literals)
│   ├── llm.py                 # MockLLM · LlamaCppLLM · TransformersLLM behind one interface
│   ├── orchestrator.py        # the Weaver: init_world / direct_turn / compact_memory
│   ├── characters.py          # the Voices: present-character actor context
│   ├── painter.py             # the Painter: prompt compose, cache, MockPainter + SDXL stub
│   ├── stt.py                 # the Ear: Whisper (mock + faster-whisper)
│   ├── tts.py                 # the Voice: per-character speech (mock + Kokoro-82M ONNX)
│   ├── state.py               # apply_directives (the ONLY mutator) + .md render
│   ├── memory.py              # bounded context assembly + compaction trigger
│   ├── engine.py              # façade: start_text/start_images · play_turn(_text/_images) → ViewState
│   ├── utils.py               # shared helpers (think-strip, JSON repair, quiet stderr)
│   ├── metrics.py             # debug-mode metrics + live monitor (no-op in mock/prod)
│   ├── trace.py               # per-turn JSONL trace (Open-Trace bonus)
│   └── smoke.py               # `python -m visualnovel.smoke`
├── templates/                 # the .md dream-memory views (rendered each turn)
├── tests/                     # state round-trip · directive apply · memory budget (mock, green)
├── scripts/download_models.py
└── docs/                      # ARCHITECTURE.md · PROMPTS.md

---

Bonus quests targeted

Badge	How
🎨 Off-Brand (custom UI)	bespoke VN frontend via gradio.Server (frontend/index.html)
🦙 Llama-Champion	LLM through llama-cpp-python + JSON-schema/GBNF grammar (run locally for the video)
🔌 Off the Grid	all weights local; demoed with the network off
🎯 Well-Tuned	the fine-tuned SDXL-Turbo anime LoRA, published on the Hub (VN_IMAGE_LORA)
📡 Open-Trace	VN_TRACE=runs/run.jsonl records every orchestration step → publish as a dataset
📓 Field-Notes	write up the diegetic conceit + one-call director pattern

⚠️ llama.cpp on the hosted ZeroGPU Space is the trickiest bit — claim it locally in the demo video and keep the transformers backend (VN_LLM_BACKEND=transformers) for the Space. See docs/ARCHITECTURE.md §7.

---

Quality

uv run ruff check . && uv run ruff format .
uv run pytest -q

Submission checklist

• Registered on the Build-Small org by June 3 + joined the Gradio Discord.
• Space is live and loads without errors.
• Total params ≤ 32B (stated).
• It's a Gradio app (gradio.Server counts).
• Demo video + social post.

Built for the Build Small Hackathon (Gradio × Hugging Face, June 2026). Code Apache-2.0 — match the licenses of the weights you ship (SDXL-Turbo and Qwen3 are permissive; check FLUX.2 if you swap the painter).