README.md

metadata

title: AI Puppet Theater
emoji: 🎭
colorFrom: yellow
colorTo: purple
sdk: gradio
sdk_version: 6.5.1
app_file: app.py
python_version: '3.10'
pinned: false
tags:
  - gradio
  - build-small-hackathon
  - track:wood
  - sponsor:openbmb
  - sponsor:modal
  - achievement:welltuned
  - achievement:llama
  - achievement:sharing
  - achievement:fieldnotes

AI Puppet Theater

Create a tiny AI puppet show, then interrupt it from the audience.

AI Puppet Theater is a Hugging Face Gradio Space for the Build Small Hackathon Thousand Token Wood track. The app turns a short user premise into a compact puppet show: it casts characters, lets a Director manage pacing, gives actors intent and state, accepts audience interruptions, performs stage effects, and exports a sanitized trace of the scene.

Table of Contents

• Live Demo
• Current Status
• Recommended Demo Settings
• What It Does
• Project Artifacts
• Small-Model Highlight
• Hackathon Fit
• How to Demo
• Architecture
• Agent Architecture
• Features
• Model and Runtime
• Hackathon Targets
• Team
• Credits
• Learning path (for contributors)
• Local Development
• Configuration
• Actor Model Backends
• Trace Export
• Fine-Tuning Artifacts
• Roadmap

Learning path (for contributors)

If you are new to Python or LLMs and want to understand how the Gradio UI connects to the Director / actor loop, read docs/LEARNING_PATH.md. It mirrors the recommended reading order for this repository and includes a UI-to-engine handler map.

Live demo

• Space: AI Puppet Theater
• Demo video: YouTube walkthrough
• Product blog: AI Puppet Theater: From Premise to Puppet Show
• Fine-tuning blog: Teaching a 1B Model to Speak Puppet JSON
• Social posts: LinkedIn · X/Twitter

Current Status

Playable final-submission demo. The Space, demo video, product blog, fine-tuning blog, social posts, Actor SFT dataset, LoRA adapter, and GGUF model are published.

Recommended Demo Settings

• Backend: HF API, or Deterministic fallback if the model path is unavailable
• Show length: Standard
• Voice: Browser TTS, or Edge TTS if available
• Flow: use Run One Beat first, throw a prop, then open Behind the Curtain

What It Does

• Creates a short puppet show from a user premise.
• Casts distinct puppet actors with goals, moods, memories, secrets, speaking styles, and optional tools.
• Uses a Director agent to choose the next speaker, beat type, pacing, stage effect, prop usage, secret reveals, and finale timing.
• Lets the audience throw props, summon an actor, or request a finale while the show is running.
• Renders an animated Gradio stage with active-speaker emphasis, transcript, Agent State, Director log, Browser TTS, optional Edge TTS, and trace export.
• Falls back to deterministic generation whenever model calls fail, keeping the demo runnable without setup.

Project artifacts

• Actor SFT dataset: AI-Puppet-Theater-Actor-SFT
• Actor LoRA adapter: AI-Puppet-Theater-MiniCPM5-Actor-LoRA
• Actor GGUF model: AI-Puppet-Theater-MiniCPM5-Actor-GGUF
• Base model: openbmb/MiniCPM5-1B
• Field Notes writeups: product blog and fine-tuning blog
• Agent design notes: AGENTS.md

Small-model highlight

AI Puppet Theater was built for the Build Small Hackathon with small-model constraints in mind.

The main fine-tuned Actor experiment uses openbmb/MiniCPM5-1B, a 1B-parameter model. The project publishes the Actor SFT dataset, a MiniCPM5 Actor LoRA adapter, and a GGUF version for local llama.cpp inference experiments.

Model-backed paths used by the app stay under the 32B limit:

• openbmb/MiniCPM5-1B: 1B base model for Actor fine-tuning.
• build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA: LoRA adapter for the Actor JSON task.
• build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF: local GGUF Actor model for llama.cpp.
• Qwen/Qwen3-4B-Instruct-2507: hosted model-backed path when configured.

The Space also keeps a deterministic fallback path for reliable no-token demos.

Hackathon fit

AI Puppet Theater was built for the Build Small Hackathon's Thousand Token Wood track as a small, playful Gradio app. The core idea is not another open-ended chatbot: a Director agent orchestrates a bounded puppet show, Actor agents return structured JSON beats, and audience actions become part of the show state.

The project stays within the small-model constraint. The main Actor fine-tuning experiment uses openbmb/MiniCPM5-1B, a 1B-parameter model, and publishes both a LoRA adapter and a GGUF version for local llama.cpp inference experiments. The hosted model-backed path uses Qwen/Qwen3-4B-Instruct-2507, which is also under the 32B limit.

The runtime treats model output as a proposal, not as trusted app state. Actor outputs are validated, optionally repaired, sanitized, traced, and backed by deterministic fallback so the stage can keep running even when a model returns invalid JSON.

How to demo

1. Enter a premise.
2. Click Create Show.
3. Use Run One Beat to watch the Director/Actor loop advance.
4. Throw a prop, summon an Actor, or request a finale from the audience controls.
5. Open Behind the Curtain to inspect Actor state, Director log, backend status, validation, tool calls, and sanitized trace.

Architecture

The diagrams below are rendered PNGs from Mermaid sources in assets/ so they display correctly on Hugging Face Spaces. Model outputs are treated as validated proposals; deterministic fallbacks keep the show bounded.

System Flow

Agent Loop

Runtime and Fallbacks

Agent Architecture

Director agent: chooses story phase, next speaker, beat type, stage effect, whether to use a prop, whether to reveal a secret, and when to end the scene. The Director can run through HF API, the experimental local OpenBMB runtime path, or deterministic policy.

Actor agents: respond as individual puppets with persona, goal, mood, memory, intent, secrets, gesture, emotion, and stage-ready short lines.

Theatrical tools: support validated actor actions: inspect_prop, consult_stage_oracle, and change_lighting. Tool calls are used to make the performance feel agentic, not to add hidden complexity.

Trace and fallbacks: every run records sanitized public events for Director decisions, actor outputs, tool results, audience actions, validation, latency, state updates, and fallback reasons. The app avoids exposing tokens, private paths, raw tracebacks, hidden reasoning, or unrevealed actor secrets.

Features

• Dynamic act length and progress-based pacing.
• LLM-assisted show bible for title, setting, backdrop description, Director, and puppet cast when a model backend is selected.
• LLM Director for speaker selection, beat planning, and finale control.
• Actor intent, memory, mood, state, goals, and secrets.
• Audience actions: prop throws, actor summons, and finale requests.
• Validated theatrical tools for prop inspection, stage-oracle hints, and lighting changes.
• Backdrop generation path: HF text-to-image from setting, LLM-picked stock URL fallback, then keyword stock fallback.
• Browser TTS with character voices or narrator mode.
• Optional Edge TTS character voices with graceful fallback.
• Actor engines: deterministic, HF API, local OpenBMB, local LoRA, and local GGUF.
• Director modes: deterministic, HF API, and local OpenBMB.
• Behind the Curtain log, Agent State panel, and downloadable trace JSON.

Model and Runtime

This project targets the hackathon's small-model spirit: model-backed paths should stay within the 32B parameter limit.

• HF API backend: Qwen/Qwen3-4B-Instruct-2507:nscale
• Local LoRA Actor backend: the published build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA adapter on openbmb/MiniCPM5-1B
• Local GGUF Actor backend: the published build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF model through llama.cpp / llama-cpp-python
• Experimental local / ZeroGPU runtime path: optional OpenBMB Transformers backend, prepared for ZeroGPU when enabled
• Optional image generation: stage backdrops may use black-forest-labs/FLUX.1-schnell; this is an optional visual path and may not be size-detected by the hackathon checker
• Fallback: deterministic Director and actor generation, requiring no token or model download

Deterministic mode is the safest no-token demo path. HF API mode is the recommended hosted model-backed path when HF_TOKEN is configured. The local LoRA, GGUF, OpenBMB, and ZeroGPU paths load lazily and may be slow or unavailable depending on Space hardware, dependencies, and model cache state. Model-backed dialogue paths are configured with models under the 32B constraint.

Hackathon Targets

• Thousand Token Wood track: playful, weird, AI-powered interactive story/toy experience.
• Tiny Titan fit: the main fine-tuned Actor path is based on openbmb/MiniCPM5-1B, a 1B-parameter model.
• OpenBMB sponsor: the Actor fine-tuning path uses openbmb/MiniCPM5-1B.
• Modal sponsor: Modal was used for the LoRA fine-tuning workflow.
• Well-Tuned: the project includes a published MiniCPM5-1B Actor LoRA adapter trained for structured Actor JSON beats.
• Llama Champion: the project includes a merged/quantized GGUF Actor model evaluated with llama.cpp.
• Sharing is Caring: the project publishes the Actor SFT dataset and supports sanitized trace export.
• Field Notes: the published product and fine-tuning blogs document what was built, what changed during the model experiments, and what was learned.

Submission notes

• The Space runs on ZeroGPU. We manually checked the account's Space count for the hackathon limit.
• We do not claim full off-grid behavior for the hosted Space because optional hosted model and image-generation paths may be configured.
• The project does include a local GGUF Actor model for llama.cpp inference experiments.

Team

• Shubham Setia
• Riya Bagaria

Credits

• OpenBMB for MiniCPM5-1B, the base model used for the Actor fine-tuning path.
• Modal for making the LoRA training workflow practical on CUDA.
• Black Forest Labs for FLUX.1-schnell, used for optional backdrop generation.

Local Development

Install dependencies:

uv sync

Run the app locally:

uv run python app.py

For live reload while editing app.py, use the Gradio CLI:

uv run gradio app.py

The Blocks instance is named app; Gradio discovers it automatically. If you need to pin the name explicitly:

uv run gradio app.py --demo-name app

Check syntax:

uv run python -m py_compile app.py puppet_theater/*.py

Generate Hugging Face Space requirements:

uv pip compile pyproject.toml -o requirements.txt

If tests are added or changed:

uv run pytest

Configuration

Running modes

Deterministic mode is the default and requires no model token or local model download. Use it for the fastest, most reliable demo path.

The optional local OpenBMB runtime path can generate actor lines or Director decisions locally, but it is experimental and may be slow on CPU Spaces because the model must be loaded into the Space process.

The optional local LoRA and GGUF actor paths use the fine-tuned MiniCPM5 Actor artifacts. They load lazily only when selected, validate every Actor JSON response, and fall back to deterministic actor generation on dependency, download, timeout, schema, or repetition failures.

The optional Hugging Face API backend uses remote inference through Hugging Face Inference Providers instead of downloading a model locally. It falls back to deterministic generation when a token is missing, the model is unavailable, the API times out, or the model returns invalid JSON.

Actor Model Backends

The Actor Engine selector supports:

• Deterministic: default and recommended for hosted demo reliability.
• Local LoRA Actor model: loads the published build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA adapter on openbmb/MiniCPM5-1B with transformers and peft.
• Local GGUF Actor model: loads a local quantized GGUF through llama-cpp-python; if ACTOR_GGUF_MODEL_PATH is not set, it downloads minicpm5-actor-q4_k_m.gguf from build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF.
• Local OpenBMB: the existing experimental local OpenBMB Transformers actor backend.
• Hugging Face API: the existing remote actor backend using Hugging Face Inference Providers.

The LoRA runtime libraries are included in the app runtime dependencies. The Space requirements.txt also includes llama-cpp-python through the upstream prebuilt CUDA wheel index so the GGUF backend can use GPU offload when the Space image has a compatible wheel. Model weights are still loaded lazily: the first selected LoRA or GGUF beat downloads/loads the model and keeps the loaded backend object in memory for the app lifecycle. If a selected backend fails, times out, errors, or returns invalid JSON, runtime output goes through first-JSON extraction, Actor JSON sanitization, validation, and then falls back to the deterministic actor generator.

Local model defaults are set for demo reliability: LoRA uses ACTOR_LORA_DEVICE=auto, GGUF uses ACTOR_GGUF_N_GPU_LAYERS=-1, and both local actor backends default to a 120 second generation timeout.

Local LoRA example:

ACTOR_MODEL_BACKEND=local_lora \
ACTOR_LORA_BASE_MODEL=openbmb/MiniCPM5-1B \
ACTOR_LORA_ADAPTER=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA \
uv run python app.py

Local GGUF example:

ACTOR_MODEL_BACKEND=local_gguf \
uv run python app.py

requirements.txt uses the upstream prebuilt CUDA 12.1 wheel index for llama-cpp-python to avoid source builds where possible on the current GPU Space. If the Space CUDA/runtime image changes, replace /cu121 with the matching CUDA wheel index from the llama-cpp-python docs, such as /cu124, /cu125, /cu130, or /cu132. For a CPU Space, replace /cu121 with /cpu and set ACTOR_GGUF_N_GPU_LAYERS=0. If no compatible prebuilt wheel exists for the Space image, pip may still fall back to a slow native build.

To force a local file instead of downloading from the Hugging Face GGUF repo:

ACTOR_MODEL_BACKEND=local_gguf \
ACTOR_GGUF_MODEL_PATH=finetune/outputs/gguf/minicpm5-actor-q4_k_m.gguf \
uv run python app.py

To point at a different GGUF repo or filename:

ACTOR_MODEL_BACKEND=local_gguf \
ACTOR_GGUF_REPO_ID=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF \
ACTOR_GGUF_FILENAME=minicpm5-actor-q4_k_m.gguf \
uv run python app.py

Limitations:

• Local LoRA needs enough GPU or CPU memory for openbmb/MiniCPM5-1B plus the adapter.
• Local GGUF needs llama-cpp-python and enough CPU/GPU memory for the quantized file; by default it downloads the configured GGUF from Hugging Face Hub, or uses ACTOR_GGUF_MODEL_PATH when set. The GGUF file is not committed to git.
• Hosted Spaces may use fallback unless a model backend is explicitly configured.
• GGUF is prompted with ChatML, matching the best eval format used for the Q4_K_M actor model.

Hugging Face API

Set HF_API_MODEL_ID to choose the remote model:

HF_API_MODEL_ID=Qwen/Qwen3-4B-Instruct-2507:nscale

The default model string includes the provider suffix used by Hugging Face Inference Providers. If that model is unavailable through your selected provider, use another chat-compatible model id from Hugging Face Inference Providers.

Premise casting (title, setting, minimal backdrop_description, director + puppets JSON) requests up to HF_API_SHOW_BIBLE_MAX_TOKENS completion tokens (default 1024, clamped between 256 and 4096). When HF_SETTING_BACKDROP_IMAGE is not disabled (default on) and HF_TOKEN is set, the app generates a JPEG backdrop from the setting sentence via Hugging Face text-to-image (model HF_BACKDROP_T2I_MODEL, default black-forest-labs/FLUX.1-schnell; optional provider HF_BACKDROP_T2I_PROVIDER; timeout HF_BACKDROP_T2I_TIMEOUT, default 90s; default stage-matched size 1152x512). If that is off or fails after an LLM cast, a follow-up chat completion can still pick a stock backdrop_image_url (up to HF_API_BACKDROP_URL_MAX_TOKENS, default 256). If that JSON was truncated at 200 tokens before, the UI fell back to the default Pip/Mina/Bolt cast; raising this cap fixes incomplete responses.

Configure the token as a Hugging Face Space secret named HF_TOKEN. HUGGINGFACEHUB_API_TOKEN is also supported. Do not commit tokens to the repository, README, app code, logs, or trace output.

For local runs, put HF_TOKEN=... in a file named .env at the repository root (next to app.py). The app loads that file on startup; you do not need to export it manually.

Local deterministic mode and app launch work without HF_TOKEN.

Stage backdrop

The show’s setting string (for example “A sun-dappled forest clearing with soft grass and distant trees, warm and quiet.”) is the narrative stage description. When HF_TOKEN is configured and HF_SETTING_BACKDROP_IMAGE is not set to 0/false/off, the app builds a text-to-image prompt from that setting (wide 16:9 puppet-stage framing, calm center, no text/watermarks) and calls the Hugging Face Inference API. Set HF_BACKDROP_T2I_PROVIDER to pin a provider such as nscale or fal-ai; leave it empty for provider auto-routing. The returned image is stored as an inline JPEG data URL on the session and used as the CSS stage background. If generation is disabled, fails, or there is no token, the app falls back to an LLM-picked stock https URL from backdrop_description, then to keyword-matched stock images in puppet_theater/session.py (_BACKDROP_RULES). Set HF_SETTING_BACKDROP_IMAGE=0 to skip image generation and use URL/keyword flow only.

Voice modes

Voice is optional. Off remains the default and the app works without audio support.

Browser TTS modes use the browser's built-in SpeechSynthesis API:

• Browser TTS: Character Voices gives each actor a deterministic browser voice style.
• Browser TTS: Narrator Only reads every latest line with the narrator style.

Edge TTS: Character Voices is experimental. It uses the optional edge-tts package to generate an MP3 for only the latest actor or tool line, then plays it through the Edge TTS audio player. Actor slots map to stable Edge voices, and repeated lines are cached in a temporary directory so they are not regenerated every time. Old generated files are cleaned up automatically.

If edge-tts is unavailable, network generation fails, or the service times out, the app shows a short friendly status and keeps the show running. Use Browser TTS or Off as the fallback. Edge TTS does not expose raw tracebacks, local cache paths, tokens, or secrets in the UI or trace output.

ZeroGPU local inference

This Space keeps sdk: gradio; it does not require Docker for ZeroGPU. To experiment with the local OpenBMB runtime path on ZeroGPU, set the Space hardware to ZeroGPU in the Hugging Face Space settings and set:

USE_ZEROGPU=true

USE_ZEROGPU defaults to false, so the app still starts locally and on regular CPU Spaces without requesting ZeroGPU. When enabled and the spaces package is available, local OpenBMB generation runs through a @spaces.GPU(duration=30) function. The app does not load local models at startup.

The Space requirements are resolved for Python 3.10 compatibility because ZeroGPU builds may use a Python 3.10 base image. The local project also supports Python 3.11. The runtime pins torch==2.8.0, which is the preferred ZeroGPU-compatible Torch version for this Space. llama-cpp-python is included in requirements.txt through its prebuilt CUDA wheel index for GGUF support on the current GPU Space; for CPU Space hardware, switch that index to /cpu and set ACTOR_GGUF_N_GPU_LAYERS=0. If the Space image cannot use the selected wheel, pip may fall back to a slow source build. If Hugging Face reports a different supported Torch version for ZeroGPU, update pyproject.toml, regenerate requirements.txt, and refresh uv.lock.

If ZeroGPU is not enabled, spaces is unavailable, local model loading fails, CUDA is unavailable inside the GPU function, or model output is invalid, the show falls back gracefully. The Hugging Face API backend remains selectable and is the recommended fallback when you have HF_TOKEN; deterministic mode remains the no-token safety path.

The Backend panel reports ZeroGPU enabled status, spaces availability, the Torch version, whether CUDA was visible inside the GPU function after a local generation attempt, and the latest fallback reason.

Trace Export

Every show keeps a lightweight agent trace in memory for the current Gradio session. The collapsed Behind the Curtain panel shows a compact human-readable trace summary with a copy button, plus the Director log. The collapsed Trace / Debug panel shows the full JSON preview.

After creating a show, use Download Trace JSON to save ai-puppet-theater-trace-<session_id>.json. The file contains a stable envelope with app_name, trace_version, session_id, created_at, premise, title, setting, show_length_mode, min/target/max beats, actor backend, Director mode, model id when used, and normalized events.

Trace events use a consistent public shape with event_type, timestamp or step, beat/story phase metadata, speaker, Director decisions, actor intent, actor memory updates, tool requests and results, audience actions, backend/model metadata, latency, validation status, fallback usage, and short fallback reasons where available.

Traces are sanitized so they are safe to publish later as a dataset: they do not include Hugging Face tokens, environment variable values, private local paths, raw tracebacks, hidden reasoning, or private secrets. Errors are reduced to short summaries and model failures are recorded as fallback events.

Fine-Tuning Artifacts

Fine-tuning scripts, publishing helpers, and model/dataset card templates live under finetune/. They are supporting project artifacts and are not part of the Space runtime.

• Fine-tuning README: finetune/README.md
• Actor SFT dataset repo: AI-Puppet-Theater-Actor-SFT
• MiniCPM5 Actor LoRA repo: AI-Puppet-Theater-MiniCPM5-Actor-LoRA
• MiniCPM5 Actor GGUF repo: AI-Puppet-Theater-MiniCPM5-Actor-GGUF

The current best adapter candidate is the v0 MiniCPM5 Actor LoRA. It was merged into openbmb/MiniCPM5-1B, converted to GGUF, and quantized as Q4_K_M for llama.cpp testing.

Best GGUF eval used llama-completion with -no-cnv, reasoning disabled, and chatml prompt formatting. On 40 Actor eval prompts, the Q4_K_M GGUF reached 39/40 extracted JSON parse, 39/40 exact top-level schema, 39/40 sanitized Actor JSON usable, and 0/40 interactive markers.

The v1 hardening run added useful validation/eval tooling but did not outperform v0 on full eval. Runtime use should still keep first-JSON extraction, Actor JSON validation, tool-request sanitization, and deterministic fallback.

Roadmap

• Publish a small sanitized trace dataset if time allows.
• Add more tested example prompts after live demo validation.
• Improve Actor SFT examples and compare local LoRA, GGUF, and hosted inference paths more systematically.
• Improve local GGUF inference UX and document recommended llama.cpp runtime presets.
• Continue polishing the stage UI and audience interaction loop.