---
title: Poetic · 诗情画意
emoji: 🖌️
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
python_version: "3.12"
license: apache-2.0
tags:
  - build-small-hackathon
  - track:thousand-token-wood
  - an-adventure-in-thousand-token-wood
  - achievement:offbrand
  - off-brand
  - achievement:bestdemo
  - best-demo
  - sponsor:openbmb
  - sponsor:modal
  - best-use-of-modal
  - minicpm
  - flux2-klein
  - cosyvoice2
  - musicgen
  - wan
  - text-to-image
  - text-to-speech
  - text-to-audio
  - image-to-video
short_description: 写一首诗，化作画、朗诵配乐与会动的小影片 — all small open models
---

# Poetic · 诗情画意 — a poem becomes a painting, a recitation, and a living film

Write (or speak) a short Chinese poem. The app **reads it back to you warmly**,
**paints it** in your chosen classical style, **recites it** with feeling over
soft background music, and finally **turns it into a short film** — a multi-scene
story drawn from the poem, animated and set to the recitation. A morning card you
can send to family.

Built for elders first: warm jargon-free Simplified Chinese, one big button per
step, and a **「一气呵成」** button that does the whole journey in one go.

## How it works — five small open models, all under 32B

| Step | 中文 | Model | Size |
|------|------|-------|------|
| Understand the poem | 读诗 | [openbmb/MiniCPM4.1-8B](https://huggingface.co/openbmb/MiniCPM4.1-8B) | 8B |
| Paint it | 作画 | [black-forest-labs/FLUX.2-klein-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) | 4B |
| Recite it | 诵诗 | [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) | 0.5B |
| Score it | 配乐 | [facebook/musicgen-medium](https://huggingface.co/facebook/musicgen-medium) | 1.5B |
| Bring it alive | 入画 | [Wan-AI/Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) | 5B |

Nothing here is a giant frontier model — the **heaviest is 8B**, most are a few
billion or less, and the voice is **half a billion** parameters. That's the whole
point of Build Small: a complete, delightful creative pipeline from models you
could (mostly) run yourself.

## The four ideas that make it work

**1. Recite, don't sing.** Open models can't sing like a closed commercial
service — vocal naturalness and mixing are exactly where they trail. So instead
of forcing a song, the poem is **recited** (诗朗诵): CosyVoice2 is driven with an
*instruction* — *「用富有感情、舒缓、抑扬顿挫的语气，像朗诵古诗一样缓缓地读这首诗」* —
so it varies pitch, emphasis and pace like a human reader. It's read
sentence-by-sentence with measured pauses for proper cadence, then mixed
**voice-forward over a soft instrumental bed** (MusicGen — a model built for
text→instrumental, so the backing is a clean calm bed, not a coerced song). Pick
the backing: 古琴 / 钢琴 / 弦乐 / 氛围.

**2. One coherent living painting, not a morph or a slideshow.** The painting is
animated as a *single continuous shot* with **Wan2.2-TI2V** — the camera drifts
slowly through the scene while mist and water move. We deliberately avoid
multi-scene interpolation (which morphs unrelated scenes into each other) and
crossfade montages (which read as slideshows): one coherent shot is the most
natural, least "obviously-AI" result. The recitation + music play underneath.

**3. Text-free generation, calligraphy on top.** Every generated image is forced
**text-free**; the poem calligraphy and an optional red **姓名章 (name seal)** are
composed on top by the app in classical vertical, right-to-left layout.

**4. A GPU-free Space with no quota to exhaust.** This Space runs on **free CPU**
and holds no weights — every step calls out, so there's **no ZeroGPU quota** for
judges to hit:

- **读诗** → OpenBMB's free hosted **MiniCPM4.1-8B** endpoint *(sponsor: OpenBMB)*.
- **作画 / 诵诗 / 配乐 / 入画** → **[Modal](https://modal.com)** serverless GPU
  (`modal_app.py`), each model in its own image so their conflicting dependencies
  never collide. The slower jobs (video, recitation) use an async spawn/poll
  pattern that dodges Modal's 150s synchronous web-endpoint limit.
  *(sponsor: Modal — Best Use of Modal.)*

### Swappable providers

Every capability sits behind a provider seam (`poetic/providers/`) resolving
**named providers** — `minicpm_api`, `modal`, `local` (on-Space ZeroGPU),
`api` (HF Inference Providers) — selected by `POETIC_MODE` bundles or
per-capability overrides (`PAINT_PROVIDER=…`). Swapping a model or backend is a
config change, not a code change.

## Run it elsewhere

```bash
pip install -r requirements.txt gradio
# hosted (this Space's default): MiniCPM free API + Modal endpoints
POETIC_MODE=hosted OPENBMB_API_KEY=sk-... \
  MODAL_PAINT_URL=... MODAL_RECITE_URL=... MODAL_SING_URL=... \
  MODAL_ANIMATE_START_URL=... MODAL_ANIMATE_RESULT_URL=... python app.py
```

Deploy the GPU backend with `modal deploy modal_app.py`. Pure logic is tested:
`pytest tests/` (interpretation parsing/repair, style clauses, text-free guards,
storyboard scene parsing, recitation config, crossfade/concat/mux commands, card
layout, provider bundles).

## Links

- **Demo video:** https://youtu.be/oGuI0VyqIpE
- **Social post:** https://x.com/ooi_yee_fei/status/2066911818359214361

— Re-imagined from the *poetic* poem-to-art project on small open models for
Build Small.