README.md

metadata

title: Poetic · 诗情画意
emoji: 🖌️
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
python_version: '3.12'
license: apache-2.0
tags:
  - build-small-hackathon
  - track:thousand-token-wood
  - an-adventure-in-thousand-token-wood
  - achievement:offbrand
  - off-brand
  - achievement:bestdemo
  - best-demo
  - sponsor:openbmb
  - sponsor:modal
  - best-use-of-modal
  - minicpm
  - flux2-klein
  - cosyvoice2
  - musicgen
  - wan
  - text-to-image
  - text-to-speech
  - text-to-audio
  - image-to-video
short_description: 写一首诗，化作画、朗诵配乐与会动的小影片 — all small open models

Poetic · 诗情画意 — a poem becomes a painting, a recitation, and a living film

Write (or speak) a short Chinese poem. The app reads it back to you warmly, paints it in your chosen classical style, recites it with feeling over soft background music, and finally turns it into a short film — a multi-scene story drawn from the poem, animated and set to the recitation. A morning card you can send to family.

Built for elders first: warm jargon-free Simplified Chinese, one big button per step, and a 「一气呵成」 button that does the whole journey in one go.

How it works — five small open models, all under 32B

Step	中文	Model	Size
Understand the poem	读诗	openbmb/MiniCPM4.1-8B	8B
Paint it	作画	black-forest-labs/FLUX.2-klein-4B	4B
Recite it	诵诗	FunAudioLLM/CosyVoice2-0.5B	0.5B
Score it	配乐	facebook/musicgen-medium	1.5B
Bring it alive	入画	Wan-AI/Wan2.2-TI2V-5B	5B

Nothing here is a giant frontier model — the heaviest is 8B, most are a few billion or less, and the voice is half a billion parameters. That's the whole point of Build Small: a complete, delightful creative pipeline from models you could (mostly) run yourself.

The four ideas that make it work

1. Recite, don't sing. Open models can't sing like a closed commercial service — vocal naturalness and mixing are exactly where they trail. So instead of forcing a song, the poem is recited (诗朗诵): CosyVoice2 is driven with an instruction — 「用富有感情、舒缓、抑扬顿挫的语气，像朗诵古诗一样缓缓地读这首诗」 — so it varies pitch, emphasis and pace like a human reader. It's read sentence-by-sentence with measured pauses for proper cadence, then mixed voice-forward over a soft instrumental bed (MusicGen — a model built for text→instrumental, so the backing is a clean calm bed, not a coerced song). Pick the backing: 古琴 / 钢琴 / 弦乐 / 氛围.

2. One coherent living painting, not a morph or a slideshow. The painting is animated as a single continuous shot with Wan2.2-TI2V — the camera drifts slowly through the scene while mist and water move. We deliberately avoid multi-scene interpolation (which morphs unrelated scenes into each other) and crossfade montages (which read as slideshows): one coherent shot is the most natural, least "obviously-AI" result. The recitation + music play underneath.

3. Text-free generation, calligraphy on top. Every generated image is forced text-free; the poem calligraphy and an optional red 姓名章 (name seal) are composed on top by the app in classical vertical, right-to-left layout.

4. A GPU-free Space with no quota to exhaust. This Space runs on free CPU and holds no weights — every step calls out, so there's no ZeroGPU quota for judges to hit:

• 读诗 → OpenBMB's free hosted MiniCPM4.1-8B endpoint (sponsor: OpenBMB).
• 作画 / 诵诗 / 配乐 / 入画 → Modal serverless GPU (modal_app.py), each model in its own image so their conflicting dependencies never collide. The slower jobs (video, recitation) use an async spawn/poll pattern that dodges Modal's 150s synchronous web-endpoint limit. (sponsor: Modal — Best Use of Modal.)

Swappable providers

Every capability sits behind a provider seam (poetic/providers/) resolving named providers — minicpm_api, modal, local (on-Space ZeroGPU), api (HF Inference Providers) — selected by POETIC_MODE bundles or per-capability overrides (PAINT_PROVIDER=…). Swapping a model or backend is a config change, not a code change.

Run it elsewhere

pip install -r requirements.txt gradio
# hosted (this Space's default): MiniCPM free API + Modal endpoints
POETIC_MODE=hosted OPENBMB_API_KEY=sk-... \
  MODAL_PAINT_URL=... MODAL_RECITE_URL=... MODAL_SING_URL=... \
  MODAL_ANIMATE_START_URL=... MODAL_ANIMATE_RESULT_URL=... python app.py

Deploy the GPU backend with modal deploy modal_app.py. Pure logic is tested: pytest tests/ (interpretation parsing/repair, style clauses, text-free guards, storyboard scene parsing, recitation config, crossfade/concat/mux commands, card layout, provider bundles).

Links

• Demo video: https://youtu.be/oGuI0VyqIpE
• Social post: https://x.com/ooi_yee_fei/status/2066911818359214361

— Re-imagined from the poetic poem-to-art project on small open models for Build Small.