Spaces:
Sleeping
Sleeping
| title: Poetic · 诗情画意 | |
| emoji: 🖌️ | |
| colorFrom: red | |
| colorTo: yellow | |
| sdk: gradio | |
| sdk_version: 6.17.3 | |
| app_file: app.py | |
| python_version: "3.12" | |
| license: apache-2.0 | |
| tags: | |
| - build-small-hackathon | |
| - track:thousand-token-wood | |
| - an-adventure-in-thousand-token-wood | |
| - achievement:offbrand | |
| - off-brand | |
| - achievement:bestdemo | |
| - best-demo | |
| - sponsor:openbmb | |
| - sponsor:modal | |
| - best-use-of-modal | |
| - minicpm | |
| - flux2-klein | |
| - cosyvoice2 | |
| - musicgen | |
| - wan | |
| - text-to-image | |
| - text-to-speech | |
| - text-to-audio | |
| - image-to-video | |
| short_description: 写一首诗，化作画、朗诵配乐与会动的小影片 — all small open models | |
| # Poetic · 诗情画意 — a poem becomes a painting, a recitation, and a living film | |
| Write (or speak) a short Chinese poem. The app **reads it back to you warmly**, | |
| **paints it** in your chosen classical style, **recites it** with feeling over | |
| soft background music, and finally **turns it into a short film** — a multi-scene | |
| story drawn from the poem, animated and set to the recitation. A morning card you | |
| can send to family. | |
| Built for elders first: warm jargon-free Simplified Chinese, one big button per | |
| step, and a **「一气呵成」** button that does the whole journey in one go. | |
| ## How it works — five small open models, all under 32B | |
| | Step | 中文 | Model | Size | | |
| |------|------|-------|------| | |
| | Understand the poem | 读诗 | [openbmb/MiniCPM4.1-8B](https://huggingface.co/openbmb/MiniCPM4.1-8B) | 8B | | |
| | Paint it | 作画 | [black-forest-labs/FLUX.2-klein-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) | 4B | | |
| | Recite it | 诵诗 | [FunAudioLLM/CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) | 0.5B | | |
| | Score it | 配乐 | [facebook/musicgen-medium](https://huggingface.co/facebook/musicgen-medium) | 1.5B | | |
| | Bring it alive | 入画 | [Wan-AI/Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) | 5B | | |
| Nothing here is a giant frontier model — the **heaviest is 8B**, most are a few | |
| billion or less, and the voice is **half a billion** parameters. That's the whole | |
| point of Build Small: a complete, delightful creative pipeline from models you | |
| could (mostly) run yourself. | |
| ## The four ideas that make it work | |
| **1. Recite, don't sing.** Open models can't sing like a closed commercial | |
| service — vocal naturalness and mixing are exactly where they trail. So instead | |
| of forcing a song, the poem is **recited** (诗朗诵): CosyVoice2 is driven with an | |
| *instruction* — *「用富有感情、舒缓、抑扬顿挫的语气，像朗诵古诗一样缓缓地读这首诗」* — | |
| so it varies pitch, emphasis and pace like a human reader. It's read | |
| sentence-by-sentence with measured pauses for proper cadence, then mixed | |
| **voice-forward over a soft instrumental bed** (MusicGen — a model built for | |
| text→instrumental, so the backing is a clean calm bed, not a coerced song). Pick | |
| the backing: 古琴 / 钢琴 / 弦乐 / 氛围. | |
| **2. One coherent living painting, not a morph or a slideshow.** The painting is | |
| animated as a *single continuous shot* with **Wan2.2-TI2V** — the camera drifts | |
| slowly through the scene while mist and water move. We deliberately avoid | |
| multi-scene interpolation (which morphs unrelated scenes into each other) and | |
| crossfade montages (which read as slideshows): one coherent shot is the most | |
| natural, least "obviously-AI" result. The recitation + music play underneath. | |
| **3. Text-free generation, calligraphy on top.** Every generated image is forced | |
| **text-free**; the poem calligraphy and an optional red **姓名章 (name seal)** are | |
| composed on top by the app in classical vertical, right-to-left layout. | |
| **4. A GPU-free Space with no quota to exhaust.** This Space runs on **free CPU** | |
| and holds no weights — every step calls out, so there's **no ZeroGPU quota** for | |
| judges to hit: | |
| - **读诗** → OpenBMB's free hosted **MiniCPM4.1-8B** endpoint *(sponsor: OpenBMB)*. | |
| - **作画 / 诵诗 / 配乐 / 入画** → **[Modal](https://modal.com)** serverless GPU | |
| (`modal_app.py`), each model in its own image so their conflicting dependencies | |
| never collide. The slower jobs (video, recitation) use an async spawn/poll | |
| pattern that dodges Modal's 150s synchronous web-endpoint limit. | |
| *(sponsor: Modal — Best Use of Modal.)* | |
| ### Swappable providers | |
| Every capability sits behind a provider seam (`poetic/providers/`) resolving | |
| **named providers** — `minicpm_api`, `modal`, `local` (on-Space ZeroGPU), | |
| `api` (HF Inference Providers) — selected by `POETIC_MODE` bundles or | |
| per-capability overrides (`PAINT_PROVIDER=…`). Swapping a model or backend is a | |
| config change, not a code change. | |
| ## Run it elsewhere | |
| ```bash | |
| pip install -r requirements.txt gradio | |
| # hosted (this Space's default): MiniCPM free API + Modal endpoints | |
| POETIC_MODE=hosted OPENBMB_API_KEY=sk-... \ | |
| MODAL_PAINT_URL=... MODAL_RECITE_URL=... MODAL_SING_URL=... \ | |
| MODAL_ANIMATE_START_URL=... MODAL_ANIMATE_RESULT_URL=... python app.py | |
| ``` | |
| Deploy the GPU backend with `modal deploy modal_app.py`. Pure logic is tested: | |
| `pytest tests/` (interpretation parsing/repair, style clauses, text-free guards, | |
| storyboard scene parsing, recitation config, crossfade/concat/mux commands, card | |
| layout, provider bundles). | |
| ## Links | |
| - **Demo video:** https://youtu.be/oGuI0VyqIpE | |
| - **Social post:** https://x.com/ooi_yee_fei/status/2066911818359214361 | |
| — Re-imagined from the *poetic* poem-to-art project on small open models for | |
| Build Small. | |