| --- |
| title: The Shape Of Words |
| emoji: 🔥 |
| colorFrom: indigo |
| colorTo: purple |
| sdk: gradio |
| sdk_version: 6.18.0 |
| python_version: '3.12' |
| app_file: app.py |
| pinned: false |
| short_description: Tell a story. Watch its feelings take shape. |
| license: mit |
| tags: |
| - track:wood |
| - sponsor:openbmb |
| - sponsor:modal |
| - achievement:offbrand |
| - achievement:llama |
| - achievement:fieldnotes |
| --- |
| |
| # The Shape of Words |
|
|
| A collaborative storytelling toy: tell a story one beat at a time, and the model |
| turns the *feeling* of each beat into an abstract shape. At the end, hidden |
| colors bloom through the shapes one by one, the model adds its own companion |
| shapes, and everything composes into a single arrangement you can rearrange, |
| sonify, save, or turn into a painting. |
|
|
| - [`Live Space`](https://huggingface.co/spaces/build-small-hackathon/the_shape_of_words) |
| - [`Demo Video`](https://youtu.be/I0XPUqYRvYE) |
| - [`Social Media Post`](https://www.linkedin.com/posts/resakemal_the-shape-of-words-field-notes-activity-7472374372591153152-Kipg) |
| - **Field notes (deep dive):** [`Hugging Face Article`](https://huggingface.co/blog/resakemal/the-shape-of-words) or [`field_notes.md`](field_notes.md) |
|
|
| Built on **gradio.Server** (FastAPI + Gradio's API engine, gradio 6.x) so the |
| custom HTML/JS frontend gets queuing, streaming, and Hugging Face Spaces hosting. |
|
|
| --- |
|
|
| ## Architecture |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ Browser — static/index.html (custom HTML/JS, no build step) │ |
| │ • renderer · mappings · layout · sound · share card · UI │ |
| │ • all rendering is client-side (JS port of engine/) │ |
| │ • calls the backend for MODEL JUDGMENT only │ |
| └───────────────┬─────────────────────────────────────────────┘ |
| │ /gradio_api/call/<name> (queued, SSE) |
| ┌───────────────▼─────────────────────────────────────────────┐ |
| │ app.py — gradio.Server │ |
| │ @app.api judge_beat · judge_beat_segmented · │ |
| │ continue_story · reveal · title_story · paint │ |
| │ @app.get "/" serves the frontend │ |
| └───────┬──────────────────────────────────────┬──────────────┘ |
| │ model/backend.py │ model/painter.py |
| │ STORY_SHAPES_BACKEND │ STORY_SHAPES_PAINT_BACKEND |
| ▼ ▼ |
| llamacpp GGUF in-process (llama.cpp) modal Modal endpoint (FLUX.2 Klein) |
| modal_llm Modal endpoint (transformers) flux_local in-process diffusers |
| ``` |
|
|
| The **model only judges affect** (valence / arousal / dominance + a few flags + |
| a voice line). Everything visual — geometry, color (OKLCH), layout, scoring — is |
| deterministic code (in `engine/`, mirrored client-side for rendering), grounded |
| in perception research (see the spec's references). |
|
|
| --- |
|
|
| ## Run locally (llama.cpp + MiniCPM4.1-8B) |
|
|
| The default backend runs the LLM in-process via `llama-cpp-python` — no daemon, |
| no separate model server. |
|
|
| ```bash |
| pip install -r requirements.txt |
| python app.py # serves http://localhost:7860 |
| ``` |
|
|
| On first run it pulls a quantized GGUF (default `openbmb/MiniCPM4.1-8B-GGUF`, |
| Q4_K_M, ~5 GB). With a GPU it offloads all layers automatically; **CPU-only?** |
| set `STORY_SHAPES_LLAMACPP_GPU_LAYERS=0`. Then open http://localhost:7860. |
|
|
| Config via env vars (all optional): |
| | var | default | meaning | |
| |-----|---------|---------| |
| | `STORY_SHAPES_BACKEND` | `llamacpp` | `llamacpp` (in-process) or `modal_llm` (Modal endpoint) | |
| | `STORY_SHAPES_LLAMACPP_REPO` | `openbmb/MiniCPM4.1-8B-GGUF` | GGUF Hub repo | |
| | `STORY_SHAPES_LLAMACPP_FILE` | `*Q4_K_M.gguf` | GGUF filename glob | |
| | `STORY_SHAPES_LLAMACPP_GPU_LAYERS` | `-1` | layers on GPU (`-1` all, `0` CPU-only) | |
| | `PORT` | `7860` | server port | |
|
|
| > The frontend falls back to a built-in keyword **stub** if the backend is |
| > unreachable, so `static/index.html` also opens standalone for quick UI testing. |
|
|
| --- |
|
|
| ## Deploy to a Hugging Face Space |
|
|
| The README YAML frontmatter above is the HF Space config (`sdk: gradio`, |
| `sdk_version: 6.18.0`, `app_file: app.py`). The gradio version is pinned there |
| because `gr.Server` requires gradio 6.x. Push this repo to a Space and set the |
| env vars below. |
|
|
| **Recommended setup — both models on one 24 GB GPU (e.g. a dedicated L4):** |
|
|
| | var | value | notes | |
| |-----|-------|-------| |
| | `STORY_SHAPES_BACKEND` | `llamacpp` | MiniCPM4.1-8B as a GGUF via llama.cpp (~5 GB at Q4_K_M) | |
| | `STORY_SHAPES_PAINT_BACKEND` | `flux_local` | FLUX.2 Klein 4B in-process with CPU offload | |
|
|
| A quantized LLM (~5 GB) + cpu-offloaded FLUX (~8–12 GB peak) fit comfortably |
| together on 24 GB. A **dedicated** GPU Space is also not ZeroGPU, which avoids |
| ZeroGPU's per-request quota attribution. |
|
|
| **Alternative — offload the heavy work to Modal** (Space then needs no GPU): |
|
|
| | var | value | notes | |
| |-----|-------|-------| |
| | `STORY_SHAPES_BACKEND` | `modal_llm` | LLM on a Modal GPU (`modal deploy modal_llm.py`) | |
| | `STORY_SHAPES_LLM_MODAL_URL` | *(from `modal deploy`)* | the printed `modal_llm` URL | |
| | `STORY_SHAPES_PAINT_BACKEND` | `modal` | painter on a Modal GPU (`modal deploy modal_painter.py`) | |
| | `STORY_SHAPES_PAINT_MODAL_URL` | *(from `modal deploy`)* | the printed `modal_painter` URL | |
|
|
| **DISCLAIMER: I was attempting the Off the Grid challenge (local models only) but I could not test the model deployment on ZeroGPU due to consistently encountering 429 Too Many Requests error (more details in the "deployment story" in the Architecture section in [`field_notes.md`](field_notes.md)). For stable/expected results on the Space I'm routing both models to Modal (shape judgment & painter; this will increase initial inference time due to cold start); I apologize for the inconvenience. I have attached a local_run_logs.txt for proof that the shape judgment model runs locally.** |
|
|
| > **Why not run MiniCPM4.1 in-process under plain `transformers`?** gradio 6 |
| > forces `transformers ≥ 5.0`, and MiniCPM4.1's `trust_remote_code` modeling |
| > code is incompatible with transformers 5.x (it loads with a shim but crashes |
| > during generation). That's exactly why the LLM runs via **llama.cpp** (no |
| > transformers at all) or on **Modal** with transformers pinned to 4.x. |
|
|
| See `requirements.txt` for the llama.cpp install (a prebuilt CUDA wheel, with a |
| one-comment CPU-fallback toggle) and the diffusers-from-source line FLUX needs. |
|
|
| --- |
|
|
| ## Layout |
|
|
| ``` |
| app.py gradio.Server: API endpoints + serves frontend |
| requirements.txt |
| engine/ deterministic core (also mirrored in the frontend JS) |
| mappings.py affect -> geometry, affect -> color (OKLCH, research-grounded) |
| renderer.py geometry -> shape points |
| scorer.py zone-based puzzle scoring (+ band + hint) |
| model/ |
| backend.py LLM abstraction: llamacpp / modal_llm |
| painter.py painting abstraction: modal / flux_local |
| modal_llm.py Modal GPU endpoint for the LLM (deploy once) |
| modal_painter.py Modal GPU endpoint for FLUX.2 Klein (deploy once) |
| prompts/ system prompts + JSON schemas / GBNF grammars per call |
| static/ |
| index.html the entire frontend (custom HTML/JS) |
| docs/ |
| make_spec_figures.py renders the spec/field-notes illustrations from the engine |
| ``` |
|
|
| --- |
|
|
| ## Features |
|
|
| - ✅ Exploration mode: full loop, hidden-color reveal, AI companion shapes, |
| narrative composition, drag / rotate / layers / hide / save. |
| - ✅ **Per-segment essence**: shape morphs through each phrase's felt character. |
| - ✅ **OKLCH color**: perceptually-uniform lightness (Python + JS parity). |
| - ✅ **Reveal sound** (Web Audio): pentatonic chimes, shape-born tick, atmospheric |
| pad, replay — togglable in Settings. |
| - ✅ **Share card** (canvas + painting): model-named title, popup, PNG export. |
| - ✅ **Origin tooltip**: hover/tap a shape or layer row to see its source beat. |
| - ✅ **Painting step (FLUX.2 Klein img2img)**: "Paint this"; free style field + |
| preset styles + 🎲 Surprise me; randomized seed; `modal` or `flux_local`. |
| - ✅ **LLM backends**: `llamacpp` (GGUF, Llama Champion) and `modal_llm`. Both |
| produce schema-constrained JSON. |
| - ✅ **Felt-quality eval** (`eval_felt_quality.py`): A/B models on a fixed set. |
| - ✅ **HF Space deployment** (gradio 6.18, `gr.Server`). |
|
|
