README.md

metadata

title: The Shape Of Words
emoji: 🔥
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.18.0
python_version: '3.12'
app_file: app.py
pinned: false
short_description: Tell a story. Watch its feelings take shape.
license: mit
tags:
  - track:wood
  - sponsor:openbmb
  - sponsor:modal
  - achievement:offbrand
  - achievement:llama
  - achievement:fieldnotes

The Shape of Words

A collaborative storytelling toy: tell a story one beat at a time, and the model turns the feeling of each beat into an abstract shape. At the end, hidden colors bloom through the shapes one by one, the model adds its own companion shapes, and everything composes into a single arrangement you can rearrange, sonify, save, or turn into a painting.

• Live Space
• Demo Video
• Social Media Post
• Field notes (deep dive): Hugging Face Article or field_notes.md

Built on gradio.Server (FastAPI + Gradio's API engine, gradio 6.x) so the custom HTML/JS frontend gets queuing, streaming, and Hugging Face Spaces hosting.

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│ Browser — static/index.html (custom HTML/JS, no build step) │
│  • renderer · mappings · layout · sound · share card · UI   │
│  • all rendering is client-side (JS port of engine/)        │
│  • calls the backend for MODEL JUDGMENT only                │
└───────────────┬─────────────────────────────────────────────┘
                │  /gradio_api/call/<name>   (queued, SSE)
┌───────────────▼─────────────────────────────────────────────┐
│ app.py — gradio.Server                                      │
│  @app.api  judge_beat · judge_beat_segmented ·              │
│            continue_story · reveal · title_story · paint    │
│  @app.get "/"  serves the frontend                          │
└───────┬──────────────────────────────────────┬──────────────┘
        │ model/backend.py                     │ model/painter.py
        │  STORY_SHAPES_BACKEND                │  STORY_SHAPES_PAINT_BACKEND
        ▼                                      ▼
  llamacpp  GGUF in-process (llama.cpp)   modal       Modal endpoint (FLUX.2 Klein)
  modal_llm Modal endpoint (transformers) flux_local  in-process diffusers

The model only judges affect (valence / arousal / dominance + a few flags + a voice line). Everything visual — geometry, color (OKLCH), layout, scoring — is deterministic code (in engine/, mirrored client-side for rendering), grounded in perception research (see the spec's references).

---

Run locally (llama.cpp + MiniCPM4.1-8B)

The default backend runs the LLM in-process via llama-cpp-python — no daemon, no separate model server.

pip install -r requirements.txt
python app.py            # serves http://localhost:7860

On first run it pulls a quantized GGUF (default openbmb/MiniCPM4.1-8B-GGUF, Q4_K_M, ~5 GB). With a GPU it offloads all layers automatically; CPU-only? set STORY_SHAPES_LLAMACPP_GPU_LAYERS=0. Then open http://localhost:7860.

Config via env vars (all optional):

var	default	meaning
STORY_SHAPES_BACKEND	llamacpp	llamacpp (in-process) or modal_llm (Modal endpoint)
STORY_SHAPES_LLAMACPP_REPO	openbmb/MiniCPM4.1-8B-GGUF	GGUF Hub repo
STORY_SHAPES_LLAMACPP_FILE	*Q4_K_M.gguf	GGUF filename glob
STORY_SHAPES_LLAMACPP_GPU_LAYERS	-1	layers on GPU (-1 all, 0 CPU-only)
PORT	7860	server port

The frontend falls back to a built-in keyword stub if the backend is unreachable, so static/index.html also opens standalone for quick UI testing.

---

Deploy to a Hugging Face Space

The README YAML frontmatter above is the HF Space config (sdk: gradio, sdk_version: 6.18.0, app_file: app.py). The gradio version is pinned there because gr.Server requires gradio 6.x. Push this repo to a Space and set the env vars below.

Recommended setup — both models on one 24 GB GPU (e.g. a dedicated L4):

var	value	notes
STORY_SHAPES_BACKEND	llamacpp	MiniCPM4.1-8B as a GGUF via llama.cpp (~5 GB at Q4_K_M)
STORY_SHAPES_PAINT_BACKEND	flux_local	FLUX.2 Klein 4B in-process with CPU offload

A quantized LLM (5 GB) + cpu-offloaded FLUX (8–12 GB peak) fit comfortably together on 24 GB. A dedicated GPU Space is also not ZeroGPU, which avoids ZeroGPU's per-request quota attribution.

Alternative — offload the heavy work to Modal (Space then needs no GPU):

var	value	notes
STORY_SHAPES_BACKEND	modal_llm	LLM on a Modal GPU (modal deploy modal_llm.py)
STORY_SHAPES_LLM_MODAL_URL	(from modal deploy)	the printed modal_llm URL
STORY_SHAPES_PAINT_BACKEND	modal	painter on a Modal GPU (modal deploy modal_painter.py)
STORY_SHAPES_PAINT_MODAL_URL	(from modal deploy)	the printed modal_painter URL

DISCLAIMER: I was attempting the Off the Grid challenge (local models only) but I could not test the model deployment on ZeroGPU due to consistently encountering 429 Too Many Requests error (more details in the "deployment story" in the Architecture section in field_notes.md). For stable/expected results on the Space I'm routing both models to Modal (shape judgment & painter; this will increase initial inference time due to cold start); I apologize for the inconvenience. I have attached a local_run_logs.txt for proof that the shape judgment model runs locally.

Why not run MiniCPM4.1 in-process under plain transformers? gradio 6 forces transformers ≥ 5.0, and MiniCPM4.1's trust_remote_code modeling code is incompatible with transformers 5.x (it loads with a shim but crashes during generation). That's exactly why the LLM runs via llama.cpp (no transformers at all) or on Modal with transformers pinned to 4.x.

See requirements.txt for the llama.cpp install (a prebuilt CUDA wheel, with a one-comment CPU-fallback toggle) and the diffusers-from-source line FLUX needs.

---

Layout

app.py                 gradio.Server: API endpoints + serves frontend
requirements.txt
engine/                deterministic core (also mirrored in the frontend JS)
  mappings.py          affect -> geometry, affect -> color (OKLCH, research-grounded)
  renderer.py          geometry -> shape points
  scorer.py            zone-based puzzle scoring (+ band + hint)
model/
  backend.py           LLM abstraction: llamacpp / modal_llm
  painter.py           painting abstraction: modal / flux_local
modal_llm.py           Modal GPU endpoint for the LLM (deploy once)
modal_painter.py       Modal GPU endpoint for FLUX.2 Klein (deploy once)
prompts/               system prompts + JSON schemas / GBNF grammars per call
static/
  index.html           the entire frontend (custom HTML/JS)
docs/
  make_spec_figures.py renders the spec/field-notes illustrations from the engine

---

Features

• ✅ Exploration mode: full loop, hidden-color reveal, AI companion shapes, narrative composition, drag / rotate / layers / hide / save.
• ✅ Per-segment essence: shape morphs through each phrase's felt character.
• ✅ OKLCH color: perceptually-uniform lightness (Python + JS parity).
• ✅ Reveal sound (Web Audio): pentatonic chimes, shape-born tick, atmospheric pad, replay — togglable in Settings.
• ✅ Share card (canvas + painting): model-named title, popup, PNG export.
• ✅ Origin tooltip: hover/tap a shape or layer row to see its source beat.
• ✅ Painting step (FLUX.2 Klein img2img): "Paint this"; free style field + preset styles + 🎲 Surprise me; randomized seed; modal or flux_local.
• ✅ LLM backends: llamacpp (GGUF, Llama Champion) and modal_llm. Both produce schema-constrained JSON.
• ✅ Felt-quality eval (eval_felt_quality.py): A/B models on a fixed set.
• ✅ HF Space deployment (gradio 6.18, gr.Server).