field_notes.md

The Shape of Words — Field Notes

A collaborative storytelling toy (https://huggingface.co/spaces/build-small-hackathon/the_shape_of_words) where you tell a story sentence by sentence, the model conjures shapes from the feeling of each story beat, and the shapes compose into a final picture. Created for the Hugging Face Build Small Hackathon 2026.

Demo video: https://youtu.be/I0XPUqYRvYE

Note: please try the toy first if you haven't — this post digs into how it works and will spoil the surprises inside!

---

0. Preface

The Hugging Face Build Small Hackathon 2026 (https://huggingface.co/build-small-hackathon) focuses on applications built with models under 32B parameters. It has three main rules:

1. No model may exceed 32B parameters (multiple models are fine, as long as each is within the limit).
2. The app must be a Gradio app hosted as a Hugging Face Space.
3. The submission also needs a short demo video and a social media post.

There are optional challenges that grant extra points:

1. Off the Grid (no cloud APIs, only local models) ✅* — MiniCPM4.1-8B (via llama.cpp) and FLUX.2 Klein 4B both run locally; a Modal deployment is offered as an optional cloud fallback.
2. Well-Tuned (use a fine-tuned model you've published on Hugging Face).
3. Off-Brand (custom frontend on Gradio, using gr.Server) ✅
4. Llama Champion (uses the llama.cpp runtime) ✅ — MiniCPM4.1-8B runs on top of llama.cpp.
5. Sharing is Caring (share an agent trace on the Hugging Face Hub).
6. Field Notes (a blog post/report about the app) ✅ — this document!

*There was some issues in deployment and testing - refer to "deployment story" in the Architecture section.

Hackathon model recommendations relevant to this app:

• Shape judgment: OpenBMB MiniCPM4.1-8B (https://huggingface.co/openbmb/MiniCPM4.1-8B)
• Image generation: FLUX.2 Klein 4B (https://huggingface.co/black-forest-labs/FLUX.2-klein-4B)

This app is for the Adventure in Thousand Token Wood track — build something delightful that wouldn't exist without AI. The track judges on:

• Genuinely delightful (would you show a friend?)
• AI is load-bearing for the experience
• Originality of concept
• Polish of the Gradio app

---

1. Concept

You tell a story one beat (1–3 sentences) at a time. The model reads each beat and, when a beat is strong enough, conjures an essence shape — the distilled essence of that beat's feeling — and you choose freely among a few shape options. After a set number of beats, hidden colors bloom through the shapes one by one in story order, the model adds its own companion shapes, and everything composes into a single layered picture you can rearrange. From there you can save the arrangement or use it as the base for a generated painting.

The shape is a companion to the story, not a literal illustration of it. Form follows evoked feeling, not depicted objects — "a flying rhinoceros with fire wings" and "shattered glass" can produce similar shapes (both sharp, energetic), while the story text is never deformed to chase a shape.

---

2. Guiding Principle

The model does the fun thing. Every creative judgment — a beat's felt form, its mood as color, whether it deserves a shape, story coherence, pacing, companion shapes, co-authoring, and per-segment phrase chunking — belongs to the model. Deterministic code only renders, scores, and enforces layout constraints; it never decides meaning. Take the model away and the toy stops being interesting — that's the bar every feature is held to.

---

3. Exploration Mode

The joy is the story and the surprise of the shapes it grows. The essence is offered as a gift, and the model curates which beats become shapes.

• Warm-up: 1 beat before the first shape is eligible, so the model has a little context to work with; the first shape can appear as early as beat 2.
• Shape budget with budget-aware escalation. On every call the model sees the whole state of the story: all beats so far, which beats already have a shape, how many beats remain, and how many shapes are left in the budget. When no shape has appeared yet it's told to be generous; after the first shape it leans toward saving the budget for the strongest remaining beats.
• You pick the shape. The model offers three options — the essence (distilled aggregate affect, pushed toward the extremes), your beat (the same affect without the distillation), and freeform (random affect, for surprise). All are greyscale; color stays hidden until the reveal.
• Pass-the-pen. You can hand the pen to the model to continue — or even start — the story with 1–3 sentences, which feed through the same judgment pipeline. A shape born from such a beat is marked "from AI's turn."

---

4. Essence Generation

The essence is the distilled core of the triggering beat: its dominant felt quality, stripped of incidental noise, at a characteristic, bounded intensity.

• Simple/understated beat → the essence is more intense than the literal words (the common case — a quiet line still has a clear feeling).
• Over-loaded beat → the essence is the clean core, found by precision rather than blandness.

---

5. Shape Layers (form, posture & color)

Each shape is built from three layers. The affective layer is the model's judgment and drives the hidden color. The geometric layer is what's rendered. The expressive layer adds personality on top.

5.1 Affective Layer

Three near-independent axes in [0,1]:

• valence — unpleasant ↔ pleasant
• arousal — calm ↔ energized
• dominance — submissive/small ↔ commanding/large

Three is the established sweet spot [PAD; Mehrabian & Russell].

5.2 Geometric Layer

• spikiness — angularity of the boundary
• compactness — tight/round ↔ thin/elongated
• segmentability — single blob ↔ distinct lobes
• symmetry — regular ↔ irregular
• scale — size; driven by dominance

Spikiness, compactness, and segmentability are the three dimensions explaining ~97% of the variance in preattentive shape perception [Huang 2020].

5.3 Expressive Layer

Two touches that give the forms more variety and intent across the emotion corners:

• Corner-aware Bézier boundary (ringToPath). The sampled polygon is converted to cubic Bézier segments via a corner-aware Catmull-Rom pass: both the handle tension and the corner-detection threshold are keyed to spikiness, so calm shapes get smooth flowing curves while tense shapes keep genuine sharp corners. The placed/rotated polygon is re-Béziered after transform (Catmull-Rom is affine-covariant, so smoothing after the transform is correct).
• Gesture / directional lean. Beyond the radial form, each shape leans: valence drives a vertical lean (pleasant rises, heavy sags) and dominance a horizontal one (commanding leans forward), with arousal amplifying the magnitude. Implemented as r·(1 + lean·cos(θ − φ)) in the radial function — this breaks the diagram-like radial symmetry the pure essence shapes had, giving each form a sense of intent.

5.4 Spikiness Renderer

Spikiness uses (2/π)·arcsin(sin(n·θ)) — a triangle wave — for genuinely sharp tips rather than smooth sine bumps. Tooth count is 5–9 (fewer reads bolder), and both amplitude and tip sharpness ramp with spikiness. Verified identical in Python and JS.

5.5 Affect → Color (hidden until reveal)

Four hue anchors, interpolated circularly. Colors are computed in OKLCH (OKLab in cylindrical form), not HSV — because OKLCH's lightness is perceptually uniform: equal lightness reads as equal brightness across hues. Under HSV, a full-"value" yellow looked far brighter than a full-"value" blue, which muddied the reveal's brightness arc. The anchor hues, in OKLCH degrees:

emotion corner	hue
calm-pleasant (low A, high V)	green (145°)
joyful (high A, high V)	yellow-orange (85°)
angry (high A, low V)	red (29°)
sad (low A, low V)	blue (264°)

• Hue ← valence + arousal (circular bilinear interpolation between the four anchors)
• Chroma ← arousal (+ a little dominance) — calm reads muted, energized reads vivid
• Lightness ← valence (− dominance) — pleasant reads bright, heavy reads dim

Conversion uses Björn Ottosson's OKLab→sRGB matrices, gamut-clamped, and is verified identical in Python (engine/mappings.py) and JS (static/index.html).

---

6. Settings

Setting	Default	Range / options
Story beats	6	3–10
Shapes	3	2–max(2, n−2)
Companion shapes	on	toggle
Per-segment essence	on	toggle

---

7. The Toy's Acknowledgment

The toy speaks as a curious listener, never an analyst. Naming the mood it senses would leak the hidden color channel, so it reacts to the story and the craft, never to the feeling.

Prompt lines:

• Ambient: "React in a few warm words as an engaged listener. Never name the emotion or mood you sense (it would spoil the hidden color). Brief or silent is fine."
• Shape moment: "Invite the choice warmly, e.g. 'Here are two shapes your beat could take — which one fits?'"

There are two channels, kept distinct. The non-verbal one is instant: the moment you submit, the sentence joins the story, a thinking indicator appears, and progress advances — covering the model's latency. The verbal one is sparse: a short warm line, guaranteed at the meaningful junctures (a shape appears, the reveal, handing the pen back).

---

8. Per-Segment Essence

The model can chunk a beat into 2–4 phrases, each with its own valence/arousal/dominance. The renderer places each segment as an anchor around the shape's boundary (clockwise from the top, weighted by phrase length) and interpolates between anchors with a cosine-bell weight — so the shape morphs continuously through each phrase's character with no visible seams.

"Calm at first, then suddenly sharp" → a shape soft on one side, spiky on the other.

Key decisions:

• Smooth interpolation, not hard sides. Hard sides produce visible seams; smooth interpolation reads as one coherent form.
• Clockwise from the top, length-weighted. The first phrase anchors at 12 o'clock; subsequent phrases sweep clockwise in proportion to word count.
• Shared rendering orientation. The render angle (θ = 0..2π) and the anchor-placement angle (θ_anchor = θ − π/2) are kept separate, so segmented and single-essence shapes stay orientation-consistent in the same composition.
• One beat = one shape. The budget is unchanged.

---

9. Reveal & Composition

Reveal sequence

1. Greyscale settle. All shapes appear in greyscale, in their final composition positions.
2. Color bloom. Colors animate in one by one in story order — a ~1s initial pause, then ~700ms per shape, each a fill 0.65s ease-in-out transition from greyscale to its real affect-driven color. This is the reveal's climax: it makes the hidden-color channel feel earned and gives the demo video a natural peak.
3. Companion shapes. The model adds shapes that balance the composition, marked "✦ AI added."
4. Narrative composition. The model assigns layout roles (region, scale, rotation, depth) reflecting narrative weight: dominant/intense beats go large and high, calm beats sink lower. A coherence dial (0–1) slides between a layered scene and a loose scatter.
5. Visibility enforcement. A floor of ≥20% visibility per shape (nudge step 0.03) — it allows juicy overlap, while hover-highlight and the layer panel do the legibility work.

Sound — sonifying the reveal (Web Audio, no asset files)

The color bloom is scored with synthesized sound so the reveal has an emotional arc you can hear. All synthesis is pure Web Audio (a lazy AudioContext, unlocked inside the user-gesture stack), togglable in Settings ("Reveal sound", persisted to localStorage).

• Reveal chimes. Each shape's bloom rings a soft bell. Pitch ← valence (sad→low, pleasant→high), quantized to an A-minor pentatonic ladder so any story's sequence is consonant in any order — the emotional arc becomes a little melody. Timbre ← arousal (a low-pass cutoff opens with energy). A sine fundamental with an octave shimmer.
• Shape-born tick. A neutral, fixed-pitch click when a shape forms during the story — deliberately not mapped to valence, so it never leaks the hidden color early.
• Atmospheric pad. A low root+fifth drone with a slow swell and filter sweep underlies the whole bloom sequence.
• Replay reveal. A "↻ Replay reveal" control re-runs the full color + chime + pad sequence over the current arrangement (your drags preserved): it smooth-scrolls the canvas into view, settles back to greyscale, then blooms again.

Composition

• A single layered picture, not a gallery — a single picture feels like an artifact; a gallery feels like outputs.
• Division of labor: the model proposes layout intent reflecting narrative role; a deterministic layout engine places the shapes and enforces the visibility floor.
• Coherence dial (model-emitted, 0–1): slides between a coherent layered scene and a playful scatter. A nonsensical story earns a nonsensical composition.
• Your arrangement: drag, rotate (↺↻ 15° per click), reorder (▲▼ depth), hide (👁), or restore to the initial state.

Authorship & Accessibility

A layer panel sits below the scene, each shape labeled by its beat. Your shapes and the model's are distinguished — "✦ AI added" for companions, "from AI's turn" for pass-the-pen shapes you chose. Hover an entry → its shape highlights with a glowing border.

Origin tooltip. Hovering (or pressing, on touch) a shape on the canvas — or its name in the layer panel — shows a custom tooltip: the shape's short essence label plus the original story beat it was born from (your verbatim 1–3 sentences). It's a single reused floating element, content set via textContent (so your prose can't inject HTML), viewport-clamped, with pointer-events:none so it never blocks a drag, and hidden on pointerup so it never lingers. Companions instead read "✦ AI added — …". It's screen-only — it never bakes into the saved image or share card. On touch, press-and-hold reveals it (a quick touch still drags).

---

10. Painting (FLUX.2 Klein)

After the reveal, an optional "Paint this" turns the composition into a full artwork:

• The current SVG canvas (including your drags; hidden shapes excluded) → base64 PNG → img2img through FLUX.2 Klein 4B.
• Integration-focused prompting. Strict-preservation prompts produce stiff, pasted-on results; a prompt that leans on FLUX's reference adherence produces paintings where the shapes feel part of the scene.
• Styles that actually steer the look. You can type any style into the theme field, and each named preset maps to a rich descriptor (e.g. Neon Glass → "luminous neon glass, glowing translucent forms, electric rim light") so it genuinely changes the result — replacing an earlier base prompt that hard-coded "art-print texture" and drowned every style out.
• 🎲 Surprise me. Picks a random style from the preset list (Art Print, Cut Paper, Ink & Wash, Neon Glass, Oil Impasto, Risograph, Dream Poster), writes it into the theme field so you can see the roll, and paints.
• Randomized seed per generation, so repeated paints (and repeat "Surprise me" rolls) differ.

Share Card

Both the shape composition and the painting can be exported as a share card — a portrait card with the model-named story title, the scene, the beat essences joined by semicolons ("the still pond; the shattering; the empty room"), and a "made with The Shape of Words" footer. It opens in a closeable popup modal (backdrop / button / Escape to dismiss) and downloads as a PNG.

• Title generation. A dedicated title_story call names the finished story in 2–5 words; the result is cached, so the canvas card and the painting card don't pay for two calls. The title fetch stays lazy (on first card open) to avoid a parallel GPU call colliding with the reveal.

---

11. Architecture

Browser (static/index.html) — custom HTML/JS, no build step
  renderer · mappings · layout engine  (deterministic, JS port of engine/)
  shape choice · reveal ritual · sound · drag · layers · rotate · hide · save
  painting UI (theme field, style presets, surprise-me) · share card · tooltips
  calls the backend for model judgment only
         │ /gradio_api/call/<name>  (queued SSE, auto-prefix detection)
app.py — gradio.Server  (satisfies the Off-Brand badge; gradio 6.x)
  @app.api  judge_beat · judge_beat_segmented · continue_story · reveal
            · title_story · paint   (each concurrency_limit=1)
         │
  model/backend.py   ← STORY_SHAPES_BACKEND
    "llamacpp"  →  GGUF in-process via llama-cpp-python  ← default; local & on-Space
                  GBNF grammar from JSON schema → guaranteed-valid JSON.
                  No transformers; ~5GB at Q4_K_M. Earns Llama Champion.
    "modal_llm" →  HTTP POST to a Modal GPU endpoint (modal_llm.py),
                  with transformers pinned to 4.x, where MiniCPM4.1 works.
         │
  model/painter.py
    "modal"      →  Modal web endpoint (A10G GPU, FLUX.2 Klein 4B)
    "flux_local" →  in-process diffusers + cpu-offload (fully local / Off-Grid)

Both Modal services (modal_llm.py, modal_painter.py) are deployed once with modal deploy; they keep one container warm (scaledown_window) and lazy-load weights on the first call.

Why llama.cpp, not in-process transformers? This was the hardest part of shipping. gr.Server (for the custom frontend) needs gradio 6.x, which forces huggingface-hub ≥ 1.2, which forces transformers ≥ 5.0. But MiniCPM4.1-8B loads via trust_remote_code, and its remote code is incompatible with transformers 5.x: loading needs a shim, and generation then still crashes deep in attention. The two clean escapes are to run it on Modal with transformers pinned to 4.x, or — better — to run it through llama.cpp as a quantized GGUF, which uses no transformers at all and is small enough (~5 GB) to sit beside FLUX on a single 24 GB GPU. The llama.cpp route also earns the Llama Champion badge.

The deployment journey. Those two backends weren't the first attempt — they're what survived after a path of dead ends, driven by the hardware I actually had (a laptop with 16 GB RAM / 8 GB VRAM, nowhere near enough for an 8B LLM and a 4B diffusion model at once):

1. Qwen3-8B on Ollama (local) + FLUX.2 Klein on Modal. The first working setup. The LLM ran locally through Ollama (which is itself llama.cpp under the hood); the painter — far too heavy for 8 GB — was pushed to a Modal GPU endpoint. This split kept the loop fast to iterate on while the expensive image step lived in the cloud.
2. MiniCPM4.1-8B via llama.cpp + FLUX.2 Klein on Modal. To target the OpenBMB prize I swapped the judge to MiniCPM4.1-8B, and moved to running it directly on llama.cpp (a quantized GGUF via llama-cpp-python) rather than Ollama — which also claims the Llama Champion badge. The painter is still on Modal due to previous RAM constraints.
3. Both on a single ZeroGPU Space — blocked. The goal was to host the whole thing on one free Hugging Face ZeroGPU Space (llama.cpp LLM + FLUX painter together). In practice this proved very difficult: ZeroGPU attributes GPU usage via an X-IP-Token header that the standard Gradio UI forwards automatically, but a custom gr.Server frontend doesn't, so every GPU call is scheduled anonymously and immediately hits a tiny per-IP quota — a constant stream of 429 Too Many Requests, while the account's real quota sits untouched. (On a campus network the anonymous quota is also shared across everyone behind the same NAT.) The solutions are a dedicated GPU Space — which isn't ZeroGPU and has no such attribution problem — or keeping the heavy work on Modal. Cracking ZeroGPU for a custom-frontend app remains open. Note: decoding the GPU schedule token showed the calls being attributed anonymously, which points to the 429s — though I didn't fully isolate the root cause so treat this as a strong lead rather than a solved case.

Engine (engine/):

• mappings.py — affect → geometry and affect → color (OKLCH)
• renderer.py — geometry → polygon boundary (triangle-wave spikiness, segmented smooth interpolation)
• scorer.py — zone-based scoring + band + directional hint (for the planned puzzle mode)

All engine math is mirrored in static/index.html (a JS port with verified-identical output), so the frontend renders independently — no server round-trip for rendering.

Model judgment:

• Full story context with per-beat status flags on every call, for globally-aware pacing.
• Pacing-note branches: "no shape yet → be generous" vs. "settled beats → skip."
• Judgment model: MiniCPM4.1-8B (the OpenBMB prize model), swappable via environment variables.

Model Selection

Both are small (8B + 4B), and together makes the whole toy runnable on a single 24 GB GPU.

Shape judgment — MiniCPM4.1-8B. The judge has 2 tasks: analytical read (mapping a beat to valence/arousal/dominance as structured JSON) and creative writing (pass-the-pen continuations and the story title). The 8B variant is perfect for the job:

• It's the OpenBMB prize model, so reaching for it is a deliberate eligibility choice — but it also does the job well.
• The MiniCPM4 family's only smaller variant is 0.5B — too weak for nuanced affect and coherent prose and switching to another family of models (e.g. Qwen, Gemma) would forfeit said prize.
• No need to downsize: at Q4_K_M the GGUF is ~5 GB, and MiniCPM4 is explicitly edge-optimized (fast decode), so it sits beside FLUX on one GPU.

Painting — FLUX.2 Klein 4B. The hackathon's recommended image model, and a clean fit for "Paint this":

• A 4-step turbo model (sub-second on a warm GPU) that natively supports img2img and multi-reference — exactly what's needed since the painter conditions on the rendered SVG composition.
• ~13 GB at bf16 (less with FP8/NVFP4 checkpoints), so it runs on Modal's A10G or in-process with CPU offload. Apache-2.0, 4B.
• Lighter alternatives exist (SDXL-Turbo runs on 8 GB) but at lower fidelity, and FLUX.1-schnell is its predecessor; Klein is the current speed / quality / license sweet spot.

---

12. Future Work

Puzzle Mode

A puzzle mode was planned to accompany exploration. For each of your story beats the model would determine its essence, then challenge you to write a new sentence whose shape resembles that essence's shape. Similarity would be scored by comparing the geometric-layer values — spikiness, compactness, and segmentability — averaged across the shape's segments, and the model would give directional feedback (e.g. "make it sharper" or "calmer") as a guide. The engine and scorer for this already exist; only the UI is unbuilt.

---

13. References

1. Huang, L. (2020). Space of preattentive shape features. Journal of Vision, 20(4):10. — A 3-D preattentive shape space (segmentability, compactness, spikiness) explains ~97% of variance. https://jov.arvojournals.org/article.aspx?articleid=2765047
2. Morgenstern, Y., Schmidt, F., Fleming, R. W. (2021). ShapeComp. PLOS Computational Biology, 17(5):e1008981. — No single descriptor captures shape similarity; MDS dimensions are not cardinal axes. https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008981
3. Mehrabian, A., & Russell, J. A. (1974). The PAD (Pleasure–Arousal–Dominance) emotional state model. — Three near-orthogonal affective dimensions. https://psycnet.apa.org/record/1996-03711-001
4. Bar, M., & Neta, M. (2006). Humans prefer curved visual objects. Psychological Science. — Angular contours → threat/negative valence. https://pubmed.ncbi.nlm.nih.gov/16913943/
5. Larson, C. L., et al. (2009). The shape of threat: simple geometric forms evoke rapid threat perception. — Angular → threat. https://pubmed.ncbi.nlm.nih.gov/17683209/
6. Blazhenkova, O., & Kumar, M. M. (2018). Angular versus curved shapes: emotion & energy (angular = energizing, round = calming). https://pubmed.ncbi.nlm.nih.gov/28927319/
7. Wilms, L., & Oberfeld, D. (2018). Color and emotion: effects of hue, saturation, and brightness. Psychological Research. — Hue arousal rises blue→red; saturated/bright = higher arousal. https://pubmed.ncbi.nlm.nih.gov/28612080/
8. Alvarado, J. (2025). Text-to-image color–emotion associations. — Higher brightness ↔ higher valence; dominance modulates chroma. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2025.1593928