docs/features/renderer-vocabulary.md

Renderer Vocabulary: Richer Geometry, Motion, and Part Counts

Review date: 2026-06-14. Reviewer: Claude (Opus 4.8), for OpenAI Codex to implement. Scope: one user goal — the vision prompt should be more concise and descriptive, and the pipeline should give the model more freedom over the built-in Three.js geometry/motion vocabulary and a larger range of parts.

This document is findings + next steps. It does not authorize any deployment or Hugging Face changes on its own; follow the normal GitHub → HF sync and verification flow in AGENTS.md. File/line references are to the repo state at review time.

User decisions captured for this spec (2026-06-14):

• Shapes: add the high-value set — cone, capsule, torus, and a spring (helix) builder — on top of the existing 5.
• Motions: add screw (helical: rotate + translate together), orbit (revolve around a pivot point), and pulse (scale breathing) on top of the existing 4.
• Parts: raise the usable cap to 6 (the renderer already allows 6).

---

TL;DR — the one architectural fact that drives everything

The prompt cannot grant freedom the renderer doesn't already have. Because of the hard security invariant — no model-authored HTML; render deterministically from validated JSON (AGENTS.md Runtime Notes, SECURITY.md Agent Guidance, docs/reviews/security-hardening.md §3) — the only things that can appear on screen are shapes/motions that the browser renderer explicitly builds in buildPartMesh (index.html:1141) and applyMotion (index.html:1195).

So today the vocabulary is capped identically in four places, and any expansion must change all four in lockstep (renderer is the source of truth):

Layer	Shapes	Motions	Parts cap
Prompt (snap2sim/prompts.py:26-27, :32)	box, cylinder, sphere, gear, rod	rotate, translate, oscillate, static	"2 to 4"
Schema enum (snap2sim/schema.py:136-139, :165-169, :214-215)	same 5	same 4	no max
Coercion (snap2sim/model_io.py:166, :169, :117)	same 5 (else→box)	same 4 (else→static)	parts[:4]
Renderer (index.html:842, :1145-1156, :1202-1210, :929)	same 5 (else→box)	same 4	slice(0,6)

Two consequences:

1. Parts is already inconsistent: renderer renders 6, but the prompt asks for ≤4 and coercion truncates to 4. Raising the usable count to 6 is mostly removing an artificial cap (see §3).
2. Adding shapes/motions is a 4-layer change, not a prompt edit. The order that avoids a broken intermediate state is: renderer → schema → coercion → prompt (build the capability, allow it, coerce toward it, then ask for it).

---

Finding 1 — The prompt is verbose and under-describes the vocabulary it already has

build_vision_prompt (snap2sim/prompts.py:13-56) is ~40 lines and embeds a full JSON skeleton, but:

• It lists shapes/motions as bare enums (prompts.py:26-27) with zero guidance on when to use each. The model gets box, cylinder, sphere, gear, rod and must guess mapping. In practice the example payload (EXAMPLE_ANALYSIS, schema.py:11-110) only ever uses box/cylinder/gear, so sphere and rod are effectively invisible to the model. The vocabulary is underused before we even expand it.
• It repeats constraints (e.g. "compact", "physically plausible", "short" appear multiple times across :15-18, :31, and the example) and spends many lines re-stating field rules already enforced by schema.py / coerced by model_io.py.
• VISION_SYSTEM_PROMPT (prompts.py:5-10) is terse and doesn't frame the model's strengths (it's a Reasoning GGUF per AGENTS.md) or the "annotated cutaway" product goal.

Net: the prompt is long on rules and short on descriptive guidance, which is the opposite of what helps a reasoning model pick good primitives.

A cheap win exists independent of any renderer change: add a terse "shape → use for" and "motion → use for" guide so the model exploits the existing 5/4 vocabulary. That guide then grows naturally when we add the new primitives.

---

Finding 2 — High-value Three.js primitives are unused; the cutaway aesthetic wants them

The renderer hand-builds geometry in buildPartMesh (index.html:1141-1167) and gearGeometry (index.html:1169-1187). Three.js ships many more primitives that map cleanly onto real mechanism elements. The user picked the high-value set:

New shape	Three.js primitive	Real mechanism elements it covers
cone	ConeGeometry(radius, height, seg)	valve cones, drill/screw tips, springs' seats, nozzles, pawl points
capsule	CapsuleGeometry(radius, length, …)	pistons, dowel pins, rollers, bearings, plungers, shafts with rounded ends
torus	TorusGeometry(radius, tube, …)	o-rings, retaining/snap rings, coils, washers (thick), seals
spring	custom helix via TubeGeometry + CatmullRomCurve3 (like gearGeometry is custom)	compression/extension springs, coils, helical elements — very common and currently impossible to depict

Note on a latent inconsistency to clean up while here: the current cylinder branch (index.html:1146) already secretly supports a cone/taper by using size[0]/2 as top radius and size[2]/2 as bottom radius — but size is documented everywhere else as [x, y, z] extents, and nothing tells the model this. Introducing an explicit cone shape (with its own size semantics) removes the need for that undocumented trick. Decide whether cylinder should revert to a true cylinder (radiusTop == radiusBottom) once cone exists; recommended yes, to keep size semantics consistent.

size semantics for the new shapes (must be documented in the prompt)

Keep the existing convention: size: [x, y, z] are bounding extents, and the builder derives radii/lengths from them so the model never has to send radius/height (which the prompt already forbids, prompts.py:28-30). Proposed:

• cone: base diameter from max(x, z), height from y. Apex +y.
• capsule: diameter from max(x, z), total length from y (long axis +y; reuse the rod pattern if a horizontal default reads better — match rod's rotateX convention at index.html:1151 for consistency).
• torus: outer diameter from max(x, z), tube thickness from y (clamp tube to a fraction of radius so it stays a ring, not a sphere).
• spring: outer diameter from max(x, z), length/height from y; expose coils (int, like teeth for gear, schema.py:158) and an optional wire thickness with sane defaults.

All four must also be reachable through the same color, rotation, and position handling that already wraps buildPartMesh (index.html:929-940, color at index.html:1158/colorFor :1228).

---

Finding 3 — Parts cap is artificially low and inconsistent

• Prompt says "Use 2 to 4 parts" (prompts.py:32).
• Coercion truncates with parts[:4] (model_io.py:117).
• Renderer already maps slice(0, 6) (index.html:929) and labels project per mesh (updateLabels, index.html:1213).
• Schema imposes no max on parts (schema.py:123-125).

Decision: usable cap = 6. This is mostly removing a cap:

• Prompt: change "Use 2 to 4 parts" → "Use 2 to 6 parts; prefer the fewest that explain the mechanism" (prompts.py:32).
• Coercion: parts[:4] → parts[:6] (model_io.py:117).
• Renderer: already 6 — no change, but double-check label legibility / overlap at 6 parts (updateLabels clamps at index.html:1221-1222).
• Schema: optionally add "maxItems": 6 to the parts array (schema.py:123) so over-long payloads fail validation predictably instead of being silently truncated only on the coercion path (note: model_io.py runs only on the Modal path, per docs/reviews/interaction-and-fallback-review.md; the Space validate_analysis path does not truncate, so a schema maxItems keeps both paths consistent).

---

Finding 4 — Motions are hand-coded, not "built-in"; expand the custom set

applyMotion (index.html:1195-1211) implements all motion by hand: rotate (continuous spin on axis), oscillate (sinusoidal rotation), translate (sinusoidal slide along axis within range), static. Three.js has no "motion" concept of its own, so "more built-in motions" = add more primitives to applyMotion. User picked:

New motion	Behavior	Mechanism examples	Params (extend schema)
screw	rotate and translate along the same axis together (helical)	screws, lead screws, drill bits, augers, twist mechanisms	reuse axis, speed, phase, range (translate extent) + e.g. pitch
orbit	revolve the part around a pivot point (not its own center)	planetary/idler gears, cranks, eccentrics, governor weights	axis, speed, phase + a pivot: [x,y,z]
pulse	sinusoidal scale breathing	diaphragms, bladders, pumps, bellows, valves opening/closing	speed, phase, amplitude

Implementation notes for applyMotion:

• It already resets to base each frame (mesh.position.copy(basePosition), mesh.rotation.copy(baseRotation), index.html:1200-1201) — good; screw and orbit can compose translate+rotate cleanly on top of that.
• orbit needs the pivot in world space; store it on mesh.userData at build time like basePosition (index.html:936). Offset = position − pivot, rotate the offset by axis, re-add pivot.
• pulse should multiply the reveal scale (revealMesh, index.html:1189) rather than fight it — guard so the staggered reveal still plays first.
• Keep the existing param defaults pattern (speed||1, phase||0, amplitude||0.25, range||[-0.25,0.25], index.html:1197-1207).

Schema additions: extend the motion enum (schema.py:165-169, _MOTIONS :215) and add optional pitch (number) and pivot (3-number list) to the motion properties (schema.py:162-186), validated with the existing _require_number_list helpers (schema.py:326-329). Coercion: extend the allowed set (model_io.py:169) and carry the new params through (mirror _axis_vector, model_io.py:173-175, :290).

---

Finding 5 — The renderability gate and validators must learn the new vocabulary too

Three guards currently encode the old vocabulary and will silently reject or downgrade the new shapes if missed:

1. Browser renderability check (index.html:840-844): the ["box","cylinder","sphere","gear","rod"].includes(geometry.shape) test decides whether a part counts as renderable (feeds render_mode / annotate-fallback selection, schema.py:select_render_mode). Add the new shapes here or a part with shape:"spring" will be treated as non-renderable and fall back to annotate-only.
2. Schema validator (schema.py:_SHAPES :214, _MOTIONS :215, enums :136-139/:165-169) — extend both sets/enums.
3. Coercion fallbacks (model_io.py:166-167, :169-170) — extend the allowed sets so a valid new shape isn't rewritten to box/static.

Also update EXAMPLE_ANALYSIS (schema.py:11-110) to exercise at least a couple of the new primitives (e.g. a spring and a capsule) so the in-prompt example actually demonstrates the wider vocabulary — the example is the single biggest lever on what the model emits.

---

Proposed prompt rewrite (concise + descriptive)

Goal: shorter scaffolding, richer guidance. Two concrete changes.

A) VISION_SYSTEM_PROMPT (prompts.py:5-10) — frame the task and the model

Keep it ~4 lines but add: this is a reasoning model building an annotated technical cutaway; reason briefly, then emit one JSON object; prefer the simplest set of primitives that truthfully explains the mechanism; lower confidence + annotate when unsure (already the policy — keep it).

B) build_vision_prompt (prompts.py:13-56) — trim rules, add a vocabulary guide

• Cut redundancy: state "final answer = one JSON object, no markdown" once; drop repeated "compact/short/plausible" restatements.

• Replace bare enums with a terse guide the model can act on, e.g.:

  Shapes (pick the closest; size = [x,y,z] extents):
    box       plates, housings, blocks, levers, selectors
    cylinder  shafts, sleeves, bushings, drums, pins
    cone      valve cones, tips, nozzles, tapers
    capsule   pistons, rollers, dowel pins, plungers, bearings
    sphere    balls, detents, ball bearings, nodes
    rod       links, tie rods, thin axles, connecting rods
    gear      toothed wheels (set teeth); ratchets, cogs
    torus     o-rings, snap/retaining rings, seals, coils (single)
    spring    helical springs, coils (set coils)
  
  Motions (axis is a numeric vector like [0,1,0]):
    static    fixed structure / housing
    rotate    continuous spin (speed)
    oscillate sinusoidal twist (amplitude, speed)
    translate slide along axis (range [min,max])
    screw     spin + advance along axis together (pitch) — screws, drills
    orbit     revolve around a pivot point (pivot [x,y,z]) — planetary/cranks
    pulse     scale breathing (amplitude) — diaphragms, pumps
  

• Update the part-count line to "Use 2 to 6 parts; prefer the fewest that explain the mechanism."

• Keep the hard rules that protect parsing/rendering: numeric size/ position/axis vectors only, no radius/height/string-axis (prompts.py:28-30), normalized [0,1] top-left annotation coords (prompts.py:31-34), and "reason first, final answer is one JSON object" (prompts.py:14-17, required by the <think>-stripping parser, docs/reviews/interaction-and-fallback-review.md Q3 / model_io.py).

• Keep the example but shrink it to a 2–3 part skeleton that now includes a new primitive, and rely on EXAMPLE_ANALYSIS (the schema sample) to show the full shape.

---

Security / invariants to preserve (do not regress)

• Deterministic rendering only. All new shapes/motions are built in buildPartMesh/applyMotion from validated JSON. Never inject model-authored HTML/JS/markup (AGENTS.md, SECURITY.md, docs/reviews/security-hardening.md §3). label/note stay textContent (index.html:1223).
• Validate before render. New enum values must be added to schema.py so validate_analysis (schema.py:263) still runs before scene generation; do not loosen validation to "accept anything."
• Coercion stays conservative. Unknown shapes/motions must still fall back to box/static (model_io.py:166-170), never crash.
• No change required to rate limiting, upload caps, decompression-bomb guards (app.py), Modal bearer auth, or the GitHub→HF sync.

---

Suggested implementation order for Codex

1. Renderer first (index.html): add cone, capsule, torus, and a spring (helix) builder to buildPartMesh/a new helper next to gearGeometry; add screw, orbit, pulse to applyMotion (store pivot on userData). Update the renderability gate at index.html:840-844. Verify each renders before touching Python.
2. Schema (snap2sim/schema.py): extend shape/motion enums + _SHAPES/ _MOTIONS; add pitch/pivot/coils properties; optional parts.maxItems: 6; update EXAMPLE_ANALYSIS to use a couple of new primitives.
3. Coercion (snap2sim/model_io.py): extend allowed shape/motion sets; carry new params; change parts[:4] → parts[:6].
4. Prompt (snap2sim/prompts.py): apply the concise rewrite (system + user), the shape/motion guide, and "2 to 6 parts".
5. Local verification (per AGENTS.md): schema/parser checks, FastAPI TestClient for /, /analyze_image, /generate_scene; and a real browser check that each new shape and motion renders and that 5–6-part scenes don't have unreadable overlapping labels.
6. Optional Modal re-measure (docs/reviews/interaction-and-fallback-review.md Q3 tuning note): the richer prompt may change token usage; confirm run_analysis_endpoint_check still returns valid JSON within the current 4096/8192/300s budgets before deploy.
7. Standard PR -> GitHub Actions HF sync -> Space verification. The final public submission is under build-small-hackathon/Snap2Sim.

Archived considerations

Final submission note, 2026-06-15: the renderer vocabulary pass shipped before the public build-small-hackathon/Snap2Sim submission.

• Whether cylinder should revert to a true (untapered) cylinder once explicit cone exists (recommended yes — see §2 note).
• Whether spring/torus need extra params (coils, wire/tube thickness) exposed to the model or just sensible fixed defaults (recommend defaults first, expose only if visual quality needs it).