jasondo OpenAI Codex commited on
Commit
53f7d61
·
1 Parent(s): 2499b0f

Migrate Snap2Sim to trusted HTML scene shell

Browse files

Co-authored-by: OpenAI Codex <codex@openai.com>

.github/workflows/sync_to_hf.yml CHANGED
@@ -57,4 +57,14 @@ jobs:
57
  run: git lfs push huggingface --all
58
 
59
  - name: Force-sync main to Hugging Face
60
- run: git push --force huggingface HEAD:main
 
 
 
 
 
 
 
 
 
 
 
57
  run: git lfs push huggingface --all
58
 
59
  - name: Force-sync main to Hugging Face
60
+ run: |
61
+ for attempt in 1 2 3 4 5; do
62
+ if git push --force huggingface HEAD:main; then
63
+ exit 0
64
+ fi
65
+ delay=$((attempt * 20))
66
+ echo "Hugging Face push failed on attempt ${attempt}; retrying in ${delay}s..." >&2
67
+ sleep "$delay"
68
+ done
69
+ echo "Hugging Face push failed after 5 attempts." >&2
70
+ exit 1
AGENTS.md CHANGED
@@ -2,10 +2,10 @@
2
 
3
  ## Project Overview
4
 
5
- Snap2Sim / Inside the Machine is a Gradio-based Hugging Face Space scaffold for
6
- the Build Small Hackathon Backyard AI track. The app is intended to accept a
7
- photo of a hardware component, infer its internal mechanism, and render an
8
- annotated technical cutaway animation.
9
 
10
  ## Runtime Notes
11
 
@@ -18,8 +18,17 @@ annotated technical cutaway animation.
18
  - Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
19
  through Transformers/custom code for the vision step if GGUF endpoint quality
20
  is not reliable enough for the demo.
21
- - Current code uses placeholder inference so the UI, schema, and endpoint
22
- contract can run without downloading large model weights.
 
 
 
 
 
 
 
 
 
23
 
24
  ## Project Structure
25
 
@@ -33,25 +42,25 @@ annotated technical cutaway animation.
33
  GitHub `main` to an existing Hugging Face Space.
34
  - `requirements.txt` - Python dependencies for the Gradio Space and Modal
35
  scaffold.
36
- - `app.py` - Hugging Face Space entry point; launches the Gradio app.
 
 
 
 
37
  - `modal_app.py` - Modal app scaffold with runtime asset caching, a
38
  llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
39
- placeholder plus experimental llama.cpp `analyze_image` / `generate_threejs`
40
  web endpoints.
41
  - `scripts/verify_runtime_assets.py` - Hugging Face metadata preflight for the
42
  selected GGUF quant and `mmproj` file.
43
  - `snap2sim/__init__.py` - package marker.
44
  - `snap2sim/backend.py` - backend config, local placeholder inference, Modal
45
  HTTP client, and image base64 encoding.
46
- - `snap2sim/fallback_scene.py` - themed animated SVG/CSS fallback shown when
47
- generated Three.js is unavailable.
48
- - `snap2sim/prompts.py` - prompt templates for the vision analysis and Three.js
49
- code-generation steps.
50
  - `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
51
- - `snap2sim/three_scene.py` - deterministic Three.js scene generation from a
52
- validated mechanism payload.
53
- - `snap2sim/ui.py` - Gradio Blocks UI, blueprint field-manual CSS theme, and
54
- pipeline orchestration.
55
 
56
  ## What Has Been Done
57
 
@@ -88,8 +97,8 @@ annotated technical cutaway animation.
88
  HTML documents.
89
  - Added a local coercion fallback for verbose/partial model JSON so the
90
  analysis endpoint does not launch a second slow repair generation.
91
- - Added experimental `analyze_image_llamacpp` and `generate_threejs_llamacpp`
92
- Modal GPU endpoints. They are not wired into the Gradio app by default; point
93
  `MODAL_ANALYZE_URL` / `MODAL_GENERATE_URL` at them only after the smoke test
94
  passes.
95
  - Fixed Modal packaging for local `snap2sim` imports by adding
@@ -102,7 +111,8 @@ annotated technical cutaway animation.
102
  - Deployed `modal_app.py` to Modal at
103
  `https://modal.com/apps/bigstonks1/main/deployed/snap2sim-inside-the-machine`.
104
  - Production endpoint checks passed for `runtime_probe`,
105
- `analyze_image_llamacpp`, and deterministic `generate_threejs`.
 
106
  - Modal web endpoints are secured with `Authorization: Bearer
107
  SNAP2SIM_API_TOKEN`; the token lives in Modal secret `snap2sim-api-auth` and
108
  Hugging Face Space secret `SNAP2SIM_API_TOKEN`.
@@ -126,6 +136,17 @@ annotated technical cutaway animation.
126
  Space sync on pushes to `main`. The workflow targets `jasondo111/Snap2Sim`
127
  and requires GitHub secret `HF_TOKEN`.
128
  - User reported the GitHub Actions `HF_TOKEN` secret has been added.
 
 
 
 
 
 
 
 
 
 
 
129
  - Documented GitHub as the source of truth:
130
  `https://github.com/Bigstonks1/Snap2Sim`. Do not edit files directly on the
131
  Hugging Face Space; they will be overwritten by the sync workflow.
@@ -133,15 +154,35 @@ annotated technical cutaway animation.
133
  assistance from OpenAI Codex.
134
  - Updated `README.md` with local run instructions, runtime preflight, Modal
135
  deployment path, and the current runtime decision.
 
 
 
 
 
 
 
 
 
 
 
 
 
136
 
137
  ## Next Work
138
 
139
- - Point the Gradio app at `analyze_image_llamacpp` for analysis and the
140
- deterministic `generate_threejs` endpoint for scene rendering.
 
 
 
 
 
 
 
 
141
  - Keep the Hugging Face Space private until the user explicitly approves making
142
  it public for submission.
143
- - Commit and push the workflow to GitHub `main`, then verify the GitHub Actions
144
- sync run succeeds.
145
- - If Three.js model generation is too brittle, keep using the deterministic
146
- local scene generator for the demo while still using Nemotron for analysis.
147
- - Document final submission links and bonus claims.
 
2
 
3
  ## Project Overview
4
 
5
+ Snap2Sim / Inside the Machine is a Build Small Hackathon Backyard AI project
6
+ for curious tinkerers and makers. The app is intended to accept a photo of a
7
+ hardware component, infer its internal mechanism, and render an annotated
8
+ technical cutaway animation.
9
 
10
  ## Runtime Notes
11
 
 
18
  - Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
19
  through Transformers/custom code for the vision step if GGUF endpoint quality
20
  is not reliable enough for the demo.
21
+ - Local code now serves a trusted `index.html` through `gradio.Server` and
22
+ exposes `/analyze_image` plus `/generate_scene`. The Space still needs a
23
+ GitHub-to-HF sync and private deployment verification for this pass.
24
+ - Model-generated scene output is A-Frame declarative HTML
25
+ (`<a-scene>...</a-scene>`), with human-written deterministic Three.js as the
26
+ fallback in `index.html`.
27
+ - Endpoint naming is `generate_scene` everywhere in active code.
28
+ - Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
29
+ Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
30
+ claim Off the Grid unless the final deployment moves inference to ZeroGPU or
31
+ another non-cloud-API path that satisfies that bonus.
32
 
33
  ## Project Structure
34
 
 
42
  GitHub `main` to an existing Hugging Face Space.
43
  - `requirements.txt` - Python dependencies for the Gradio Space and Modal
44
  scaffold.
45
+ - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
46
+ `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
47
+ - `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
48
+ includes A-Frame, Three.js, Gradio JS client, upload UI, pipeline
49
+ orchestration, and deterministic Three.js fallback.
50
  - `modal_app.py` - Modal app scaffold with runtime asset caching, a
51
  llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
52
+ placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
53
  web endpoints.
54
  - `scripts/verify_runtime_assets.py` - Hugging Face metadata preflight for the
55
  selected GGUF quant and `mmproj` file.
56
  - `snap2sim/__init__.py` - package marker.
57
  - `snap2sim/backend.py` - backend config, local placeholder inference, Modal
58
  HTTP client, and image base64 encoding.
59
+ - `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
60
+ local and placeholder Modal mode.
61
+ - `snap2sim/prompts.py` - prompt templates for the vision analysis and A-Frame
62
+ scene-generation steps.
63
  - `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
 
 
 
 
64
 
65
  ## What Has Been Done
66
 
 
97
  HTML documents.
98
  - Added a local coercion fallback for verbose/partial model JSON so the
99
  analysis endpoint does not launch a second slow repair generation.
100
+ - Added experimental `analyze_image_llamacpp` and scene-generation Modal GPU
101
+ endpoints. They are not wired into the Gradio app by default; point
102
  `MODAL_ANALYZE_URL` / `MODAL_GENERATE_URL` at them only after the smoke test
103
  passes.
104
  - Fixed Modal packaging for local `snap2sim` imports by adding
 
111
  - Deployed `modal_app.py` to Modal at
112
  `https://modal.com/apps/bigstonks1/main/deployed/snap2sim-inside-the-machine`.
113
  - Production endpoint checks passed for `runtime_probe`,
114
+ `analyze_image_llamacpp`, and deterministic scene generation before the
115
+ current endpoint rename.
116
  - Modal web endpoints are secured with `Authorization: Bearer
117
  SNAP2SIM_API_TOKEN`; the token lives in Modal secret `snap2sim-api-auth` and
118
  Hugging Face Space secret `SNAP2SIM_API_TOKEN`.
 
136
  Space sync on pushes to `main`. The workflow targets `jasondo111/Snap2Sim`
137
  and requires GitHub secret `HF_TOKEN`.
138
  - User reported the GitHub Actions `HF_TOKEN` secret has been added.
139
+ - PR #1 was merged to `main`. The first GitHub-to-HF sync run reached the final
140
+ push step but failed with Hugging Face HTTP `429`; the workflow was updated
141
+ to retry that final push with bounded backoff.
142
+ - The next GitHub Actions sync run succeeded:
143
+ `https://github.com/Bigstonks1/Snap2Sim/actions/runs/27487136019`.
144
+ - Hugging Face Space `jasondo111/Snap2Sim` later reported SHA
145
+ `bfe49606478d6e922b0f973409fa3433f9d9ed9d`, matching GitHub `main`, after
146
+ the final handoff-note sync.
147
+ - `PROMPT.md` was updated with a new critical path: `gradio.Server`,
148
+ `index.html`, A-Frame model scene generation, `generate_scene` naming, and a
149
+ deterministic Three.js fallback in browser JS.
150
  - Documented GitHub as the source of truth:
151
  `https://github.com/Bigstonks1/Snap2Sim`. Do not edit files directly on the
152
  Hugging Face Space; they will be overwritten by the sync workflow.
 
154
  assistance from OpenAI Codex.
155
  - Updated `README.md` with local run instructions, runtime preflight, Modal
156
  deployment path, and the current runtime decision.
157
+ - Rewrote `app.py` as a `gradio.Server` app with `/`, `/analyze_image`, and
158
+ `/generate_scene` routes.
159
+ - Added `index.html` with upload orchestration, analysis readout, A-Frame scene
160
+ injection, 3-second blank-scene fallback, play/pause, and deterministic
161
+ browser-side Three.js fallback.
162
+ - Renamed active backend and Modal scene generation paths to `generate_scene`.
163
+ - Updated scene-generation prompts and parsing for `<a-scene>...</a-scene>`
164
+ output.
165
+ - Deleted the legacy Gradio Blocks UI, Python iframe Three.js generator, and
166
+ SVG fallback modules.
167
+ - Local verification passed with FastAPI `TestClient`: `/` returned the HTML
168
+ shell, `/analyze_image` returned the placeholder analysis, and
169
+ `/generate_scene` returned an `<a-scene>` block.
170
 
171
  ## Next Work
172
 
173
+ - Deploy the current branch through GitHub-to-HF sync, keep the Space private,
174
+ and confirm the private Space loads the trusted `index.html` shell.
175
+ - Update Space/Modal endpoint variables if needed so `MODAL_GENERATE_URL`
176
+ points at the deployed `generate_scene` endpoint, then verify
177
+ `/analyze_image` and `/generate_scene` with the secured bearer-token flow.
178
+ - Run an end-to-end private Space check with a real image: upload ->
179
+ `/analyze_image` -> panel population -> `/generate_scene` -> A-Frame render
180
+ or deterministic Three.js fallback.
181
+ - Optional polish only after private deployment verification: component
182
+ watermark, noise texture, vignette, then scan-line reveal refinements.
183
  - Keep the Hugging Face Space private until the user explicitly approves making
184
  it public for submission.
185
+ - Keep the README opening hook intact: "You find a small metal cylinder at a
186
+ flea market. What is it? How does it work inside?" After implementation,
187
+ expand README with the updated model stack, rendering-stack rationale, and
188
+ bonus quest claims from `PROMPT.md`.
 
PROMPT.md CHANGED
@@ -1,185 +1,343 @@
1
- PROJECT: "Inside the Machine" — AI-powered teardown visualizer for the
2
- Build Small Hackathon (Backyard AI track, huggingface.co/build-small-hackathon)
3
-
4
- GOAL
5
- Build a Gradio app, deployed as a Hugging Face Space, that takes a photo of
6
- a hardware component (e.g. a gear, valve, hinge, pump, lock, engine part)
7
- and produces an animated 3D visualization showing how that component works
8
- internally at a finer-grained scale than the photo shows i.e. "open it up
9
- and show me the moving parts and the physics/mechanism."
10
-
11
- HARD CONSTRAINTS
12
- - Total model parameters across the entire pipeline must be ≤ 32B.
13
- - Must be a Gradio app, hosted as a Hugging Face Space.
14
- - Prefer local/open-weight models (target the "Off the Grid" bonus: no
15
- cloud APIs at inference time, where deployment allows).
16
-
17
- MODEL STACK
18
- Primary model: NVIDIA Nemotron 3 Nano Omni (30B-A3B, MoE, ~3B active
19
- params), used for both the vision-understanding step AND the Three.js
20
- code generation step (two prompts/turns against the same model). This
21
- targets the NVIDIA Nemotron Quest sponsor award and stays comfortably
22
- under the 32B cap as a single model.
23
-
24
- Fallback split pipeline (use only if the primary path's code-gen quality
25
- is too weak, or if the multimodal runtime issue below can't be resolved):
26
- - NVIDIA Nemotron Nano V2 VL (12B) for image understanding/analysis.
27
- - Qwen2.5-Coder-14B for Three.js/animation code generation.
28
- - Total ~26B, still leaves headroom and still qualifies for the Nemotron
29
- Quest award via the vision step.
30
-
31
- MODEL RUNTIME
32
- - Primary path (targets "Llama Champion" bonus): use
33
- unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF via llama.cpp.
34
- Start with a mid-size quant (e.g. UD-Q4_K_M, ~24 GB) for quality, and
35
- have a smaller quant (e.g. UD-IQ2_M, ~18.5 GB) ready as a fallback for
36
- constrained hardware.
37
- - IMPORTANT verify multimodal support first: as a setup/diagnostic step,
38
- confirm whether this GGUF repo includes a vision-encoder ("mmproj") file
39
- for the CRADIO v4-H encoder, and whether llama.cpp's current build
40
- actually accepts image input for this architecture (nemotron_h_moe).
41
- Multimodal GGUF conversions often ship the LLM only, with vision support
42
- lagging behind.
43
- - If image input works end-to-end via llama.cpp: use this single model
44
- for both the vision-analysis step and the Three.js code-gen step —
45
- this is the ideal "Llama Champion" + Nemotron Quest combo.
46
- - If image input is NOT yet supported in the GGUF/llama.cpp path: use
47
- the NVFP4 safetensors checkpoint (nvidia/Nemotron-3-Nano-Omni-30B-A3B-
48
- Reasoning-NVFP4, ~20.9 GB) via vLLM or transformers for the vision-
49
- analysis step only, and use the GGUF + llama.cpp build for the
50
- Three.js code-gen step (text-only, so GGUF support is solid). Note
51
- in the README that "Llama Champion" applies to the code-gen stage.
52
- - Either way, total parameters stay at ~31B (single model), comfortably
53
- under the 32B cap, and the project remains eligible for the NVIDIA
54
- Nemotron Quest award.
55
-
56
- DEPLOYMENT ARCHITECTURE two-tier with Modal
57
- - Tier 1 (frontend): the Gradio app, hosted as a HF Space on the standard
58
- CPU tier. Handles UI, image upload, calling the inference backend, and
59
- rendering the returned JSON + Three.js HTML.
60
- - Tier 2 (inference backend): a Modal app exposing two functions as web
61
- endpoints, running on a Modal GPU container:
62
- - analyze_image(image) -> structured JSON (vision step, per the
63
- pipeline below)
64
- - generate_threejs(json) -> Three.js scene HTML/JS (code-gen step)
65
- Both load Nemotron 3 Nano Omni once at container start (use a Modal
66
- Volume to cache model weights across cold starts so they aren't
67
- re-downloaded every time). Use whichever runtime (llama.cpp/GGUF or
68
- vLLM/NVFP4) was determined in the MODEL RUNTIME verification step.
69
- - The Gradio app calls these endpoints over HTTP (e.g. via `requests`),
70
- passing the image as base64 and receiving JSON/HTML back.
71
- - Make the inference backend swappable via a config flag (e.g.
72
- INFERENCE_BACKEND=modal | zerogpu | local), so the same Gradio code can
73
- run against:
74
- - a Modal endpoint (primary path generous credits, no VRAM
75
- constraints, targets the Modal Award)
76
- - HF Spaces ZeroGPU (alternative — model loads inside the Space itself
77
- on an A10G; try the smallest GGUF quant here)
78
- - a local GPU (for development/testing)
79
- - Surface Modal cold-start latency in the UI — e.g. a "WAKING THE
80
- WORKSHOP..." loading message in the established visual theme, since
81
- cold starts on a 30B model can take tens of seconds.
82
- - README must state which backend the submitted Space actually uses, and
83
- adjust bonus-quest claims accordingly: if Modal is the deployed backend,
84
- "Off the Grid" is not claimed (inference happens on Modal's cloud GPUs),
85
- but "Llama Champion", "Nemotron Quest", and the Modal Award still apply.
86
- If ZeroGPU ends up sufficient, "Off the Grid" can additionally be claimed.
87
-
88
- PIPELINE / APPLICATION FLOW
89
- 1. User uploads a photo of a hardware component via gr.Image.
90
- 2. Vision step (Nemotron VL): identify the component, enumerate its
91
- internal parts, and describe its operating mechanism in structured
92
- terms — e.g. part names, how each part moves (rotate/translate/
93
- oscillate), axes/pivots, sequence of motion, and what triggers it
94
- (e.g. "input shaft rotates cam lobe pushes follower valve opens").
95
- Output this as structured JSON (part list with geometry hints: shape,
96
- approximate size/position, motion type, motion parameters).
97
- 3. Code-gen step: feed that JSON into the second prompt/model with
98
- instructions to generate a single self-contained Three.js scene (HTML
99
- + inline JS/CSS) that:
100
- - builds simple primitive-based 3D representations of each part
101
- (boxes, cylinders, gears via extruded shapes, etc.)
102
- - animates them according to the described motion (rotation speed,
103
- translation range, timing/sequence)
104
- - includes basic camera controls (OrbitControls) and a play/pause toggle
105
- - includes on-screen labels for each part
106
- 4. Gradio renders the generated HTML via gr.HTML (sandboxed iframe).
107
- 5. Below the visualization, display the structured explanation text from
108
- step 2 (plain-language "how it works" writeup) so the app is useful
109
- even if the 3D fails to render.
110
-
111
- VISUAL DESIGN DIRECTION — "Technical Cutaway / Field Manual" (Off-Brand bonus)
112
- Commit fully to this aesthetic; do not fall back to default Gradio styling.
113
-
114
- - Concept: the app should feel like a page from a vintage engineering
115
- service manual or a reverse-engineering field notebook — the kind of
116
- diagram you'd find annotating a cutaway drawing of an engine.
117
- - Theme: dark "blueprint" canvas as the dominant surface — deep navy/
118
- charcoal background (not pure black), with fine 1px grid lines at low
119
- opacity to evoke graph/blueprint paper.
120
- - Color system (define as CSS variables, used consistently):
121
- - Background: deep charcoal-navy (e.g. #14181F)
122
- - Primary accent: warm amber/safety-orange (e.g. #E8A33D) for
123
- annotations, active states, and the "how it works" highlights
124
- - Secondary accent: cool cyan (e.g. #5FD4D0) for part labels and
125
- motion-path indicators in the 3D scene
126
- - Text: off-white / warm gray, never pure white
127
- - Typography: pair a condensed, slightly industrial display/grotesk font
128
- (e.g. for headings/labels something like a stencil-adjacent or
129
- technical condensed sans) with a monospace font (e.g. for the part
130
- list, JSON-derived data, and annotations) avoid Inter, Roboto, Arial,
131
- and system-default fonts entirely.
132
- - Layout: two-pane, asymmetric split (3D viewport larger/left, analysis
133
- panel narrower/right or as an overlay drawer). Part labels in the 3D
134
- scene should look like annotation callouts (thin leader lines + small
135
- monospace tags), echoing real exploded-diagram conventions.
136
- - Motion: one signature moment — when the model finishes analysis and
137
- the 3D scene loads, animate a "power-on" reveal (e.g. parts fade/slide
138
- into place sequentially, or a scanning-line sweep across the blueprint
139
- grid before the model appears). Avoid scattered micro-animations
140
- elsewhere; spend the motion budget on this one moment.
141
- - Texture/atmosphere: subtle grid-paper background, faint scanline or
142
- vignette effect on the canvas edges — restrained, not noisy.
143
- - Loading states should reinforce the theme (e.g. "ANALYZING ASSEMBLY...",
144
- "RENDERING CUTAWAY...", "WAKING THE WORKSHOP...") in the monospace font
145
- rather than generic spinners.
146
-
147
- UI / UX REQUIREMENTS
148
- - Clean two-pane layout per the design direction above: 3D visualization
149
- + part-by-part "how it works" panel.
150
- - Loading states for both inference steps (vision analysis can be slow,
151
- Modal cold starts add latency), styled per the theme above.
152
- - Graceful fallback: if Three.js code generation fails or errors, show
153
- the textual explanation plus a simple 2D animated SVG/CSS fallback
154
- instead of a blank pane — keep this fallback in the same visual theme.
155
- - Investigate gr.Server if needed to achieve full styling control beyond
156
- what gr.HTML / custom CSS in Blocks allows.
157
-
158
- DELIVERABLES
159
- - Working Gradio app code (app.py + requirements.txt) ready to push to a
160
- Hugging Face Space under the build-small-hackathon org.
161
- - Modal app code (modal_app.py or similar) implementing the two inference
162
- endpoints described above.
163
- - README explaining: the problem this solves and for whom (Backyard AI
164
- track requires a real person/use case — e.g. a hobbyist mechanic, a
165
- shop teacher, a repair-cafe volunteer who wants to explain how parts
166
- work to customers/students), the model stack and parameter count
167
- (must total ≤32B, state the exact breakdown), the deployment
168
- architecture, and which bonus quests/sponsor awards are targeted
169
- (Llama Champion, Off-Brand, NVIDIA Nemotron Quest, Modal Award, Field
170
- Notes, and "Off the Grid" if applicable to the final deployment).
171
- - Keep code modular: separate files/functions for (1) vision analysis,
172
- (2) code generation, (3) Gradio UI/styling, (4) inference backend
173
- client/config, so each piece can be swapped or debugged independently.
174
-
175
- START by:
176
- 1. Verifying the llama.cpp multimodal support question above for the
177
- Nemotron 3 Nano Omni GGUF — this determines the runtime path for the
178
- rest of the build.
179
- 2. Scaffolding the Modal app with the two inference endpoints (using
180
- placeholder/echo logic initially), and the Gradio app structure with
181
- the backend-config flag and CSS theme/variables for the visual design
182
- direction above.
183
- 3. Designing the JSON schema for step 2 of the pipeline this is the
184
- critical interface between the vision and code-gen steps.
185
- 4. Iterating on prompt templates for steps 2 and 3.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Snap2Sim — "Inside the Machine"
2
+ **Build Small Hackathon** · Backyard AI Track · [huggingface.co/build-small-hackathon](https://huggingface.co/build-small-hackathon)
3
+
4
+ ---
5
+
6
+ ## Goal
7
+
8
+ Build a Gradio app deployed as a Hugging Face Space that takes a photo of a hardware component (gear, valve, pump, lock, engine part, etc.) and produces an animated 3D visualization showing how that component works internally — "open it up and show me the moving parts and the mechanism."
9
+
10
+ ---
11
+
12
+ ## Hard Constraints
13
+
14
+ - Total model parameters across the entire pipeline **≤ 32B**
15
+ - Must be a **Gradio app** hosted as a **Hugging Face Space**
16
+ - No cloud AI APIs at inference time where possible (targets "Off the Grid" bonus)
17
+ - **Plain HTML/CSS/JS frontend** — no React, no build step, no bundler
18
+
19
+ ---
20
+
21
+ ## Target User The Curious Tinkerer / Maker
22
+
23
+ Someone who pulls apart old electronics, finds a mystery component at a thrift store or salvage yard, or cracks open a broken appliance wondering: *"what does this actually do and how does it work inside?"* Not an engineer — a curious, hands-on person who learns by taking things apart.
24
+
25
+ > **README must open with:** *"You find a small metal cylinder at a flea market. What is it? How does it work inside?"* — before any technical description.
26
+
27
+ ---
28
+
29
+ ## Current State (June 13, 2026)
30
+
31
+ Scaffold exists at [github.com/Bigstonks1/Snap2Sim](https://github.com/Bigstonks1/Snap2Sim), synced to HF Space `jasondo111/Snap2Sim` (private, keep private until user approves).
32
+
33
+ **Confirmed working:**
34
+ - Modal deployment: `snap2sim-inside-the-machine` (bigstonks1 workspace)
35
+ - `smoke_test_llamacpp_image` `"ok": true` for `UD-Q4_K_M` + `mmproj-F16.gguf`
36
+ - `analyze_image_llamacpp` and scene-generation Modal endpoints deployed
37
+ - GitHub HF Space sync workflow live and passing
38
+ - Local app code now uses `gradio.Server` plus a trusted `index.html`
39
+
40
+ **Primary remaining tasks:**
41
+ 1. Deploy the current branch through GitHub HF sync
42
+ 2. Confirm the private Space loads the trusted `index.html`
43
+ 3. Confirm `/analyze_image` and `/generate_scene` respond through the secured
44
+ Modal bearer-token flow
45
+
46
+ ---
47
+
48
+ ## Model Stack
49
+
50
+ **Primary model:** NVIDIA Nemotron 3 Nano Omni (30B-A3B, MoE, ~3B active params)
51
+ Used for both vision analysis and A-Frame scene generation (two prompt turns, same model). Targets the **NVIDIA Nemotron Quest** sponsor award. ~31B total — under the 32B cap.
52
+
53
+ ### GGUF Path (confirmed working)
54
+ | Setting | Value |
55
+ |---|---|
56
+ | Repo | `unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF` |
57
+ | Primary quant | `UD-Q4_K_M` (~24 GB) + `mmproj-F16.gguf` |
58
+ | Fallback quant | `UD-IQ2_M` (~18.5 GB) |
59
+ | Runtime | `llama-mtmd-cli` via llama.cpp on Modal GPU |
60
+
61
+ ### Fallback Split Pipeline
62
+ Only use if primary model code-gen quality is too weak:
63
+ - **Vision:** NVIDIA Nemotron Nano V2 VL (12B)
64
+ - **Scene gen:** Qwen2.5-Coder-14B
65
+ - Total ~26B · still qualifies for Nemotron Quest
66
+
67
+ ---
68
+
69
+ ## Model Runtime
70
+
71
+ - Inference via **llama.cpp / GGUF** targets **"Llama Champion"** bonus
72
+ - Modal GPU endpoints called over HTTP with Bearer token auth (`SNAP2SIM_API_TOKEN`)
73
+ - Backend swappable via `INFERENCE_BACKEND=modal | zerogpu | local`
74
+ - **If Modal deployed:** "Off the Grid" not claimed, but Llama Champion + Nemotron Quest + Modal Award all apply
75
+ - **If ZeroGPU sufficient:** "Off the Grid" additionally claimable
76
+
77
+ ---
78
+
79
+ ## Deployment Architecture
80
+
81
+ ```
82
+ [HF Space CPU tier] [Modal GPU tier]
83
+ gradio.Server analyze_image_llamacpp
84
+ @app.get("/") index.html ←→ generate_scene
85
+ @app.api() /analyze_image (weights cached in Modal Volume)
86
+ @app.api() /generate_scene
87
+ ```
88
+
89
+ - Gradio app calls Modal endpoints over HTTP via `requests`, image passed as base64
90
+ - Modal cold starts on 30B model can take tens of seconds → show `"WAKING THE WORKSHOP..."` loading state
91
+
92
+ ---
93
+
94
+ ## Architectural Shift `gr.Blocks``gradio.Server`
95
+
96
+ > **This is the core change. Do not skip or partially implement it.**
97
+
98
+ ### Why
99
+ `gr.HTML` strips `<script>` tags for security and HF Spaces CSP blocks external CDN imports in `js_on_load`. Any WebGL/Three.js/A-Frame output piped through `gr.HTML` will fail on the live Space — scripts get stripped, nothing renders. This is a confirmed, known issue.
100
+
101
+ ### How `gradio.Server` Fixes It
102
+ `gradio.Server` extends FastAPI. `@app.get("/")` serves `index.html` as a first-class trusted FastAPI response the browser receives a full page with no stripping, no sandboxing, no CSP conflicts from Gradio's component system. A-Frame and Three.js CDN scripts load normally.
103
+
104
+ ```python
105
+ from gradio import Server
106
+
107
+ app = Server()
108
+
109
+ @app.get("/")
110
+ async def homepage():
111
+ with open("index.html") as f:
112
+ return HTMLResponse(f.read())
113
+
114
+ @app.api(name="analyze_image")
115
+ def analyze_image(image_b64: str) -> dict:
116
+ return backend.run_analysis(image_b64) # calls Modal or local placeholder
117
+
118
+ @app.api(name="generate_scene")
119
+ def generate_scene(mechanism_json: dict) -> str:
120
+ return backend.run_scene_gen(mechanism_json) # returns A-Frame HTML string
121
+
122
+ app.launch()
123
+ ```
124
+
125
+ ### What to Change in the Scaffold
126
+ | File | Action |
127
+ |---|---|
128
+ | `app.py` | `gradio.Server` app serving `index.html` and API routes |
129
+ | `index.html` | Trusted HTML/CSS/JS shell with pipeline orchestration |
130
+ | `modal_app.py` | `generate_scene` endpoints and A-Frame prompt |
131
+ | `snap2sim/backend.py` | `generate_scene` backend method |
132
+ | `snap2sim/prompts.py` | A-Frame scene-generation prompt |
133
+ | `snap2sim/aframe_scene.py` | Deterministic A-Frame placeholder scene |
134
+
135
+ ---
136
+
137
+ ## Rendering Stack Two Layers
138
+
139
+ ### Layer 1 Model-Generated Scene: A-Frame (declarative HTML)
140
+
141
+ A-Frame is a web framework built on Three.js that uses declarative HTML tags for 3D scenes. The model outputs HTML, not JavaScript — far more reliable for LLM generation.
142
+
143
+ **Why A-Frame for model output:**
144
+ - LLMs generate HTML tags far more reliably than imperative JS
145
+ - Injected via `innerHTML`, not `eval()` — no script execution risk
146
+ - A-Frame runtime (already loaded in `<head>`) renders injected tags automatically
147
+ - Built-in `animation` attribute handles motion without JS animation loops
148
+ - Camera, lighting, and sky added automatically less boilerplate to get wrong
149
+
150
+ **Loading A-Frame in `index.html`:**
151
+ ```html
152
+ <head>
153
+ <script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
154
+ </head>
155
+ ```
156
+
157
+ > **CDN fallback:** If HF Spaces blocks `aframe.io`, vendor the minified A-Frame JS (~1.1MB) as a static file served via `gradio.Server`'s FastAPI static file mounting.
158
+
159
+ **Injecting model output:**
160
+ ```javascript
161
+ document.getElementById('viewport').innerHTML = modelGeneratedAframeHTML;
162
+ // A-Frame runtime picks up the new <a-scene> tags automatically
163
+ ```
164
+
165
+ **Example of what the model should output:**
166
+ ```html
167
+ <a-scene>
168
+ <a-sky color="#0F1318"></a-sky>
169
+ <a-cylinder color="#E8A33D" radius="0.3" height="1" position="0 1 -3"
170
+ animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear">
171
+ </a-cylinder>
172
+ <a-box color="#5FD4D0" position="0.8 0.5 -3"
173
+ animation="property: position; to: 0.8 1 -3; dir: alternate; loop: true; dur: 1000">
174
+ </a-box>
175
+ <a-text value="Drive Shaft" position="0 2 -3" color="#5FD4D0" scale="0.5 0.5 0.5">
176
+ </a-text>
177
+ </a-scene>
178
+ ```
179
+
180
+ **Prompt engineering for `generate_scene` endpoint:**
181
+ - Instruct the model to output **only** the `<a-scene>...</a-scene>` block no preamble, no markdown fences, no explanation
182
+ - A-Frame primitives to use: `<a-box>`, `<a-cylinder>`, `<a-sphere>`, `<a-torus>`, `<a-cone>`, `<a-entity>`
183
+ - Animation format: `animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear"`
184
+ - Keep scenes to **3–6 parts maximum** for clarity
185
+ - Set `<a-sky color="#0F1318">` to match the page background
186
+
187
+ ### Layer 2 — Deterministic Fallback: Three.js (human-written)
188
+
189
+ If A-Frame output is empty, malformed, or renders blank after 3 seconds, immediately swap to `buildDeterministicScene(json)` — a JS function that reads the mechanism JSON and builds a reliable Three.js scene from geometric primitives.
190
+
191
+ ```javascript
192
+ function buildDeterministicScene(mechanismJson) {
193
+ // Human-written. Always works given valid JSON.
194
+ // Uses: BoxGeometry, CylinderGeometry, TorusGeometry per part shape
195
+ // Applies rotation/translation per motion_type
196
+ // Adds OrbitControls, annotation labels
197
+ // The viewport must never be blank or show an error
198
+ }
199
+ ```
200
+
201
+ > Load Three.js in `index.html` alongside A-Frame:
202
+ > ```html
203
+ > <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
204
+ > ```
205
+
206
+ ---
207
+
208
+ ## Pipeline / Application Flow
209
+
210
+ ```
211
+ 1. User uploads photo
212
+
213
+ 2. Frontend encodes as base64 → calls /analyze_image (Gradio JS client)
214
+
215
+ 3. Nemotron vision step → structured JSON:
216
+ {
217
+ "component_name": "Solenoid Valve",
218
+ "parts": [
219
+ { "name": "Coil", "shape": "cylinder", "color": "#E8A33D",
220
+ "position": [0, 0, 0], "motion_type": "none", "motion_params": {} },
221
+ { "name": "Plunger", "shape": "cylinder", "color": "#5FD4D0",
222
+ "position": [0, 0.5, 0], "motion_type": "translate",
223
+ "motion_params": { "axis": "y", "range": 0.3, "dur": 800 } }
224
+ ],
225
+ "summary": "When current flows through the coil, it generates a magnetic
226
+ field that pulls the plunger upward, opening the valve port."
227
+ }
228
+
229
+ 4. Frontend populates analysis panel (name, part list, summary)
230
+ → immediately calls /generate_scene with JSON
231
+
232
+ 5. Nemotron scene gen step → A-Frame HTML string (<a-scene>...</a-scene>)
233
+
234
+ 6. Frontend injects A-Frame HTML → innerHTML of #viewport
235
+ A-Frame runtime renders automatically
236
+
237
+ 7. If A-Frame blank/failed after 3s → buildDeterministicScene(json)
238
+ Viewport is NEVER blank
239
+ ```
240
+
241
+ ---
242
+
243
+ ## Visual Design — "Industrial Instrument Panel / Field Cutaway"
244
+
245
+ > Implement the shell and CSS in Step 2. Do **not** work on `[POLISH]` items until Steps 1–4 are done.
246
+
247
+ ### Color System
248
+ ```css
249
+ :root {
250
+ --bg: #0F1318;
251
+ --bg-panel: #161B22;
252
+ --bg-lift: #1E2530;
253
+ --amber: #E8A33D;
254
+ --amber-dim: #7A5420;
255
+ --cyan: #5FD4D0;
256
+ --cyan-dim: #2A5E5C;
257
+ --text: #C8C0AC;
258
+ --text-muted: #6B7280;
259
+ --grid: rgba(255,255,255,0.04);
260
+ }
261
+ ```
262
+
263
+ ### Typography
264
+ Load from **Bunny Fonts** (not Google Fonts):
265
+ - **Display / headings / UI labels:** `Chakra Petch` — technical, instrument-panel character
266
+ - **Monospace / data / callouts:** `Fira Code`
267
+ - **Never use:** Inter, Roboto, Arial, Space Grotesk, or any system font
268
+
269
+ ### Layout
270
+ - Two-pane asymmetric split: **63% viewport** (left) · **37% analysis panel** (right)
271
+ - Blueprint grid: 1px lines at `--grid` opacity, 32px spacing, on `--bg` base
272
+ - Panel separator: 1px vertical line in `--amber-dim`
273
+ - Upload drop zone: fills viewport · dashed 1px `--amber-dim` border (no `border-radius`) · centered `"DROP COMPONENT PHOTO"` in Chakra Petch uppercase `--text-muted` · on drag-over: border → `--amber`, text → `--amber`
274
+ - Play/pause: minimal amber rectangle (no `border-radius`), Chakra Petch uppercase `"PAUSE"` / `"RESUME"`, controls A-Frame animation playback via JS
275
+
276
+ ### Loading States
277
+ All in Chakra Petch uppercase, `--amber` color, with thin `--amber` indeterminate progress bar across viewport top:
278
+
279
+ | State | Message |
280
+ |---|---|
281
+ | Modal cold start | `WAKING THE WORKSHOP...` |
282
+ | Vision inference | `ANALYZING ASSEMBLY...` |
283
+ | Scene generation | `RENDERING CUTAWAY...` |
284
+
285
+ ### [POLISH] — Only After Steps 1–4 Work
286
+ Implement in this sub-order:
287
+
288
+ 1. **Component name watermark** — large (`clamp(4rem, 8vw, 9rem)`) Chakra Petch uppercase in `--bg-lift`, absolutely positioned bleeding across both panes from bottom-left, `z-index` below content. Populated from `component_name` in JSON.
289
+
290
+ 2. **Noise texture** — SVG `feTurbulence` grain at 3% opacity on viewport pane, inline `data:` URI, no external file. Makes the surface feel physical.
291
+
292
+ 3. **Vignette** — radial gradient overlay on viewport edges, `pointer-events: none` so it floats above the A-Frame canvas without blocking interaction.
293
+
294
+ 4. **Scan-line reveal** — when A-Frame scene first loads:
295
+ - 2px `--cyan` scan-line sweeps top→bottom over 0.7s
296
+ - Each A-Frame entity fades in as line passes it (`opacity 0→1`, `translateY 12px→0`, 0.35s ease-out, staggered by part index via `animation-delay`)
297
+ - Part labels fade in together (0.25s)
298
+ - Progress bar dissolves (0.2s)
299
+ - Total: ~1.2s · this is the signature moment
300
+
301
+ ---
302
+
303
+ ## Deliverables
304
+
305
+ - `app.py` — `gradio.Server` app (~50 lines)
306
+ - `index.html` — self-contained HTML/CSS/JS; A-Frame + Three.js from CDN; Gradio JS client from CDN
307
+ - `snap2sim/backend.py` — `generate_scene`
308
+ - `modal_app.py` — `generate_scene`; A-Frame prompt
309
+ - `snap2sim/prompts.py` — updated A-Frame scene generation prompt
310
+ - `requirements.txt` — updated if needed for `gradio.Server`
311
+ - `README.md` — tinkerer/maker story hook → project description → model stack with exact parameter breakdown (≤32B) → rendering stack rationale → bonus quest claims:
312
+
313
+ | Quest | Status |
314
+ |---|---|
315
+ | Llama Champion | ✅ Confirmed |
316
+ | NVIDIA Nemotron Quest | ✅ Confirmed |
317
+ | Off-Brand | ✅ Confirmed |
318
+ | Modal Award | ✅ Confirmed |
319
+ | Off the Grid | ⚡ If ZeroGPU used in final deploy |
320
+ | Field Notes | 🎯 Stretch |
321
+
322
+ ---
323
+
324
+ ## Start Order
325
+
326
+ > Follow this exactly. Do not skip ahead.
327
+
328
+ ### Step 1 — Critical Path
329
+ Deploy the current `gradio.Server` + `index.html` implementation through
330
+ GitHub → HF sync. Confirm the private Space loads and both `/analyze_image`
331
+ and `/generate_scene` respond.
332
+
333
+ ### Step 2 — Make It Functional
334
+ Build `buildDeterministicScene(json)` in JS — Three.js scene from geometric primitives, always works given valid JSON. Wire the full pipeline: upload → `/analyze_image` → populate analysis panel → `/generate_scene` → inject A-Frame HTML via `innerHTML` → fallback to `buildDeterministicScene(json)` if A-Frame fails or blanks after 3 seconds. **Confirm end-to-end with a real image and live Modal endpoints.**
335
+
336
+ ### Step 3 — Apply the Design Shell
337
+ Add CSS variable system, Chakra Petch + Fira Code from Bunny Fonts, two-pane asymmetric layout, blueprint grid, loading states with progress bar, upload drop zone, panel separator. App should match the design direction above — minus `[POLISH]` items.
338
+
339
+ ### Step 4 — Harden
340
+ Error handling, 3-second blank-scene timeout before fallback triggers, Modal cold-start messaging, A-Frame `<a-sky color="#0F1318">` matching page background, play/pause toggle wired to A-Frame animation playback. Verify GitHub → HF Space sync pushes cleanly and the private Space runs correctly.
341
+
342
+ ### Step 5 — [POLISH] Only if Time Remains
343
+ Component name watermark → noise texture → vignette → scan-line reveal. In that order.
README.md CHANGED
@@ -9,13 +9,18 @@ colorTo: yellow
9
 
10
  # Snap2Sim
11
 
12
- Inside the Machine is a Gradio app scaffold for the Build Small Hackathon
13
- Backyard AI track. It takes a hardware component photo, produces a structured
14
- mechanism analysis, and renders a technical cutaway visualization.
15
 
16
- The local app still defaults to the placeholder backend, but the Modal
17
- llama.cpp path has been smoke-tested with the selected Nemotron GGUF and
18
- projector.
 
 
 
 
 
 
19
 
20
  ## Run Locally
21
 
@@ -25,7 +30,8 @@ python app.py
25
  ```
26
 
27
  Set `INFERENCE_BACKEND=modal`, `MODAL_ANALYZE_URL`, and `MODAL_GENERATE_URL`
28
- to point the Gradio app at deployed Modal endpoints.
 
29
 
30
  For the deployed demo configuration, copy `.env.example` into local environment
31
  variables or Hugging Face Space variables. Do not put tokens in `.env.example`;
@@ -68,20 +74,60 @@ Primary references:
68
  - https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
69
  - https://github.com/ggml-org/llama.cpp
70
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
  ## Project Structure
72
 
73
  - `.env.example` - public runtime variables for the deployed Modal demo.
74
  - `SECURITY.md` - public/private data handling guidance for humans and agents.
75
- - `app.py` - Hugging Face Space entry point.
 
 
 
76
  - `modal_app.py` - Modal web endpoint scaffold.
77
  - `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
78
  - `snap2sim/backend.py` - backend selection and HTTP client.
79
- - `snap2sim/fallback_scene.py` - themed 2D fallback animation.
80
- - `snap2sim/prompts.py` - prompt templates for vision and Three.js generation.
 
81
  - `snap2sim/schema.py` - JSON schema and sample analysis payload.
82
- - `snap2sim/three_scene.py` - deterministic Three.js scene generation from the
83
- validated mechanism JSON.
84
- - `snap2sim/ui.py` - Gradio Blocks UI and visual theme.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
86
  ## Modal Deployment Path
87
 
@@ -113,9 +159,9 @@ Useful deployment functions/endpoints:
113
  synthetic image and confirms it returns a validated mechanism payload.
114
  - `runtime_probe` reports the configured model repo, quant, projector file, and
115
  whether placeholder inference is still active.
116
- - `analyze_image` and `generate_threejs` preserve the current HTTP contract for
117
  the Gradio app.
118
- - `analyze_image_llamacpp` and `generate_threejs_llamacpp` are experimental GPU
119
  endpoints for the llama.cpp runtime path after the smoke test passes.
120
 
121
  Runtime environment knobs:
@@ -130,7 +176,7 @@ Runtime environment knobs:
130
 
131
  Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
132
  `MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint and keep
133
- `MODAL_GENERATE_URL` on deterministic `generate_threejs`.
134
 
135
  Run the deployment preflight in this order:
136
 
 
9
 
10
  # Snap2Sim
11
 
12
+ You find a small metal cylinder at a flea market. What is it? How does it work
13
+ inside?
 
14
 
15
+ Inside the Machine is a Build Small Hackathon Backyard AI project for curious
16
+ tinkerers and makers. It takes a hardware component photo, produces a
17
+ structured mechanism analysis, and renders an animated technical cutaway
18
+ visualization.
19
+
20
+ The local app now serves a trusted `index.html` shell through `gradio.Server`
21
+ and exposes `/analyze_image` plus `/generate_scene` routes. It still defaults
22
+ to the placeholder backend locally, while the Modal llama.cpp path has been
23
+ smoke-tested with the selected Nemotron GGUF and projector.
24
 
25
  ## Run Locally
26
 
 
30
  ```
31
 
32
  Set `INFERENCE_BACKEND=modal`, `MODAL_ANALYZE_URL`, and `MODAL_GENERATE_URL`
33
+ to point the Gradio app at deployed Modal `/analyze_image` and
34
+ `/generate_scene` endpoints.
35
 
36
  For the deployed demo configuration, copy `.env.example` into local environment
37
  variables or Hugging Face Space variables. Do not put tokens in `.env.example`;
 
74
  - https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
75
  - https://github.com/ggml-org/llama.cpp
76
 
77
+ ## Model Stack and Quest Claims
78
+
79
+ Primary inference uses NVIDIA Nemotron 3 Nano Omni 30B-A3B through llama.cpp
80
+ GGUF. The model has about 30B total parameters with roughly 3B active per MoE
81
+ token, keeping the single-model pipeline under the hackathon 32B cap.
82
+
83
+ Fallback, only if the primary endpoint quality is not demo-ready, is
84
+ NVIDIA Nemotron Nano V2 VL 12B for vision plus Qwen2.5-Coder-14B for scene
85
+ generation, for about 26B total parameters.
86
+
87
+ | Quest | Status |
88
+ |---|---|
89
+ | Llama Champion | Confirmed via llama.cpp / GGUF runtime |
90
+ | NVIDIA Nemotron Quest | Confirmed via Nemotron primary model |
91
+ | Off-Brand | Confirmed |
92
+ | Modal Award | Confirmed via deployed Modal endpoints |
93
+ | Off the Grid | Not claimed while inference runs on Modal |
94
+ | Field Notes | Stretch |
95
+
96
  ## Project Structure
97
 
98
  - `.env.example` - public runtime variables for the deployed Modal demo.
99
  - `SECURITY.md` - public/private data handling guidance for humans and agents.
100
+ - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
101
+ `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
102
+ - `index.html` - plain HTML/CSS/JS shell with upload orchestration,
103
+ A-Frame injection, and deterministic browser-side Three.js fallback.
104
  - `modal_app.py` - Modal web endpoint scaffold.
105
  - `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
106
  - `snap2sim/backend.py` - backend selection and HTTP client.
107
+ - `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
108
+ local and placeholder Modal mode.
109
+ - `snap2sim/prompts.py` - prompt templates for vision and A-Frame generation.
110
  - `snap2sim/schema.py` - JSON schema and sample analysis payload.
111
+
112
+ ## Current Rendering Architecture
113
+
114
+ The app uses `gradio.Server` to serve `index.html` directly. This avoids
115
+ Gradio component script stripping and lets the page load A-Frame and Three.js
116
+ normally.
117
+
118
+ Runtime flow:
119
+
120
+ 1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
121
+ 2. Backend returns the validated mechanism JSON.
122
+ 3. Browser posts the JSON to `/generate_scene`.
123
+ 4. The scene endpoint returns only an `<a-scene>...</a-scene>` block.
124
+ 5. Browser injects the A-Frame scene. If it is malformed or does not create a
125
+ canvas within 3 seconds, `buildDeterministicScene(json)` renders a
126
+ human-written Three.js fallback in the same viewport.
127
+
128
+ The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
129
+ 63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
130
+ colors, explicit Modal cold-start messaging, and a play/pause control.
131
 
132
  ## Modal Deployment Path
133
 
 
159
  synthetic image and confirms it returns a validated mechanism payload.
160
  - `runtime_probe` reports the configured model repo, quant, projector file, and
161
  whether placeholder inference is still active.
162
+ - `analyze_image` and `generate_scene` preserve the current HTTP contract for
163
  the Gradio app.
164
+ - `analyze_image_llamacpp` and `generate_scene_llamacpp` are experimental GPU
165
  endpoints for the llama.cpp runtime path after the smoke test passes.
166
 
167
  Runtime environment knobs:
 
176
 
177
  Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
178
  `MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint and keep
179
+ `MODAL_GENERATE_URL` on deterministic `generate_scene`.
180
 
181
  Run the deployment preflight in this order:
182
 
SECURITY.md CHANGED
@@ -82,6 +82,12 @@ Do not configure jobs that pull changes back from Hugging Face into GitHub.
82
 
83
  - Prefer Hugging Face Space variables for public configuration and Space secrets
84
  for credentials.
 
 
 
 
 
 
85
  - Do not print secret-bearing environment variables in logs.
86
  - Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
87
  or Codex summaries.
 
82
 
83
  - Prefer Hugging Face Space variables for public configuration and Space secrets
84
  for credentials.
85
+ - The planned `index.html` / `gradio.Server` frontend is public client-side
86
+ code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
87
+ tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
88
+ static assets, browser local storage, or query strings. Browser JS should call
89
+ same-origin `@app.api()` endpoints; server-side Python should attach secrets
90
+ when calling Modal.
91
  - Do not print secret-bearing environment variables in logs.
92
  - Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
93
  or Codex summaries.
app.py CHANGED
@@ -1,11 +1,96 @@
1
- """Hugging Face Space entry point."""
2
 
3
- from snap2sim.ui import build_app
4
 
 
 
 
 
5
 
6
- demo = build_app()
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
- if __name__ == "__main__":
10
- demo.launch()
11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Hugging Face Space entry point for the trusted HTML shell."""
2
 
3
+ from __future__ import annotations
4
 
5
+ import base64
6
+ from io import BytesIO
7
+ from pathlib import Path
8
+ from typing import Any, Callable
9
 
10
+ from fastapi.responses import HTMLResponse
11
+ from PIL import Image
12
 
13
+ from snap2sim.backend import InferenceClient, Settings
14
+
15
+ try:
16
+ from gradio import Server
17
+ except ImportError:
18
+ from fastapi import FastAPI
19
+ import uvicorn
20
+
21
+ class Server(FastAPI): # type: ignore[no-redef]
22
+ """Local compatibility shim for environments older than Gradio Server."""
23
+
24
+ def api(self, name: str) -> Callable[[Callable[..., Any]], Callable[..., Any]]:
25
+ def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
26
+ return func
27
+
28
+ return decorator
29
+
30
+ def launch(self, **kwargs: Any) -> None:
31
+ uvicorn.run(
32
+ self,
33
+ host=kwargs.get("server_name", "0.0.0.0"),
34
+ port=kwargs.get("server_port", 7860),
35
+ )
36
+
37
+
38
+ app = Server()
39
+ INDEX_PATH = Path(__file__).with_name("index.html")
40
+
41
+
42
+ @app.get("/", response_class=HTMLResponse)
43
+ async def homepage() -> str:
44
+ return INDEX_PATH.read_text(encoding="utf-8")
45
+
46
+
47
+ @app.get("/manifest.json")
48
+ async def manifest() -> dict[str, Any]:
49
+ return {
50
+ "name": "Snap2Sim Inside the Machine",
51
+ "short_name": "Snap2Sim",
52
+ "start_url": "/",
53
+ "display": "standalone",
54
+ "background_color": "#0F1318",
55
+ "theme_color": "#E8A33D",
56
+ }
57
+
58
+
59
+ @app.api(name="analyze_image")
60
+ def analyze_image_api(image_base64: str) -> dict[str, Any]:
61
+ return _analyze_image(image_base64)
62
 
 
 
63
 
64
+ @app.post("/analyze_image")
65
+ def analyze_image_http(payload: dict[str, Any]) -> dict[str, Any]:
66
+ return _analyze_image(str(payload.get("image_base64", "")))
67
+
68
+
69
+ @app.api(name="generate_scene")
70
+ def generate_scene_api(analysis: dict[str, Any]) -> str:
71
+ return _generate_scene(analysis)
72
+
73
+
74
+ @app.post("/generate_scene")
75
+ def generate_scene_http(payload: dict[str, Any]) -> dict[str, str]:
76
+ return {"html": _generate_scene(payload.get("analysis") or {})}
77
+
78
+
79
+ def _analyze_image(image_base64: str) -> dict[str, Any]:
80
+ image = _decode_image(image_base64) if image_base64 else None
81
+ return InferenceClient(Settings()).analyze_image(image)
82
+
83
+
84
+ def _generate_scene(analysis: dict[str, Any]) -> str:
85
+ return InferenceClient(Settings()).generate_scene(analysis)
86
+
87
+
88
+ def _decode_image(image_base64: str) -> Image.Image:
89
+ if "," in image_base64 and image_base64.lstrip().startswith("data:"):
90
+ image_base64 = image_base64.split(",", 1)[1]
91
+ raw = base64.b64decode(image_base64)
92
+ return Image.open(BytesIO(raw)).convert("RGB")
93
+
94
+
95
+ if __name__ == "__main__":
96
+ app.launch()
index.html ADDED
@@ -0,0 +1,808 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>Inside the Machine</title>
7
+ <link rel="preconnect" href="https://fonts.bunny.net">
8
+ <link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
9
+ <script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
10
+ <script type="module">
11
+ import { Client } from "https://cdn.jsdelivr.net/npm/@gradio/client/dist/index.min.js";
12
+ window.snap2simGradioClient = Client;
13
+ </script>
14
+ <style>
15
+ :root {
16
+ --bg: #0F1318;
17
+ --bg-panel: #161B22;
18
+ --bg-lift: #1E2530;
19
+ --amber: #E8A33D;
20
+ --amber-dim: #7A5420;
21
+ --cyan: #5FD4D0;
22
+ --cyan-dim: #2A5E5C;
23
+ --text: #C8C0AC;
24
+ --text-muted: #6B7280;
25
+ --grid: rgba(255,255,255,0.04);
26
+ --danger: #F07F5A;
27
+ }
28
+
29
+ * {
30
+ box-sizing: border-box;
31
+ }
32
+
33
+ html,
34
+ body {
35
+ width: 100%;
36
+ height: 100%;
37
+ margin: 0;
38
+ overflow: hidden;
39
+ color: var(--text);
40
+ background:
41
+ linear-gradient(var(--grid) 1px, transparent 1px),
42
+ linear-gradient(90deg, var(--grid) 1px, transparent 1px),
43
+ var(--bg);
44
+ background-size: 32px 32px;
45
+ font-family: "Chakra Petch", "Fira Code", monospace;
46
+ }
47
+
48
+ button,
49
+ input {
50
+ font: inherit;
51
+ }
52
+
53
+ .shell {
54
+ display: grid;
55
+ grid-template-columns: minmax(0, 63fr) minmax(340px, 37fr);
56
+ width: 100vw;
57
+ height: 100vh;
58
+ }
59
+
60
+ .viewport-pane {
61
+ position: relative;
62
+ min-width: 0;
63
+ overflow: hidden;
64
+ background:
65
+ radial-gradient(circle at 50% 40%, rgba(95, 212, 208, 0.06), transparent 42%),
66
+ var(--bg);
67
+ }
68
+
69
+ .viewport-pane::after {
70
+ content: "";
71
+ position: absolute;
72
+ inset: 0;
73
+ pointer-events: none;
74
+ background: radial-gradient(circle at center, transparent 58%, rgba(0, 0, 0, 0.36));
75
+ z-index: 7;
76
+ }
77
+
78
+ #viewport {
79
+ position: absolute;
80
+ inset: 0;
81
+ z-index: 1;
82
+ }
83
+
84
+ #viewport a-scene {
85
+ width: 100%;
86
+ height: 100%;
87
+ }
88
+
89
+ #viewport .a-enter-vr,
90
+ #viewport .a-orientation-modal {
91
+ display: none !important;
92
+ }
93
+
94
+ .drop-zone {
95
+ position: absolute;
96
+ inset: 18px;
97
+ z-index: 5;
98
+ display: grid;
99
+ place-items: center;
100
+ border: 1px dashed var(--amber-dim);
101
+ color: var(--text-muted);
102
+ background: rgba(15, 19, 24, 0.72);
103
+ text-transform: uppercase;
104
+ letter-spacing: 0;
105
+ font-size: clamp(1.15rem, 2vw, 2.1rem);
106
+ font-weight: 600;
107
+ transition: color 160ms ease, border-color 160ms ease, background 160ms ease;
108
+ }
109
+
110
+ .drop-zone.dragging {
111
+ color: var(--amber);
112
+ border-color: var(--amber);
113
+ background: rgba(122, 84, 32, 0.16);
114
+ }
115
+
116
+ .drop-zone.hidden {
117
+ display: none;
118
+ }
119
+
120
+ .scan-line {
121
+ position: absolute;
122
+ left: 0;
123
+ right: 0;
124
+ top: -2px;
125
+ z-index: 8;
126
+ height: 2px;
127
+ opacity: 0;
128
+ background: var(--cyan);
129
+ box-shadow: 0 0 18px rgba(95, 212, 208, 0.8);
130
+ pointer-events: none;
131
+ }
132
+
133
+ .scan-line.active {
134
+ animation: scan 700ms ease-out;
135
+ }
136
+
137
+ .progress {
138
+ position: absolute;
139
+ left: 0;
140
+ right: 0;
141
+ top: 0;
142
+ z-index: 9;
143
+ height: 2px;
144
+ overflow: hidden;
145
+ opacity: 0;
146
+ background: rgba(122, 84, 32, 0.35);
147
+ transition: opacity 180ms ease;
148
+ }
149
+
150
+ .progress.active {
151
+ opacity: 1;
152
+ }
153
+
154
+ .progress::before {
155
+ content: "";
156
+ position: absolute;
157
+ top: 0;
158
+ bottom: 0;
159
+ width: 34%;
160
+ background: var(--amber);
161
+ animation: loading 1050ms linear infinite;
162
+ }
163
+
164
+ .toolbar {
165
+ position: absolute;
166
+ left: 18px;
167
+ top: 18px;
168
+ z-index: 10;
169
+ display: flex;
170
+ gap: 8px;
171
+ }
172
+
173
+ .tool-button {
174
+ min-width: 82px;
175
+ min-height: 34px;
176
+ border: 1px solid var(--amber-dim);
177
+ border-radius: 0;
178
+ color: var(--amber);
179
+ background: rgba(15, 19, 24, 0.86);
180
+ text-transform: uppercase;
181
+ letter-spacing: 0;
182
+ cursor: pointer;
183
+ }
184
+
185
+ .tool-button:hover,
186
+ .tool-button:focus-visible {
187
+ border-color: var(--amber);
188
+ outline: none;
189
+ }
190
+
191
+ .tool-button[disabled] {
192
+ color: var(--text-muted);
193
+ border-color: var(--cyan-dim);
194
+ cursor: default;
195
+ }
196
+
197
+ .panel {
198
+ min-width: 0;
199
+ overflow: auto;
200
+ border-left: 1px solid var(--amber-dim);
201
+ background: linear-gradient(180deg, rgba(30, 37, 48, 0.92), var(--bg-panel));
202
+ }
203
+
204
+ .panel-inner {
205
+ min-height: 100%;
206
+ padding: 24px;
207
+ display: grid;
208
+ grid-template-rows: auto auto 1fr auto;
209
+ gap: 22px;
210
+ }
211
+
212
+ .brand {
213
+ display: grid;
214
+ gap: 8px;
215
+ border-bottom: 1px solid rgba(122, 84, 32, 0.7);
216
+ padding-bottom: 18px;
217
+ }
218
+
219
+ .kicker,
220
+ .label {
221
+ color: var(--amber);
222
+ font-size: 0.78rem;
223
+ font-weight: 600;
224
+ text-transform: uppercase;
225
+ }
226
+
227
+ h1 {
228
+ margin: 0;
229
+ color: var(--text);
230
+ font-size: clamp(2rem, 4vw, 4.4rem);
231
+ line-height: 0.9;
232
+ font-weight: 700;
233
+ letter-spacing: 0;
234
+ text-transform: uppercase;
235
+ }
236
+
237
+ .status {
238
+ min-height: 24px;
239
+ color: var(--amber);
240
+ font-size: 0.92rem;
241
+ font-weight: 600;
242
+ text-transform: uppercase;
243
+ }
244
+
245
+ .status.error {
246
+ color: var(--danger);
247
+ }
248
+
249
+ .readout {
250
+ display: grid;
251
+ gap: 18px;
252
+ }
253
+
254
+ .metric-row {
255
+ display: grid;
256
+ grid-template-columns: 1fr auto;
257
+ gap: 16px;
258
+ align-items: baseline;
259
+ border-bottom: 1px solid rgba(95, 212, 208, 0.16);
260
+ padding-bottom: 10px;
261
+ }
262
+
263
+ .component {
264
+ color: var(--cyan);
265
+ font-size: clamp(1.4rem, 2.2vw, 2.4rem);
266
+ font-weight: 700;
267
+ line-height: 1;
268
+ text-transform: uppercase;
269
+ }
270
+
271
+ .confidence {
272
+ color: var(--amber);
273
+ font-family: "Fira Code", monospace;
274
+ font-size: 0.95rem;
275
+ }
276
+
277
+ .summary,
278
+ .trigger,
279
+ .sequence,
280
+ .parts {
281
+ margin: 0;
282
+ color: var(--text);
283
+ font-size: 0.98rem;
284
+ line-height: 1.5;
285
+ }
286
+
287
+ .trigger {
288
+ color: var(--text-muted);
289
+ }
290
+
291
+ .sequence,
292
+ .parts {
293
+ display: grid;
294
+ gap: 8px;
295
+ padding: 0;
296
+ list-style: none;
297
+ }
298
+
299
+ .sequence li,
300
+ .parts li {
301
+ border-left: 1px solid var(--cyan-dim);
302
+ padding-left: 10px;
303
+ }
304
+
305
+ .parts strong {
306
+ display: block;
307
+ color: var(--cyan);
308
+ font-weight: 600;
309
+ text-transform: uppercase;
310
+ }
311
+
312
+ .json-box {
313
+ max-height: 240px;
314
+ overflow: auto;
315
+ margin: 0;
316
+ border-top: 1px solid rgba(122, 84, 32, 0.7);
317
+ padding-top: 14px;
318
+ color: var(--text-muted);
319
+ font: 0.78rem/1.45 "Fira Code", monospace;
320
+ white-space: pre-wrap;
321
+ }
322
+
323
+ .fallback-stage,
324
+ .scene-mount,
325
+ .label-layer {
326
+ position: absolute;
327
+ inset: 0;
328
+ }
329
+
330
+ .scene-label {
331
+ position: absolute;
332
+ max-width: min(180px, 34vw);
333
+ transform: translate(-50%, -50%);
334
+ border-left: 1px solid var(--cyan);
335
+ padding: 4px 0 4px 8px;
336
+ color: var(--cyan);
337
+ background: rgba(15, 19, 24, 0.72);
338
+ font: 0.72rem/1.2 "Fira Code", monospace;
339
+ text-transform: uppercase;
340
+ pointer-events: none;
341
+ }
342
+
343
+ @keyframes loading {
344
+ from { transform: translateX(-100%); }
345
+ to { transform: translateX(300%); }
346
+ }
347
+
348
+ @keyframes scan {
349
+ 0% { transform: translateY(0); opacity: 0; }
350
+ 12%, 75% { opacity: 1; }
351
+ 100% { transform: translateY(100vh); opacity: 0; }
352
+ }
353
+
354
+ @media (max-width: 860px) {
355
+ body {
356
+ overflow: auto;
357
+ }
358
+
359
+ .shell {
360
+ grid-template-columns: 1fr;
361
+ grid-template-rows: minmax(420px, 62vh) auto;
362
+ min-height: 100vh;
363
+ height: auto;
364
+ }
365
+
366
+ .panel {
367
+ border-left: 0;
368
+ border-top: 1px solid var(--amber-dim);
369
+ }
370
+ }
371
+ </style>
372
+ </head>
373
+ <body>
374
+ <main class="shell">
375
+ <section class="viewport-pane" aria-label="Cutaway viewport">
376
+ <div id="progress" class="progress"></div>
377
+ <div class="toolbar">
378
+ <button id="uploadButton" class="tool-button" type="button">Load</button>
379
+ <button id="playButton" class="tool-button" type="button" disabled>Pause</button>
380
+ </div>
381
+ <div id="viewport"></div>
382
+ <label id="dropZone" class="drop-zone" for="fileInput">Drop component photo</label>
383
+ <div id="scanLine" class="scan-line"></div>
384
+ <input id="fileInput" type="file" accept="image/*" hidden>
385
+ </section>
386
+
387
+ <aside class="panel">
388
+ <div class="panel-inner">
389
+ <header class="brand">
390
+ <div class="kicker">Snap2Sim</div>
391
+ <h1>Inside the Machine</h1>
392
+ <div id="status" class="status">Standby</div>
393
+ </header>
394
+
395
+ <section class="readout">
396
+ <div class="metric-row">
397
+ <div id="component" class="component">Awaiting Photo</div>
398
+ <div id="confidence" class="confidence">--</div>
399
+ </div>
400
+ <p id="summary" class="summary"></p>
401
+ <p id="trigger" class="trigger"></p>
402
+ </section>
403
+
404
+ <section class="readout">
405
+ <div>
406
+ <div class="label">Motion Sequence</div>
407
+ <ol id="sequence" class="sequence"></ol>
408
+ </div>
409
+ <div>
410
+ <div class="label">Parts</div>
411
+ <ul id="parts" class="parts"></ul>
412
+ </div>
413
+ </section>
414
+
415
+ <pre id="rawJson" class="json-box">{}</pre>
416
+ </div>
417
+ </aside>
418
+ </main>
419
+
420
+ <script>
421
+ const fileInput = document.getElementById("fileInput");
422
+ const uploadButton = document.getElementById("uploadButton");
423
+ const playButton = document.getElementById("playButton");
424
+ const dropZone = document.getElementById("dropZone");
425
+ const viewport = document.getElementById("viewport");
426
+ const progress = document.getElementById("progress");
427
+ const scanLine = document.getElementById("scanLine");
428
+ const statusEl = document.getElementById("status");
429
+ const componentEl = document.getElementById("component");
430
+ const confidenceEl = document.getElementById("confidence");
431
+ const summaryEl = document.getElementById("summary");
432
+ const triggerEl = document.getElementById("trigger");
433
+ const sequenceEl = document.getElementById("sequence");
434
+ const partsEl = document.getElementById("parts");
435
+ const rawJsonEl = document.getElementById("rawJson");
436
+
437
+ let activeMode = "idle";
438
+ let paused = false;
439
+ let coldStartTimer = 0;
440
+ let fallbackRuntime = null;
441
+
442
+ uploadButton.addEventListener("click", () => fileInput.click());
443
+ fileInput.addEventListener("change", () => {
444
+ const file = fileInput.files && fileInput.files[0];
445
+ if (file) runPipeline(file);
446
+ });
447
+
448
+ for (const eventName of ["dragenter", "dragover"]) {
449
+ dropZone.addEventListener(eventName, (event) => {
450
+ event.preventDefault();
451
+ dropZone.classList.add("dragging");
452
+ });
453
+ }
454
+
455
+ for (const eventName of ["dragleave", "drop"]) {
456
+ dropZone.addEventListener(eventName, (event) => {
457
+ event.preventDefault();
458
+ dropZone.classList.remove("dragging");
459
+ });
460
+ }
461
+
462
+ dropZone.addEventListener("drop", (event) => {
463
+ const file = event.dataTransfer.files && event.dataTransfer.files[0];
464
+ if (file) runPipeline(file);
465
+ });
466
+
467
+ playButton.addEventListener("click", () => {
468
+ paused = !paused;
469
+ playButton.textContent = paused ? "Resume" : "Pause";
470
+ if (activeMode === "aframe") {
471
+ const scene = viewport.querySelector("a-scene");
472
+ if (scene) paused ? scene.pause() : scene.play();
473
+ }
474
+ if (fallbackRuntime) fallbackRuntime.playing = !paused;
475
+ });
476
+
477
+ async function runPipeline(file) {
478
+ resetScene();
479
+ setBusy(true);
480
+ setStatus("ANALYZING ASSEMBLY...");
481
+ coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
482
+
483
+ try {
484
+ const imageBase64 = await fileToDataUrl(file);
485
+ const analysis = await postJson("/analyze_image", { image_base64: imageBase64 });
486
+ window.clearTimeout(coldStartTimer);
487
+ populateAnalysis(analysis);
488
+
489
+ setStatus("RENDERING CUTAWAY...");
490
+ const sceneResponse = await postJson("/generate_scene", { analysis });
491
+ const sceneHtml = typeof sceneResponse === "string" ? sceneResponse : sceneResponse.html;
492
+ renderAframe(sceneHtml, analysis);
493
+ } catch (error) {
494
+ window.clearTimeout(coldStartTimer);
495
+ setStatus(error.message || String(error), true);
496
+ if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
497
+ } finally {
498
+ setBusy(false);
499
+ }
500
+ }
501
+
502
+ function resetScene() {
503
+ if (fallbackRuntime && fallbackRuntime.cleanup) fallbackRuntime.cleanup();
504
+ fallbackRuntime = null;
505
+ activeMode = "idle";
506
+ paused = false;
507
+ playButton.textContent = "Pause";
508
+ playButton.disabled = true;
509
+ viewport.replaceChildren();
510
+ dropZone.classList.add("hidden");
511
+ }
512
+
513
+ function populateAnalysis(analysis) {
514
+ window.lastAnalysis = analysis;
515
+ componentEl.textContent = String(analysis.component || "Component").toUpperCase();
516
+ confidenceEl.textContent = typeof analysis.confidence === "number"
517
+ ? Math.round(analysis.confidence * 100) + "%"
518
+ : "--";
519
+ summaryEl.textContent = analysis.summary || "";
520
+ triggerEl.textContent = analysis.trigger ? "Trigger: " + analysis.trigger : "";
521
+ rawJsonEl.textContent = JSON.stringify(analysis, null, 2);
522
+ sequenceEl.replaceChildren(...(analysis.motion_sequence || []).map((step) => {
523
+ const item = document.createElement("li");
524
+ item.textContent = step;
525
+ return item;
526
+ }));
527
+ partsEl.replaceChildren(...(analysis.parts || []).map((part) => {
528
+ const item = document.createElement("li");
529
+ const name = document.createElement("strong");
530
+ name.textContent = part.name || part.id || "part";
531
+ const role = document.createElement("span");
532
+ role.textContent = part.role || "";
533
+ item.append(name, role);
534
+ return item;
535
+ }));
536
+ }
537
+
538
+ function renderAframe(sceneHtml, analysis) {
539
+ if (!sceneHtml || !/<a-scene[\s>]/i.test(sceneHtml) || !/<\/a-scene>/i.test(sceneHtml)) {
540
+ buildDeterministicScene(analysis);
541
+ return;
542
+ }
543
+
544
+ activeMode = "aframe";
545
+ viewport.innerHTML = sceneHtml;
546
+ const scene = viewport.querySelector("a-scene");
547
+ if (scene) {
548
+ scene.setAttribute("embedded", "");
549
+ scene.setAttribute("vr-mode-ui", "enabled: false");
550
+ scene.setAttribute("device-orientation-permission-ui", "enabled: false");
551
+ scene.setAttribute("background", "color: #0F1318");
552
+ }
553
+ playButton.disabled = false;
554
+ revealScan();
555
+
556
+ window.setTimeout(() => {
557
+ const liveScene = viewport.querySelector("a-scene");
558
+ const hasCanvas = Boolean(liveScene && liveScene.canvas);
559
+ if (!hasCanvas) buildDeterministicScene(analysis);
560
+ }, 3000);
561
+
562
+ setStatus("CUTAWAY READY");
563
+ }
564
+
565
+ function buildDeterministicScene(analysis) {
566
+ const threeRuntime = window.THREE || (window.AFRAME && window.AFRAME.THREE);
567
+ if (!threeRuntime) {
568
+ setStatus("3D RUNTIME UNAVAILABLE", true);
569
+ return;
570
+ }
571
+ window.THREE = threeRuntime;
572
+
573
+ if (fallbackRuntime && fallbackRuntime.cleanup) fallbackRuntime.cleanup();
574
+ activeMode = "three";
575
+ paused = false;
576
+ playButton.textContent = "Pause";
577
+ playButton.disabled = false;
578
+ viewport.innerHTML = '<div class="fallback-stage"><div class="scene-mount"></div><div class="label-layer"></div></div>';
579
+
580
+ const mount = viewport.querySelector(".scene-mount");
581
+ const labelLayer = viewport.querySelector(".label-layer");
582
+ const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
583
+ renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
584
+ mount.appendChild(renderer.domElement);
585
+
586
+ const scene = new THREE.Scene();
587
+ scene.fog = new THREE.Fog(0x0f1318, 8, 20);
588
+
589
+ const camera = new THREE.PerspectiveCamera(42, 1, 0.1, 100);
590
+ camera.position.set(5.4, 3.9, 6.2);
591
+
592
+ const controls = window.THREE.OrbitControls
593
+ ? new THREE.OrbitControls(camera, renderer.domElement)
594
+ : null;
595
+ if (controls) {
596
+ controls.enableDamping = true;
597
+ controls.target.set(0, 0.1, 0);
598
+ }
599
+
600
+ scene.add(new THREE.HemisphereLight(0x5fd4d0, 0x0f1318, 1.2));
601
+ const key = new THREE.DirectionalLight(0xe8a33d, 1.9);
602
+ key.position.set(4, 6, 5);
603
+ scene.add(key);
604
+
605
+ const grid = new THREE.GridHelper(8, 32, 0x2a5e5c, 0x1e2530);
606
+ grid.material.transparent = true;
607
+ grid.material.opacity = 0.75;
608
+ scene.add(grid);
609
+
610
+ const meshes = (analysis.parts || []).slice(0, 6).map((part, index) => {
611
+ const mesh = buildPartMesh(part, index);
612
+ const position = part.geometry && part.geometry.position || [0, 0, 0];
613
+ mesh.position.set(position[0], position[1], position[2]);
614
+ if (part.geometry && Array.isArray(part.geometry.rotation)) {
615
+ mesh.rotation.fromArray(part.geometry.rotation);
616
+ }
617
+ mesh.userData.basePosition = mesh.position.clone();
618
+ mesh.userData.baseRotation = mesh.rotation.clone();
619
+ mesh.userData.part = part;
620
+ mesh.scale.setScalar(0.001);
621
+ scene.add(mesh);
622
+ return mesh;
623
+ });
624
+
625
+ const clock = new THREE.Clock();
626
+ let frameId = 0;
627
+ let disposed = false;
628
+ fallbackRuntime = {
629
+ playing: true,
630
+ cleanup() {
631
+ disposed = true;
632
+ cancelAnimationFrame(frameId);
633
+ renderer.dispose();
634
+ }
635
+ };
636
+
637
+ function resize() {
638
+ const rect = viewport.getBoundingClientRect();
639
+ camera.aspect = Math.max(1, rect.width) / Math.max(1, rect.height);
640
+ camera.updateProjectionMatrix();
641
+ renderer.setSize(Math.max(1, rect.width), Math.max(1, rect.height));
642
+ }
643
+
644
+ function animate() {
645
+ if (disposed) return;
646
+ frameId = requestAnimationFrame(animate);
647
+ const elapsed = clock.getElapsedTime();
648
+ for (const [index, mesh] of meshes.entries()) {
649
+ revealMesh(mesh, elapsed, index);
650
+ if (fallbackRuntime.playing) applyMotion(mesh, elapsed);
651
+ }
652
+ if (controls) controls.update();
653
+ renderer.render(scene, camera);
654
+ updateLabels(camera, labelLayer, meshes);
655
+ }
656
+
657
+ window.addEventListener("resize", resize);
658
+ const originalCleanup = fallbackRuntime.cleanup;
659
+ fallbackRuntime.cleanup = () => {
660
+ window.removeEventListener("resize", resize);
661
+ originalCleanup();
662
+ };
663
+
664
+ resize();
665
+ animate();
666
+ revealScan();
667
+ setStatus("CUTAWAY READY");
668
+ }
669
+
670
+ function buildPartMesh(part, index) {
671
+ const geometryData = part.geometry || {};
672
+ const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
673
+ let geometry;
674
+ if (geometryData.shape === "cylinder") {
675
+ geometry = new THREE.CylinderGeometry(size[0] / 2, size[2] / 2, size[1], 48);
676
+ } else if (geometryData.shape === "sphere") {
677
+ geometry = new THREE.SphereGeometry(Math.max(size[0], size[1], size[2]) / 2, 36, 18);
678
+ } else if (geometryData.shape === "rod") {
679
+ geometry = new THREE.CylinderGeometry(Math.max(size[0], size[1]) / 2, Math.max(size[0], size[1]) / 2, size[2], 24);
680
+ geometry.rotateX(Math.PI / 2);
681
+ } else if (geometryData.shape === "gear") {
682
+ geometry = gearGeometry(Math.max(size[0], size[2]) / 2, size[1], geometryData.teeth || 18);
683
+ } else {
684
+ geometry = new THREE.BoxGeometry(size[0], size[1], size[2]);
685
+ }
686
+
687
+ const color = colorFor(geometryData.color, index);
688
+ const material = new THREE.MeshStandardMaterial({
689
+ color,
690
+ metalness: 0.48,
691
+ roughness: 0.34,
692
+ emissive: color,
693
+ emissiveIntensity: 0.04
694
+ });
695
+ return new THREE.Mesh(geometry, material);
696
+ }
697
+
698
+ function gearGeometry(radius, depth, teeth) {
699
+ const shape = new THREE.Shape();
700
+ const steps = teeth * 2;
701
+ for (let i = 0; i <= steps; i += 1) {
702
+ const angle = (i / steps) * Math.PI * 2;
703
+ const r = radius * (i % 2 === 0 ? 1 : 0.84);
704
+ const x = Math.cos(angle) * r;
705
+ const y = Math.sin(angle) * r;
706
+ if (i === 0) shape.moveTo(x, y);
707
+ else shape.lineTo(x, y);
708
+ }
709
+ const hole = new THREE.Path();
710
+ hole.absarc(0, 0, radius * 0.26, 0, Math.PI * 2, true);
711
+ shape.holes.push(hole);
712
+ const geometry = new THREE.ExtrudeGeometry(shape, { depth, bevelEnabled: false });
713
+ geometry.center();
714
+ geometry.rotateX(Math.PI / 2);
715
+ return geometry;
716
+ }
717
+
718
+ function revealMesh(mesh, elapsed, index) {
719
+ const local = Math.max(0, Math.min(1, (elapsed - index * 0.1) / 0.55));
720
+ const eased = 1 - Math.pow(1 - local, 3);
721
+ mesh.scale.setScalar(eased);
722
+ }
723
+
724
+ function applyMotion(mesh, elapsed) {
725
+ const motion = mesh.userData.part.motion || { type: "static" };
726
+ const speed = Number(motion.speed || 1);
727
+ const phase = Number(motion.phase || 0);
728
+ const axis = new THREE.Vector3(...(motion.axis || [0, 1, 0])).normalize();
729
+ mesh.position.copy(mesh.userData.basePosition);
730
+ mesh.rotation.copy(mesh.userData.baseRotation);
731
+ if (motion.type === "rotate") {
732
+ mesh.rotateOnAxis(axis, elapsed * speed + phase);
733
+ } else if (motion.type === "oscillate") {
734
+ mesh.rotateOnAxis(axis, Math.sin(elapsed * speed + phase) * Number(motion.amplitude || 0.25));
735
+ } else if (motion.type === "translate") {
736
+ const range = motion.range || [-0.25, 0.25];
737
+ const offset = range[0] + (range[1] - range[0]) * ((Math.sin(elapsed * speed + phase) + 1) / 2);
738
+ mesh.position.add(axis.multiplyScalar(offset));
739
+ }
740
+ }
741
+
742
+ function updateLabels(camera, labelLayer, meshes) {
743
+ const rect = viewport.getBoundingClientRect();
744
+ labelLayer.replaceChildren();
745
+ for (const mesh of meshes) {
746
+ const projected = mesh.position.clone().project(camera);
747
+ if (projected.z < -1 || projected.z > 1) continue;
748
+ const label = document.createElement("div");
749
+ label.className = "scene-label";
750
+ label.style.left = Math.min(rect.width - 72, Math.max(72, (projected.x * 0.5 + 0.5) * rect.width)) + "px";
751
+ label.style.top = Math.min(rect.height - 36, Math.max(56, (-projected.y * 0.5 + 0.5) * rect.height)) + "px";
752
+ label.textContent = mesh.userData.part.name || mesh.userData.part.id || "part";
753
+ labelLayer.appendChild(label);
754
+ }
755
+ }
756
+
757
+ function colorFor(name, index) {
758
+ const key = String(name || "").toLowerCase();
759
+ if (key.includes("amber") || key.includes("orange")) return 0xe8a33d;
760
+ if (key.includes("cyan")) return 0x5fd4d0;
761
+ if (key.includes("steel")) return 0x9aa4a6;
762
+ return [0x5fd4d0, 0xe8a33d, 0xc8c0ac, 0x80b8ff, 0xf07f5a, 0xa7d676][index % 6];
763
+ }
764
+
765
+ async function postJson(url, payload) {
766
+ const response = await fetch(url, {
767
+ method: "POST",
768
+ headers: { "Content-Type": "application/json" },
769
+ body: JSON.stringify(payload)
770
+ });
771
+ if (!response.ok) {
772
+ let message = response.statusText || "Request failed";
773
+ try {
774
+ const data = await response.json();
775
+ message = data.detail || data.error || message;
776
+ } catch (_) {}
777
+ throw new Error(message);
778
+ }
779
+ return response.json();
780
+ }
781
+
782
+ function fileToDataUrl(file) {
783
+ return new Promise((resolve, reject) => {
784
+ const reader = new FileReader();
785
+ reader.onload = () => resolve(String(reader.result || ""));
786
+ reader.onerror = () => reject(reader.error || new Error("File read failed"));
787
+ reader.readAsDataURL(file);
788
+ });
789
+ }
790
+
791
+ function setBusy(active) {
792
+ progress.classList.toggle("active", active);
793
+ uploadButton.disabled = active;
794
+ }
795
+
796
+ function setStatus(message, error) {
797
+ statusEl.textContent = String(message || "STANDBY").toUpperCase();
798
+ statusEl.classList.toggle("error", Boolean(error));
799
+ }
800
+
801
+ function revealScan() {
802
+ scanLine.classList.remove("active");
803
+ void scanLine.offsetWidth;
804
+ scanLine.classList.add("active");
805
+ }
806
+ </script>
807
+ </body>
808
+ </html>
modal_app.py CHANGED
@@ -16,10 +16,10 @@ import secrets as token_secrets
16
  import modal
17
  from fastapi import Header, HTTPException
18
 
19
- from snap2sim.model_io import coerce_analysis_response, parse_analysis_response, parse_html_response
20
- from snap2sim.prompts import build_threejs_prompt, build_vision_prompt
 
21
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
22
- from snap2sim.three_scene import build_threejs_html
23
 
24
 
25
  DEFAULT_MODEL_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
@@ -81,14 +81,14 @@ def check_remote_imports() -> dict[str, Any]:
81
  """Lightweight Modal check that local project modules are packaged."""
82
  import snap2sim.model_io
83
  import snap2sim.schema
84
- import snap2sim.three_scene
85
 
86
  return {
87
  "ok": True,
88
  "modules": [
89
  snap2sim.model_io.__name__,
90
  snap2sim.schema.__name__,
91
- snap2sim.three_scene.__name__,
92
  ],
93
  }
94
 
@@ -413,7 +413,7 @@ def runtime_probe(authorization: str = Header(default="")) -> dict[str, Any]:
413
  else "runtime selected"
414
  ),
415
  "verified_endpoint": "analyze_image_llamacpp",
416
- "recommended_generate_endpoint": "generate_threejs",
417
  }
418
 
419
 
@@ -447,23 +447,23 @@ def analyze_image_llamacpp(payload: dict[str, Any], authorization: str = Header(
447
 
448
  @app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
449
  @modal.fastapi_endpoint(method="POST")
450
- def generate_threejs(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
451
  require_authorization(authorization)
452
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
453
- _prompt = build_threejs_prompt(analysis)
454
  if runtime_config()["runtime_mode"] != "placeholder":
455
  raise NotImplementedError(
456
  "Nemotron runtime is selected but not wired yet. Keep placeholder "
457
  "mode active until the llama.cpp/vLLM adapter is implemented."
458
  )
459
- return {"html": build_threejs_html(analysis)}
460
 
461
 
462
  @app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
463
  @modal.fastapi_endpoint(method="POST")
464
- def generate_threejs_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
465
- """Experimental GPU endpoint for llama.cpp Three.js code generation."""
466
  require_authorization(authorization)
467
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
468
- response = run_llamacpp_prompt(build_threejs_prompt(analysis), max_tokens=4096)
469
- return {"html": parse_html_response(response)}
 
16
  import modal
17
  from fastapi import Header, HTTPException
18
 
19
+ from snap2sim.aframe_scene import build_aframe_scene
20
+ from snap2sim.model_io import coerce_analysis_response, parse_analysis_response, parse_scene_response
21
+ from snap2sim.prompts import build_scene_prompt, build_vision_prompt
22
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
 
23
 
24
 
25
  DEFAULT_MODEL_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
 
81
  """Lightweight Modal check that local project modules are packaged."""
82
  import snap2sim.model_io
83
  import snap2sim.schema
84
+ import snap2sim.aframe_scene
85
 
86
  return {
87
  "ok": True,
88
  "modules": [
89
  snap2sim.model_io.__name__,
90
  snap2sim.schema.__name__,
91
+ snap2sim.aframe_scene.__name__,
92
  ],
93
  }
94
 
 
413
  else "runtime selected"
414
  ),
415
  "verified_endpoint": "analyze_image_llamacpp",
416
+ "recommended_generate_endpoint": "generate_scene",
417
  }
418
 
419
 
 
447
 
448
  @app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
449
  @modal.fastapi_endpoint(method="POST")
450
+ def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
451
  require_authorization(authorization)
452
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
453
+ _prompt = build_scene_prompt(analysis)
454
  if runtime_config()["runtime_mode"] != "placeholder":
455
  raise NotImplementedError(
456
  "Nemotron runtime is selected but not wired yet. Keep placeholder "
457
  "mode active until the llama.cpp/vLLM adapter is implemented."
458
  )
459
+ return {"html": build_aframe_scene(analysis)}
460
 
461
 
462
  @app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
463
  @modal.fastapi_endpoint(method="POST")
464
+ def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
465
+ """Experimental GPU endpoint for llama.cpp A-Frame scene generation."""
466
  require_authorization(authorization)
467
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
468
+ response = run_llamacpp_prompt(build_scene_prompt(analysis), max_tokens=4096)
469
+ return {"html": parse_scene_response(response)}
requirements.txt CHANGED
@@ -1,4 +1,4 @@
1
- gradio>=4.44
2
  huggingface_hub[hf_xet]>=0.36
3
  requests>=2.32
4
  pillow>=10.4
 
1
+ gradio>=6.0.1
2
  huggingface_hub[hf_xet]>=0.36
3
  requests>=2.32
4
  pillow>=10.4
snap2sim/aframe_scene.py ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Deterministic A-Frame scene generation from a validated mechanism payload."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import html
6
+ from typing import Any
7
+
8
+ from snap2sim.schema import validate_analysis
9
+
10
+
11
+ def build_aframe_scene(analysis: dict[str, Any]) -> str:
12
+ """Return a declarative A-Frame scene for the browser to inject."""
13
+ valid_analysis = validate_analysis(analysis)
14
+ title = html.escape(str(valid_analysis.get("component", "mechanism")).upper(), quote=True)
15
+ summary = html.escape(str(valid_analysis.get("summary", "")), quote=True)
16
+ entities = "\n".join(_part_entity(part, index) for index, part in enumerate(valid_analysis["parts"][:6]))
17
+ return f"""<a-scene embedded vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false" background="color: #0F1318" renderer="colorManagement: true">
18
+ <a-sky color="#0F1318"></a-sky>
19
+ <a-entity light="type: ambient; intensity: 0.55; color: #5FD4D0"></a-entity>
20
+ <a-entity light="type: directional; intensity: 1.35; color: #E8A33D" position="3 5 4"></a-entity>
21
+ <a-entity position="0 1.1 5.6">
22
+ <a-camera look-controls wasd-controls="enabled: false"></a-camera>
23
+ </a-entity>
24
+ <a-plane color="#111820" opacity="0.72" rotation="-90 0 0" position="0 -1.05 -0.2" width="8" height="8"></a-plane>
25
+ <a-text value="{title}" position="-2.8 2.35 -2.6" color="#C8C0AC" width="4.8" align="left"></a-text>
26
+ <a-text value="{summary}" position="-2.8 2.05 -2.6" color="#6B7280" width="5.2" align="left"></a-text>
27
+ {entities}
28
+ </a-scene>"""
29
+
30
+
31
+ def _part_entity(part: dict[str, Any], index: int) -> str:
32
+ geometry = part["geometry"]
33
+ shape = geometry["shape"]
34
+ size = geometry["size"]
35
+ position = _vec(geometry["position"])
36
+ rotation = _rotation(geometry.get("rotation", [0, 0, 0]))
37
+ color = _color(geometry.get("color"), index)
38
+ label = html.escape(str(part.get("name", part.get("id", "part"))), quote=True)
39
+ animation = _animation(part)
40
+
41
+ if shape in {"cylinder", "gear"}:
42
+ radius = max(float(size[0]), float(size[2])) / 2
43
+ primitive = (
44
+ f'<a-cylinder radius="{radius:.3f}" height="{float(size[1]):.3f}" '
45
+ f'segments-radial="{int(geometry.get("teeth", 32)) if shape == "gear" else 48}"'
46
+ )
47
+ elif shape == "sphere":
48
+ primitive = f'<a-sphere radius="{max(float(item) for item in size) / 2:.3f}"'
49
+ elif shape == "rod":
50
+ primitive = f'<a-cylinder radius="{max(float(size[0]), float(size[1])) / 2:.3f}" height="{float(size[2]):.3f}"'
51
+ rotation = _rotation([1.5708, 0, 0])
52
+ else:
53
+ primitive = f'<a-box width="{float(size[0]):.3f}" height="{float(size[1]):.3f}" depth="{float(size[2]):.3f}"'
54
+
55
+ return f""" <a-entity position="{position}">
56
+ {primitive} material="color: {color}; metalness: 0.45; roughness: 0.38" rotation="{rotation}" {animation}></a-cylinder>
57
+ <a-text value="{label}" position="0 {float(size[1]) + 0.28:.3f} 0" color="#5FD4D0" width="2.4" align="center"></a-text>
58
+ </a-entity>""".replace("</a-cylinder>", _closing_tag(shape))
59
+
60
+
61
+ def _closing_tag(shape: str) -> str:
62
+ if shape == "sphere":
63
+ return "</a-sphere>"
64
+ if shape == "box":
65
+ return "</a-box>"
66
+ return "</a-cylinder>"
67
+
68
+
69
+ def _animation(part: dict[str, Any]) -> str:
70
+ motion = part.get("motion") or {"type": "static"}
71
+ motion_type = motion.get("type")
72
+ if motion_type == "static":
73
+ return ""
74
+
75
+ speed = max(0.1, abs(float(motion.get("speed", 1))))
76
+ duration = int(max(700, min(9000, 3600 / speed)))
77
+ axis = motion.get("axis") if isinstance(motion.get("axis"), list) else [0, 1, 0]
78
+
79
+ if motion_type == "translate":
80
+ move_range = motion.get("range") if isinstance(motion.get("range"), list) else [-0.25, 0.25]
81
+ target = [float(axis[i]) * float(move_range[-1]) for i in range(3)]
82
+ return (
83
+ f'animation="property: position; to: {_vec(target)}; dir: alternate; '
84
+ f'loop: true; dur: {duration}; easing: easeInOutSine"'
85
+ )
86
+
87
+ rotation = [0, 0, 0]
88
+ if abs(float(axis[0])) >= max(abs(float(axis[1])), abs(float(axis[2]))):
89
+ rotation[0] = 360
90
+ elif abs(float(axis[2])) >= abs(float(axis[1])):
91
+ rotation[2] = 360
92
+ else:
93
+ rotation[1] = 360
94
+ if motion_type == "oscillate":
95
+ amplitude = int(float(motion.get("amplitude", 0.25)) * 90)
96
+ rotation = [value and amplitude for value in rotation]
97
+ return (
98
+ f'animation="property: rotation; to: {_vec(rotation)}; dir: alternate; '
99
+ f'loop: true; dur: {duration}; easing: easeInOutSine"'
100
+ )
101
+ return f'animation="property: rotation; to: {_vec(rotation)}; loop: true; dur: {duration}; easing: linear"'
102
+
103
+
104
+ def _vec(values: list[Any]) -> str:
105
+ return " ".join(f"{float(value):.3f}" for value in values)
106
+
107
+
108
+ def _rotation(values: list[Any]) -> str:
109
+ return " ".join(f"{float(value) * 57.2958:.2f}" for value in values)
110
+
111
+
112
+ def _color(value: Any, index: int) -> str:
113
+ key = str(value or "").lower()
114
+ if "amber" in key or "orange" in key:
115
+ return "#E8A33D"
116
+ if "cyan" in key:
117
+ return "#5FD4D0"
118
+ if "steel" in key:
119
+ return "#9AA4A6"
120
+ return ["#5FD4D0", "#E8A33D", "#C8C0AC", "#80B8FF", "#F07F5A", "#A7D676"][index % 6]
snap2sim/backend.py CHANGED
@@ -11,8 +11,8 @@ from typing import Any
11
  import requests
12
  from PIL import Image
13
 
 
14
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
15
- from snap2sim.three_scene import build_threejs_html
16
 
17
 
18
  @dataclass(frozen=True)
@@ -45,7 +45,7 @@ class InferenceClient:
45
 
46
  return validate_analysis(dict(EXAMPLE_ANALYSIS))
47
 
48
- def generate_threejs(self, analysis: dict[str, Any]) -> str:
49
  valid_analysis = validate_analysis(analysis)
50
  if self.settings.backend == "modal":
51
  response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
@@ -54,7 +54,7 @@ class InferenceClient:
54
  raise RuntimeError("Modal response did not include generated HTML.")
55
  return str(html)
56
 
57
- return build_threejs_html(valid_analysis)
58
 
59
  def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
60
  if not url:
 
11
  import requests
12
  from PIL import Image
13
 
14
+ from snap2sim.aframe_scene import build_aframe_scene
15
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
 
16
 
17
 
18
  @dataclass(frozen=True)
 
45
 
46
  return validate_analysis(dict(EXAMPLE_ANALYSIS))
47
 
48
+ def generate_scene(self, analysis: dict[str, Any]) -> str:
49
  valid_analysis = validate_analysis(analysis)
50
  if self.settings.backend == "modal":
51
  response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
 
54
  raise RuntimeError("Modal response did not include generated HTML.")
55
  return str(html)
56
 
57
+ return build_aframe_scene(valid_analysis)
58
 
59
  def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
60
  if not url:
snap2sim/fallback_scene.py DELETED
@@ -1,72 +0,0 @@
1
- """Themed fallback visualization used when generated Three.js is unavailable."""
2
-
3
- from __future__ import annotations
4
-
5
-
6
- def build_fallback_html(component: str) -> str:
7
- title = component.upper()
8
- return f"""
9
- <div class="fallback-cutaway" role="img" aria-label="Fallback mechanism animation for {component}">
10
- <div class="scan"></div>
11
- <svg viewBox="0 0 760 430" xmlns="http://www.w3.org/2000/svg">
12
- <defs>
13
- <pattern id="grid" width="24" height="24" patternUnits="userSpaceOnUse">
14
- <path d="M 24 0 L 0 0 0 24" fill="none" stroke="rgba(95,212,208,.14)" stroke-width="1"/>
15
- </pattern>
16
- </defs>
17
- <rect width="760" height="430" fill="#14181F"/>
18
- <rect width="760" height="430" fill="url(#grid)"/>
19
- <g transform="translate(380 220)">
20
- <circle r="104" fill="none" stroke="#5FD4D0" stroke-width="2" opacity=".8"/>
21
- <circle class="fallback-gear" r="72" fill="none" stroke="#E8A33D" stroke-width="18"
22
- stroke-dasharray="18 10"/>
23
- <rect class="fallback-pawl" x="72" y="-16" width="132" height="32" rx="2"
24
- fill="#E8A33D" opacity=".9"/>
25
- <line x1="108" y1="-54" x2="215" y2="-112" stroke="#5FD4D0" stroke-width="1"/>
26
- <text x="224" y="-116" fill="#5FD4D0" font-family="monospace" font-size="15">LOCKING PAWL</text>
27
- <line x1="-54" y1="52" x2="-198" y2="122" stroke="#5FD4D0" stroke-width="1"/>
28
- <text x="-342" y="132" fill="#5FD4D0" font-family="monospace" font-size="15">DRIVE GEAR</text>
29
- </g>
30
- <text x="32" y="44" fill="#D9D3C7" font-family="monospace" font-size="18">{title}</text>
31
- <text x="32" y="394" fill="#E8A33D" font-family="monospace" font-size="13">2D FALLBACK: GENERATED 3D SCENE UNAVAILABLE</text>
32
- </svg>
33
- </div>
34
- <style>
35
- .fallback-cutaway {{
36
- position: relative;
37
- overflow: hidden;
38
- border: 1px solid rgba(95, 212, 208, .28);
39
- background: #14181F;
40
- min-height: 430px;
41
- }}
42
- .fallback-cutaway svg {{
43
- display: block;
44
- width: 100%;
45
- height: min(58vh, 560px);
46
- }}
47
- .fallback-gear {{
48
- transform-origin: center;
49
- animation: fallback-spin 3.8s linear infinite;
50
- }}
51
- .fallback-pawl {{
52
- transform-origin: 74px 0;
53
- animation: fallback-pawl 1.05s steps(2, end) infinite;
54
- }}
55
- .scan {{
56
- position: absolute;
57
- inset: 0;
58
- background: linear-gradient(180deg, transparent, rgba(95, 212, 208, .16), transparent);
59
- height: 26%;
60
- animation: fallback-scan 2.6s ease-in-out infinite;
61
- pointer-events: none;
62
- }}
63
- @keyframes fallback-spin {{ to {{ transform: rotate(360deg); }} }}
64
- @keyframes fallback-pawl {{ 50% {{ transform: rotate(-10deg); }} }}
65
- @keyframes fallback-scan {{
66
- 0% {{ transform: translateY(-120%); opacity: 0; }}
67
- 20%, 80% {{ opacity: 1; }}
68
- 100% {{ transform: translateY(430%); opacity: 0; }}
69
- }}
70
- </style>
71
- """
72
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
snap2sim/model_io.py CHANGED
@@ -46,26 +46,21 @@ def coerce_analysis_response(text: str) -> dict[str, Any]:
46
  return validate_analysis(_generic_analysis(fallback_component))
47
 
48
 
49
- def parse_html_response(text: str) -> str:
50
- """Extract a complete HTML document from a model response."""
51
  raw = _strip_fences(text).strip()
52
- html_start = _find_html_start(raw)
53
- html_end = raw.lower().rfind("</html>")
54
- if html_start < 0 or html_end < 0:
55
- raise ValueError("Model response did not contain a complete HTML document.")
56
- return raw[html_start : html_end + len("</html>")]
 
57
 
58
 
59
  def _strip_fences(text: str) -> str:
60
  return _FENCE_RE.sub("", text.strip()).strip()
61
 
62
 
63
- def _find_html_start(text: str) -> int:
64
- lowered = text.lower()
65
- starts = [index for index in [lowered.find("<!doctype"), lowered.find("<html")] if index >= 0]
66
- return min(starts) if starts else -1
67
-
68
-
69
  def _json_object_candidates(text: str) -> list[tuple[int, str]]:
70
  candidates = []
71
  for index, char in enumerate(text):
 
46
  return validate_analysis(_generic_analysis(fallback_component))
47
 
48
 
49
+ def parse_scene_response(text: str) -> str:
50
+ """Extract a complete A-Frame scene block from a model response."""
51
  raw = _strip_fences(text).strip()
52
+ lowered = raw.lower()
53
+ scene_start = lowered.find("<a-scene")
54
+ scene_end = lowered.rfind("</a-scene>")
55
+ if scene_start < 0 or scene_end < 0:
56
+ raise ValueError("Model response did not contain a complete <a-scene> block.")
57
+ return raw[scene_start : scene_end + len("</a-scene>")]
58
 
59
 
60
  def _strip_fences(text: str) -> str:
61
  return _FENCE_RE.sub("", text.strip()).strip()
62
 
63
 
 
 
 
 
 
 
64
  def _json_object_candidates(text: str) -> list[tuple[int, str]]:
65
  candidates = []
66
  for index, char in enumerate(text):
snap2sim/prompts.py CHANGED
@@ -18,7 +18,7 @@ def build_vision_prompt() -> str:
18
  "Analyze the uploaded hardware component as a cutaway mechanism. "
19
  "Answer with only one JSON object. Do not include markdown. Do not "
20
  "include a reasoning trace or <think> tags. Keep the payload compact and physically "
21
- "plausible for primitive Three.js rendering.\n\n"
22
  "Required top-level keys: component, confidence, summary, trigger, "
23
  "motion_sequence, parts.\n"
24
  "Each part requires: id, name, role, geometry, motion.\n"
@@ -48,17 +48,23 @@ def build_vision_prompt() -> str:
48
  )
49
 
50
 
51
- THREEJS_SYSTEM_PROMPT = """You generate self-contained Three.js cutaway scenes.
52
- Return a complete HTML document with inline CSS and JavaScript only. Use
53
- primitive geometry, labels, OrbitControls, a play/pause control, and one
54
- power-on reveal animation. Do not include markdown fences."""
55
 
56
 
57
- def build_threejs_prompt(analysis: dict[str, Any]) -> str:
58
  return (
59
- "Build a technical cutaway / field manual Three.js animation for this "
60
- "mechanism analysis. Use a deep navy blueprint surface, warm amber "
61
- "annotations, cool cyan motion paths, and monospace labels. Include "
62
- "a robust fallback message inside the HTML if WebGL fails.\n\n"
 
 
 
 
 
 
63
  f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
64
  )
 
18
  "Analyze the uploaded hardware component as a cutaway mechanism. "
19
  "Answer with only one JSON object. Do not include markdown. Do not "
20
  "include a reasoning trace or <think> tags. Keep the payload compact and physically "
21
+ "plausible for primitive A-Frame or Three.js rendering.\n\n"
22
  "Required top-level keys: component, confidence, summary, trigger, "
23
  "motion_sequence, parts.\n"
24
  "Each part requires: id, name, role, geometry, motion.\n"
 
48
  )
49
 
50
 
51
+ AFRAME_SYSTEM_PROMPT = """You generate declarative A-Frame cutaway scenes.
52
+ Return only one <a-scene>...</a-scene> block. Use primitive A-Frame entities,
53
+ short labels, and animation attributes. Do not include scripts, markdown fences,
54
+ explanations, or a complete HTML document."""
55
 
56
 
57
+ def build_scene_prompt(analysis: dict[str, Any]) -> str:
58
  return (
59
+ "Build a technical cutaway / field manual A-Frame animation for this "
60
+ "mechanism analysis. Return only the <a-scene>...</a-scene> block. "
61
+ "Use 3 to 6 visible primitive parts maximum, selected from <a-box>, "
62
+ "<a-cylinder>, <a-sphere>, <a-torus>, <a-cone>, and <a-entity>. "
63
+ "Include <a-sky color=\"#0F1318\">, a camera, lights, short <a-text> "
64
+ "labels, warm amber annotations, and cool cyan moving parts. Use "
65
+ "A-Frame animation attributes for motion, for example "
66
+ "animation=\"property: rotation; to: 0 360 0; loop: true; dur: 2000; "
67
+ "easing: linear\". Do not include scripts, markdown, explanations, "
68
+ "or a complete HTML document.\n\n"
69
  f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
70
  )
snap2sim/three_scene.py DELETED
@@ -1,397 +0,0 @@
1
- """Deterministic Three.js scene generation from a validated mechanism payload."""
2
-
3
- from __future__ import annotations
4
-
5
- import html
6
- import json
7
- from typing import Any
8
-
9
- from snap2sim.schema import validate_analysis
10
-
11
-
12
- def build_threejs_html(analysis: dict[str, Any]) -> str:
13
- """Return a sandboxed iframe containing a self-contained scene document."""
14
- valid_analysis = validate_analysis(analysis)
15
- document = _build_scene_document(valid_analysis)
16
- escaped = html.escape(document, quote=True)
17
- title = html.escape(str(valid_analysis.get("component", "mechanism")), quote=True)
18
- return (
19
- f'<iframe class="snap-scene-frame" title="Animated cutaway of {title}" '
20
- 'sandbox="allow-scripts allow-same-origin" '
21
- f'srcdoc="{escaped}"></iframe>'
22
- )
23
-
24
-
25
- def _build_scene_document(analysis: dict[str, Any]) -> str:
26
- analysis_json = json.dumps(analysis)
27
- component = html.escape(str(analysis.get("component", "mechanism")).upper())
28
- return f"""<!doctype html>
29
- <html lang="en">
30
- <head>
31
- <meta charset="utf-8">
32
- <meta name="viewport" content="width=device-width, initial-scale=1">
33
- <style>
34
- :root {{
35
- --bg: #14181F;
36
- --panel: #1B222B;
37
- --amber: #E8A33D;
38
- --cyan: #5FD4D0;
39
- --text: #D9D3C7;
40
- --muted: #8F968F;
41
- }}
42
- * {{ box-sizing: border-box; }}
43
- html, body {{
44
- width: 100%;
45
- height: 100%;
46
- margin: 0;
47
- overflow: hidden;
48
- color: var(--text);
49
- background:
50
- linear-gradient(rgba(95, 212, 208, .07) 1px, transparent 1px),
51
- linear-gradient(90deg, rgba(95, 212, 208, .07) 1px, transparent 1px),
52
- var(--bg);
53
- background-size: 28px 28px;
54
- font-family: "IBM Plex Mono", "Consolas", monospace;
55
- }}
56
- #scene {{
57
- position: fixed;
58
- inset: 0;
59
- }}
60
- .hud {{
61
- position: fixed;
62
- left: 18px;
63
- right: 18px;
64
- top: 16px;
65
- z-index: 3;
66
- display: flex;
67
- align-items: flex-start;
68
- justify-content: space-between;
69
- gap: 16px;
70
- pointer-events: none;
71
- }}
72
- .title {{
73
- max-width: min(680px, 70vw);
74
- }}
75
- .title b {{
76
- display: block;
77
- color: var(--text);
78
- font-size: 14px;
79
- letter-spacing: 0;
80
- text-transform: uppercase;
81
- }}
82
- .title span {{
83
- display: block;
84
- margin-top: 6px;
85
- color: var(--muted);
86
- font-size: 12px;
87
- line-height: 1.45;
88
- }}
89
- #toggle {{
90
- pointer-events: auto;
91
- min-width: 92px;
92
- border: 1px solid rgba(232, 163, 61, .8);
93
- border-radius: 2px;
94
- padding: 8px 12px;
95
- color: #16130E;
96
- background: linear-gradient(180deg, #F0B65B, #C97923);
97
- font: 700 12px "IBM Plex Mono", monospace;
98
- text-transform: uppercase;
99
- cursor: pointer;
100
- }}
101
- #labels {{
102
- position: fixed;
103
- inset: 0;
104
- z-index: 2;
105
- pointer-events: none;
106
- }}
107
- .tag {{
108
- position: absolute;
109
- max-width: min(190px, 42vw);
110
- transform: translate(-50%, -50%);
111
- border-left: 1px solid var(--cyan);
112
- padding: 4px 0 4px 8px;
113
- color: var(--cyan);
114
- background: rgba(20, 24, 31, .76);
115
- font-size: 11px;
116
- line-height: 1.25;
117
- text-transform: uppercase;
118
- white-space: normal;
119
- }}
120
- .tag::before {{
121
- content: "";
122
- position: absolute;
123
- left: -42px;
124
- top: 50%;
125
- width: 40px;
126
- border-top: 1px solid rgba(95, 212, 208, .7);
127
- }}
128
- .scan {{
129
- position: fixed;
130
- inset: -30% 0 auto;
131
- height: 28%;
132
- z-index: 4;
133
- background: linear-gradient(180deg, transparent, rgba(95, 212, 208, .18), transparent);
134
- animation: scan 2.2s ease-in-out 1;
135
- pointer-events: none;
136
- }}
137
- .failure {{
138
- position: fixed;
139
- left: 50%;
140
- top: 50%;
141
- width: min(520px, calc(100vw - 40px));
142
- transform: translate(-50%, -50%);
143
- border: 1px solid rgba(232, 163, 61, .62);
144
- padding: 18px;
145
- color: var(--amber);
146
- background: rgba(20, 24, 31, .92);
147
- font-size: 13px;
148
- line-height: 1.55;
149
- }}
150
- .hidden {{ display: none; }}
151
- @keyframes scan {{
152
- 0% {{ transform: translateY(-100%); opacity: 0; }}
153
- 20%, 75% {{ opacity: 1; }}
154
- 100% {{ transform: translateY(430%); opacity: 0; }}
155
- }}
156
- </style>
157
- </head>
158
- <body>
159
- <div id="scene"></div>
160
- <div id="labels"></div>
161
- <div class="scan"></div>
162
- <div class="hud">
163
- <div class="title">
164
- <b>{component}</b>
165
- <span>Primitive cutaway generated from the validated mechanism schema.</span>
166
- </div>
167
- <button id="toggle" type="button">Pause</button>
168
- </div>
169
- <div id="failure" class="failure hidden">WebGL scene failed to initialize.</div>
170
- <script type="module">
171
- const analysis = {analysis_json};
172
- const failure = document.getElementById("failure");
173
-
174
- function fail(error) {{
175
- failure.classList.remove("hidden");
176
- failure.textContent = "3D renderer unavailable: " + (error && error.message ? error.message : error);
177
- }}
178
-
179
- try {{
180
- const [threeModule, controlsModule] = await Promise.all([
181
- import("https://esm.sh/three@0.166.1"),
182
- import("https://esm.sh/three@0.166.1/examples/jsm/controls/OrbitControls.js?deps=three@0.166.1")
183
- ]);
184
- initialize(threeModule, controlsModule.OrbitControls);
185
- }} catch (error) {{
186
- fail(error);
187
- }}
188
-
189
- function initialize(THREE, OrbitControls) {{
190
- const root = document.getElementById("scene");
191
- const labelsRoot = document.getElementById("labels");
192
- const renderer = new THREE.WebGLRenderer({{ antialias: true, alpha: true, preserveDrawingBuffer: true }});
193
- renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
194
- renderer.setSize(window.innerWidth, window.innerHeight);
195
- renderer.outputColorSpace = THREE.SRGBColorSpace;
196
- root.appendChild(renderer.domElement);
197
-
198
- const scene = new THREE.Scene();
199
- scene.fog = new THREE.Fog(0x14181f, 9, 22);
200
-
201
- const camera = new THREE.PerspectiveCamera(42, window.innerWidth / window.innerHeight, 0.1, 100);
202
- camera.position.set(5.8, 4.6, 6.8);
203
-
204
- const controls = new OrbitControls(camera, renderer.domElement);
205
- controls.enableDamping = true;
206
- controls.target.set(0, 0.15, 0);
207
-
208
- scene.add(new THREE.HemisphereLight(0x5fd4d0, 0x14181f, 1.35));
209
- const key = new THREE.DirectionalLight(0xffc36a, 2.15);
210
- key.position.set(4, 7, 5);
211
- scene.add(key);
212
-
213
- const grid = new THREE.GridHelper(8, 32, 0x5fd4d0, 0x24505a);
214
- grid.material.transparent = true;
215
- grid.material.opacity = 0.42;
216
- scene.add(grid);
217
-
218
- const motionLineMaterial = new THREE.LineBasicMaterial({{ color: 0x5fd4d0, transparent: true, opacity: 0.56 }});
219
- const parts = [];
220
-
221
- for (const [index, part] of analysis.parts.entries()) {{
222
- const mesh = buildMesh(THREE, part, index);
223
- mesh.position.fromArray(part.geometry.position);
224
- if (Array.isArray(part.geometry.rotation)) mesh.rotation.fromArray(part.geometry.rotation);
225
- mesh.userData.basePosition = mesh.position.clone();
226
- mesh.userData.baseRotation = mesh.rotation.clone();
227
- mesh.userData.part = part;
228
- mesh.scale.setScalar(0.001);
229
- scene.add(mesh);
230
- parts.push(mesh);
231
-
232
- if (part.motion && part.motion.type !== "static") {{
233
- const curve = motionCurve(THREE, part);
234
- const line = new THREE.Line(
235
- new THREE.BufferGeometry().setFromPoints(curve),
236
- motionLineMaterial
237
- );
238
- scene.add(line);
239
- }}
240
- }}
241
-
242
- let playing = true;
243
- document.getElementById("toggle").addEventListener("click", (event) => {{
244
- playing = !playing;
245
- event.currentTarget.textContent = playing ? "Pause" : "Play";
246
- }});
247
-
248
- const clock = new THREE.Clock();
249
- function animate() {{
250
- requestAnimationFrame(animate);
251
- const elapsed = clock.getElapsedTime();
252
- const t = playing ? elapsed : 0;
253
- for (const [index, mesh] of parts.entries()) {{
254
- reveal(mesh, elapsed, index);
255
- applyMotion(THREE, mesh, t);
256
- }}
257
- controls.update();
258
- renderer.render(scene, camera);
259
- updateLabels(THREE, camera, labelsRoot, parts);
260
- }}
261
- animate();
262
-
263
- window.addEventListener("resize", () => {{
264
- camera.aspect = window.innerWidth / window.innerHeight;
265
- camera.updateProjectionMatrix();
266
- renderer.setSize(window.innerWidth, window.innerHeight);
267
- }});
268
- }}
269
-
270
- function buildMesh(THREE, part, index) {{
271
- const g = part.geometry || {{}};
272
- const size = Array.isArray(g.size) ? g.size : [1, 1, 1];
273
- let geometry;
274
- if (g.shape === "cylinder") {{
275
- geometry = new THREE.CylinderGeometry(size[0] / 2, size[2] / 2, size[1], 48);
276
- }} else if (g.shape === "sphere") {{
277
- geometry = new THREE.SphereGeometry(Math.max(size[0], size[1], size[2]) / 2, 36, 18);
278
- }} else if (g.shape === "rod") {{
279
- geometry = new THREE.CylinderGeometry(size[0] / 2, size[0] / 2, size[2], 24);
280
- geometry.rotateX(Math.PI / 2);
281
- }} else if (g.shape === "gear") {{
282
- geometry = gearGeometry(THREE, Math.max(size[0], size[2]) / 2, size[1], g.teeth || 18);
283
- }} else {{
284
- geometry = new THREE.BoxGeometry(size[0], size[1], size[2]);
285
- }}
286
- const color = colorFor(g.color, index);
287
- const material = new THREE.MeshStandardMaterial({{
288
- color,
289
- metalness: 0.55,
290
- roughness: 0.36,
291
- emissive: color,
292
- emissiveIntensity: 0.05
293
- }});
294
- return new THREE.Mesh(geometry, material);
295
- }}
296
-
297
- function gearGeometry(THREE, radius, depth, teeth) {{
298
- const shape = new THREE.Shape();
299
- const steps = teeth * 2;
300
- for (let i = 0; i <= steps; i++) {{
301
- const angle = (i / steps) * Math.PI * 2;
302
- const r = radius * (i % 2 === 0 ? 1.0 : 0.84);
303
- const x = Math.cos(angle) * r;
304
- const y = Math.sin(angle) * r;
305
- if (i === 0) shape.moveTo(x, y);
306
- else shape.lineTo(x, y);
307
- }}
308
- const hole = new THREE.Path();
309
- hole.absarc(0, 0, radius * 0.28, 0, Math.PI * 2, true);
310
- shape.holes.push(hole);
311
- const geometry = new THREE.ExtrudeGeometry(shape, {{ depth, bevelEnabled: false }});
312
- geometry.center();
313
- geometry.rotateX(Math.PI / 2);
314
- return geometry;
315
- }}
316
-
317
- function colorFor(name, index) {{
318
- const key = String(name || "").toLowerCase();
319
- if (key.includes("amber") || key.includes("orange")) return 0xe8a33d;
320
- if (key.includes("cyan")) return 0x5fd4d0;
321
- if (key.includes("steel")) return 0x9aa4a6;
322
- const palette = [0x5fd4d0, 0xe8a33d, 0xd9d3c7, 0x80b8ff, 0xf07f5a];
323
- return palette[index % palette.length];
324
- }}
325
-
326
- function motionCurve(THREE, part) {{
327
- const p = part.geometry.position || [0, 0, 0];
328
- const points = [];
329
- for (let i = 0; i < 48; i++) {{
330
- const a = (i / 47) * Math.PI * 2;
331
- points.push(new THREE.Vector3(p[0] + Math.cos(a) * 0.52, p[1] + 0.04, p[2] + Math.sin(a) * 0.52));
332
- }}
333
- return points;
334
- }}
335
-
336
- function reveal(mesh, elapsed, index) {{
337
- const local = Math.max(0, Math.min(1, (elapsed - index * 0.12) / 0.75));
338
- const eased = 1 - Math.pow(1 - local, 3);
339
- mesh.scale.setScalar(eased);
340
- }}
341
-
342
- function applyMotion(THREE, mesh, elapsed) {{
343
- const motion = mesh.userData.part.motion || {{ type: "static" }};
344
- const speed = Number(motion.speed ?? 1);
345
- const phase = Number(motion.phase ?? 0);
346
- const axis = new THREE.Vector3(...(motion.axis || [0, 1, 0])).normalize();
347
- mesh.position.copy(mesh.userData.basePosition);
348
- mesh.rotation.copy(mesh.userData.baseRotation);
349
- if (motion.type === "rotate") {{
350
- mesh.rotateOnAxis(axis, elapsed * speed + phase);
351
- }} else if (motion.type === "oscillate") {{
352
- mesh.rotateOnAxis(axis, Math.sin(elapsed * speed + phase) * Number(motion.amplitude ?? 0.25));
353
- }} else if (motion.type === "translate") {{
354
- const range = motion.range || [-0.35, 0.35];
355
- const offset = range[0] + (range[1] - range[0]) * ((Math.sin(elapsed * speed + phase) + 1) / 2);
356
- mesh.position.add(axis.multiplyScalar(offset));
357
- }}
358
- }}
359
-
360
- function updateLabels(THREE, camera, labelsRoot, parts) {{
361
- labelsRoot.replaceChildren();
362
- const width = window.innerWidth;
363
- const height = window.innerHeight;
364
- const entries = [];
365
- for (const mesh of parts) {{
366
- const pos = mesh.position.clone().project(camera);
367
- if (pos.z < -1 || pos.z > 1) continue;
368
- entries.push({{
369
- x: clamp((pos.x * 0.5 + 0.5) * width, 76, Math.max(76, width - 76)),
370
- y: clamp((-pos.y * 0.5 + 0.5) * height, 72, Math.max(72, height - 32)),
371
- text: mesh.userData.part.name || mesh.userData.part.id || "part"
372
- }});
373
- }}
374
- entries.sort((a, b) => a.y - b.y);
375
- for (let i = 1; i < entries.length; i++) {{
376
- entries[i].y = Math.max(entries[i].y, entries[i - 1].y + 38);
377
- }}
378
- const overflow = entries.length ? entries[entries.length - 1].y - (height - 32) : 0;
379
- if (overflow > 0) {{
380
- for (const entry of entries) entry.y -= overflow;
381
- }}
382
- for (const entry of entries) {{
383
- const label = document.createElement("div");
384
- label.className = "tag";
385
- label.style.left = entry.x + "px";
386
- label.style.top = entry.y + "px";
387
- label.textContent = entry.text;
388
- labelsRoot.appendChild(label);
389
- }}
390
- }}
391
-
392
- function clamp(value, min, max) {{
393
- return Math.min(max, Math.max(min, value));
394
- }}
395
- </script>
396
- </body>
397
- </html>"""
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
snap2sim/ui.py DELETED
@@ -1,225 +0,0 @@
1
- """Gradio interface for Snap2Sim."""
2
-
3
- from __future__ import annotations
4
-
5
- import json
6
- from typing import Any
7
-
8
- import gradio as gr
9
- from PIL import Image
10
-
11
- from snap2sim.backend import InferenceClient, Settings
12
- from snap2sim.fallback_scene import build_fallback_html
13
- from snap2sim.schema import EXAMPLE_ANALYSIS
14
-
15
-
16
- CSS = """
17
- :root {
18
- --snap-bg: #14181F;
19
- --snap-panel: #1B222B;
20
- --snap-line: rgba(95, 212, 208, 0.22);
21
- --snap-amber: #E8A33D;
22
- --snap-cyan: #5FD4D0;
23
- --snap-text: #D9D3C7;
24
- --snap-muted: #8F968F;
25
- }
26
-
27
- .gradio-container {
28
- min-height: 100vh;
29
- color: var(--snap-text);
30
- background:
31
- linear-gradient(rgba(95, 212, 208, .055) 1px, transparent 1px),
32
- linear-gradient(90deg, rgba(95, 212, 208, .055) 1px, transparent 1px),
33
- radial-gradient(circle at 85% 15%, rgba(232, 163, 61, .12), transparent 30%),
34
- var(--snap-bg) !important;
35
- background-size: 28px 28px, 28px 28px, auto, auto !important;
36
- font-family: 'IBM Plex Mono', monospace !important;
37
- }
38
-
39
- .snap-shell {
40
- max-width: 1540px;
41
- margin: 0 auto;
42
- }
43
-
44
- .snap-title h1 {
45
- margin: 0 0 4px;
46
- color: var(--snap-text);
47
- font-family: 'Archivo Narrow', sans-serif;
48
- font-size: clamp(38px, 5vw, 76px);
49
- line-height: .9;
50
- letter-spacing: 0;
51
- text-transform: uppercase;
52
- }
53
-
54
- .snap-title p {
55
- max-width: 860px;
56
- color: var(--snap-muted);
57
- font-size: 14px;
58
- line-height: 1.6;
59
- }
60
-
61
- .snap-frame {
62
- border: 1px solid var(--snap-line);
63
- background: rgba(20, 24, 31, .8);
64
- box-shadow: inset 0 0 0 1px rgba(232, 163, 61, .06);
65
- }
66
-
67
- .snap-frame label,
68
- .snap-frame .label-wrap span {
69
- color: var(--snap-cyan) !important;
70
- font-family: 'IBM Plex Mono', monospace !important;
71
- text-transform: uppercase;
72
- }
73
-
74
- .snap-frame textarea,
75
- .snap-frame input {
76
- background: #10141A !important;
77
- color: var(--snap-text) !important;
78
- border-color: var(--snap-line) !important;
79
- font-family: 'IBM Plex Mono', monospace !important;
80
- }
81
-
82
- .snap-run {
83
- border: 1px solid rgba(232, 163, 61, .7) !important;
84
- background: linear-gradient(180deg, #F0B65B, #C97923) !important;
85
- color: #16130E !important;
86
- font-family: 'Archivo Narrow', sans-serif !important;
87
- font-size: 18px !important;
88
- font-weight: 700 !important;
89
- text-transform: uppercase;
90
- border-radius: 2px !important;
91
- }
92
-
93
- .snap-run:hover {
94
- filter: brightness(1.08);
95
- }
96
-
97
- .snap-status {
98
- min-height: 42px;
99
- color: var(--snap-amber);
100
- font-size: 13px;
101
- text-transform: uppercase;
102
- }
103
-
104
- .snap-html iframe,
105
- .snap-html > div {
106
- min-height: 560px;
107
- }
108
-
109
- .snap-scene-frame {
110
- display: block;
111
- width: 100%;
112
- min-height: 560px;
113
- border: 0;
114
- background: #14181F;
115
- }
116
-
117
- .snap-json pre {
118
- color: var(--snap-text) !important;
119
- background: #10141A !important;
120
- }
121
-
122
- .snap-explain {
123
- color: var(--snap-text);
124
- line-height: 1.65;
125
- }
126
- """
127
-
128
-
129
- def format_explanation(analysis: dict[str, Any]) -> str:
130
- parts = analysis.get("parts", [])
131
- part_lines = "\n".join(
132
- f"- **{part.get('name', 'part')}**: {part.get('role', 'role unknown')}"
133
- for part in parts
134
- )
135
- sequence = "\n".join(f"{index + 1}. {step}" for index, step in enumerate(analysis.get("motion_sequence", [])))
136
- return (
137
- f"### {analysis.get('component', 'Component')}\n\n"
138
- f"{analysis.get('summary', '')}\n\n"
139
- f"**Trigger:** {analysis.get('trigger', 'unknown')}\n\n"
140
- f"**Motion sequence**\n{sequence}\n\n"
141
- f"**Parts**\n{part_lines}"
142
- )
143
-
144
-
145
- def initial_status() -> str:
146
- settings = Settings()
147
- if settings.backend == "modal":
148
- configured = bool(settings.analyze_url and settings.generate_url)
149
- return "STANDBY: MODAL BACKEND CONFIGURED" if configured else "STANDBY: MODAL BACKEND MISSING URLS"
150
- return "STANDBY: LOCAL PLACEHOLDER BACKEND"
151
-
152
-
153
- def run_pipeline(image: Image.Image | None) -> tuple[str, str, str, str]:
154
- client = InferenceClient(Settings())
155
- try:
156
- analysis = client.analyze_image(image)
157
- html = client.generate_threejs(analysis)
158
- status = "CUTAWAY READY"
159
- except Exception as exc:
160
- analysis = dict(EXAMPLE_ANALYSIS)
161
- html = build_fallback_html(str(analysis.get("component", "mechanism")))
162
- status = f"FALLBACK RENDER ACTIVE: {exc}"
163
-
164
- return (
165
- html,
166
- format_explanation(analysis),
167
- json.dumps(analysis, indent=2),
168
- status,
169
- )
170
-
171
-
172
- def build_app() -> gr.Blocks:
173
- with gr.Blocks(css=CSS, title="Inside the Machine") as demo:
174
- with gr.Column(elem_classes=["snap-shell"]):
175
- gr.HTML(
176
- """
177
- <link rel="preconnect" href="https://fonts.googleapis.com">
178
- <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
179
- <link href="https://fonts.googleapis.com/css2?family=Archivo+Narrow:wght@500;700&family=IBM+Plex+Mono:wght@400;500;700&display=swap" rel="stylesheet">
180
- <section class="snap-title">
181
- <h1>Inside the Machine</h1>
182
- <p>Upload a component photo. Snap2Sim drafts a mechanical teardown,
183
- then renders an annotated cutaway animation for repair cafes,
184
- shop classrooms, and curious builders.</p>
185
- </section>
186
- """
187
- )
188
-
189
- with gr.Row(equal_height=False):
190
- with gr.Column(scale=5, elem_classes=["snap-frame"]):
191
- scene = gr.HTML(
192
- build_fallback_html("waiting for upload"),
193
- label="Cutaway viewport",
194
- elem_classes=["snap-html"],
195
- )
196
- with gr.Column(scale=3, elem_classes=["snap-frame"]):
197
- image = gr.Image(type="pil", label="Component photo")
198
- run = gr.Button("Analyze Assembly", elem_classes=["snap-run"])
199
- status = gr.Textbox(
200
- value=initial_status(),
201
- label="Workshop status",
202
- interactive=False,
203
- elem_classes=["snap-status"],
204
- )
205
- explanation = gr.Markdown(
206
- "Upload a photo and run the analyzer.",
207
- elem_classes=["snap-explain"],
208
- )
209
-
210
- with gr.Accordion("Structured mechanism JSON", open=False):
211
- raw_json = gr.Code(
212
- value="{}",
213
- language="json",
214
- label="Vision-to-scene contract",
215
- elem_classes=["snap-json"],
216
- )
217
-
218
- run.click(
219
- fn=run_pipeline,
220
- inputs=[image],
221
- outputs=[scene, explanation, raw_json, status],
222
- show_progress="full",
223
- )
224
-
225
- return demo