Spaces:
Running
Running
| # Snap2Sim — "Inside the Machine" | |
| **Build Small Hackathon** · Backyard AI Track · [huggingface.co/build-small-hackathon](https://huggingface.co/build-small-hackathon) | |
| --- | |
| ## Final Status (June 15, 2026) | |
| This file is the original build prompt and contains historical implementation | |
| directions, including early A-Frame and private-Space assumptions that were | |
| superseded during development. The finished public submission uses deterministic | |
| browser-side Three.js from validated JSON, is hosted under | |
| `build-small-hackathon/Snap2Sim`, and is linked from the README. | |
| - Public Space: https://huggingface.co/spaces/build-small-hackathon/Snap2Sim | |
| - App host: https://build-small-hackathon-snap2sim.hf.space | |
| - Demo video: https://youtu.be/nuisDKMyyF8 | |
| - X post: https://x.com/Ryno67114241/status/2066660199558152411 | |
| --- | |
| ## Goal | |
| Build a Gradio app deployed as a Hugging Face Space that takes a photo of a hardware component (gear, valve, pump, lock, engine part, etc.) and produces an animated 3D visualization showing how that component works internally — "open it up and show me the moving parts and the mechanism." | |
| --- | |
| ## Hard Constraints | |
| - Total model parameters across the entire pipeline **≤ 32B** | |
| - Must be a **Gradio app** hosted as a **Hugging Face Space** | |
| - No cloud AI APIs at inference time where possible (targets "Off the Grid" bonus) | |
| - **Plain HTML/CSS/JS frontend** — no React, no build step, no bundler | |
| --- | |
| ## Target User — The Curious Tinkerer / Maker | |
| Someone who pulls apart old electronics, finds a mystery component at a thrift store or salvage yard, or cracks open a broken appliance wondering: *"what does this actually do and how does it work inside?"* Not an engineer — a curious, hands-on person who learns by taking things apart. | |
| > **README must open with:** *"You find a small metal cylinder at a flea market. What is it? How does it work inside?"* — before any technical description. | |
| --- | |
| ## Archived Current State (June 13, 2026) | |
| Scaffold existed at [github.com/Bigstonks1/Snap2Sim](https://github.com/Bigstonks1/Snap2Sim), initially synced to HF Space `jasondo111/Snap2Sim`. The finished project was later transferred to the public Build Small Hackathon Space listed above. | |
| **Confirmed working:** | |
| - Modal deployment: `snap2sim-inside-the-machine` (bigstonks1 workspace) | |
| - `smoke_test_llamacpp_image` → `"ok": true` for `UD-Q4_K_M` + `mmproj-F16.gguf` | |
| - `analyze_image_llamacpp` and scene-generation Modal endpoints deployed | |
| - GitHub → HF Space sync workflow live and passing | |
| - Local app code now uses `gradio.Server` plus a trusted `index.html` | |
| **Archived primary tasks (completed):** | |
| 1. Deploy the current branch through GitHub → HF sync | |
| 2. Confirm the deployed Space loads the trusted `index.html` | |
| 3. Confirm `/analyze_image` and `/generate_scene` respond through the secured | |
| Modal bearer-token flow | |
| Final result: deployed through GitHub-to-Hugging Face sync, verified on the | |
| public org-owned Space, and completed with public demo video and X post links. | |
| --- | |
| ## Model Stack | |
| **Primary model:** NVIDIA Nemotron 3 Nano Omni (30B-A3B, MoE, ~3B active params) | |
| Used for both vision analysis and A-Frame scene generation (two prompt turns, same model). Targets the **NVIDIA Nemotron Quest** sponsor award. ~31B total — under the 32B cap. | |
| ### GGUF Path (confirmed working) | |
| | Setting | Value | | |
| |---|---| | |
| | Repo | `unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF` | | |
| | Primary quant | `UD-Q4_K_M` (~24 GB) + `mmproj-F16.gguf` | | |
| | Fallback quant | `UD-IQ2_M` (~18.5 GB) | | |
| | Runtime | `llama-mtmd-cli` via llama.cpp on Modal GPU | | |
| ### Fallback Split Pipeline | |
| Only use if primary model code-gen quality is too weak: | |
| - **Vision:** NVIDIA Nemotron Nano V2 VL (12B) | |
| - **Scene gen:** Qwen2.5-Coder-14B | |
| - Total ~26B · still qualifies for Nemotron Quest | |
| --- | |
| ## Model Runtime | |
| - Inference via **llama.cpp / GGUF** → targets **"Llama Champion"** bonus | |
| - Modal GPU endpoints called over HTTP with Bearer token auth (`SNAP2SIM_API_TOKEN`) | |
| - Backend swappable via `INFERENCE_BACKEND=modal | zerogpu | local` | |
| - **If Modal deployed:** "Off the Grid" not claimed, but Llama Champion + Nemotron Quest + Modal Award all apply | |
| - **If ZeroGPU sufficient:** "Off the Grid" additionally claimable | |
| --- | |
| ## Deployment Architecture | |
| ``` | |
| [HF Space — CPU tier] [Modal — GPU tier] | |
| gradio.Server analyze_image_llamacpp | |
| @app.get("/") → index.html ←→ generate_scene | |
| @app.api() → /analyze_image (weights cached in Modal Volume) | |
| @app.api() → /generate_scene | |
| ``` | |
| - Gradio app calls Modal endpoints over HTTP via `requests`, image passed as base64 | |
| - Modal cold starts on 30B model can take tens of seconds → show `"WAKING THE WORKSHOP..."` loading state | |
| --- | |
| ## Architectural Shift — `gr.Blocks` → `gradio.Server` | |
| > **This is the core change. Do not skip or partially implement it.** | |
| ### Why | |
| `gr.HTML` strips `<script>` tags for security and HF Spaces CSP blocks external CDN imports in `js_on_load`. Any WebGL/Three.js/A-Frame output piped through `gr.HTML` will fail on the live Space — scripts get stripped, nothing renders. This is a confirmed, known issue. | |
| ### How `gradio.Server` Fixes It | |
| `gradio.Server` extends FastAPI. `@app.get("/")` serves `index.html` as a first-class trusted FastAPI response — the browser receives a full page with no stripping, no sandboxing, no CSP conflicts from Gradio's component system. A-Frame and Three.js CDN scripts load normally. | |
| ```python | |
| from gradio import Server | |
| app = Server() | |
| @app.get("/") | |
| async def homepage(): | |
| with open("index.html") as f: | |
| return HTMLResponse(f.read()) | |
| @app.api(name="analyze_image") | |
| def analyze_image(image_b64: str) -> dict: | |
| return backend.run_analysis(image_b64) # calls Modal or local placeholder | |
| @app.api(name="generate_scene") | |
| def generate_scene(mechanism_json: dict) -> str: | |
| return backend.run_scene_gen(mechanism_json) # returns A-Frame HTML string | |
| app.launch() | |
| ``` | |
| ### What to Change in the Scaffold | |
| | File | Action | | |
| |---|---| | |
| | `app.py` | `gradio.Server` app serving `index.html` and API routes | | |
| | `index.html` | Trusted HTML/CSS/JS shell with pipeline orchestration | | |
| | `modal_app.py` | `generate_scene` endpoints and A-Frame prompt | | |
| | `snap2sim/backend.py` | `generate_scene` backend method | | |
| | `snap2sim/prompts.py` | A-Frame scene-generation prompt | | |
| | `snap2sim/aframe_scene.py` | Deterministic A-Frame placeholder scene | | |
| --- | |
| ## Rendering Stack — Two Layers | |
| ### Layer 1 — Model-Generated Scene: A-Frame (declarative HTML) | |
| A-Frame is a web framework built on Three.js that uses declarative HTML tags for 3D scenes. The model outputs HTML, not JavaScript — far more reliable for LLM generation. | |
| **Why A-Frame for model output:** | |
| - LLMs generate HTML tags far more reliably than imperative JS | |
| - Injected via `innerHTML`, not `eval()` — no script execution risk | |
| - A-Frame runtime (already loaded in `<head>`) renders injected tags automatically | |
| - Built-in `animation` attribute handles motion without JS animation loops | |
| - Camera, lighting, and sky added automatically — less boilerplate to get wrong | |
| **Loading A-Frame in `index.html`:** | |
| ```html | |
| <head> | |
| <script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script> | |
| </head> | |
| ``` | |
| > **CDN fallback:** If HF Spaces blocks `aframe.io`, vendor the minified A-Frame JS (~1.1MB) as a static file served via `gradio.Server`'s FastAPI static file mounting. | |
| **Injecting model output:** | |
| ```javascript | |
| document.getElementById('viewport').innerHTML = modelGeneratedAframeHTML; | |
| // A-Frame runtime picks up the new <a-scene> tags automatically | |
| ``` | |
| **Example of what the model should output:** | |
| ```html | |
| <a-scene> | |
| <a-sky color="#0F1318"></a-sky> | |
| <a-cylinder color="#E8A33D" radius="0.3" height="1" position="0 1 -3" | |
| animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear"> | |
| </a-cylinder> | |
| <a-box color="#5FD4D0" position="0.8 0.5 -3" | |
| animation="property: position; to: 0.8 1 -3; dir: alternate; loop: true; dur: 1000"> | |
| </a-box> | |
| <a-text value="Drive Shaft" position="0 2 -3" color="#5FD4D0" scale="0.5 0.5 0.5"> | |
| </a-text> | |
| </a-scene> | |
| ``` | |
| **Prompt engineering for `generate_scene` endpoint:** | |
| - Instruct the model to output **only** the `<a-scene>...</a-scene>` block — no preamble, no markdown fences, no explanation | |
| - A-Frame primitives to use: `<a-box>`, `<a-cylinder>`, `<a-sphere>`, `<a-torus>`, `<a-cone>`, `<a-entity>` | |
| - Animation format: `animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear"` | |
| - Keep scenes to **3–6 parts maximum** for clarity | |
| - Set `<a-sky color="#0F1318">` to match the page background | |
| ### Layer 2 — Deterministic Fallback: Three.js (human-written) | |
| If A-Frame output is empty, malformed, or renders blank after 3 seconds, immediately swap to `buildDeterministicScene(json)` — a JS function that reads the mechanism JSON and builds a reliable Three.js scene from geometric primitives. | |
| ```javascript | |
| function buildDeterministicScene(mechanismJson) { | |
| // Human-written. Always works given valid JSON. | |
| // Uses: BoxGeometry, CylinderGeometry, TorusGeometry per part shape | |
| // Applies rotation/translation per motion_type | |
| // Adds OrbitControls, annotation labels | |
| // The viewport must never be blank or show an error | |
| } | |
| ``` | |
| > Load Three.js in `index.html` alongside A-Frame: | |
| > ```html | |
| > <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script> | |
| > ``` | |
| --- | |
| ## Pipeline / Application Flow | |
| ``` | |
| 1. User uploads photo | |
| ↓ | |
| 2. Frontend encodes as base64 → calls /analyze_image (Gradio JS client) | |
| ↓ | |
| 3. Nemotron vision step → structured JSON: | |
| { | |
| "component_name": "Solenoid Valve", | |
| "parts": [ | |
| { "name": "Coil", "shape": "cylinder", "color": "#E8A33D", | |
| "position": [0, 0, 0], "motion_type": "none", "motion_params": {} }, | |
| { "name": "Plunger", "shape": "cylinder", "color": "#5FD4D0", | |
| "position": [0, 0.5, 0], "motion_type": "translate", | |
| "motion_params": { "axis": "y", "range": 0.3, "dur": 800 } } | |
| ], | |
| "summary": "When current flows through the coil, it generates a magnetic | |
| field that pulls the plunger upward, opening the valve port." | |
| } | |
| ↓ | |
| 4. Frontend populates analysis panel (name, part list, summary) | |
| → immediately calls /generate_scene with JSON | |
| ↓ | |
| 5. Nemotron scene gen step → A-Frame HTML string (<a-scene>...</a-scene>) | |
| ↓ | |
| 6. Frontend injects A-Frame HTML → innerHTML of #viewport | |
| A-Frame runtime renders automatically | |
| ↓ | |
| 7. If A-Frame blank/failed after 3s → buildDeterministicScene(json) | |
| Viewport is NEVER blank | |
| ``` | |
| --- | |
| ## Visual Design — "Industrial Instrument Panel / Field Cutaway" | |
| > Implement the shell and CSS in Step 2. Do **not** work on `[POLISH]` items until Steps 1–4 are done. | |
| ### Color System | |
| ```css | |
| :root { | |
| --bg: #0F1318; | |
| --bg-panel: #161B22; | |
| --bg-lift: #1E2530; | |
| --amber: #E8A33D; | |
| --amber-dim: #7A5420; | |
| --cyan: #5FD4D0; | |
| --cyan-dim: #2A5E5C; | |
| --text: #C8C0AC; | |
| --text-muted: #6B7280; | |
| --grid: rgba(255,255,255,0.04); | |
| } | |
| ``` | |
| ### Typography | |
| Load from **Bunny Fonts** (not Google Fonts): | |
| - **Display / headings / UI labels:** `Chakra Petch` — technical, instrument-panel character | |
| - **Monospace / data / callouts:** `Fira Code` | |
| - **Never use:** Inter, Roboto, Arial, Space Grotesk, or any system font | |
| ### Layout | |
| - Two-pane asymmetric split: **63% viewport** (left) · **37% analysis panel** (right) | |
| - Blueprint grid: 1px lines at `--grid` opacity, 32px spacing, on `--bg` base | |
| - Panel separator: 1px vertical line in `--amber-dim` | |
| - Upload drop zone: fills viewport · dashed 1px `--amber-dim` border (no `border-radius`) · centered `"DROP COMPONENT PHOTO"` in Chakra Petch uppercase `--text-muted` · on drag-over: border → `--amber`, text → `--amber` | |
| - Play/pause: minimal amber rectangle (no `border-radius`), Chakra Petch uppercase `"PAUSE"` / `"RESUME"`, controls A-Frame animation playback via JS | |
| ### Loading States | |
| All in Chakra Petch uppercase, `--amber` color, with thin `--amber` indeterminate progress bar across viewport top: | |
| | State | Message | | |
| |---|---| | |
| | Modal cold start | `WAKING THE WORKSHOP...` | | |
| | Vision inference | `ANALYZING ASSEMBLY...` | | |
| | Scene generation | `RENDERING CUTAWAY...` | | |
| ### [POLISH] — Only After Steps 1–4 Work | |
| Implement in this sub-order: | |
| 1. **Component name watermark** — large (`clamp(4rem, 8vw, 9rem)`) Chakra Petch uppercase in `--bg-lift`, absolutely positioned bleeding across both panes from bottom-left, `z-index` below content. Populated from `component_name` in JSON. | |
| 2. **Noise texture** — SVG `feTurbulence` grain at 3% opacity on viewport pane, inline `data:` URI, no external file. Makes the surface feel physical. | |
| 3. **Vignette** — radial gradient overlay on viewport edges, `pointer-events: none` so it floats above the A-Frame canvas without blocking interaction. | |
| 4. **Scan-line reveal** — when A-Frame scene first loads: | |
| - 2px `--cyan` scan-line sweeps top→bottom over 0.7s | |
| - Each A-Frame entity fades in as line passes it (`opacity 0→1`, `translateY 12px→0`, 0.35s ease-out, staggered by part index via `animation-delay`) | |
| - Part labels fade in together (0.25s) | |
| - Progress bar dissolves (0.2s) | |
| - Total: ~1.2s · this is the signature moment | |
| --- | |
| ## Deliverables | |
| - `app.py` — `gradio.Server` app (~50 lines) | |
| - `index.html` — self-contained HTML/CSS/JS; A-Frame + Three.js from CDN; Gradio JS client from CDN | |
| - `snap2sim/backend.py` — `generate_scene` | |
| - `modal_app.py` — `generate_scene`; A-Frame prompt | |
| - `snap2sim/prompts.py` — updated A-Frame scene generation prompt | |
| - `requirements.txt` — updated if needed for `gradio.Server` | |
| - `README.md` — tinkerer/maker story hook → project description → model stack with exact parameter breakdown (≤32B) → rendering stack rationale → bonus quest claims: | |
| | Quest | Status | | |
| |---|---| | |
| | Llama Champion | ✅ Confirmed | | |
| | NVIDIA Nemotron Quest | ✅ Confirmed | | |
| | Off-Brand | ✅ Confirmed | | |
| | Modal Award | ✅ Confirmed | | |
| | Off the Grid | ⚡ If ZeroGPU used in final deploy | | |
| | Field Notes | 🎯 Stretch | | |
| --- | |
| ## Original Start Order (completed) | |
| > Follow this exactly. Do not skip ahead. | |
| ### Step 1 — Critical Path | |
| Deploy the current `gradio.Server` + `index.html` implementation through | |
| GitHub -> HF sync. Confirm the deployed Space loads and both `/analyze_image` | |
| and `/generate_scene` respond. | |
| ### Step 2 — Make It Functional | |
| Build `buildDeterministicScene(json)` in JS — Three.js scene from geometric primitives, always works given valid JSON. Wire the full pipeline: upload → `/analyze_image` → populate analysis panel → `/generate_scene` → inject A-Frame HTML via `innerHTML` → fallback to `buildDeterministicScene(json)` if A-Frame fails or blanks after 3 seconds. **Confirm end-to-end with a real image and live Modal endpoints.** | |
| ### Step 3 — Apply the Design Shell | |
| Add CSS variable system, Chakra Petch + Fira Code from Bunny Fonts, two-pane asymmetric layout, blueprint grid, loading states with progress bar, upload drop zone, panel separator. App should match the design direction above — minus `[POLISH]` items. | |
| ### Step 4 — Harden | |
| Error handling, 3-second blank-scene timeout before fallback triggers, Modal cold-start messaging, A-Frame `<a-sky color="#0F1318">` matching page background, play/pause toggle wired to A-Frame animation playback. Verify GitHub -> HF Space sync pushes cleanly and the deployed Space runs correctly. | |
| ### Step 5 — [POLISH] Only if Time Remains | |
| Component name watermark → noise texture → vignette → scan-line reveal. In that order. | |