# Snap2Sim — "Inside the Machine"
**Build Small Hackathon** · Backyard AI Track · [huggingface.co/build-small-hackathon](https://huggingface.co/build-small-hackathon)

---

## Final Status (June 15, 2026)

This file is the original build prompt and contains historical implementation
directions, including early A-Frame and private-Space assumptions that were
superseded during development. The finished public submission uses deterministic
browser-side Three.js from validated JSON, is hosted under
`build-small-hackathon/Snap2Sim`, and is linked from the README.

- Public Space: https://huggingface.co/spaces/build-small-hackathon/Snap2Sim
- App host: https://build-small-hackathon-snap2sim.hf.space
- Demo video: https://youtu.be/nuisDKMyyF8
- X post: https://x.com/Ryno67114241/status/2066660199558152411

---

## Goal

Build a Gradio app deployed as a Hugging Face Space that takes a photo of a hardware component (gear, valve, pump, lock, engine part, etc.) and produces an animated 3D visualization showing how that component works internally — "open it up and show me the moving parts and the mechanism."

---

## Hard Constraints

- Total model parameters across the entire pipeline **≤ 32B**
- Must be a **Gradio app** hosted as a **Hugging Face Space**
- No cloud AI APIs at inference time where possible (targets "Off the Grid" bonus)
- **Plain HTML/CSS/JS frontend** — no React, no build step, no bundler

---

## Target User — The Curious Tinkerer / Maker

Someone who pulls apart old electronics, finds a mystery component at a thrift store or salvage yard, or cracks open a broken appliance wondering: *"what does this actually do and how does it work inside?"* Not an engineer — a curious, hands-on person who learns by taking things apart.

> **README must open with:** *"You find a small metal cylinder at a flea market. What is it? How does it work inside?"* — before any technical description.

---

## Archived Current State (June 13, 2026)

Scaffold existed at [github.com/Bigstonks1/Snap2Sim](https://github.com/Bigstonks1/Snap2Sim), initially synced to HF Space `jasondo111/Snap2Sim`. The finished project was later transferred to the public Build Small Hackathon Space listed above.

**Confirmed working:**
- Modal deployment: `snap2sim-inside-the-machine` (bigstonks1 workspace)
- `smoke_test_llamacpp_image` → `"ok": true` for `UD-Q4_K_M` + `mmproj-F16.gguf`
- `analyze_image_llamacpp` and scene-generation Modal endpoints deployed
- GitHub → HF Space sync workflow live and passing
- Local app code now uses `gradio.Server` plus a trusted `index.html`

**Archived primary tasks (completed):**
1. Deploy the current branch through GitHub → HF sync
2. Confirm the deployed Space loads the trusted `index.html`
3. Confirm `/analyze_image` and `/generate_scene` respond through the secured
   Modal bearer-token flow

Final result: deployed through GitHub-to-Hugging Face sync, verified on the
public org-owned Space, and completed with public demo video and X post links.

---

## Model Stack

**Primary model:** NVIDIA Nemotron 3 Nano Omni (30B-A3B, MoE, ~3B active params)
Used for both vision analysis and A-Frame scene generation (two prompt turns, same model). Targets the **NVIDIA Nemotron Quest** sponsor award. ~31B total — under the 32B cap.

### GGUF Path (confirmed working)
| Setting | Value |
|---|---|
| Repo | `unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF` |
| Primary quant | `UD-Q4_K_M` (~24 GB) + `mmproj-F16.gguf` |
| Fallback quant | `UD-IQ2_M` (~18.5 GB) |
| Runtime | `llama-mtmd-cli` via llama.cpp on Modal GPU |

### Fallback Split Pipeline
Only use if primary model code-gen quality is too weak:
- **Vision:** NVIDIA Nemotron Nano V2 VL (12B)
- **Scene gen:** Qwen2.5-Coder-14B
- Total ~26B · still qualifies for Nemotron Quest

---

## Model Runtime

- Inference via **llama.cpp / GGUF** → targets **"Llama Champion"** bonus
- Modal GPU endpoints called over HTTP with Bearer token auth (`SNAP2SIM_API_TOKEN`)
- Backend swappable via `INFERENCE_BACKEND=modal | zerogpu | local`
- **If Modal deployed:** "Off the Grid" not claimed, but Llama Champion + Nemotron Quest + Modal Award all apply
- **If ZeroGPU sufficient:** "Off the Grid" additionally claimable

---

## Deployment Architecture

```
[HF Space — CPU tier]              [Modal — GPU tier]
  gradio.Server                      analyze_image_llamacpp
  @app.get("/") → index.html    ←→   generate_scene
  @app.api() → /analyze_image        (weights cached in Modal Volume)
  @app.api() → /generate_scene
```

- Gradio app calls Modal endpoints over HTTP via `requests`, image passed as base64
- Modal cold starts on 30B model can take tens of seconds → show `"WAKING THE WORKSHOP..."` loading state

---

## Architectural Shift — `gr.Blocks` → `gradio.Server`

> **This is the core change. Do not skip or partially implement it.**

### Why
`gr.HTML` strips `<script>` tags for security and HF Spaces CSP blocks external CDN imports in `js_on_load`. Any WebGL/Three.js/A-Frame output piped through `gr.HTML` will fail on the live Space — scripts get stripped, nothing renders. This is a confirmed, known issue.

### How `gradio.Server` Fixes It
`gradio.Server` extends FastAPI. `@app.get("/")` serves `index.html` as a first-class trusted FastAPI response — the browser receives a full page with no stripping, no sandboxing, no CSP conflicts from Gradio's component system. A-Frame and Three.js CDN scripts load normally.

```python
from gradio import Server

app = Server()

@app.get("/")
async def homepage():
    with open("index.html") as f:
        return HTMLResponse(f.read())

@app.api(name="analyze_image")
def analyze_image(image_b64: str) -> dict:
    return backend.run_analysis(image_b64)  # calls Modal or local placeholder

@app.api(name="generate_scene")
def generate_scene(mechanism_json: dict) -> str:
    return backend.run_scene_gen(mechanism_json)  # returns A-Frame HTML string

app.launch()
```

### What to Change in the Scaffold
| File | Action |
|---|---|
| `app.py` | `gradio.Server` app serving `index.html` and API routes |
| `index.html` | Trusted HTML/CSS/JS shell with pipeline orchestration |
| `modal_app.py` | `generate_scene` endpoints and A-Frame prompt |
| `snap2sim/backend.py` | `generate_scene` backend method |
| `snap2sim/prompts.py` | A-Frame scene-generation prompt |
| `snap2sim/aframe_scene.py` | Deterministic A-Frame placeholder scene |

---

## Rendering Stack — Two Layers

### Layer 1 — Model-Generated Scene: A-Frame (declarative HTML)

A-Frame is a web framework built on Three.js that uses declarative HTML tags for 3D scenes. The model outputs HTML, not JavaScript — far more reliable for LLM generation.

**Why A-Frame for model output:**
- LLMs generate HTML tags far more reliably than imperative JS
- Injected via `innerHTML`, not `eval()` — no script execution risk
- A-Frame runtime (already loaded in `<head>`) renders injected tags automatically
- Built-in `animation` attribute handles motion without JS animation loops
- Camera, lighting, and sky added automatically — less boilerplate to get wrong

**Loading A-Frame in `index.html`:**
```html
<head>
  <script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
</head>
```

> **CDN fallback:** If HF Spaces blocks `aframe.io`, vendor the minified A-Frame JS (~1.1MB) as a static file served via `gradio.Server`'s FastAPI static file mounting.

**Injecting model output:**
```javascript
document.getElementById('viewport').innerHTML = modelGeneratedAframeHTML;
// A-Frame runtime picks up the new <a-scene> tags automatically
```

**Example of what the model should output:**
```html
<a-scene>
  <a-sky color="#0F1318"></a-sky>
  <a-cylinder color="#E8A33D" radius="0.3" height="1" position="0 1 -3"
    animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear">
  </a-cylinder>
  <a-box color="#5FD4D0" position="0.8 0.5 -3"
    animation="property: position; to: 0.8 1 -3; dir: alternate; loop: true; dur: 1000">
  </a-box>
  <a-text value="Drive Shaft" position="0 2 -3" color="#5FD4D0" scale="0.5 0.5 0.5">
  </a-text>
</a-scene>
```

**Prompt engineering for `generate_scene` endpoint:**
- Instruct the model to output **only** the `<a-scene>...</a-scene>` block — no preamble, no markdown fences, no explanation
- A-Frame primitives to use: `<a-box>`, `<a-cylinder>`, `<a-sphere>`, `<a-torus>`, `<a-cone>`, `<a-entity>`
- Animation format: `animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear"`
- Keep scenes to **3–6 parts maximum** for clarity
- Set `<a-sky color="#0F1318">` to match the page background

### Layer 2 — Deterministic Fallback: Three.js (human-written)

If A-Frame output is empty, malformed, or renders blank after 3 seconds, immediately swap to `buildDeterministicScene(json)` — a JS function that reads the mechanism JSON and builds a reliable Three.js scene from geometric primitives.

```javascript
function buildDeterministicScene(mechanismJson) {
  // Human-written. Always works given valid JSON.
  // Uses: BoxGeometry, CylinderGeometry, TorusGeometry per part shape
  // Applies rotation/translation per motion_type
  // Adds OrbitControls, annotation labels
  // The viewport must never be blank or show an error
}
```

> Load Three.js in `index.html` alongside A-Frame:
> ```html
> <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
> ```

---

## Pipeline / Application Flow

```
1. User uploads photo
        ↓
2. Frontend encodes as base64 → calls /analyze_image (Gradio JS client)
        ↓
3. Nemotron vision step → structured JSON:
   {
     "component_name": "Solenoid Valve",
     "parts": [
       { "name": "Coil", "shape": "cylinder", "color": "#E8A33D",
         "position": [0, 0, 0], "motion_type": "none", "motion_params": {} },
       { "name": "Plunger", "shape": "cylinder", "color": "#5FD4D0",
         "position": [0, 0.5, 0], "motion_type": "translate",
         "motion_params": { "axis": "y", "range": 0.3, "dur": 800 } }
     ],
     "summary": "When current flows through the coil, it generates a magnetic
                 field that pulls the plunger upward, opening the valve port."
   }
        ↓
4. Frontend populates analysis panel (name, part list, summary)
   → immediately calls /generate_scene with JSON
        ↓
5. Nemotron scene gen step → A-Frame HTML string (<a-scene>...</a-scene>)
        ↓
6. Frontend injects A-Frame HTML → innerHTML of #viewport
   A-Frame runtime renders automatically
        ↓
7. If A-Frame blank/failed after 3s → buildDeterministicScene(json)
   Viewport is NEVER blank
```

---

## Visual Design — "Industrial Instrument Panel / Field Cutaway"

> Implement the shell and CSS in Step 2. Do **not** work on `[POLISH]` items until Steps 1–4 are done.

### Color System
```css
:root {
  --bg:         #0F1318;
  --bg-panel:   #161B22;
  --bg-lift:    #1E2530;
  --amber:      #E8A33D;
  --amber-dim:  #7A5420;
  --cyan:       #5FD4D0;
  --cyan-dim:   #2A5E5C;
  --text:       #C8C0AC;
  --text-muted: #6B7280;
  --grid:       rgba(255,255,255,0.04);
}
```

### Typography
Load from **Bunny Fonts** (not Google Fonts):
- **Display / headings / UI labels:** `Chakra Petch` — technical, instrument-panel character
- **Monospace / data / callouts:** `Fira Code`
- **Never use:** Inter, Roboto, Arial, Space Grotesk, or any system font

### Layout
- Two-pane asymmetric split: **63% viewport** (left) · **37% analysis panel** (right)
- Blueprint grid: 1px lines at `--grid` opacity, 32px spacing, on `--bg` base
- Panel separator: 1px vertical line in `--amber-dim`
- Upload drop zone: fills viewport · dashed 1px `--amber-dim` border (no `border-radius`) · centered `"DROP COMPONENT PHOTO"` in Chakra Petch uppercase `--text-muted` · on drag-over: border → `--amber`, text → `--amber`
- Play/pause: minimal amber rectangle (no `border-radius`), Chakra Petch uppercase `"PAUSE"` / `"RESUME"`, controls A-Frame animation playback via JS

### Loading States
All in Chakra Petch uppercase, `--amber` color, with thin `--amber` indeterminate progress bar across viewport top:

| State | Message |
|---|---|
| Modal cold start | `WAKING THE WORKSHOP...` |
| Vision inference | `ANALYZING ASSEMBLY...` |
| Scene generation | `RENDERING CUTAWAY...` |

### [POLISH] — Only After Steps 1–4 Work
Implement in this sub-order:

1. **Component name watermark** — large (`clamp(4rem, 8vw, 9rem)`) Chakra Petch uppercase in `--bg-lift`, absolutely positioned bleeding across both panes from bottom-left, `z-index` below content. Populated from `component_name` in JSON.

2. **Noise texture** — SVG `feTurbulence` grain at 3% opacity on viewport pane, inline `data:` URI, no external file. Makes the surface feel physical.

3. **Vignette** — radial gradient overlay on viewport edges, `pointer-events: none` so it floats above the A-Frame canvas without blocking interaction.

4. **Scan-line reveal** — when A-Frame scene first loads:
   - 2px `--cyan` scan-line sweeps top→bottom over 0.7s
   - Each A-Frame entity fades in as line passes it (`opacity 0→1`, `translateY 12px→0`, 0.35s ease-out, staggered by part index via `animation-delay`)
   - Part labels fade in together (0.25s)
   - Progress bar dissolves (0.2s)
   - Total: ~1.2s · this is the signature moment

---

## Deliverables

- `app.py` — `gradio.Server` app (~50 lines)
- `index.html` — self-contained HTML/CSS/JS; A-Frame + Three.js from CDN; Gradio JS client from CDN
- `snap2sim/backend.py` — `generate_scene`
- `modal_app.py` — `generate_scene`; A-Frame prompt
- `snap2sim/prompts.py` — updated A-Frame scene generation prompt
- `requirements.txt` — updated if needed for `gradio.Server`
- `README.md` — tinkerer/maker story hook → project description → model stack with exact parameter breakdown (≤32B) → rendering stack rationale → bonus quest claims:

| Quest | Status |
|---|---|
| Llama Champion | ✅ Confirmed |
| NVIDIA Nemotron Quest | ✅ Confirmed |
| Off-Brand | ✅ Confirmed |
| Modal Award | ✅ Confirmed |
| Off the Grid | ⚡ If ZeroGPU used in final deploy |
| Field Notes | 🎯 Stretch |

---

## Original Start Order (completed)

> Follow this exactly. Do not skip ahead.

### Step 1 — Critical Path
Deploy the current `gradio.Server` + `index.html` implementation through
GitHub -> HF sync. Confirm the deployed Space loads and both `/analyze_image`
and `/generate_scene` respond.

### Step 2 — Make It Functional
Build `buildDeterministicScene(json)` in JS — Three.js scene from geometric primitives, always works given valid JSON. Wire the full pipeline: upload → `/analyze_image` → populate analysis panel → `/generate_scene` → inject A-Frame HTML via `innerHTML` → fallback to `buildDeterministicScene(json)` if A-Frame fails or blanks after 3 seconds. **Confirm end-to-end with a real image and live Modal endpoints.**

### Step 3 — Apply the Design Shell
Add CSS variable system, Chakra Petch + Fira Code from Bunny Fonts, two-pane asymmetric layout, blueprint grid, loading states with progress bar, upload drop zone, panel separator. App should match the design direction above — minus `[POLISH]` items.

### Step 4 — Harden
Error handling, 3-second blank-scene timeout before fallback triggers, Modal cold-start messaging, A-Frame `<a-sky color="#0F1318">` matching page background, play/pause toggle wired to A-Frame animation playback. Verify GitHub -> HF Space sync pushes cleanly and the deployed Space runs correctly.

### Step 5 — [POLISH] Only if Time Remains
Component name watermark → noise texture → vignette → scan-line reveal. In that order.