jasondo111 commited on
Commit
62cfa90
·
unverified ·
2 Parent(s): 2207810e40d130

Merge pull request #3 from Bigstonks1/dev-3

Browse files
Files changed (10) hide show
  1. AGENTS.md +20 -12
  2. README.md +29 -22
  3. SECURITY.md +8 -1
  4. app.py +85 -7
  5. index.html +113 -58
  6. modal_app.py +62 -29
  7. snap2sim/aframe_scene.py +0 -120
  8. snap2sim/backend.py +2 -10
  9. snap2sim/model_io.py +0 -11
  10. snap2sim/prompts.py +2 -28
AGENTS.md CHANGED
@@ -26,12 +26,13 @@ technical cutaway animation.
26
  - Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
27
  through Transformers/custom code for the vision step if GGUF endpoint quality
28
  is not reliable enough for the demo.
29
- - Local code now serves a trusted `index.html` through `gradio.Server` and
30
- exposes `/analyze_image` plus `/generate_scene`. The Space still needs a
31
- GitHub-to-HF sync and private deployment verification for this pass.
32
- - Model-generated scene output is A-Frame declarative HTML
33
- (`<a-scene>...</a-scene>`), with human-written deterministic Three.js as the
34
- fallback in `index.html`.
 
35
  - Endpoint naming is `generate_scene` everywhere in active code.
36
  - Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
37
  Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
@@ -53,8 +54,8 @@ technical cutaway animation.
53
  - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
54
  `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
55
  - `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
56
- includes A-Frame, Three.js, Gradio JS client, upload UI, pipeline
57
- orchestration, and deterministic Three.js fallback.
58
  - `modal_app.py` - Modal app scaffold with runtime asset caching, a
59
  llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
60
  placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
@@ -64,10 +65,7 @@ technical cutaway animation.
64
  - `snap2sim/__init__.py` - package marker.
65
  - `snap2sim/backend.py` - backend config, local placeholder inference, Modal
66
  HTTP client, and image base64 encoding.
67
- - `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
68
- local and placeholder Modal mode.
69
- - `snap2sim/prompts.py` - prompt templates for the vision analysis and A-Frame
70
- scene-generation steps.
71
  - `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
72
 
73
  ## What Has Been Done
@@ -191,6 +189,16 @@ technical cutaway animation.
191
  on June 14, 2026: `/analyze_image` returned a validated mechanism payload
192
  through the secured Modal bearer-token flow, and `/generate_scene` returned
193
  an A-Frame `<a-scene>` block.
 
 
 
 
 
 
 
 
 
 
194
 
195
  ## Next Work
196
 
 
26
  - Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
27
  through Transformers/custom code for the vision step if GGUF endpoint quality
28
  is not reliable enough for the demo.
29
+ - Local code serves a trusted `index.html` through `gradio.Server` and exposes
30
+ `/analyze_image` plus compatibility `/generate_scene`. The latest deployed
31
+ private Space was verified before the June 14, 2026 review-security pass; any
32
+ new code changes still need the normal GitHub-to-HF sync and deployment
33
+ verification.
34
+ - Scene rendering is deterministic browser-side Three.js from the validated
35
+ analysis JSON. Do not reintroduce model-authored HTML scene injection.
36
  - Endpoint naming is `generate_scene` everywhere in active code.
37
  - Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
38
  Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
 
54
  - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
55
  `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
56
  - `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
57
+ includes Three.js, upload UI, source-photo preview, pipeline orchestration,
58
+ and deterministic Three.js rendering.
59
  - `modal_app.py` - Modal app scaffold with runtime asset caching, a
60
  llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
61
  placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
 
65
  - `snap2sim/__init__.py` - package marker.
66
  - `snap2sim/backend.py` - backend config, local placeholder inference, Modal
67
  HTTP client, and image base64 encoding.
68
+ - `snap2sim/prompts.py` - prompt template for the vision analysis step.
 
 
 
69
  - `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
70
 
71
  ## What Has Been Done
 
189
  on June 14, 2026: `/analyze_image` returned a validated mechanism payload
190
  through the secured Modal bearer-token flow, and `/generate_scene` returned
191
  an A-Frame `<a-scene>` block.
192
+ - Implemented the `REVIEW.md` security pass on June 14, 2026:
193
+ Space-layer rate limiting for `/analyze_image` and `/generate_scene`,
194
+ upload-size caps, PIL decompression-bomb handling, clean image decode errors,
195
+ unique Modal temp image files with cleanup, and no model-authored scene HTML.
196
+ - Consolidated rendering on deterministic browser-side Three.js from validated
197
+ analysis JSON. Removed the A-Frame runtime import, unused Gradio client
198
+ import, A-Frame scene builder, A-Frame scene prompt, and raw scene parser.
199
+ - Updated the UI with source-photo preview, screen-reader live status, lighter
200
+ muted text, client-side image validation, keyboard-accessible drop zone, a
201
+ retry affordance, and a disabled initial Play state.
202
 
203
  ## Next Work
204
 
README.md CHANGED
@@ -17,10 +17,10 @@ tinkerers and makers. It takes a hardware component photo, produces a
17
  structured mechanism analysis, and renders an animated technical cutaway
18
  visualization.
19
 
20
- The local app now serves a trusted `index.html` shell through `gradio.Server`
21
- and exposes `/analyze_image` plus `/generate_scene` routes. It still defaults
22
- to the placeholder backend locally, while the Modal llama.cpp path has been
23
- smoke-tested with the selected Nemotron GGUF and projector.
24
 
25
  ## Run Locally
26
 
@@ -100,34 +100,36 @@ generation, for about 26B total parameters.
100
  - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
101
  `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
102
  - `index.html` - plain HTML/CSS/JS shell with upload orchestration,
103
- A-Frame injection, and deterministic browser-side Three.js fallback.
104
  - `modal_app.py` - Modal web endpoint scaffold.
105
  - `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
106
  - `snap2sim/backend.py` - backend selection and HTTP client.
107
- - `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
108
- local and placeholder Modal mode.
109
- - `snap2sim/prompts.py` - prompt templates for vision and A-Frame generation.
110
  - `snap2sim/schema.py` - JSON schema and sample analysis payload.
111
 
112
  ## Current Rendering Architecture
113
 
114
  The app uses `gradio.Server` to serve `index.html` directly. This avoids
115
- Gradio component script stripping and lets the page load A-Frame and Three.js
116
- normally.
117
 
118
  Runtime flow:
119
 
120
  1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
121
  2. Backend returns the validated mechanism JSON.
122
- 3. Browser posts the JSON to `/generate_scene`.
123
- 4. The scene endpoint returns only an `<a-scene>...</a-scene>` block.
124
- 5. Browser injects the A-Frame scene. If it is malformed or does not create a
125
- canvas within 3 seconds, `buildDeterministicScene(json)` renders a
126
- human-written Three.js fallback in the same viewport.
127
 
128
  The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
129
  63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
130
- colors, explicit Modal cold-start messaging, and a play/pause control.
 
 
 
 
131
 
132
  ## Modal Deployment Path
133
 
@@ -159,10 +161,11 @@ Useful deployment functions/endpoints:
159
  synthetic image and confirms it returns a validated mechanism payload.
160
  - `runtime_probe` reports the configured model repo, quant, projector file, and
161
  whether placeholder inference is still active.
162
- - `analyze_image` and `generate_scene` preserve the current HTTP contract for
163
- the Gradio app.
164
- - `analyze_image_llamacpp` and `generate_scene_llamacpp` are experimental GPU
165
- endpoints for the llama.cpp runtime path after the smoke test passes.
 
166
 
167
  Runtime environment knobs:
168
 
@@ -175,8 +178,8 @@ Runtime environment knobs:
175
  - `SNAP2SIM_RUNTIME_GPU`, default `L40S`
176
 
177
  Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
178
- `MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint and keep
179
- `MODAL_GENERATE_URL` on deterministic `generate_scene`.
180
 
181
  Run the deployment preflight in this order:
182
 
@@ -220,6 +223,10 @@ The Space also needs `SNAP2SIM_API_TOKEN` as a Hugging Face Space secret. The
220
  same value must be present in the Modal `snap2sim-api-auth` secret. Do not put
221
  that token in `.env.example`, README, logs, or prompts.
222
 
 
 
 
 
223
  From this repo, deploy after authenticating with Hugging Face:
224
 
225
  ```powershell
 
17
  structured mechanism analysis, and renders an animated technical cutaway
18
  visualization.
19
 
20
+ The local app serves a trusted `index.html` shell through `gradio.Server` and
21
+ exposes `/analyze_image` plus a compatibility `/generate_scene` route. It still
22
+ defaults to the placeholder backend locally, while the Modal llama.cpp analysis
23
+ path has been smoke-tested with the selected Nemotron GGUF and projector.
24
 
25
  ## Run Locally
26
 
 
100
  - `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
101
  `index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
102
  - `index.html` - plain HTML/CSS/JS shell with upload orchestration,
103
+ source-photo preview, and deterministic browser-side Three.js rendering.
104
  - `modal_app.py` - Modal web endpoint scaffold.
105
  - `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
106
  - `snap2sim/backend.py` - backend selection and HTTP client.
107
+ - `snap2sim/prompts.py` - prompt template for vision analysis.
 
 
108
  - `snap2sim/schema.py` - JSON schema and sample analysis payload.
109
 
110
  ## Current Rendering Architecture
111
 
112
  The app uses `gradio.Server` to serve `index.html` directly. This avoids
113
+ Gradio component script stripping and lets the page load the trusted Three.js
114
+ renderer normally.
115
 
116
  Runtime flow:
117
 
118
  1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
119
  2. Backend returns the validated mechanism JSON.
120
+ 3. Browser renders the cutaway directly from that JSON with deterministic
121
+ Three.js primitives.
122
+ 4. `/generate_scene` remains as a compatibility endpoint, but it returns a
123
+ validated `{ "renderer": "three", "analysis": ... }` scene descriptor
124
+ instead of model-authored HTML.
125
 
126
  The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
127
  63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
128
+ colors, explicit Modal cold-start messaging, source-photo preview, and a
129
+ play/pause control.
130
+
131
+ The browser no longer injects model-authored HTML into the DOM. The model's
132
+ job is limited to the structured analysis JSON contract in `snap2sim/schema.py`.
133
 
134
  ## Modal Deployment Path
135
 
 
161
  synthetic image and confirms it returns a validated mechanism payload.
162
  - `runtime_probe` reports the configured model repo, quant, projector file, and
163
  whether placeholder inference is still active.
164
+ - `analyze_image` preserves the HTTP contract for the Gradio app.
165
+ - `generate_scene` and `generate_scene_llamacpp` are compatibility endpoints
166
+ that return the deterministic Three.js scene descriptor.
167
+ - `analyze_image_llamacpp` is the experimental GPU endpoint for the llama.cpp
168
+ runtime path after the smoke test passes.
169
 
170
  Runtime environment knobs:
171
 
 
178
  - `SNAP2SIM_RUNTIME_GPU`, default `L40S`
179
 
180
  Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
181
+ `MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint. The
182
+ browser renders scenes deterministically from the validated analysis JSON.
183
 
184
  Run the deployment preflight in this order:
185
 
 
223
  same value must be present in the Modal `snap2sim-api-auth` secret. Do not put
224
  that token in `.env.example`, README, logs, or prompts.
225
 
226
+ Before making the Space public, keep the Space-layer protections active:
227
+ `app.py` rate-limits `/analyze_image` and `/generate_scene`, rejects oversized
228
+ base64/image payloads, and treats decompression-bomb images as request errors.
229
+
230
  From this repo, deploy after authenticating with Hugging Face:
231
 
232
  ```powershell
SECURITY.md CHANGED
@@ -44,6 +44,11 @@ The Hugging Face Space at `jasondo111/Snap2Sim` is private as of the latest
44
  deployment. Future agents must not change the Space visibility to public unless
45
  the user explicitly asks for that action.
46
 
 
 
 
 
 
47
  ## Environment Variables
48
 
49
  Safe to expose in public docs or `.env.example`:
@@ -82,12 +87,14 @@ Do not configure jobs that pull changes back from Hugging Face into GitHub.
82
 
83
  - Prefer Hugging Face Space variables for public configuration and Space secrets
84
  for credentials.
85
- - The planned `index.html` / `gradio.Server` frontend is public client-side
86
  code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
87
  tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
88
  static assets, browser local storage, or query strings. Browser JS should call
89
  same-origin `@app.api()` endpoints; server-side Python should attach secrets
90
  when calling Modal.
 
 
91
  - Do not print secret-bearing environment variables in logs.
92
  - Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
93
  or Codex summaries.
 
44
  deployment. Future agents must not change the Space visibility to public unless
45
  the user explicitly asks for that action.
46
 
47
+ The Space endpoints are same-origin browser APIs, not public credentials. Keep
48
+ the server-side rate limits, upload-size caps, and image decompression-bomb
49
+ guards in `app.py` active before any public launch. These controls reduce abuse
50
+ of the Space as a trusted Modal proxy but do not replace the Modal bearer token.
51
+
52
  ## Environment Variables
53
 
54
  Safe to expose in public docs or `.env.example`:
 
87
 
88
  - Prefer Hugging Face Space variables for public configuration and Space secrets
89
  for credentials.
90
+ - The `index.html` / `gradio.Server` frontend is public client-side
91
  code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
92
  tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
93
  static assets, browser local storage, or query strings. Browser JS should call
94
  same-origin `@app.api()` endpoints; server-side Python should attach secrets
95
  when calling Modal.
96
+ - Do not inject model-authored HTML, JavaScript, or A-Frame markup into the DOM.
97
+ Scene rendering should stay deterministic Three.js built from validated JSON.
98
  - Do not print secret-bearing environment variables in logs.
99
  - Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
100
  or Codex summaries.
app.py CHANGED
@@ -3,12 +3,18 @@
3
  from __future__ import annotations
4
 
5
  import base64
 
 
 
 
6
  from io import BytesIO
7
  from pathlib import Path
8
  from typing import Any, Callable
9
 
 
10
  from fastapi.responses import HTMLResponse
11
- from PIL import Image
 
12
 
13
  from snap2sim.backend import InferenceClient, Settings
14
 
@@ -37,6 +43,32 @@ except ImportError:
37
 
38
  app = Server()
39
  INDEX_PATH = Path(__file__).with_name("index.html")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
 
41
 
42
  @app.get("/", response_class=HTMLResponse)
@@ -67,13 +99,13 @@ def analyze_image_http(payload: dict[str, Any]) -> dict[str, Any]:
67
 
68
 
69
  @app.api(name="generate_scene")
70
- def generate_scene_api(analysis: dict[str, Any]) -> str:
71
  return _generate_scene(analysis)
72
 
73
 
74
  @app.post("/generate_scene")
75
- def generate_scene_http(payload: dict[str, Any]) -> dict[str, str]:
76
- return {"html": _generate_scene(payload.get("analysis") or {})}
77
 
78
 
79
  def _analyze_image(image_base64: str) -> dict[str, Any]:
@@ -81,15 +113,61 @@ def _analyze_image(image_base64: str) -> dict[str, Any]:
81
  return InferenceClient(Settings()).analyze_image(image)
82
 
83
 
84
- def _generate_scene(analysis: dict[str, Any]) -> str:
85
  return InferenceClient(Settings()).generate_scene(analysis)
86
 
87
 
88
  def _decode_image(image_base64: str) -> Image.Image:
89
  if "," in image_base64 and image_base64.lstrip().startswith("data:"):
90
  image_base64 = image_base64.split(",", 1)[1]
91
- raw = base64.b64decode(image_base64)
92
- return Image.open(BytesIO(raw)).convert("RGB")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
 
94
 
95
  if __name__ == "__main__":
 
3
  from __future__ import annotations
4
 
5
  import base64
6
+ import binascii
7
+ import threading
8
+ import time
9
+ import warnings
10
  from io import BytesIO
11
  from pathlib import Path
12
  from typing import Any, Callable
13
 
14
+ from fastapi import HTTPException, Request
15
  from fastapi.responses import HTMLResponse
16
+ from fastapi.responses import JSONResponse
17
+ from PIL import Image, UnidentifiedImageError
18
 
19
  from snap2sim.backend import InferenceClient, Settings
20
 
 
43
 
44
  app = Server()
45
  INDEX_PATH = Path(__file__).with_name("index.html")
46
+ MAX_IMAGE_BASE64_CHARS = 12 * 1024 * 1024
47
+ MAX_IMAGE_BYTES = 9 * 1024 * 1024
48
+ MAX_IMAGE_PIXELS = 12_000_000
49
+ RATE_LIMIT_WINDOW_SECONDS = 60
50
+ RATE_LIMIT_PER_CLIENT = 12
51
+ RATE_LIMIT_GLOBAL = 72
52
+ RATE_LIMIT_PATHS = {"/analyze_image", "/generate_scene"}
53
+
54
+ Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
55
+
56
+ _rate_lock = threading.Lock()
57
+ _client_hits: dict[str, list[float]] = {}
58
+ _global_hits: list[float] = []
59
+
60
+
61
+ @app.middleware("http")
62
+ async def rate_limit_api(request: Request, call_next: Callable[..., Any]) -> Any:
63
+ if request.method == "POST" and request.url.path in RATE_LIMIT_PATHS:
64
+ allowed, retry_after = _record_request(_client_id(request))
65
+ if not allowed:
66
+ return JSONResponse(
67
+ {"detail": f"Rate limit exceeded. Retry after {retry_after} seconds."},
68
+ status_code=429,
69
+ headers={"Retry-After": str(retry_after)},
70
+ )
71
+ return await call_next(request)
72
 
73
 
74
  @app.get("/", response_class=HTMLResponse)
 
99
 
100
 
101
  @app.api(name="generate_scene")
102
+ def generate_scene_api(analysis: dict[str, Any]) -> dict[str, Any]:
103
  return _generate_scene(analysis)
104
 
105
 
106
  @app.post("/generate_scene")
107
+ def generate_scene_http(payload: dict[str, Any]) -> dict[str, Any]:
108
+ return _generate_scene(payload.get("analysis") or {})
109
 
110
 
111
  def _analyze_image(image_base64: str) -> dict[str, Any]:
 
113
  return InferenceClient(Settings()).analyze_image(image)
114
 
115
 
116
+ def _generate_scene(analysis: dict[str, Any]) -> dict[str, Any]:
117
  return InferenceClient(Settings()).generate_scene(analysis)
118
 
119
 
120
  def _decode_image(image_base64: str) -> Image.Image:
121
  if "," in image_base64 and image_base64.lstrip().startswith("data:"):
122
  image_base64 = image_base64.split(",", 1)[1]
123
+ if len(image_base64) > MAX_IMAGE_BASE64_CHARS:
124
+ raise HTTPException(status_code=413, detail="Image upload is too large.")
125
+ try:
126
+ raw = base64.b64decode(image_base64, validate=True)
127
+ except (binascii.Error, ValueError) as exc:
128
+ raise HTTPException(status_code=400, detail="Image payload is not valid base64.") from exc
129
+ if len(raw) > MAX_IMAGE_BYTES:
130
+ raise HTTPException(status_code=413, detail="Image upload is too large.")
131
+
132
+ try:
133
+ with warnings.catch_warnings():
134
+ warnings.simplefilter("error", Image.DecompressionBombWarning)
135
+ image = Image.open(BytesIO(raw))
136
+ image.load()
137
+ except Image.DecompressionBombWarning as exc:
138
+ raise HTTPException(status_code=413, detail="Image dimensions are too large.") from exc
139
+ except Image.DecompressionBombError as exc:
140
+ raise HTTPException(status_code=413, detail="Image dimensions are too large.") from exc
141
+ except (UnidentifiedImageError, OSError, ValueError) as exc:
142
+ raise HTTPException(status_code=400, detail="Upload a valid image file.") from exc
143
+
144
+ if image.width * image.height > MAX_IMAGE_PIXELS:
145
+ raise HTTPException(status_code=413, detail="Image dimensions are too large.")
146
+ return image.convert("RGB")
147
+
148
+
149
+ def _client_id(request: Request) -> str:
150
+ forwarded_for = request.headers.get("x-forwarded-for", "")
151
+ if forwarded_for:
152
+ return forwarded_for.split(",", 1)[0].strip()
153
+ return request.client.host if request.client else "unknown"
154
+
155
+
156
+ def _record_request(client_id: str) -> tuple[bool, int]:
157
+ now = time.monotonic()
158
+ cutoff = now - RATE_LIMIT_WINDOW_SECONDS
159
+ with _rate_lock:
160
+ _global_hits[:] = [hit for hit in _global_hits if hit >= cutoff]
161
+ hits = [hit for hit in _client_hits.get(client_id, []) if hit >= cutoff]
162
+ if len(hits) >= RATE_LIMIT_PER_CLIENT or len(_global_hits) >= RATE_LIMIT_GLOBAL:
163
+ oldest = min(hits[0] if hits else now, _global_hits[0] if _global_hits else now)
164
+ retry_after = max(1, int(RATE_LIMIT_WINDOW_SECONDS - (now - oldest)))
165
+ _client_hits[client_id] = hits
166
+ return False, retry_after
167
+ hits.append(now)
168
+ _global_hits.append(now)
169
+ _client_hits[client_id] = hits
170
+ return True, 0
171
 
172
 
173
  if __name__ == "__main__":
index.html CHANGED
@@ -6,10 +6,19 @@
6
  <title>Inside the Machine</title>
7
  <link rel="preconnect" href="https://fonts.bunny.net">
8
  <link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
9
- <script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
 
 
 
 
 
 
 
10
  <script type="module">
11
- import { Client } from "https://cdn.jsdelivr.net/npm/@gradio/client/dist/index.min.js";
12
- window.snap2simGradioClient = Client;
 
 
13
  </script>
14
  <style>
15
  :root {
@@ -21,7 +30,7 @@
21
  --cyan: #5FD4D0;
22
  --cyan-dim: #2A5E5C;
23
  --text: #C8C0AC;
24
- --text-muted: #6B7280;
25
  --grid: rgba(255,255,255,0.04);
26
  --danger: #F07F5A;
27
  }
@@ -81,16 +90,6 @@
81
  z-index: 1;
82
  }
83
 
84
- #viewport a-scene {
85
- width: 100%;
86
- height: 100%;
87
- }
88
-
89
- #viewport .a-enter-vr,
90
- #viewport .a-orientation-modal {
91
- display: none !important;
92
- }
93
-
94
  .drop-zone {
95
  position: absolute;
96
  inset: 18px;
@@ -246,6 +245,41 @@
246
  color: var(--danger);
247
  }
248
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
249
  .readout {
250
  display: grid;
251
  gap: 18px;
@@ -376,12 +410,12 @@
376
  <div id="progress" class="progress"></div>
377
  <div class="toolbar">
378
  <button id="uploadButton" class="tool-button" type="button">Load</button>
379
- <button id="playButton" class="tool-button" type="button" disabled>Pause</button>
380
  </div>
381
- <div id="viewport"></div>
382
- <label id="dropZone" class="drop-zone" for="fileInput">Drop component photo</label>
383
  <div id="scanLine" class="scan-line"></div>
384
- <input id="fileInput" type="file" accept="image/*" hidden>
385
  </section>
386
 
387
  <aside class="panel">
@@ -389,9 +423,15 @@
389
  <header class="brand">
390
  <div class="kicker">Snap2Sim</div>
391
  <h1>Inside the Machine</h1>
392
- <div id="status" class="status">Standby</div>
 
393
  </header>
394
 
 
 
 
 
 
395
  <section class="readout">
396
  <div class="metric-row">
397
  <div id="component" class="component">Awaiting Photo</div>
@@ -426,6 +466,9 @@
426
  const progress = document.getElementById("progress");
427
  const scanLine = document.getElementById("scanLine");
428
  const statusEl = document.getElementById("status");
 
 
 
429
  const componentEl = document.getElementById("component");
430
  const confidenceEl = document.getElementById("confidence");
431
  const summaryEl = document.getElementById("summary");
@@ -438,8 +481,11 @@
438
  let paused = false;
439
  let coldStartTimer = 0;
440
  let fallbackRuntime = null;
 
 
441
 
442
  uploadButton.addEventListener("click", () => fileInput.click());
 
443
  fileInput.addEventListener("change", () => {
444
  const file = fileInput.files && fileInput.files[0];
445
  if (file) runPipeline(file);
@@ -459,6 +505,13 @@
459
  });
460
  }
461
 
 
 
 
 
 
 
 
462
  dropZone.addEventListener("drop", (event) => {
463
  const file = event.dataTransfer.files && event.dataTransfer.files[0];
464
  if (file) runPipeline(file);
@@ -467,15 +520,20 @@
467
  playButton.addEventListener("click", () => {
468
  paused = !paused;
469
  playButton.textContent = paused ? "Resume" : "Pause";
470
- if (activeMode === "aframe") {
471
- const scene = viewport.querySelector("a-scene");
472
- if (scene) paused ? scene.pause() : scene.play();
473
- }
474
  if (fallbackRuntime) fallbackRuntime.playing = !paused;
475
  });
476
 
477
  async function runPipeline(file) {
 
 
 
 
 
 
 
 
478
  resetScene();
 
479
  setBusy(true);
480
  setStatus("ANALYZING ASSEMBLY...");
481
  coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
@@ -487,12 +545,12 @@
487
  populateAnalysis(analysis);
488
 
489
  setStatus("RENDERING CUTAWAY...");
490
- const sceneResponse = await postJson("/generate_scene", { analysis });
491
- const sceneHtml = typeof sceneResponse === "string" ? sceneResponse : sceneResponse.html;
492
- renderAframe(sceneHtml, analysis);
493
  } catch (error) {
494
  window.clearTimeout(coldStartTimer);
495
  setStatus(error.message || String(error), true);
 
 
496
  if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
497
  } finally {
498
  setBusy(false);
@@ -504,8 +562,9 @@
504
  fallbackRuntime = null;
505
  activeMode = "idle";
506
  paused = false;
507
- playButton.textContent = "Pause";
508
  playButton.disabled = true;
 
509
  viewport.replaceChildren();
510
  dropZone.classList.add("hidden");
511
  }
@@ -535,35 +594,8 @@
535
  }));
536
  }
537
 
538
- function renderAframe(sceneHtml, analysis) {
539
- if (!sceneHtml || !/<a-scene[\s>]/i.test(sceneHtml) || !/<\/a-scene>/i.test(sceneHtml)) {
540
- buildDeterministicScene(analysis);
541
- return;
542
- }
543
-
544
- activeMode = "aframe";
545
- viewport.innerHTML = sceneHtml;
546
- const scene = viewport.querySelector("a-scene");
547
- if (scene) {
548
- scene.setAttribute("embedded", "");
549
- scene.setAttribute("vr-mode-ui", "enabled: false");
550
- scene.setAttribute("device-orientation-permission-ui", "enabled: false");
551
- scene.setAttribute("background", "color: #0F1318");
552
- }
553
- playButton.disabled = false;
554
- revealScan();
555
-
556
- window.setTimeout(() => {
557
- const liveScene = viewport.querySelector("a-scene");
558
- const hasCanvas = Boolean(liveScene && liveScene.canvas);
559
- if (!hasCanvas) buildDeterministicScene(analysis);
560
- }, 3000);
561
-
562
- setStatus("CUTAWAY READY");
563
- }
564
-
565
  function buildDeterministicScene(analysis) {
566
- const threeRuntime = window.THREE || (window.AFRAME && window.AFRAME.THREE);
567
  if (!threeRuntime) {
568
  setStatus("3D RUNTIME UNAVAILABLE", true);
569
  return;
@@ -575,10 +607,16 @@
575
  paused = false;
576
  playButton.textContent = "Pause";
577
  playButton.disabled = false;
578
- viewport.innerHTML = '<div class="fallback-stage"><div class="scene-mount"></div><div class="label-layer"></div></div>';
 
 
 
 
 
 
 
 
579
 
580
- const mount = viewport.querySelector(".scene-mount");
581
- const labelLayer = viewport.querySelector(".label-layer");
582
  const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
583
  renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
584
  mount.appendChild(renderer.domElement);
@@ -667,6 +705,23 @@
667
  setStatus("CUTAWAY READY");
668
  }
669
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
670
  function buildPartMesh(part, index) {
671
  const geometryData = part.geometry || {};
672
  const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
 
6
  <title>Inside the Machine</title>
7
  <link rel="preconnect" href="https://fonts.bunny.net">
8
  <link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
9
+ <script type="importmap">
10
+ {
11
+ "imports": {
12
+ "three": "https://cdn.jsdelivr.net/npm/three@0.160.1/build/three.module.js",
13
+ "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.160.1/examples/jsm/"
14
+ }
15
+ }
16
+ </script>
17
  <script type="module">
18
+ import * as THREE from "three";
19
+ import { OrbitControls } from "three/addons/controls/OrbitControls.js";
20
+
21
+ window.THREE = { ...THREE, OrbitControls };
22
  </script>
23
  <style>
24
  :root {
 
30
  --cyan: #5FD4D0;
31
  --cyan-dim: #2A5E5C;
32
  --text: #C8C0AC;
33
+ --text-muted: #9CA3AF;
34
  --grid: rgba(255,255,255,0.04);
35
  --danger: #F07F5A;
36
  }
 
90
  z-index: 1;
91
  }
92
 
 
 
 
 
 
 
 
 
 
 
93
  .drop-zone {
94
  position: absolute;
95
  inset: 18px;
 
245
  color: var(--danger);
246
  }
247
 
248
+ .retry-button {
249
+ justify-self: start;
250
+ min-height: 32px;
251
+ border: 1px solid var(--danger);
252
+ border-radius: 0;
253
+ color: var(--danger);
254
+ background: transparent;
255
+ text-transform: uppercase;
256
+ letter-spacing: 0;
257
+ cursor: pointer;
258
+ }
259
+
260
+ .retry-button[hidden] {
261
+ display: none;
262
+ }
263
+
264
+ .source-card {
265
+ display: grid;
266
+ gap: 8px;
267
+ border-bottom: 1px solid rgba(95, 212, 208, 0.16);
268
+ padding-bottom: 14px;
269
+ }
270
+
271
+ .source-card[hidden] {
272
+ display: none;
273
+ }
274
+
275
+ .source-image {
276
+ width: 100%;
277
+ max-height: 150px;
278
+ object-fit: cover;
279
+ border: 1px solid rgba(122, 84, 32, 0.72);
280
+ background: var(--bg);
281
+ }
282
+
283
  .readout {
284
  display: grid;
285
  gap: 18px;
 
410
  <div id="progress" class="progress"></div>
411
  <div class="toolbar">
412
  <button id="uploadButton" class="tool-button" type="button">Load</button>
413
+ <button id="playButton" class="tool-button" type="button" disabled>Play</button>
414
  </div>
415
+ <div id="viewport" aria-hidden="true"></div>
416
+ <label id="dropZone" class="drop-zone" for="fileInput" tabindex="0">Drop component photo</label>
417
  <div id="scanLine" class="scan-line"></div>
418
+ <input id="fileInput" type="file" accept="image/*" capture="environment" hidden>
419
  </section>
420
 
421
  <aside class="panel">
 
423
  <header class="brand">
424
  <div class="kicker">Snap2Sim</div>
425
  <h1>Inside the Machine</h1>
426
+ <div id="status" class="status" role="status" aria-live="polite">Standby</div>
427
+ <button id="retryButton" class="retry-button" type="button" hidden>Try another photo</button>
428
  </header>
429
 
430
+ <section id="sourceCard" class="source-card" hidden>
431
+ <div class="label">Source Photo</div>
432
+ <img id="sourceImage" class="source-image" alt="Uploaded component photo preview">
433
+ </section>
434
+
435
  <section class="readout">
436
  <div class="metric-row">
437
  <div id="component" class="component">Awaiting Photo</div>
 
466
  const progress = document.getElementById("progress");
467
  const scanLine = document.getElementById("scanLine");
468
  const statusEl = document.getElementById("status");
469
+ const retryButton = document.getElementById("retryButton");
470
+ const sourceCard = document.getElementById("sourceCard");
471
+ const sourceImage = document.getElementById("sourceImage");
472
  const componentEl = document.getElementById("component");
473
  const confidenceEl = document.getElementById("confidence");
474
  const summaryEl = document.getElementById("summary");
 
481
  let paused = false;
482
  let coldStartTimer = 0;
483
  let fallbackRuntime = null;
484
+ let currentPreviewUrl = "";
485
+ const MAX_CLIENT_IMAGE_BYTES = 8 * 1024 * 1024;
486
 
487
  uploadButton.addEventListener("click", () => fileInput.click());
488
+ retryButton.addEventListener("click", () => fileInput.click());
489
  fileInput.addEventListener("change", () => {
490
  const file = fileInput.files && fileInput.files[0];
491
  if (file) runPipeline(file);
 
505
  });
506
  }
507
 
508
+ dropZone.addEventListener("keydown", (event) => {
509
+ if (event.key === "Enter" || event.key === " ") {
510
+ event.preventDefault();
511
+ fileInput.click();
512
+ }
513
+ });
514
+
515
  dropZone.addEventListener("drop", (event) => {
516
  const file = event.dataTransfer.files && event.dataTransfer.files[0];
517
  if (file) runPipeline(file);
 
520
  playButton.addEventListener("click", () => {
521
  paused = !paused;
522
  playButton.textContent = paused ? "Resume" : "Pause";
 
 
 
 
523
  if (fallbackRuntime) fallbackRuntime.playing = !paused;
524
  });
525
 
526
  async function runPipeline(file) {
527
+ const validationError = validateFile(file);
528
+ if (validationError) {
529
+ setStatus(validationError, true);
530
+ retryButton.hidden = false;
531
+ dropZone.classList.remove("hidden");
532
+ return;
533
+ }
534
+
535
  resetScene();
536
+ showSourcePreview(file);
537
  setBusy(true);
538
  setStatus("ANALYZING ASSEMBLY...");
539
  coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
 
545
  populateAnalysis(analysis);
546
 
547
  setStatus("RENDERING CUTAWAY...");
548
+ buildDeterministicScene(analysis);
 
 
549
  } catch (error) {
550
  window.clearTimeout(coldStartTimer);
551
  setStatus(error.message || String(error), true);
552
+ retryButton.hidden = false;
553
+ dropZone.classList.remove("hidden");
554
  if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
555
  } finally {
556
  setBusy(false);
 
562
  fallbackRuntime = null;
563
  activeMode = "idle";
564
  paused = false;
565
+ playButton.textContent = "Play";
566
  playButton.disabled = true;
567
+ retryButton.hidden = true;
568
  viewport.replaceChildren();
569
  dropZone.classList.add("hidden");
570
  }
 
594
  }));
595
  }
596
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
597
  function buildDeterministicScene(analysis) {
598
+ const threeRuntime = window.THREE;
599
  if (!threeRuntime) {
600
  setStatus("3D RUNTIME UNAVAILABLE", true);
601
  return;
 
607
  paused = false;
608
  playButton.textContent = "Pause";
609
  playButton.disabled = false;
610
+ viewport.replaceChildren();
611
+ const stage = document.createElement("div");
612
+ stage.className = "fallback-stage";
613
+ const mount = document.createElement("div");
614
+ mount.className = "scene-mount";
615
+ const labelLayer = document.createElement("div");
616
+ labelLayer.className = "label-layer";
617
+ stage.append(mount, labelLayer);
618
+ viewport.append(stage);
619
 
 
 
620
  const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
621
  renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
622
  mount.appendChild(renderer.domElement);
 
705
  setStatus("CUTAWAY READY");
706
  }
707
 
708
+ function validateFile(file) {
709
+ if (!file.type || !file.type.startsWith("image/")) {
710
+ return "Upload an image file.";
711
+ }
712
+ if (file.size > MAX_CLIENT_IMAGE_BYTES) {
713
+ return "Image must be 8 MB or smaller.";
714
+ }
715
+ return "";
716
+ }
717
+
718
+ function showSourcePreview(file) {
719
+ if (currentPreviewUrl) URL.revokeObjectURL(currentPreviewUrl);
720
+ currentPreviewUrl = URL.createObjectURL(file);
721
+ sourceImage.src = currentPreviewUrl;
722
+ sourceCard.hidden = false;
723
+ }
724
+
725
  function buildPartMesh(part, index) {
726
  const geometryData = part.geometry || {};
727
  const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
modal_app.py CHANGED
@@ -16,9 +16,8 @@ import secrets as token_secrets
16
  import modal
17
  from fastapi import Header, HTTPException
18
 
19
- from snap2sim.aframe_scene import build_aframe_scene
20
- from snap2sim.model_io import coerce_analysis_response, parse_analysis_response, parse_scene_response
21
- from snap2sim.prompts import build_scene_prompt, build_vision_prompt
22
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
23
 
24
 
@@ -29,6 +28,9 @@ DEFAULT_RUNTIME_MODE = "placeholder"
29
  CACHE_DIR = "/cache"
30
  HF_CACHE_DIR = f"{CACHE_DIR}/huggingface"
31
  MODEL_ASSET_DIR = f"{CACHE_DIR}/models"
 
 
 
32
 
33
 
34
  model_cache = modal.Volume.from_name("snap2sim-hf-cache", create_if_missing=True)
@@ -81,14 +83,12 @@ def check_remote_imports() -> dict[str, Any]:
81
  """Lightweight Modal check that local project modules are packaged."""
82
  import snap2sim.model_io
83
  import snap2sim.schema
84
- import snap2sim.aframe_scene
85
 
86
  return {
87
  "ok": True,
88
  "modules": [
89
  snap2sim.model_io.__name__,
90
  snap2sim.schema.__name__,
91
- snap2sim.aframe_scene.__name__,
92
  ],
93
  }
94
 
@@ -191,12 +191,15 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
191
  """Run one image prompt through llama.cpp's multimodal CLI on a GPU."""
192
  import base64
193
  import subprocess
 
194
  import time
195
 
196
  from PIL import Image, ImageDraw
197
 
198
  model_path, mmproj_path = ensure_runtime_assets()
199
- test_image = Path("/tmp/snap2sim-smoke-input.jpg")
 
 
200
  img = Image.new("RGB", (512, 384), "#d8d0bd")
201
  draw = ImageDraw.Draw(img)
202
  draw.rectangle((82, 96, 430, 288), outline="#2c3138", width=8)
@@ -265,6 +268,8 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
265
  and "image input is not supported" not in combined_output.lower()
266
  and "failed to load projector" not in combined_output.lower()
267
  )
 
 
268
  return {
269
  "ok": image_supported and valid_json,
270
  "image_supported": image_supported,
@@ -275,7 +280,7 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
275
  "elapsed_seconds": elapsed_seconds,
276
  "model_path": str(model_path),
277
  "mmproj_path": str(mmproj_path),
278
- "image_base64_prefix": base64.b64encode(test_image.read_bytes()).decode("ascii")[:80],
279
  "stdout_tail": stdout[-4000:],
280
  "stderr_tail": stderr[-4000:],
281
  }
@@ -327,35 +332,66 @@ def run_llamacpp_prompt(
327
  def write_payload_image(payload: dict[str, Any]) -> Path:
328
  """Decode an image_base64 payload into a temporary RGB JPEG."""
329
  import base64
 
 
 
330
  from io import BytesIO
331
 
332
- from PIL import Image
333
 
 
334
  image_base64 = payload.get("image_base64")
335
  if not isinstance(image_base64, str) or not image_base64:
336
  raise ValueError("Request payload must include image_base64.")
337
  if "," in image_base64 and image_base64.lstrip().startswith("data:"):
338
  image_base64 = image_base64.split(",", 1)[1]
 
 
339
 
340
- raw = base64.b64decode(image_base64)
341
- image = Image.open(BytesIO(raw)).convert("RGB")
342
- path = Path("/tmp/snap2sim-request-image.jpg")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
343
  image.save(path, format="JPEG", quality=92)
344
  return path
345
 
346
 
347
  def analyze_image_llamacpp_payload(payload: dict[str, Any]) -> dict[str, Any]:
348
  image_path = write_payload_image(payload)
349
- response = run_llamacpp_prompt(
350
- build_vision_prompt(),
351
- image_path=image_path,
352
- max_tokens=1536,
353
- timeout_seconds=180,
354
- )
355
  try:
356
- return parse_analysis_response(response)
357
- except Exception:
358
- return coerce_analysis_response(response)
 
 
 
 
 
 
 
 
 
359
 
360
 
361
  @app.local_entrypoint()
@@ -447,23 +483,20 @@ def analyze_image_llamacpp(payload: dict[str, Any], authorization: str = Header(
447
 
448
  @app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
449
  @modal.fastapi_endpoint(method="POST")
450
- def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
451
  require_authorization(authorization)
452
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
453
- _prompt = build_scene_prompt(analysis)
454
  if runtime_config()["runtime_mode"] != "placeholder":
455
  raise NotImplementedError(
456
- "Nemotron runtime is selected but not wired yet. Keep placeholder "
457
- "mode active until the llama.cpp/vLLM adapter is implemented."
458
  )
459
- return {"html": build_aframe_scene(analysis)}
460
 
461
 
462
  @app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
463
  @modal.fastapi_endpoint(method="POST")
464
- def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
465
- """Experimental GPU endpoint for llama.cpp A-Frame scene generation."""
466
  require_authorization(authorization)
467
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
468
- response = run_llamacpp_prompt(build_scene_prompt(analysis), max_tokens=4096)
469
- return {"html": parse_scene_response(response)}
 
16
  import modal
17
  from fastapi import Header, HTTPException
18
 
19
+ from snap2sim.model_io import coerce_analysis_response, parse_analysis_response
20
+ from snap2sim.prompts import build_vision_prompt
 
21
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
22
 
23
 
 
28
  CACHE_DIR = "/cache"
29
  HF_CACHE_DIR = f"{CACHE_DIR}/huggingface"
30
  MODEL_ASSET_DIR = f"{CACHE_DIR}/models"
31
+ MAX_IMAGE_BASE64_CHARS = 12 * 1024 * 1024
32
+ MAX_IMAGE_BYTES = 9 * 1024 * 1024
33
+ MAX_IMAGE_PIXELS = 12_000_000
34
 
35
 
36
  model_cache = modal.Volume.from_name("snap2sim-hf-cache", create_if_missing=True)
 
83
  """Lightweight Modal check that local project modules are packaged."""
84
  import snap2sim.model_io
85
  import snap2sim.schema
 
86
 
87
  return {
88
  "ok": True,
89
  "modules": [
90
  snap2sim.model_io.__name__,
91
  snap2sim.schema.__name__,
 
92
  ],
93
  }
94
 
 
191
  """Run one image prompt through llama.cpp's multimodal CLI on a GPU."""
192
  import base64
193
  import subprocess
194
+ import tempfile
195
  import time
196
 
197
  from PIL import Image, ImageDraw
198
 
199
  model_path, mmproj_path = ensure_runtime_assets()
200
+ temp_file = tempfile.NamedTemporaryFile(prefix="snap2sim-smoke-", suffix=".jpg", delete=False)
201
+ temp_file.close()
202
+ test_image = Path(temp_file.name)
203
  img = Image.new("RGB", (512, 384), "#d8d0bd")
204
  draw = ImageDraw.Draw(img)
205
  draw.rectangle((82, 96, 430, 288), outline="#2c3138", width=8)
 
268
  and "image input is not supported" not in combined_output.lower()
269
  and "failed to load projector" not in combined_output.lower()
270
  )
271
+ image_base64_prefix = base64.b64encode(test_image.read_bytes()).decode("ascii")[:80]
272
+ test_image.unlink(missing_ok=True)
273
  return {
274
  "ok": image_supported and valid_json,
275
  "image_supported": image_supported,
 
280
  "elapsed_seconds": elapsed_seconds,
281
  "model_path": str(model_path),
282
  "mmproj_path": str(mmproj_path),
283
+ "image_base64_prefix": image_base64_prefix,
284
  "stdout_tail": stdout[-4000:],
285
  "stderr_tail": stderr[-4000:],
286
  }
 
332
  def write_payload_image(payload: dict[str, Any]) -> Path:
333
  """Decode an image_base64 payload into a temporary RGB JPEG."""
334
  import base64
335
+ import binascii
336
+ import tempfile
337
+ import warnings
338
  from io import BytesIO
339
 
340
+ from PIL import Image, UnidentifiedImageError
341
 
342
+ Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
343
  image_base64 = payload.get("image_base64")
344
  if not isinstance(image_base64, str) or not image_base64:
345
  raise ValueError("Request payload must include image_base64.")
346
  if "," in image_base64 and image_base64.lstrip().startswith("data:"):
347
  image_base64 = image_base64.split(",", 1)[1]
348
+ if len(image_base64) > MAX_IMAGE_BASE64_CHARS:
349
+ raise ValueError("Image upload is too large.")
350
 
351
+ try:
352
+ raw = base64.b64decode(image_base64, validate=True)
353
+ except (binascii.Error, ValueError) as exc:
354
+ raise ValueError("Image payload is not valid base64.") from exc
355
+ if len(raw) > MAX_IMAGE_BYTES:
356
+ raise ValueError("Image upload is too large.")
357
+
358
+ try:
359
+ with warnings.catch_warnings():
360
+ warnings.simplefilter("error", Image.DecompressionBombWarning)
361
+ image = Image.open(BytesIO(raw))
362
+ image.load()
363
+ except Image.DecompressionBombWarning as exc:
364
+ raise ValueError("Image dimensions are too large.") from exc
365
+ except Image.DecompressionBombError as exc:
366
+ raise ValueError("Image dimensions are too large.") from exc
367
+ except (UnidentifiedImageError, OSError, ValueError) as exc:
368
+ raise ValueError("Upload a valid image file.") from exc
369
+ if image.width * image.height > MAX_IMAGE_PIXELS:
370
+ raise ValueError("Image dimensions are too large.")
371
+
372
+ temp_file = tempfile.NamedTemporaryFile(prefix="snap2sim-request-", suffix=".jpg", delete=False)
373
+ temp_file.close()
374
+ path = Path(temp_file.name)
375
+ image = image.convert("RGB")
376
  image.save(path, format="JPEG", quality=92)
377
  return path
378
 
379
 
380
  def analyze_image_llamacpp_payload(payload: dict[str, Any]) -> dict[str, Any]:
381
  image_path = write_payload_image(payload)
 
 
 
 
 
 
382
  try:
383
+ response = run_llamacpp_prompt(
384
+ build_vision_prompt(),
385
+ image_path=image_path,
386
+ max_tokens=1536,
387
+ timeout_seconds=180,
388
+ )
389
+ try:
390
+ return parse_analysis_response(response)
391
+ except Exception:
392
+ return coerce_analysis_response(response)
393
+ finally:
394
+ image_path.unlink(missing_ok=True)
395
 
396
 
397
  @app.local_entrypoint()
 
483
 
484
  @app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
485
  @modal.fastapi_endpoint(method="POST")
486
+ def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, Any]:
487
  require_authorization(authorization)
488
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
 
489
  if runtime_config()["runtime_mode"] != "placeholder":
490
  raise NotImplementedError(
491
+ "Scene generation is deterministic in the browser from validated JSON."
 
492
  )
493
+ return {"renderer": "three", "analysis": analysis}
494
 
495
 
496
  @app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
497
  @modal.fastapi_endpoint(method="POST")
498
+ def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, Any]:
499
+ """Compatibility endpoint; scene rendering is deterministic browser-side."""
500
  require_authorization(authorization)
501
  analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
502
+ return {"renderer": "three", "analysis": analysis}
 
snap2sim/aframe_scene.py DELETED
@@ -1,120 +0,0 @@
1
- """Deterministic A-Frame scene generation from a validated mechanism payload."""
2
-
3
- from __future__ import annotations
4
-
5
- import html
6
- from typing import Any
7
-
8
- from snap2sim.schema import validate_analysis
9
-
10
-
11
- def build_aframe_scene(analysis: dict[str, Any]) -> str:
12
- """Return a declarative A-Frame scene for the browser to inject."""
13
- valid_analysis = validate_analysis(analysis)
14
- title = html.escape(str(valid_analysis.get("component", "mechanism")).upper(), quote=True)
15
- summary = html.escape(str(valid_analysis.get("summary", "")), quote=True)
16
- entities = "\n".join(_part_entity(part, index) for index, part in enumerate(valid_analysis["parts"][:6]))
17
- return f"""<a-scene embedded vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false" background="color: #0F1318" renderer="colorManagement: true">
18
- <a-sky color="#0F1318"></a-sky>
19
- <a-entity light="type: ambient; intensity: 0.55; color: #5FD4D0"></a-entity>
20
- <a-entity light="type: directional; intensity: 1.35; color: #E8A33D" position="3 5 4"></a-entity>
21
- <a-entity position="0 1.1 5.6">
22
- <a-camera look-controls wasd-controls="enabled: false"></a-camera>
23
- </a-entity>
24
- <a-plane color="#111820" opacity="0.72" rotation="-90 0 0" position="0 -1.05 -0.2" width="8" height="8"></a-plane>
25
- <a-text value="{title}" position="-2.8 2.35 -2.6" color="#C8C0AC" width="4.8" align="left"></a-text>
26
- <a-text value="{summary}" position="-2.8 2.05 -2.6" color="#6B7280" width="5.2" align="left"></a-text>
27
- {entities}
28
- </a-scene>"""
29
-
30
-
31
- def _part_entity(part: dict[str, Any], index: int) -> str:
32
- geometry = part["geometry"]
33
- shape = geometry["shape"]
34
- size = geometry["size"]
35
- position = _vec(geometry["position"])
36
- rotation = _rotation(geometry.get("rotation", [0, 0, 0]))
37
- color = _color(geometry.get("color"), index)
38
- label = html.escape(str(part.get("name", part.get("id", "part"))), quote=True)
39
- animation = _animation(part)
40
-
41
- if shape in {"cylinder", "gear"}:
42
- radius = max(float(size[0]), float(size[2])) / 2
43
- primitive = (
44
- f'<a-cylinder radius="{radius:.3f}" height="{float(size[1]):.3f}" '
45
- f'segments-radial="{int(geometry.get("teeth", 32)) if shape == "gear" else 48}"'
46
- )
47
- elif shape == "sphere":
48
- primitive = f'<a-sphere radius="{max(float(item) for item in size) / 2:.3f}"'
49
- elif shape == "rod":
50
- primitive = f'<a-cylinder radius="{max(float(size[0]), float(size[1])) / 2:.3f}" height="{float(size[2]):.3f}"'
51
- rotation = _rotation([1.5708, 0, 0])
52
- else:
53
- primitive = f'<a-box width="{float(size[0]):.3f}" height="{float(size[1]):.3f}" depth="{float(size[2]):.3f}"'
54
-
55
- return f""" <a-entity position="{position}">
56
- {primitive} material="color: {color}; metalness: 0.45; roughness: 0.38" rotation="{rotation}" {animation}></a-cylinder>
57
- <a-text value="{label}" position="0 {float(size[1]) + 0.28:.3f} 0" color="#5FD4D0" width="2.4" align="center"></a-text>
58
- </a-entity>""".replace("</a-cylinder>", _closing_tag(shape))
59
-
60
-
61
- def _closing_tag(shape: str) -> str:
62
- if shape == "sphere":
63
- return "</a-sphere>"
64
- if shape == "box":
65
- return "</a-box>"
66
- return "</a-cylinder>"
67
-
68
-
69
- def _animation(part: dict[str, Any]) -> str:
70
- motion = part.get("motion") or {"type": "static"}
71
- motion_type = motion.get("type")
72
- if motion_type == "static":
73
- return ""
74
-
75
- speed = max(0.1, abs(float(motion.get("speed", 1))))
76
- duration = int(max(700, min(9000, 3600 / speed)))
77
- axis = motion.get("axis") if isinstance(motion.get("axis"), list) else [0, 1, 0]
78
-
79
- if motion_type == "translate":
80
- move_range = motion.get("range") if isinstance(motion.get("range"), list) else [-0.25, 0.25]
81
- target = [float(axis[i]) * float(move_range[-1]) for i in range(3)]
82
- return (
83
- f'animation="property: position; to: {_vec(target)}; dir: alternate; '
84
- f'loop: true; dur: {duration}; easing: easeInOutSine"'
85
- )
86
-
87
- rotation = [0, 0, 0]
88
- if abs(float(axis[0])) >= max(abs(float(axis[1])), abs(float(axis[2]))):
89
- rotation[0] = 360
90
- elif abs(float(axis[2])) >= abs(float(axis[1])):
91
- rotation[2] = 360
92
- else:
93
- rotation[1] = 360
94
- if motion_type == "oscillate":
95
- amplitude = int(float(motion.get("amplitude", 0.25)) * 90)
96
- rotation = [value and amplitude for value in rotation]
97
- return (
98
- f'animation="property: rotation; to: {_vec(rotation)}; dir: alternate; '
99
- f'loop: true; dur: {duration}; easing: easeInOutSine"'
100
- )
101
- return f'animation="property: rotation; to: {_vec(rotation)}; loop: true; dur: {duration}; easing: linear"'
102
-
103
-
104
- def _vec(values: list[Any]) -> str:
105
- return " ".join(f"{float(value):.3f}" for value in values)
106
-
107
-
108
- def _rotation(values: list[Any]) -> str:
109
- return " ".join(f"{float(value) * 57.2958:.2f}" for value in values)
110
-
111
-
112
- def _color(value: Any, index: int) -> str:
113
- key = str(value or "").lower()
114
- if "amber" in key or "orange" in key:
115
- return "#E8A33D"
116
- if "cyan" in key:
117
- return "#5FD4D0"
118
- if "steel" in key:
119
- return "#9AA4A6"
120
- return ["#5FD4D0", "#E8A33D", "#C8C0AC", "#80B8FF", "#F07F5A", "#A7D676"][index % 6]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
snap2sim/backend.py CHANGED
@@ -11,7 +11,6 @@ from typing import Any
11
  import requests
12
  from PIL import Image
13
 
14
- from snap2sim.aframe_scene import build_aframe_scene
15
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
16
 
17
 
@@ -45,16 +44,9 @@ class InferenceClient:
45
 
46
  return validate_analysis(dict(EXAMPLE_ANALYSIS))
47
 
48
- def generate_scene(self, analysis: dict[str, Any]) -> str:
49
  valid_analysis = validate_analysis(analysis)
50
- if self.settings.backend == "modal":
51
- response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
52
- html = response.get("html", "")
53
- if not html:
54
- raise RuntimeError("Modal response did not include generated HTML.")
55
- return str(html)
56
-
57
- return build_aframe_scene(valid_analysis)
58
 
59
  def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
60
  if not url:
 
11
  import requests
12
  from PIL import Image
13
 
 
14
  from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
15
 
16
 
 
44
 
45
  return validate_analysis(dict(EXAMPLE_ANALYSIS))
46
 
47
+ def generate_scene(self, analysis: dict[str, Any]) -> dict[str, Any]:
48
  valid_analysis = validate_analysis(analysis)
49
+ return {"renderer": "three", "analysis": valid_analysis}
 
 
 
 
 
 
 
50
 
51
  def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
52
  if not url:
snap2sim/model_io.py CHANGED
@@ -46,17 +46,6 @@ def coerce_analysis_response(text: str) -> dict[str, Any]:
46
  return validate_analysis(_generic_analysis(fallback_component))
47
 
48
 
49
- def parse_scene_response(text: str) -> str:
50
- """Extract a complete A-Frame scene block from a model response."""
51
- raw = _strip_fences(text).strip()
52
- lowered = raw.lower()
53
- scene_start = lowered.find("<a-scene")
54
- scene_end = lowered.rfind("</a-scene>")
55
- if scene_start < 0 or scene_end < 0:
56
- raise ValueError("Model response did not contain a complete <a-scene> block.")
57
- return raw[scene_start : scene_end + len("</a-scene>")]
58
-
59
-
60
  def _strip_fences(text: str) -> str:
61
  return _FENCE_RE.sub("", text.strip()).strip()
62
 
 
46
  return validate_analysis(_generic_analysis(fallback_component))
47
 
48
 
 
 
 
 
 
 
 
 
 
 
 
49
  def _strip_fences(text: str) -> str:
50
  return _FENCE_RE.sub("", text.strip()).strip()
51
 
snap2sim/prompts.py CHANGED
@@ -1,11 +1,7 @@
1
- """Prompt templates for the two model calls."""
2
 
3
  from __future__ import annotations
4
 
5
- import json
6
- from typing import Any
7
-
8
-
9
  VISION_SYSTEM_PROMPT = """You are a mechanical teardown analyst.
10
  Given an image of a hardware component, infer the most likely internal
11
  mechanism and return only JSON matching the provided schema. Prefer clear,
@@ -18,7 +14,7 @@ def build_vision_prompt() -> str:
18
  "Analyze the uploaded hardware component as a cutaway mechanism. "
19
  "Answer with only one JSON object. Do not include markdown. Do not "
20
  "include a reasoning trace or <think> tags. Keep the payload compact and physically "
21
- "plausible for primitive A-Frame or Three.js rendering.\n\n"
22
  "Required top-level keys: component, confidence, summary, trigger, "
23
  "motion_sequence, parts.\n"
24
  "Each part requires: id, name, role, geometry, motion.\n"
@@ -46,25 +42,3 @@ def build_vision_prompt() -> str:
46
  "fields: axis, speed, amplitude, phase, range. Include optional fields "
47
  "only when useful."
48
  )
49
-
50
-
51
- AFRAME_SYSTEM_PROMPT = """You generate declarative A-Frame cutaway scenes.
52
- Return only one <a-scene>...</a-scene> block. Use primitive A-Frame entities,
53
- short labels, and animation attributes. Do not include scripts, markdown fences,
54
- explanations, or a complete HTML document."""
55
-
56
-
57
- def build_scene_prompt(analysis: dict[str, Any]) -> str:
58
- return (
59
- "Build a technical cutaway / field manual A-Frame animation for this "
60
- "mechanism analysis. Return only the <a-scene>...</a-scene> block. "
61
- "Use 3 to 6 visible primitive parts maximum, selected from <a-box>, "
62
- "<a-cylinder>, <a-sphere>, <a-torus>, <a-cone>, and <a-entity>. "
63
- "Include <a-sky color=\"#0F1318\">, a camera, lights, short <a-text> "
64
- "labels, warm amber annotations, and cool cyan moving parts. Use "
65
- "A-Frame animation attributes for motion, for example "
66
- "animation=\"property: rotation; to: 0 360 0; loop: true; dur: 2000; "
67
- "easing: linear\". Do not include scripts, markdown, explanations, "
68
- "or a complete HTML document.\n\n"
69
- f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
70
- )
 
1
+ """Prompt templates for model calls."""
2
 
3
  from __future__ import annotations
4
 
 
 
 
 
5
  VISION_SYSTEM_PROMPT = """You are a mechanical teardown analyst.
6
  Given an image of a hardware component, infer the most likely internal
7
  mechanism and return only JSON matching the provided schema. Prefer clear,
 
14
  "Analyze the uploaded hardware component as a cutaway mechanism. "
15
  "Answer with only one JSON object. Do not include markdown. Do not "
16
  "include a reasoning trace or <think> tags. Keep the payload compact and physically "
17
+ "plausible for primitive Three.js rendering.\n\n"
18
  "Required top-level keys: component, confidence, summary, trigger, "
19
  "motion_sequence, parts.\n"
20
  "Each part requires: id, name, role, geometry, motion.\n"
 
42
  "fields: axis, speed, amplitude, phase, range. Include optional fields "
43
  "only when useful."
44
  )