Spaces:
Running
Running
Merge pull request #3 from Bigstonks1/dev-3
Browse files- AGENTS.md +20 -12
- README.md +29 -22
- SECURITY.md +8 -1
- app.py +85 -7
- index.html +113 -58
- modal_app.py +62 -29
- snap2sim/aframe_scene.py +0 -120
- snap2sim/backend.py +2 -10
- snap2sim/model_io.py +0 -11
- snap2sim/prompts.py +2 -28
AGENTS.md
CHANGED
|
@@ -26,12 +26,13 @@ technical cutaway animation.
|
|
| 26 |
- Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
|
| 27 |
through Transformers/custom code for the vision step if GGUF endpoint quality
|
| 28 |
is not reliable enough for the demo.
|
| 29 |
-
- Local code
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
|
|
|
| 35 |
- Endpoint naming is `generate_scene` everywhere in active code.
|
| 36 |
- Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
|
| 37 |
Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
|
|
@@ -53,8 +54,8 @@ technical cutaway animation.
|
|
| 53 |
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 54 |
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 55 |
- `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
|
| 56 |
-
includes
|
| 57 |
-
|
| 58 |
- `modal_app.py` - Modal app scaffold with runtime asset caching, a
|
| 59 |
llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
|
| 60 |
placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
|
|
@@ -64,10 +65,7 @@ technical cutaway animation.
|
|
| 64 |
- `snap2sim/__init__.py` - package marker.
|
| 65 |
- `snap2sim/backend.py` - backend config, local placeholder inference, Modal
|
| 66 |
HTTP client, and image base64 encoding.
|
| 67 |
-
- `snap2sim/
|
| 68 |
-
local and placeholder Modal mode.
|
| 69 |
-
- `snap2sim/prompts.py` - prompt templates for the vision analysis and A-Frame
|
| 70 |
-
scene-generation steps.
|
| 71 |
- `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
|
| 72 |
|
| 73 |
## What Has Been Done
|
|
@@ -191,6 +189,16 @@ technical cutaway animation.
|
|
| 191 |
on June 14, 2026: `/analyze_image` returned a validated mechanism payload
|
| 192 |
through the secured Modal bearer-token flow, and `/generate_scene` returned
|
| 193 |
an A-Frame `<a-scene>` block.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 194 |
|
| 195 |
## Next Work
|
| 196 |
|
|
|
|
| 26 |
- Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
|
| 27 |
through Transformers/custom code for the vision step if GGUF endpoint quality
|
| 28 |
is not reliable enough for the demo.
|
| 29 |
+
- Local code serves a trusted `index.html` through `gradio.Server` and exposes
|
| 30 |
+
`/analyze_image` plus compatibility `/generate_scene`. The latest deployed
|
| 31 |
+
private Space was verified before the June 14, 2026 review-security pass; any
|
| 32 |
+
new code changes still need the normal GitHub-to-HF sync and deployment
|
| 33 |
+
verification.
|
| 34 |
+
- Scene rendering is deterministic browser-side Three.js from the validated
|
| 35 |
+
analysis JSON. Do not reintroduce model-authored HTML scene injection.
|
| 36 |
- Endpoint naming is `generate_scene` everywhere in active code.
|
| 37 |
- Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
|
| 38 |
Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
|
|
|
|
| 54 |
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 55 |
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 56 |
- `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
|
| 57 |
+
includes Three.js, upload UI, source-photo preview, pipeline orchestration,
|
| 58 |
+
and deterministic Three.js rendering.
|
| 59 |
- `modal_app.py` - Modal app scaffold with runtime asset caching, a
|
| 60 |
llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
|
| 61 |
placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
|
|
|
|
| 65 |
- `snap2sim/__init__.py` - package marker.
|
| 66 |
- `snap2sim/backend.py` - backend config, local placeholder inference, Modal
|
| 67 |
HTTP client, and image base64 encoding.
|
| 68 |
+
- `snap2sim/prompts.py` - prompt template for the vision analysis step.
|
|
|
|
|
|
|
|
|
|
| 69 |
- `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
|
| 70 |
|
| 71 |
## What Has Been Done
|
|
|
|
| 189 |
on June 14, 2026: `/analyze_image` returned a validated mechanism payload
|
| 190 |
through the secured Modal bearer-token flow, and `/generate_scene` returned
|
| 191 |
an A-Frame `<a-scene>` block.
|
| 192 |
+
- Implemented the `REVIEW.md` security pass on June 14, 2026:
|
| 193 |
+
Space-layer rate limiting for `/analyze_image` and `/generate_scene`,
|
| 194 |
+
upload-size caps, PIL decompression-bomb handling, clean image decode errors,
|
| 195 |
+
unique Modal temp image files with cleanup, and no model-authored scene HTML.
|
| 196 |
+
- Consolidated rendering on deterministic browser-side Three.js from validated
|
| 197 |
+
analysis JSON. Removed the A-Frame runtime import, unused Gradio client
|
| 198 |
+
import, A-Frame scene builder, A-Frame scene prompt, and raw scene parser.
|
| 199 |
+
- Updated the UI with source-photo preview, screen-reader live status, lighter
|
| 200 |
+
muted text, client-side image validation, keyboard-accessible drop zone, a
|
| 201 |
+
retry affordance, and a disabled initial Play state.
|
| 202 |
|
| 203 |
## Next Work
|
| 204 |
|
README.md
CHANGED
|
@@ -17,10 +17,10 @@ tinkerers and makers. It takes a hardware component photo, produces a
|
|
| 17 |
structured mechanism analysis, and renders an animated technical cutaway
|
| 18 |
visualization.
|
| 19 |
|
| 20 |
-
The local app
|
| 21 |
-
|
| 22 |
-
to the placeholder backend locally, while the Modal llama.cpp
|
| 23 |
-
smoke-tested with the selected Nemotron GGUF and projector.
|
| 24 |
|
| 25 |
## Run Locally
|
| 26 |
|
|
@@ -100,34 +100,36 @@ generation, for about 26B total parameters.
|
|
| 100 |
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 101 |
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 102 |
- `index.html` - plain HTML/CSS/JS shell with upload orchestration,
|
| 103 |
-
|
| 104 |
- `modal_app.py` - Modal web endpoint scaffold.
|
| 105 |
- `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
|
| 106 |
- `snap2sim/backend.py` - backend selection and HTTP client.
|
| 107 |
-
- `snap2sim/
|
| 108 |
-
local and placeholder Modal mode.
|
| 109 |
-
- `snap2sim/prompts.py` - prompt templates for vision and A-Frame generation.
|
| 110 |
- `snap2sim/schema.py` - JSON schema and sample analysis payload.
|
| 111 |
|
| 112 |
## Current Rendering Architecture
|
| 113 |
|
| 114 |
The app uses `gradio.Server` to serve `index.html` directly. This avoids
|
| 115 |
-
Gradio component script stripping and lets the page load
|
| 116 |
-
normally.
|
| 117 |
|
| 118 |
Runtime flow:
|
| 119 |
|
| 120 |
1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
|
| 121 |
2. Backend returns the validated mechanism JSON.
|
| 122 |
-
3. Browser
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
|
| 128 |
The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
|
| 129 |
63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
|
| 130 |
-
colors, explicit Modal cold-start messaging,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
## Modal Deployment Path
|
| 133 |
|
|
@@ -159,10 +161,11 @@ Useful deployment functions/endpoints:
|
|
| 159 |
synthetic image and confirms it returns a validated mechanism payload.
|
| 160 |
- `runtime_probe` reports the configured model repo, quant, projector file, and
|
| 161 |
whether placeholder inference is still active.
|
| 162 |
-
- `analyze_image`
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
|
|
|
| 166 |
|
| 167 |
Runtime environment knobs:
|
| 168 |
|
|
@@ -175,8 +178,8 @@ Runtime environment knobs:
|
|
| 175 |
- `SNAP2SIM_RUNTIME_GPU`, default `L40S`
|
| 176 |
|
| 177 |
Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
|
| 178 |
-
`MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint
|
| 179 |
-
|
| 180 |
|
| 181 |
Run the deployment preflight in this order:
|
| 182 |
|
|
@@ -220,6 +223,10 @@ The Space also needs `SNAP2SIM_API_TOKEN` as a Hugging Face Space secret. The
|
|
| 220 |
same value must be present in the Modal `snap2sim-api-auth` secret. Do not put
|
| 221 |
that token in `.env.example`, README, logs, or prompts.
|
| 222 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
From this repo, deploy after authenticating with Hugging Face:
|
| 224 |
|
| 225 |
```powershell
|
|
|
|
| 17 |
structured mechanism analysis, and renders an animated technical cutaway
|
| 18 |
visualization.
|
| 19 |
|
| 20 |
+
The local app serves a trusted `index.html` shell through `gradio.Server` and
|
| 21 |
+
exposes `/analyze_image` plus a compatibility `/generate_scene` route. It still
|
| 22 |
+
defaults to the placeholder backend locally, while the Modal llama.cpp analysis
|
| 23 |
+
path has been smoke-tested with the selected Nemotron GGUF and projector.
|
| 24 |
|
| 25 |
## Run Locally
|
| 26 |
|
|
|
|
| 100 |
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 101 |
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 102 |
- `index.html` - plain HTML/CSS/JS shell with upload orchestration,
|
| 103 |
+
source-photo preview, and deterministic browser-side Three.js rendering.
|
| 104 |
- `modal_app.py` - Modal web endpoint scaffold.
|
| 105 |
- `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
|
| 106 |
- `snap2sim/backend.py` - backend selection and HTTP client.
|
| 107 |
+
- `snap2sim/prompts.py` - prompt template for vision analysis.
|
|
|
|
|
|
|
| 108 |
- `snap2sim/schema.py` - JSON schema and sample analysis payload.
|
| 109 |
|
| 110 |
## Current Rendering Architecture
|
| 111 |
|
| 112 |
The app uses `gradio.Server` to serve `index.html` directly. This avoids
|
| 113 |
+
Gradio component script stripping and lets the page load the trusted Three.js
|
| 114 |
+
renderer normally.
|
| 115 |
|
| 116 |
Runtime flow:
|
| 117 |
|
| 118 |
1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
|
| 119 |
2. Backend returns the validated mechanism JSON.
|
| 120 |
+
3. Browser renders the cutaway directly from that JSON with deterministic
|
| 121 |
+
Three.js primitives.
|
| 122 |
+
4. `/generate_scene` remains as a compatibility endpoint, but it returns a
|
| 123 |
+
validated `{ "renderer": "three", "analysis": ... }` scene descriptor
|
| 124 |
+
instead of model-authored HTML.
|
| 125 |
|
| 126 |
The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
|
| 127 |
63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
|
| 128 |
+
colors, explicit Modal cold-start messaging, source-photo preview, and a
|
| 129 |
+
play/pause control.
|
| 130 |
+
|
| 131 |
+
The browser no longer injects model-authored HTML into the DOM. The model's
|
| 132 |
+
job is limited to the structured analysis JSON contract in `snap2sim/schema.py`.
|
| 133 |
|
| 134 |
## Modal Deployment Path
|
| 135 |
|
|
|
|
| 161 |
synthetic image and confirms it returns a validated mechanism payload.
|
| 162 |
- `runtime_probe` reports the configured model repo, quant, projector file, and
|
| 163 |
whether placeholder inference is still active.
|
| 164 |
+
- `analyze_image` preserves the HTTP contract for the Gradio app.
|
| 165 |
+
- `generate_scene` and `generate_scene_llamacpp` are compatibility endpoints
|
| 166 |
+
that return the deterministic Three.js scene descriptor.
|
| 167 |
+
- `analyze_image_llamacpp` is the experimental GPU endpoint for the llama.cpp
|
| 168 |
+
runtime path after the smoke test passes.
|
| 169 |
|
| 170 |
Runtime environment knobs:
|
| 171 |
|
|
|
|
| 178 |
- `SNAP2SIM_RUNTIME_GPU`, default `L40S`
|
| 179 |
|
| 180 |
Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
|
| 181 |
+
`MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint. The
|
| 182 |
+
browser renders scenes deterministically from the validated analysis JSON.
|
| 183 |
|
| 184 |
Run the deployment preflight in this order:
|
| 185 |
|
|
|
|
| 223 |
same value must be present in the Modal `snap2sim-api-auth` secret. Do not put
|
| 224 |
that token in `.env.example`, README, logs, or prompts.
|
| 225 |
|
| 226 |
+
Before making the Space public, keep the Space-layer protections active:
|
| 227 |
+
`app.py` rate-limits `/analyze_image` and `/generate_scene`, rejects oversized
|
| 228 |
+
base64/image payloads, and treats decompression-bomb images as request errors.
|
| 229 |
+
|
| 230 |
From this repo, deploy after authenticating with Hugging Face:
|
| 231 |
|
| 232 |
```powershell
|
SECURITY.md
CHANGED
|
@@ -44,6 +44,11 @@ The Hugging Face Space at `jasondo111/Snap2Sim` is private as of the latest
|
|
| 44 |
deployment. Future agents must not change the Space visibility to public unless
|
| 45 |
the user explicitly asks for that action.
|
| 46 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
## Environment Variables
|
| 48 |
|
| 49 |
Safe to expose in public docs or `.env.example`:
|
|
@@ -82,12 +87,14 @@ Do not configure jobs that pull changes back from Hugging Face into GitHub.
|
|
| 82 |
|
| 83 |
- Prefer Hugging Face Space variables for public configuration and Space secrets
|
| 84 |
for credentials.
|
| 85 |
-
- The
|
| 86 |
code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
|
| 87 |
tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
|
| 88 |
static assets, browser local storage, or query strings. Browser JS should call
|
| 89 |
same-origin `@app.api()` endpoints; server-side Python should attach secrets
|
| 90 |
when calling Modal.
|
|
|
|
|
|
|
| 91 |
- Do not print secret-bearing environment variables in logs.
|
| 92 |
- Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
|
| 93 |
or Codex summaries.
|
|
|
|
| 44 |
deployment. Future agents must not change the Space visibility to public unless
|
| 45 |
the user explicitly asks for that action.
|
| 46 |
|
| 47 |
+
The Space endpoints are same-origin browser APIs, not public credentials. Keep
|
| 48 |
+
the server-side rate limits, upload-size caps, and image decompression-bomb
|
| 49 |
+
guards in `app.py` active before any public launch. These controls reduce abuse
|
| 50 |
+
of the Space as a trusted Modal proxy but do not replace the Modal bearer token.
|
| 51 |
+
|
| 52 |
## Environment Variables
|
| 53 |
|
| 54 |
Safe to expose in public docs or `.env.example`:
|
|
|
|
| 87 |
|
| 88 |
- Prefer Hugging Face Space variables for public configuration and Space secrets
|
| 89 |
for credentials.
|
| 90 |
+
- The `index.html` / `gradio.Server` frontend is public client-side
|
| 91 |
code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
|
| 92 |
tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
|
| 93 |
static assets, browser local storage, or query strings. Browser JS should call
|
| 94 |
same-origin `@app.api()` endpoints; server-side Python should attach secrets
|
| 95 |
when calling Modal.
|
| 96 |
+
- Do not inject model-authored HTML, JavaScript, or A-Frame markup into the DOM.
|
| 97 |
+
Scene rendering should stay deterministic Three.js built from validated JSON.
|
| 98 |
- Do not print secret-bearing environment variables in logs.
|
| 99 |
- Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
|
| 100 |
or Codex summaries.
|
app.py
CHANGED
|
@@ -3,12 +3,18 @@
|
|
| 3 |
from __future__ import annotations
|
| 4 |
|
| 5 |
import base64
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
from io import BytesIO
|
| 7 |
from pathlib import Path
|
| 8 |
from typing import Any, Callable
|
| 9 |
|
|
|
|
| 10 |
from fastapi.responses import HTMLResponse
|
| 11 |
-
from
|
|
|
|
| 12 |
|
| 13 |
from snap2sim.backend import InferenceClient, Settings
|
| 14 |
|
|
@@ -37,6 +43,32 @@ except ImportError:
|
|
| 37 |
|
| 38 |
app = Server()
|
| 39 |
INDEX_PATH = Path(__file__).with_name("index.html")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
|
| 42 |
@app.get("/", response_class=HTMLResponse)
|
|
@@ -67,13 +99,13 @@ def analyze_image_http(payload: dict[str, Any]) -> dict[str, Any]:
|
|
| 67 |
|
| 68 |
|
| 69 |
@app.api(name="generate_scene")
|
| 70 |
-
def generate_scene_api(analysis: dict[str, Any]) -> str:
|
| 71 |
return _generate_scene(analysis)
|
| 72 |
|
| 73 |
|
| 74 |
@app.post("/generate_scene")
|
| 75 |
-
def generate_scene_http(payload: dict[str, Any]) -> dict[str,
|
| 76 |
-
return
|
| 77 |
|
| 78 |
|
| 79 |
def _analyze_image(image_base64: str) -> dict[str, Any]:
|
|
@@ -81,15 +113,61 @@ def _analyze_image(image_base64: str) -> dict[str, Any]:
|
|
| 81 |
return InferenceClient(Settings()).analyze_image(image)
|
| 82 |
|
| 83 |
|
| 84 |
-
def _generate_scene(analysis: dict[str, Any]) -> str:
|
| 85 |
return InferenceClient(Settings()).generate_scene(analysis)
|
| 86 |
|
| 87 |
|
| 88 |
def _decode_image(image_base64: str) -> Image.Image:
|
| 89 |
if "," in image_base64 and image_base64.lstrip().startswith("data:"):
|
| 90 |
image_base64 = image_base64.split(",", 1)[1]
|
| 91 |
-
|
| 92 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
|
| 95 |
if __name__ == "__main__":
|
|
|
|
| 3 |
from __future__ import annotations
|
| 4 |
|
| 5 |
import base64
|
| 6 |
+
import binascii
|
| 7 |
+
import threading
|
| 8 |
+
import time
|
| 9 |
+
import warnings
|
| 10 |
from io import BytesIO
|
| 11 |
from pathlib import Path
|
| 12 |
from typing import Any, Callable
|
| 13 |
|
| 14 |
+
from fastapi import HTTPException, Request
|
| 15 |
from fastapi.responses import HTMLResponse
|
| 16 |
+
from fastapi.responses import JSONResponse
|
| 17 |
+
from PIL import Image, UnidentifiedImageError
|
| 18 |
|
| 19 |
from snap2sim.backend import InferenceClient, Settings
|
| 20 |
|
|
|
|
| 43 |
|
| 44 |
app = Server()
|
| 45 |
INDEX_PATH = Path(__file__).with_name("index.html")
|
| 46 |
+
MAX_IMAGE_BASE64_CHARS = 12 * 1024 * 1024
|
| 47 |
+
MAX_IMAGE_BYTES = 9 * 1024 * 1024
|
| 48 |
+
MAX_IMAGE_PIXELS = 12_000_000
|
| 49 |
+
RATE_LIMIT_WINDOW_SECONDS = 60
|
| 50 |
+
RATE_LIMIT_PER_CLIENT = 12
|
| 51 |
+
RATE_LIMIT_GLOBAL = 72
|
| 52 |
+
RATE_LIMIT_PATHS = {"/analyze_image", "/generate_scene"}
|
| 53 |
+
|
| 54 |
+
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
|
| 55 |
+
|
| 56 |
+
_rate_lock = threading.Lock()
|
| 57 |
+
_client_hits: dict[str, list[float]] = {}
|
| 58 |
+
_global_hits: list[float] = []
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
@app.middleware("http")
|
| 62 |
+
async def rate_limit_api(request: Request, call_next: Callable[..., Any]) -> Any:
|
| 63 |
+
if request.method == "POST" and request.url.path in RATE_LIMIT_PATHS:
|
| 64 |
+
allowed, retry_after = _record_request(_client_id(request))
|
| 65 |
+
if not allowed:
|
| 66 |
+
return JSONResponse(
|
| 67 |
+
{"detail": f"Rate limit exceeded. Retry after {retry_after} seconds."},
|
| 68 |
+
status_code=429,
|
| 69 |
+
headers={"Retry-After": str(retry_after)},
|
| 70 |
+
)
|
| 71 |
+
return await call_next(request)
|
| 72 |
|
| 73 |
|
| 74 |
@app.get("/", response_class=HTMLResponse)
|
|
|
|
| 99 |
|
| 100 |
|
| 101 |
@app.api(name="generate_scene")
|
| 102 |
+
def generate_scene_api(analysis: dict[str, Any]) -> dict[str, Any]:
|
| 103 |
return _generate_scene(analysis)
|
| 104 |
|
| 105 |
|
| 106 |
@app.post("/generate_scene")
|
| 107 |
+
def generate_scene_http(payload: dict[str, Any]) -> dict[str, Any]:
|
| 108 |
+
return _generate_scene(payload.get("analysis") or {})
|
| 109 |
|
| 110 |
|
| 111 |
def _analyze_image(image_base64: str) -> dict[str, Any]:
|
|
|
|
| 113 |
return InferenceClient(Settings()).analyze_image(image)
|
| 114 |
|
| 115 |
|
| 116 |
+
def _generate_scene(analysis: dict[str, Any]) -> dict[str, Any]:
|
| 117 |
return InferenceClient(Settings()).generate_scene(analysis)
|
| 118 |
|
| 119 |
|
| 120 |
def _decode_image(image_base64: str) -> Image.Image:
|
| 121 |
if "," in image_base64 and image_base64.lstrip().startswith("data:"):
|
| 122 |
image_base64 = image_base64.split(",", 1)[1]
|
| 123 |
+
if len(image_base64) > MAX_IMAGE_BASE64_CHARS:
|
| 124 |
+
raise HTTPException(status_code=413, detail="Image upload is too large.")
|
| 125 |
+
try:
|
| 126 |
+
raw = base64.b64decode(image_base64, validate=True)
|
| 127 |
+
except (binascii.Error, ValueError) as exc:
|
| 128 |
+
raise HTTPException(status_code=400, detail="Image payload is not valid base64.") from exc
|
| 129 |
+
if len(raw) > MAX_IMAGE_BYTES:
|
| 130 |
+
raise HTTPException(status_code=413, detail="Image upload is too large.")
|
| 131 |
+
|
| 132 |
+
try:
|
| 133 |
+
with warnings.catch_warnings():
|
| 134 |
+
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
| 135 |
+
image = Image.open(BytesIO(raw))
|
| 136 |
+
image.load()
|
| 137 |
+
except Image.DecompressionBombWarning as exc:
|
| 138 |
+
raise HTTPException(status_code=413, detail="Image dimensions are too large.") from exc
|
| 139 |
+
except Image.DecompressionBombError as exc:
|
| 140 |
+
raise HTTPException(status_code=413, detail="Image dimensions are too large.") from exc
|
| 141 |
+
except (UnidentifiedImageError, OSError, ValueError) as exc:
|
| 142 |
+
raise HTTPException(status_code=400, detail="Upload a valid image file.") from exc
|
| 143 |
+
|
| 144 |
+
if image.width * image.height > MAX_IMAGE_PIXELS:
|
| 145 |
+
raise HTTPException(status_code=413, detail="Image dimensions are too large.")
|
| 146 |
+
return image.convert("RGB")
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _client_id(request: Request) -> str:
|
| 150 |
+
forwarded_for = request.headers.get("x-forwarded-for", "")
|
| 151 |
+
if forwarded_for:
|
| 152 |
+
return forwarded_for.split(",", 1)[0].strip()
|
| 153 |
+
return request.client.host if request.client else "unknown"
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
def _record_request(client_id: str) -> tuple[bool, int]:
|
| 157 |
+
now = time.monotonic()
|
| 158 |
+
cutoff = now - RATE_LIMIT_WINDOW_SECONDS
|
| 159 |
+
with _rate_lock:
|
| 160 |
+
_global_hits[:] = [hit for hit in _global_hits if hit >= cutoff]
|
| 161 |
+
hits = [hit for hit in _client_hits.get(client_id, []) if hit >= cutoff]
|
| 162 |
+
if len(hits) >= RATE_LIMIT_PER_CLIENT or len(_global_hits) >= RATE_LIMIT_GLOBAL:
|
| 163 |
+
oldest = min(hits[0] if hits else now, _global_hits[0] if _global_hits else now)
|
| 164 |
+
retry_after = max(1, int(RATE_LIMIT_WINDOW_SECONDS - (now - oldest)))
|
| 165 |
+
_client_hits[client_id] = hits
|
| 166 |
+
return False, retry_after
|
| 167 |
+
hits.append(now)
|
| 168 |
+
_global_hits.append(now)
|
| 169 |
+
_client_hits[client_id] = hits
|
| 170 |
+
return True, 0
|
| 171 |
|
| 172 |
|
| 173 |
if __name__ == "__main__":
|
index.html
CHANGED
|
@@ -6,10 +6,19 @@
|
|
| 6 |
<title>Inside the Machine</title>
|
| 7 |
<link rel="preconnect" href="https://fonts.bunny.net">
|
| 8 |
<link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
|
| 9 |
-
<script
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
<script type="module">
|
| 11 |
-
import
|
| 12 |
-
|
|
|
|
|
|
|
| 13 |
</script>
|
| 14 |
<style>
|
| 15 |
:root {
|
|
@@ -21,7 +30,7 @@
|
|
| 21 |
--cyan: #5FD4D0;
|
| 22 |
--cyan-dim: #2A5E5C;
|
| 23 |
--text: #C8C0AC;
|
| 24 |
-
--text-muted: #
|
| 25 |
--grid: rgba(255,255,255,0.04);
|
| 26 |
--danger: #F07F5A;
|
| 27 |
}
|
|
@@ -81,16 +90,6 @@
|
|
| 81 |
z-index: 1;
|
| 82 |
}
|
| 83 |
|
| 84 |
-
#viewport a-scene {
|
| 85 |
-
width: 100%;
|
| 86 |
-
height: 100%;
|
| 87 |
-
}
|
| 88 |
-
|
| 89 |
-
#viewport .a-enter-vr,
|
| 90 |
-
#viewport .a-orientation-modal {
|
| 91 |
-
display: none !important;
|
| 92 |
-
}
|
| 93 |
-
|
| 94 |
.drop-zone {
|
| 95 |
position: absolute;
|
| 96 |
inset: 18px;
|
|
@@ -246,6 +245,41 @@
|
|
| 246 |
color: var(--danger);
|
| 247 |
}
|
| 248 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 249 |
.readout {
|
| 250 |
display: grid;
|
| 251 |
gap: 18px;
|
|
@@ -376,12 +410,12 @@
|
|
| 376 |
<div id="progress" class="progress"></div>
|
| 377 |
<div class="toolbar">
|
| 378 |
<button id="uploadButton" class="tool-button" type="button">Load</button>
|
| 379 |
-
<button id="playButton" class="tool-button" type="button" disabled>
|
| 380 |
</div>
|
| 381 |
-
<div id="viewport"></div>
|
| 382 |
-
<label id="dropZone" class="drop-zone" for="fileInput">Drop component photo</label>
|
| 383 |
<div id="scanLine" class="scan-line"></div>
|
| 384 |
-
<input id="fileInput" type="file" accept="image/*" hidden>
|
| 385 |
</section>
|
| 386 |
|
| 387 |
<aside class="panel">
|
|
@@ -389,9 +423,15 @@
|
|
| 389 |
<header class="brand">
|
| 390 |
<div class="kicker">Snap2Sim</div>
|
| 391 |
<h1>Inside the Machine</h1>
|
| 392 |
-
<div id="status" class="status">Standby</div>
|
|
|
|
| 393 |
</header>
|
| 394 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 395 |
<section class="readout">
|
| 396 |
<div class="metric-row">
|
| 397 |
<div id="component" class="component">Awaiting Photo</div>
|
|
@@ -426,6 +466,9 @@
|
|
| 426 |
const progress = document.getElementById("progress");
|
| 427 |
const scanLine = document.getElementById("scanLine");
|
| 428 |
const statusEl = document.getElementById("status");
|
|
|
|
|
|
|
|
|
|
| 429 |
const componentEl = document.getElementById("component");
|
| 430 |
const confidenceEl = document.getElementById("confidence");
|
| 431 |
const summaryEl = document.getElementById("summary");
|
|
@@ -438,8 +481,11 @@
|
|
| 438 |
let paused = false;
|
| 439 |
let coldStartTimer = 0;
|
| 440 |
let fallbackRuntime = null;
|
|
|
|
|
|
|
| 441 |
|
| 442 |
uploadButton.addEventListener("click", () => fileInput.click());
|
|
|
|
| 443 |
fileInput.addEventListener("change", () => {
|
| 444 |
const file = fileInput.files && fileInput.files[0];
|
| 445 |
if (file) runPipeline(file);
|
|
@@ -459,6 +505,13 @@
|
|
| 459 |
});
|
| 460 |
}
|
| 461 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 462 |
dropZone.addEventListener("drop", (event) => {
|
| 463 |
const file = event.dataTransfer.files && event.dataTransfer.files[0];
|
| 464 |
if (file) runPipeline(file);
|
|
@@ -467,15 +520,20 @@
|
|
| 467 |
playButton.addEventListener("click", () => {
|
| 468 |
paused = !paused;
|
| 469 |
playButton.textContent = paused ? "Resume" : "Pause";
|
| 470 |
-
if (activeMode === "aframe") {
|
| 471 |
-
const scene = viewport.querySelector("a-scene");
|
| 472 |
-
if (scene) paused ? scene.pause() : scene.play();
|
| 473 |
-
}
|
| 474 |
if (fallbackRuntime) fallbackRuntime.playing = !paused;
|
| 475 |
});
|
| 476 |
|
| 477 |
async function runPipeline(file) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 478 |
resetScene();
|
|
|
|
| 479 |
setBusy(true);
|
| 480 |
setStatus("ANALYZING ASSEMBLY...");
|
| 481 |
coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
|
|
@@ -487,12 +545,12 @@
|
|
| 487 |
populateAnalysis(analysis);
|
| 488 |
|
| 489 |
setStatus("RENDERING CUTAWAY...");
|
| 490 |
-
|
| 491 |
-
const sceneHtml = typeof sceneResponse === "string" ? sceneResponse : sceneResponse.html;
|
| 492 |
-
renderAframe(sceneHtml, analysis);
|
| 493 |
} catch (error) {
|
| 494 |
window.clearTimeout(coldStartTimer);
|
| 495 |
setStatus(error.message || String(error), true);
|
|
|
|
|
|
|
| 496 |
if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
|
| 497 |
} finally {
|
| 498 |
setBusy(false);
|
|
@@ -504,8 +562,9 @@
|
|
| 504 |
fallbackRuntime = null;
|
| 505 |
activeMode = "idle";
|
| 506 |
paused = false;
|
| 507 |
-
playButton.textContent = "
|
| 508 |
playButton.disabled = true;
|
|
|
|
| 509 |
viewport.replaceChildren();
|
| 510 |
dropZone.classList.add("hidden");
|
| 511 |
}
|
|
@@ -535,35 +594,8 @@
|
|
| 535 |
}));
|
| 536 |
}
|
| 537 |
|
| 538 |
-
function renderAframe(sceneHtml, analysis) {
|
| 539 |
-
if (!sceneHtml || !/<a-scene[\s>]/i.test(sceneHtml) || !/<\/a-scene>/i.test(sceneHtml)) {
|
| 540 |
-
buildDeterministicScene(analysis);
|
| 541 |
-
return;
|
| 542 |
-
}
|
| 543 |
-
|
| 544 |
-
activeMode = "aframe";
|
| 545 |
-
viewport.innerHTML = sceneHtml;
|
| 546 |
-
const scene = viewport.querySelector("a-scene");
|
| 547 |
-
if (scene) {
|
| 548 |
-
scene.setAttribute("embedded", "");
|
| 549 |
-
scene.setAttribute("vr-mode-ui", "enabled: false");
|
| 550 |
-
scene.setAttribute("device-orientation-permission-ui", "enabled: false");
|
| 551 |
-
scene.setAttribute("background", "color: #0F1318");
|
| 552 |
-
}
|
| 553 |
-
playButton.disabled = false;
|
| 554 |
-
revealScan();
|
| 555 |
-
|
| 556 |
-
window.setTimeout(() => {
|
| 557 |
-
const liveScene = viewport.querySelector("a-scene");
|
| 558 |
-
const hasCanvas = Boolean(liveScene && liveScene.canvas);
|
| 559 |
-
if (!hasCanvas) buildDeterministicScene(analysis);
|
| 560 |
-
}, 3000);
|
| 561 |
-
|
| 562 |
-
setStatus("CUTAWAY READY");
|
| 563 |
-
}
|
| 564 |
-
|
| 565 |
function buildDeterministicScene(analysis) {
|
| 566 |
-
const threeRuntime = window.THREE
|
| 567 |
if (!threeRuntime) {
|
| 568 |
setStatus("3D RUNTIME UNAVAILABLE", true);
|
| 569 |
return;
|
|
@@ -575,10 +607,16 @@
|
|
| 575 |
paused = false;
|
| 576 |
playButton.textContent = "Pause";
|
| 577 |
playButton.disabled = false;
|
| 578 |
-
viewport.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 579 |
|
| 580 |
-
const mount = viewport.querySelector(".scene-mount");
|
| 581 |
-
const labelLayer = viewport.querySelector(".label-layer");
|
| 582 |
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
|
| 583 |
renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
|
| 584 |
mount.appendChild(renderer.domElement);
|
|
@@ -667,6 +705,23 @@
|
|
| 667 |
setStatus("CUTAWAY READY");
|
| 668 |
}
|
| 669 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 670 |
function buildPartMesh(part, index) {
|
| 671 |
const geometryData = part.geometry || {};
|
| 672 |
const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
|
|
|
|
| 6 |
<title>Inside the Machine</title>
|
| 7 |
<link rel="preconnect" href="https://fonts.bunny.net">
|
| 8 |
<link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
|
| 9 |
+
<script type="importmap">
|
| 10 |
+
{
|
| 11 |
+
"imports": {
|
| 12 |
+
"three": "https://cdn.jsdelivr.net/npm/three@0.160.1/build/three.module.js",
|
| 13 |
+
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.160.1/examples/jsm/"
|
| 14 |
+
}
|
| 15 |
+
}
|
| 16 |
+
</script>
|
| 17 |
<script type="module">
|
| 18 |
+
import * as THREE from "three";
|
| 19 |
+
import { OrbitControls } from "three/addons/controls/OrbitControls.js";
|
| 20 |
+
|
| 21 |
+
window.THREE = { ...THREE, OrbitControls };
|
| 22 |
</script>
|
| 23 |
<style>
|
| 24 |
:root {
|
|
|
|
| 30 |
--cyan: #5FD4D0;
|
| 31 |
--cyan-dim: #2A5E5C;
|
| 32 |
--text: #C8C0AC;
|
| 33 |
+
--text-muted: #9CA3AF;
|
| 34 |
--grid: rgba(255,255,255,0.04);
|
| 35 |
--danger: #F07F5A;
|
| 36 |
}
|
|
|
|
| 90 |
z-index: 1;
|
| 91 |
}
|
| 92 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
.drop-zone {
|
| 94 |
position: absolute;
|
| 95 |
inset: 18px;
|
|
|
|
| 245 |
color: var(--danger);
|
| 246 |
}
|
| 247 |
|
| 248 |
+
.retry-button {
|
| 249 |
+
justify-self: start;
|
| 250 |
+
min-height: 32px;
|
| 251 |
+
border: 1px solid var(--danger);
|
| 252 |
+
border-radius: 0;
|
| 253 |
+
color: var(--danger);
|
| 254 |
+
background: transparent;
|
| 255 |
+
text-transform: uppercase;
|
| 256 |
+
letter-spacing: 0;
|
| 257 |
+
cursor: pointer;
|
| 258 |
+
}
|
| 259 |
+
|
| 260 |
+
.retry-button[hidden] {
|
| 261 |
+
display: none;
|
| 262 |
+
}
|
| 263 |
+
|
| 264 |
+
.source-card {
|
| 265 |
+
display: grid;
|
| 266 |
+
gap: 8px;
|
| 267 |
+
border-bottom: 1px solid rgba(95, 212, 208, 0.16);
|
| 268 |
+
padding-bottom: 14px;
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
.source-card[hidden] {
|
| 272 |
+
display: none;
|
| 273 |
+
}
|
| 274 |
+
|
| 275 |
+
.source-image {
|
| 276 |
+
width: 100%;
|
| 277 |
+
max-height: 150px;
|
| 278 |
+
object-fit: cover;
|
| 279 |
+
border: 1px solid rgba(122, 84, 32, 0.72);
|
| 280 |
+
background: var(--bg);
|
| 281 |
+
}
|
| 282 |
+
|
| 283 |
.readout {
|
| 284 |
display: grid;
|
| 285 |
gap: 18px;
|
|
|
|
| 410 |
<div id="progress" class="progress"></div>
|
| 411 |
<div class="toolbar">
|
| 412 |
<button id="uploadButton" class="tool-button" type="button">Load</button>
|
| 413 |
+
<button id="playButton" class="tool-button" type="button" disabled>Play</button>
|
| 414 |
</div>
|
| 415 |
+
<div id="viewport" aria-hidden="true"></div>
|
| 416 |
+
<label id="dropZone" class="drop-zone" for="fileInput" tabindex="0">Drop component photo</label>
|
| 417 |
<div id="scanLine" class="scan-line"></div>
|
| 418 |
+
<input id="fileInput" type="file" accept="image/*" capture="environment" hidden>
|
| 419 |
</section>
|
| 420 |
|
| 421 |
<aside class="panel">
|
|
|
|
| 423 |
<header class="brand">
|
| 424 |
<div class="kicker">Snap2Sim</div>
|
| 425 |
<h1>Inside the Machine</h1>
|
| 426 |
+
<div id="status" class="status" role="status" aria-live="polite">Standby</div>
|
| 427 |
+
<button id="retryButton" class="retry-button" type="button" hidden>Try another photo</button>
|
| 428 |
</header>
|
| 429 |
|
| 430 |
+
<section id="sourceCard" class="source-card" hidden>
|
| 431 |
+
<div class="label">Source Photo</div>
|
| 432 |
+
<img id="sourceImage" class="source-image" alt="Uploaded component photo preview">
|
| 433 |
+
</section>
|
| 434 |
+
|
| 435 |
<section class="readout">
|
| 436 |
<div class="metric-row">
|
| 437 |
<div id="component" class="component">Awaiting Photo</div>
|
|
|
|
| 466 |
const progress = document.getElementById("progress");
|
| 467 |
const scanLine = document.getElementById("scanLine");
|
| 468 |
const statusEl = document.getElementById("status");
|
| 469 |
+
const retryButton = document.getElementById("retryButton");
|
| 470 |
+
const sourceCard = document.getElementById("sourceCard");
|
| 471 |
+
const sourceImage = document.getElementById("sourceImage");
|
| 472 |
const componentEl = document.getElementById("component");
|
| 473 |
const confidenceEl = document.getElementById("confidence");
|
| 474 |
const summaryEl = document.getElementById("summary");
|
|
|
|
| 481 |
let paused = false;
|
| 482 |
let coldStartTimer = 0;
|
| 483 |
let fallbackRuntime = null;
|
| 484 |
+
let currentPreviewUrl = "";
|
| 485 |
+
const MAX_CLIENT_IMAGE_BYTES = 8 * 1024 * 1024;
|
| 486 |
|
| 487 |
uploadButton.addEventListener("click", () => fileInput.click());
|
| 488 |
+
retryButton.addEventListener("click", () => fileInput.click());
|
| 489 |
fileInput.addEventListener("change", () => {
|
| 490 |
const file = fileInput.files && fileInput.files[0];
|
| 491 |
if (file) runPipeline(file);
|
|
|
|
| 505 |
});
|
| 506 |
}
|
| 507 |
|
| 508 |
+
dropZone.addEventListener("keydown", (event) => {
|
| 509 |
+
if (event.key === "Enter" || event.key === " ") {
|
| 510 |
+
event.preventDefault();
|
| 511 |
+
fileInput.click();
|
| 512 |
+
}
|
| 513 |
+
});
|
| 514 |
+
|
| 515 |
dropZone.addEventListener("drop", (event) => {
|
| 516 |
const file = event.dataTransfer.files && event.dataTransfer.files[0];
|
| 517 |
if (file) runPipeline(file);
|
|
|
|
| 520 |
playButton.addEventListener("click", () => {
|
| 521 |
paused = !paused;
|
| 522 |
playButton.textContent = paused ? "Resume" : "Pause";
|
|
|
|
|
|
|
|
|
|
|
|
|
| 523 |
if (fallbackRuntime) fallbackRuntime.playing = !paused;
|
| 524 |
});
|
| 525 |
|
| 526 |
async function runPipeline(file) {
|
| 527 |
+
const validationError = validateFile(file);
|
| 528 |
+
if (validationError) {
|
| 529 |
+
setStatus(validationError, true);
|
| 530 |
+
retryButton.hidden = false;
|
| 531 |
+
dropZone.classList.remove("hidden");
|
| 532 |
+
return;
|
| 533 |
+
}
|
| 534 |
+
|
| 535 |
resetScene();
|
| 536 |
+
showSourcePreview(file);
|
| 537 |
setBusy(true);
|
| 538 |
setStatus("ANALYZING ASSEMBLY...");
|
| 539 |
coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
|
|
|
|
| 545 |
populateAnalysis(analysis);
|
| 546 |
|
| 547 |
setStatus("RENDERING CUTAWAY...");
|
| 548 |
+
buildDeterministicScene(analysis);
|
|
|
|
|
|
|
| 549 |
} catch (error) {
|
| 550 |
window.clearTimeout(coldStartTimer);
|
| 551 |
setStatus(error.message || String(error), true);
|
| 552 |
+
retryButton.hidden = false;
|
| 553 |
+
dropZone.classList.remove("hidden");
|
| 554 |
if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
|
| 555 |
} finally {
|
| 556 |
setBusy(false);
|
|
|
|
| 562 |
fallbackRuntime = null;
|
| 563 |
activeMode = "idle";
|
| 564 |
paused = false;
|
| 565 |
+
playButton.textContent = "Play";
|
| 566 |
playButton.disabled = true;
|
| 567 |
+
retryButton.hidden = true;
|
| 568 |
viewport.replaceChildren();
|
| 569 |
dropZone.classList.add("hidden");
|
| 570 |
}
|
|
|
|
| 594 |
}));
|
| 595 |
}
|
| 596 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 597 |
function buildDeterministicScene(analysis) {
|
| 598 |
+
const threeRuntime = window.THREE;
|
| 599 |
if (!threeRuntime) {
|
| 600 |
setStatus("3D RUNTIME UNAVAILABLE", true);
|
| 601 |
return;
|
|
|
|
| 607 |
paused = false;
|
| 608 |
playButton.textContent = "Pause";
|
| 609 |
playButton.disabled = false;
|
| 610 |
+
viewport.replaceChildren();
|
| 611 |
+
const stage = document.createElement("div");
|
| 612 |
+
stage.className = "fallback-stage";
|
| 613 |
+
const mount = document.createElement("div");
|
| 614 |
+
mount.className = "scene-mount";
|
| 615 |
+
const labelLayer = document.createElement("div");
|
| 616 |
+
labelLayer.className = "label-layer";
|
| 617 |
+
stage.append(mount, labelLayer);
|
| 618 |
+
viewport.append(stage);
|
| 619 |
|
|
|
|
|
|
|
| 620 |
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
|
| 621 |
renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
|
| 622 |
mount.appendChild(renderer.domElement);
|
|
|
|
| 705 |
setStatus("CUTAWAY READY");
|
| 706 |
}
|
| 707 |
|
| 708 |
+
function validateFile(file) {
|
| 709 |
+
if (!file.type || !file.type.startsWith("image/")) {
|
| 710 |
+
return "Upload an image file.";
|
| 711 |
+
}
|
| 712 |
+
if (file.size > MAX_CLIENT_IMAGE_BYTES) {
|
| 713 |
+
return "Image must be 8 MB or smaller.";
|
| 714 |
+
}
|
| 715 |
+
return "";
|
| 716 |
+
}
|
| 717 |
+
|
| 718 |
+
function showSourcePreview(file) {
|
| 719 |
+
if (currentPreviewUrl) URL.revokeObjectURL(currentPreviewUrl);
|
| 720 |
+
currentPreviewUrl = URL.createObjectURL(file);
|
| 721 |
+
sourceImage.src = currentPreviewUrl;
|
| 722 |
+
sourceCard.hidden = false;
|
| 723 |
+
}
|
| 724 |
+
|
| 725 |
function buildPartMesh(part, index) {
|
| 726 |
const geometryData = part.geometry || {};
|
| 727 |
const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
|
modal_app.py
CHANGED
|
@@ -16,9 +16,8 @@ import secrets as token_secrets
|
|
| 16 |
import modal
|
| 17 |
from fastapi import Header, HTTPException
|
| 18 |
|
| 19 |
-
from snap2sim.
|
| 20 |
-
from snap2sim.
|
| 21 |
-
from snap2sim.prompts import build_scene_prompt, build_vision_prompt
|
| 22 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 23 |
|
| 24 |
|
|
@@ -29,6 +28,9 @@ DEFAULT_RUNTIME_MODE = "placeholder"
|
|
| 29 |
CACHE_DIR = "/cache"
|
| 30 |
HF_CACHE_DIR = f"{CACHE_DIR}/huggingface"
|
| 31 |
MODEL_ASSET_DIR = f"{CACHE_DIR}/models"
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
|
| 34 |
model_cache = modal.Volume.from_name("snap2sim-hf-cache", create_if_missing=True)
|
|
@@ -81,14 +83,12 @@ def check_remote_imports() -> dict[str, Any]:
|
|
| 81 |
"""Lightweight Modal check that local project modules are packaged."""
|
| 82 |
import snap2sim.model_io
|
| 83 |
import snap2sim.schema
|
| 84 |
-
import snap2sim.aframe_scene
|
| 85 |
|
| 86 |
return {
|
| 87 |
"ok": True,
|
| 88 |
"modules": [
|
| 89 |
snap2sim.model_io.__name__,
|
| 90 |
snap2sim.schema.__name__,
|
| 91 |
-
snap2sim.aframe_scene.__name__,
|
| 92 |
],
|
| 93 |
}
|
| 94 |
|
|
@@ -191,12 +191,15 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
|
|
| 191 |
"""Run one image prompt through llama.cpp's multimodal CLI on a GPU."""
|
| 192 |
import base64
|
| 193 |
import subprocess
|
|
|
|
| 194 |
import time
|
| 195 |
|
| 196 |
from PIL import Image, ImageDraw
|
| 197 |
|
| 198 |
model_path, mmproj_path = ensure_runtime_assets()
|
| 199 |
-
|
|
|
|
|
|
|
| 200 |
img = Image.new("RGB", (512, 384), "#d8d0bd")
|
| 201 |
draw = ImageDraw.Draw(img)
|
| 202 |
draw.rectangle((82, 96, 430, 288), outline="#2c3138", width=8)
|
|
@@ -265,6 +268,8 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
|
|
| 265 |
and "image input is not supported" not in combined_output.lower()
|
| 266 |
and "failed to load projector" not in combined_output.lower()
|
| 267 |
)
|
|
|
|
|
|
|
| 268 |
return {
|
| 269 |
"ok": image_supported and valid_json,
|
| 270 |
"image_supported": image_supported,
|
|
@@ -275,7 +280,7 @@ def smoke_test_llamacpp_image() -> dict[str, Any]:
|
|
| 275 |
"elapsed_seconds": elapsed_seconds,
|
| 276 |
"model_path": str(model_path),
|
| 277 |
"mmproj_path": str(mmproj_path),
|
| 278 |
-
"image_base64_prefix":
|
| 279 |
"stdout_tail": stdout[-4000:],
|
| 280 |
"stderr_tail": stderr[-4000:],
|
| 281 |
}
|
|
@@ -327,35 +332,66 @@ def run_llamacpp_prompt(
|
|
| 327 |
def write_payload_image(payload: dict[str, Any]) -> Path:
|
| 328 |
"""Decode an image_base64 payload into a temporary RGB JPEG."""
|
| 329 |
import base64
|
|
|
|
|
|
|
|
|
|
| 330 |
from io import BytesIO
|
| 331 |
|
| 332 |
-
from PIL import Image
|
| 333 |
|
|
|
|
| 334 |
image_base64 = payload.get("image_base64")
|
| 335 |
if not isinstance(image_base64, str) or not image_base64:
|
| 336 |
raise ValueError("Request payload must include image_base64.")
|
| 337 |
if "," in image_base64 and image_base64.lstrip().startswith("data:"):
|
| 338 |
image_base64 = image_base64.split(",", 1)[1]
|
|
|
|
|
|
|
| 339 |
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 343 |
image.save(path, format="JPEG", quality=92)
|
| 344 |
return path
|
| 345 |
|
| 346 |
|
| 347 |
def analyze_image_llamacpp_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
| 348 |
image_path = write_payload_image(payload)
|
| 349 |
-
response = run_llamacpp_prompt(
|
| 350 |
-
build_vision_prompt(),
|
| 351 |
-
image_path=image_path,
|
| 352 |
-
max_tokens=1536,
|
| 353 |
-
timeout_seconds=180,
|
| 354 |
-
)
|
| 355 |
try:
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 359 |
|
| 360 |
|
| 361 |
@app.local_entrypoint()
|
|
@@ -447,23 +483,20 @@ def analyze_image_llamacpp(payload: dict[str, Any], authorization: str = Header(
|
|
| 447 |
|
| 448 |
@app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
|
| 449 |
@modal.fastapi_endpoint(method="POST")
|
| 450 |
-
def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str,
|
| 451 |
require_authorization(authorization)
|
| 452 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 453 |
-
_prompt = build_scene_prompt(analysis)
|
| 454 |
if runtime_config()["runtime_mode"] != "placeholder":
|
| 455 |
raise NotImplementedError(
|
| 456 |
-
"
|
| 457 |
-
"mode active until the llama.cpp/vLLM adapter is implemented."
|
| 458 |
)
|
| 459 |
-
return {"
|
| 460 |
|
| 461 |
|
| 462 |
@app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
|
| 463 |
@modal.fastapi_endpoint(method="POST")
|
| 464 |
-
def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str,
|
| 465 |
-
"""
|
| 466 |
require_authorization(authorization)
|
| 467 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 468 |
-
|
| 469 |
-
return {"html": parse_scene_response(response)}
|
|
|
|
| 16 |
import modal
|
| 17 |
from fastapi import Header, HTTPException
|
| 18 |
|
| 19 |
+
from snap2sim.model_io import coerce_analysis_response, parse_analysis_response
|
| 20 |
+
from snap2sim.prompts import build_vision_prompt
|
|
|
|
| 21 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 22 |
|
| 23 |
|
|
|
|
| 28 |
CACHE_DIR = "/cache"
|
| 29 |
HF_CACHE_DIR = f"{CACHE_DIR}/huggingface"
|
| 30 |
MODEL_ASSET_DIR = f"{CACHE_DIR}/models"
|
| 31 |
+
MAX_IMAGE_BASE64_CHARS = 12 * 1024 * 1024
|
| 32 |
+
MAX_IMAGE_BYTES = 9 * 1024 * 1024
|
| 33 |
+
MAX_IMAGE_PIXELS = 12_000_000
|
| 34 |
|
| 35 |
|
| 36 |
model_cache = modal.Volume.from_name("snap2sim-hf-cache", create_if_missing=True)
|
|
|
|
| 83 |
"""Lightweight Modal check that local project modules are packaged."""
|
| 84 |
import snap2sim.model_io
|
| 85 |
import snap2sim.schema
|
|
|
|
| 86 |
|
| 87 |
return {
|
| 88 |
"ok": True,
|
| 89 |
"modules": [
|
| 90 |
snap2sim.model_io.__name__,
|
| 91 |
snap2sim.schema.__name__,
|
|
|
|
| 92 |
],
|
| 93 |
}
|
| 94 |
|
|
|
|
| 191 |
"""Run one image prompt through llama.cpp's multimodal CLI on a GPU."""
|
| 192 |
import base64
|
| 193 |
import subprocess
|
| 194 |
+
import tempfile
|
| 195 |
import time
|
| 196 |
|
| 197 |
from PIL import Image, ImageDraw
|
| 198 |
|
| 199 |
model_path, mmproj_path = ensure_runtime_assets()
|
| 200 |
+
temp_file = tempfile.NamedTemporaryFile(prefix="snap2sim-smoke-", suffix=".jpg", delete=False)
|
| 201 |
+
temp_file.close()
|
| 202 |
+
test_image = Path(temp_file.name)
|
| 203 |
img = Image.new("RGB", (512, 384), "#d8d0bd")
|
| 204 |
draw = ImageDraw.Draw(img)
|
| 205 |
draw.rectangle((82, 96, 430, 288), outline="#2c3138", width=8)
|
|
|
|
| 268 |
and "image input is not supported" not in combined_output.lower()
|
| 269 |
and "failed to load projector" not in combined_output.lower()
|
| 270 |
)
|
| 271 |
+
image_base64_prefix = base64.b64encode(test_image.read_bytes()).decode("ascii")[:80]
|
| 272 |
+
test_image.unlink(missing_ok=True)
|
| 273 |
return {
|
| 274 |
"ok": image_supported and valid_json,
|
| 275 |
"image_supported": image_supported,
|
|
|
|
| 280 |
"elapsed_seconds": elapsed_seconds,
|
| 281 |
"model_path": str(model_path),
|
| 282 |
"mmproj_path": str(mmproj_path),
|
| 283 |
+
"image_base64_prefix": image_base64_prefix,
|
| 284 |
"stdout_tail": stdout[-4000:],
|
| 285 |
"stderr_tail": stderr[-4000:],
|
| 286 |
}
|
|
|
|
| 332 |
def write_payload_image(payload: dict[str, Any]) -> Path:
|
| 333 |
"""Decode an image_base64 payload into a temporary RGB JPEG."""
|
| 334 |
import base64
|
| 335 |
+
import binascii
|
| 336 |
+
import tempfile
|
| 337 |
+
import warnings
|
| 338 |
from io import BytesIO
|
| 339 |
|
| 340 |
+
from PIL import Image, UnidentifiedImageError
|
| 341 |
|
| 342 |
+
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
|
| 343 |
image_base64 = payload.get("image_base64")
|
| 344 |
if not isinstance(image_base64, str) or not image_base64:
|
| 345 |
raise ValueError("Request payload must include image_base64.")
|
| 346 |
if "," in image_base64 and image_base64.lstrip().startswith("data:"):
|
| 347 |
image_base64 = image_base64.split(",", 1)[1]
|
| 348 |
+
if len(image_base64) > MAX_IMAGE_BASE64_CHARS:
|
| 349 |
+
raise ValueError("Image upload is too large.")
|
| 350 |
|
| 351 |
+
try:
|
| 352 |
+
raw = base64.b64decode(image_base64, validate=True)
|
| 353 |
+
except (binascii.Error, ValueError) as exc:
|
| 354 |
+
raise ValueError("Image payload is not valid base64.") from exc
|
| 355 |
+
if len(raw) > MAX_IMAGE_BYTES:
|
| 356 |
+
raise ValueError("Image upload is too large.")
|
| 357 |
+
|
| 358 |
+
try:
|
| 359 |
+
with warnings.catch_warnings():
|
| 360 |
+
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
| 361 |
+
image = Image.open(BytesIO(raw))
|
| 362 |
+
image.load()
|
| 363 |
+
except Image.DecompressionBombWarning as exc:
|
| 364 |
+
raise ValueError("Image dimensions are too large.") from exc
|
| 365 |
+
except Image.DecompressionBombError as exc:
|
| 366 |
+
raise ValueError("Image dimensions are too large.") from exc
|
| 367 |
+
except (UnidentifiedImageError, OSError, ValueError) as exc:
|
| 368 |
+
raise ValueError("Upload a valid image file.") from exc
|
| 369 |
+
if image.width * image.height > MAX_IMAGE_PIXELS:
|
| 370 |
+
raise ValueError("Image dimensions are too large.")
|
| 371 |
+
|
| 372 |
+
temp_file = tempfile.NamedTemporaryFile(prefix="snap2sim-request-", suffix=".jpg", delete=False)
|
| 373 |
+
temp_file.close()
|
| 374 |
+
path = Path(temp_file.name)
|
| 375 |
+
image = image.convert("RGB")
|
| 376 |
image.save(path, format="JPEG", quality=92)
|
| 377 |
return path
|
| 378 |
|
| 379 |
|
| 380 |
def analyze_image_llamacpp_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
| 381 |
image_path = write_payload_image(payload)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 382 |
try:
|
| 383 |
+
response = run_llamacpp_prompt(
|
| 384 |
+
build_vision_prompt(),
|
| 385 |
+
image_path=image_path,
|
| 386 |
+
max_tokens=1536,
|
| 387 |
+
timeout_seconds=180,
|
| 388 |
+
)
|
| 389 |
+
try:
|
| 390 |
+
return parse_analysis_response(response)
|
| 391 |
+
except Exception:
|
| 392 |
+
return coerce_analysis_response(response)
|
| 393 |
+
finally:
|
| 394 |
+
image_path.unlink(missing_ok=True)
|
| 395 |
|
| 396 |
|
| 397 |
@app.local_entrypoint()
|
|
|
|
| 483 |
|
| 484 |
@app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
|
| 485 |
@modal.fastapi_endpoint(method="POST")
|
| 486 |
+
def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, Any]:
|
| 487 |
require_authorization(authorization)
|
| 488 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
|
|
|
| 489 |
if runtime_config()["runtime_mode"] != "placeholder":
|
| 490 |
raise NotImplementedError(
|
| 491 |
+
"Scene generation is deterministic in the browser from validated JSON."
|
|
|
|
| 492 |
)
|
| 493 |
+
return {"renderer": "three", "analysis": analysis}
|
| 494 |
|
| 495 |
|
| 496 |
@app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
|
| 497 |
@modal.fastapi_endpoint(method="POST")
|
| 498 |
+
def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, Any]:
|
| 499 |
+
"""Compatibility endpoint; scene rendering is deterministic browser-side."""
|
| 500 |
require_authorization(authorization)
|
| 501 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 502 |
+
return {"renderer": "three", "analysis": analysis}
|
|
|
snap2sim/aframe_scene.py
DELETED
|
@@ -1,120 +0,0 @@
|
|
| 1 |
-
"""Deterministic A-Frame scene generation from a validated mechanism payload."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
import html
|
| 6 |
-
from typing import Any
|
| 7 |
-
|
| 8 |
-
from snap2sim.schema import validate_analysis
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
def build_aframe_scene(analysis: dict[str, Any]) -> str:
|
| 12 |
-
"""Return a declarative A-Frame scene for the browser to inject."""
|
| 13 |
-
valid_analysis = validate_analysis(analysis)
|
| 14 |
-
title = html.escape(str(valid_analysis.get("component", "mechanism")).upper(), quote=True)
|
| 15 |
-
summary = html.escape(str(valid_analysis.get("summary", "")), quote=True)
|
| 16 |
-
entities = "\n".join(_part_entity(part, index) for index, part in enumerate(valid_analysis["parts"][:6]))
|
| 17 |
-
return f"""<a-scene embedded vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false" background="color: #0F1318" renderer="colorManagement: true">
|
| 18 |
-
<a-sky color="#0F1318"></a-sky>
|
| 19 |
-
<a-entity light="type: ambient; intensity: 0.55; color: #5FD4D0"></a-entity>
|
| 20 |
-
<a-entity light="type: directional; intensity: 1.35; color: #E8A33D" position="3 5 4"></a-entity>
|
| 21 |
-
<a-entity position="0 1.1 5.6">
|
| 22 |
-
<a-camera look-controls wasd-controls="enabled: false"></a-camera>
|
| 23 |
-
</a-entity>
|
| 24 |
-
<a-plane color="#111820" opacity="0.72" rotation="-90 0 0" position="0 -1.05 -0.2" width="8" height="8"></a-plane>
|
| 25 |
-
<a-text value="{title}" position="-2.8 2.35 -2.6" color="#C8C0AC" width="4.8" align="left"></a-text>
|
| 26 |
-
<a-text value="{summary}" position="-2.8 2.05 -2.6" color="#6B7280" width="5.2" align="left"></a-text>
|
| 27 |
-
{entities}
|
| 28 |
-
</a-scene>"""
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
def _part_entity(part: dict[str, Any], index: int) -> str:
|
| 32 |
-
geometry = part["geometry"]
|
| 33 |
-
shape = geometry["shape"]
|
| 34 |
-
size = geometry["size"]
|
| 35 |
-
position = _vec(geometry["position"])
|
| 36 |
-
rotation = _rotation(geometry.get("rotation", [0, 0, 0]))
|
| 37 |
-
color = _color(geometry.get("color"), index)
|
| 38 |
-
label = html.escape(str(part.get("name", part.get("id", "part"))), quote=True)
|
| 39 |
-
animation = _animation(part)
|
| 40 |
-
|
| 41 |
-
if shape in {"cylinder", "gear"}:
|
| 42 |
-
radius = max(float(size[0]), float(size[2])) / 2
|
| 43 |
-
primitive = (
|
| 44 |
-
f'<a-cylinder radius="{radius:.3f}" height="{float(size[1]):.3f}" '
|
| 45 |
-
f'segments-radial="{int(geometry.get("teeth", 32)) if shape == "gear" else 48}"'
|
| 46 |
-
)
|
| 47 |
-
elif shape == "sphere":
|
| 48 |
-
primitive = f'<a-sphere radius="{max(float(item) for item in size) / 2:.3f}"'
|
| 49 |
-
elif shape == "rod":
|
| 50 |
-
primitive = f'<a-cylinder radius="{max(float(size[0]), float(size[1])) / 2:.3f}" height="{float(size[2]):.3f}"'
|
| 51 |
-
rotation = _rotation([1.5708, 0, 0])
|
| 52 |
-
else:
|
| 53 |
-
primitive = f'<a-box width="{float(size[0]):.3f}" height="{float(size[1]):.3f}" depth="{float(size[2]):.3f}"'
|
| 54 |
-
|
| 55 |
-
return f""" <a-entity position="{position}">
|
| 56 |
-
{primitive} material="color: {color}; metalness: 0.45; roughness: 0.38" rotation="{rotation}" {animation}></a-cylinder>
|
| 57 |
-
<a-text value="{label}" position="0 {float(size[1]) + 0.28:.3f} 0" color="#5FD4D0" width="2.4" align="center"></a-text>
|
| 58 |
-
</a-entity>""".replace("</a-cylinder>", _closing_tag(shape))
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
def _closing_tag(shape: str) -> str:
|
| 62 |
-
if shape == "sphere":
|
| 63 |
-
return "</a-sphere>"
|
| 64 |
-
if shape == "box":
|
| 65 |
-
return "</a-box>"
|
| 66 |
-
return "</a-cylinder>"
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
def _animation(part: dict[str, Any]) -> str:
|
| 70 |
-
motion = part.get("motion") or {"type": "static"}
|
| 71 |
-
motion_type = motion.get("type")
|
| 72 |
-
if motion_type == "static":
|
| 73 |
-
return ""
|
| 74 |
-
|
| 75 |
-
speed = max(0.1, abs(float(motion.get("speed", 1))))
|
| 76 |
-
duration = int(max(700, min(9000, 3600 / speed)))
|
| 77 |
-
axis = motion.get("axis") if isinstance(motion.get("axis"), list) else [0, 1, 0]
|
| 78 |
-
|
| 79 |
-
if motion_type == "translate":
|
| 80 |
-
move_range = motion.get("range") if isinstance(motion.get("range"), list) else [-0.25, 0.25]
|
| 81 |
-
target = [float(axis[i]) * float(move_range[-1]) for i in range(3)]
|
| 82 |
-
return (
|
| 83 |
-
f'animation="property: position; to: {_vec(target)}; dir: alternate; '
|
| 84 |
-
f'loop: true; dur: {duration}; easing: easeInOutSine"'
|
| 85 |
-
)
|
| 86 |
-
|
| 87 |
-
rotation = [0, 0, 0]
|
| 88 |
-
if abs(float(axis[0])) >= max(abs(float(axis[1])), abs(float(axis[2]))):
|
| 89 |
-
rotation[0] = 360
|
| 90 |
-
elif abs(float(axis[2])) >= abs(float(axis[1])):
|
| 91 |
-
rotation[2] = 360
|
| 92 |
-
else:
|
| 93 |
-
rotation[1] = 360
|
| 94 |
-
if motion_type == "oscillate":
|
| 95 |
-
amplitude = int(float(motion.get("amplitude", 0.25)) * 90)
|
| 96 |
-
rotation = [value and amplitude for value in rotation]
|
| 97 |
-
return (
|
| 98 |
-
f'animation="property: rotation; to: {_vec(rotation)}; dir: alternate; '
|
| 99 |
-
f'loop: true; dur: {duration}; easing: easeInOutSine"'
|
| 100 |
-
)
|
| 101 |
-
return f'animation="property: rotation; to: {_vec(rotation)}; loop: true; dur: {duration}; easing: linear"'
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
def _vec(values: list[Any]) -> str:
|
| 105 |
-
return " ".join(f"{float(value):.3f}" for value in values)
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
def _rotation(values: list[Any]) -> str:
|
| 109 |
-
return " ".join(f"{float(value) * 57.2958:.2f}" for value in values)
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
def _color(value: Any, index: int) -> str:
|
| 113 |
-
key = str(value or "").lower()
|
| 114 |
-
if "amber" in key or "orange" in key:
|
| 115 |
-
return "#E8A33D"
|
| 116 |
-
if "cyan" in key:
|
| 117 |
-
return "#5FD4D0"
|
| 118 |
-
if "steel" in key:
|
| 119 |
-
return "#9AA4A6"
|
| 120 |
-
return ["#5FD4D0", "#E8A33D", "#C8C0AC", "#80B8FF", "#F07F5A", "#A7D676"][index % 6]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
snap2sim/backend.py
CHANGED
|
@@ -11,7 +11,6 @@ from typing import Any
|
|
| 11 |
import requests
|
| 12 |
from PIL import Image
|
| 13 |
|
| 14 |
-
from snap2sim.aframe_scene import build_aframe_scene
|
| 15 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 16 |
|
| 17 |
|
|
@@ -45,16 +44,9 @@ class InferenceClient:
|
|
| 45 |
|
| 46 |
return validate_analysis(dict(EXAMPLE_ANALYSIS))
|
| 47 |
|
| 48 |
-
def generate_scene(self, analysis: dict[str, Any]) -> str:
|
| 49 |
valid_analysis = validate_analysis(analysis)
|
| 50 |
-
|
| 51 |
-
response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
|
| 52 |
-
html = response.get("html", "")
|
| 53 |
-
if not html:
|
| 54 |
-
raise RuntimeError("Modal response did not include generated HTML.")
|
| 55 |
-
return str(html)
|
| 56 |
-
|
| 57 |
-
return build_aframe_scene(valid_analysis)
|
| 58 |
|
| 59 |
def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
|
| 60 |
if not url:
|
|
|
|
| 11 |
import requests
|
| 12 |
from PIL import Image
|
| 13 |
|
|
|
|
| 14 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 15 |
|
| 16 |
|
|
|
|
| 44 |
|
| 45 |
return validate_analysis(dict(EXAMPLE_ANALYSIS))
|
| 46 |
|
| 47 |
+
def generate_scene(self, analysis: dict[str, Any]) -> dict[str, Any]:
|
| 48 |
valid_analysis = validate_analysis(analysis)
|
| 49 |
+
return {"renderer": "three", "analysis": valid_analysis}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
|
| 52 |
if not url:
|
snap2sim/model_io.py
CHANGED
|
@@ -46,17 +46,6 @@ def coerce_analysis_response(text: str) -> dict[str, Any]:
|
|
| 46 |
return validate_analysis(_generic_analysis(fallback_component))
|
| 47 |
|
| 48 |
|
| 49 |
-
def parse_scene_response(text: str) -> str:
|
| 50 |
-
"""Extract a complete A-Frame scene block from a model response."""
|
| 51 |
-
raw = _strip_fences(text).strip()
|
| 52 |
-
lowered = raw.lower()
|
| 53 |
-
scene_start = lowered.find("<a-scene")
|
| 54 |
-
scene_end = lowered.rfind("</a-scene>")
|
| 55 |
-
if scene_start < 0 or scene_end < 0:
|
| 56 |
-
raise ValueError("Model response did not contain a complete <a-scene> block.")
|
| 57 |
-
return raw[scene_start : scene_end + len("</a-scene>")]
|
| 58 |
-
|
| 59 |
-
|
| 60 |
def _strip_fences(text: str) -> str:
|
| 61 |
return _FENCE_RE.sub("", text.strip()).strip()
|
| 62 |
|
|
|
|
| 46 |
return validate_analysis(_generic_analysis(fallback_component))
|
| 47 |
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
def _strip_fences(text: str) -> str:
|
| 50 |
return _FENCE_RE.sub("", text.strip()).strip()
|
| 51 |
|
snap2sim/prompts.py
CHANGED
|
@@ -1,11 +1,7 @@
|
|
| 1 |
-
"""Prompt templates for
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
| 5 |
-
import json
|
| 6 |
-
from typing import Any
|
| 7 |
-
|
| 8 |
-
|
| 9 |
VISION_SYSTEM_PROMPT = """You are a mechanical teardown analyst.
|
| 10 |
Given an image of a hardware component, infer the most likely internal
|
| 11 |
mechanism and return only JSON matching the provided schema. Prefer clear,
|
|
@@ -18,7 +14,7 @@ def build_vision_prompt() -> str:
|
|
| 18 |
"Analyze the uploaded hardware component as a cutaway mechanism. "
|
| 19 |
"Answer with only one JSON object. Do not include markdown. Do not "
|
| 20 |
"include a reasoning trace or <think> tags. Keep the payload compact and physically "
|
| 21 |
-
"plausible for primitive
|
| 22 |
"Required top-level keys: component, confidence, summary, trigger, "
|
| 23 |
"motion_sequence, parts.\n"
|
| 24 |
"Each part requires: id, name, role, geometry, motion.\n"
|
|
@@ -46,25 +42,3 @@ def build_vision_prompt() -> str:
|
|
| 46 |
"fields: axis, speed, amplitude, phase, range. Include optional fields "
|
| 47 |
"only when useful."
|
| 48 |
)
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
AFRAME_SYSTEM_PROMPT = """You generate declarative A-Frame cutaway scenes.
|
| 52 |
-
Return only one <a-scene>...</a-scene> block. Use primitive A-Frame entities,
|
| 53 |
-
short labels, and animation attributes. Do not include scripts, markdown fences,
|
| 54 |
-
explanations, or a complete HTML document."""
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
def build_scene_prompt(analysis: dict[str, Any]) -> str:
|
| 58 |
-
return (
|
| 59 |
-
"Build a technical cutaway / field manual A-Frame animation for this "
|
| 60 |
-
"mechanism analysis. Return only the <a-scene>...</a-scene> block. "
|
| 61 |
-
"Use 3 to 6 visible primitive parts maximum, selected from <a-box>, "
|
| 62 |
-
"<a-cylinder>, <a-sphere>, <a-torus>, <a-cone>, and <a-entity>. "
|
| 63 |
-
"Include <a-sky color=\"#0F1318\">, a camera, lights, short <a-text> "
|
| 64 |
-
"labels, warm amber annotations, and cool cyan moving parts. Use "
|
| 65 |
-
"A-Frame animation attributes for motion, for example "
|
| 66 |
-
"animation=\"property: rotation; to: 0 360 0; loop: true; dur: 2000; "
|
| 67 |
-
"easing: linear\". Do not include scripts, markdown, explanations, "
|
| 68 |
-
"or a complete HTML document.\n\n"
|
| 69 |
-
f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
|
| 70 |
-
)
|
|
|
|
| 1 |
+
"""Prompt templates for model calls."""
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
VISION_SYSTEM_PROMPT = """You are a mechanical teardown analyst.
|
| 6 |
Given an image of a hardware component, infer the most likely internal
|
| 7 |
mechanism and return only JSON matching the provided schema. Prefer clear,
|
|
|
|
| 14 |
"Analyze the uploaded hardware component as a cutaway mechanism. "
|
| 15 |
"Answer with only one JSON object. Do not include markdown. Do not "
|
| 16 |
"include a reasoning trace or <think> tags. Keep the payload compact and physically "
|
| 17 |
+
"plausible for primitive Three.js rendering.\n\n"
|
| 18 |
"Required top-level keys: component, confidence, summary, trigger, "
|
| 19 |
"motion_sequence, parts.\n"
|
| 20 |
"Each part requires: id, name, role, geometry, motion.\n"
|
|
|
|
| 42 |
"fields: axis, speed, amplitude, phase, range. Include optional fields "
|
| 43 |
"only when useful."
|
| 44 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|