Spaces:
Running
Running
jasondo OpenAI Codex commited on
Commit ·
53f7d61
1
Parent(s): 2499b0f
Migrate Snap2Sim to trusted HTML scene shell
Browse filesCo-authored-by: OpenAI Codex <codex@openai.com>
- .github/workflows/sync_to_hf.yml +11 -1
- AGENTS.md +67 -26
- PROMPT.md +343 -185
- README.md +62 -16
- SECURITY.md +6 -0
- app.py +90 -5
- index.html +808 -0
- modal_app.py +13 -13
- requirements.txt +1 -1
- snap2sim/aframe_scene.py +120 -0
- snap2sim/backend.py +3 -3
- snap2sim/fallback_scene.py +0 -72
- snap2sim/model_io.py +8 -13
- snap2sim/prompts.py +16 -10
- snap2sim/three_scene.py +0 -397
- snap2sim/ui.py +0 -225
.github/workflows/sync_to_hf.yml
CHANGED
|
@@ -57,4 +57,14 @@ jobs:
|
|
| 57 |
run: git lfs push huggingface --all
|
| 58 |
|
| 59 |
- name: Force-sync main to Hugging Face
|
| 60 |
-
run:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
run: git lfs push huggingface --all
|
| 58 |
|
| 59 |
- name: Force-sync main to Hugging Face
|
| 60 |
+
run: |
|
| 61 |
+
for attempt in 1 2 3 4 5; do
|
| 62 |
+
if git push --force huggingface HEAD:main; then
|
| 63 |
+
exit 0
|
| 64 |
+
fi
|
| 65 |
+
delay=$((attempt * 20))
|
| 66 |
+
echo "Hugging Face push failed on attempt ${attempt}; retrying in ${delay}s..." >&2
|
| 67 |
+
sleep "$delay"
|
| 68 |
+
done
|
| 69 |
+
echo "Hugging Face push failed after 5 attempts." >&2
|
| 70 |
+
exit 1
|
AGENTS.md
CHANGED
|
@@ -2,10 +2,10 @@
|
|
| 2 |
|
| 3 |
## Project Overview
|
| 4 |
|
| 5 |
-
Snap2Sim / Inside the Machine is a
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
|
| 10 |
## Runtime Notes
|
| 11 |
|
|
@@ -18,8 +18,17 @@ annotated technical cutaway animation.
|
|
| 18 |
- Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
|
| 19 |
through Transformers/custom code for the vision step if GGUF endpoint quality
|
| 20 |
is not reliable enough for the demo.
|
| 21 |
-
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
## Project Structure
|
| 25 |
|
|
@@ -33,25 +42,25 @@ annotated technical cutaway animation.
|
|
| 33 |
GitHub `main` to an existing Hugging Face Space.
|
| 34 |
- `requirements.txt` - Python dependencies for the Gradio Space and Modal
|
| 35 |
scaffold.
|
| 36 |
-
- `app.py` - Hugging Face Space entry point
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
- `modal_app.py` - Modal app scaffold with runtime asset caching, a
|
| 38 |
llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
|
| 39 |
-
placeholder plus experimental llama.cpp `analyze_image` / `
|
| 40 |
web endpoints.
|
| 41 |
- `scripts/verify_runtime_assets.py` - Hugging Face metadata preflight for the
|
| 42 |
selected GGUF quant and `mmproj` file.
|
| 43 |
- `snap2sim/__init__.py` - package marker.
|
| 44 |
- `snap2sim/backend.py` - backend config, local placeholder inference, Modal
|
| 45 |
HTTP client, and image base64 encoding.
|
| 46 |
-
- `snap2sim/
|
| 47 |
-
|
| 48 |
-
- `snap2sim/prompts.py` - prompt templates for the vision analysis and
|
| 49 |
-
|
| 50 |
- `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
|
| 51 |
-
- `snap2sim/three_scene.py` - deterministic Three.js scene generation from a
|
| 52 |
-
validated mechanism payload.
|
| 53 |
-
- `snap2sim/ui.py` - Gradio Blocks UI, blueprint field-manual CSS theme, and
|
| 54 |
-
pipeline orchestration.
|
| 55 |
|
| 56 |
## What Has Been Done
|
| 57 |
|
|
@@ -88,8 +97,8 @@ annotated technical cutaway animation.
|
|
| 88 |
HTML documents.
|
| 89 |
- Added a local coercion fallback for verbose/partial model JSON so the
|
| 90 |
analysis endpoint does not launch a second slow repair generation.
|
| 91 |
-
- Added experimental `analyze_image_llamacpp` and
|
| 92 |
-
|
| 93 |
`MODAL_ANALYZE_URL` / `MODAL_GENERATE_URL` at them only after the smoke test
|
| 94 |
passes.
|
| 95 |
- Fixed Modal packaging for local `snap2sim` imports by adding
|
|
@@ -102,7 +111,8 @@ annotated technical cutaway animation.
|
|
| 102 |
- Deployed `modal_app.py` to Modal at
|
| 103 |
`https://modal.com/apps/bigstonks1/main/deployed/snap2sim-inside-the-machine`.
|
| 104 |
- Production endpoint checks passed for `runtime_probe`,
|
| 105 |
-
`analyze_image_llamacpp`, and deterministic
|
|
|
|
| 106 |
- Modal web endpoints are secured with `Authorization: Bearer
|
| 107 |
SNAP2SIM_API_TOKEN`; the token lives in Modal secret `snap2sim-api-auth` and
|
| 108 |
Hugging Face Space secret `SNAP2SIM_API_TOKEN`.
|
|
@@ -126,6 +136,17 @@ annotated technical cutaway animation.
|
|
| 126 |
Space sync on pushes to `main`. The workflow targets `jasondo111/Snap2Sim`
|
| 127 |
and requires GitHub secret `HF_TOKEN`.
|
| 128 |
- User reported the GitHub Actions `HF_TOKEN` secret has been added.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
- Documented GitHub as the source of truth:
|
| 130 |
`https://github.com/Bigstonks1/Snap2Sim`. Do not edit files directly on the
|
| 131 |
Hugging Face Space; they will be overwritten by the sync workflow.
|
|
@@ -133,15 +154,35 @@ annotated technical cutaway animation.
|
|
| 133 |
assistance from OpenAI Codex.
|
| 134 |
- Updated `README.md` with local run instructions, runtime preflight, Modal
|
| 135 |
deployment path, and the current runtime decision.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
|
| 137 |
## Next Work
|
| 138 |
|
| 139 |
-
-
|
| 140 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
- Keep the Hugging Face Space private until the user explicitly approves making
|
| 142 |
it public for submission.
|
| 143 |
-
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
- Document final submission links and bonus claims.
|
|
|
|
| 2 |
|
| 3 |
## Project Overview
|
| 4 |
|
| 5 |
+
Snap2Sim / Inside the Machine is a Build Small Hackathon Backyard AI project
|
| 6 |
+
for curious tinkerers and makers. The app is intended to accept a photo of a
|
| 7 |
+
hardware component, infer its internal mechanism, and render an annotated
|
| 8 |
+
technical cutaway animation.
|
| 9 |
|
| 10 |
## Runtime Notes
|
| 11 |
|
|
|
|
| 18 |
- Fallback model path: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`
|
| 19 |
through Transformers/custom code for the vision step if GGUF endpoint quality
|
| 20 |
is not reliable enough for the demo.
|
| 21 |
+
- Local code now serves a trusted `index.html` through `gradio.Server` and
|
| 22 |
+
exposes `/analyze_image` plus `/generate_scene`. The Space still needs a
|
| 23 |
+
GitHub-to-HF sync and private deployment verification for this pass.
|
| 24 |
+
- Model-generated scene output is A-Frame declarative HTML
|
| 25 |
+
(`<a-scene>...</a-scene>`), with human-written deterministic Three.js as the
|
| 26 |
+
fallback in `index.html`.
|
| 27 |
+
- Endpoint naming is `generate_scene` everywhere in active code.
|
| 28 |
+
- Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA
|
| 29 |
+
Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not
|
| 30 |
+
claim Off the Grid unless the final deployment moves inference to ZeroGPU or
|
| 31 |
+
another non-cloud-API path that satisfies that bonus.
|
| 32 |
|
| 33 |
## Project Structure
|
| 34 |
|
|
|
|
| 42 |
GitHub `main` to an existing Hugging Face Space.
|
| 43 |
- `requirements.txt` - Python dependencies for the Gradio Space and Modal
|
| 44 |
scaffold.
|
| 45 |
+
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 46 |
+
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 47 |
+
- `index.html` - self-contained HTML/CSS/JS shell loaded by `gradio.Server`;
|
| 48 |
+
includes A-Frame, Three.js, Gradio JS client, upload UI, pipeline
|
| 49 |
+
orchestration, and deterministic Three.js fallback.
|
| 50 |
- `modal_app.py` - Modal app scaffold with runtime asset caching, a
|
| 51 |
llama.cpp GPU smoke-test function, a `runtime_probe` diagnostic endpoint, and
|
| 52 |
+
placeholder plus experimental llama.cpp `analyze_image` / `generate_scene`
|
| 53 |
web endpoints.
|
| 54 |
- `scripts/verify_runtime_assets.py` - Hugging Face metadata preflight for the
|
| 55 |
selected GGUF quant and `mmproj` file.
|
| 56 |
- `snap2sim/__init__.py` - package marker.
|
| 57 |
- `snap2sim/backend.py` - backend config, local placeholder inference, Modal
|
| 58 |
HTTP client, and image base64 encoding.
|
| 59 |
+
- `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
|
| 60 |
+
local and placeholder Modal mode.
|
| 61 |
+
- `snap2sim/prompts.py` - prompt templates for the vision analysis and A-Frame
|
| 62 |
+
scene-generation steps.
|
| 63 |
- `snap2sim/schema.py` - structured JSON schema plus a sample mechanism payload.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
## What Has Been Done
|
| 66 |
|
|
|
|
| 97 |
HTML documents.
|
| 98 |
- Added a local coercion fallback for verbose/partial model JSON so the
|
| 99 |
analysis endpoint does not launch a second slow repair generation.
|
| 100 |
+
- Added experimental `analyze_image_llamacpp` and scene-generation Modal GPU
|
| 101 |
+
endpoints. They are not wired into the Gradio app by default; point
|
| 102 |
`MODAL_ANALYZE_URL` / `MODAL_GENERATE_URL` at them only after the smoke test
|
| 103 |
passes.
|
| 104 |
- Fixed Modal packaging for local `snap2sim` imports by adding
|
|
|
|
| 111 |
- Deployed `modal_app.py` to Modal at
|
| 112 |
`https://modal.com/apps/bigstonks1/main/deployed/snap2sim-inside-the-machine`.
|
| 113 |
- Production endpoint checks passed for `runtime_probe`,
|
| 114 |
+
`analyze_image_llamacpp`, and deterministic scene generation before the
|
| 115 |
+
current endpoint rename.
|
| 116 |
- Modal web endpoints are secured with `Authorization: Bearer
|
| 117 |
SNAP2SIM_API_TOKEN`; the token lives in Modal secret `snap2sim-api-auth` and
|
| 118 |
Hugging Face Space secret `SNAP2SIM_API_TOKEN`.
|
|
|
|
| 136 |
Space sync on pushes to `main`. The workflow targets `jasondo111/Snap2Sim`
|
| 137 |
and requires GitHub secret `HF_TOKEN`.
|
| 138 |
- User reported the GitHub Actions `HF_TOKEN` secret has been added.
|
| 139 |
+
- PR #1 was merged to `main`. The first GitHub-to-HF sync run reached the final
|
| 140 |
+
push step but failed with Hugging Face HTTP `429`; the workflow was updated
|
| 141 |
+
to retry that final push with bounded backoff.
|
| 142 |
+
- The next GitHub Actions sync run succeeded:
|
| 143 |
+
`https://github.com/Bigstonks1/Snap2Sim/actions/runs/27487136019`.
|
| 144 |
+
- Hugging Face Space `jasondo111/Snap2Sim` later reported SHA
|
| 145 |
+
`bfe49606478d6e922b0f973409fa3433f9d9ed9d`, matching GitHub `main`, after
|
| 146 |
+
the final handoff-note sync.
|
| 147 |
+
- `PROMPT.md` was updated with a new critical path: `gradio.Server`,
|
| 148 |
+
`index.html`, A-Frame model scene generation, `generate_scene` naming, and a
|
| 149 |
+
deterministic Three.js fallback in browser JS.
|
| 150 |
- Documented GitHub as the source of truth:
|
| 151 |
`https://github.com/Bigstonks1/Snap2Sim`. Do not edit files directly on the
|
| 152 |
Hugging Face Space; they will be overwritten by the sync workflow.
|
|
|
|
| 154 |
assistance from OpenAI Codex.
|
| 155 |
- Updated `README.md` with local run instructions, runtime preflight, Modal
|
| 156 |
deployment path, and the current runtime decision.
|
| 157 |
+
- Rewrote `app.py` as a `gradio.Server` app with `/`, `/analyze_image`, and
|
| 158 |
+
`/generate_scene` routes.
|
| 159 |
+
- Added `index.html` with upload orchestration, analysis readout, A-Frame scene
|
| 160 |
+
injection, 3-second blank-scene fallback, play/pause, and deterministic
|
| 161 |
+
browser-side Three.js fallback.
|
| 162 |
+
- Renamed active backend and Modal scene generation paths to `generate_scene`.
|
| 163 |
+
- Updated scene-generation prompts and parsing for `<a-scene>...</a-scene>`
|
| 164 |
+
output.
|
| 165 |
+
- Deleted the legacy Gradio Blocks UI, Python iframe Three.js generator, and
|
| 166 |
+
SVG fallback modules.
|
| 167 |
+
- Local verification passed with FastAPI `TestClient`: `/` returned the HTML
|
| 168 |
+
shell, `/analyze_image` returned the placeholder analysis, and
|
| 169 |
+
`/generate_scene` returned an `<a-scene>` block.
|
| 170 |
|
| 171 |
## Next Work
|
| 172 |
|
| 173 |
+
- Deploy the current branch through GitHub-to-HF sync, keep the Space private,
|
| 174 |
+
and confirm the private Space loads the trusted `index.html` shell.
|
| 175 |
+
- Update Space/Modal endpoint variables if needed so `MODAL_GENERATE_URL`
|
| 176 |
+
points at the deployed `generate_scene` endpoint, then verify
|
| 177 |
+
`/analyze_image` and `/generate_scene` with the secured bearer-token flow.
|
| 178 |
+
- Run an end-to-end private Space check with a real image: upload ->
|
| 179 |
+
`/analyze_image` -> panel population -> `/generate_scene` -> A-Frame render
|
| 180 |
+
or deterministic Three.js fallback.
|
| 181 |
+
- Optional polish only after private deployment verification: component
|
| 182 |
+
watermark, noise texture, vignette, then scan-line reveal refinements.
|
| 183 |
- Keep the Hugging Face Space private until the user explicitly approves making
|
| 184 |
it public for submission.
|
| 185 |
+
- Keep the README opening hook intact: "You find a small metal cylinder at a
|
| 186 |
+
flea market. What is it? How does it work inside?" After implementation,
|
| 187 |
+
expand README with the updated model stack, rendering-stack rationale, and
|
| 188 |
+
bonus quest claims from `PROMPT.md`.
|
|
|
PROMPT.md
CHANGED
|
@@ -1,185 +1,343 @@
|
|
| 1 |
-
|
| 2 |
-
Build Small Hackathon
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
"
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
-
|
| 142 |
-
|
| 143 |
-
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Snap2Sim — "Inside the Machine"
|
| 2 |
+
**Build Small Hackathon** · Backyard AI Track · [huggingface.co/build-small-hackathon](https://huggingface.co/build-small-hackathon)
|
| 3 |
+
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
## Goal
|
| 7 |
+
|
| 8 |
+
Build a Gradio app deployed as a Hugging Face Space that takes a photo of a hardware component (gear, valve, pump, lock, engine part, etc.) and produces an animated 3D visualization showing how that component works internally — "open it up and show me the moving parts and the mechanism."
|
| 9 |
+
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
## Hard Constraints
|
| 13 |
+
|
| 14 |
+
- Total model parameters across the entire pipeline **≤ 32B**
|
| 15 |
+
- Must be a **Gradio app** hosted as a **Hugging Face Space**
|
| 16 |
+
- No cloud AI APIs at inference time where possible (targets "Off the Grid" bonus)
|
| 17 |
+
- **Plain HTML/CSS/JS frontend** — no React, no build step, no bundler
|
| 18 |
+
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
## Target User — The Curious Tinkerer / Maker
|
| 22 |
+
|
| 23 |
+
Someone who pulls apart old electronics, finds a mystery component at a thrift store or salvage yard, or cracks open a broken appliance wondering: *"what does this actually do and how does it work inside?"* Not an engineer — a curious, hands-on person who learns by taking things apart.
|
| 24 |
+
|
| 25 |
+
> **README must open with:** *"You find a small metal cylinder at a flea market. What is it? How does it work inside?"* — before any technical description.
|
| 26 |
+
|
| 27 |
+
---
|
| 28 |
+
|
| 29 |
+
## Current State (June 13, 2026)
|
| 30 |
+
|
| 31 |
+
Scaffold exists at [github.com/Bigstonks1/Snap2Sim](https://github.com/Bigstonks1/Snap2Sim), synced to HF Space `jasondo111/Snap2Sim` (private, keep private until user approves).
|
| 32 |
+
|
| 33 |
+
**Confirmed working:**
|
| 34 |
+
- Modal deployment: `snap2sim-inside-the-machine` (bigstonks1 workspace)
|
| 35 |
+
- `smoke_test_llamacpp_image` → `"ok": true` for `UD-Q4_K_M` + `mmproj-F16.gguf`
|
| 36 |
+
- `analyze_image_llamacpp` and scene-generation Modal endpoints deployed
|
| 37 |
+
- GitHub → HF Space sync workflow live and passing
|
| 38 |
+
- Local app code now uses `gradio.Server` plus a trusted `index.html`
|
| 39 |
+
|
| 40 |
+
**Primary remaining tasks:**
|
| 41 |
+
1. Deploy the current branch through GitHub → HF sync
|
| 42 |
+
2. Confirm the private Space loads the trusted `index.html`
|
| 43 |
+
3. Confirm `/analyze_image` and `/generate_scene` respond through the secured
|
| 44 |
+
Modal bearer-token flow
|
| 45 |
+
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
## Model Stack
|
| 49 |
+
|
| 50 |
+
**Primary model:** NVIDIA Nemotron 3 Nano Omni (30B-A3B, MoE, ~3B active params)
|
| 51 |
+
Used for both vision analysis and A-Frame scene generation (two prompt turns, same model). Targets the **NVIDIA Nemotron Quest** sponsor award. ~31B total — under the 32B cap.
|
| 52 |
+
|
| 53 |
+
### GGUF Path (confirmed working)
|
| 54 |
+
| Setting | Value |
|
| 55 |
+
|---|---|
|
| 56 |
+
| Repo | `unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF` |
|
| 57 |
+
| Primary quant | `UD-Q4_K_M` (~24 GB) + `mmproj-F16.gguf` |
|
| 58 |
+
| Fallback quant | `UD-IQ2_M` (~18.5 GB) |
|
| 59 |
+
| Runtime | `llama-mtmd-cli` via llama.cpp on Modal GPU |
|
| 60 |
+
|
| 61 |
+
### Fallback Split Pipeline
|
| 62 |
+
Only use if primary model code-gen quality is too weak:
|
| 63 |
+
- **Vision:** NVIDIA Nemotron Nano V2 VL (12B)
|
| 64 |
+
- **Scene gen:** Qwen2.5-Coder-14B
|
| 65 |
+
- Total ~26B · still qualifies for Nemotron Quest
|
| 66 |
+
|
| 67 |
+
---
|
| 68 |
+
|
| 69 |
+
## Model Runtime
|
| 70 |
+
|
| 71 |
+
- Inference via **llama.cpp / GGUF** → targets **"Llama Champion"** bonus
|
| 72 |
+
- Modal GPU endpoints called over HTTP with Bearer token auth (`SNAP2SIM_API_TOKEN`)
|
| 73 |
+
- Backend swappable via `INFERENCE_BACKEND=modal | zerogpu | local`
|
| 74 |
+
- **If Modal deployed:** "Off the Grid" not claimed, but Llama Champion + Nemotron Quest + Modal Award all apply
|
| 75 |
+
- **If ZeroGPU sufficient:** "Off the Grid" additionally claimable
|
| 76 |
+
|
| 77 |
+
---
|
| 78 |
+
|
| 79 |
+
## Deployment Architecture
|
| 80 |
+
|
| 81 |
+
```
|
| 82 |
+
[HF Space — CPU tier] [Modal — GPU tier]
|
| 83 |
+
gradio.Server analyze_image_llamacpp
|
| 84 |
+
@app.get("/") → index.html ←→ generate_scene
|
| 85 |
+
@app.api() → /analyze_image (weights cached in Modal Volume)
|
| 86 |
+
@app.api() → /generate_scene
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
- Gradio app calls Modal endpoints over HTTP via `requests`, image passed as base64
|
| 90 |
+
- Modal cold starts on 30B model can take tens of seconds → show `"WAKING THE WORKSHOP..."` loading state
|
| 91 |
+
|
| 92 |
+
---
|
| 93 |
+
|
| 94 |
+
## Architectural Shift — `gr.Blocks` → `gradio.Server`
|
| 95 |
+
|
| 96 |
+
> **This is the core change. Do not skip or partially implement it.**
|
| 97 |
+
|
| 98 |
+
### Why
|
| 99 |
+
`gr.HTML` strips `<script>` tags for security and HF Spaces CSP blocks external CDN imports in `js_on_load`. Any WebGL/Three.js/A-Frame output piped through `gr.HTML` will fail on the live Space — scripts get stripped, nothing renders. This is a confirmed, known issue.
|
| 100 |
+
|
| 101 |
+
### How `gradio.Server` Fixes It
|
| 102 |
+
`gradio.Server` extends FastAPI. `@app.get("/")` serves `index.html` as a first-class trusted FastAPI response — the browser receives a full page with no stripping, no sandboxing, no CSP conflicts from Gradio's component system. A-Frame and Three.js CDN scripts load normally.
|
| 103 |
+
|
| 104 |
+
```python
|
| 105 |
+
from gradio import Server
|
| 106 |
+
|
| 107 |
+
app = Server()
|
| 108 |
+
|
| 109 |
+
@app.get("/")
|
| 110 |
+
async def homepage():
|
| 111 |
+
with open("index.html") as f:
|
| 112 |
+
return HTMLResponse(f.read())
|
| 113 |
+
|
| 114 |
+
@app.api(name="analyze_image")
|
| 115 |
+
def analyze_image(image_b64: str) -> dict:
|
| 116 |
+
return backend.run_analysis(image_b64) # calls Modal or local placeholder
|
| 117 |
+
|
| 118 |
+
@app.api(name="generate_scene")
|
| 119 |
+
def generate_scene(mechanism_json: dict) -> str:
|
| 120 |
+
return backend.run_scene_gen(mechanism_json) # returns A-Frame HTML string
|
| 121 |
+
|
| 122 |
+
app.launch()
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
### What to Change in the Scaffold
|
| 126 |
+
| File | Action |
|
| 127 |
+
|---|---|
|
| 128 |
+
| `app.py` | `gradio.Server` app serving `index.html` and API routes |
|
| 129 |
+
| `index.html` | Trusted HTML/CSS/JS shell with pipeline orchestration |
|
| 130 |
+
| `modal_app.py` | `generate_scene` endpoints and A-Frame prompt |
|
| 131 |
+
| `snap2sim/backend.py` | `generate_scene` backend method |
|
| 132 |
+
| `snap2sim/prompts.py` | A-Frame scene-generation prompt |
|
| 133 |
+
| `snap2sim/aframe_scene.py` | Deterministic A-Frame placeholder scene |
|
| 134 |
+
|
| 135 |
+
---
|
| 136 |
+
|
| 137 |
+
## Rendering Stack — Two Layers
|
| 138 |
+
|
| 139 |
+
### Layer 1 — Model-Generated Scene: A-Frame (declarative HTML)
|
| 140 |
+
|
| 141 |
+
A-Frame is a web framework built on Three.js that uses declarative HTML tags for 3D scenes. The model outputs HTML, not JavaScript — far more reliable for LLM generation.
|
| 142 |
+
|
| 143 |
+
**Why A-Frame for model output:**
|
| 144 |
+
- LLMs generate HTML tags far more reliably than imperative JS
|
| 145 |
+
- Injected via `innerHTML`, not `eval()` — no script execution risk
|
| 146 |
+
- A-Frame runtime (already loaded in `<head>`) renders injected tags automatically
|
| 147 |
+
- Built-in `animation` attribute handles motion without JS animation loops
|
| 148 |
+
- Camera, lighting, and sky added automatically — less boilerplate to get wrong
|
| 149 |
+
|
| 150 |
+
**Loading A-Frame in `index.html`:**
|
| 151 |
+
```html
|
| 152 |
+
<head>
|
| 153 |
+
<script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
|
| 154 |
+
</head>
|
| 155 |
+
```
|
| 156 |
+
|
| 157 |
+
> **CDN fallback:** If HF Spaces blocks `aframe.io`, vendor the minified A-Frame JS (~1.1MB) as a static file served via `gradio.Server`'s FastAPI static file mounting.
|
| 158 |
+
|
| 159 |
+
**Injecting model output:**
|
| 160 |
+
```javascript
|
| 161 |
+
document.getElementById('viewport').innerHTML = modelGeneratedAframeHTML;
|
| 162 |
+
// A-Frame runtime picks up the new <a-scene> tags automatically
|
| 163 |
+
```
|
| 164 |
+
|
| 165 |
+
**Example of what the model should output:**
|
| 166 |
+
```html
|
| 167 |
+
<a-scene>
|
| 168 |
+
<a-sky color="#0F1318"></a-sky>
|
| 169 |
+
<a-cylinder color="#E8A33D" radius="0.3" height="1" position="0 1 -3"
|
| 170 |
+
animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear">
|
| 171 |
+
</a-cylinder>
|
| 172 |
+
<a-box color="#5FD4D0" position="0.8 0.5 -3"
|
| 173 |
+
animation="property: position; to: 0.8 1 -3; dir: alternate; loop: true; dur: 1000">
|
| 174 |
+
</a-box>
|
| 175 |
+
<a-text value="Drive Shaft" position="0 2 -3" color="#5FD4D0" scale="0.5 0.5 0.5">
|
| 176 |
+
</a-text>
|
| 177 |
+
</a-scene>
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
**Prompt engineering for `generate_scene` endpoint:**
|
| 181 |
+
- Instruct the model to output **only** the `<a-scene>...</a-scene>` block — no preamble, no markdown fences, no explanation
|
| 182 |
+
- A-Frame primitives to use: `<a-box>`, `<a-cylinder>`, `<a-sphere>`, `<a-torus>`, `<a-cone>`, `<a-entity>`
|
| 183 |
+
- Animation format: `animation="property: rotation; to: 0 360 0; loop: true; dur: 2000; easing: linear"`
|
| 184 |
+
- Keep scenes to **3–6 parts maximum** for clarity
|
| 185 |
+
- Set `<a-sky color="#0F1318">` to match the page background
|
| 186 |
+
|
| 187 |
+
### Layer 2 — Deterministic Fallback: Three.js (human-written)
|
| 188 |
+
|
| 189 |
+
If A-Frame output is empty, malformed, or renders blank after 3 seconds, immediately swap to `buildDeterministicScene(json)` — a JS function that reads the mechanism JSON and builds a reliable Three.js scene from geometric primitives.
|
| 190 |
+
|
| 191 |
+
```javascript
|
| 192 |
+
function buildDeterministicScene(mechanismJson) {
|
| 193 |
+
// Human-written. Always works given valid JSON.
|
| 194 |
+
// Uses: BoxGeometry, CylinderGeometry, TorusGeometry per part shape
|
| 195 |
+
// Applies rotation/translation per motion_type
|
| 196 |
+
// Adds OrbitControls, annotation labels
|
| 197 |
+
// The viewport must never be blank or show an error
|
| 198 |
+
}
|
| 199 |
+
```
|
| 200 |
+
|
| 201 |
+
> Load Three.js in `index.html` alongside A-Frame:
|
| 202 |
+
> ```html
|
| 203 |
+
> <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
|
| 204 |
+
> ```
|
| 205 |
+
|
| 206 |
+
---
|
| 207 |
+
|
| 208 |
+
## Pipeline / Application Flow
|
| 209 |
+
|
| 210 |
+
```
|
| 211 |
+
1. User uploads photo
|
| 212 |
+
↓
|
| 213 |
+
2. Frontend encodes as base64 → calls /analyze_image (Gradio JS client)
|
| 214 |
+
↓
|
| 215 |
+
3. Nemotron vision step → structured JSON:
|
| 216 |
+
{
|
| 217 |
+
"component_name": "Solenoid Valve",
|
| 218 |
+
"parts": [
|
| 219 |
+
{ "name": "Coil", "shape": "cylinder", "color": "#E8A33D",
|
| 220 |
+
"position": [0, 0, 0], "motion_type": "none", "motion_params": {} },
|
| 221 |
+
{ "name": "Plunger", "shape": "cylinder", "color": "#5FD4D0",
|
| 222 |
+
"position": [0, 0.5, 0], "motion_type": "translate",
|
| 223 |
+
"motion_params": { "axis": "y", "range": 0.3, "dur": 800 } }
|
| 224 |
+
],
|
| 225 |
+
"summary": "When current flows through the coil, it generates a magnetic
|
| 226 |
+
field that pulls the plunger upward, opening the valve port."
|
| 227 |
+
}
|
| 228 |
+
↓
|
| 229 |
+
4. Frontend populates analysis panel (name, part list, summary)
|
| 230 |
+
→ immediately calls /generate_scene with JSON
|
| 231 |
+
↓
|
| 232 |
+
5. Nemotron scene gen step → A-Frame HTML string (<a-scene>...</a-scene>)
|
| 233 |
+
↓
|
| 234 |
+
6. Frontend injects A-Frame HTML → innerHTML of #viewport
|
| 235 |
+
A-Frame runtime renders automatically
|
| 236 |
+
↓
|
| 237 |
+
7. If A-Frame blank/failed after 3s → buildDeterministicScene(json)
|
| 238 |
+
Viewport is NEVER blank
|
| 239 |
+
```
|
| 240 |
+
|
| 241 |
+
---
|
| 242 |
+
|
| 243 |
+
## Visual Design — "Industrial Instrument Panel / Field Cutaway"
|
| 244 |
+
|
| 245 |
+
> Implement the shell and CSS in Step 2. Do **not** work on `[POLISH]` items until Steps 1–4 are done.
|
| 246 |
+
|
| 247 |
+
### Color System
|
| 248 |
+
```css
|
| 249 |
+
:root {
|
| 250 |
+
--bg: #0F1318;
|
| 251 |
+
--bg-panel: #161B22;
|
| 252 |
+
--bg-lift: #1E2530;
|
| 253 |
+
--amber: #E8A33D;
|
| 254 |
+
--amber-dim: #7A5420;
|
| 255 |
+
--cyan: #5FD4D0;
|
| 256 |
+
--cyan-dim: #2A5E5C;
|
| 257 |
+
--text: #C8C0AC;
|
| 258 |
+
--text-muted: #6B7280;
|
| 259 |
+
--grid: rgba(255,255,255,0.04);
|
| 260 |
+
}
|
| 261 |
+
```
|
| 262 |
+
|
| 263 |
+
### Typography
|
| 264 |
+
Load from **Bunny Fonts** (not Google Fonts):
|
| 265 |
+
- **Display / headings / UI labels:** `Chakra Petch` — technical, instrument-panel character
|
| 266 |
+
- **Monospace / data / callouts:** `Fira Code`
|
| 267 |
+
- **Never use:** Inter, Roboto, Arial, Space Grotesk, or any system font
|
| 268 |
+
|
| 269 |
+
### Layout
|
| 270 |
+
- Two-pane asymmetric split: **63% viewport** (left) · **37% analysis panel** (right)
|
| 271 |
+
- Blueprint grid: 1px lines at `--grid` opacity, 32px spacing, on `--bg` base
|
| 272 |
+
- Panel separator: 1px vertical line in `--amber-dim`
|
| 273 |
+
- Upload drop zone: fills viewport · dashed 1px `--amber-dim` border (no `border-radius`) · centered `"DROP COMPONENT PHOTO"` in Chakra Petch uppercase `--text-muted` · on drag-over: border → `--amber`, text → `--amber`
|
| 274 |
+
- Play/pause: minimal amber rectangle (no `border-radius`), Chakra Petch uppercase `"PAUSE"` / `"RESUME"`, controls A-Frame animation playback via JS
|
| 275 |
+
|
| 276 |
+
### Loading States
|
| 277 |
+
All in Chakra Petch uppercase, `--amber` color, with thin `--amber` indeterminate progress bar across viewport top:
|
| 278 |
+
|
| 279 |
+
| State | Message |
|
| 280 |
+
|---|---|
|
| 281 |
+
| Modal cold start | `WAKING THE WORKSHOP...` |
|
| 282 |
+
| Vision inference | `ANALYZING ASSEMBLY...` |
|
| 283 |
+
| Scene generation | `RENDERING CUTAWAY...` |
|
| 284 |
+
|
| 285 |
+
### [POLISH] — Only After Steps 1–4 Work
|
| 286 |
+
Implement in this sub-order:
|
| 287 |
+
|
| 288 |
+
1. **Component name watermark** — large (`clamp(4rem, 8vw, 9rem)`) Chakra Petch uppercase in `--bg-lift`, absolutely positioned bleeding across both panes from bottom-left, `z-index` below content. Populated from `component_name` in JSON.
|
| 289 |
+
|
| 290 |
+
2. **Noise texture** — SVG `feTurbulence` grain at 3% opacity on viewport pane, inline `data:` URI, no external file. Makes the surface feel physical.
|
| 291 |
+
|
| 292 |
+
3. **Vignette** — radial gradient overlay on viewport edges, `pointer-events: none` so it floats above the A-Frame canvas without blocking interaction.
|
| 293 |
+
|
| 294 |
+
4. **Scan-line reveal** — when A-Frame scene first loads:
|
| 295 |
+
- 2px `--cyan` scan-line sweeps top→bottom over 0.7s
|
| 296 |
+
- Each A-Frame entity fades in as line passes it (`opacity 0→1`, `translateY 12px→0`, 0.35s ease-out, staggered by part index via `animation-delay`)
|
| 297 |
+
- Part labels fade in together (0.25s)
|
| 298 |
+
- Progress bar dissolves (0.2s)
|
| 299 |
+
- Total: ~1.2s · this is the signature moment
|
| 300 |
+
|
| 301 |
+
---
|
| 302 |
+
|
| 303 |
+
## Deliverables
|
| 304 |
+
|
| 305 |
+
- `app.py` — `gradio.Server` app (~50 lines)
|
| 306 |
+
- `index.html` — self-contained HTML/CSS/JS; A-Frame + Three.js from CDN; Gradio JS client from CDN
|
| 307 |
+
- `snap2sim/backend.py` — `generate_scene`
|
| 308 |
+
- `modal_app.py` — `generate_scene`; A-Frame prompt
|
| 309 |
+
- `snap2sim/prompts.py` — updated A-Frame scene generation prompt
|
| 310 |
+
- `requirements.txt` — updated if needed for `gradio.Server`
|
| 311 |
+
- `README.md` — tinkerer/maker story hook → project description → model stack with exact parameter breakdown (≤32B) → rendering stack rationale → bonus quest claims:
|
| 312 |
+
|
| 313 |
+
| Quest | Status |
|
| 314 |
+
|---|---|
|
| 315 |
+
| Llama Champion | ✅ Confirmed |
|
| 316 |
+
| NVIDIA Nemotron Quest | ✅ Confirmed |
|
| 317 |
+
| Off-Brand | ✅ Confirmed |
|
| 318 |
+
| Modal Award | ✅ Confirmed |
|
| 319 |
+
| Off the Grid | ⚡ If ZeroGPU used in final deploy |
|
| 320 |
+
| Field Notes | 🎯 Stretch |
|
| 321 |
+
|
| 322 |
+
---
|
| 323 |
+
|
| 324 |
+
## Start Order
|
| 325 |
+
|
| 326 |
+
> Follow this exactly. Do not skip ahead.
|
| 327 |
+
|
| 328 |
+
### Step 1 — Critical Path
|
| 329 |
+
Deploy the current `gradio.Server` + `index.html` implementation through
|
| 330 |
+
GitHub → HF sync. Confirm the private Space loads and both `/analyze_image`
|
| 331 |
+
and `/generate_scene` respond.
|
| 332 |
+
|
| 333 |
+
### Step 2 — Make It Functional
|
| 334 |
+
Build `buildDeterministicScene(json)` in JS — Three.js scene from geometric primitives, always works given valid JSON. Wire the full pipeline: upload → `/analyze_image` → populate analysis panel → `/generate_scene` → inject A-Frame HTML via `innerHTML` → fallback to `buildDeterministicScene(json)` if A-Frame fails or blanks after 3 seconds. **Confirm end-to-end with a real image and live Modal endpoints.**
|
| 335 |
+
|
| 336 |
+
### Step 3 — Apply the Design Shell
|
| 337 |
+
Add CSS variable system, Chakra Petch + Fira Code from Bunny Fonts, two-pane asymmetric layout, blueprint grid, loading states with progress bar, upload drop zone, panel separator. App should match the design direction above — minus `[POLISH]` items.
|
| 338 |
+
|
| 339 |
+
### Step 4 — Harden
|
| 340 |
+
Error handling, 3-second blank-scene timeout before fallback triggers, Modal cold-start messaging, A-Frame `<a-sky color="#0F1318">` matching page background, play/pause toggle wired to A-Frame animation playback. Verify GitHub → HF Space sync pushes cleanly and the private Space runs correctly.
|
| 341 |
+
|
| 342 |
+
### Step 5 — [POLISH] Only if Time Remains
|
| 343 |
+
Component name watermark → noise texture → vignette → scan-line reveal. In that order.
|
README.md
CHANGED
|
@@ -9,13 +9,18 @@ colorTo: yellow
|
|
| 9 |
|
| 10 |
# Snap2Sim
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
mechanism analysis, and renders a technical cutaway visualization.
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Run Locally
|
| 21 |
|
|
@@ -25,7 +30,8 @@ python app.py
|
|
| 25 |
```
|
| 26 |
|
| 27 |
Set `INFERENCE_BACKEND=modal`, `MODAL_ANALYZE_URL`, and `MODAL_GENERATE_URL`
|
| 28 |
-
to point the Gradio app at deployed Modal
|
|
|
|
| 29 |
|
| 30 |
For the deployed demo configuration, copy `.env.example` into local environment
|
| 31 |
variables or Hugging Face Space variables. Do not put tokens in `.env.example`;
|
|
@@ -68,20 +74,60 @@ Primary references:
|
|
| 68 |
- https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
|
| 69 |
- https://github.com/ggml-org/llama.cpp
|
| 70 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
## Project Structure
|
| 72 |
|
| 73 |
- `.env.example` - public runtime variables for the deployed Modal demo.
|
| 74 |
- `SECURITY.md` - public/private data handling guidance for humans and agents.
|
| 75 |
-
- `app.py` - Hugging Face Space entry point.
|
|
|
|
|
|
|
|
|
|
| 76 |
- `modal_app.py` - Modal web endpoint scaffold.
|
| 77 |
- `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
|
| 78 |
- `snap2sim/backend.py` - backend selection and HTTP client.
|
| 79 |
-
- `snap2sim/
|
| 80 |
-
|
|
|
|
| 81 |
- `snap2sim/schema.py` - JSON schema and sample analysis payload.
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
## Modal Deployment Path
|
| 87 |
|
|
@@ -113,9 +159,9 @@ Useful deployment functions/endpoints:
|
|
| 113 |
synthetic image and confirms it returns a validated mechanism payload.
|
| 114 |
- `runtime_probe` reports the configured model repo, quant, projector file, and
|
| 115 |
whether placeholder inference is still active.
|
| 116 |
-
- `analyze_image` and `
|
| 117 |
the Gradio app.
|
| 118 |
-
- `analyze_image_llamacpp` and `
|
| 119 |
endpoints for the llama.cpp runtime path after the smoke test passes.
|
| 120 |
|
| 121 |
Runtime environment knobs:
|
|
@@ -130,7 +176,7 @@ Runtime environment knobs:
|
|
| 130 |
|
| 131 |
Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
|
| 132 |
`MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint and keep
|
| 133 |
-
`MODAL_GENERATE_URL` on deterministic `
|
| 134 |
|
| 135 |
Run the deployment preflight in this order:
|
| 136 |
|
|
|
|
| 9 |
|
| 10 |
# Snap2Sim
|
| 11 |
|
| 12 |
+
You find a small metal cylinder at a flea market. What is it? How does it work
|
| 13 |
+
inside?
|
|
|
|
| 14 |
|
| 15 |
+
Inside the Machine is a Build Small Hackathon Backyard AI project for curious
|
| 16 |
+
tinkerers and makers. It takes a hardware component photo, produces a
|
| 17 |
+
structured mechanism analysis, and renders an animated technical cutaway
|
| 18 |
+
visualization.
|
| 19 |
+
|
| 20 |
+
The local app now serves a trusted `index.html` shell through `gradio.Server`
|
| 21 |
+
and exposes `/analyze_image` plus `/generate_scene` routes. It still defaults
|
| 22 |
+
to the placeholder backend locally, while the Modal llama.cpp path has been
|
| 23 |
+
smoke-tested with the selected Nemotron GGUF and projector.
|
| 24 |
|
| 25 |
## Run Locally
|
| 26 |
|
|
|
|
| 30 |
```
|
| 31 |
|
| 32 |
Set `INFERENCE_BACKEND=modal`, `MODAL_ANALYZE_URL`, and `MODAL_GENERATE_URL`
|
| 33 |
+
to point the Gradio app at deployed Modal `/analyze_image` and
|
| 34 |
+
`/generate_scene` endpoints.
|
| 35 |
|
| 36 |
For the deployed demo configuration, copy `.env.example` into local environment
|
| 37 |
variables or Hugging Face Space variables. Do not put tokens in `.env.example`;
|
|
|
|
| 74 |
- https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
|
| 75 |
- https://github.com/ggml-org/llama.cpp
|
| 76 |
|
| 77 |
+
## Model Stack and Quest Claims
|
| 78 |
+
|
| 79 |
+
Primary inference uses NVIDIA Nemotron 3 Nano Omni 30B-A3B through llama.cpp
|
| 80 |
+
GGUF. The model has about 30B total parameters with roughly 3B active per MoE
|
| 81 |
+
token, keeping the single-model pipeline under the hackathon 32B cap.
|
| 82 |
+
|
| 83 |
+
Fallback, only if the primary endpoint quality is not demo-ready, is
|
| 84 |
+
NVIDIA Nemotron Nano V2 VL 12B for vision plus Qwen2.5-Coder-14B for scene
|
| 85 |
+
generation, for about 26B total parameters.
|
| 86 |
+
|
| 87 |
+
| Quest | Status |
|
| 88 |
+
|---|---|
|
| 89 |
+
| Llama Champion | Confirmed via llama.cpp / GGUF runtime |
|
| 90 |
+
| NVIDIA Nemotron Quest | Confirmed via Nemotron primary model |
|
| 91 |
+
| Off-Brand | Confirmed |
|
| 92 |
+
| Modal Award | Confirmed via deployed Modal endpoints |
|
| 93 |
+
| Off the Grid | Not claimed while inference runs on Modal |
|
| 94 |
+
| Field Notes | Stretch |
|
| 95 |
+
|
| 96 |
## Project Structure
|
| 97 |
|
| 98 |
- `.env.example` - public runtime variables for the deployed Modal demo.
|
| 99 |
- `SECURITY.md` - public/private data handling guidance for humans and agents.
|
| 100 |
+
- `app.py` - Hugging Face Space entry point using `gradio.Server`; serves
|
| 101 |
+
`index.html` at `/` and exposes `/analyze_image` plus `/generate_scene`.
|
| 102 |
+
- `index.html` - plain HTML/CSS/JS shell with upload orchestration,
|
| 103 |
+
A-Frame injection, and deterministic browser-side Three.js fallback.
|
| 104 |
- `modal_app.py` - Modal web endpoint scaffold.
|
| 105 |
- `scripts/verify_runtime_assets.py` - GGUF/mmproj metadata preflight.
|
| 106 |
- `snap2sim/backend.py` - backend selection and HTTP client.
|
| 107 |
+
- `snap2sim/aframe_scene.py` - deterministic A-Frame scene generation for
|
| 108 |
+
local and placeholder Modal mode.
|
| 109 |
+
- `snap2sim/prompts.py` - prompt templates for vision and A-Frame generation.
|
| 110 |
- `snap2sim/schema.py` - JSON schema and sample analysis payload.
|
| 111 |
+
|
| 112 |
+
## Current Rendering Architecture
|
| 113 |
+
|
| 114 |
+
The app uses `gradio.Server` to serve `index.html` directly. This avoids
|
| 115 |
+
Gradio component script stripping and lets the page load A-Frame and Three.js
|
| 116 |
+
normally.
|
| 117 |
+
|
| 118 |
+
Runtime flow:
|
| 119 |
+
|
| 120 |
+
1. Browser encodes the uploaded photo and posts it to `/analyze_image`.
|
| 121 |
+
2. Backend returns the validated mechanism JSON.
|
| 122 |
+
3. Browser posts the JSON to `/generate_scene`.
|
| 123 |
+
4. The scene endpoint returns only an `<a-scene>...</a-scene>` block.
|
| 124 |
+
5. Browser injects the A-Frame scene. If it is malformed or does not create a
|
| 125 |
+
canvas within 3 seconds, `buildDeterministicScene(json)` renders a
|
| 126 |
+
human-written Three.js fallback in the same viewport.
|
| 127 |
+
|
| 128 |
+
The shell uses Chakra Petch and Fira Code from Bunny Fonts, an asymmetric
|
| 129 |
+
63/37 viewport/readout split, a blueprint grid, amber/cyan instrument-panel
|
| 130 |
+
colors, explicit Modal cold-start messaging, and a play/pause control.
|
| 131 |
|
| 132 |
## Modal Deployment Path
|
| 133 |
|
|
|
|
| 159 |
synthetic image and confirms it returns a validated mechanism payload.
|
| 160 |
- `runtime_probe` reports the configured model repo, quant, projector file, and
|
| 161 |
whether placeholder inference is still active.
|
| 162 |
+
- `analyze_image` and `generate_scene` preserve the current HTTP contract for
|
| 163 |
the Gradio app.
|
| 164 |
+
- `analyze_image_llamacpp` and `generate_scene_llamacpp` are experimental GPU
|
| 165 |
endpoints for the llama.cpp runtime path after the smoke test passes.
|
| 166 |
|
| 167 |
Runtime environment knobs:
|
|
|
|
| 176 |
|
| 177 |
Keep `SNAP2SIM_RUNTIME_MODE=placeholder` for the public demo unless you point
|
| 178 |
`MODAL_ANALYZE_URL` at the validated `analyze_image_llamacpp` endpoint and keep
|
| 179 |
+
`MODAL_GENERATE_URL` on deterministic `generate_scene`.
|
| 180 |
|
| 181 |
Run the deployment preflight in this order:
|
| 182 |
|
SECURITY.md
CHANGED
|
@@ -82,6 +82,12 @@ Do not configure jobs that pull changes back from Hugging Face into GitHub.
|
|
| 82 |
|
| 83 |
- Prefer Hugging Face Space variables for public configuration and Space secrets
|
| 84 |
for credentials.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
- Do not print secret-bearing environment variables in logs.
|
| 86 |
- Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
|
| 87 |
or Codex summaries.
|
|
|
|
| 82 |
|
| 83 |
- Prefer Hugging Face Space variables for public configuration and Space secrets
|
| 84 |
for credentials.
|
| 85 |
+
- The planned `index.html` / `gradio.Server` frontend is public client-side
|
| 86 |
+
code. Never embed `SNAP2SIM_API_TOKEN`, Modal endpoint URLs, Hugging Face
|
| 87 |
+
tokens, or any credential-bearing values in HTML, JavaScript, CSS, bundled
|
| 88 |
+
static assets, browser local storage, or query strings. Browser JS should call
|
| 89 |
+
same-origin `@app.api()` endpoints; server-side Python should attach secrets
|
| 90 |
+
when calling Modal.
|
| 91 |
- Do not print secret-bearing environment variables in logs.
|
| 92 |
- Do not paste full Modal or Hugging Face auth files into issues, docs, prompts,
|
| 93 |
or Codex summaries.
|
app.py
CHANGED
|
@@ -1,11 +1,96 @@
|
|
| 1 |
-
"""Hugging Face Space entry point."""
|
| 2 |
|
| 3 |
-
from
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
|
| 6 |
-
|
|
|
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
-
if __name__ == "__main__":
|
| 10 |
-
demo.launch()
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Hugging Face Space entry point for the trusted HTML shell."""
|
| 2 |
|
| 3 |
+
from __future__ import annotations
|
| 4 |
|
| 5 |
+
import base64
|
| 6 |
+
from io import BytesIO
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
from typing import Any, Callable
|
| 9 |
|
| 10 |
+
from fastapi.responses import HTMLResponse
|
| 11 |
+
from PIL import Image
|
| 12 |
|
| 13 |
+
from snap2sim.backend import InferenceClient, Settings
|
| 14 |
+
|
| 15 |
+
try:
|
| 16 |
+
from gradio import Server
|
| 17 |
+
except ImportError:
|
| 18 |
+
from fastapi import FastAPI
|
| 19 |
+
import uvicorn
|
| 20 |
+
|
| 21 |
+
class Server(FastAPI): # type: ignore[no-redef]
|
| 22 |
+
"""Local compatibility shim for environments older than Gradio Server."""
|
| 23 |
+
|
| 24 |
+
def api(self, name: str) -> Callable[[Callable[..., Any]], Callable[..., Any]]:
|
| 25 |
+
def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
|
| 26 |
+
return func
|
| 27 |
+
|
| 28 |
+
return decorator
|
| 29 |
+
|
| 30 |
+
def launch(self, **kwargs: Any) -> None:
|
| 31 |
+
uvicorn.run(
|
| 32 |
+
self,
|
| 33 |
+
host=kwargs.get("server_name", "0.0.0.0"),
|
| 34 |
+
port=kwargs.get("server_port", 7860),
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
app = Server()
|
| 39 |
+
INDEX_PATH = Path(__file__).with_name("index.html")
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
@app.get("/", response_class=HTMLResponse)
|
| 43 |
+
async def homepage() -> str:
|
| 44 |
+
return INDEX_PATH.read_text(encoding="utf-8")
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@app.get("/manifest.json")
|
| 48 |
+
async def manifest() -> dict[str, Any]:
|
| 49 |
+
return {
|
| 50 |
+
"name": "Snap2Sim Inside the Machine",
|
| 51 |
+
"short_name": "Snap2Sim",
|
| 52 |
+
"start_url": "/",
|
| 53 |
+
"display": "standalone",
|
| 54 |
+
"background_color": "#0F1318",
|
| 55 |
+
"theme_color": "#E8A33D",
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
@app.api(name="analyze_image")
|
| 60 |
+
def analyze_image_api(image_base64: str) -> dict[str, Any]:
|
| 61 |
+
return _analyze_image(image_base64)
|
| 62 |
|
|
|
|
|
|
|
| 63 |
|
| 64 |
+
@app.post("/analyze_image")
|
| 65 |
+
def analyze_image_http(payload: dict[str, Any]) -> dict[str, Any]:
|
| 66 |
+
return _analyze_image(str(payload.get("image_base64", "")))
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
@app.api(name="generate_scene")
|
| 70 |
+
def generate_scene_api(analysis: dict[str, Any]) -> str:
|
| 71 |
+
return _generate_scene(analysis)
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
@app.post("/generate_scene")
|
| 75 |
+
def generate_scene_http(payload: dict[str, Any]) -> dict[str, str]:
|
| 76 |
+
return {"html": _generate_scene(payload.get("analysis") or {})}
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _analyze_image(image_base64: str) -> dict[str, Any]:
|
| 80 |
+
image = _decode_image(image_base64) if image_base64 else None
|
| 81 |
+
return InferenceClient(Settings()).analyze_image(image)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def _generate_scene(analysis: dict[str, Any]) -> str:
|
| 85 |
+
return InferenceClient(Settings()).generate_scene(analysis)
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def _decode_image(image_base64: str) -> Image.Image:
|
| 89 |
+
if "," in image_base64 and image_base64.lstrip().startswith("data:"):
|
| 90 |
+
image_base64 = image_base64.split(",", 1)[1]
|
| 91 |
+
raw = base64.b64decode(image_base64)
|
| 92 |
+
return Image.open(BytesIO(raw)).convert("RGB")
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
if __name__ == "__main__":
|
| 96 |
+
app.launch()
|
index.html
ADDED
|
@@ -0,0 +1,808 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
+
<title>Inside the Machine</title>
|
| 7 |
+
<link rel="preconnect" href="https://fonts.bunny.net">
|
| 8 |
+
<link href="https://fonts.bunny.net/css?family=chakra-petch:400,500,600,700|fira-code:400,500,600" rel="stylesheet">
|
| 9 |
+
<script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script>
|
| 10 |
+
<script type="module">
|
| 11 |
+
import { Client } from "https://cdn.jsdelivr.net/npm/@gradio/client/dist/index.min.js";
|
| 12 |
+
window.snap2simGradioClient = Client;
|
| 13 |
+
</script>
|
| 14 |
+
<style>
|
| 15 |
+
:root {
|
| 16 |
+
--bg: #0F1318;
|
| 17 |
+
--bg-panel: #161B22;
|
| 18 |
+
--bg-lift: #1E2530;
|
| 19 |
+
--amber: #E8A33D;
|
| 20 |
+
--amber-dim: #7A5420;
|
| 21 |
+
--cyan: #5FD4D0;
|
| 22 |
+
--cyan-dim: #2A5E5C;
|
| 23 |
+
--text: #C8C0AC;
|
| 24 |
+
--text-muted: #6B7280;
|
| 25 |
+
--grid: rgba(255,255,255,0.04);
|
| 26 |
+
--danger: #F07F5A;
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
* {
|
| 30 |
+
box-sizing: border-box;
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
html,
|
| 34 |
+
body {
|
| 35 |
+
width: 100%;
|
| 36 |
+
height: 100%;
|
| 37 |
+
margin: 0;
|
| 38 |
+
overflow: hidden;
|
| 39 |
+
color: var(--text);
|
| 40 |
+
background:
|
| 41 |
+
linear-gradient(var(--grid) 1px, transparent 1px),
|
| 42 |
+
linear-gradient(90deg, var(--grid) 1px, transparent 1px),
|
| 43 |
+
var(--bg);
|
| 44 |
+
background-size: 32px 32px;
|
| 45 |
+
font-family: "Chakra Petch", "Fira Code", monospace;
|
| 46 |
+
}
|
| 47 |
+
|
| 48 |
+
button,
|
| 49 |
+
input {
|
| 50 |
+
font: inherit;
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
.shell {
|
| 54 |
+
display: grid;
|
| 55 |
+
grid-template-columns: minmax(0, 63fr) minmax(340px, 37fr);
|
| 56 |
+
width: 100vw;
|
| 57 |
+
height: 100vh;
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
.viewport-pane {
|
| 61 |
+
position: relative;
|
| 62 |
+
min-width: 0;
|
| 63 |
+
overflow: hidden;
|
| 64 |
+
background:
|
| 65 |
+
radial-gradient(circle at 50% 40%, rgba(95, 212, 208, 0.06), transparent 42%),
|
| 66 |
+
var(--bg);
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
.viewport-pane::after {
|
| 70 |
+
content: "";
|
| 71 |
+
position: absolute;
|
| 72 |
+
inset: 0;
|
| 73 |
+
pointer-events: none;
|
| 74 |
+
background: radial-gradient(circle at center, transparent 58%, rgba(0, 0, 0, 0.36));
|
| 75 |
+
z-index: 7;
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
#viewport {
|
| 79 |
+
position: absolute;
|
| 80 |
+
inset: 0;
|
| 81 |
+
z-index: 1;
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
#viewport a-scene {
|
| 85 |
+
width: 100%;
|
| 86 |
+
height: 100%;
|
| 87 |
+
}
|
| 88 |
+
|
| 89 |
+
#viewport .a-enter-vr,
|
| 90 |
+
#viewport .a-orientation-modal {
|
| 91 |
+
display: none !important;
|
| 92 |
+
}
|
| 93 |
+
|
| 94 |
+
.drop-zone {
|
| 95 |
+
position: absolute;
|
| 96 |
+
inset: 18px;
|
| 97 |
+
z-index: 5;
|
| 98 |
+
display: grid;
|
| 99 |
+
place-items: center;
|
| 100 |
+
border: 1px dashed var(--amber-dim);
|
| 101 |
+
color: var(--text-muted);
|
| 102 |
+
background: rgba(15, 19, 24, 0.72);
|
| 103 |
+
text-transform: uppercase;
|
| 104 |
+
letter-spacing: 0;
|
| 105 |
+
font-size: clamp(1.15rem, 2vw, 2.1rem);
|
| 106 |
+
font-weight: 600;
|
| 107 |
+
transition: color 160ms ease, border-color 160ms ease, background 160ms ease;
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
.drop-zone.dragging {
|
| 111 |
+
color: var(--amber);
|
| 112 |
+
border-color: var(--amber);
|
| 113 |
+
background: rgba(122, 84, 32, 0.16);
|
| 114 |
+
}
|
| 115 |
+
|
| 116 |
+
.drop-zone.hidden {
|
| 117 |
+
display: none;
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
.scan-line {
|
| 121 |
+
position: absolute;
|
| 122 |
+
left: 0;
|
| 123 |
+
right: 0;
|
| 124 |
+
top: -2px;
|
| 125 |
+
z-index: 8;
|
| 126 |
+
height: 2px;
|
| 127 |
+
opacity: 0;
|
| 128 |
+
background: var(--cyan);
|
| 129 |
+
box-shadow: 0 0 18px rgba(95, 212, 208, 0.8);
|
| 130 |
+
pointer-events: none;
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
.scan-line.active {
|
| 134 |
+
animation: scan 700ms ease-out;
|
| 135 |
+
}
|
| 136 |
+
|
| 137 |
+
.progress {
|
| 138 |
+
position: absolute;
|
| 139 |
+
left: 0;
|
| 140 |
+
right: 0;
|
| 141 |
+
top: 0;
|
| 142 |
+
z-index: 9;
|
| 143 |
+
height: 2px;
|
| 144 |
+
overflow: hidden;
|
| 145 |
+
opacity: 0;
|
| 146 |
+
background: rgba(122, 84, 32, 0.35);
|
| 147 |
+
transition: opacity 180ms ease;
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
.progress.active {
|
| 151 |
+
opacity: 1;
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
.progress::before {
|
| 155 |
+
content: "";
|
| 156 |
+
position: absolute;
|
| 157 |
+
top: 0;
|
| 158 |
+
bottom: 0;
|
| 159 |
+
width: 34%;
|
| 160 |
+
background: var(--amber);
|
| 161 |
+
animation: loading 1050ms linear infinite;
|
| 162 |
+
}
|
| 163 |
+
|
| 164 |
+
.toolbar {
|
| 165 |
+
position: absolute;
|
| 166 |
+
left: 18px;
|
| 167 |
+
top: 18px;
|
| 168 |
+
z-index: 10;
|
| 169 |
+
display: flex;
|
| 170 |
+
gap: 8px;
|
| 171 |
+
}
|
| 172 |
+
|
| 173 |
+
.tool-button {
|
| 174 |
+
min-width: 82px;
|
| 175 |
+
min-height: 34px;
|
| 176 |
+
border: 1px solid var(--amber-dim);
|
| 177 |
+
border-radius: 0;
|
| 178 |
+
color: var(--amber);
|
| 179 |
+
background: rgba(15, 19, 24, 0.86);
|
| 180 |
+
text-transform: uppercase;
|
| 181 |
+
letter-spacing: 0;
|
| 182 |
+
cursor: pointer;
|
| 183 |
+
}
|
| 184 |
+
|
| 185 |
+
.tool-button:hover,
|
| 186 |
+
.tool-button:focus-visible {
|
| 187 |
+
border-color: var(--amber);
|
| 188 |
+
outline: none;
|
| 189 |
+
}
|
| 190 |
+
|
| 191 |
+
.tool-button[disabled] {
|
| 192 |
+
color: var(--text-muted);
|
| 193 |
+
border-color: var(--cyan-dim);
|
| 194 |
+
cursor: default;
|
| 195 |
+
}
|
| 196 |
+
|
| 197 |
+
.panel {
|
| 198 |
+
min-width: 0;
|
| 199 |
+
overflow: auto;
|
| 200 |
+
border-left: 1px solid var(--amber-dim);
|
| 201 |
+
background: linear-gradient(180deg, rgba(30, 37, 48, 0.92), var(--bg-panel));
|
| 202 |
+
}
|
| 203 |
+
|
| 204 |
+
.panel-inner {
|
| 205 |
+
min-height: 100%;
|
| 206 |
+
padding: 24px;
|
| 207 |
+
display: grid;
|
| 208 |
+
grid-template-rows: auto auto 1fr auto;
|
| 209 |
+
gap: 22px;
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
.brand {
|
| 213 |
+
display: grid;
|
| 214 |
+
gap: 8px;
|
| 215 |
+
border-bottom: 1px solid rgba(122, 84, 32, 0.7);
|
| 216 |
+
padding-bottom: 18px;
|
| 217 |
+
}
|
| 218 |
+
|
| 219 |
+
.kicker,
|
| 220 |
+
.label {
|
| 221 |
+
color: var(--amber);
|
| 222 |
+
font-size: 0.78rem;
|
| 223 |
+
font-weight: 600;
|
| 224 |
+
text-transform: uppercase;
|
| 225 |
+
}
|
| 226 |
+
|
| 227 |
+
h1 {
|
| 228 |
+
margin: 0;
|
| 229 |
+
color: var(--text);
|
| 230 |
+
font-size: clamp(2rem, 4vw, 4.4rem);
|
| 231 |
+
line-height: 0.9;
|
| 232 |
+
font-weight: 700;
|
| 233 |
+
letter-spacing: 0;
|
| 234 |
+
text-transform: uppercase;
|
| 235 |
+
}
|
| 236 |
+
|
| 237 |
+
.status {
|
| 238 |
+
min-height: 24px;
|
| 239 |
+
color: var(--amber);
|
| 240 |
+
font-size: 0.92rem;
|
| 241 |
+
font-weight: 600;
|
| 242 |
+
text-transform: uppercase;
|
| 243 |
+
}
|
| 244 |
+
|
| 245 |
+
.status.error {
|
| 246 |
+
color: var(--danger);
|
| 247 |
+
}
|
| 248 |
+
|
| 249 |
+
.readout {
|
| 250 |
+
display: grid;
|
| 251 |
+
gap: 18px;
|
| 252 |
+
}
|
| 253 |
+
|
| 254 |
+
.metric-row {
|
| 255 |
+
display: grid;
|
| 256 |
+
grid-template-columns: 1fr auto;
|
| 257 |
+
gap: 16px;
|
| 258 |
+
align-items: baseline;
|
| 259 |
+
border-bottom: 1px solid rgba(95, 212, 208, 0.16);
|
| 260 |
+
padding-bottom: 10px;
|
| 261 |
+
}
|
| 262 |
+
|
| 263 |
+
.component {
|
| 264 |
+
color: var(--cyan);
|
| 265 |
+
font-size: clamp(1.4rem, 2.2vw, 2.4rem);
|
| 266 |
+
font-weight: 700;
|
| 267 |
+
line-height: 1;
|
| 268 |
+
text-transform: uppercase;
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
.confidence {
|
| 272 |
+
color: var(--amber);
|
| 273 |
+
font-family: "Fira Code", monospace;
|
| 274 |
+
font-size: 0.95rem;
|
| 275 |
+
}
|
| 276 |
+
|
| 277 |
+
.summary,
|
| 278 |
+
.trigger,
|
| 279 |
+
.sequence,
|
| 280 |
+
.parts {
|
| 281 |
+
margin: 0;
|
| 282 |
+
color: var(--text);
|
| 283 |
+
font-size: 0.98rem;
|
| 284 |
+
line-height: 1.5;
|
| 285 |
+
}
|
| 286 |
+
|
| 287 |
+
.trigger {
|
| 288 |
+
color: var(--text-muted);
|
| 289 |
+
}
|
| 290 |
+
|
| 291 |
+
.sequence,
|
| 292 |
+
.parts {
|
| 293 |
+
display: grid;
|
| 294 |
+
gap: 8px;
|
| 295 |
+
padding: 0;
|
| 296 |
+
list-style: none;
|
| 297 |
+
}
|
| 298 |
+
|
| 299 |
+
.sequence li,
|
| 300 |
+
.parts li {
|
| 301 |
+
border-left: 1px solid var(--cyan-dim);
|
| 302 |
+
padding-left: 10px;
|
| 303 |
+
}
|
| 304 |
+
|
| 305 |
+
.parts strong {
|
| 306 |
+
display: block;
|
| 307 |
+
color: var(--cyan);
|
| 308 |
+
font-weight: 600;
|
| 309 |
+
text-transform: uppercase;
|
| 310 |
+
}
|
| 311 |
+
|
| 312 |
+
.json-box {
|
| 313 |
+
max-height: 240px;
|
| 314 |
+
overflow: auto;
|
| 315 |
+
margin: 0;
|
| 316 |
+
border-top: 1px solid rgba(122, 84, 32, 0.7);
|
| 317 |
+
padding-top: 14px;
|
| 318 |
+
color: var(--text-muted);
|
| 319 |
+
font: 0.78rem/1.45 "Fira Code", monospace;
|
| 320 |
+
white-space: pre-wrap;
|
| 321 |
+
}
|
| 322 |
+
|
| 323 |
+
.fallback-stage,
|
| 324 |
+
.scene-mount,
|
| 325 |
+
.label-layer {
|
| 326 |
+
position: absolute;
|
| 327 |
+
inset: 0;
|
| 328 |
+
}
|
| 329 |
+
|
| 330 |
+
.scene-label {
|
| 331 |
+
position: absolute;
|
| 332 |
+
max-width: min(180px, 34vw);
|
| 333 |
+
transform: translate(-50%, -50%);
|
| 334 |
+
border-left: 1px solid var(--cyan);
|
| 335 |
+
padding: 4px 0 4px 8px;
|
| 336 |
+
color: var(--cyan);
|
| 337 |
+
background: rgba(15, 19, 24, 0.72);
|
| 338 |
+
font: 0.72rem/1.2 "Fira Code", monospace;
|
| 339 |
+
text-transform: uppercase;
|
| 340 |
+
pointer-events: none;
|
| 341 |
+
}
|
| 342 |
+
|
| 343 |
+
@keyframes loading {
|
| 344 |
+
from { transform: translateX(-100%); }
|
| 345 |
+
to { transform: translateX(300%); }
|
| 346 |
+
}
|
| 347 |
+
|
| 348 |
+
@keyframes scan {
|
| 349 |
+
0% { transform: translateY(0); opacity: 0; }
|
| 350 |
+
12%, 75% { opacity: 1; }
|
| 351 |
+
100% { transform: translateY(100vh); opacity: 0; }
|
| 352 |
+
}
|
| 353 |
+
|
| 354 |
+
@media (max-width: 860px) {
|
| 355 |
+
body {
|
| 356 |
+
overflow: auto;
|
| 357 |
+
}
|
| 358 |
+
|
| 359 |
+
.shell {
|
| 360 |
+
grid-template-columns: 1fr;
|
| 361 |
+
grid-template-rows: minmax(420px, 62vh) auto;
|
| 362 |
+
min-height: 100vh;
|
| 363 |
+
height: auto;
|
| 364 |
+
}
|
| 365 |
+
|
| 366 |
+
.panel {
|
| 367 |
+
border-left: 0;
|
| 368 |
+
border-top: 1px solid var(--amber-dim);
|
| 369 |
+
}
|
| 370 |
+
}
|
| 371 |
+
</style>
|
| 372 |
+
</head>
|
| 373 |
+
<body>
|
| 374 |
+
<main class="shell">
|
| 375 |
+
<section class="viewport-pane" aria-label="Cutaway viewport">
|
| 376 |
+
<div id="progress" class="progress"></div>
|
| 377 |
+
<div class="toolbar">
|
| 378 |
+
<button id="uploadButton" class="tool-button" type="button">Load</button>
|
| 379 |
+
<button id="playButton" class="tool-button" type="button" disabled>Pause</button>
|
| 380 |
+
</div>
|
| 381 |
+
<div id="viewport"></div>
|
| 382 |
+
<label id="dropZone" class="drop-zone" for="fileInput">Drop component photo</label>
|
| 383 |
+
<div id="scanLine" class="scan-line"></div>
|
| 384 |
+
<input id="fileInput" type="file" accept="image/*" hidden>
|
| 385 |
+
</section>
|
| 386 |
+
|
| 387 |
+
<aside class="panel">
|
| 388 |
+
<div class="panel-inner">
|
| 389 |
+
<header class="brand">
|
| 390 |
+
<div class="kicker">Snap2Sim</div>
|
| 391 |
+
<h1>Inside the Machine</h1>
|
| 392 |
+
<div id="status" class="status">Standby</div>
|
| 393 |
+
</header>
|
| 394 |
+
|
| 395 |
+
<section class="readout">
|
| 396 |
+
<div class="metric-row">
|
| 397 |
+
<div id="component" class="component">Awaiting Photo</div>
|
| 398 |
+
<div id="confidence" class="confidence">--</div>
|
| 399 |
+
</div>
|
| 400 |
+
<p id="summary" class="summary"></p>
|
| 401 |
+
<p id="trigger" class="trigger"></p>
|
| 402 |
+
</section>
|
| 403 |
+
|
| 404 |
+
<section class="readout">
|
| 405 |
+
<div>
|
| 406 |
+
<div class="label">Motion Sequence</div>
|
| 407 |
+
<ol id="sequence" class="sequence"></ol>
|
| 408 |
+
</div>
|
| 409 |
+
<div>
|
| 410 |
+
<div class="label">Parts</div>
|
| 411 |
+
<ul id="parts" class="parts"></ul>
|
| 412 |
+
</div>
|
| 413 |
+
</section>
|
| 414 |
+
|
| 415 |
+
<pre id="rawJson" class="json-box">{}</pre>
|
| 416 |
+
</div>
|
| 417 |
+
</aside>
|
| 418 |
+
</main>
|
| 419 |
+
|
| 420 |
+
<script>
|
| 421 |
+
const fileInput = document.getElementById("fileInput");
|
| 422 |
+
const uploadButton = document.getElementById("uploadButton");
|
| 423 |
+
const playButton = document.getElementById("playButton");
|
| 424 |
+
const dropZone = document.getElementById("dropZone");
|
| 425 |
+
const viewport = document.getElementById("viewport");
|
| 426 |
+
const progress = document.getElementById("progress");
|
| 427 |
+
const scanLine = document.getElementById("scanLine");
|
| 428 |
+
const statusEl = document.getElementById("status");
|
| 429 |
+
const componentEl = document.getElementById("component");
|
| 430 |
+
const confidenceEl = document.getElementById("confidence");
|
| 431 |
+
const summaryEl = document.getElementById("summary");
|
| 432 |
+
const triggerEl = document.getElementById("trigger");
|
| 433 |
+
const sequenceEl = document.getElementById("sequence");
|
| 434 |
+
const partsEl = document.getElementById("parts");
|
| 435 |
+
const rawJsonEl = document.getElementById("rawJson");
|
| 436 |
+
|
| 437 |
+
let activeMode = "idle";
|
| 438 |
+
let paused = false;
|
| 439 |
+
let coldStartTimer = 0;
|
| 440 |
+
let fallbackRuntime = null;
|
| 441 |
+
|
| 442 |
+
uploadButton.addEventListener("click", () => fileInput.click());
|
| 443 |
+
fileInput.addEventListener("change", () => {
|
| 444 |
+
const file = fileInput.files && fileInput.files[0];
|
| 445 |
+
if (file) runPipeline(file);
|
| 446 |
+
});
|
| 447 |
+
|
| 448 |
+
for (const eventName of ["dragenter", "dragover"]) {
|
| 449 |
+
dropZone.addEventListener(eventName, (event) => {
|
| 450 |
+
event.preventDefault();
|
| 451 |
+
dropZone.classList.add("dragging");
|
| 452 |
+
});
|
| 453 |
+
}
|
| 454 |
+
|
| 455 |
+
for (const eventName of ["dragleave", "drop"]) {
|
| 456 |
+
dropZone.addEventListener(eventName, (event) => {
|
| 457 |
+
event.preventDefault();
|
| 458 |
+
dropZone.classList.remove("dragging");
|
| 459 |
+
});
|
| 460 |
+
}
|
| 461 |
+
|
| 462 |
+
dropZone.addEventListener("drop", (event) => {
|
| 463 |
+
const file = event.dataTransfer.files && event.dataTransfer.files[0];
|
| 464 |
+
if (file) runPipeline(file);
|
| 465 |
+
});
|
| 466 |
+
|
| 467 |
+
playButton.addEventListener("click", () => {
|
| 468 |
+
paused = !paused;
|
| 469 |
+
playButton.textContent = paused ? "Resume" : "Pause";
|
| 470 |
+
if (activeMode === "aframe") {
|
| 471 |
+
const scene = viewport.querySelector("a-scene");
|
| 472 |
+
if (scene) paused ? scene.pause() : scene.play();
|
| 473 |
+
}
|
| 474 |
+
if (fallbackRuntime) fallbackRuntime.playing = !paused;
|
| 475 |
+
});
|
| 476 |
+
|
| 477 |
+
async function runPipeline(file) {
|
| 478 |
+
resetScene();
|
| 479 |
+
setBusy(true);
|
| 480 |
+
setStatus("ANALYZING ASSEMBLY...");
|
| 481 |
+
coldStartTimer = window.setTimeout(() => setStatus("WAKING THE WORKSHOP..."), 6500);
|
| 482 |
+
|
| 483 |
+
try {
|
| 484 |
+
const imageBase64 = await fileToDataUrl(file);
|
| 485 |
+
const analysis = await postJson("/analyze_image", { image_base64: imageBase64 });
|
| 486 |
+
window.clearTimeout(coldStartTimer);
|
| 487 |
+
populateAnalysis(analysis);
|
| 488 |
+
|
| 489 |
+
setStatus("RENDERING CUTAWAY...");
|
| 490 |
+
const sceneResponse = await postJson("/generate_scene", { analysis });
|
| 491 |
+
const sceneHtml = typeof sceneResponse === "string" ? sceneResponse : sceneResponse.html;
|
| 492 |
+
renderAframe(sceneHtml, analysis);
|
| 493 |
+
} catch (error) {
|
| 494 |
+
window.clearTimeout(coldStartTimer);
|
| 495 |
+
setStatus(error.message || String(error), true);
|
| 496 |
+
if (window.lastAnalysis) buildDeterministicScene(window.lastAnalysis);
|
| 497 |
+
} finally {
|
| 498 |
+
setBusy(false);
|
| 499 |
+
}
|
| 500 |
+
}
|
| 501 |
+
|
| 502 |
+
function resetScene() {
|
| 503 |
+
if (fallbackRuntime && fallbackRuntime.cleanup) fallbackRuntime.cleanup();
|
| 504 |
+
fallbackRuntime = null;
|
| 505 |
+
activeMode = "idle";
|
| 506 |
+
paused = false;
|
| 507 |
+
playButton.textContent = "Pause";
|
| 508 |
+
playButton.disabled = true;
|
| 509 |
+
viewport.replaceChildren();
|
| 510 |
+
dropZone.classList.add("hidden");
|
| 511 |
+
}
|
| 512 |
+
|
| 513 |
+
function populateAnalysis(analysis) {
|
| 514 |
+
window.lastAnalysis = analysis;
|
| 515 |
+
componentEl.textContent = String(analysis.component || "Component").toUpperCase();
|
| 516 |
+
confidenceEl.textContent = typeof analysis.confidence === "number"
|
| 517 |
+
? Math.round(analysis.confidence * 100) + "%"
|
| 518 |
+
: "--";
|
| 519 |
+
summaryEl.textContent = analysis.summary || "";
|
| 520 |
+
triggerEl.textContent = analysis.trigger ? "Trigger: " + analysis.trigger : "";
|
| 521 |
+
rawJsonEl.textContent = JSON.stringify(analysis, null, 2);
|
| 522 |
+
sequenceEl.replaceChildren(...(analysis.motion_sequence || []).map((step) => {
|
| 523 |
+
const item = document.createElement("li");
|
| 524 |
+
item.textContent = step;
|
| 525 |
+
return item;
|
| 526 |
+
}));
|
| 527 |
+
partsEl.replaceChildren(...(analysis.parts || []).map((part) => {
|
| 528 |
+
const item = document.createElement("li");
|
| 529 |
+
const name = document.createElement("strong");
|
| 530 |
+
name.textContent = part.name || part.id || "part";
|
| 531 |
+
const role = document.createElement("span");
|
| 532 |
+
role.textContent = part.role || "";
|
| 533 |
+
item.append(name, role);
|
| 534 |
+
return item;
|
| 535 |
+
}));
|
| 536 |
+
}
|
| 537 |
+
|
| 538 |
+
function renderAframe(sceneHtml, analysis) {
|
| 539 |
+
if (!sceneHtml || !/<a-scene[\s>]/i.test(sceneHtml) || !/<\/a-scene>/i.test(sceneHtml)) {
|
| 540 |
+
buildDeterministicScene(analysis);
|
| 541 |
+
return;
|
| 542 |
+
}
|
| 543 |
+
|
| 544 |
+
activeMode = "aframe";
|
| 545 |
+
viewport.innerHTML = sceneHtml;
|
| 546 |
+
const scene = viewport.querySelector("a-scene");
|
| 547 |
+
if (scene) {
|
| 548 |
+
scene.setAttribute("embedded", "");
|
| 549 |
+
scene.setAttribute("vr-mode-ui", "enabled: false");
|
| 550 |
+
scene.setAttribute("device-orientation-permission-ui", "enabled: false");
|
| 551 |
+
scene.setAttribute("background", "color: #0F1318");
|
| 552 |
+
}
|
| 553 |
+
playButton.disabled = false;
|
| 554 |
+
revealScan();
|
| 555 |
+
|
| 556 |
+
window.setTimeout(() => {
|
| 557 |
+
const liveScene = viewport.querySelector("a-scene");
|
| 558 |
+
const hasCanvas = Boolean(liveScene && liveScene.canvas);
|
| 559 |
+
if (!hasCanvas) buildDeterministicScene(analysis);
|
| 560 |
+
}, 3000);
|
| 561 |
+
|
| 562 |
+
setStatus("CUTAWAY READY");
|
| 563 |
+
}
|
| 564 |
+
|
| 565 |
+
function buildDeterministicScene(analysis) {
|
| 566 |
+
const threeRuntime = window.THREE || (window.AFRAME && window.AFRAME.THREE);
|
| 567 |
+
if (!threeRuntime) {
|
| 568 |
+
setStatus("3D RUNTIME UNAVAILABLE", true);
|
| 569 |
+
return;
|
| 570 |
+
}
|
| 571 |
+
window.THREE = threeRuntime;
|
| 572 |
+
|
| 573 |
+
if (fallbackRuntime && fallbackRuntime.cleanup) fallbackRuntime.cleanup();
|
| 574 |
+
activeMode = "three";
|
| 575 |
+
paused = false;
|
| 576 |
+
playButton.textContent = "Pause";
|
| 577 |
+
playButton.disabled = false;
|
| 578 |
+
viewport.innerHTML = '<div class="fallback-stage"><div class="scene-mount"></div><div class="label-layer"></div></div>';
|
| 579 |
+
|
| 580 |
+
const mount = viewport.querySelector(".scene-mount");
|
| 581 |
+
const labelLayer = viewport.querySelector(".label-layer");
|
| 582 |
+
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
|
| 583 |
+
renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
|
| 584 |
+
mount.appendChild(renderer.domElement);
|
| 585 |
+
|
| 586 |
+
const scene = new THREE.Scene();
|
| 587 |
+
scene.fog = new THREE.Fog(0x0f1318, 8, 20);
|
| 588 |
+
|
| 589 |
+
const camera = new THREE.PerspectiveCamera(42, 1, 0.1, 100);
|
| 590 |
+
camera.position.set(5.4, 3.9, 6.2);
|
| 591 |
+
|
| 592 |
+
const controls = window.THREE.OrbitControls
|
| 593 |
+
? new THREE.OrbitControls(camera, renderer.domElement)
|
| 594 |
+
: null;
|
| 595 |
+
if (controls) {
|
| 596 |
+
controls.enableDamping = true;
|
| 597 |
+
controls.target.set(0, 0.1, 0);
|
| 598 |
+
}
|
| 599 |
+
|
| 600 |
+
scene.add(new THREE.HemisphereLight(0x5fd4d0, 0x0f1318, 1.2));
|
| 601 |
+
const key = new THREE.DirectionalLight(0xe8a33d, 1.9);
|
| 602 |
+
key.position.set(4, 6, 5);
|
| 603 |
+
scene.add(key);
|
| 604 |
+
|
| 605 |
+
const grid = new THREE.GridHelper(8, 32, 0x2a5e5c, 0x1e2530);
|
| 606 |
+
grid.material.transparent = true;
|
| 607 |
+
grid.material.opacity = 0.75;
|
| 608 |
+
scene.add(grid);
|
| 609 |
+
|
| 610 |
+
const meshes = (analysis.parts || []).slice(0, 6).map((part, index) => {
|
| 611 |
+
const mesh = buildPartMesh(part, index);
|
| 612 |
+
const position = part.geometry && part.geometry.position || [0, 0, 0];
|
| 613 |
+
mesh.position.set(position[0], position[1], position[2]);
|
| 614 |
+
if (part.geometry && Array.isArray(part.geometry.rotation)) {
|
| 615 |
+
mesh.rotation.fromArray(part.geometry.rotation);
|
| 616 |
+
}
|
| 617 |
+
mesh.userData.basePosition = mesh.position.clone();
|
| 618 |
+
mesh.userData.baseRotation = mesh.rotation.clone();
|
| 619 |
+
mesh.userData.part = part;
|
| 620 |
+
mesh.scale.setScalar(0.001);
|
| 621 |
+
scene.add(mesh);
|
| 622 |
+
return mesh;
|
| 623 |
+
});
|
| 624 |
+
|
| 625 |
+
const clock = new THREE.Clock();
|
| 626 |
+
let frameId = 0;
|
| 627 |
+
let disposed = false;
|
| 628 |
+
fallbackRuntime = {
|
| 629 |
+
playing: true,
|
| 630 |
+
cleanup() {
|
| 631 |
+
disposed = true;
|
| 632 |
+
cancelAnimationFrame(frameId);
|
| 633 |
+
renderer.dispose();
|
| 634 |
+
}
|
| 635 |
+
};
|
| 636 |
+
|
| 637 |
+
function resize() {
|
| 638 |
+
const rect = viewport.getBoundingClientRect();
|
| 639 |
+
camera.aspect = Math.max(1, rect.width) / Math.max(1, rect.height);
|
| 640 |
+
camera.updateProjectionMatrix();
|
| 641 |
+
renderer.setSize(Math.max(1, rect.width), Math.max(1, rect.height));
|
| 642 |
+
}
|
| 643 |
+
|
| 644 |
+
function animate() {
|
| 645 |
+
if (disposed) return;
|
| 646 |
+
frameId = requestAnimationFrame(animate);
|
| 647 |
+
const elapsed = clock.getElapsedTime();
|
| 648 |
+
for (const [index, mesh] of meshes.entries()) {
|
| 649 |
+
revealMesh(mesh, elapsed, index);
|
| 650 |
+
if (fallbackRuntime.playing) applyMotion(mesh, elapsed);
|
| 651 |
+
}
|
| 652 |
+
if (controls) controls.update();
|
| 653 |
+
renderer.render(scene, camera);
|
| 654 |
+
updateLabels(camera, labelLayer, meshes);
|
| 655 |
+
}
|
| 656 |
+
|
| 657 |
+
window.addEventListener("resize", resize);
|
| 658 |
+
const originalCleanup = fallbackRuntime.cleanup;
|
| 659 |
+
fallbackRuntime.cleanup = () => {
|
| 660 |
+
window.removeEventListener("resize", resize);
|
| 661 |
+
originalCleanup();
|
| 662 |
+
};
|
| 663 |
+
|
| 664 |
+
resize();
|
| 665 |
+
animate();
|
| 666 |
+
revealScan();
|
| 667 |
+
setStatus("CUTAWAY READY");
|
| 668 |
+
}
|
| 669 |
+
|
| 670 |
+
function buildPartMesh(part, index) {
|
| 671 |
+
const geometryData = part.geometry || {};
|
| 672 |
+
const size = Array.isArray(geometryData.size) ? geometryData.size : [1, 1, 1];
|
| 673 |
+
let geometry;
|
| 674 |
+
if (geometryData.shape === "cylinder") {
|
| 675 |
+
geometry = new THREE.CylinderGeometry(size[0] / 2, size[2] / 2, size[1], 48);
|
| 676 |
+
} else if (geometryData.shape === "sphere") {
|
| 677 |
+
geometry = new THREE.SphereGeometry(Math.max(size[0], size[1], size[2]) / 2, 36, 18);
|
| 678 |
+
} else if (geometryData.shape === "rod") {
|
| 679 |
+
geometry = new THREE.CylinderGeometry(Math.max(size[0], size[1]) / 2, Math.max(size[0], size[1]) / 2, size[2], 24);
|
| 680 |
+
geometry.rotateX(Math.PI / 2);
|
| 681 |
+
} else if (geometryData.shape === "gear") {
|
| 682 |
+
geometry = gearGeometry(Math.max(size[0], size[2]) / 2, size[1], geometryData.teeth || 18);
|
| 683 |
+
} else {
|
| 684 |
+
geometry = new THREE.BoxGeometry(size[0], size[1], size[2]);
|
| 685 |
+
}
|
| 686 |
+
|
| 687 |
+
const color = colorFor(geometryData.color, index);
|
| 688 |
+
const material = new THREE.MeshStandardMaterial({
|
| 689 |
+
color,
|
| 690 |
+
metalness: 0.48,
|
| 691 |
+
roughness: 0.34,
|
| 692 |
+
emissive: color,
|
| 693 |
+
emissiveIntensity: 0.04
|
| 694 |
+
});
|
| 695 |
+
return new THREE.Mesh(geometry, material);
|
| 696 |
+
}
|
| 697 |
+
|
| 698 |
+
function gearGeometry(radius, depth, teeth) {
|
| 699 |
+
const shape = new THREE.Shape();
|
| 700 |
+
const steps = teeth * 2;
|
| 701 |
+
for (let i = 0; i <= steps; i += 1) {
|
| 702 |
+
const angle = (i / steps) * Math.PI * 2;
|
| 703 |
+
const r = radius * (i % 2 === 0 ? 1 : 0.84);
|
| 704 |
+
const x = Math.cos(angle) * r;
|
| 705 |
+
const y = Math.sin(angle) * r;
|
| 706 |
+
if (i === 0) shape.moveTo(x, y);
|
| 707 |
+
else shape.lineTo(x, y);
|
| 708 |
+
}
|
| 709 |
+
const hole = new THREE.Path();
|
| 710 |
+
hole.absarc(0, 0, radius * 0.26, 0, Math.PI * 2, true);
|
| 711 |
+
shape.holes.push(hole);
|
| 712 |
+
const geometry = new THREE.ExtrudeGeometry(shape, { depth, bevelEnabled: false });
|
| 713 |
+
geometry.center();
|
| 714 |
+
geometry.rotateX(Math.PI / 2);
|
| 715 |
+
return geometry;
|
| 716 |
+
}
|
| 717 |
+
|
| 718 |
+
function revealMesh(mesh, elapsed, index) {
|
| 719 |
+
const local = Math.max(0, Math.min(1, (elapsed - index * 0.1) / 0.55));
|
| 720 |
+
const eased = 1 - Math.pow(1 - local, 3);
|
| 721 |
+
mesh.scale.setScalar(eased);
|
| 722 |
+
}
|
| 723 |
+
|
| 724 |
+
function applyMotion(mesh, elapsed) {
|
| 725 |
+
const motion = mesh.userData.part.motion || { type: "static" };
|
| 726 |
+
const speed = Number(motion.speed || 1);
|
| 727 |
+
const phase = Number(motion.phase || 0);
|
| 728 |
+
const axis = new THREE.Vector3(...(motion.axis || [0, 1, 0])).normalize();
|
| 729 |
+
mesh.position.copy(mesh.userData.basePosition);
|
| 730 |
+
mesh.rotation.copy(mesh.userData.baseRotation);
|
| 731 |
+
if (motion.type === "rotate") {
|
| 732 |
+
mesh.rotateOnAxis(axis, elapsed * speed + phase);
|
| 733 |
+
} else if (motion.type === "oscillate") {
|
| 734 |
+
mesh.rotateOnAxis(axis, Math.sin(elapsed * speed + phase) * Number(motion.amplitude || 0.25));
|
| 735 |
+
} else if (motion.type === "translate") {
|
| 736 |
+
const range = motion.range || [-0.25, 0.25];
|
| 737 |
+
const offset = range[0] + (range[1] - range[0]) * ((Math.sin(elapsed * speed + phase) + 1) / 2);
|
| 738 |
+
mesh.position.add(axis.multiplyScalar(offset));
|
| 739 |
+
}
|
| 740 |
+
}
|
| 741 |
+
|
| 742 |
+
function updateLabels(camera, labelLayer, meshes) {
|
| 743 |
+
const rect = viewport.getBoundingClientRect();
|
| 744 |
+
labelLayer.replaceChildren();
|
| 745 |
+
for (const mesh of meshes) {
|
| 746 |
+
const projected = mesh.position.clone().project(camera);
|
| 747 |
+
if (projected.z < -1 || projected.z > 1) continue;
|
| 748 |
+
const label = document.createElement("div");
|
| 749 |
+
label.className = "scene-label";
|
| 750 |
+
label.style.left = Math.min(rect.width - 72, Math.max(72, (projected.x * 0.5 + 0.5) * rect.width)) + "px";
|
| 751 |
+
label.style.top = Math.min(rect.height - 36, Math.max(56, (-projected.y * 0.5 + 0.5) * rect.height)) + "px";
|
| 752 |
+
label.textContent = mesh.userData.part.name || mesh.userData.part.id || "part";
|
| 753 |
+
labelLayer.appendChild(label);
|
| 754 |
+
}
|
| 755 |
+
}
|
| 756 |
+
|
| 757 |
+
function colorFor(name, index) {
|
| 758 |
+
const key = String(name || "").toLowerCase();
|
| 759 |
+
if (key.includes("amber") || key.includes("orange")) return 0xe8a33d;
|
| 760 |
+
if (key.includes("cyan")) return 0x5fd4d0;
|
| 761 |
+
if (key.includes("steel")) return 0x9aa4a6;
|
| 762 |
+
return [0x5fd4d0, 0xe8a33d, 0xc8c0ac, 0x80b8ff, 0xf07f5a, 0xa7d676][index % 6];
|
| 763 |
+
}
|
| 764 |
+
|
| 765 |
+
async function postJson(url, payload) {
|
| 766 |
+
const response = await fetch(url, {
|
| 767 |
+
method: "POST",
|
| 768 |
+
headers: { "Content-Type": "application/json" },
|
| 769 |
+
body: JSON.stringify(payload)
|
| 770 |
+
});
|
| 771 |
+
if (!response.ok) {
|
| 772 |
+
let message = response.statusText || "Request failed";
|
| 773 |
+
try {
|
| 774 |
+
const data = await response.json();
|
| 775 |
+
message = data.detail || data.error || message;
|
| 776 |
+
} catch (_) {}
|
| 777 |
+
throw new Error(message);
|
| 778 |
+
}
|
| 779 |
+
return response.json();
|
| 780 |
+
}
|
| 781 |
+
|
| 782 |
+
function fileToDataUrl(file) {
|
| 783 |
+
return new Promise((resolve, reject) => {
|
| 784 |
+
const reader = new FileReader();
|
| 785 |
+
reader.onload = () => resolve(String(reader.result || ""));
|
| 786 |
+
reader.onerror = () => reject(reader.error || new Error("File read failed"));
|
| 787 |
+
reader.readAsDataURL(file);
|
| 788 |
+
});
|
| 789 |
+
}
|
| 790 |
+
|
| 791 |
+
function setBusy(active) {
|
| 792 |
+
progress.classList.toggle("active", active);
|
| 793 |
+
uploadButton.disabled = active;
|
| 794 |
+
}
|
| 795 |
+
|
| 796 |
+
function setStatus(message, error) {
|
| 797 |
+
statusEl.textContent = String(message || "STANDBY").toUpperCase();
|
| 798 |
+
statusEl.classList.toggle("error", Boolean(error));
|
| 799 |
+
}
|
| 800 |
+
|
| 801 |
+
function revealScan() {
|
| 802 |
+
scanLine.classList.remove("active");
|
| 803 |
+
void scanLine.offsetWidth;
|
| 804 |
+
scanLine.classList.add("active");
|
| 805 |
+
}
|
| 806 |
+
</script>
|
| 807 |
+
</body>
|
| 808 |
+
</html>
|
modal_app.py
CHANGED
|
@@ -16,10 +16,10 @@ import secrets as token_secrets
|
|
| 16 |
import modal
|
| 17 |
from fastapi import Header, HTTPException
|
| 18 |
|
| 19 |
-
from snap2sim.
|
| 20 |
-
from snap2sim.
|
|
|
|
| 21 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 22 |
-
from snap2sim.three_scene import build_threejs_html
|
| 23 |
|
| 24 |
|
| 25 |
DEFAULT_MODEL_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
|
|
@@ -81,14 +81,14 @@ def check_remote_imports() -> dict[str, Any]:
|
|
| 81 |
"""Lightweight Modal check that local project modules are packaged."""
|
| 82 |
import snap2sim.model_io
|
| 83 |
import snap2sim.schema
|
| 84 |
-
import snap2sim.
|
| 85 |
|
| 86 |
return {
|
| 87 |
"ok": True,
|
| 88 |
"modules": [
|
| 89 |
snap2sim.model_io.__name__,
|
| 90 |
snap2sim.schema.__name__,
|
| 91 |
-
snap2sim.
|
| 92 |
],
|
| 93 |
}
|
| 94 |
|
|
@@ -413,7 +413,7 @@ def runtime_probe(authorization: str = Header(default="")) -> dict[str, Any]:
|
|
| 413 |
else "runtime selected"
|
| 414 |
),
|
| 415 |
"verified_endpoint": "analyze_image_llamacpp",
|
| 416 |
-
"recommended_generate_endpoint": "
|
| 417 |
}
|
| 418 |
|
| 419 |
|
|
@@ -447,23 +447,23 @@ def analyze_image_llamacpp(payload: dict[str, Any], authorization: str = Header(
|
|
| 447 |
|
| 448 |
@app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
|
| 449 |
@modal.fastapi_endpoint(method="POST")
|
| 450 |
-
def
|
| 451 |
require_authorization(authorization)
|
| 452 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 453 |
-
_prompt =
|
| 454 |
if runtime_config()["runtime_mode"] != "placeholder":
|
| 455 |
raise NotImplementedError(
|
| 456 |
"Nemotron runtime is selected but not wired yet. Keep placeholder "
|
| 457 |
"mode active until the llama.cpp/vLLM adapter is implemented."
|
| 458 |
)
|
| 459 |
-
return {"html":
|
| 460 |
|
| 461 |
|
| 462 |
@app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
|
| 463 |
@modal.fastapi_endpoint(method="POST")
|
| 464 |
-
def
|
| 465 |
-
"""Experimental GPU endpoint for llama.cpp
|
| 466 |
require_authorization(authorization)
|
| 467 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 468 |
-
response = run_llamacpp_prompt(
|
| 469 |
-
return {"html":
|
|
|
|
| 16 |
import modal
|
| 17 |
from fastapi import Header, HTTPException
|
| 18 |
|
| 19 |
+
from snap2sim.aframe_scene import build_aframe_scene
|
| 20 |
+
from snap2sim.model_io import coerce_analysis_response, parse_analysis_response, parse_scene_response
|
| 21 |
+
from snap2sim.prompts import build_scene_prompt, build_vision_prompt
|
| 22 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
|
|
|
| 23 |
|
| 24 |
|
| 25 |
DEFAULT_MODEL_REPO = "unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
|
|
|
|
| 81 |
"""Lightweight Modal check that local project modules are packaged."""
|
| 82 |
import snap2sim.model_io
|
| 83 |
import snap2sim.schema
|
| 84 |
+
import snap2sim.aframe_scene
|
| 85 |
|
| 86 |
return {
|
| 87 |
"ok": True,
|
| 88 |
"modules": [
|
| 89 |
snap2sim.model_io.__name__,
|
| 90 |
snap2sim.schema.__name__,
|
| 91 |
+
snap2sim.aframe_scene.__name__,
|
| 92 |
],
|
| 93 |
}
|
| 94 |
|
|
|
|
| 413 |
else "runtime selected"
|
| 414 |
),
|
| 415 |
"verified_endpoint": "analyze_image_llamacpp",
|
| 416 |
+
"recommended_generate_endpoint": "generate_scene",
|
| 417 |
}
|
| 418 |
|
| 419 |
|
|
|
|
| 447 |
|
| 448 |
@app.function(image=image, volumes={CACHE_DIR: model_cache}, timeout=600, secrets=[api_auth_secret])
|
| 449 |
@modal.fastapi_endpoint(method="POST")
|
| 450 |
+
def generate_scene(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
|
| 451 |
require_authorization(authorization)
|
| 452 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 453 |
+
_prompt = build_scene_prompt(analysis)
|
| 454 |
if runtime_config()["runtime_mode"] != "placeholder":
|
| 455 |
raise NotImplementedError(
|
| 456 |
"Nemotron runtime is selected but not wired yet. Keep placeholder "
|
| 457 |
"mode active until the llama.cpp/vLLM adapter is implemented."
|
| 458 |
)
|
| 459 |
+
return {"html": build_aframe_scene(analysis)}
|
| 460 |
|
| 461 |
|
| 462 |
@app.function(image=llamacpp_image, gpu=os.getenv("SNAP2SIM_RUNTIME_GPU", "L40S"), volumes={CACHE_DIR: model_cache}, timeout=60 * 60, secrets=[api_auth_secret])
|
| 463 |
@modal.fastapi_endpoint(method="POST")
|
| 464 |
+
def generate_scene_llamacpp(payload: dict[str, Any], authorization: str = Header(default="")) -> dict[str, str]:
|
| 465 |
+
"""Experimental GPU endpoint for llama.cpp A-Frame scene generation."""
|
| 466 |
require_authorization(authorization)
|
| 467 |
analysis = validate_analysis(payload.get("analysis") or EXAMPLE_ANALYSIS)
|
| 468 |
+
response = run_llamacpp_prompt(build_scene_prompt(analysis), max_tokens=4096)
|
| 469 |
+
return {"html": parse_scene_response(response)}
|
requirements.txt
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
gradio>=
|
| 2 |
huggingface_hub[hf_xet]>=0.36
|
| 3 |
requests>=2.32
|
| 4 |
pillow>=10.4
|
|
|
|
| 1 |
+
gradio>=6.0.1
|
| 2 |
huggingface_hub[hf_xet]>=0.36
|
| 3 |
requests>=2.32
|
| 4 |
pillow>=10.4
|
snap2sim/aframe_scene.py
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Deterministic A-Frame scene generation from a validated mechanism payload."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import html
|
| 6 |
+
from typing import Any
|
| 7 |
+
|
| 8 |
+
from snap2sim.schema import validate_analysis
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def build_aframe_scene(analysis: dict[str, Any]) -> str:
|
| 12 |
+
"""Return a declarative A-Frame scene for the browser to inject."""
|
| 13 |
+
valid_analysis = validate_analysis(analysis)
|
| 14 |
+
title = html.escape(str(valid_analysis.get("component", "mechanism")).upper(), quote=True)
|
| 15 |
+
summary = html.escape(str(valid_analysis.get("summary", "")), quote=True)
|
| 16 |
+
entities = "\n".join(_part_entity(part, index) for index, part in enumerate(valid_analysis["parts"][:6]))
|
| 17 |
+
return f"""<a-scene embedded vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false" background="color: #0F1318" renderer="colorManagement: true">
|
| 18 |
+
<a-sky color="#0F1318"></a-sky>
|
| 19 |
+
<a-entity light="type: ambient; intensity: 0.55; color: #5FD4D0"></a-entity>
|
| 20 |
+
<a-entity light="type: directional; intensity: 1.35; color: #E8A33D" position="3 5 4"></a-entity>
|
| 21 |
+
<a-entity position="0 1.1 5.6">
|
| 22 |
+
<a-camera look-controls wasd-controls="enabled: false"></a-camera>
|
| 23 |
+
</a-entity>
|
| 24 |
+
<a-plane color="#111820" opacity="0.72" rotation="-90 0 0" position="0 -1.05 -0.2" width="8" height="8"></a-plane>
|
| 25 |
+
<a-text value="{title}" position="-2.8 2.35 -2.6" color="#C8C0AC" width="4.8" align="left"></a-text>
|
| 26 |
+
<a-text value="{summary}" position="-2.8 2.05 -2.6" color="#6B7280" width="5.2" align="left"></a-text>
|
| 27 |
+
{entities}
|
| 28 |
+
</a-scene>"""
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _part_entity(part: dict[str, Any], index: int) -> str:
|
| 32 |
+
geometry = part["geometry"]
|
| 33 |
+
shape = geometry["shape"]
|
| 34 |
+
size = geometry["size"]
|
| 35 |
+
position = _vec(geometry["position"])
|
| 36 |
+
rotation = _rotation(geometry.get("rotation", [0, 0, 0]))
|
| 37 |
+
color = _color(geometry.get("color"), index)
|
| 38 |
+
label = html.escape(str(part.get("name", part.get("id", "part"))), quote=True)
|
| 39 |
+
animation = _animation(part)
|
| 40 |
+
|
| 41 |
+
if shape in {"cylinder", "gear"}:
|
| 42 |
+
radius = max(float(size[0]), float(size[2])) / 2
|
| 43 |
+
primitive = (
|
| 44 |
+
f'<a-cylinder radius="{radius:.3f}" height="{float(size[1]):.3f}" '
|
| 45 |
+
f'segments-radial="{int(geometry.get("teeth", 32)) if shape == "gear" else 48}"'
|
| 46 |
+
)
|
| 47 |
+
elif shape == "sphere":
|
| 48 |
+
primitive = f'<a-sphere radius="{max(float(item) for item in size) / 2:.3f}"'
|
| 49 |
+
elif shape == "rod":
|
| 50 |
+
primitive = f'<a-cylinder radius="{max(float(size[0]), float(size[1])) / 2:.3f}" height="{float(size[2]):.3f}"'
|
| 51 |
+
rotation = _rotation([1.5708, 0, 0])
|
| 52 |
+
else:
|
| 53 |
+
primitive = f'<a-box width="{float(size[0]):.3f}" height="{float(size[1]):.3f}" depth="{float(size[2]):.3f}"'
|
| 54 |
+
|
| 55 |
+
return f""" <a-entity position="{position}">
|
| 56 |
+
{primitive} material="color: {color}; metalness: 0.45; roughness: 0.38" rotation="{rotation}" {animation}></a-cylinder>
|
| 57 |
+
<a-text value="{label}" position="0 {float(size[1]) + 0.28:.3f} 0" color="#5FD4D0" width="2.4" align="center"></a-text>
|
| 58 |
+
</a-entity>""".replace("</a-cylinder>", _closing_tag(shape))
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def _closing_tag(shape: str) -> str:
|
| 62 |
+
if shape == "sphere":
|
| 63 |
+
return "</a-sphere>"
|
| 64 |
+
if shape == "box":
|
| 65 |
+
return "</a-box>"
|
| 66 |
+
return "</a-cylinder>"
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def _animation(part: dict[str, Any]) -> str:
|
| 70 |
+
motion = part.get("motion") or {"type": "static"}
|
| 71 |
+
motion_type = motion.get("type")
|
| 72 |
+
if motion_type == "static":
|
| 73 |
+
return ""
|
| 74 |
+
|
| 75 |
+
speed = max(0.1, abs(float(motion.get("speed", 1))))
|
| 76 |
+
duration = int(max(700, min(9000, 3600 / speed)))
|
| 77 |
+
axis = motion.get("axis") if isinstance(motion.get("axis"), list) else [0, 1, 0]
|
| 78 |
+
|
| 79 |
+
if motion_type == "translate":
|
| 80 |
+
move_range = motion.get("range") if isinstance(motion.get("range"), list) else [-0.25, 0.25]
|
| 81 |
+
target = [float(axis[i]) * float(move_range[-1]) for i in range(3)]
|
| 82 |
+
return (
|
| 83 |
+
f'animation="property: position; to: {_vec(target)}; dir: alternate; '
|
| 84 |
+
f'loop: true; dur: {duration}; easing: easeInOutSine"'
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
rotation = [0, 0, 0]
|
| 88 |
+
if abs(float(axis[0])) >= max(abs(float(axis[1])), abs(float(axis[2]))):
|
| 89 |
+
rotation[0] = 360
|
| 90 |
+
elif abs(float(axis[2])) >= abs(float(axis[1])):
|
| 91 |
+
rotation[2] = 360
|
| 92 |
+
else:
|
| 93 |
+
rotation[1] = 360
|
| 94 |
+
if motion_type == "oscillate":
|
| 95 |
+
amplitude = int(float(motion.get("amplitude", 0.25)) * 90)
|
| 96 |
+
rotation = [value and amplitude for value in rotation]
|
| 97 |
+
return (
|
| 98 |
+
f'animation="property: rotation; to: {_vec(rotation)}; dir: alternate; '
|
| 99 |
+
f'loop: true; dur: {duration}; easing: easeInOutSine"'
|
| 100 |
+
)
|
| 101 |
+
return f'animation="property: rotation; to: {_vec(rotation)}; loop: true; dur: {duration}; easing: linear"'
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def _vec(values: list[Any]) -> str:
|
| 105 |
+
return " ".join(f"{float(value):.3f}" for value in values)
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def _rotation(values: list[Any]) -> str:
|
| 109 |
+
return " ".join(f"{float(value) * 57.2958:.2f}" for value in values)
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def _color(value: Any, index: int) -> str:
|
| 113 |
+
key = str(value or "").lower()
|
| 114 |
+
if "amber" in key or "orange" in key:
|
| 115 |
+
return "#E8A33D"
|
| 116 |
+
if "cyan" in key:
|
| 117 |
+
return "#5FD4D0"
|
| 118 |
+
if "steel" in key:
|
| 119 |
+
return "#9AA4A6"
|
| 120 |
+
return ["#5FD4D0", "#E8A33D", "#C8C0AC", "#80B8FF", "#F07F5A", "#A7D676"][index % 6]
|
snap2sim/backend.py
CHANGED
|
@@ -11,8 +11,8 @@ from typing import Any
|
|
| 11 |
import requests
|
| 12 |
from PIL import Image
|
| 13 |
|
|
|
|
| 14 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
| 15 |
-
from snap2sim.three_scene import build_threejs_html
|
| 16 |
|
| 17 |
|
| 18 |
@dataclass(frozen=True)
|
|
@@ -45,7 +45,7 @@ class InferenceClient:
|
|
| 45 |
|
| 46 |
return validate_analysis(dict(EXAMPLE_ANALYSIS))
|
| 47 |
|
| 48 |
-
def
|
| 49 |
valid_analysis = validate_analysis(analysis)
|
| 50 |
if self.settings.backend == "modal":
|
| 51 |
response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
|
|
@@ -54,7 +54,7 @@ class InferenceClient:
|
|
| 54 |
raise RuntimeError("Modal response did not include generated HTML.")
|
| 55 |
return str(html)
|
| 56 |
|
| 57 |
-
return
|
| 58 |
|
| 59 |
def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
|
| 60 |
if not url:
|
|
|
|
| 11 |
import requests
|
| 12 |
from PIL import Image
|
| 13 |
|
| 14 |
+
from snap2sim.aframe_scene import build_aframe_scene
|
| 15 |
from snap2sim.schema import EXAMPLE_ANALYSIS, validate_analysis
|
|
|
|
| 16 |
|
| 17 |
|
| 18 |
@dataclass(frozen=True)
|
|
|
|
| 45 |
|
| 46 |
return validate_analysis(dict(EXAMPLE_ANALYSIS))
|
| 47 |
|
| 48 |
+
def generate_scene(self, analysis: dict[str, Any]) -> str:
|
| 49 |
valid_analysis = validate_analysis(analysis)
|
| 50 |
if self.settings.backend == "modal":
|
| 51 |
response = self._post_json(self.settings.generate_url, {"analysis": valid_analysis})
|
|
|
|
| 54 |
raise RuntimeError("Modal response did not include generated HTML.")
|
| 55 |
return str(html)
|
| 56 |
|
| 57 |
+
return build_aframe_scene(valid_analysis)
|
| 58 |
|
| 59 |
def _post_json(self, url: str, payload: dict[str, Any]) -> dict[str, Any]:
|
| 60 |
if not url:
|
snap2sim/fallback_scene.py
DELETED
|
@@ -1,72 +0,0 @@
|
|
| 1 |
-
"""Themed fallback visualization used when generated Three.js is unavailable."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
def build_fallback_html(component: str) -> str:
|
| 7 |
-
title = component.upper()
|
| 8 |
-
return f"""
|
| 9 |
-
<div class="fallback-cutaway" role="img" aria-label="Fallback mechanism animation for {component}">
|
| 10 |
-
<div class="scan"></div>
|
| 11 |
-
<svg viewBox="0 0 760 430" xmlns="http://www.w3.org/2000/svg">
|
| 12 |
-
<defs>
|
| 13 |
-
<pattern id="grid" width="24" height="24" patternUnits="userSpaceOnUse">
|
| 14 |
-
<path d="M 24 0 L 0 0 0 24" fill="none" stroke="rgba(95,212,208,.14)" stroke-width="1"/>
|
| 15 |
-
</pattern>
|
| 16 |
-
</defs>
|
| 17 |
-
<rect width="760" height="430" fill="#14181F"/>
|
| 18 |
-
<rect width="760" height="430" fill="url(#grid)"/>
|
| 19 |
-
<g transform="translate(380 220)">
|
| 20 |
-
<circle r="104" fill="none" stroke="#5FD4D0" stroke-width="2" opacity=".8"/>
|
| 21 |
-
<circle class="fallback-gear" r="72" fill="none" stroke="#E8A33D" stroke-width="18"
|
| 22 |
-
stroke-dasharray="18 10"/>
|
| 23 |
-
<rect class="fallback-pawl" x="72" y="-16" width="132" height="32" rx="2"
|
| 24 |
-
fill="#E8A33D" opacity=".9"/>
|
| 25 |
-
<line x1="108" y1="-54" x2="215" y2="-112" stroke="#5FD4D0" stroke-width="1"/>
|
| 26 |
-
<text x="224" y="-116" fill="#5FD4D0" font-family="monospace" font-size="15">LOCKING PAWL</text>
|
| 27 |
-
<line x1="-54" y1="52" x2="-198" y2="122" stroke="#5FD4D0" stroke-width="1"/>
|
| 28 |
-
<text x="-342" y="132" fill="#5FD4D0" font-family="monospace" font-size="15">DRIVE GEAR</text>
|
| 29 |
-
</g>
|
| 30 |
-
<text x="32" y="44" fill="#D9D3C7" font-family="monospace" font-size="18">{title}</text>
|
| 31 |
-
<text x="32" y="394" fill="#E8A33D" font-family="monospace" font-size="13">2D FALLBACK: GENERATED 3D SCENE UNAVAILABLE</text>
|
| 32 |
-
</svg>
|
| 33 |
-
</div>
|
| 34 |
-
<style>
|
| 35 |
-
.fallback-cutaway {{
|
| 36 |
-
position: relative;
|
| 37 |
-
overflow: hidden;
|
| 38 |
-
border: 1px solid rgba(95, 212, 208, .28);
|
| 39 |
-
background: #14181F;
|
| 40 |
-
min-height: 430px;
|
| 41 |
-
}}
|
| 42 |
-
.fallback-cutaway svg {{
|
| 43 |
-
display: block;
|
| 44 |
-
width: 100%;
|
| 45 |
-
height: min(58vh, 560px);
|
| 46 |
-
}}
|
| 47 |
-
.fallback-gear {{
|
| 48 |
-
transform-origin: center;
|
| 49 |
-
animation: fallback-spin 3.8s linear infinite;
|
| 50 |
-
}}
|
| 51 |
-
.fallback-pawl {{
|
| 52 |
-
transform-origin: 74px 0;
|
| 53 |
-
animation: fallback-pawl 1.05s steps(2, end) infinite;
|
| 54 |
-
}}
|
| 55 |
-
.scan {{
|
| 56 |
-
position: absolute;
|
| 57 |
-
inset: 0;
|
| 58 |
-
background: linear-gradient(180deg, transparent, rgba(95, 212, 208, .16), transparent);
|
| 59 |
-
height: 26%;
|
| 60 |
-
animation: fallback-scan 2.6s ease-in-out infinite;
|
| 61 |
-
pointer-events: none;
|
| 62 |
-
}}
|
| 63 |
-
@keyframes fallback-spin {{ to {{ transform: rotate(360deg); }} }}
|
| 64 |
-
@keyframes fallback-pawl {{ 50% {{ transform: rotate(-10deg); }} }}
|
| 65 |
-
@keyframes fallback-scan {{
|
| 66 |
-
0% {{ transform: translateY(-120%); opacity: 0; }}
|
| 67 |
-
20%, 80% {{ opacity: 1; }}
|
| 68 |
-
100% {{ transform: translateY(430%); opacity: 0; }}
|
| 69 |
-
}}
|
| 70 |
-
</style>
|
| 71 |
-
"""
|
| 72 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
snap2sim/model_io.py
CHANGED
|
@@ -46,26 +46,21 @@ def coerce_analysis_response(text: str) -> dict[str, Any]:
|
|
| 46 |
return validate_analysis(_generic_analysis(fallback_component))
|
| 47 |
|
| 48 |
|
| 49 |
-
def
|
| 50 |
-
"""Extract a complete
|
| 51 |
raw = _strip_fences(text).strip()
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
| 57 |
|
| 58 |
|
| 59 |
def _strip_fences(text: str) -> str:
|
| 60 |
return _FENCE_RE.sub("", text.strip()).strip()
|
| 61 |
|
| 62 |
|
| 63 |
-
def _find_html_start(text: str) -> int:
|
| 64 |
-
lowered = text.lower()
|
| 65 |
-
starts = [index for index in [lowered.find("<!doctype"), lowered.find("<html")] if index >= 0]
|
| 66 |
-
return min(starts) if starts else -1
|
| 67 |
-
|
| 68 |
-
|
| 69 |
def _json_object_candidates(text: str) -> list[tuple[int, str]]:
|
| 70 |
candidates = []
|
| 71 |
for index, char in enumerate(text):
|
|
|
|
| 46 |
return validate_analysis(_generic_analysis(fallback_component))
|
| 47 |
|
| 48 |
|
| 49 |
+
def parse_scene_response(text: str) -> str:
|
| 50 |
+
"""Extract a complete A-Frame scene block from a model response."""
|
| 51 |
raw = _strip_fences(text).strip()
|
| 52 |
+
lowered = raw.lower()
|
| 53 |
+
scene_start = lowered.find("<a-scene")
|
| 54 |
+
scene_end = lowered.rfind("</a-scene>")
|
| 55 |
+
if scene_start < 0 or scene_end < 0:
|
| 56 |
+
raise ValueError("Model response did not contain a complete <a-scene> block.")
|
| 57 |
+
return raw[scene_start : scene_end + len("</a-scene>")]
|
| 58 |
|
| 59 |
|
| 60 |
def _strip_fences(text: str) -> str:
|
| 61 |
return _FENCE_RE.sub("", text.strip()).strip()
|
| 62 |
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
def _json_object_candidates(text: str) -> list[tuple[int, str]]:
|
| 65 |
candidates = []
|
| 66 |
for index, char in enumerate(text):
|
snap2sim/prompts.py
CHANGED
|
@@ -18,7 +18,7 @@ def build_vision_prompt() -> str:
|
|
| 18 |
"Analyze the uploaded hardware component as a cutaway mechanism. "
|
| 19 |
"Answer with only one JSON object. Do not include markdown. Do not "
|
| 20 |
"include a reasoning trace or <think> tags. Keep the payload compact and physically "
|
| 21 |
-
"plausible for primitive Three.js rendering.\n\n"
|
| 22 |
"Required top-level keys: component, confidence, summary, trigger, "
|
| 23 |
"motion_sequence, parts.\n"
|
| 24 |
"Each part requires: id, name, role, geometry, motion.\n"
|
|
@@ -48,17 +48,23 @@ def build_vision_prompt() -> str:
|
|
| 48 |
)
|
| 49 |
|
| 50 |
|
| 51 |
-
|
| 52 |
-
Return
|
| 53 |
-
|
| 54 |
-
|
| 55 |
|
| 56 |
|
| 57 |
-
def
|
| 58 |
return (
|
| 59 |
-
"Build a technical cutaway / field manual
|
| 60 |
-
"mechanism analysis.
|
| 61 |
-
"
|
| 62 |
-
"a
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
|
| 64 |
)
|
|
|
|
| 18 |
"Analyze the uploaded hardware component as a cutaway mechanism. "
|
| 19 |
"Answer with only one JSON object. Do not include markdown. Do not "
|
| 20 |
"include a reasoning trace or <think> tags. Keep the payload compact and physically "
|
| 21 |
+
"plausible for primitive A-Frame or Three.js rendering.\n\n"
|
| 22 |
"Required top-level keys: component, confidence, summary, trigger, "
|
| 23 |
"motion_sequence, parts.\n"
|
| 24 |
"Each part requires: id, name, role, geometry, motion.\n"
|
|
|
|
| 48 |
)
|
| 49 |
|
| 50 |
|
| 51 |
+
AFRAME_SYSTEM_PROMPT = """You generate declarative A-Frame cutaway scenes.
|
| 52 |
+
Return only one <a-scene>...</a-scene> block. Use primitive A-Frame entities,
|
| 53 |
+
short labels, and animation attributes. Do not include scripts, markdown fences,
|
| 54 |
+
explanations, or a complete HTML document."""
|
| 55 |
|
| 56 |
|
| 57 |
+
def build_scene_prompt(analysis: dict[str, Any]) -> str:
|
| 58 |
return (
|
| 59 |
+
"Build a technical cutaway / field manual A-Frame animation for this "
|
| 60 |
+
"mechanism analysis. Return only the <a-scene>...</a-scene> block. "
|
| 61 |
+
"Use 3 to 6 visible primitive parts maximum, selected from <a-box>, "
|
| 62 |
+
"<a-cylinder>, <a-sphere>, <a-torus>, <a-cone>, and <a-entity>. "
|
| 63 |
+
"Include <a-sky color=\"#0F1318\">, a camera, lights, short <a-text> "
|
| 64 |
+
"labels, warm amber annotations, and cool cyan moving parts. Use "
|
| 65 |
+
"A-Frame animation attributes for motion, for example "
|
| 66 |
+
"animation=\"property: rotation; to: 0 360 0; loop: true; dur: 2000; "
|
| 67 |
+
"easing: linear\". Do not include scripts, markdown, explanations, "
|
| 68 |
+
"or a complete HTML document.\n\n"
|
| 69 |
f"Mechanism JSON:\n{json.dumps(analysis, indent=2)}"
|
| 70 |
)
|
snap2sim/three_scene.py
DELETED
|
@@ -1,397 +0,0 @@
|
|
| 1 |
-
"""Deterministic Three.js scene generation from a validated mechanism payload."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
import html
|
| 6 |
-
import json
|
| 7 |
-
from typing import Any
|
| 8 |
-
|
| 9 |
-
from snap2sim.schema import validate_analysis
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
def build_threejs_html(analysis: dict[str, Any]) -> str:
|
| 13 |
-
"""Return a sandboxed iframe containing a self-contained scene document."""
|
| 14 |
-
valid_analysis = validate_analysis(analysis)
|
| 15 |
-
document = _build_scene_document(valid_analysis)
|
| 16 |
-
escaped = html.escape(document, quote=True)
|
| 17 |
-
title = html.escape(str(valid_analysis.get("component", "mechanism")), quote=True)
|
| 18 |
-
return (
|
| 19 |
-
f'<iframe class="snap-scene-frame" title="Animated cutaway of {title}" '
|
| 20 |
-
'sandbox="allow-scripts allow-same-origin" '
|
| 21 |
-
f'srcdoc="{escaped}"></iframe>'
|
| 22 |
-
)
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
def _build_scene_document(analysis: dict[str, Any]) -> str:
|
| 26 |
-
analysis_json = json.dumps(analysis)
|
| 27 |
-
component = html.escape(str(analysis.get("component", "mechanism")).upper())
|
| 28 |
-
return f"""<!doctype html>
|
| 29 |
-
<html lang="en">
|
| 30 |
-
<head>
|
| 31 |
-
<meta charset="utf-8">
|
| 32 |
-
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 33 |
-
<style>
|
| 34 |
-
:root {{
|
| 35 |
-
--bg: #14181F;
|
| 36 |
-
--panel: #1B222B;
|
| 37 |
-
--amber: #E8A33D;
|
| 38 |
-
--cyan: #5FD4D0;
|
| 39 |
-
--text: #D9D3C7;
|
| 40 |
-
--muted: #8F968F;
|
| 41 |
-
}}
|
| 42 |
-
* {{ box-sizing: border-box; }}
|
| 43 |
-
html, body {{
|
| 44 |
-
width: 100%;
|
| 45 |
-
height: 100%;
|
| 46 |
-
margin: 0;
|
| 47 |
-
overflow: hidden;
|
| 48 |
-
color: var(--text);
|
| 49 |
-
background:
|
| 50 |
-
linear-gradient(rgba(95, 212, 208, .07) 1px, transparent 1px),
|
| 51 |
-
linear-gradient(90deg, rgba(95, 212, 208, .07) 1px, transparent 1px),
|
| 52 |
-
var(--bg);
|
| 53 |
-
background-size: 28px 28px;
|
| 54 |
-
font-family: "IBM Plex Mono", "Consolas", monospace;
|
| 55 |
-
}}
|
| 56 |
-
#scene {{
|
| 57 |
-
position: fixed;
|
| 58 |
-
inset: 0;
|
| 59 |
-
}}
|
| 60 |
-
.hud {{
|
| 61 |
-
position: fixed;
|
| 62 |
-
left: 18px;
|
| 63 |
-
right: 18px;
|
| 64 |
-
top: 16px;
|
| 65 |
-
z-index: 3;
|
| 66 |
-
display: flex;
|
| 67 |
-
align-items: flex-start;
|
| 68 |
-
justify-content: space-between;
|
| 69 |
-
gap: 16px;
|
| 70 |
-
pointer-events: none;
|
| 71 |
-
}}
|
| 72 |
-
.title {{
|
| 73 |
-
max-width: min(680px, 70vw);
|
| 74 |
-
}}
|
| 75 |
-
.title b {{
|
| 76 |
-
display: block;
|
| 77 |
-
color: var(--text);
|
| 78 |
-
font-size: 14px;
|
| 79 |
-
letter-spacing: 0;
|
| 80 |
-
text-transform: uppercase;
|
| 81 |
-
}}
|
| 82 |
-
.title span {{
|
| 83 |
-
display: block;
|
| 84 |
-
margin-top: 6px;
|
| 85 |
-
color: var(--muted);
|
| 86 |
-
font-size: 12px;
|
| 87 |
-
line-height: 1.45;
|
| 88 |
-
}}
|
| 89 |
-
#toggle {{
|
| 90 |
-
pointer-events: auto;
|
| 91 |
-
min-width: 92px;
|
| 92 |
-
border: 1px solid rgba(232, 163, 61, .8);
|
| 93 |
-
border-radius: 2px;
|
| 94 |
-
padding: 8px 12px;
|
| 95 |
-
color: #16130E;
|
| 96 |
-
background: linear-gradient(180deg, #F0B65B, #C97923);
|
| 97 |
-
font: 700 12px "IBM Plex Mono", monospace;
|
| 98 |
-
text-transform: uppercase;
|
| 99 |
-
cursor: pointer;
|
| 100 |
-
}}
|
| 101 |
-
#labels {{
|
| 102 |
-
position: fixed;
|
| 103 |
-
inset: 0;
|
| 104 |
-
z-index: 2;
|
| 105 |
-
pointer-events: none;
|
| 106 |
-
}}
|
| 107 |
-
.tag {{
|
| 108 |
-
position: absolute;
|
| 109 |
-
max-width: min(190px, 42vw);
|
| 110 |
-
transform: translate(-50%, -50%);
|
| 111 |
-
border-left: 1px solid var(--cyan);
|
| 112 |
-
padding: 4px 0 4px 8px;
|
| 113 |
-
color: var(--cyan);
|
| 114 |
-
background: rgba(20, 24, 31, .76);
|
| 115 |
-
font-size: 11px;
|
| 116 |
-
line-height: 1.25;
|
| 117 |
-
text-transform: uppercase;
|
| 118 |
-
white-space: normal;
|
| 119 |
-
}}
|
| 120 |
-
.tag::before {{
|
| 121 |
-
content: "";
|
| 122 |
-
position: absolute;
|
| 123 |
-
left: -42px;
|
| 124 |
-
top: 50%;
|
| 125 |
-
width: 40px;
|
| 126 |
-
border-top: 1px solid rgba(95, 212, 208, .7);
|
| 127 |
-
}}
|
| 128 |
-
.scan {{
|
| 129 |
-
position: fixed;
|
| 130 |
-
inset: -30% 0 auto;
|
| 131 |
-
height: 28%;
|
| 132 |
-
z-index: 4;
|
| 133 |
-
background: linear-gradient(180deg, transparent, rgba(95, 212, 208, .18), transparent);
|
| 134 |
-
animation: scan 2.2s ease-in-out 1;
|
| 135 |
-
pointer-events: none;
|
| 136 |
-
}}
|
| 137 |
-
.failure {{
|
| 138 |
-
position: fixed;
|
| 139 |
-
left: 50%;
|
| 140 |
-
top: 50%;
|
| 141 |
-
width: min(520px, calc(100vw - 40px));
|
| 142 |
-
transform: translate(-50%, -50%);
|
| 143 |
-
border: 1px solid rgba(232, 163, 61, .62);
|
| 144 |
-
padding: 18px;
|
| 145 |
-
color: var(--amber);
|
| 146 |
-
background: rgba(20, 24, 31, .92);
|
| 147 |
-
font-size: 13px;
|
| 148 |
-
line-height: 1.55;
|
| 149 |
-
}}
|
| 150 |
-
.hidden {{ display: none; }}
|
| 151 |
-
@keyframes scan {{
|
| 152 |
-
0% {{ transform: translateY(-100%); opacity: 0; }}
|
| 153 |
-
20%, 75% {{ opacity: 1; }}
|
| 154 |
-
100% {{ transform: translateY(430%); opacity: 0; }}
|
| 155 |
-
}}
|
| 156 |
-
</style>
|
| 157 |
-
</head>
|
| 158 |
-
<body>
|
| 159 |
-
<div id="scene"></div>
|
| 160 |
-
<div id="labels"></div>
|
| 161 |
-
<div class="scan"></div>
|
| 162 |
-
<div class="hud">
|
| 163 |
-
<div class="title">
|
| 164 |
-
<b>{component}</b>
|
| 165 |
-
<span>Primitive cutaway generated from the validated mechanism schema.</span>
|
| 166 |
-
</div>
|
| 167 |
-
<button id="toggle" type="button">Pause</button>
|
| 168 |
-
</div>
|
| 169 |
-
<div id="failure" class="failure hidden">WebGL scene failed to initialize.</div>
|
| 170 |
-
<script type="module">
|
| 171 |
-
const analysis = {analysis_json};
|
| 172 |
-
const failure = document.getElementById("failure");
|
| 173 |
-
|
| 174 |
-
function fail(error) {{
|
| 175 |
-
failure.classList.remove("hidden");
|
| 176 |
-
failure.textContent = "3D renderer unavailable: " + (error && error.message ? error.message : error);
|
| 177 |
-
}}
|
| 178 |
-
|
| 179 |
-
try {{
|
| 180 |
-
const [threeModule, controlsModule] = await Promise.all([
|
| 181 |
-
import("https://esm.sh/three@0.166.1"),
|
| 182 |
-
import("https://esm.sh/three@0.166.1/examples/jsm/controls/OrbitControls.js?deps=three@0.166.1")
|
| 183 |
-
]);
|
| 184 |
-
initialize(threeModule, controlsModule.OrbitControls);
|
| 185 |
-
}} catch (error) {{
|
| 186 |
-
fail(error);
|
| 187 |
-
}}
|
| 188 |
-
|
| 189 |
-
function initialize(THREE, OrbitControls) {{
|
| 190 |
-
const root = document.getElementById("scene");
|
| 191 |
-
const labelsRoot = document.getElementById("labels");
|
| 192 |
-
const renderer = new THREE.WebGLRenderer({{ antialias: true, alpha: true, preserveDrawingBuffer: true }});
|
| 193 |
-
renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
|
| 194 |
-
renderer.setSize(window.innerWidth, window.innerHeight);
|
| 195 |
-
renderer.outputColorSpace = THREE.SRGBColorSpace;
|
| 196 |
-
root.appendChild(renderer.domElement);
|
| 197 |
-
|
| 198 |
-
const scene = new THREE.Scene();
|
| 199 |
-
scene.fog = new THREE.Fog(0x14181f, 9, 22);
|
| 200 |
-
|
| 201 |
-
const camera = new THREE.PerspectiveCamera(42, window.innerWidth / window.innerHeight, 0.1, 100);
|
| 202 |
-
camera.position.set(5.8, 4.6, 6.8);
|
| 203 |
-
|
| 204 |
-
const controls = new OrbitControls(camera, renderer.domElement);
|
| 205 |
-
controls.enableDamping = true;
|
| 206 |
-
controls.target.set(0, 0.15, 0);
|
| 207 |
-
|
| 208 |
-
scene.add(new THREE.HemisphereLight(0x5fd4d0, 0x14181f, 1.35));
|
| 209 |
-
const key = new THREE.DirectionalLight(0xffc36a, 2.15);
|
| 210 |
-
key.position.set(4, 7, 5);
|
| 211 |
-
scene.add(key);
|
| 212 |
-
|
| 213 |
-
const grid = new THREE.GridHelper(8, 32, 0x5fd4d0, 0x24505a);
|
| 214 |
-
grid.material.transparent = true;
|
| 215 |
-
grid.material.opacity = 0.42;
|
| 216 |
-
scene.add(grid);
|
| 217 |
-
|
| 218 |
-
const motionLineMaterial = new THREE.LineBasicMaterial({{ color: 0x5fd4d0, transparent: true, opacity: 0.56 }});
|
| 219 |
-
const parts = [];
|
| 220 |
-
|
| 221 |
-
for (const [index, part] of analysis.parts.entries()) {{
|
| 222 |
-
const mesh = buildMesh(THREE, part, index);
|
| 223 |
-
mesh.position.fromArray(part.geometry.position);
|
| 224 |
-
if (Array.isArray(part.geometry.rotation)) mesh.rotation.fromArray(part.geometry.rotation);
|
| 225 |
-
mesh.userData.basePosition = mesh.position.clone();
|
| 226 |
-
mesh.userData.baseRotation = mesh.rotation.clone();
|
| 227 |
-
mesh.userData.part = part;
|
| 228 |
-
mesh.scale.setScalar(0.001);
|
| 229 |
-
scene.add(mesh);
|
| 230 |
-
parts.push(mesh);
|
| 231 |
-
|
| 232 |
-
if (part.motion && part.motion.type !== "static") {{
|
| 233 |
-
const curve = motionCurve(THREE, part);
|
| 234 |
-
const line = new THREE.Line(
|
| 235 |
-
new THREE.BufferGeometry().setFromPoints(curve),
|
| 236 |
-
motionLineMaterial
|
| 237 |
-
);
|
| 238 |
-
scene.add(line);
|
| 239 |
-
}}
|
| 240 |
-
}}
|
| 241 |
-
|
| 242 |
-
let playing = true;
|
| 243 |
-
document.getElementById("toggle").addEventListener("click", (event) => {{
|
| 244 |
-
playing = !playing;
|
| 245 |
-
event.currentTarget.textContent = playing ? "Pause" : "Play";
|
| 246 |
-
}});
|
| 247 |
-
|
| 248 |
-
const clock = new THREE.Clock();
|
| 249 |
-
function animate() {{
|
| 250 |
-
requestAnimationFrame(animate);
|
| 251 |
-
const elapsed = clock.getElapsedTime();
|
| 252 |
-
const t = playing ? elapsed : 0;
|
| 253 |
-
for (const [index, mesh] of parts.entries()) {{
|
| 254 |
-
reveal(mesh, elapsed, index);
|
| 255 |
-
applyMotion(THREE, mesh, t);
|
| 256 |
-
}}
|
| 257 |
-
controls.update();
|
| 258 |
-
renderer.render(scene, camera);
|
| 259 |
-
updateLabels(THREE, camera, labelsRoot, parts);
|
| 260 |
-
}}
|
| 261 |
-
animate();
|
| 262 |
-
|
| 263 |
-
window.addEventListener("resize", () => {{
|
| 264 |
-
camera.aspect = window.innerWidth / window.innerHeight;
|
| 265 |
-
camera.updateProjectionMatrix();
|
| 266 |
-
renderer.setSize(window.innerWidth, window.innerHeight);
|
| 267 |
-
}});
|
| 268 |
-
}}
|
| 269 |
-
|
| 270 |
-
function buildMesh(THREE, part, index) {{
|
| 271 |
-
const g = part.geometry || {{}};
|
| 272 |
-
const size = Array.isArray(g.size) ? g.size : [1, 1, 1];
|
| 273 |
-
let geometry;
|
| 274 |
-
if (g.shape === "cylinder") {{
|
| 275 |
-
geometry = new THREE.CylinderGeometry(size[0] / 2, size[2] / 2, size[1], 48);
|
| 276 |
-
}} else if (g.shape === "sphere") {{
|
| 277 |
-
geometry = new THREE.SphereGeometry(Math.max(size[0], size[1], size[2]) / 2, 36, 18);
|
| 278 |
-
}} else if (g.shape === "rod") {{
|
| 279 |
-
geometry = new THREE.CylinderGeometry(size[0] / 2, size[0] / 2, size[2], 24);
|
| 280 |
-
geometry.rotateX(Math.PI / 2);
|
| 281 |
-
}} else if (g.shape === "gear") {{
|
| 282 |
-
geometry = gearGeometry(THREE, Math.max(size[0], size[2]) / 2, size[1], g.teeth || 18);
|
| 283 |
-
}} else {{
|
| 284 |
-
geometry = new THREE.BoxGeometry(size[0], size[1], size[2]);
|
| 285 |
-
}}
|
| 286 |
-
const color = colorFor(g.color, index);
|
| 287 |
-
const material = new THREE.MeshStandardMaterial({{
|
| 288 |
-
color,
|
| 289 |
-
metalness: 0.55,
|
| 290 |
-
roughness: 0.36,
|
| 291 |
-
emissive: color,
|
| 292 |
-
emissiveIntensity: 0.05
|
| 293 |
-
}});
|
| 294 |
-
return new THREE.Mesh(geometry, material);
|
| 295 |
-
}}
|
| 296 |
-
|
| 297 |
-
function gearGeometry(THREE, radius, depth, teeth) {{
|
| 298 |
-
const shape = new THREE.Shape();
|
| 299 |
-
const steps = teeth * 2;
|
| 300 |
-
for (let i = 0; i <= steps; i++) {{
|
| 301 |
-
const angle = (i / steps) * Math.PI * 2;
|
| 302 |
-
const r = radius * (i % 2 === 0 ? 1.0 : 0.84);
|
| 303 |
-
const x = Math.cos(angle) * r;
|
| 304 |
-
const y = Math.sin(angle) * r;
|
| 305 |
-
if (i === 0) shape.moveTo(x, y);
|
| 306 |
-
else shape.lineTo(x, y);
|
| 307 |
-
}}
|
| 308 |
-
const hole = new THREE.Path();
|
| 309 |
-
hole.absarc(0, 0, radius * 0.28, 0, Math.PI * 2, true);
|
| 310 |
-
shape.holes.push(hole);
|
| 311 |
-
const geometry = new THREE.ExtrudeGeometry(shape, {{ depth, bevelEnabled: false }});
|
| 312 |
-
geometry.center();
|
| 313 |
-
geometry.rotateX(Math.PI / 2);
|
| 314 |
-
return geometry;
|
| 315 |
-
}}
|
| 316 |
-
|
| 317 |
-
function colorFor(name, index) {{
|
| 318 |
-
const key = String(name || "").toLowerCase();
|
| 319 |
-
if (key.includes("amber") || key.includes("orange")) return 0xe8a33d;
|
| 320 |
-
if (key.includes("cyan")) return 0x5fd4d0;
|
| 321 |
-
if (key.includes("steel")) return 0x9aa4a6;
|
| 322 |
-
const palette = [0x5fd4d0, 0xe8a33d, 0xd9d3c7, 0x80b8ff, 0xf07f5a];
|
| 323 |
-
return palette[index % palette.length];
|
| 324 |
-
}}
|
| 325 |
-
|
| 326 |
-
function motionCurve(THREE, part) {{
|
| 327 |
-
const p = part.geometry.position || [0, 0, 0];
|
| 328 |
-
const points = [];
|
| 329 |
-
for (let i = 0; i < 48; i++) {{
|
| 330 |
-
const a = (i / 47) * Math.PI * 2;
|
| 331 |
-
points.push(new THREE.Vector3(p[0] + Math.cos(a) * 0.52, p[1] + 0.04, p[2] + Math.sin(a) * 0.52));
|
| 332 |
-
}}
|
| 333 |
-
return points;
|
| 334 |
-
}}
|
| 335 |
-
|
| 336 |
-
function reveal(mesh, elapsed, index) {{
|
| 337 |
-
const local = Math.max(0, Math.min(1, (elapsed - index * 0.12) / 0.75));
|
| 338 |
-
const eased = 1 - Math.pow(1 - local, 3);
|
| 339 |
-
mesh.scale.setScalar(eased);
|
| 340 |
-
}}
|
| 341 |
-
|
| 342 |
-
function applyMotion(THREE, mesh, elapsed) {{
|
| 343 |
-
const motion = mesh.userData.part.motion || {{ type: "static" }};
|
| 344 |
-
const speed = Number(motion.speed ?? 1);
|
| 345 |
-
const phase = Number(motion.phase ?? 0);
|
| 346 |
-
const axis = new THREE.Vector3(...(motion.axis || [0, 1, 0])).normalize();
|
| 347 |
-
mesh.position.copy(mesh.userData.basePosition);
|
| 348 |
-
mesh.rotation.copy(mesh.userData.baseRotation);
|
| 349 |
-
if (motion.type === "rotate") {{
|
| 350 |
-
mesh.rotateOnAxis(axis, elapsed * speed + phase);
|
| 351 |
-
}} else if (motion.type === "oscillate") {{
|
| 352 |
-
mesh.rotateOnAxis(axis, Math.sin(elapsed * speed + phase) * Number(motion.amplitude ?? 0.25));
|
| 353 |
-
}} else if (motion.type === "translate") {{
|
| 354 |
-
const range = motion.range || [-0.35, 0.35];
|
| 355 |
-
const offset = range[0] + (range[1] - range[0]) * ((Math.sin(elapsed * speed + phase) + 1) / 2);
|
| 356 |
-
mesh.position.add(axis.multiplyScalar(offset));
|
| 357 |
-
}}
|
| 358 |
-
}}
|
| 359 |
-
|
| 360 |
-
function updateLabels(THREE, camera, labelsRoot, parts) {{
|
| 361 |
-
labelsRoot.replaceChildren();
|
| 362 |
-
const width = window.innerWidth;
|
| 363 |
-
const height = window.innerHeight;
|
| 364 |
-
const entries = [];
|
| 365 |
-
for (const mesh of parts) {{
|
| 366 |
-
const pos = mesh.position.clone().project(camera);
|
| 367 |
-
if (pos.z < -1 || pos.z > 1) continue;
|
| 368 |
-
entries.push({{
|
| 369 |
-
x: clamp((pos.x * 0.5 + 0.5) * width, 76, Math.max(76, width - 76)),
|
| 370 |
-
y: clamp((-pos.y * 0.5 + 0.5) * height, 72, Math.max(72, height - 32)),
|
| 371 |
-
text: mesh.userData.part.name || mesh.userData.part.id || "part"
|
| 372 |
-
}});
|
| 373 |
-
}}
|
| 374 |
-
entries.sort((a, b) => a.y - b.y);
|
| 375 |
-
for (let i = 1; i < entries.length; i++) {{
|
| 376 |
-
entries[i].y = Math.max(entries[i].y, entries[i - 1].y + 38);
|
| 377 |
-
}}
|
| 378 |
-
const overflow = entries.length ? entries[entries.length - 1].y - (height - 32) : 0;
|
| 379 |
-
if (overflow > 0) {{
|
| 380 |
-
for (const entry of entries) entry.y -= overflow;
|
| 381 |
-
}}
|
| 382 |
-
for (const entry of entries) {{
|
| 383 |
-
const label = document.createElement("div");
|
| 384 |
-
label.className = "tag";
|
| 385 |
-
label.style.left = entry.x + "px";
|
| 386 |
-
label.style.top = entry.y + "px";
|
| 387 |
-
label.textContent = entry.text;
|
| 388 |
-
labelsRoot.appendChild(label);
|
| 389 |
-
}}
|
| 390 |
-
}}
|
| 391 |
-
|
| 392 |
-
function clamp(value, min, max) {{
|
| 393 |
-
return Math.min(max, Math.max(min, value));
|
| 394 |
-
}}
|
| 395 |
-
</script>
|
| 396 |
-
</body>
|
| 397 |
-
</html>"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
snap2sim/ui.py
DELETED
|
@@ -1,225 +0,0 @@
|
|
| 1 |
-
"""Gradio interface for Snap2Sim."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
import json
|
| 6 |
-
from typing import Any
|
| 7 |
-
|
| 8 |
-
import gradio as gr
|
| 9 |
-
from PIL import Image
|
| 10 |
-
|
| 11 |
-
from snap2sim.backend import InferenceClient, Settings
|
| 12 |
-
from snap2sim.fallback_scene import build_fallback_html
|
| 13 |
-
from snap2sim.schema import EXAMPLE_ANALYSIS
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
CSS = """
|
| 17 |
-
:root {
|
| 18 |
-
--snap-bg: #14181F;
|
| 19 |
-
--snap-panel: #1B222B;
|
| 20 |
-
--snap-line: rgba(95, 212, 208, 0.22);
|
| 21 |
-
--snap-amber: #E8A33D;
|
| 22 |
-
--snap-cyan: #5FD4D0;
|
| 23 |
-
--snap-text: #D9D3C7;
|
| 24 |
-
--snap-muted: #8F968F;
|
| 25 |
-
}
|
| 26 |
-
|
| 27 |
-
.gradio-container {
|
| 28 |
-
min-height: 100vh;
|
| 29 |
-
color: var(--snap-text);
|
| 30 |
-
background:
|
| 31 |
-
linear-gradient(rgba(95, 212, 208, .055) 1px, transparent 1px),
|
| 32 |
-
linear-gradient(90deg, rgba(95, 212, 208, .055) 1px, transparent 1px),
|
| 33 |
-
radial-gradient(circle at 85% 15%, rgba(232, 163, 61, .12), transparent 30%),
|
| 34 |
-
var(--snap-bg) !important;
|
| 35 |
-
background-size: 28px 28px, 28px 28px, auto, auto !important;
|
| 36 |
-
font-family: 'IBM Plex Mono', monospace !important;
|
| 37 |
-
}
|
| 38 |
-
|
| 39 |
-
.snap-shell {
|
| 40 |
-
max-width: 1540px;
|
| 41 |
-
margin: 0 auto;
|
| 42 |
-
}
|
| 43 |
-
|
| 44 |
-
.snap-title h1 {
|
| 45 |
-
margin: 0 0 4px;
|
| 46 |
-
color: var(--snap-text);
|
| 47 |
-
font-family: 'Archivo Narrow', sans-serif;
|
| 48 |
-
font-size: clamp(38px, 5vw, 76px);
|
| 49 |
-
line-height: .9;
|
| 50 |
-
letter-spacing: 0;
|
| 51 |
-
text-transform: uppercase;
|
| 52 |
-
}
|
| 53 |
-
|
| 54 |
-
.snap-title p {
|
| 55 |
-
max-width: 860px;
|
| 56 |
-
color: var(--snap-muted);
|
| 57 |
-
font-size: 14px;
|
| 58 |
-
line-height: 1.6;
|
| 59 |
-
}
|
| 60 |
-
|
| 61 |
-
.snap-frame {
|
| 62 |
-
border: 1px solid var(--snap-line);
|
| 63 |
-
background: rgba(20, 24, 31, .8);
|
| 64 |
-
box-shadow: inset 0 0 0 1px rgba(232, 163, 61, .06);
|
| 65 |
-
}
|
| 66 |
-
|
| 67 |
-
.snap-frame label,
|
| 68 |
-
.snap-frame .label-wrap span {
|
| 69 |
-
color: var(--snap-cyan) !important;
|
| 70 |
-
font-family: 'IBM Plex Mono', monospace !important;
|
| 71 |
-
text-transform: uppercase;
|
| 72 |
-
}
|
| 73 |
-
|
| 74 |
-
.snap-frame textarea,
|
| 75 |
-
.snap-frame input {
|
| 76 |
-
background: #10141A !important;
|
| 77 |
-
color: var(--snap-text) !important;
|
| 78 |
-
border-color: var(--snap-line) !important;
|
| 79 |
-
font-family: 'IBM Plex Mono', monospace !important;
|
| 80 |
-
}
|
| 81 |
-
|
| 82 |
-
.snap-run {
|
| 83 |
-
border: 1px solid rgba(232, 163, 61, .7) !important;
|
| 84 |
-
background: linear-gradient(180deg, #F0B65B, #C97923) !important;
|
| 85 |
-
color: #16130E !important;
|
| 86 |
-
font-family: 'Archivo Narrow', sans-serif !important;
|
| 87 |
-
font-size: 18px !important;
|
| 88 |
-
font-weight: 700 !important;
|
| 89 |
-
text-transform: uppercase;
|
| 90 |
-
border-radius: 2px !important;
|
| 91 |
-
}
|
| 92 |
-
|
| 93 |
-
.snap-run:hover {
|
| 94 |
-
filter: brightness(1.08);
|
| 95 |
-
}
|
| 96 |
-
|
| 97 |
-
.snap-status {
|
| 98 |
-
min-height: 42px;
|
| 99 |
-
color: var(--snap-amber);
|
| 100 |
-
font-size: 13px;
|
| 101 |
-
text-transform: uppercase;
|
| 102 |
-
}
|
| 103 |
-
|
| 104 |
-
.snap-html iframe,
|
| 105 |
-
.snap-html > div {
|
| 106 |
-
min-height: 560px;
|
| 107 |
-
}
|
| 108 |
-
|
| 109 |
-
.snap-scene-frame {
|
| 110 |
-
display: block;
|
| 111 |
-
width: 100%;
|
| 112 |
-
min-height: 560px;
|
| 113 |
-
border: 0;
|
| 114 |
-
background: #14181F;
|
| 115 |
-
}
|
| 116 |
-
|
| 117 |
-
.snap-json pre {
|
| 118 |
-
color: var(--snap-text) !important;
|
| 119 |
-
background: #10141A !important;
|
| 120 |
-
}
|
| 121 |
-
|
| 122 |
-
.snap-explain {
|
| 123 |
-
color: var(--snap-text);
|
| 124 |
-
line-height: 1.65;
|
| 125 |
-
}
|
| 126 |
-
"""
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
def format_explanation(analysis: dict[str, Any]) -> str:
|
| 130 |
-
parts = analysis.get("parts", [])
|
| 131 |
-
part_lines = "\n".join(
|
| 132 |
-
f"- **{part.get('name', 'part')}**: {part.get('role', 'role unknown')}"
|
| 133 |
-
for part in parts
|
| 134 |
-
)
|
| 135 |
-
sequence = "\n".join(f"{index + 1}. {step}" for index, step in enumerate(analysis.get("motion_sequence", [])))
|
| 136 |
-
return (
|
| 137 |
-
f"### {analysis.get('component', 'Component')}\n\n"
|
| 138 |
-
f"{analysis.get('summary', '')}\n\n"
|
| 139 |
-
f"**Trigger:** {analysis.get('trigger', 'unknown')}\n\n"
|
| 140 |
-
f"**Motion sequence**\n{sequence}\n\n"
|
| 141 |
-
f"**Parts**\n{part_lines}"
|
| 142 |
-
)
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
def initial_status() -> str:
|
| 146 |
-
settings = Settings()
|
| 147 |
-
if settings.backend == "modal":
|
| 148 |
-
configured = bool(settings.analyze_url and settings.generate_url)
|
| 149 |
-
return "STANDBY: MODAL BACKEND CONFIGURED" if configured else "STANDBY: MODAL BACKEND MISSING URLS"
|
| 150 |
-
return "STANDBY: LOCAL PLACEHOLDER BACKEND"
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
def run_pipeline(image: Image.Image | None) -> tuple[str, str, str, str]:
|
| 154 |
-
client = InferenceClient(Settings())
|
| 155 |
-
try:
|
| 156 |
-
analysis = client.analyze_image(image)
|
| 157 |
-
html = client.generate_threejs(analysis)
|
| 158 |
-
status = "CUTAWAY READY"
|
| 159 |
-
except Exception as exc:
|
| 160 |
-
analysis = dict(EXAMPLE_ANALYSIS)
|
| 161 |
-
html = build_fallback_html(str(analysis.get("component", "mechanism")))
|
| 162 |
-
status = f"FALLBACK RENDER ACTIVE: {exc}"
|
| 163 |
-
|
| 164 |
-
return (
|
| 165 |
-
html,
|
| 166 |
-
format_explanation(analysis),
|
| 167 |
-
json.dumps(analysis, indent=2),
|
| 168 |
-
status,
|
| 169 |
-
)
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
def build_app() -> gr.Blocks:
|
| 173 |
-
with gr.Blocks(css=CSS, title="Inside the Machine") as demo:
|
| 174 |
-
with gr.Column(elem_classes=["snap-shell"]):
|
| 175 |
-
gr.HTML(
|
| 176 |
-
"""
|
| 177 |
-
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 178 |
-
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 179 |
-
<link href="https://fonts.googleapis.com/css2?family=Archivo+Narrow:wght@500;700&family=IBM+Plex+Mono:wght@400;500;700&display=swap" rel="stylesheet">
|
| 180 |
-
<section class="snap-title">
|
| 181 |
-
<h1>Inside the Machine</h1>
|
| 182 |
-
<p>Upload a component photo. Snap2Sim drafts a mechanical teardown,
|
| 183 |
-
then renders an annotated cutaway animation for repair cafes,
|
| 184 |
-
shop classrooms, and curious builders.</p>
|
| 185 |
-
</section>
|
| 186 |
-
"""
|
| 187 |
-
)
|
| 188 |
-
|
| 189 |
-
with gr.Row(equal_height=False):
|
| 190 |
-
with gr.Column(scale=5, elem_classes=["snap-frame"]):
|
| 191 |
-
scene = gr.HTML(
|
| 192 |
-
build_fallback_html("waiting for upload"),
|
| 193 |
-
label="Cutaway viewport",
|
| 194 |
-
elem_classes=["snap-html"],
|
| 195 |
-
)
|
| 196 |
-
with gr.Column(scale=3, elem_classes=["snap-frame"]):
|
| 197 |
-
image = gr.Image(type="pil", label="Component photo")
|
| 198 |
-
run = gr.Button("Analyze Assembly", elem_classes=["snap-run"])
|
| 199 |
-
status = gr.Textbox(
|
| 200 |
-
value=initial_status(),
|
| 201 |
-
label="Workshop status",
|
| 202 |
-
interactive=False,
|
| 203 |
-
elem_classes=["snap-status"],
|
| 204 |
-
)
|
| 205 |
-
explanation = gr.Markdown(
|
| 206 |
-
"Upload a photo and run the analyzer.",
|
| 207 |
-
elem_classes=["snap-explain"],
|
| 208 |
-
)
|
| 209 |
-
|
| 210 |
-
with gr.Accordion("Structured mechanism JSON", open=False):
|
| 211 |
-
raw_json = gr.Code(
|
| 212 |
-
value="{}",
|
| 213 |
-
language="json",
|
| 214 |
-
label="Vision-to-scene contract",
|
| 215 |
-
elem_classes=["snap-json"],
|
| 216 |
-
)
|
| 217 |
-
|
| 218 |
-
run.click(
|
| 219 |
-
fn=run_pipeline,
|
| 220 |
-
inputs=[image],
|
| 221 |
-
outputs=[scene, explanation, raw_json, status],
|
| 222 |
-
show_progress="full",
|
| 223 |
-
)
|
| 224 |
-
|
| 225 |
-
return demo
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|