AGENTS.md

AGENTS.md

Project Overview

Snap2Sim / Inside the Machine is a Build Small Hackathon Backyard AI project for curious tinkerers and makers. The app is intended to accept a photo of a hardware component, infer its internal mechanism, and render an annotated technical cutaway animation.

Commit Instructions

• When Codex creates a git commit in this repository, include this trailer in the commit message: Co-authored-by: OpenAI Codex <codex@openai.com>
• Do not rewrite already-pushed public history only to add a co-author trailer unless the user explicitly asks for that history rewrite.

Runtime Notes

• Preferred model path: unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF through llama.cpp.
• Verification note from June 13, 2026: the Unsloth GGUF repo lists mmproj files and llama.cpp usage instructions. Modal GPU smoke testing confirms the selected UD-Q4_K_M GGUF and mmproj-F16.gguf work with llama-mtmd-cli image input.
• Fallback model path: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 through Transformers/custom code for the vision step if GGUF endpoint quality is not reliable enough for the demo.
• Local code serves a trusted index.html through gradio.Server and exposes /analyze_image plus /generate_scene. The final public Space is build-small-hackathon/Snap2Sim at https://build-small-hackathon-snap2sim.hf.space; any new code changes still need the normal GitHub-to-HF sync and deployment verification.
• Scene rendering is deterministic browser-side Three.js from the validated analysis JSON. Do not reintroduce model-authored HTML scene injection.
• Endpoint naming is generate_scene everywhere in active code.
• Bonus claims after Modal deployment: Llama Champion confirmed, NVIDIA Nemotron Quest confirmed, Off-Brand confirmed, Modal Award confirmed. Do not claim Off the Grid unless the final deployment moves inference to ZeroGPU or another non-cloud-API path that satisfies that bonus.

Project Structure

• prompts/ - archived build prompts and Codex task specs.
• docs/ - implementation notes, review passes, and representative fixtures.
• README.md - quickstart, runtime decision, and repo map.
• SECURITY.md - public deployment data handling rules for humans and agents.
• .env.example - public Hugging Face Space / local demo variables.
• .hfignore - excludes local credentials, caches, GGUF files, and artifacts from Hugging Face uploads.
• .github/workflows/sync_to_hf.yml - GitHub Actions workflow that syncs GitHub main to an existing Hugging Face Space.
• requirements.txt - Python dependencies for the Gradio Space and Modal scaffold.
• app.py - Hugging Face Space entry point using gradio.Server; serves index.html at / and exposes /analyze_image plus /generate_scene.
• index.html - self-contained HTML/CSS/JS shell loaded by gradio.Server; includes Three.js, upload UI, source-photo preview, pipeline orchestration, and deterministic Three.js rendering.
• modal_app.py - Modal app scaffold with runtime asset caching, a llama.cpp GPU smoke-test function, a runtime_probe diagnostic endpoint, and placeholder plus experimental llama.cpp analyze_image / generate_scene web endpoints.
• scripts/verify_runtime_assets.py - Hugging Face metadata preflight for the selected GGUF quant and mmproj file.
• snap2sim/__init__.py - package marker.
• snap2sim/backend.py - backend config, local placeholder inference, Modal HTTP client, and image base64 encoding.
• snap2sim/prompts.py - prompt template for the vision analysis step.
• snap2sim/schema.py - structured JSON schema plus a sample mechanism payload.

What Has Been Done

• Read prompts/PROMPT.md and implemented the requested initial scaffold.
• Added a modular Gradio app with a two-pane technical cutaway / field manual visual direction.
• Added loading/status language and a styled fallback visualization path so the viewport is not blank when Three.js generation is unavailable.
• Added a backend abstraction controlled by INFERENCE_BACKEND.
• Added placeholder Modal endpoints for the two required inference steps.
• Added a Modal Volume-based runtime asset cache helper and runtime probe.
• Added smoke_test_llamacpp_image, which builds llama.cpp with CUDA on Modal and runs one image prompt through llama-mtmd-cli using the selected GGUF and mmproj-F16.gguf.
• Confirmed smoke_test_llamacpp_image returns "ok": true.
• Confirmed run_analysis_endpoint_check returns a validated mechanism payload from the real llama.cpp analysis task.
• Changed runtime asset resolution so endpoint calls use cached Modal Volume files directly; repeated Fetching 3 files logs were cached Hugging Face metadata checks, not full model downloads.
• Added run_runtime_preflight local entrypoint to cache assets and run the GPU smoke test in one Modal command.
• Added a local Hugging Face metadata preflight script and confirmed it passes for UD-Q4_K_M plus mmproj-F16.gguf.
• Modal is authenticated under the bigstonks1 workspace; use $env:PYTHONIOENCODING='utf-8'; python -m modal ... on Windows to avoid CLI Unicode/charmap errors.
• Defined the vision-to-scene JSON schema for parts, geometry hints, and motion.
• Added JSON-schema validation before scene generation.
• Added deterministic Three.js scene generation for local and Modal placeholder mode, so validated payloads render as an animated 3D cutaway before model code-generation is wired in.
• Added model response parsing helpers for JSON analysis payloads and complete HTML documents.
• Added a local coercion fallback for verbose/partial model JSON so the analysis endpoint does not launch a second slow repair generation.
• Added experimental analyze_image_llamacpp and scene-generation Modal GPU endpoints. They are not wired into the Gradio app by default; point MODAL_ANALYZE_URL / MODAL_GENERATE_URL at them only after the smoke test passes.
• Fixed Modal packaging for local snap2sim imports by adding add_local_python_source("snap2sim") as the final image step for both Modal images.
• Added check_remote_imports; python -m modal run modal_app.py::check_remote_imports now creates the PythonPackage:snap2sim mount and hydrates the web functions without ModuleNotFoundError.
• Added prompt templates for image analysis and Three.js scene generation.
• Deployed modal_app.py to Modal at https://modal.com/apps/bigstonks1/main/deployed/snap2sim-inside-the-machine.
• Production endpoint checks passed for runtime_probe, analyze_image_llamacpp, and deterministic scene generation before the current endpoint rename.
• Modal web endpoints are secured with Authorization: Bearer SNAP2SIM_API_TOKEN; the token lives in Modal secret snap2sim-api-auth and Hugging Face Space secret SNAP2SIM_API_TOKEN.
• Verified unauthenticated Modal requests return 401 Unauthorized.
• Added SECURITY.md, .env.example, and .hfignore so public deployment config is explicit while credentials and runtime caches stay out of uploads.
• Added Hugging Face Space metadata to README.md.
• Hugging Face auth is now valid for user jasondo111.
• Uploaded the Space to jasondo111/Snap2Sim; the Space is RUNNING on cpu-basic at https://jasondo111-snap2sim.hf.space.
• The Space remains private as of the latest deployment.
• User decision on June 13, 2026: keep the Hugging Face Space private for now. Do not make it public unless the user explicitly asks.
• Removed accidentally uploaded Playwright artifacts/screenshots from the Space and expanded .gitignore so future hf upload runs skip them.
• Verified the private Space with an authenticated gradio_client call using handle_file; the /run_pipeline API returned CUTAWAY READY.
• Re-verified /run_pipeline after endpoint auth was added; it still returns CUTAWAY READY.
• Added .github/workflows/sync_to_hf.yml for one-way GitHub to Hugging Face Space sync on pushes to main. The workflow targets jasondo111/Snap2Sim and requires GitHub secret HF_TOKEN.
• User reported the GitHub Actions HF_TOKEN secret has been added.
• PR #1 was merged to main. The first GitHub-to-HF sync run reached the final push step but failed with Hugging Face HTTP 429; the workflow was updated to retry that final push with bounded backoff.
• The next GitHub Actions sync run succeeded: https://github.com/Bigstonks1/Snap2Sim/actions/runs/27487136019.
• Hugging Face Space jasondo111/Snap2Sim reports SHA d70e374bea16acaa41c322942815d3cb213745f4, matching GitHub main.
• prompts/PROMPT.md was updated with a new critical path: gradio.Server, index.html, A-Frame model scene generation, generate_scene naming, and a deterministic Three.js fallback in browser JS.
• Documented GitHub as the source of truth: https://github.com/Bigstonks1/Snap2Sim. Do not edit files directly on the Hugging Face Space; they will be overwritten by the sync workflow.
• Added README acknowledgement: built by Jason Do with implementation assistance from OpenAI Codex.
• Updated README.md with local run instructions, runtime preflight, Modal deployment path, and the current runtime decision.
• Rewrote app.py as a gradio.Server app with /, /analyze_image, and /generate_scene routes.
• Added index.html with upload orchestration, analysis readout, A-Frame scene injection, 3-second blank-scene fallback, play/pause, and deterministic browser-side Three.js fallback.
• Renamed active backend and Modal scene generation paths to generate_scene.
• Updated scene-generation prompts and parsing for <a-scene>...</a-scene> output.
• Deleted the legacy Gradio Blocks UI, Python iframe Three.js generator, and SVG fallback modules.
• Local verification passed with FastAPI TestClient: / returned the HTML shell, /analyze_image returned the placeholder analysis, and /generate_scene returned an <a-scene> block.
• Deployed the renamed Modal app on June 14, 2026. Modal now exposes generate_scene and generate_scene_llamacpp web endpoints.
• Updated Hugging Face Space variable MODAL_GENERATE_URL to the deployed generate_scene endpoint. MODAL_ANALYZE_URL remains pointed at the verified analyze_image_llamacpp endpoint.
• PR #2 merged to main; GitHub Actions sync run https://github.com/Bigstonks1/Snap2Sim/actions/runs/27492198915 completed successfully.
• Hugging Face Space jasondo111/Snap2Sim reported SHA 9175771c1d3a2e39d7191b24e5729473fdaae0ef, matching GitHub main, and remained private/running on cpu-basic.
• Verified the private Space root with authenticated HTTP on June 14, 2026: status 200, trusted index.html shell text present.
• Verified the private Space endpoint chain with an authenticated image request on June 14, 2026: /analyze_image returned a validated mechanism payload through the secured Modal bearer-token flow, and /generate_scene returned an A-Frame <a-scene> block.
• Implemented the docs/reviews/security-hardening.md security pass on June 14, 2026: Space-layer rate limiting for /analyze_image and /generate_scene, upload-size caps, PIL decompression-bomb handling, clean image decode errors, unique Modal temp image files with cleanup, and no model-authored scene HTML.
• Consolidated rendering on deterministic browser-side Three.js from validated analysis JSON. Removed the A-Frame runtime import, unused Gradio client import, A-Frame scene builder, A-Frame scene prompt, and raw scene parser.
• Updated the UI with source-photo preview, screen-reader live status, lighter muted text, client-side image validation, keyboard-accessible drop zone, a retry affordance, and a disabled initial Play state.
• PR #3 merged to main; GitHub Actions sync run https://github.com/Bigstonks1/Snap2Sim/actions/runs/27495583577 completed successfully.
• Hugging Face Space jasondo111/Snap2Sim reported SHA 62cfa90093cc5d7de8d12dec831210d142c08a9d, matching GitHub main, and the private root served the trusted Three.js shell.
• Verified the private Space endpoint chain after the security pass: /analyze_image returned a validated mechanism payload through the secured Modal bearer-token flow, and /generate_scene returned { "renderer": "three", "analysis": ... } with no HTML field.
• Implemented the docs/reviews/interaction-and-fallback-review.md local code pass on June 14, 2026: fixed the label-overlay pointer-events bug that blocked OrbitControls, added bounded camera controls plus Reset view and orbit/zoom hint, changed the source-photo preview to a stable contained image box, added optional normalized per-part annotation fields, added render_mode scene descriptors, and added a browser-side annotated-photo fallback plus an honest unavailable state.
• Updated model prompting/parsing for longer reasoning: prompts may allow reasoning before the final JSON, parser strips <think> blocks and prefers the last complete JSON object, and llama.cpp analysis now uses larger token/context/timeout budgets (4096 output tokens, 8192 context, 300s timeout at the analysis call site).
• Local verification after the docs/reviews/interaction-and-fallback-review.md pass: schema/parser checks passed, FastAPI TestClient confirmed /, /analyze_image, and /generate_scene, and browser verification confirmed left-drag orbit, right-drag pan, wheel zoom, mobile canvas event targeting, contained source preview, annotated-photo fallback, and unavailable retry state.
• Modal reasoning-budget follow-up on June 14, 2026: an initial run_analysis_endpoint_check returned a generic fallback, so a local-only run_analysis_raw_check diagnostic was added. The raw diagnostic showed the model used schema-adjacent fields (radius, height, string axes). The prompt now explicitly requires size vectors and numeric axis vectors, and coercion preserves common radius/height/length/axis-string mistakes instead of accepting nested JSON fragments as generic fallbacks. After the fix, run_analysis_raw_check parsed strict JSON successfully in 35.12s, and run_analysis_endpoint_check returned a parsed per-image targeting reticle payload at 0.7 confidence.
• User decision after the docs/reviews/interaction-and-fallback-review.md pass: use the real Modal analyze_image_llamacpp analysis path for the demo. INFERENCE_BACKEND now defaults to modal; INFERENCE_BACKEND=local is an explicit UI-development sample mode only and should not be used for public submission.
• Deployed the updated Modal app after the real-analysis decision on June 14,
  2026. The stable deployment exposes analyze_image_llamacpp with the latest prompt/parsing/coercion fixes. A lightweight deployed runtime_probe request without credentials returned 401 Unauthorized. Authenticated deployed web verification could not be run from the local shell because SNAP2SIM_API_TOKEN was not set locally; use the Hugging Face Space secret or a local secret store for the normal authenticated check.
• Committed and pushed 51992f4 (Use real Modal analysis path) to GitHub main. GitHub Actions sync run 27509085142 completed successfully, and the private Hugging Face Space reported SHA 51992f4d9aff7f298fc377791228999e98054579.
• Authenticated private Space verification after the real-analysis sync: root returned 200, a blank tan synthetic image returned the honest unavailable photo-render state, and the target-like synthetic image returned a real Modal analysis for reticle assembly at 0.7 confidence with 4 parts; /generate_scene returned render_mode: three and renderer: three.
• Implemented the optional visual-polish pass after private deployment verification: component watermark populated from the analysis component name, subtle static noise overlay, stronger viewport vignette, and refined scan-line reveal. Local browser verification confirmed the scene still renders, orbit drag still works, the canvas remains the pointer target, the source preview remains contained, and the mobile layout has no horizontal overflow.
• Implemented the docs/features/confidence-threshold.md confidence-threshold control on June 14, 2026: added a toolbar slider defaulting to 0.5, made the browser recompute render mode from cached analysis plus the current slider value, and kept slider changes local with no extra /analyze_image or /generate_scene calls. Added DEFAULT_CONFIDENCE_THRESHOLD = 0.5 for the server's advisory fallback. Local verification passed for schema/parser checks, FastAPI TestClient, slider downgrade/promote behavior, annotated-photo fallback, keyboard slider operation, mobile no-overflow layout, and canvas pointer targeting.
• Committed and pushed a0540e9 (Add confidence threshold control) to GitHub main. GitHub Actions sync run 27515143658 completed successfully, and the private Hugging Face Space reported SHA a0540e92f19174f8b828374b201c1da94ef125a4.
• Authenticated private Space verification after the confidence-threshold sync: root returned 200 with the slider shell present, a synthetic target image returned real Modal analysis for Target Reticle at 0.7 confidence with 3 parts, and /generate_scene returned renderer: three, render_mode: three, and no HTML field.
• Implemented the docs/features/confidence-threshold.md confidence-threshold re-spec on June 14, 2026: the slider is enabled before the first upload, sends confidence_threshold only with /generate_scene, no longer re-renders cached analysis on slider movement, and the server clamps/coerces the threshold before selecting render_mode. The browser now trusts the server's render_mode and only downgrades when geometry or annotation data is missing.
• Implemented the docs/features/annotated-photo-fallback.md fallback pass: fixed portrait and landscape image containment by giving the annotation frame a definite viewport-bounded box, added numbered marker dots, leader lines, and optional annotation bounding boxes, and kept all annotation text rendered through textContent.
• Implemented the docs/features/renderer-vocabulary.md renderer/prompt vocabulary pass: added deterministic cone, capsule, torus, and spring shapes; added screw, orbit, and pulse motions; raised the usable part cap to 6 across prompt, schema, coercion, and browser renderability; and updated the sample analysis to exercise all four new primitives.
• Local verification after the docs/features/annotated-photo-fallback.md / docs/features/renderer-vocabulary.md pass: schema and coercion checks passed, FastAPI TestClient confirmed /, /analyze_image, and /generate_scene, browser verification confirmed contained portrait and landscape annotated-photo fallbacks with labels in bounds and no mobile horizontal overflow, and the deterministic Three.js path reached CUTAWAY READY with a nonblank six-part scene.
• Committed and pushed 682623a (Expand renderer vocabulary and annotation fallback) to GitHub main. GitHub Actions sync run 27518719440 completed successfully, and the private Hugging Face Space reported SHA 682623a979e351c3fb6d5b5b9ade907c7a979aea.
• Deployed the updated Modal app after the docs/features/annotated-photo-fallback.md / docs/features/renderer-vocabulary.md pass so analyze_image_llamacpp uses the richer prompt plus expanded schema and coercion. Authenticated private Space verification passed: root returned 200 with the updated vocabulary shell, a synthetic target image returned a real Modal analysis for piston mechanism at 0.85 confidence with 3 parts, /generate_scene returned renderer: three, render_mode: three, and no HTML field, and a high confidence threshold returned render_mode: annotate.
• Implemented the docs/features/vision-prompt-roles.md prompt-role split on June 15, 2026: moved the invariant analysis contract into VISION_SYSTEM_PROMPT, reduced build_vision_prompt() to a backward-compatible per-image user-prompt alias, added build_vision_messages(), and wired Modal llama.cpp analysis calls to pass -sys with a fallback that prepends the system text only if the deployed binary rejects the system-prompt flag.
• Verification after the docs/features/vision-prompt-roles.md pass: prompt-helper checks, schema/parser/coercion checks, FastAPI TestClient checks for /, /analyze_image, and /generate_scene, and a modal_app.py import/helper check all passed locally. Modal dev raw analysis parsed strict JSON in 28.59s with coerced_render_mode: three, and the endpoint-style Modal diagnostic returned a validated two-part mechanism payload. Stable Modal deploy completed on June 15, 2026, and the stable unauthenticated runtime_probe request returned 401 Unauthorized.
• Committed and pushed f24826e (Split vision prompt roles) to GitHub main. GitHub Actions sync run 27530875972 completed successfully, and the private Hugging Face Space reported SHA f24826e7a4f7335cb4310f88c01baef7d8155860.
• Authenticated private Space verification after the prompt-role sync: root returned 200, the trusted shell was present, the synthetic target image returned a real Modal analysis for cutaway mechanism at 0.7 confidence with 3 parts, and /generate_scene returned renderer: three, render_mode: three, and no HTML field.
• Implemented the docs/reviews/scene-geometry-review.md deterministic renderer pass on June 15, 2026: axial and disk-like primitives now infer their axis from geometry.size, radius comes from the cross-section pair, explicit geometry.rotation remains additive, the Three.js assembly recenters before reveal animation, camera/OrbitControls fit is derived from the bounding sphere and viewport aspect, the grid moves to the recentered assembly bottom, and label projection uses world positions. Prompt wording now tells the model that the renderer infers primitive orientation from the odd-one-out size dimension.
• Local verification after the docs/reviews/scene-geometry-review.md pass: prompt/schema imports, schema validation, parser/coercion validation, FastAPI TestClient checks in INFERENCE_BACKEND=local, browser verification for docs/fixtures/mini-fan-analysis.json and EXAMPLE_ANALYSIS, desktop/mobile nonblank screenshots, mobile no-overflow measurement, orbit drag, and Reset view all passed.
• Reworked README.md for Build Small judging on June 15, 2026: added judge-facing YAML tags for the Backyard AI track plus Llama Champion, NVIDIA Nemotron Quest, Modal Award, OpenAI Codex, Off-Brand, Best Demo, and Sharing is Caring; shifted the body from a maintainer runbook to a public pitch with concise idea, demo, tech stack, verification, and setup sections; kept private Modal endpoint URLs and credentials out of public docs.
• Cleaned up repository notes on June 15, 2026: moved feature specs to docs/features/, review passes to docs/reviews/, and the mini-fan model output fixture to docs/fixtures/mini-fan-analysis.json; added docs/README.md as an index.
• Implemented the docs/reviews/motion-axis-bias.md pass on June 15, 2026: de-anchored the vision prompt from [0, 1, 0], added balanced X/Y/Z axis examples and a shaft exemplar, made missing browser-side motion axes derive from the same size-based primitive axis used for geometry orientation, and removed the hard-coded Y axis from the generic fallback rotor.
• Verification after the motion-axis pass: schema/parser/coercion checks passed, FastAPI TestClient checks passed in INFERENCE_BACKEND=local, Playwright renderer probes confirmed missing-axis X, Y, and Z parts infer the expected axes with no horizontal overflow, Modal dev run_analysis_endpoint_check returned a rotating dial with axis: [1, 0, 0], stable Modal deploy completed, and unauthenticated stable runtime_probe still returned 401 Unauthorized.
• Committed and pushed 5e65d06 (Reduce motion axis bias) to GitHub main. The first GitHub-to-HF sync failed because the new README short_description exceeded Hugging Face's 60-character metadata limit. Committed and pushed e7369eb (Fix Space metadata length), after which GitHub Actions sync run 27544185461 completed successfully.
• Hugging Face Space jasondo111/Snap2Sim reported SHA e7369eb7e22f8e25bbbac95a1531bacf242446cb, remained private, and was running on cpu-basic. Authenticated private Space verification after the motion-axis sync: root returned 200 with the trusted shell present, the synthetic target image returned real Modal analysis for rotating shaft assembly at 0.65 confidence with an emitted motion axis of [1.0, 0.0, 0.0], and /generate_scene returned renderer: three, render_mode: three, and no HTML field.
• User made Hugging Face Space jasondo111/Snap2Sim public on June 15, 2026. Verification confirmed Hub metadata private=false, the public Space page returned 200, and the public app host https://jasondo111-snap2sim.hf.space returned the trusted shell with the confidence slider present. The Space has not been transferred to build-small-hackathon/Snap2Sim; that target repo did not exist at the time of verification.
• Public endpoint smoke verification on June 15, 2026 passed: /generate_scene returned renderer: three with no HTML field for EXAMPLE_ANALYSIS, and /analyze_image returned a real Modal analysis for a synthetic target-like image as rotary indexer at 0.6 confidence with 3 parts.
• Committed and pushed 92853cc (Add hackathon presentation deck) to GitHub main. GitHub Actions sync run 27576590128 completed successfully, and the public Hugging Face Space reported SHA 92853cc9acf19555524a7f65b189926e9088f4fb. The synced Space includes prompts/PRESENTATION_PROMPT.md, presentation/deck.html, and presentation/assets/.gitkeep.
• User transferred the public Space to build-small-hackathon/Snap2Sim on June 15, 2026. Verification confirmed the org-owned Space is public, running on cpu-basic, and serves https://build-small-hackathon-snap2sim.hf.space. The transferred Space kept SHA f4134f7379db5abd0b6b58130da2d677c0fcb218 at verification time.
• Org-owned Space endpoint smoke verification on June 15, 2026 passed: /generate_scene returned renderer: three, render_mode: three, and no HTML field for EXAMPLE_ANALYSIS; /analyze_image returned a real Modal analysis for a synthetic target-like image as rotary indexing mechanism at 0.85 confidence with 3 parts.
• Final public submission links were added on June 15, 2026: YouTube demo https://youtu.be/nuisDKMyyF8 and X post https://x.com/Ryno67114241/status/2066660199558152411.

Final State / Maintenance Notes

• The Hugging Face Space is now public under build-small-hackathon/Snap2Sim. Keep README, presentation links, and .github/workflows/sync_to_hf.yml pointed at that org-owned Space.
• The public submission is complete with the app, README, presentation deck, YouTube demo, and X post linked from the repository.
• Keep the README opening hook intact: "You find a small metal cylinder at a flea market. What is it? How does it work inside?" Maintain the judge-facing YAML tags and public pitch if future README updates are needed.