README.md

metadata

title: Snap2Sim Inside The Machine
sdk: gradio
app_file: app.py
license: mit
colorFrom: blue
colorTo: yellow
short_description: Photograph a part. Get a mechanical cutaway.
models:
  - unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
tags:
  - build-small-hackathon
  - backyard-ai
  - llama-champion
  - nvidia-nemotron-quest
  - modal-award
  - openai-codex
  - off-brand
  - best-demo
  - sharing-is-caring
  - llama-cpp
  - gguf
  - gradio
  - three-js
  - track:backyard
  - sponsor:openai
  - sponsor:nvidia
  - sponsor:modal
  - achievement:offbrand
  - achievement:llama
  - achievement:sharing

Snap2Sim: Inside the Machine

You find a small metal cylinder at a flea market. What is it? How does it work inside?

Snap2Sim turns a photo of a hardware component into a narrated mechanical cutaway. Drop in a picture of a valve, lock, pump, gear train, fan, latch, or other mystery part; the app identifies the likely mechanism, names the internal pieces, and renders a moving 3D field-manual view that explains how the object works.

The goal is not just recognition. It is the moment a curious maker gets to see the hidden motion: the plunger sliding, the spring loading, the rotor spinning, the pawl catching, the port opening.

Demo

• Hugging Face Space: https://huggingface.co/spaces/build-small-hackathon/Snap2Sim
• App host: https://build-small-hackathon-snap2sim.hf.space
• Source of truth: https://github.com/Bigstonks1/Snap2Sim
• Demo video: https://youtu.be/nuisDKMyyF8
• X post: https://x.com/Ryno67114241/status/2066660199558152411

The Space is public for submission under the Build Small Hackathon organization. The app is built as a Gradio Space with a custom gradio.Server shell; GitHub remains the source of truth for code changes.

Hackathon Fit

Snap2Sim is a Backyard AI project for people who learn by taking things apart: garage tinkerers, repair hobbyists, thrift-store scavengers, robotics students, and anyone holding an unknown part and wondering what is going on inside.

The README metadata tags request consideration for:

Track or badge	Why it applies
Backyard AI	Explains real-world objects from a user's own photo.
Llama Champion	Runs the primary model through llama.cpp with GGUF assets.
NVIDIA Nemotron Quest	Uses NVIDIA Nemotron 3 Nano Omni as the vision-reasoning model.
Modal Award	GPU inference is deployed through Modal with cached model assets.
OpenAI Codex	Built with Codex assistance and Codex-attributed commits in the linked GitHub repo.
Off-Brand	The experience is a branded maker tool, not a generic chatbot or image demo.
Best Demo / Sharing is Caring	Public demo video and X post are linked in the Demo section.
Off the Grid	Not claimed while inference runs on Modal.

How It Works

1. The browser accepts a component photo and sends it to the Space's same-origin /analyze_image API.
2. The Space calls a secured Modal endpoint running unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF through llama.cpp and llama-mtmd-cli.
3. Nemotron returns a structured mechanism JSON payload: component name, confidence, internal parts, geometry hints, motion types, annotations, and a plain-language explanation.
4. /generate_scene validates the analysis and chooses the render mode.
5. The browser renders the result with trusted deterministic Three.js code. If a confident 3D reconstruction is not justified, it falls back to annotated callouts on the original photo instead of pretending.

The model describes the mechanism. The browser owns the rendering. That keeps the demo reliable and avoids injecting model-authored HTML or JavaScript into the page.

Tech Stack

• Frontend: plain index.html, CSS, and JavaScript served directly by gradio.Server.
• Rendering: deterministic Three.js primitives, labels, OrbitControls, camera fitting, and an annotated-photo fallback.
• Model runtime: NVIDIA Nemotron 3 Nano Omni 30B-A3B GGUF, UD-Q4_K_M, with mmproj-F16.gguf through llama.cpp.
• GPU service: Modal, with weights cached in a Modal Volume and bearer-token protection between the Space and Modal.
• Space runtime: Hugging Face cpu-basic; expensive inference happens on Modal.
• Contract: JSON schema validation in snap2sim/schema.py before anything is rendered.

The primary model is approximately 30B total parameters with roughly 3B active per MoE token, keeping the single-model pipeline under the hackathon's 32B model budget.

What Makes It Different

Most image demos stop at a label. Snap2Sim tries to answer the maker's next question: "what would I see if I cut this open?"

• It produces a mechanism explanation, not just a class name.
• It maps internal parts into a moving technical cutaway.
• It shows uncertainty honestly with a confidence threshold and a photo annotation fallback.
• It uses a custom public-facing interface instead of a default chat or form UI.
• It keeps secrets and GPU endpoints server-side while the client calls only same-origin APIs.

Current Verification

Recent local and deployed checks confirmed:

• The public Space serves the trusted index.html shell.
• /analyze_image returns validated mechanism payloads through the secured Modal bearer-token flow.
• A public smoke test on the org-owned Space on June 15, 2026 returned a real Modal analysis for a synthetic target-like image as rotary indexing mechanism at 0.85 confidence with 3 parts.
• /generate_scene returns a deterministic Three.js scene descriptor, not raw HTML; a public smoke test returned renderer: three.
• The renderer handles six-part scenes, axial orientation from geometry sizes, geometry-aware motion-axis fallback, fitted camera framing, orbit controls, reset view, mobile layout, and annotated-photo fallbacks.
• Unauthenticated Modal requests return 401 Unauthorized.

Run Locally

pip install -r requirements.txt
python app.py

For UI-only development, set:

$env:INFERENCE_BACKEND="local"
python app.py

For the real demo path, configure the public variables and private token as described in .env.example and SECURITY.md. Do not commit tokens, Modal endpoint URLs, uploaded user photos, or downloaded model weights.

Runtime Preflight

Check the selected GGUF and projector metadata before spending GPU time:

python scripts/verify_runtime_assets.py

Useful Modal checks:

python -m modal run modal_app.py::check_remote_imports
python -m modal run modal_app.py::download_runtime_assets
python -m modal run modal_app.py::smoke_test_llamacpp_image
python -m modal run modal_app.py::run_analysis_endpoint_check

On Windows, use UTF-8 output when running Modal commands:

$env:PYTHONIOENCODING="utf-8"; python -m modal run modal_app.py::run_runtime_preflight

Repository Map

• app.py - Gradio Server app, same-origin APIs, image validation, rate limits.
• index.html - upload flow, analysis panel, Three.js renderer, fallback UI.
• modal_app.py - Modal deployment, llama.cpp runtime, diagnostics.
• prompts/ - archived original build prompt, GitHub sync prompt, and deck prompt.
• presentation/deck.html - reveal.js judging deck for the hackathon pitch.
• docs/ - implementation notes, review passes, and representative fixtures.
• snap2sim/backend.py - backend selection and authenticated Modal client.
• snap2sim/prompts.py - vision prompt contract.
• snap2sim/schema.py - mechanism JSON schema and sample payload.
• scripts/verify_runtime_assets.py - Hugging Face model metadata preflight.
• SECURITY.md - public deployment handling rules.

Acknowledgements

Built by Jason Do with implementation assistance from OpenAI Codex.