Spaces:
Running
Running
| # Vision Prompt Roles: Split System and User Prompt | |
| ## Question being answered | |
| > Is it better to split the **analysis** portion and the **generation** portion | |
| > into different system prompts and user prompts? Would this help the model | |
| > interpret the input and produce better output? | |
| Short answer: **Yes, split — but split by chat *role* inside the single | |
| analysis call, not into two model calls.** The "generation" step is not a model | |
| step in this codebase, so the only useful split is system-vs-user within the one | |
| vision call. Wiring that up is a small, safe change that aligns the prompt with | |
| how the instruct/reasoning model was trained. | |
| --- | |
| ## Findings | |
| ### Finding 1 — There is only one model step; "generation" is deterministic | |
| The pipeline is: | |
| ``` | |
| image -> [MODEL] vision analysis -> validated JSON -> [NO MODEL] Three.js scene | |
| ``` | |
| - Scene rendering is deterministic browser-side Three.js built from the | |
| validated analysis JSON (`AGENTS.md:34-35`, `AGENTS.md:196-198`). | |
| - `InferenceClient.generate_scene` (`snap2sim/backend.py:47-51`) does **no** | |
| model inference — it validates the analysis and picks a `render_mode`. Model- | |
| authored scene HTML/JS was intentionally removed (`SECURITY.md:96-97`). | |
| **Implication:** "split analysis and generation into different prompts" cannot | |
| mean *two model calls*. There is no generation model call to separate, and | |
| adding a second round-trip would double an already slow path (~35s observed, | |
| `300s` timeout at `modal_app.py:392`) for zero quality gain, since the scene is | |
| deterministic. **Do not introduce a second inference step.** | |
| ### Finding 2 — The system prompt is currently dead code | |
| - `VISION_SYSTEM_PROMPT` (`snap2sim/prompts.py:5-10`) is defined but **never | |
| imported or used**. `modal_app.py` imports only `build_vision_prompt` | |
| (`modal_app.py:20`). | |
| - The real call, `run_llamacpp_prompt` (`modal_app.py:291-334`), passes the | |
| prompt through a single `-p` flag (`modal_app.py:308-309`). There is **no** | |
| system message. | |
| - Net effect: the model receives one large *user* turn that mixes four different | |
| concerns — role framing, the analytical task, the full output schema, and | |
| field-by-field formatting rules (`build_vision_prompt`, | |
| `snap2sim/prompts.py:12-70`). | |
| So today there is effectively **no** system/user separation, and the one | |
| "system" string we wrote is doing nothing. | |
| ### Finding 3 — The runtime supports a real role split | |
| - `llama-mtmd-cli` accepts a separate system message via `-sys` alongside the | |
| user prompt `-p`, and applies the model's chat template. (Confirmed against | |
| llama.cpp `mtmd-cli` usage.) | |
| - The model is `unsloth/NVIDIA-Nemotron-3-Nano-...-Reasoning-GGUF` | |
| (`AGENTS.md:20`), an instruct/reasoning model. Such models are trained to | |
| treat the **system** turn as persistent role + constraints and the **user** | |
| turn as the immediate request. Collapsing everything into the user turn is | |
| off-distribution and dilutes the per-request ask. | |
| ### Why a role split should help interpretation and output | |
| 1. **Aligns with training.** Stable behavior/output-contract belongs in system; | |
| the moment-to-moment ask belongs in user. The model already expects this | |
| shape. | |
| 2. **Sharpens the ask.** The user turn becomes short and image-focused | |
| ("Analyze the component in this photo …") instead of being buried under ~40 | |
| lines of schema rules. | |
| 3. **Separates invariants from the request.** The schema, shape/motion | |
| vocabulary, and hard rules are constant across every image; they read as | |
| *policy* in system, not as part of *this* request. | |
| 4. **Stops wasting the role channel.** We already wrote a system prompt; right | |
| now it is ignored. | |
| This is a low-risk change: it does not touch the schema, validator, coercion | |
| (`snap2sim/schema.py`, `snap2sim/model_io.py`), or the deterministic renderer. | |
| --- | |
| ## Recommendation | |
| **Adopt a system/user role split within the single vision call.** Concretely: | |
| - **System message** = the invariant output contract: | |
| - role + reasoning frame (it is a reasoning model; brief reasoning then JSON), | |
| - the "emit exactly one JSON object, no markdown" rule, | |
| - the JSON skeleton, | |
| - the shape -> use-for and motion -> use-for vocabulary guide, | |
| - the hard field rules (`size: [x,y,z]`, numeric axis vectors, 2–6 parts, no | |
| `radius/height/length/width/depth`). | |
| - **User message** = only the per-image ask: "Analyze the hardware component in | |
| this photo as a cutaway mechanism and return the analysis JSON." (The image is | |
| attached via `--image`.) | |
| Keep it one inference call. Keep temperature, token, context, and timeout | |
| budgets as-is (`modal_app.py:388-394`) — this is a prompt-structure change, not | |
| a budget change. | |
| --- | |
| ## Implementation plan (completed) | |
| 1. **`snap2sim/prompts.py`** | |
| - Repurpose `VISION_SYSTEM_PROMPT` to hold the full invariant contract | |
| (role + JSON-only rule + skeleton + shape/motion vocabulary guide + hard | |
| field rules) currently living inside `build_vision_prompt`. | |
| - Reduce `build_vision_prompt()` to the short per-image ask only. Consider | |
| renaming it `build_vision_user_prompt()` (keep a back-compat alias if any | |
| diagnostic entrypoint imports the old name). | |
| - Optionally add `build_vision_messages()` returning `(system, user)` so call | |
| sites have one source of truth. | |
| 2. **`snap2sim/prompts.py` (smoke test parity, optional)** | |
| - The smoke-test prompt is a separate inline string | |
| (`modal_app.py:211-234`). Leave functionally as-is, but it can reuse the | |
| same system message for realism. Not required for the demo. | |
| 3. **`modal_app.py`** | |
| - In `run_llamacpp_prompt` (`:291-334`), add an optional `system_prompt: | |
| str | None = None` parameter and append `-sys <system_prompt>` to `cmd` | |
| when provided. Verify the exact flag name against the deployed | |
| `llama-mtmd-cli` build before relying on it (`-sys`); if the installed | |
| build does not accept it, fall back to prepending the system text to `-p` | |
| so behavior never regresses. | |
| - In `analyze_image_llamacpp_payload` (`:385-400`), pass the new system | |
| message + the short user prompt. | |
| - Update the diagnostic entrypoints that build the prompt | |
| (`run_analysis_raw_check` near `:489`, `run_analysis_endpoint_check`, | |
| remote check near `:512`) to use the same two-part prompt so diagnostics | |
| match production. | |
| 4. **Verify before/after (no regressions, measured improvement)** | |
| - Local: schema/parser checks + FastAPI `TestClient` for `/`, | |
| `/analyze_image`, `/generate_scene` still pass. | |
| - Modal: run `run_analysis_raw_check` and `run_analysis_endpoint_check` | |
| against the synthetic target image; confirm strict JSON still parses and | |
| latency is comparable. Compare parse success / field quality vs. the | |
| current single-`-p` prompt on a few representative real photos before | |
| committing the prompt change. | |
| - Confirm the flag actually took effect (e.g. inspect that `-sys` is honored, | |
| not silently ignored) before trusting the split. | |
| 5. **Then deploy** following the normal path (Modal deploy of | |
| `analyze_image_llamacpp`, GitHub `main` push, GitHub->HF sync, and Space | |
| verification), per the existing workflow in `AGENTS.md`. | |
| Final submission note, 2026-06-15: this prompt-role split shipped before the | |
| public `build-small-hackathon/Snap2Sim` submission. | |
| --- | |
| ## Explicitly NOT recommended | |
| - **Two separate model calls** (an "analysis" call feeding a "generation" | |
| call). There is no model generation step; the scene is deterministic. This | |
| would only add latency and a failure mode. | |
| - **Re-introducing model-authored scene HTML/JS.** Prohibited by | |
| `SECURITY.md:96-97` and `AGENTS.md:35`. | |
| ## Open question for the user | |
| The recommendation assumes you meant "should the prompt be *structured* into | |
| system vs user roles" (yes), not "should the app make two separate model | |
| requests" (no — there is no model generation step). If you specifically wanted | |
| the model to also generate the scene/animation (re-adding a model generation | |
| call), that conflicts with the current deterministic-renderer security decision | |
| and should be discussed separately before implementing. | |