# Vision Prompt Roles: Split System and User Prompt ## Question being answered > Is it better to split the **analysis** portion and the **generation** portion > into different system prompts and user prompts? Would this help the model > interpret the input and produce better output? Short answer: **Yes, split — but split by chat *role* inside the single analysis call, not into two model calls.** The "generation" step is not a model step in this codebase, so the only useful split is system-vs-user within the one vision call. Wiring that up is a small, safe change that aligns the prompt with how the instruct/reasoning model was trained. --- ## Findings ### Finding 1 — There is only one model step; "generation" is deterministic The pipeline is: ``` image -> [MODEL] vision analysis -> validated JSON -> [NO MODEL] Three.js scene ``` - Scene rendering is deterministic browser-side Three.js built from the validated analysis JSON (`AGENTS.md:34-35`, `AGENTS.md:196-198`). - `InferenceClient.generate_scene` (`snap2sim/backend.py:47-51`) does **no** model inference — it validates the analysis and picks a `render_mode`. Model- authored scene HTML/JS was intentionally removed (`SECURITY.md:96-97`). **Implication:** "split analysis and generation into different prompts" cannot mean *two model calls*. There is no generation model call to separate, and adding a second round-trip would double an already slow path (~35s observed, `300s` timeout at `modal_app.py:392`) for zero quality gain, since the scene is deterministic. **Do not introduce a second inference step.** ### Finding 2 — The system prompt is currently dead code - `VISION_SYSTEM_PROMPT` (`snap2sim/prompts.py:5-10`) is defined but **never imported or used**. `modal_app.py` imports only `build_vision_prompt` (`modal_app.py:20`). - The real call, `run_llamacpp_prompt` (`modal_app.py:291-334`), passes the prompt through a single `-p` flag (`modal_app.py:308-309`). There is **no** system message. - Net effect: the model receives one large *user* turn that mixes four different concerns — role framing, the analytical task, the full output schema, and field-by-field formatting rules (`build_vision_prompt`, `snap2sim/prompts.py:12-70`). So today there is effectively **no** system/user separation, and the one "system" string we wrote is doing nothing. ### Finding 3 — The runtime supports a real role split - `llama-mtmd-cli` accepts a separate system message via `-sys` alongside the user prompt `-p`, and applies the model's chat template. (Confirmed against llama.cpp `mtmd-cli` usage.) - The model is `unsloth/NVIDIA-Nemotron-3-Nano-...-Reasoning-GGUF` (`AGENTS.md:20`), an instruct/reasoning model. Such models are trained to treat the **system** turn as persistent role + constraints and the **user** turn as the immediate request. Collapsing everything into the user turn is off-distribution and dilutes the per-request ask. ### Why a role split should help interpretation and output 1. **Aligns with training.** Stable behavior/output-contract belongs in system; the moment-to-moment ask belongs in user. The model already expects this shape. 2. **Sharpens the ask.** The user turn becomes short and image-focused ("Analyze the component in this photo …") instead of being buried under ~40 lines of schema rules. 3. **Separates invariants from the request.** The schema, shape/motion vocabulary, and hard rules are constant across every image; they read as *policy* in system, not as part of *this* request. 4. **Stops wasting the role channel.** We already wrote a system prompt; right now it is ignored. This is a low-risk change: it does not touch the schema, validator, coercion (`snap2sim/schema.py`, `snap2sim/model_io.py`), or the deterministic renderer. --- ## Recommendation **Adopt a system/user role split within the single vision call.** Concretely: - **System message** = the invariant output contract: - role + reasoning frame (it is a reasoning model; brief reasoning then JSON), - the "emit exactly one JSON object, no markdown" rule, - the JSON skeleton, - the shape -> use-for and motion -> use-for vocabulary guide, - the hard field rules (`size: [x,y,z]`, numeric axis vectors, 2–6 parts, no `radius/height/length/width/depth`). - **User message** = only the per-image ask: "Analyze the hardware component in this photo as a cutaway mechanism and return the analysis JSON." (The image is attached via `--image`.) Keep it one inference call. Keep temperature, token, context, and timeout budgets as-is (`modal_app.py:388-394`) — this is a prompt-structure change, not a budget change. --- ## Implementation plan (completed) 1. **`snap2sim/prompts.py`** - Repurpose `VISION_SYSTEM_PROMPT` to hold the full invariant contract (role + JSON-only rule + skeleton + shape/motion vocabulary guide + hard field rules) currently living inside `build_vision_prompt`. - Reduce `build_vision_prompt()` to the short per-image ask only. Consider renaming it `build_vision_user_prompt()` (keep a back-compat alias if any diagnostic entrypoint imports the old name). - Optionally add `build_vision_messages()` returning `(system, user)` so call sites have one source of truth. 2. **`snap2sim/prompts.py` (smoke test parity, optional)** - The smoke-test prompt is a separate inline string (`modal_app.py:211-234`). Leave functionally as-is, but it can reuse the same system message for realism. Not required for the demo. 3. **`modal_app.py`** - In `run_llamacpp_prompt` (`:291-334`), add an optional `system_prompt: str | None = None` parameter and append `-sys ` to `cmd` when provided. Verify the exact flag name against the deployed `llama-mtmd-cli` build before relying on it (`-sys`); if the installed build does not accept it, fall back to prepending the system text to `-p` so behavior never regresses. - In `analyze_image_llamacpp_payload` (`:385-400`), pass the new system message + the short user prompt. - Update the diagnostic entrypoints that build the prompt (`run_analysis_raw_check` near `:489`, `run_analysis_endpoint_check`, remote check near `:512`) to use the same two-part prompt so diagnostics match production. 4. **Verify before/after (no regressions, measured improvement)** - Local: schema/parser checks + FastAPI `TestClient` for `/`, `/analyze_image`, `/generate_scene` still pass. - Modal: run `run_analysis_raw_check` and `run_analysis_endpoint_check` against the synthetic target image; confirm strict JSON still parses and latency is comparable. Compare parse success / field quality vs. the current single-`-p` prompt on a few representative real photos before committing the prompt change. - Confirm the flag actually took effect (e.g. inspect that `-sys` is honored, not silently ignored) before trusting the split. 5. **Then deploy** following the normal path (Modal deploy of `analyze_image_llamacpp`, GitHub `main` push, GitHub->HF sync, and Space verification), per the existing workflow in `AGENTS.md`. Final submission note, 2026-06-15: this prompt-role split shipped before the public `build-small-hackathon/Snap2Sim` submission. --- ## Explicitly NOT recommended - **Two separate model calls** (an "analysis" call feeding a "generation" call). There is no model generation step; the scene is deterministic. This would only add latency and a failure mode. - **Re-introducing model-authored scene HTML/JS.** Prohibited by `SECURITY.md:96-97` and `AGENTS.md:35`. ## Open question for the user The recommendation assumes you meant "should the prompt be *structured* into system vs user roles" (yes), not "should the app make two separate model requests" (no — there is no model generation step). If you specifically wanted the model to also generate the scene/animation (re-adding a model generation call), that conflicts with the current deterministic-renderer security decision and should be discussed separately before implementing.