docs/features/vision-prompt-roles.md

Vision Prompt Roles: Split System and User Prompt

Question being answered

Is it better to split the analysis portion and the generation portion into different system prompts and user prompts? Would this help the model interpret the input and produce better output?

Short answer: Yes, split — but split by chat role inside the single analysis call, not into two model calls. The "generation" step is not a model step in this codebase, so the only useful split is system-vs-user within the one vision call. Wiring that up is a small, safe change that aligns the prompt with how the instruct/reasoning model was trained.

---

Findings

Finding 1 — There is only one model step; "generation" is deterministic

The pipeline is:

image -> [MODEL] vision analysis -> validated JSON -> [NO MODEL] Three.js scene

• Scene rendering is deterministic browser-side Three.js built from the validated analysis JSON (AGENTS.md:34-35, AGENTS.md:196-198).
• InferenceClient.generate_scene (snap2sim/backend.py:47-51) does no model inference — it validates the analysis and picks a render_mode. Model- authored scene HTML/JS was intentionally removed (SECURITY.md:96-97).

Implication: "split analysis and generation into different prompts" cannot mean two model calls. There is no generation model call to separate, and adding a second round-trip would double an already slow path (~35s observed, 300s timeout at modal_app.py:392) for zero quality gain, since the scene is deterministic. Do not introduce a second inference step.

Finding 2 — The system prompt is currently dead code

• VISION_SYSTEM_PROMPT (snap2sim/prompts.py:5-10) is defined but never imported or used. modal_app.py imports only build_vision_prompt (modal_app.py:20).
• The real call, run_llamacpp_prompt (modal_app.py:291-334), passes the prompt through a single -p flag (modal_app.py:308-309). There is no system message.
• Net effect: the model receives one large user turn that mixes four different concerns — role framing, the analytical task, the full output schema, and field-by-field formatting rules (build_vision_prompt, snap2sim/prompts.py:12-70).

So today there is effectively no system/user separation, and the one "system" string we wrote is doing nothing.

Finding 3 — The runtime supports a real role split

• llama-mtmd-cli accepts a separate system message via -sys alongside the user prompt -p, and applies the model's chat template. (Confirmed against llama.cpp mtmd-cli usage.)
• The model is unsloth/NVIDIA-Nemotron-3-Nano-...-Reasoning-GGUF (AGENTS.md:20), an instruct/reasoning model. Such models are trained to treat the system turn as persistent role + constraints and the user turn as the immediate request. Collapsing everything into the user turn is off-distribution and dilutes the per-request ask.

Why a role split should help interpretation and output

1. Aligns with training. Stable behavior/output-contract belongs in system; the moment-to-moment ask belongs in user. The model already expects this shape.
2. Sharpens the ask. The user turn becomes short and image-focused ("Analyze the component in this photo …") instead of being buried under ~40 lines of schema rules.
3. Separates invariants from the request. The schema, shape/motion vocabulary, and hard rules are constant across every image; they read as policy in system, not as part of this request.
4. Stops wasting the role channel. We already wrote a system prompt; right now it is ignored.

This is a low-risk change: it does not touch the schema, validator, coercion (snap2sim/schema.py, snap2sim/model_io.py), or the deterministic renderer.

---

Recommendation

Adopt a system/user role split within the single vision call. Concretely:

• System message = the invariant output contract:
  • role + reasoning frame (it is a reasoning model; brief reasoning then JSON),
  • the "emit exactly one JSON object, no markdown" rule,
  • the JSON skeleton,
  • the shape -> use-for and motion -> use-for vocabulary guide,
  • the hard field rules (size: [x,y,z], numeric axis vectors, 2–6 parts, no radius/height/length/width/depth).
• User message = only the per-image ask: "Analyze the hardware component in this photo as a cutaway mechanism and return the analysis JSON." (The image is attached via --image.)

Keep it one inference call. Keep temperature, token, context, and timeout budgets as-is (modal_app.py:388-394) — this is a prompt-structure change, not a budget change.

---

Implementation plan (completed)

1. snap2sim/prompts.py

   • Repurpose VISION_SYSTEM_PROMPT to hold the full invariant contract (role + JSON-only rule + skeleton + shape/motion vocabulary guide + hard field rules) currently living inside build_vision_prompt.
   • Reduce build_vision_prompt() to the short per-image ask only. Consider renaming it build_vision_user_prompt() (keep a back-compat alias if any diagnostic entrypoint imports the old name).
   • Optionally add build_vision_messages() returning (system, user) so call sites have one source of truth.

2. snap2sim/prompts.py (smoke test parity, optional)

   • The smoke-test prompt is a separate inline string (modal_app.py:211-234). Leave functionally as-is, but it can reuse the same system message for realism. Not required for the demo.

3. modal_app.py

   • In run_llamacpp_prompt (:291-334), add an optional system_prompt: str | None = None parameter and append -sys <system_prompt> to cmd when provided. Verify the exact flag name against the deployed llama-mtmd-cli build before relying on it (-sys); if the installed build does not accept it, fall back to prepending the system text to -p so behavior never regresses.
   • In analyze_image_llamacpp_payload (:385-400), pass the new system message + the short user prompt.
   • Update the diagnostic entrypoints that build the prompt (run_analysis_raw_check near :489, run_analysis_endpoint_check, remote check near :512) to use the same two-part prompt so diagnostics match production.

4. Verify before/after (no regressions, measured improvement)

   • Local: schema/parser checks + FastAPI TestClient for /, /analyze_image, /generate_scene still pass.
   • Modal: run run_analysis_raw_check and run_analysis_endpoint_check against the synthetic target image; confirm strict JSON still parses and latency is comparable. Compare parse success / field quality vs. the current single--p prompt on a few representative real photos before committing the prompt change.
   • Confirm the flag actually took effect (e.g. inspect that -sys is honored, not silently ignored) before trusting the split.

5. Then deploy following the normal path (Modal deploy of analyze_image_llamacpp, GitHub main push, GitHub->HF sync, and Space verification), per the existing workflow in AGENTS.md.

Final submission note, 2026-06-15: this prompt-role split shipped before the public build-small-hackathon/Snap2Sim submission.

---

Explicitly NOT recommended

• Two separate model calls (an "analysis" call feeding a "generation" call). There is no model generation step; the scene is deterministic. This would only add latency and a failure mode.
• Re-introducing model-authored scene HTML/JS. Prohibited by SECURITY.md:96-97 and AGENTS.md:35.

Open question for the user

The recommendation assumes you meant "should the prompt be structured into system vs user roles" (yes), not "should the app make two separate model requests" (no — there is no model generation step). If you specifically wanted the model to also generate the scene/animation (re-adding a model generation call), that conflicts with the current deterministic-renderer security decision and should be discussed separately before implementing.