docs/features/confidence-threshold.md

Confidence Threshold: Apply Only at Analysis and Generation

Status: Implemented and synced. Final public submission is under build-small-hackathon/Snap2Sim as of June 15, 2026. Supersedes the prior implemented version of this file (commit a0540e9, "Add confidence threshold control"). Author of spec: codebase review pass, June 14, 2026.

Why this supersedes the prior spec. The slider already exists, but it was built as a client-authoritative, live re-render control: moving it instantly recomputes the render mode from cached analysis in the browser, and the server never sees the chosen threshold. The user has reversed all three of those decisions. The threshold must now be applied only when analysis and generation run, enforced server-side, and the slider must be usable before the first upload.

Findings — prior behavior before this re-spec

A confidence-threshold slider already existed and worked, but not the way the user wanted in this re-spec.

1. Slider UI exists. index.html:593-597 — #confidenceThreshold (type="range", min=0 max=1 step=0.05 value=0.5), with a live #thresholdValue label. It carries the disabled attribute and is only enabled after a run.

2. Slider applies live, client-side, from cache. The input handler at index.html:722-725 calls updateConfidenceThreshold() then scheduleThresholdRender() (index.html:1279-1286), which debounces 90 ms and re-runs renderScenePayload(window.lastScenePayload) against the cached analysis. No network call — but it re-renders on every drag, decoupled from any analysis/generation step.

3. Threshold is applied client-side only. chooseRenderMode() (index.html:821-833) compares analysis.confidence against the JS confidenceThreshold var. It ignores the render_mode the server already returned in the payload.

4. The server never receives the threshold. /generate_scene (app.py:101-108 → _generate_scene at app.py:116-117 → InferenceClient.generate_scene at backend.py:47-51) calls select_render_mode(valid_analysis) (schema.py:219-236), which always uses the hardcoded DEFAULT_CONFIDENCE_THRESHOLD = 0.5 (schema.py:216). The server's render_mode is computed with 0.5 regardless of the slider.

5. Slider availability is gated on a completed run. It is disabled in markup (index.html:595), disabled in resetScene() (index.html:769), and setBusy() (index.html:1263) keeps it disabled whenever !window.lastScenePayload. So it cannot be set before the first analysis.

Conclusion: the threshold is currently a live, browser-only control that never reaches the analysis/generation pipeline — the opposite of "only used upon analysis and generation." This spec changes it to a value that is captured and enforced at generation time, server-side, and that only takes effect on the next run.

Product decisions (confirmed with user, June 14, 2026)

1. Apply timing: next run only. Moving the slider does nothing immediately — no live re-render of cached analysis. The new threshold is captured and applied only the next time analysis + generation runs (next upload / re-run). Remove the live client-side re-render entirely.
2. Enforcement: server pipeline. The slider value is sent to /generate_scene, and the server's select_render_mode() uses the user's threshold instead of the hardcoded 0.5. The server's returned render_mode becomes the source of truth; the client trusts it.
3. Pre-run state: enabled up front. The slider is usable before the first upload so the chosen threshold is in effect for the very first analysis/generation.

Unchanged from prior spec (still true):

• Downgrade chain stays: below threshold, skip the 3D Three.js render and fall back to annotate (annotated source photo), then unavailable. The slider only moves the cutoff; no new hard-block state.
• No re-running model inference on slider move (analysis is the expensive Modal GPU call; it is never re-triggered by the slider).

Implementation plan (for Codex)

Changes span the browser (index.html) and the server (app.py, snap2sim/backend.py, snap2sim/schema.py).

1. Server: accept and enforce a threshold at generation

• snap2sim/schema.py — select_render_mode(analysis, threshold=DEFAULT_CONFIDENCE_THRESHOLD):
  • Add a threshold parameter, defaulting to DEFAULT_CONFIDENCE_THRESHOLD.
  • Coerce/clamp: if threshold is not a number, fall back to the default; clamp into [0.0, 1.0].
  • Replace the hardcoded DEFAULT_CONFIDENCE_THRESHOLD in the low_confidence line (schema.py:229) with the (clamped) threshold.
• snap2sim/backend.py — InferenceClient.generate_scene(analysis, threshold=None) (backend.py:47):
  • Accept an optional threshold; when None/invalid use DEFAULT_CONFIDENCE_THRESHOLD.
  • Pass it through to select_render_mode(valid_analysis, threshold).
• app.py:
  • /generate_scene HTTP route (app.py:106-108): read payload.get("confidence_threshold") and pass to _generate_scene.
  • @app.api(name="generate_scene") (app.py:101-103): add an optional confidence_threshold parameter (default keeps the existing /run_pipeline Gradio API backward compatible).
  • _generate_scene(analysis, threshold) (app.py:116-117): forward the threshold to InferenceClient(...).generate_scene(analysis, threshold).
  • Validate at the boundary: coerce to float, clamp [0, 1], default on missing/invalid. Do not raise on a bad threshold — fall back to the default so a malformed client value can't break generation.

Result: /generate_scene returns a render_mode computed with the user's threshold. Existing callers that omit confidence_threshold still get the 0.5 default — backward compatible.

2. Client: send the threshold at generation, stop live re-render

• Capture and send at run time. In runPipeline() (index.html:727-759), include the current threshold in the generate call: postJson("/generate_scene", { analysis, confidence_threshold: confidenceThreshold }) (index.html:749). The value is read at the moment of the call, so later slider drags don't affect the in-flight run.
• Trust the server's render mode. Change renderScenePayload() (index.html:805-819) / chooseRenderMode() (index.html:821-833) so the decision uses payload.render_mode as the primary choice, with the existing capability guards (hasUsableGeometry, hasAnnotations) only to downgrade when data is missing — never to upgrade past what the server allowed. Remove the client-side confidence vs confidenceThreshold comparison (the server now owns that). chooseRenderMode should take the payload (or render_mode) rather than recomputing from confidence.
• Remove the live re-render. Delete scheduleThresholdRender() (index.html:1279-1286) and the thresholdRenderTimer state (index.html:675). The slider input handler (index.html:722-725) should now only call updateConfidenceThreshold() — update the confidenceThreshold var and the #thresholdValue label. No render, no network call.

3. Client: enable the slider up front

• Remove the disabled attribute from the markup (index.html:595).
• In resetScene() (index.html:761-776), stop disabling the slider (index.html:769) — it should remain available between runs.
• In setBusy() (index.html:1259-1264), disable the slider only while a request is in flight (active), not based on window.lastScenePayload (index.html:1263). This lets the user set the threshold before the first upload and adjust it between runs, while preventing edits mid-request.
• Keep the default at 0.5 / 50% so first-run behavior is unchanged when the user never touches the slider.

4. Optional: reflect "applies on next run" in the UI

Because the slider no longer re-renders live, consider a subtle affordance so the change isn't silent — e.g. update the label to hint the value applies to the next analysis (tooltip or helper text). Low priority; keep it lightweight and accessible (don't regress the existing aria-live label).

Out of scope / explicitly NOT doing

• No live re-render from cached analysis (decision #1 — this is the behavior being removed).
• No re-running model inference (/analyze_image) when the slider moves (analysis is the expensive Modal GPU call).
• No hard-block "confidence too low" state — the three -> annotate -> unavailable downgrade chain stays.
• No model-authored HTML/JS/markup injection. Rendering stays deterministic Three.js from validated JSON (SECURITY.md Agent Guidance). The threshold is a non-credential UX/quality control; sending it to a same-origin endpoint is fine.

Verification checklist

• Server, default: /generate_scene without confidence_threshold returns the same render_mode as today (regression — 0.5 default). Add/extend a unit check: select_render_mode(analysis, 0.9) downgrades a 0.7-confidence geometry payload to annotate/unavailable; select_render_mode(analysis, 0.1) keeps it three.
• Server, clamp/coerce: out-of-range (-1, 5) and non-numeric thresholds fall back/clamp without raising; /generate_scene still returns a valid payload.
• Client, next-run-only: moving the slider after a run does not trigger any network request and does not change the current render (confirm via devtools Network + visual). The new value only takes effect after the next upload / re-run.
• Client, enforced server-side: raising the threshold above the analysis's confidence and re-running downgrades the 3D cutaway to annotated photo / then unavailable; lowering it and re-running promotes back to 3D when geometry exists.
• Client, pre-run: the slider is interactive before the first upload, disabled only while a request is in flight, and re-enabled afterward.
• Accessibility/layout (regression): slider keyboard-operable, #thresholdValue announced, does not block canvas OrbitControls (the pointer-events bug fixed in the docs/reviews/interaction-and-fallback-review.md pass), and no mobile horizontal overflow.
• INFERENCE_BACKEND=local sample mode still renders the example analysis with the slider present and the threshold honored at generation.
• Existing local checks pass: schema/parser checks and FastAPI TestClient for /, /analyze_image, /generate_scene.

Touch points (file/line reference)

• snap2sim/schema.py:219-236 select_render_mode() — add threshold param, clamp, use it instead of the hardcoded default at schema.py:229.
• snap2sim/backend.py:47-51 generate_scene() — accept + forward threshold.
• app.py:101-108 generate_scene_api / generate_scene_http — accept confidence_threshold; app.py:116-117 _generate_scene — forward it; clamp at the boundary.
• index.html:595 slider markup — remove disabled.
• index.html:722-725 slider input handler — drop the live re-render call.
• index.html:749 /generate_scene call — send confidence_threshold.
• index.html:761-776 resetScene() / index.html:1259-1264 setBusy() — enable slider up front, disable only while busy.
• index.html:805-833 renderScenePayload() / chooseRenderMode() — trust server render_mode; drop client-side threshold comparison.
• index.html:675 thresholdRenderTimer + index.html:1279-1286 scheduleThresholdRender() — remove.

Implementation result

• Implemented in commit a6f63e9 (Enforce confidence threshold during generation).
• The slider is enabled before the first upload and disabled only while a request is in flight.
• The browser sends confidence_threshold only with /generate_scene; slider movement does not call /analyze_image, call /generate_scene, or re-render cached analysis.
• /generate_scene clamps/coerces the threshold server-side and returns the authoritative render_mode; the browser only downgrades when geometry or annotation data is missing.
• Local verification passed for default/high/low/malformed/clamped thresholds, FastAPI TestClient, next-run-only browser behavior, high-threshold downgrade, low-threshold promotion, keyboard slider operation, mobile no-overflow layout, and canvas pointer targeting.
• GitHub Actions sync run 27515950105 deployed commit a6f63e9 to the then-private Hugging Face Space. The Space reported SHA a6f63e9a0b76315bb223a09a71f4c027a29877fb.
• Authenticated Space verification passed: the root served the updated shell, a synthetic image returned optical sight at 0.7 confidence with 3 parts, high threshold returned photo / annotate, low threshold returned three / three, and no HTML field was present.