docs/features/annotated-photo-fallback.md

Annotated-Photo Fallback: Image Sizing and Annotation Accuracy

Review date: 2026-06-14. Scope: one user report — when the annotated-photo fallback triggers (instead of the 3D cutaway), the uploaded image is "too large to display," and the annotation placement should be confirmed accurate.

This document is findings + next steps for Codex to implement. It does not authorize any deployment or Hugging Face changes on its own; follow the normal GitHub → HF sync and verification flow in AGENTS.md. All file/line references are to index.html unless noted.

---

TL;DR

1. The image-sizing bug is real and confirmed in a browser. The annotated-photo image renders at its intrinsic aspect height at full panel width, ignoring the viewport height, so tall photos overflow the viewport by ~2x. object-fit: contain never engages because the box it should fit inside is itself oversized.
2. Root cause: nested display: grid; place-items: center containers (.annotated-stage → .annotation-frame) with auto-sized rows. A child's height: 100% resolves against an auto (content-sized) grid row, not the container's pixel height, so it collapses back to the image's natural size.
3. The annotation accuracy is mostly broken as a side effect of the sizing bug — labels are positioned against the oversized, overflowing image box, so any callout in the lower half of a tall photo lands below the visible viewport and is clipped. Fixing the sizing fixes most of the misplacement. A few annotation-accuracy improvements remain that are independent of sizing (no marker dot at the point, edge-clamp detaches labels, unused box).

---

How this was verified (not just code inspection)

I reproduced the exact buildAnnotatedPhoto DOM (index.html:996-1076) and the exact CSS rules (.fallback-stage/.annotated-stage/.annotation-frame/ .annotation-image, index.html:466-511) in a standalone page, loaded a 3000 × 4000 portrait test image into a 900 × 600 viewport pane, and measured the rendered geometry in a real browser (Chromium via Playwright):

Scenario	Image rendered (W×H)	Fits 600px-tall pane?
Current code (baseline)	864 × 1151	❌ overflows (image bottom at 1209px)
Fix: pin image absolutely to padded box	864 × 498	✅ contained (top 58 → bottom 556)
Fix: definite minmax(0,1fr) tracks at both grid levels	864 × 498	✅ contained

864 = 900 − 18px×2 (panel width minus horizontal padding); 1151 ≈ 864 × (4000/3000) — i.e. the image is laid out at panel width and natural aspect, with zero vertical constraint. That is the "too large to display" symptom. (The repro page was temporary and has been removed; re-create it if you want to re-measure.)

---

Root cause (precise)

The fallback markup is three nested boxes:

#viewport (absolute, inset:0)                       <- definite size, good
└─ .fallback-stage.annotated-stage                  index.html:1011-1012
     position:absolute; inset:0                      (definite height ✔)
     display:grid; place-items:center; padding:58/18/44   index.html:490-495
   └─ .annotation-frame                              index.html:1013-1014
        width:100%; height:100%                       index.html:497-503
        display:grid; place-items:center               <-- second nested grid
      ├─ img.annotation-image                         index.html:1015-1018
      │    width:100%; height:100%; object-fit:contain  index.html:505-511
      └─ .label-layer (absolute, inset:0)

Two compounding CSS facts:

• place-items: center sets align-items: center, not stretch. Combined with the default grid-auto-rows: auto, the single grid row is content sized, not stretched to the container's definite height.
• A percentage height resolves against the grid area (the track), and an auto track is indefinite. So height: 100% on a child of a place-items: center grid computes as auto.

This happens twice:

1. .annotation-frame { height: 100% } inside .annotated-stage → the frame's track is auto → frame height becomes auto.
2. img.annotation-image { height: 100% } inside .annotation-frame → the image's track is auto → image height becomes auto → the <img> uses its intrinsic aspect ratio at the available width (864px) → 1151px tall.

Measured proof that the bug is the inner grid too: even after forcing .annotation-frame to a correct definite 498px box, the image still rendered 1151px, because its height:100% resolves against .annotation-frame's inner auto row, not the frame's element box. Both grid levels must be fixed (or the nested-grid sizing dropped entirely).

Note: .source-image in the side panel (index.html:384-390) does not have this bug because it has a fixed height: 180px — a definite height, so object-fit: contain works there. The viewport fallback wants to fill available space rather than a fixed height, which is why it hit the trap.

---

Fix options (both verified to contain the image at 864 × 498)

Pick one. Option A is recommended — fewest moving parts, no dependence on fragile nested-grid percentage resolution.

Option A (recommended): pin the image to the padded content box

Stop relying on height: 100% chaining through two grids. Make the image fill a single absolutely-positioned box and let object-fit: contain do the framing.

/* .annotated-stage can keep its background; drop the grid centering */
.annotation-frame {
  position: absolute;
  inset: 58px 18px 44px;     /* same as the old .annotated-stage padding */
  /* remove width/height:100% + display:grid + place-items:center */
}
.annotation-image {
  position: absolute;
  inset: 0;
  width: 100%;
  height: 100%;
  object-fit: contain;       /* now fits a definite box -> letterboxed, contained */
}

.label-layer already is position:absolute; inset:0 (index.html:466-471), so it keeps lining up with the frame. The 58/18/44 padding currently lives on .annotated-stage (index.html:493); move that inset onto .annotation-frame (or keep the padding on the stage and set .annotation-frame { position:absolute; inset:0 }). Either way the frame becomes a definite box and the image is contained.

Option B: keep the grids but make every track definite

If you prefer to keep display:grid centering, the breakage is the auto rows — give both grids definite tracks and let the items stretch:

.annotated-stage {
  display: grid;
  grid-template-columns: minmax(0, 1fr);
  grid-template-rows: minmax(0, 1fr);
  place-items: stretch;          /* not center: stretch the frame to fill */
  padding: 58px 18px 44px;
}
.annotation-frame {
  display: grid;
  grid-template-columns: minmax(0, 1fr);
  grid-template-rows: minmax(0, 1fr);
  min-width: 0;
  min-height: 0;
  width: auto;                   /* track sizing handles it; 100% no longer needed */
  height: auto;
}
.annotation-image { width: 100%; height: 100%; object-fit: contain; }

This also measured 864 × 498, contained. It is more brittle (relies on readers understanding the minmax(0,1fr) + min-height:0 idiom), hence Option A is preferred.

Whichever option: this is CSS-only. No change to buildAnnotatedPhoto's JS, the schema, the backend, or app.py is required for the sizing fix.

---

Annotation accuracy

A) Most misplacement is a symptom of the sizing bug — fixing sizing fixes it

updatePhotoLabels (index.html:1031-1062) positions each callout from:

• containedImageRect(image) (index.html:1096-1118) — computes the letterboxed sub-rect of the image from image.naturalWidth/Height and the <img> bounding box, then
• label.left = imageRect.left − layerRect.left + point[0] * imageRect.width (and the point[1] equivalent for top), clamped to the layer.

This math is correct only when the <img> box matches the visible image area. With the current sizing bug the <img> box is 864 × 1151 and overflows the viewport, so:

• imageRect.width/height come from the oversized box, so a point[1] = 0.8 callout is placed at ~0.8 × 1151 ≈ 920px down — far below the 600px viewport, which is overflow: hidden (.viewport-pane, index.html:105), so the label is clipped and the user never sees it.
• The clamp uses layerRect.height (also 1151), so the "keep on screen" clamp (index.html:1044-1047) clamps to the wrong, oversized bounds.

Once the image is contained (Option A/B), containedImageRect returns the real letterboxed rect and the existing placement math is accurate. So the sizing fix is also the primary annotation-accuracy fix. Re-verify after fixing.

B) Independent annotation-accuracy improvements (do after the sizing fix)

User decision (2026-06-14): in scope — implement sizing fix and this annotation-accuracy polish. Do items #1–#3 below; #4 (1px border) and #5 (coercion clamp) are optional robustness items, implement if cheap.

1. No marker at the actual point. The callout box is centered on the point via .scene-label { transform: translate(-50%, -50%) } (index.html:476), so the text occludes the component instead of pointing at it, and an edge-clamped label has nothing tying it back to the real location. Recommend drawing a small dot/crosshair at the exact point (a 1–2px element, not transformed) and offsetting the text label, optionally with a short leader line. Numbered prefixes already exist ((index + 1) + ". ", index.html:1051) but there's no matching number on the photo, so the number is currently meaningless spatially.

2. Edge clamp silently detaches the label from its point. Math.min(width − 72, Math.max(72, …)) (index.html:1040-1047) keeps the label on screen but, with no leader/marker (see #1), a clamped label no longer indicates which component it describes. The marker dot from #1 is the fix — keep the dot at the true (clamped-to-image) point and only clamp the text.

3. annotation.box is parsed and validated but never drawn. The schema and coercion carry an optional normalized box: [x, y, w, h] (snap2sim/schema.py:197-202, snap2sim/model_io.py:325-327), but buildAnnotatedPhoto ignores it. When present, drawing the bounding box over the contained image would be a much stronger, more accurate annotation than a single point. Optional enhancement.

4. 1px border offset. .annotation-image has a 1px border (index.html:509) and box-sizing: border-box is global (index.html:38-40), so object-fit: contain fits the content box (inside the border) while containedImageRect measures the border box — a 1px placement error. Negligible; mention only. If you want it exact, account for the border or use outline instead of border.

5. Coercion rejects (drops) out-of-range points instead of clamping. _unit_number_list returns None if any coordinate is outside [0,1] (snap2sim/model_io.py:311-315), so a point like [1.02, 0.5] discards the entire annotation and the part may then fall through to "unavailable". The browser already clamp01s at render (index.html:1120-1122), so consider clamping (not rejecting) server-side for resilience. Minor robustness item; the prompt already asks for [0,1] (snap2sim/prompts.py:31-34).

The point-coordinate contract itself is consistent end-to-end: normalized [0,1], origin top-left, asserted in the prompt (snap2sim/prompts.py:31-34) and consumed as point[0]→x*width, point[1]→y*height from the image's top-left (index.html:1040-1047). No change needed there.

---

Security / invariants to preserve

• This is a client-side CSS/JS change to an existing render path; keep the "no model-authored HTML" rule from AGENTS.md/SECURITY.md. label/note must stay textContent, never innerHTML (already correct, index.html:1051-1058). A new marker-dot/box element must be built with document.createElement + style, not injected markup.
• Keep using the local currentPreviewUrl object URL for the photo (index.html:1018, index.html:1134-1139); do not round-trip the image back from the server for display.
• No change to rate limiting, upload caps, or decompression-bomb guards in app.py is required.

---

Suggested implementation order for Codex

1. Sizing fix first (Option A). CSS-only in index.html. This is the bug the user reported and it also corrects most annotation placement.
2. Re-verify in a real browser with both a tall (portrait) and a wide (landscape) photo that triggers the annotate fallback: the whole image is visible and contained, no viewport overflow/scroll, and callouts land on the correct components within the image (especially lower-half points that were previously clipped). Check the ≤860px stacked layout (index.html:555-581).
3. Then the annotation-accuracy items (in scope per the user): marker dot at the true point + offset/leader label (#1, #2), and render annotation.box when present (#3). Items #4/#5 optional.
4. Run the standard local checks per AGENTS.md (schema/parser + FastAPI TestClient for /, /analyze_image, /generate_scene), then the normal PR -> GitHub Actions HF sync -> Space verification.

Resolved scope

• Sizing fix + annotation-accuracy polish are both in scope (user decision, 2026-06-14): the CSS sizing fix, marker dots at the true point, offset/leader labels, and rendering annotation.box when the model provides it.
• Final submission note, 2026-06-15: this work shipped before the public build-small-hackathon/Snap2Sim submission.