Deploy: 12-task vision extraction + fusion ZeroGPU showcase

README.md

@@ -1,13 +1,73 @@
 ---
-title: Qwen Caption Array
-emoji: 🌖
-colorFrom: red
+title: Qwen Runner Vision — 12-Task Extraction + Fusion
+emoji: 🧩
+colorFrom: indigo
 colorTo: blue
 sdk: gradio
-sdk_version: 6.20.0
-python_version: '3.12'
+sdk_version: 4.44.1
 app_file: app.py
 pinned: false
+license: apache-2.0
+short_description: Image → 12-task JSON + fused prompt on ZeroGPU
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Qwen Runner Vision — deterministic 12-task extraction + fusion
+
+**Stick an image in → get a full JSON readout.** This Space showcases the vision
+half of [`qwen-test-runner`](https://github.com/AbstractEyes/qwen-test-runner): a
+**deterministic-first** pipeline that replaces a hallucinating VLM with hand-picked
+Apache/MIT specialist models, one per task, then **fuses** everything into one
+relational scene and a byte-deterministic prompt.
+
+It runs the **batched extraction structure** on **ZeroGPU**.
+
+## What comes out
+
+Per image, the full readout:
+
+- **12 task JSONs** — 11 from the specialist/derive engine
+  (`image_classification`, `bbox_grounding`, `ocr_text`, `data_type_differentiation`,
+  `data_type_utilization`, `structural_spatial_awareness`, `depth_analysis`,
+  `subject_fixation`, `segmentation`, `outline_association`, `style_structural_awareness`)
+  plus `semantic_association` from the fusion tier — with a per-task **schema-validity** map.
+- **`FusedScene`** — entities (dedup + left-to-right ids), relations, the attribute
+  **ownership cascade** and the **shared basin** (uncertainty stored, never guessed),
+  a voted scene block, and a quality/accounting block.
+- **`prompt_fused`** — the deterministic natural-language prompt, plus `fusion_confidence`.
+- **Overlays** — detection boxes, SAM mask fills, subject box, outline, and a depth heatmap.
+- **Download** — one row in the production column shape
+  (`tasks_json`, `tasks_valid`, `fused_json`, `prompt_fused`, `fusion_confidence`,
+  `proc_width/height`, plus `struct_*`/`age_audit` when those toggles are on).
+
+## Everything is a toggle
+
+It's a *multiple-possibility system*: **structurer** (`off` / Qwen3.5-0.8B / Qwen3.5-9B for
+caption enrichment) · **tasks** (which to run/show) · **vocab** (COCO-80 / shapes / custom
+phrases) · **specialists** (OCR, SAM masks, depth on/off) · **detection** (box/text
+thresholds) · **fusion** (`t_own`, `t_margin`, `dedup_iou`, coord space) · **batch**
+(`extract_batch`, `gdino_batch`) · optional **age-gate** pre-filter.
+
+The core path needs no captions: the fusion attribute-ownership and shared-basin machinery
+light up when you supply captions and pick a structurer, but entities/relations/scene come
+from the image alone.
+
+## Model ledger (Apache-2.0 — redistributable)
+
+| role | checkpoint |
+|---|---|
+| detection (hub) | `IDEA-Research/grounding-dino-base` |
+| segmentation | `facebook/sam-vit-base` |
+| depth | `depth-anything/Depth-Anything-V2-Small-hf` |
+| classification / style | `google/siglip2-so400m-patch14-384` |
+| OCR | EasyOCR |
+| structurer (optional) | `Qwen/Qwen3.5-0.8B` · `Qwen/Qwen3.5-9B` |
+| age gate (optional) | `nateraw/vit-age-classifier` |
+
+## Hardware
+
+Select **ZeroGPU** in the Space's hardware settings. `large` (48 GB) is enough for the
+default config and the 0.8B structurer; the 9B structurer wants `xlarge` (96 GB). The
+fusion tier is CPU-only (torch-free) and runs off the GPU.
+
+Full-corpus production (streaming an HF dataset → published `{src}-fused` parquet shards)
+runs in Colab via `colab/produce_fused_dataset.py` — this Space is the interactive showcase.

app.py

@@ -0,0 +1,669 @@
+"""app.py — HuggingFace ZeroGPU Space: the deterministic 12-task vision
+extraction + fusion pipeline as an interactive showcase.
+
+Stick an image in → get the full JSON readout (12 task JSONs + FusedScene +
+deterministic fused prompt + overlays). Every possibility in the system is a
+selectable toggle.
+
+ZeroGPU teardown-friendly design
+--------------------------------
+* PYTORCH_CUDA_ALLOC_CONF is set BEFORE torch imports (OOM-probing batched path).
+* The always-on specialist models load ONCE at module level (CUDA-emulation
+  outside `@spaces.GPU`; real CUDA inside) — the efficient, fork-friendly residency.
+* Optional structurer (0.8B / 9B) + age gate load on demand, single-resident.
+* GPU functions return only picklable CPU data (task/digest dicts + rendered PIL
+  overlays). fuse()/fused_prompt()/build_semantic_association() run on the CPU in
+  the main process — no GPU is held during fusion.
+
+The pipeline modules themselves are the real `qwen_test_runner` package, vendored
+verbatim by ../build_space.py.
+"""
+from __future__ import annotations
+
+import os
+
+# ── ZeroGPU rule: set the allocator conf BEFORE torch is imported ────────────
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import json
+import tempfile
+import time
+
+import numpy as np
+from PIL import Image, ImageDraw
+
+import gradio as gr
+
+# spaces (ZeroGPU). Degrade to a no-op decorator when running off-platform.
+try:
+    import spaces
+
+    _HAS_SPACES = True
+except Exception:  # pragma: no cover - local/CPU dev
+    _HAS_SPACES = False
+
+    class _NoSpaces:
+        @staticmethod
+        def GPU(*_a, **_k):
+            def deco(fn):
+                return fn
+
+            return deco
+
+    spaces = _NoSpaces()  # type: ignore
+
+import torch
+
+# ── real pipeline (vendored package) ─────────────────────────────────────────
+import qwen_test_runner.vision.specialists_gpu as g
+from qwen_test_runner.vision.specialists import Solids
+from qwen_test_runner.vision import derive
+from qwen_test_runner.vision.fuse import (
+    solids_digest,
+    fuse,
+    phrases_for_grounding,
+    build_semantic_association,
+)
+from qwen_test_runner.vision.fuse_prompt import fused_prompt
+from qwen_test_runner.vision.tasks_vision import get_task, model_for
+from qwen_test_runner.vision.coords import CoordSpace
+from qwen_test_runner.model_runner import SYSTEM_PROMPT_JSON
+from qwen_test_runner.evaluator import parse_safely
+
+
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+IS_GPU_ENV = bool(os.environ.get("SPACES_ZERO_GPU")) or DEVICE == "cuda"
+MAX_DIM = 1024                      # match production DECODE_MAX_DIM
+BATCH_CAP = 24                      # interactive batch ceiling (ZeroGPU quota)
+
+# 12 deterministic tasks (11 from _build_tasks + semantic_association from fusion)
+DET_TASKS = [
+    "image_classification", "bbox_grounding", "ocr_text",
+    "data_type_differentiation", "data_type_utilization",
+    "structural_spatial_awareness", "depth_analysis", "subject_fixation",
+    "segmentation", "outline_association", "style_structural_awareness",
+    "semantic_association",
+]
+# registry entries with no deterministic builder (shown, disabled)
+VLM_TASKS = ["vit_accuracy_to_prompt", "geometric_3d_object_id", "camera_rotational_offset"]
+
+VOCABS = {"COCO-80": g.COCO_CLASSES, "shapes": g.SHAPE_CLASSES}
+STRUCTURERS = {"off": None, "Qwen3.5-0.8B": "Qwen/Qwen3.5-0.8B", "Qwen3.5-9B": "Qwen/Qwen3.5-9B"}
+COORD_SPACES = ["norm_0_1000", "norm_0_1", "pixel_abs"]
+
+_PALETTE = [
+    (239, 71, 111), (17, 138, 178), (6, 214, 160), (255, 209, 102),
+    (155, 93, 229), (241, 91, 181), (0, 187, 249), (254, 127, 45),
+]
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Model residency (teardown-friendly)
+# ═════════════════════════════════════════════════════════════════════════════
+
+_PIPE = None
+_OCR = None
+_AGE = None
+_STRUCT: dict = {}
+
+
+def get_pipe():
+    """The always-on specialist pipeline (GroundingDINO/SAM/Depth/SigLIP2[/OCR])."""
+    global _PIPE
+    if _PIPE is None:
+        with_ocr = os.environ.get("SPACE_WITH_OCR", "1") == "1"
+        _PIPE = g.SpecialistPipeline(device=DEVICE, with_ocr=with_ocr)
+    return _PIPE
+
+
+def _get_ocr(pipe):
+    """OCR reader — from the pipeline if it loaded there, else a lazy singleton
+    (the teardown-safe fallback when EasyOCR misbehaves at module level)."""
+    global _OCR
+    if pipe.ocr is not None:
+        return pipe.ocr
+    if _OCR is None:
+        _OCR = g.load_ocr(DEVICE)
+    return _OCR
+
+
+def _get_age_filter():
+    """Age-gate pre-filter — imported lazily (the module loads its model at import)."""
+    global _AGE
+    if _AGE is None:
+        import importlib
+
+        faf = importlib.import_module("face_age_filter")
+        _AGE = faf.FaceAgeFilter(decision_mode="strict", batch_size=32)
+    return _AGE
+
+
+class _Structurer:
+    """Caption→struct (slot-registry JSON), mirroring the production ModelPack."""
+
+    def __init__(self, model_id: str):
+        from transformers import AutoProcessor
+
+        try:
+            from transformers import AutoModelForMultimodalLM as _M
+        except ImportError:  # pragma: no cover
+            from transformers import AutoModelForImageTextToText as _M
+
+        self.proc = AutoProcessor.from_pretrained(model_id)
+        tok = getattr(self.proc, "tokenizer", self.proc)
+        tok.padding_side = "left"
+        if tok.pad_token_id is None:
+            tok.pad_token = tok.eos_token
+        self.pad_id = tok.pad_token_id
+        if DEVICE == "cuda":
+            dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
+            self.model = _M.from_pretrained(model_id, dtype=dtype, device_map="cuda").eval()
+        else:
+            self.model = _M.from_pretrained(model_id).to("cpu").eval()
+
+    @torch.no_grad()
+    def structure(self, captions: list, max_tok: int = 512) -> list:
+        msgs = [[{"role": "system", "content": SYSTEM_PROMPT_JSON},
+                 {"role": "user", "content": c}] for c in captions]
+        enc = self.proc.apply_chat_template(
+            msgs, add_generation_prompt=True, tokenize=True, return_dict=True,
+            return_tensors="pt", padding=True, enable_thinking=False).to(self.model.device)
+        n_in = enc["input_ids"].shape[1]
+        gen = self.model.generate(**enc, max_new_tokens=max_tok, do_sample=False,
+                                  pad_token_id=self.pad_id)
+        outs = [self.proc.decode(s, skip_special_tokens=True).strip() for s in gen[:, n_in:]]
+        structs = []
+        for raw in outs:
+            pr = parse_safely(raw)
+            if pr.schema_valid and pr.parsed is not None:
+                m = pr.parsed
+                structs.append(m.model_dump() if hasattr(m, "model_dump") else m.dict())
+            else:
+                structs.append(None)
+        return structs
+
+
+def _get_structurer(model_id: str):
+    if model_id in _STRUCT:
+        return _STRUCT[model_id]
+    _STRUCT.clear()                      # single-resident (evict on switch)
+    if DEVICE == "cuda":
+        torch.cuda.empty_cache()
+    _STRUCT[model_id] = _Structurer(model_id)
+    return _STRUCT[model_id]
+
+
+# Preload the always-on specialists at module level on a GPU/ZeroGPU env
+# (lazy on a CPU dev box so the module imports cheaply for tests).
+if IS_GPU_ENV:
+    try:
+        get_pipe()
+    except Exception as e:  # pragma: no cover
+        print(f"[app] specialist preload deferred: {type(e).__name__}: {e}")
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Solidify orchestration (public batched primitives + threshold / skip control)
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _resolve_vocab(vocab_choice: str, custom: str) -> list:
+    if vocab_choice == "custom":
+        toks = [t.strip() for t in (custom or "").split(",") if t.strip()]
+        return toks or g.COCO_CLASSES
+    return VOCABS.get(vocab_choice, g.COCO_CLASSES)
+
+
+def _solidify(pipe, images, vocab, phrases_list, box_thr, text_thr,
+              use_ocr, use_masks, use_depth, batch, gdino_batch) -> list:
+    """Mirror SpecialistPipeline.solidify_batch, but pass detection thresholds and
+    honour the specialist on/off toggles."""
+    images = list(images)
+    solids = []
+    ocr_reader = _get_ocr(pipe) if use_ocr else None
+    for start in range(0, len(images), batch):
+        chunk = images[start:start + batch]
+        p_chunk = phrases_list[start:start + batch] if phrases_list is not None else None
+
+        boxes_list = []
+        for s2 in range(0, len(chunk), gdino_batch):
+            boxes_list.extend(g.detect_batch(
+                pipe.gdino, chunk[s2:s2 + gdino_batch], vocab,
+                box_threshold=box_thr, text_threshold=text_thr, device=DEVICE))
+        if use_masks and pipe.sam is not None:
+            boxes_list = g.segment_batch(pipe.sam, chunk, boxes_list, device=DEVICE)
+        depths = (g.depth_map_batch(pipe.depth, chunk)
+                  if (use_depth and pipe.depth is not None) else [None] * len(chunk))
+        classes = (g.zero_shot_batch(pipe.siglip, chunk, vocab, device=DEVICE)
+                   if pipe.siglip is not None else [None] * len(chunk))
+        styles = (g.zero_shot_batch(pipe.siglip, chunk, g.STYLE_LABELS, device=DEVICE)
+                  if pipe.siglip is not None else [None] * len(chunk))
+        if p_chunk is not None and any(p_chunk):
+            attrs = []
+            for s2 in range(0, len(chunk), gdino_batch):
+                attrs.extend(g.ground_phrases_batch(
+                    pipe.gdino, chunk[s2:s2 + gdino_batch],
+                    p_chunk[s2:s2 + gdino_batch], device=DEVICE))
+        else:
+            attrs = [[] for _ in chunk]
+
+        for k, im in enumerate(chunk):
+            s = Solids(size=im.size)
+            s.boxes = boxes_list[k]
+            s.depth = depths[k]
+            s.gray = np.asarray(im.convert("L"), dtype=np.float32)
+            if classes[k] is not None:
+                s.class_top = classes[k][:5]
+                s.style = styles[k][0]["label"]
+            if ocr_reader is not None:
+                s.ocr = g.ocr_read(ocr_reader, im)
+            s.attr_boxes = attrs[k]
+            solids.append(s)
+    return solids
+
+
+def _solidify_oom(pipe, images, vocab, phrases_list, box_thr, text_thr,
+                  use_ocr, use_masks, use_depth, batch, gdino_batch) -> list:
+    """OOM-halving wrapper (mirrors produce_fused_dataset's guard)."""
+    solids, i, bs = [], 0, batch
+    while i < len(images):
+        chunk = images[i:i + bs]
+        p_chunk = phrases_list[i:i + bs] if phrases_list is not None else None
+        try:
+            solids.extend(_solidify(pipe, chunk, vocab, p_chunk, box_thr, text_thr,
+                                    use_ocr, use_masks, use_depth, bs, gdino_batch))
+            i += len(chunk)
+            bs = batch
+        except torch.cuda.OutOfMemoryError:  # pragma: no cover
+            torch.cuda.empty_cache()
+            if bs == 1:
+                solids.append(Solids(size=images[i].size))
+                i += 1
+                bs = batch
+            else:
+                bs = max(1, bs // 2)
+    return solids
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# GPU stage (everything that touches CUDA) — teardown-friendly
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _gpu_duration(images, *_a, **_k):
+    n = len(images) if images else 1
+    return int(min(240, 25 + 9 * n))
+
+
+@spaces.GPU(duration=_gpu_duration)
+def gpu_extract(images, vocab, box_thr, text_thr, use_ocr, use_masks, use_depth,
+                structurer_id, captions_list, use_age, batch, gdino_batch, render):
+    """All CUDA work in one allocation. Returns picklable CPU data:
+    per-image (tasks, digest, overlays) + caption structs + age audits + timing."""
+    pipe = get_pipe()
+    timing = {}
+    n = len(images)
+
+    audits = None
+    if use_age:
+        t = time.perf_counter()
+        audits = [r.to_audit() for r in _get_age_filter().check_batch(images)]
+        timing["age_s"] = round(time.perf_counter() - t, 3)
+
+    structs_rows = [{} for _ in images]
+    raws_rows = [{} for _ in images]
+    if structurer_id and any(captions_list or []):
+        t = time.perf_counter()
+        st = _get_structurer(structurer_id)
+        for idx, caps in enumerate(captions_list or []):
+            caps = [c for c in (caps or []) if c and str(c).strip()]
+            if not caps:
+                continue
+            got = st.structure(caps)
+            structs_rows[idx] = {f"cap_{j}": s for j, s in enumerate(got)}
+            raws_rows[idx] = {f"cap_{j}": c for j, c in enumerate(caps)}
+        timing["struct_s"] = round(time.perf_counter() - t, 3)
+
+    phrases_list = [phrases_for_grounding(sr) for sr in structs_rows]
+
+    t = time.perf_counter()
+    solids = _solidify_oom(pipe, images, vocab, phrases_list, box_thr, text_thr,
+                           use_ocr, use_masks, use_depth, batch, gdino_batch)
+    timing["extract_s"] = round(time.perf_counter() - t, 3)
+
+    results = []
+    for s, im in zip(solids, images):
+        tasks = g.SpecialistPipeline._build_tasks(s)      # CPU, torch-free, fast
+        digest = solids_digest(s)
+        overlays = _render_overlays(im, s) if render else None
+        results.append({"tasks": tasks, "digest": digest, "overlays": overlays})
+
+    if DEVICE == "cuda":
+        torch.cuda.empty_cache()
+    timing["n_images"] = n
+    return results, structs_rows, raws_rows, audits, timing
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# CPU fusion + assembly (no GPU held)
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _task_valid(task: str, pred) -> bool:
+    try:
+        m = model_for(get_task(task))
+        m.model_validate(pred) if hasattr(m, "model_validate") else m(**pred)
+        return True
+    except Exception:
+        return False
+
+
+def _assemble(results, structs_rows, raws_rows, audits, sizes,
+              t_own, t_margin, dedup_iou, coord_space, task_filter):
+    """Fuse each image's digest + structs → scene + prompt + one output row."""
+    rows = []
+    cs = CoordSpace(coord_space)
+    for i, r in enumerate(results):
+        tasks = dict(r["tasks"])
+        try:
+            scene = fuse(r["digest"], structs_rows[i] or {}, raws_rows[i] or {},
+                         t_own=t_own, t_margin=t_margin, dedup_iou=dedup_iou, coord_space=cs)
+            tasks["semantic_association"] = build_semantic_association(scene)
+            prompt = fused_prompt(scene)
+            conf = float(scene["quality"]["overall_confidence"])
+        except Exception as e:                          # pragma: no cover
+            scene, prompt, conf = {"__error__": f"{type(e).__name__}: {e}"}, "", 0.0
+        valid = {t: _task_valid(t, p) for t, p in tasks.items() if t != "__error__"}
+        shown = {t: tasks[t] for t in tasks if (not task_filter or t in task_filter)}
+        W, H = sizes[i]
+        rows.append({
+            "tasks_json": shown, "tasks_valid": valid, "fused_json": scene,
+            "prompt_fused": prompt, "fusion_confidence": round(conf, 4),
+            "struct": structs_rows[i] or {}, "age_audit": (audits[i] if audits else None),
+            "proc_width": W, "proc_height": H,
+            "overlays": r.get("overlays"),
+        })
+    return rows
+
+
+def _download_row(row: dict) -> str:
+    payload = {
+        "tasks_json": json.dumps(row["tasks_json"]),
+        "tasks_valid": json.dumps(row["tasks_valid"]),
+        "fused_json": json.dumps(row["fused_json"]),
+        "prompt_fused": row["prompt_fused"],
+        "fusion_confidence": row["fusion_confidence"],
+        "struct": json.dumps(row["struct"]),
+        "age_audit": json.dumps(row["age_audit"]),
+        "proc_width": row["proc_width"], "proc_height": row["proc_height"],
+    }
+    f = tempfile.NamedTemporaryFile("w", suffix=".json", delete=False, encoding="utf-8")
+    json.dump(payload, f, indent=2)
+    f.close()
+    return f.name
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Overlay rendering (from Solids, pixel space)
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _colorize_depth(depth: np.ndarray) -> Image.Image:
+    d = np.asarray(depth, dtype=np.float32)
+    lo, hi = float(d.min()), float(d.max())
+    n = (d - lo) / (hi - lo + 1e-6)                     # 0=far, 1=near
+    # 3-stop gradient far(indigo)→mid(teal)→near(amber)
+    stops = np.array([[40, 30, 90], [17, 138, 178], [255, 209, 102]], dtype=np.float32)
+    x = n * 2.0
+    lo_i = np.clip(np.floor(x).astype(int), 0, 1)
+    frac = (x - lo_i)[..., None]
+    rgb = (stops[lo_i] * (1 - frac) + stops[lo_i + 1] * frac).astype(np.uint8)
+    return Image.fromarray(rgb, "RGB")
+
+
+def _render_overlays(image: Image.Image, s: Solids) -> dict:
+    base = image.convert("RGB")
+    annotated = base.copy()
+    overlay = Image.new("RGBA", annotated.size, (0, 0, 0, 0))
+    od = ImageDraw.Draw(overlay)
+    dr = ImageDraw.Draw(annotated)
+
+    for i, b in enumerate(s.boxes):
+        color = _PALETTE[i % len(_PALETTE)]
+        x1, y1, x2, y2 = [int(v) for v in b["box"]]
+        mask = b.get("mask")
+        if mask is not None:
+            m = np.asarray(mask, dtype=bool)
+            fill = np.zeros((*m.shape, 4), dtype=np.uint8)
+            fill[m] = (*color, 90)
+            overlay.alpha_composite(Image.fromarray(fill, "RGBA"))
+        dr.rectangle([x1, y1, x2, y2], outline=color, width=3)
+        label = f'{b.get("label", "?")} {b.get("score", 0):.2f}'
+        dr.text((x1 + 3, max(0, y1 - 12)), label, fill=color)
+
+    annotated = Image.alpha_composite(annotated.convert("RGBA"), overlay).convert("RGB")
+    dr = ImageDraw.Draw(annotated)
+
+    # subject box (thick white)
+    subj = derive.subject_fixation(s.boxes, s.size).get("primary_subject", {})
+    if subj.get("box"):
+        x1, y1, x2, y2 = [int(v) for v in subj["box"]]
+        dr.rectangle([x1, y1, x2, y2], outline=(255, 255, 255), width=4)
+
+    # outline of the largest mask
+    masked = [b for b in s.boxes if b.get("mask") is not None]
+    if masked:
+        big = max(masked, key=lambda b: np.asarray(b["mask"]).sum())
+        poly = derive.outline_polygon(big["mask"], big["label"])["outline"]
+        if len(poly) >= 6:
+            pts = [(poly[j], poly[j + 1]) for j in range(0, len(poly) - 1, 2)]
+            dr.line(pts + [pts[0]], fill=(255, 0, 128), width=2)
+
+    depth_img = _colorize_depth(s.depth) if s.depth is not None else None
+    return {"annotated": annotated, "depth": depth_img}
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Gradio callbacks
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _prep(image) -> Image.Image:
+    im = image if isinstance(image, Image.Image) else Image.open(image)
+    im = im.convert("RGB")
+    if max(im.size) > MAX_DIM:
+        im.thumbnail((MAX_DIM, MAX_DIM))
+    return im
+
+
+def run_single(image, vocab_choice, custom_vocab, tasks_sel, use_ocr, use_masks,
+               use_depth, box_thr, text_thr, structurer_choice, captions_text,
+               use_age, t_own, t_margin, dedup_iou, coord_space):
+    if image is None:
+        raise gr.Error("Upload or pick an image first.")
+    im = _prep(image)
+    vocab = _resolve_vocab(vocab_choice, custom_vocab)
+    struct_id = STRUCTURERS.get(structurer_choice)
+    caps = [c.strip() for c in (captions_text or "").splitlines() if c.strip()]
+
+    results, structs_rows, raws_rows, audits, timing = gpu_extract(
+        [im], vocab, float(box_thr), float(text_thr), bool(use_ocr), bool(use_masks),
+        bool(use_depth), struct_id, [caps], bool(use_age), 1, 2, True)
+
+    row = _assemble(results, structs_rows, raws_rows, audits, [im.size],
+                    float(t_own), float(t_margin), float(dedup_iou), coord_space,
+                    set(tasks_sel or []))[0]
+    ov = row["overlays"] or {}
+    return (
+        ov.get("annotated"), ov.get("depth"),
+        row["prompt_fused"], row["fusion_confidence"],
+        row["tasks_json"], row["tasks_valid"], row["fused_json"],
+        row["struct"], (row["age_audit"] or {}), timing,
+        _download_row(row),
+    )
+
+
+def run_batch(files, vocab_choice, custom_vocab, use_ocr, use_masks, use_depth,
+              box_thr, text_thr, structurer_choice, use_age, t_own, t_margin,
+              dedup_iou, coord_space, batch, gdino_batch):
+    if not files:
+        raise gr.Error("Upload at least one image.")
+    files = files[:BATCH_CAP]
+    ims = [_prep(f) for f in files]
+    vocab = _resolve_vocab(vocab_choice, custom_vocab)
+    struct_id = STRUCTURERS.get(structurer_choice)
+
+    t0 = time.perf_counter()
+    results, structs_rows, raws_rows, audits, timing = gpu_extract(
+        ims, vocab, float(box_thr), float(text_thr), bool(use_ocr), bool(use_masks),
+        bool(use_depth), struct_id, [[] for _ in ims], bool(use_age),
+        int(batch), int(gdino_batch), False)
+    rows = _assemble(results, structs_rows, raws_rows, audits, [im.size for im in ims],
+                     float(t_own), float(t_margin), float(dedup_iou), coord_space, None)
+    wall = time.perf_counter() - t0
+
+    table, jsonl = [], []
+    for i, row in enumerate(rows):
+        cls = row["tasks_json"].get("image_classification", {}) if row["tasks_json"] else {}
+        n_ent = len(row["fused_json"].get("entities", [])) if isinstance(row["fused_json"], dict) else 0
+        nvalid = sum(1 for v in row["tasks_valid"].values() if v)
+        table.append([i, cls.get("label", ""), n_ent, row["fusion_confidence"],
+                      f"{nvalid}/{len(row['tasks_valid'])}", (row["prompt_fused"] or "")[:90]])
+        jsonl.append({
+            "idx": i, "tasks_json": json.dumps(row["tasks_json"]),
+            "tasks_valid": json.dumps(row["tasks_valid"]),
+            "fused_json": json.dumps(row["fused_json"]),
+            "prompt_fused": row["prompt_fused"], "fusion_confidence": row["fusion_confidence"],
+            "proc_width": row["proc_width"], "proc_height": row["proc_height"],
+        })
+
+    f = tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False, encoding="utf-8")
+    for r in jsonl:
+        f.write(json.dumps(r) + "\n")
+    f.close()
+
+    summary = {
+        "images": len(ims), "wall_s": round(wall, 2),
+        "img_per_s": round(len(ims) / max(0.001, wall), 2),
+        **{k: v for k, v in timing.items() if k != "n_images"},
+    }
+    return table, summary, f.name
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# UI
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _controls():
+    """Shared control widgets — returned so both tabs can wire them."""
+    vocab_choice = gr.Radio(list(VOCABS) + ["custom"], value="COCO-80", label="Detection vocab")
+    custom_vocab = gr.Textbox(label="Custom phrases (comma-separated)", visible=False,
+                              placeholder="person, red circle, laptop")
+    with gr.Row():
+        use_ocr = gr.Checkbox(True, label="OCR")
+        use_masks = gr.Checkbox(True, label="SAM masks")
+        use_depth = gr.Checkbox(True, label="Depth")
+    with gr.Row():
+        box_thr = gr.Slider(0.05, 0.6, 0.30, step=0.01, label="box threshold")
+        text_thr = gr.Slider(0.05, 0.6, 0.25, step=0.01, label="text threshold")
+    structurer = gr.Radio(list(STRUCTURERS), value="off", label="Caption structurer")
+    with gr.Row():
+        t_own = gr.Slider(0.0, 1.0, 0.60, step=0.01, label="t_own")
+        t_margin = gr.Slider(0.0, 1.0, 0.25, step=0.01, label="t_margin")
+        dedup_iou = gr.Slider(0.0, 1.0, 0.75, step=0.01, label="dedup_iou")
+    coord_space = gr.Radio(COORD_SPACES, value="norm_0_1000", label="Fused-scene coord space")
+    use_age = gr.Checkbox(False, label="Age-gate pre-filter (nateraw/vit-age-classifier)")
+
+    vocab_choice.change(lambda c: gr.update(visible=(c == "custom")),
+                        vocab_choice, custom_vocab)
+    return (vocab_choice, custom_vocab, use_ocr, use_masks, use_depth, box_thr,
+            text_thr, structurer, coord_space, use_age, t_own, t_margin, dedup_iou)
+
+
+with gr.Blocks(title="Qwen Runner Vision — 12-task extraction + fusion") as demo:
+    gr.Markdown(
+        "# 🧩 Qwen Runner Vision\n"
+        "Deterministic **12-task** extraction + **fusion** — stick an image in, get the "
+        "full JSON readout (task JSONs + `FusedScene` + fused prompt). Specialists run on "
+        "**ZeroGPU**; fusion is CPU-only. Every option below is a live toggle."
+    )
+
+    with gr.Tab("Single image"):
+        with gr.Row():
+            with gr.Column(scale=1):
+                img_in = gr.Image(type="pil", label="Image", height=320)
+                tasks_sel = gr.CheckboxGroup(DET_TASKS, value=DET_TASKS,
+                                             label="Tasks to show (all 12 always computed)")
+                gr.CheckboxGroup(VLM_TASKS, label="VLM/DEFER (no deterministic builder)",
+                                 interactive=False)
+                captions = gr.Textbox(lines=3, label="Captions (one per line — enrich fusion)",
+                                      placeholder="a woman with long red hair in a blue coat")
+                with gr.Accordion("Settings", open=False):
+                    ctl = _controls()
+                run_b = gr.Button("Extract", variant="primary")
+            with gr.Column(scale=1):
+                with gr.Row():
+                    annotated = gr.Image(label="Detections · masks · subject · outline", height=280)
+                    depth_img = gr.Image(label="Depth (near → far)", height=280)
+                prompt_out = gr.Textbox(label="Fused prompt", lines=3)
+                conf_out = gr.Number(label="Fusion confidence")
+                dl = gr.File(label="Download row (JSON)")
+        with gr.Accordion("Full JSON readout", open=True):
+            tasks_out = gr.JSON(label="tasks_json (12 tasks)")
+            with gr.Row():
+                valid_out = gr.JSON(label="tasks_valid")
+                struct_out = gr.JSON(label="caption structs")
+            fused_out = gr.JSON(label="fused_json (FusedScene)")
+            with gr.Row():
+                age_out = gr.JSON(label="age_audit")
+                timing_out = gr.JSON(label="timing")
+
+        (vocab_choice, custom_vocab, use_ocr, use_masks, use_depth, box_thr,
+         text_thr, structurer, coord_space, use_age, t_own, t_margin, dedup_iou) = ctl
+
+        ex_dir = os.path.join(os.path.dirname(__file__), "examples")
+        if os.path.isdir(ex_dir):
+            ex_imgs = [[os.path.join(ex_dir, f)] for f in sorted(os.listdir(ex_dir))
+                       if f.lower().endswith((".png", ".jpg", ".jpeg"))]
+            if ex_imgs:
+                gr.Examples(ex_imgs, inputs=img_in, label="Examples")
+
+        run_b.click(
+            run_single,
+            inputs=[img_in, vocab_choice, custom_vocab, tasks_sel, use_ocr, use_masks,
+                    use_depth, box_thr, text_thr, structurer, captions, use_age,
+                    t_own, t_margin, dedup_iou, coord_space],
+            outputs=[annotated, depth_img, prompt_out, conf_out, tasks_out, valid_out,
+                     fused_out, struct_out, age_out, timing_out, dl],
+        )
+
+    with gr.Tab("Batch (the batched structure)"):
+        gr.Markdown(
+            f"Upload up to **{BATCH_CAP}** images → batched `solidify_batch` on ZeroGPU "
+            "(`gdino_batch=2`, GDINO anti-scales) + per-image CPU fusion. Throughput mirrors "
+            "`runs/extract_throughput_results.md`."
+        )
+        with gr.Row():
+            with gr.Column(scale=1):
+                files_in = gr.Files(label="Images", file_types=["image"])
+                with gr.Accordion("Settings", open=False):
+                    bctl = _controls()
+                with gr.Row():
+                    batch_sl = gr.Slider(1, 24, 16, step=1, label="extract_batch")
+                    gdino_sl = gr.Slider(1, 8, 2, step=1, label="gdino_batch (keep ~2)")
+                run_batch_b = gr.Button("Run batch", variant="primary")
+            with gr.Column(scale=1):
+                batch_table = gr.Dataframe(
+                    headers=["#", "label", "entities", "fusion_conf", "valid", "prompt…"],
+                    label="Per-image results", wrap=True)
+                batch_summary = gr.JSON(label="Throughput")
+                batch_dl = gr.File(label="Download rows (JSONL)")
+
+        (b_vocab, b_custom, b_ocr, b_masks, b_depth, b_box, b_text, b_struct,
+         b_coord, b_age, b_town, b_tmargin, b_dedup) = bctl
+
+        run_batch_b.click(
+            run_batch,
+            inputs=[files_in, b_vocab, b_custom, b_ocr, b_masks, b_depth, b_box, b_text,
+                    b_struct, b_age, b_town, b_tmargin, b_dedup, b_coord, batch_sl, gdino_sl],
+            outputs=[batch_table, batch_summary, batch_dl],
+        )
+
+
+if __name__ == "__main__":
+    demo.queue().launch()

face_age_filter.py

@@ -0,0 +1,312 @@
+# ─────────────────────────────────────────────────────────────────────────────
+# face_age_filter.py — age-classification pre-filter (no face detection deps).
+#
+# This rewrite drops facenet-pytorch / MTCNN entirely — they pull torchvision
+# which collides with Colab's current Pillow (the classic "_util.is_directory"
+# ImportError). The project CLAUDE.md flags this exact failure mode.
+#
+# Strategy:
+#   - The age classifier (HF nateraw/vit-age-classifier) runs on PIL images
+#     directly. For datasets where the image IS a centered face (FFHQ) or
+#     where face bbox coords are provided (IMDB has `rect` in its CSV), no
+#     face detector is needed.
+#   - For deepfashion (face position unknown, possibly cropped out) we'll add
+#     a lightweight detector later — a separate concern.
+#
+# Threshold logic unchanged from the previous draft:
+#   reject if expected age < 24  OR  P(0-2)+P(3-9)+P(10-19) > 0.20
+#
+# Paste this cell ONCE per Colab session, after super_dataset_lib.py.
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# 1. ENSURE DEPS (no force-upgrades — Colab's stock transformers/torch/PIL
+#    are kept untouched to avoid the torchvision↔Pillow ImportError chain
+#    documented in the project CLAUDE.md).
+# ═════════════════════════════════════════════════════════════════════════════
+
+import importlib, subprocess, sys
+
+def _ensure(pkg_spec: str, import_name: str | None = None):
+    name = import_name or pkg_spec.split(">=")[0].split("==")[0].split("<")[0]
+    try:
+        importlib.import_module(name)
+    except ImportError:
+        print(f"  installing missing dep: {pkg_spec}")
+        subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pkg_spec])
+
+_ensure("transformers")
+_ensure("torch")
+print("face_age_filter deps OK (no force-upgrades).")
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# 2. IMPORTS  +  MODEL CONFIG
+# ═════════════════════════════════════════════════════════════════════════════
+
+from dataclasses import dataclass
+from typing import Optional
+
+import numpy as np
+import torch
+from PIL import Image as _PILImage
+from transformers import AutoImageProcessor, AutoModelForImageClassification
+
+AGE_MODEL_ID    = "nateraw/vit-age-classifier"
+AGE_THRESHOLD   = 24.0
+MINOR_MASS_MAX  = 0.20
+
+# Device selection.
+#   DEVICE_OVERRIDE = None   → auto-detect, GPU-test-then-fallback (default)
+#   DEVICE_OVERRIDE = "cuda" → force GPU (ignore warnings)
+#   DEVICE_OVERRIDE = "cpu"  → force CPU (~10× slower but always works)
+#
+# Auto-detect catches the case where the installed PyTorch's bundled CUDA
+# kernels don't include your GPU's compute capability (e.g. stock Colab torch
+# topping out at sm_90 vs an RTX 6000 Blackwell at sm_120). We detect by
+# running a tiny model forward; if it crashes, fall back to CPU.
+DEVICE_OVERRIDE = None
+
+
+def _select_device() -> str:
+    if DEVICE_OVERRIDE in ("cpu", "cuda"):
+        return DEVICE_OVERRIDE
+    if not torch.cuda.is_available():
+        return "cpu"
+    # Check that the GPU's capability is in torch's compiled-for list.
+    try:
+        cap_major, cap_minor = torch.cuda.get_device_capability(0)
+        my_sm = f"sm_{cap_major}{cap_minor}"
+        # Some torch builds expose get_arch_list, some don't.
+        arch_list = getattr(torch.cuda, "get_arch_list", lambda: [])()
+        # arch_list entries look like "sm_80" / "compute_80"; normalize.
+        compiled_sm = {a.replace("compute_", "sm_") for a in arch_list}
+        if compiled_sm and my_sm not in compiled_sm:
+            print(f"  GPU is {my_sm} but PyTorch was compiled for {sorted(compiled_sm)}.")
+            print(f"  Trying GPU anyway — if forward fails we'll fall back to CPU.")
+    except Exception:
+        pass
+    return "cuda"
+
+
+DEVICE = _select_device()
+
+# Bucket → midpoint mapping. Multiplied by per-bucket probability to get a
+# continuous expected age estimate.
+AGE_BUCKETS = [
+    ("0-2",         1.0),
+    ("3-9",         6.0),
+    ("10-19",      14.0),
+    ("20-29",      24.0),
+    ("30-39",      34.0),
+    ("40-49",      44.0),
+    ("50-59",      54.0),
+    ("60-69",      64.0),
+    ("more than 70", 75.0),
+]
+MINOR_BUCKETS = {"0-2", "3-9", "10-19"}
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# 3. MODEL LOAD  (singleton)
+# ═════════════════════════════════════════════════════════════════════════════
+
+print(f"Loading age classifier {AGE_MODEL_ID} ({DEVICE}) …")
+# Fast (Rust-backed) image preprocessing is the default in current transformers;
+# passing use_fast= now deprecation-warns, so we pass nothing.
+_AGE_PROCESSOR = AutoImageProcessor.from_pretrained(AGE_MODEL_ID)
+_AGE_MODEL     = AutoModelForImageClassification.from_pretrained(AGE_MODEL_ID).to(DEVICE).eval()
+
+_MODEL_LABELS = [_AGE_MODEL.config.id2label[i] for i in range(_AGE_MODEL.config.num_labels)]
+_LABEL_TO_MIDPOINT = dict(AGE_BUCKETS)
+
+_missing = [lbl for lbl, _ in AGE_BUCKETS if lbl not in _MODEL_LABELS]
+if _missing:
+    print(f"  WARNING — model labels don't include AGE_BUCKETS entries: {_missing}")
+    print(f"  model labels: {_MODEL_LABELS}")
+
+# GPU smoke test: run a tiny zero-tensor forward to confirm the GPU kernels
+# actually execute on this device. If PyTorch was compiled without our SM
+# version (Blackwell sm_120 on stock Colab torch) this fails immediately
+# rather than crashing mid-ingest.
+if DEVICE == "cuda":
+    try:
+        with torch.no_grad():
+            _test_in = torch.zeros(1, 3, 224, 224, device=DEVICE)
+            _ = _AGE_MODEL(_test_in)
+        print(f"  GPU smoke test passed.  VRAM: {torch.cuda.memory_allocated()/1024**3:.2f} GB")
+    except RuntimeError as e:
+        msg = str(e).splitlines()[0]
+        print(f"  GPU smoke test FAILED ({msg!r}) — falling back to CPU.")
+        DEVICE = "cpu"
+        _AGE_MODEL = _AGE_MODEL.to(DEVICE)
+        print(f"  age model relocated to CPU.")
+else:
+    print(f"  running on CPU (slower, but compatible).")
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# 4. RESULT TYPE
+# ═════════════════════════════════════════════════════════════════════════════
+
+@dataclass
+class FaceCheckResult:
+    """Outcome of running the age filter on ONE image."""
+    decision: str                # "pass" | "fail"
+    expected_age: float          # continuous age estimate
+    minor_mass: float            # P(0-2)+P(3-9)+P(10-19)
+    most_likely_bucket: str      # argmax bucket label
+    most_likely_prob: float      # probability of argmax bucket
+    reasons: list                # human-readable reasons for fail
+
+    def to_audit(self) -> dict:
+        return {
+            "decision":          self.decision,
+            "expected_age":      round(self.expected_age, 1),
+            "minor_mass":        round(self.minor_mass, 3),
+            "most_likely":       f"{self.most_likely_bucket} ({self.most_likely_prob:.2f})",
+            "reasons":           self.reasons,
+        }
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# 5. FaceAgeFilter — age-classifier-only variant
+# ═════════════════════════════════════════════════════════════════════════════
+
+class FaceAgeFilter:
+    """Runs the age classifier over images (or pre-cropped face regions).
+
+    Entry points:
+      .check_one(pil, bbox=None)        — single image (optional face bbox crop)
+      .check_batch(pils, bboxes=None)   — N images, batched on GPU
+
+    `bbox` (if provided) is an (x1, y1, x2, y2) tuple in pixel coords —
+    the image is cropped to that region before classification. Useful for
+    IMDB where the CSV provides face bbox coords. For FFHQ leave bbox=None
+    and the whole image is classified (each FFHQ image is a centered face crop).
+
+    decision_mode controls how strict the reject rule is:
+      "strict"   — fail if expected_age < age_threshold OR minor_mass > minor_mass_max
+                   (catches every borderline; gives ~30-40% reject rate on FFHQ)
+      "balanced" — fail only if most_likely bucket is a minor bucket OR minor_mass > 0.40
+                   (single-bucket-argmax + relaxed mass; ~10-20% reject rate)
+      "loose"    — fail only if most_likely bucket is a minor bucket
+                   (most permissive; only rejects model-confident minors)
+    """
+
+    def __init__(self,
+                 age_threshold: float = AGE_THRESHOLD,
+                 minor_mass_max: float = MINOR_MASS_MAX,
+                 decision_mode: str = "strict",       # "strict" | "balanced" | "loose"
+                 batch_size: int = 32):
+        assert decision_mode in ("strict", "balanced", "loose")
+        self.age_threshold = age_threshold
+        self.minor_mass_max = minor_mass_max
+        self.decision_mode = decision_mode
+        self.batch_size = batch_size
+
+    # ── core ────────────────────────────────────────────────────────────────
+
+    def _prep_one(self, img: _PILImage.Image,
+                  bbox: Optional[tuple] = None) -> _PILImage.Image:
+        if img.mode != "RGB":
+            img = img.convert("RGB")
+        if bbox is not None:
+            x1, y1, x2, y2 = bbox
+            W, H = img.size
+            x1, y1 = max(0, int(x1)), max(0, int(y1))
+            x2, y2 = min(W, int(x2)), min(H, int(y2))
+            if x2 > x1 and y2 > y1:
+                img = img.crop((x1, y1, x2, y2))
+        return img
+
+    def _classify_batch(self, crops: list) -> tuple:
+        """Returns (expected_ages, minor_masses, most_likely_buckets, most_likely_probs)
+        per crop. Each is a list aligned with `crops`."""
+        if not crops:
+            return [], [], [], []
+        inputs = _AGE_PROCESSOR(images=crops, return_tensors="pt").to(DEVICE)
+        with torch.no_grad():
+            logits = _AGE_MODEL(**inputs).logits
+        probs = torch.softmax(logits, dim=-1).cpu().numpy()
+        expected_ages, minor_masses = [], []
+        most_likely_buckets, most_likely_probs = [], []
+        for row in probs:
+            exp_age, minor_mass = 0.0, 0.0
+            for i, label in enumerate(_MODEL_LABELS):
+                p = float(row[i])
+                exp_age += p * _LABEL_TO_MIDPOINT.get(label, 0.0)
+                if label in MINOR_BUCKETS:
+                    minor_mass += p
+            expected_ages.append(exp_age)
+            minor_masses.append(minor_mass)
+            mli = int(row.argmax())
+            most_likely_buckets.append(_MODEL_LABELS[mli])
+            most_likely_probs.append(float(row[mli]))
+        return expected_ages, minor_masses, most_likely_buckets, most_likely_probs
+
+    def _decide(self, exp_age: float, minor_mass: float,
+                most_likely_bucket: str, most_likely_prob: float) -> tuple:
+        reasons = []
+        mode = self.decision_mode
+        if mode == "strict":
+            if exp_age < self.age_threshold:
+                reasons.append(f"expected_age={exp_age:.1f} < {self.age_threshold}")
+            if minor_mass > self.minor_mass_max:
+                reasons.append(f"minor_mass={minor_mass:.2f} > {self.minor_mass_max}")
+        elif mode == "balanced":
+            if most_likely_bucket in MINOR_BUCKETS:
+                reasons.append(f"most_likely={most_likely_bucket} ({most_likely_prob:.2f}) is minor bucket")
+            elif minor_mass > 0.40:
+                reasons.append(f"minor_mass={minor_mass:.2f} > 0.40")
+        elif mode == "loose":
+            if most_likely_bucket in MINOR_BUCKETS:
+                reasons.append(f"most_likely={most_likely_bucket} ({most_likely_prob:.2f}) is minor bucket")
+        return (("fail", reasons) if reasons else ("pass", []))
+
+    # ── public ──────────────────────────────────────────────────────────────
+
+    def check_one(self, img: _PILImage.Image,
+                  bbox: Optional[tuple] = None) -> FaceCheckResult:
+        prepped = self._prep_one(img, bbox)
+        ea, mm, mlb, mlp = self._classify_batch([prepped])
+        decision, reasons = self._decide(ea[0], mm[0], mlb[0], mlp[0])
+        return FaceCheckResult(
+            decision=decision, expected_age=ea[0], minor_mass=mm[0],
+            most_likely_bucket=mlb[0], most_likely_prob=mlp[0],
+            reasons=reasons,
+        )
+
+    def check_batch(self, images: list,
+                    bboxes: Optional[list] = None) -> list:
+        """Process N images. `bboxes`, if given, must have same length as `images`
+        (use None for items where no crop should happen)."""
+        if not images:
+            return []
+        if bboxes is None:
+            bboxes = [None] * len(images)
+        assert len(bboxes) == len(images), "bboxes and images must align"
+
+        prepped = [self._prep_one(im, bb) for im, bb in zip(images, bboxes)]
+
+        all_exp, all_mm, all_mlb, all_mlp = [], [], [], []
+        bs = self.batch_size
+        for start in range(0, len(prepped), bs):
+            ea, mm, mlb, mlp = self._classify_batch(prepped[start:start + bs])
+            all_exp.extend(ea); all_mm.extend(mm)
+            all_mlb.extend(mlb); all_mlp.extend(mlp)
+
+        results = []
+        for ea, mm, mlb, mlp in zip(all_exp, all_mm, all_mlb, all_mlp):
+            decision, reasons = self._decide(ea, mm, mlb, mlp)
+            results.append(FaceCheckResult(
+                decision=decision, expected_age=ea, minor_mass=mm,
+                most_likely_bucket=mlb, most_likely_prob=mlp,
+                reasons=reasons,
+            ))
+        return results
+
+
+print(f"face_age_filter loaded. threshold={AGE_THRESHOLD}, "
+      f"minor_mass_max={MINOR_MASS_MAX}, batch={32}")

qwen_test_runner/__init__.py

@@ -0,0 +1,89 @@
+"""qwen_test_runner — testbed for evaluating small Qwen models on JSON schema.
+
+Public API:
+    from qwen_test_runner import Caption, QwenRunner, score_sample, score_run
+    from qwen_test_runner import SLOT_REGISTRY, SlotSpec   # v0.2 registry
+
+CLI:
+    qwen-bench --help
+"""
+
+from __future__ import annotations
+
+__version__ = "0.2.0"
+
+# Registry — single source of truth for all slot definitions
+from .registry import (
+    SLOT_REGISTRY,
+    SlotSpec,
+    SubjectValue,
+    Category,
+    Cardinality,
+    Vocabulary,
+    Groundedness,
+    slots_by_category,
+    slot_names,
+    get_slot,
+    all_closed_vocab,
+)
+
+# Schema — generated from registry at import time
+from .schema import (
+    Caption,
+    Subject,                     # alias for SubjectValue, kept for back-compat
+    CAPTION_JSON_SCHEMA,
+    CAPTION_GRAMMAR_GBNF,
+    build_gbnf_grammar,
+)
+
+# Evaluation — scoring functions
+from .evaluator import (
+    parse_safely,
+    ground_check,
+    coverage_check,
+    score_sample,
+    score_run,
+    SampleResult,
+    RunMetrics,
+    GroundingReport,
+    CoverageReport,
+)
+
+# Eval data
+from .eval_set import BUILTIN_CAPTIONS, load_eval_set
+
+
+# Model runner — imported lazily so `import qwen_test_runner` doesn't drag in
+# torch unless the user actually needs it. The names are still importable as
+# `from qwen_test_runner import QwenRunner` thanks to __getattr__.
+def __getattr__(name: str):
+    if name == "QwenRunner":
+        from .model_runner import QwenRunner
+        return QwenRunner
+    if name == "GenResult":
+        from .model_runner import GenResult
+        return GenResult
+    if name == "ClaudeProvider":
+        from .providers.claude_api import ClaudeProvider
+        return ClaudeProvider
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = [
+    "__version__",
+    # registry
+    "SLOT_REGISTRY", "SlotSpec", "SubjectValue",
+    "Category", "Cardinality", "Vocabulary", "Groundedness",
+    "slots_by_category", "slot_names", "get_slot", "all_closed_vocab",
+    # schema (generated)
+    "Caption", "Subject",
+    "CAPTION_JSON_SCHEMA", "CAPTION_GRAMMAR_GBNF", "build_gbnf_grammar",
+    # evaluator
+    "parse_safely", "ground_check", "coverage_check",
+    "score_sample", "score_run",
+    "SampleResult", "RunMetrics", "GroundingReport", "CoverageReport",
+    # eval data
+    "BUILTIN_CAPTIONS", "load_eval_set",
+    # runners (lazy)
+    "QwenRunner", "GenResult", "ClaudeProvider",
+]

qwen_test_runner/data_gen.py

@@ -0,0 +1,199 @@
+"""
+data_gen.py — Generate SFT-ready caption→schema training data.
+
+Pipeline:
+  1. Load source captions from a file (one per line) or the builtin eval set.
+  2. Pass each through a provider (Claude by default) to produce structured JSON.
+  3. Score each result against the registry's grounding rules.
+  4. Filter: keep only rows where grounding_rate == 1.0 (no hallucinations).
+  5. Write one JSONL row per kept sample, in the OpenAI-chat format that
+     trl.SFTTrainer accepts directly.
+
+The "filter on grounding" step is essential: Claude is excellent but not
+perfect, and we don't want Claude's stray hallucinations leaking into the
+Qwen training set. Roughly 30-50% rejection is normal on diverse inputs;
+that's a feature, not a bug.
+
+Usage:
+    qwen-datagen --source captions.txt --output train.jsonl --n 1000
+    qwen-datagen --source builtin --prompt strict
+    qwen-datagen --source captions.txt --provider claude --model claude-haiku-4-5
+"""
+
+from __future__ import annotations
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+from typing import Iterable, Optional
+
+from .registry import SLOT_REGISTRY
+from .schema import CAPTION_JSON_SCHEMA
+from .evaluator import score_sample
+from .eval_set import load_eval_set
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# SFT row formatting — OpenAI chat format consumed directly by trl.SFTTrainer.
+# Single system + single user + single assistant. Assistant emits raw JSON.
+# ──────────────────────────────────────────────────────────────────────────────
+
+SFT_SYSTEM_PROMPT = """You are a caption-structuring assistant. Convert each
+image caption into JSON matching the schema. Only include subjects, attributes,
+and actions explicitly mentioned in the caption. Use null/[] for unspecified
+fields.""".strip()
+
+
+def make_sft_row(caption: str, structured_json: str) -> dict:
+    """Build one SFTTrainer-compatible row."""
+    return {
+        "messages": [
+            {"role": "system", "content": SFT_SYSTEM_PROMPT},
+            {"role": "user", "content": caption},
+            {"role": "assistant", "content": structured_json},
+        ]
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Source loaders
+# ──────────────────────────────────────────────────────────────────────────────
+
+def load_captions(source: str, limit: Optional[int] = None) -> list[str]:
+    """Load captions from `builtin`, a .txt (one per line), or a .json (list)."""
+    captions = load_eval_set(source)  # `load_eval_set` already handles all three
+    if limit is not None:
+        captions = captions[:limit]
+    return captions
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Generation loop
+# ──────────────────────────────────────────────────────────────────────────────
+
+def generate_dataset(
+    captions: list[str],
+    provider,
+    prompt: str = "strict",
+    grounding_threshold: float = 1.0,
+    on_progress=None,
+) -> tuple[list[dict], dict]:
+    """Run captions through the provider, filter on grounding, return SFT rows + stats.
+
+    Returns:
+        (rows, stats)
+        rows  — list of SFT-format dicts (ready to json.dump line-by-line)
+        stats — {"total", "kept", "rejected_halluc", "rejected_invalid", "total_cost_usd"}
+    """
+    rows: list[dict] = []
+    stats = {"total": 0, "kept": 0, "rejected_halluc": 0,
+             "rejected_invalid": 0, "total_cost_usd": 0.0}
+
+    for i, cap in enumerate(captions):
+        stats["total"] += 1
+        try:
+            result = provider.process(cap, prompt=prompt)
+        except Exception as e:
+            stats["rejected_invalid"] += 1
+            if on_progress:
+                on_progress(i, cap, status=f"provider error: {e}")
+            continue
+
+        stats["total_cost_usd"] += result.cost_usd
+        scored = score_sample(cap, result.raw_text, mode=result.mode,
+                              n_input_tokens=result.n_input_tokens,
+                              n_output_tokens=result.n_output_tokens)
+
+        if not scored.schema_valid:
+            stats["rejected_invalid"] += 1
+            if on_progress:
+                on_progress(i, cap, status=f"invalid: {scored.parse_error}",
+                            cost=result.cost_usd)
+            continue
+
+        if scored.grounding_rate < grounding_threshold:
+            stats["rejected_halluc"] += 1
+            if on_progress:
+                on_progress(i, cap, status=f"halluc: {scored.hallucinations}",
+                            cost=result.cost_usd)
+            continue
+
+        rows.append(make_sft_row(cap, result.raw_text))
+        stats["kept"] += 1
+        if on_progress:
+            on_progress(i, cap, status="kept", cost=result.cost_usd)
+
+    return rows, stats
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# CLI
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _print_progress(i: int, caption: str, status: str, cost: float = 0.0):
+    short = caption[:60] + ("…" if len(caption) > 60 else "")
+    cost_str = f" ${cost:.4f}" if cost else ""
+    print(f"  [{i + 1:4d}] {status[:30]:30s}{cost_str}  → {short}")
+
+
+def main(argv: Optional[list[str]] = None) -> int:
+    p = argparse.ArgumentParser(description="Generate SFT-ready caption→schema dataset.")
+    p.add_argument("--source", default="builtin",
+                   help="builtin | path to .txt (one per line) | path to .json (list)")
+    p.add_argument("--output", default="train.jsonl",
+                   help="output JSONL file (overwritten if exists)")
+    p.add_argument("--n", type=int, default=None,
+                   help="cap captions to this many (default: all)")
+    p.add_argument("--provider", choices=["claude"], default="claude",
+                   help="backend to use (more added later)")
+    p.add_argument("--model", default="claude-sonnet-4-6",
+                   help="model id for the provider")
+    p.add_argument("--prompt", choices=["strict", "enhance"], default="strict",
+                   help="strict: descriptive only; enhance: license style/mood inference")
+    p.add_argument("--grounding-threshold", type=float, default=1.0,
+                   help="reject samples below this grounding rate (default: 1.0 = strict)")
+    args = p.parse_args(argv)
+
+    captions = load_captions(args.source, limit=args.n)
+    print(f"Loaded {len(captions)} source captions from {args.source}")
+
+    if args.provider == "claude":
+        from .providers.claude_api import ClaudeProvider
+        provider = ClaudeProvider(model=args.model)
+    else:
+        raise NotImplementedError(args.provider)
+
+    print(f"Provider: {args.provider} ({args.model})  prompt={args.prompt}  "
+          f"grounding>={args.grounding_threshold}")
+    t0 = time.time()
+
+    rows, stats = generate_dataset(
+        captions=captions,
+        provider=provider,
+        prompt=args.prompt,
+        grounding_threshold=args.grounding_threshold,
+        on_progress=_print_progress,
+    )
+
+    dt = time.time() - t0
+
+    out_path = Path(args.output)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    with out_path.open("w") as fh:
+        for row in rows:
+            fh.write(json.dumps(row) + "\n")
+
+    print("\n=== summary ===")
+    print(f"  total           : {stats['total']}")
+    print(f"  kept            : {stats['kept']}  ({stats['kept']/max(stats['total'], 1):.1%})")
+    print(f"  rejected halluc : {stats['rejected_halluc']}")
+    print(f"  rejected invalid: {stats['rejected_invalid']}")
+    print(f"  total cost      : ${stats['total_cost_usd']:.4f}")
+    print(f"  wall time       : {dt:.1f}s")
+    print(f"  output          : {out_path}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

qwen_test_runner/eval_set.py

@@ -0,0 +1,103 @@
+"""
+eval_set.py — Seed evaluation captions.
+
+The set is designed to stress different failure modes:
+  * short captions (where the model is most tempted to invent)
+  * long descriptive captions (where it's most tempted to drop information)
+  * captions with explicit setting words ("kitchen", "park")
+  * captions with NO setting cues (force "unknown")
+  * captions with abstract nouns (no clear "subject")
+  * captions in different domains (photo, painting, screenshot)
+"""
+
+from __future__ import annotations
+from typing import List
+import json
+from pathlib import Path
+
+
+BUILTIN_CAPTIONS: List[str] = [
+    # short, sparse — model will be tempted to invent details
+    "a dog",
+    "a red car",
+    "two people talking",
+
+    # everyday scenes with clear setting cues
+    "A golden retriever catching a red frisbee in a sunny park.",
+    "A child eating cereal at a kitchen table.",
+    "Three commuters waiting at a subway platform during rush hour.",
+    "An elderly woman knitting on a porch swing.",
+    "A chef plating pasta in a busy restaurant kitchen.",
+
+    # outdoor / landscape (no people, no explicit framing)
+    "A snow-covered mountain ridge under a clear blue sky.",
+    "Waves crashing against jagged coastal rocks at sunset.",
+    "A field of yellow sunflowers stretching to the horizon.",
+
+    # indoor / no setting word (model must infer)
+    "Books stacked haphazardly on a worn wooden desk.",
+    "A laptop showing a half-finished email beside a steaming mug.",
+    "A single candle burning in an otherwise dark room.",
+
+    # explicit composition / framing words present
+    "Close-up of a bumblebee on a lavender flower, side view.",
+    "Wide shot of a marching band crossing a stadium field.",
+    "Overhead view of a chess game in progress.",
+
+    # action-heavy
+    "A skateboarder grinding a metal rail at a skatepark.",
+    "Two boxers exchanging punches in a brightly lit ring.",
+    "Firefighters carrying hoses up a smoke-filled stairwell.",
+
+    # mood-laden
+    "An empty playground at dusk, swings creaking in the wind.",
+    "A bride laughing as she dances with her father at a wedding reception.",
+    "A lone wolf howling at the moon on a snowy ridge.",
+
+    # abstract / art / non-photographic
+    "An abstract painting of swirling reds and oranges.",
+    "A digital illustration of a cyberpunk city at night with neon signs.",
+    "A black and white sketch of a hand holding a pencil.",
+
+    # screenshots / UI / unusual
+    "A screenshot of a video game character standing in a forest clearing.",
+    "A satellite image of a hurricane over the Atlantic Ocean.",
+    "A microscope photograph of red blood cells.",
+
+    # tricky — multiple subjects, multiple actions
+    "A barista pouring milk into a latte while a customer types on a laptop in the background.",
+    "A cat watching from the windowsill as squirrels chase each other on the lawn outside.",
+]
+
+
+def load_eval_set(name_or_path: str = "builtin") -> List[str]:
+    """
+    Load captions. If `name_or_path == "builtin"`, return the hand-curated set.
+    Otherwise treat as a path to a .txt (one per line) or .json (list of strings).
+    """
+    if name_or_path == "builtin":
+        return list(BUILTIN_CAPTIONS)
+
+    path = Path(name_or_path)
+    if not path.exists():
+        raise FileNotFoundError(name_or_path)
+
+    if path.suffix == ".json":
+        data = json.loads(path.read_text())
+        if not isinstance(data, list) or not all(isinstance(x, str) for x in data):
+            raise ValueError("JSON eval set must be a list of strings")
+        return data
+
+    # .txt — one caption per line, ignore blank lines and lines starting with #
+    return [
+        line.strip()
+        for line in path.read_text().splitlines()
+        if line.strip() and not line.strip().startswith("#")
+    ]
+
+
+if __name__ == "__main__":
+    captions = load_eval_set("builtin")
+    print(f"builtin eval set: {len(captions)} captions")
+    lengths = [len(c.split()) for c in captions]
+    print(f"  word counts: min={min(lengths)} median={sorted(lengths)[len(lengths)//2]} max={max(lengths)}")

qwen_test_runner/evaluator.py

@@ -0,0 +1,427 @@
+"""
+evaluator.py — Scores model output along three orthogonal axes:
+
+  1. SCHEMA VALIDITY:  does it parse as JSON, and validate against the Pydantic schema?
+  2. GROUNDING:        is every leaf string traceable to the input caption?
+                       This is the hallucination metric. v0.2: per-slot rule
+                       driven by `groundedness` in registry.SLOT_REGISTRY.
+  3. COVERAGE:         did the model surface the obvious nouns/verbs from the input,
+                       or did it drop information? (cheap recall signal)
+
+Grounding rules (per slot, read from registry):
+  - must_ground   : every leaf MUST trace to input. Otherwise hallucinated.
+  - may_infer     : leaf is allowed regardless of input. Counted as grounded.
+                    Closed-vocab values (e.g. "indoor") are also auto-grounded
+                    because the grammar enforces the value space anyway.
+  - derived_only  : leaf is expected to be inferred. Auto-grounded, never penalized.
+"""
+
+from __future__ import annotations
+import json
+import re
+from dataclasses import dataclass, field, asdict
+from typing import Any, List, Optional, Tuple
+
+from pydantic import BaseModel, ValidationError
+
+from .schema import Caption
+from .registry import SLOT_REGISTRY, SubjectValue, all_closed_vocab
+
+
+# Auto-grounded values — anything in any closed vocab is always counted as
+# grounded since the grammar pins the value space.
+CLOSED_VOCAB: set[str] = all_closed_vocab()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Parsing — recover JSON from messy model output (markdown fences, prose, etc.)
+# ──────────────────────────────────────────────────────────────────────────────
+
+_FENCE_RE = re.compile(r"```(?:json)?\s*(.*?)```", re.DOTALL)
+
+
+def _strip_fences(text: str) -> str:
+    """If the model wrapped output in ```json ... ```, peel it off. First fence wins."""
+    m = _FENCE_RE.search(text)
+    return m.group(1).strip() if m else text.strip()
+
+
+def _extract_first_json_object(text: str) -> Optional[str]:
+    """
+    Walk the text and return the first balanced {...} substring.
+    Tolerates leading prose. Returns None if no balanced object found.
+    """
+    text = _strip_fences(text)
+    start = text.find("{")
+    if start < 0:
+        return None
+    depth = 0
+    in_str = False
+    esc = False
+    for i in range(start, len(text)):
+        c = text[i]
+        if esc:
+            esc = False
+            continue
+        if c == "\\":
+            esc = True
+            continue
+        if c == '"':
+            in_str = not in_str
+            continue
+        if in_str:
+            continue
+        if c == "{":
+            depth += 1
+        elif c == "}":
+            depth -= 1
+            if depth == 0:
+                return text[start:i + 1]
+    return None
+
+
+@dataclass
+class GenericParseReport:
+    """Result of recovering + validating JSON against an arbitrary Pydantic model.
+
+    Two repair signals, because not all repair is equal:
+      `needed_repair`            — any recovery was needed (not clean bare JSON).
+      `needed_structural_repair` — recovery needed MORE than stripping a markdown
+                                   code fence (prose prefix, trailing tokens,
+                                   runaway generation). Fence-stripping is benign
+                                   and 100% deterministic, so the vision
+                                   `json_robustness` metric keys off the STRUCTURAL
+                                   signal — a model that only wraps clean JSON in
+                                   ```fences``` is still robust; one that buries it
+                                   in prose is not.
+    """
+    parsed: Optional[Any]
+    schema_valid: bool
+    error: Optional[str]
+    needed_repair: bool = False
+    needed_structural_repair: bool = False
+
+
+def parse_against(raw_text: str, model: type[BaseModel]) -> GenericParseReport:
+    """Recover the first JSON object from raw model output and validate it against
+    `model`. Never raises. Used by both the caption path (model=Caption) and the
+    vision metrics (model=per-category schema)."""
+    obj_str = _extract_first_json_object(raw_text)
+    if obj_str is None:
+        return GenericParseReport(None, False, "no JSON object found",
+                                  needed_repair=True, needed_structural_repair=True)
+    stripped = raw_text.strip()
+    needed_repair = stripped != obj_str
+    # If the ONLY thing between the raw text and the object was a code fence, the
+    # fence-stripped text equals the object → benign (fence-only) repair.
+    fenced_inner = _strip_fences(raw_text).strip()
+    fence_only = needed_repair and (fenced_inner == obj_str)
+    needed_structural_repair = needed_repair and not fence_only
+    try:
+        as_dict = json.loads(obj_str)
+    except json.JSONDecodeError as e:
+        return GenericParseReport(None, False, f"json decode: {e}",
+                                  needed_repair=True, needed_structural_repair=True)
+    try:
+        parsed = model.model_validate(as_dict)
+    except ValidationError as e:
+        return GenericParseReport(None, False, f"schema: {e.errors()[:2]}",
+                                  needed_repair=needed_repair,
+                                  needed_structural_repair=needed_structural_repair)
+    return GenericParseReport(parsed, True, None, needed_repair=needed_repair,
+                              needed_structural_repair=needed_structural_repair)
+
+
+@dataclass
+class ParseReport:
+    parsed: Optional[Caption]
+    schema_valid: bool
+    error: Optional[str]
+
+
+def parse_safely(raw_text: str) -> ParseReport:
+    """Try to recover a Caption object from raw model output. Never raises."""
+    r = parse_against(raw_text, Caption)
+    return ParseReport(r.parsed, r.schema_valid, r.error)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Grounding — the hallucination metric
+# ──────────────────────────────────────────────────────────────────────────────
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+")
+
+
+def _normalize(s: str) -> str:
+    return s.lower().strip()
+
+
+def _tokens(s: str) -> List[str]:
+    return _TOKEN_RE.findall(s.lower())
+
+
+def _depluralize(token: str) -> str:
+    """Cheap singularization: drop trailing -s, -es, -ies. No NLTK dependency.
+
+    LIMITATION: irregular plurals (children, mice, geese, men) are not handled.
+    Those will surface as false-positive hallucinations. Upgrade to a real
+    lemmatizer if irregular-plural FPs become a problem in production data.
+    """
+    if len(token) <= 3:
+        return token
+    if token.endswith("ies"):
+        return token[:-3] + "y"
+    if token.endswith("es"):
+        return token[:-2]
+    if token.endswith("s"):
+        return token[:-1]
+    return token
+
+
+def _is_grounded(leaf: str, input_caption: str) -> bool:
+    """Does `leaf` trace back to `input_caption`?"""
+    leaf_norm = _normalize(leaf)
+    if leaf_norm in CLOSED_VOCAB:
+        return True
+
+    cap_norm = _normalize(input_caption)
+    # Direct substring (handles multi-word phrases like "blue car")
+    if leaf_norm in cap_norm:
+        return True
+
+    # Token-level: every token of the leaf (after singularization) must appear in caption
+    cap_tokens = {_depluralize(t) for t in _tokens(input_caption)}
+    leaf_tokens = [_depluralize(t) for t in _tokens(leaf)]
+    if leaf_tokens and all(t in cap_tokens for t in leaf_tokens):
+        return True
+    return False
+
+
+@dataclass
+class GroundingReport:
+    leaves_total: int
+    leaves_grounded: int
+    hallucinated: List[Tuple[str, str]]  # (field_path, value)
+
+    @property
+    def grounding_rate(self) -> float:
+        return self.leaves_grounded / self.leaves_total if self.leaves_total else 1.0
+
+
+def _collect_leaves(caption: Caption) -> List[Tuple[str, str, str]]:
+    """Walk the caption and return (path, value, groundedness) for every leaf.
+
+    Closed-vocab single-value slots are NOT included — their value space is
+    grammar-enforced, so they can't hallucinate by definition.
+    """
+    leaves: List[Tuple[str, str, str]] = []
+    for slot_name, spec in SLOT_REGISTRY.items():
+        val = getattr(caption, slot_name)
+
+        if spec.cardinality == "list":
+            if spec.nested_model is SubjectValue:
+                for i, subj in enumerate(val):
+                    leaves.append((f"{slot_name}[{i}].name", subj.name, spec.groundedness))
+                    for j, attr in enumerate(subj.attributes):
+                        leaves.append(
+                            (f"{slot_name}[{i}].attributes[{j}]", attr, spec.groundedness)
+                        )
+            else:
+                for i, item in enumerate(val):
+                    leaves.append((f"{slot_name}[{i}]", item, spec.groundedness))
+        else:
+            if val is None:
+                continue
+            if spec.vocabulary == "closed":
+                # Value space is grammar-enforced — auto-grounded, not a leaf.
+                continue
+            leaves.append((slot_name, val, spec.groundedness))
+    return leaves
+
+
+def ground_check(caption: Caption, input_text: str) -> GroundingReport:
+    """Walk every leaf in the parsed caption; flag per the slot's groundedness rule.
+
+      - must_ground: leaf must trace to input or it's hallucinated
+      - may_infer:   leaf auto-counts as grounded (closed enums + soft slots)
+      - derived_only: leaf auto-counts as grounded (model is expected to infer)
+    """
+    leaves = _collect_leaves(caption)
+    grounded = 0
+    halluc: List[Tuple[str, str]] = []
+
+    for path, val, groundedness in leaves:
+        if groundedness in ("may_infer", "derived_only"):
+            grounded += 1
+            continue
+        # must_ground — strict check
+        if _is_grounded(val, input_text):
+            grounded += 1
+        else:
+            halluc.append((path, val))
+
+    return GroundingReport(
+        leaves_total=len(leaves),
+        leaves_grounded=grounded,
+        hallucinated=halluc,
+    )
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Coverage — did the model surface the obvious nouns from input? (cheap recall)
+# ──────────────────────────────────────────────────────────────────────────────
+
+# Common English stop-tokens we don't expect to appear as caption subjects/actions
+_STOP = {
+    "a", "an", "the", "of", "in", "on", "at", "to", "and", "or", "with",
+    "is", "are", "was", "were", "be", "been", "being",
+    "this", "that", "these", "those", "it", "its",
+    "for", "from", "by", "as", "into", "onto", "over", "under",
+}
+
+
+def _content_tokens(text: str) -> set[str]:
+    return {_depluralize(t) for t in _tokens(text) if t not in _STOP and len(t) > 2}
+
+
+@dataclass
+class CoverageReport:
+    input_content_tokens: int
+    output_coverage: int
+
+    @property
+    def coverage_rate(self) -> float:
+        return self.output_coverage / self.input_content_tokens if self.input_content_tokens else 1.0
+
+
+def _collect_output_strings(caption: Caption) -> list[str]:
+    """All string content the model produced, for coverage / recall scoring.
+
+    Iterates the registry so new slots automatically participate in coverage.
+    Closed-vocab single-value slots are excluded — their values come from the
+    enum, not from input content, so they're not informative for recall.
+    """
+    out: list[str] = []
+    for slot_name, spec in SLOT_REGISTRY.items():
+        val = getattr(caption, slot_name)
+        if spec.cardinality == "list":
+            if spec.nested_model is SubjectValue:
+                for subj in val:
+                    out.append(subj.name)
+                    out.extend(subj.attributes)
+            else:
+                out.extend(val)
+        else:
+            if val is None:
+                continue
+            if spec.vocabulary == "closed":
+                continue
+            out.append(val)
+    return out
+
+
+def coverage_check(caption: Caption, input_text: str) -> CoverageReport:
+    in_tokens = _content_tokens(input_text)
+    out_blob = " ".join(_collect_output_strings(caption))
+    out_tokens = _content_tokens(out_blob)
+    overlap = in_tokens & out_tokens
+    return CoverageReport(len(in_tokens), len(overlap))
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Per-sample and per-run aggregation
+# ──────────────────────────────────────────────────────────────────────────────
+
+@dataclass
+class SampleResult:
+    input_caption: str
+    mode: str
+    raw_output: str
+    schema_valid: bool
+    parse_error: Optional[str]
+    grounding_rate: float
+    hallucinations: List[Tuple[str, str]]
+    coverage_rate: float
+    n_input_tokens: int
+    n_output_tokens: int
+
+    def to_dict(self) -> dict:
+        return asdict(self)
+
+
+def score_sample(
+    input_caption: str,
+    raw_output: str,
+    mode: str,
+    n_input_tokens: int = 0,
+    n_output_tokens: int = 0,
+) -> SampleResult:
+    parse = parse_safely(raw_output)
+    if not parse.schema_valid or parse.parsed is None:
+        return SampleResult(
+            input_caption=input_caption,
+            mode=mode,
+            raw_output=raw_output,
+            schema_valid=False,
+            parse_error=parse.error,
+            grounding_rate=0.0,
+            hallucinations=[],
+            coverage_rate=0.0,
+            n_input_tokens=n_input_tokens,
+            n_output_tokens=n_output_tokens,
+        )
+
+    g = ground_check(parse.parsed, input_caption)
+    c = coverage_check(parse.parsed, input_caption)
+    return SampleResult(
+        input_caption=input_caption,
+        mode=mode,
+        raw_output=raw_output,
+        schema_valid=True,
+        parse_error=None,
+        grounding_rate=g.grounding_rate,
+        hallucinations=g.hallucinated,
+        coverage_rate=c.coverage_rate,
+        n_input_tokens=n_input_tokens,
+        n_output_tokens=n_output_tokens,
+    )
+
+
+@dataclass
+class RunMetrics:
+    mode: str
+    n_samples: int
+    schema_valid_rate: float
+    mean_grounding_rate: float
+    mean_coverage_rate: float
+    total_hallucinations: int
+    samples_with_zero_hallucinations: int
+
+    def __str__(self) -> str:
+        return (
+            f"[{self.mode}] n={self.n_samples}  "
+            f"schema_valid={self.schema_valid_rate:.1%}  "
+            f"grounding={self.mean_grounding_rate:.1%}  "
+            f"coverage={self.mean_coverage_rate:.1%}  "
+            f"clean_samples={self.samples_with_zero_hallucinations}/{self.n_samples}  "
+            f"halluc_total={self.total_hallucinations}"
+        )
+
+
+def score_run(results: List[SampleResult]) -> RunMetrics:
+    if not results:
+        return RunMetrics("empty", 0, 0.0, 0.0, 0.0, 0, 0)
+    mode = results[0].mode
+    n = len(results)
+    valid = [r for r in results if r.schema_valid]
+    return RunMetrics(
+        mode=mode,
+        n_samples=n,
+        schema_valid_rate=len(valid) / n,
+        mean_grounding_rate=sum(r.grounding_rate for r in valid) / len(valid) if valid else 0.0,
+        mean_coverage_rate=sum(r.coverage_rate for r in valid) / len(valid) if valid else 0.0,
+        total_hallucinations=sum(len(r.hallucinations) for r in results),
+        samples_with_zero_hallucinations=sum(
+            1 for r in results if r.schema_valid and not r.hallucinations
+        ),
+    )

qwen_test_runner/model_runner.py

@@ -0,0 +1,361 @@
+"""
+model_runner.py — Loads a Qwen instruct model once and exposes three generation modes.
+
+Modes:
+  1. free          — raw chat, no JSON instruction. Establishes a "what does the model
+                     do unprompted?" floor.
+  2. json_mode     — chat with a strong system prompt asking for JSON-only output.
+                     No decoder-level constraint. Tests in-context schema obedience.
+  3. constrained   — uses xgrammar (preferred) or outlines (fallback) to enforce the
+                     grammar at decode time. Schema validity becomes guaranteed; the
+                     interesting question is whether faithfulness survives.
+
+The model is loaded ONCE in __init__. All three modes share the same weights.
+
+Optional dependencies (xgrammar, outlines) degrade gracefully — if neither is installed,
+generate_constrained falls back to json_mode and emits a warning.
+"""
+
+from __future__ import annotations
+import json
+import warnings
+from dataclasses import dataclass
+from typing import Optional
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Optional backends — import lazily and tolerate missing
+try:
+    import xgrammar as xgr
+    _HAS_XGRAMMAR = True
+except ImportError:
+    _HAS_XGRAMMAR = False
+
+try:
+    import outlines
+    _HAS_OUTLINES = True
+except ImportError:
+    _HAS_OUTLINES = False
+
+
+SYSTEM_PROMPT_FREE = (
+    "You are a vision-language assistant. Given an image caption, describe what the "
+    "image shows."
+)
+
+# NOTE: this schema block mirrors SLOT_REGISTRY (registry.py) — the registry is
+# the source of truth. If a slot is added/removed there, update this block too
+# (a stale prompt validates fine because pydantic ignores extras, but the model
+# wastes output budget on fields that get silently dropped — caught 2026-07).
+SYSTEM_PROMPT_JSON = """You are a caption structuring assistant. Given an image caption,
+extract its content into JSON matching this exact schema:
+
+{
+  "subjects": [{"name": str, "attributes": [str]}],
+  "actions":  [str],
+  "setting":  "indoor" | "outdoor" | "unknown",
+  "style":    str or null,
+  "mood":     str or null
+}
+
+Rules:
+- Only include subjects, attributes, and actions that are EXPLICITLY mentioned in the caption.
+- Never invent details that aren't in the input.
+- If the caption doesn't specify the setting, use "unknown".
+- If no style or mood is evident, use null.
+- Limits (hard): at most 8 subjects and at most 8 actions (attributes per subject are
+  unlimited), and every string under 64 characters. If the caption has more, keep only
+  the most important ones.
+- Output ONLY the JSON object. No prose, no markdown, no code fences.
+""".strip()
+
+
+@dataclass
+class GenResult:
+    """Output of a single generation call."""
+    mode: str               # "free" | "json_mode" | "constrained"
+    raw_text: str           # exactly what the model decoded (after chat template strip)
+    backend: str            # "transformers" | "xgrammar" | "outlines"
+    n_input_tokens: int
+    n_output_tokens: int
+
+
+class QwenRunner:
+    """Loads a Qwen instruct model once, runs three generation modes against it."""
+
+    def __init__(
+        self,
+        model_id: str = "Qwen/Qwen3.5-0.8B",
+        device: Optional[str] = None,
+        dtype: torch.dtype = torch.bfloat16,
+        trust_remote_code: bool = True,
+        enable_thinking: bool = False,
+    ):
+        """
+        Loads a Qwen3.5 post-trained checkpoint.
+
+        Notes on Qwen3.5-0.8B specifically:
+          * It is a vision-language model (image-text-to-text). For text-only use
+            (this benchmark), just don't pass image content; the chat template
+            handles it. The vision encoder still gets loaded into VRAM (~0.1 GB).
+          * model_type=qwen3_5 needs transformers from git main:
+              pip install "transformers @ git+https://github.com/huggingface/transformers.git@main"
+          * Default is non-thinking mode. Qwen3.5-0.8B is prone to thinking loops,
+            so leave enable_thinking=False unless you have a reason.
+        """
+        self.model_id = model_id
+        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+        self.dtype = dtype
+        self.enable_thinking = enable_thinking
+
+        print(f"[QwenRunner] loading {model_id} on {self.device} ({dtype})")
+        self.tokenizer = AutoTokenizer.from_pretrained(
+            model_id, trust_remote_code=trust_remote_code
+        )
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_id,
+            torch_dtype=dtype,
+            device_map=self.device,
+            trust_remote_code=trust_remote_code,
+        )
+        self.model.eval()
+
+        # xgrammar compiler is reusable across calls — build once.
+        self._xgr_compiled_grammar = None
+        self._xgr_tokenizer_info = None
+        if _HAS_XGRAMMAR:
+            try:
+                self._xgr_tokenizer_info = xgr.TokenizerInfo.from_huggingface(self.tokenizer)
+                self._xgr_compiler = xgr.GrammarCompiler(self._xgr_tokenizer_info)
+            except Exception as e:
+                warnings.warn(f"xgrammar tokenizer init failed: {e}; falling back")
+                self._xgr_compiler = None
+        else:
+            self._xgr_compiler = None
+
+        print(f"[QwenRunner] ready. xgrammar={_HAS_XGRAMMAR}, outlines={_HAS_OUTLINES}")
+
+    # ── prompt construction ──────────────────────────────────────────────
+
+    def _build_chat(self, system: str, user: str) -> str:
+        """Apply chat template; returns the formatted prompt string.
+
+        Per the Qwen3.5 card, thinking mode is toggled via the `enable_thinking`
+        template variable (the legacy /think /nothink soft switch was removed).
+        When calling apply_chat_template directly, pass it as a regular kwarg;
+        when calling via OpenAI-compat APIs, nest it under chat_template_kwargs.
+        """
+        msgs = [
+            {"role": "system", "content": system},
+            {"role": "user", "content": user},
+        ]
+        return self.tokenizer.apply_chat_template(
+            msgs,
+            tokenize=False,
+            add_generation_prompt=True,
+            enable_thinking=self.enable_thinking,
+        )
+
+    # Recommended sampling for Qwen3.5-0.8B non-thinking text tasks (per model card).
+    # Keep top_k since transformers supports it; min_p, presence_penalty likewise.
+    RECOMMENDED_SAMPLING_NONTHINKING = dict(
+        temperature=1.0, top_p=1.0, top_k=20, min_p=0.0,
+        repetition_penalty=1.0,  # presence_penalty=2.0 not directly supported in HF generate
+    )
+    RECOMMENDED_SAMPLING_THINKING = dict(
+        temperature=1.0, top_p=0.95, top_k=20, min_p=0.0,
+        repetition_penalty=1.0,
+    )
+
+    def _generate_unconstrained(
+        self,
+        prompt_str: str,
+        max_new_tokens: int,
+        temperature: float,
+        sampling_preset: Optional[str] = None,
+    ) -> tuple[str, int, int]:
+        """Plain HF generation; returns (decoded, n_in, n_out).
+
+        sampling_preset:
+            None        — greedy (or sampled at given temperature), default top_p/top_k
+            "recommended" — apply Qwen3.5 paper's recommended params for current mode
+        """
+        inputs = self.tokenizer(prompt_str, return_tensors="pt").to(self.device)
+        n_in = inputs["input_ids"].shape[1]
+
+        gen_kwargs = dict(
+            max_new_tokens=max_new_tokens,
+            pad_token_id=self.tokenizer.eos_token_id,
+        )
+
+        if sampling_preset == "recommended":
+            preset = (
+                self.RECOMMENDED_SAMPLING_THINKING
+                if self.enable_thinking else self.RECOMMENDED_SAMPLING_NONTHINKING
+            )
+            gen_kwargs.update(preset)
+            gen_kwargs["do_sample"] = True
+        else:
+            gen_kwargs["do_sample"] = (temperature > 0)
+            gen_kwargs["temperature"] = temperature if temperature > 0 else 1.0
+
+        with torch.no_grad():
+            out = self.model.generate(**inputs, **gen_kwargs)
+
+        # Strip the prompt to keep only newly generated tokens
+        new_tokens = out[0, n_in:]
+        n_out = int(new_tokens.shape[0])
+        text = self.tokenizer.decode(new_tokens, skip_special_tokens=True)
+        return text, n_in, n_out
+
+    # ── public modes ─────────────────────────────────────────────────────
+
+    def generate_free(
+        self, caption: str, max_new_tokens: int = 256, temperature: float = 0.0,
+        sampling_preset: Optional[str] = None,
+    ) -> GenResult:
+        prompt = self._build_chat(SYSTEM_PROMPT_FREE, caption)
+        text, n_in, n_out = self._generate_unconstrained(
+            prompt, max_new_tokens, temperature, sampling_preset
+        )
+        return GenResult("free", text, "transformers", n_in, n_out)
+
+    def generate_json_mode(
+        self, caption: str, max_new_tokens: int = 256, temperature: float = 0.0,
+        sampling_preset: Optional[str] = None,
+    ) -> GenResult:
+        prompt = self._build_chat(SYSTEM_PROMPT_JSON, caption)
+        text, n_in, n_out = self._generate_unconstrained(
+            prompt, max_new_tokens, temperature, sampling_preset
+        )
+        return GenResult("json_mode", text, "transformers", n_in, n_out)
+
+    def generate_constrained(
+        self,
+        caption: str,
+        grammar_gbnf: Optional[str] = None,
+        json_schema: Optional[dict] = None,
+        max_new_tokens: int = 256,
+        temperature: float = 0.0,
+        sampling_preset: Optional[str] = None,
+    ) -> GenResult:
+        """
+        Grammar-constrained decoding. Prefers xgrammar (fastest), falls back to outlines,
+        then to plain json_mode with a warning.
+
+        Provide EITHER grammar_gbnf (xgrammar path) OR json_schema (outlines path).
+        If both are provided, xgrammar wins when available.
+        """
+        prompt = self._build_chat(SYSTEM_PROMPT_JSON, caption)
+
+        # xgrammar path
+        if self._xgr_compiler is not None and grammar_gbnf is not None:
+            return self._generate_xgrammar(
+                prompt, grammar_gbnf, max_new_tokens, temperature, sampling_preset
+            )
+
+        # outlines path — keep as fallback; install instructions in dependencies.txt
+        if _HAS_OUTLINES and json_schema is not None:
+            warnings.warn("outlines path not yet implemented; falling back to json_mode")
+
+        # final fallback
+        warnings.warn(
+            "No constrained-decoding backend active; falling back to json_mode. "
+            "Install xgrammar for true grammar-constrained generation."
+        )
+        text, n_in, n_out = self._generate_unconstrained(
+            prompt, max_new_tokens, temperature, sampling_preset
+        )
+        return GenResult("constrained_fallback", text, "transformers", n_in, n_out)
+
+    def _generate_xgrammar(
+        self, prompt_str: str, grammar_gbnf: str, max_new_tokens: int,
+        temperature: float, sampling_preset: Optional[str] = None,
+    ) -> GenResult:
+        """xgrammar-backed constrained generation.
+
+        Uses a hand-rolled LogitsProcessor instead of `xgr.contrib.hf.LogitsProcessor`
+        because the latter passes a tensor scalar to `matcher.accept_token`, which
+        the current xgrammar tvm-ffi binding rejects (it requires a Python int).
+        Calling `.item()` on the token id, as every official xgrammar tutorial does,
+        sidesteps the bug.
+        """
+        compiled = self._xgr_compiler.compile_grammar(grammar_gbnf)
+
+        inputs = self.tokenizer(prompt_str, return_tensors="pt").to(self.device)
+        n_in = inputs["input_ids"].shape[1]
+
+        logits_processor = _XGrammarLogitsProcessor(
+            compiled_grammar=compiled,
+            vocab_size=self._xgr_tokenizer_info.vocab_size,
+            prompt_len=n_in,
+        )
+
+        gen_kwargs = dict(
+            max_new_tokens=max_new_tokens,
+            pad_token_id=self.tokenizer.eos_token_id,
+            logits_processor=[logits_processor],
+        )
+        if sampling_preset == "recommended":
+            preset = (
+                self.RECOMMENDED_SAMPLING_THINKING
+                if self.enable_thinking else self.RECOMMENDED_SAMPLING_NONTHINKING
+            )
+            gen_kwargs.update(preset)
+            gen_kwargs["do_sample"] = True
+        else:
+            gen_kwargs["do_sample"] = (temperature > 0)
+            gen_kwargs["temperature"] = temperature if temperature > 0 else 1.0
+
+        with torch.no_grad():
+            out = self.model.generate(**inputs, **gen_kwargs)
+
+        new_tokens = out[0, n_in:]
+        n_out = int(new_tokens.shape[0])
+        text = self.tokenizer.decode(new_tokens, skip_special_tokens=True)
+        return GenResult("constrained", text, "xgrammar", n_in, n_out)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Custom xgrammar LogitsProcessor.
+#
+# Replaces the broken `xgr.contrib.hf.LogitsProcessor` (it passes a tensor scalar
+# to `accept_token`, which the current tvm-ffi binding rejects with
+# "Expected int but got ffi.Tensor"). We track previously-accepted positions and
+# convert every token to a plain int via `.item()` before passing it to xgrammar.
+# ──────────────────────────────────────────────────────────────────────────────
+
+class _XGrammarLogitsProcessor:
+    """Constrains HF `generate` output to a compiled xgrammar grammar."""
+
+    def __init__(self, compiled_grammar, vocab_size: int, prompt_len: int):
+        if not _HAS_XGRAMMAR:  # pragma: no cover
+            raise RuntimeError("xgrammar is not installed")
+        self.matcher = xgr.GrammarMatcher(compiled_grammar)
+        # bitmask must be int32 CPU per xgrammar docs; we move to logits.device
+        # on apply.
+        self.bitmask = xgr.allocate_token_bitmask(1, vocab_size)
+        self.prompt_len = prompt_len
+        self.accepted_up_to = prompt_len  # next position to accept from
+
+    def __call__(self, input_ids, scores):
+        # input_ids: (batch=1, cur_len)   scores: (batch=1, vocab_size)
+        cur_len = int(input_ids.shape[1])
+
+        # Accept every token generated since we last ran. On the first call
+        # cur_len == prompt_len, so this loop is a no-op.
+        for pos in range(self.accepted_up_to, cur_len):
+            tok = int(input_ids[0, pos].item())  # ← the critical .item() fix
+            ok = self.matcher.accept_token(tok)
+            if not ok:  # pragma: no cover — shouldn't happen with constrained sampling
+                break
+        self.accepted_up_to = cur_len
+
+        if self.matcher.is_terminated():
+            return scores
+
+        # Fill bitmask and apply to current-step logits.
+        self.matcher.fill_next_token_bitmask(self.bitmask)
+        xgr.apply_token_bitmask_inplace(scores, self.bitmask.to(scores.device))
+        return scores

qwen_test_runner/providers/__init__.py

@@ -0,0 +1,35 @@
+"""
+providers — pluggable backends that produce structured caption JSON.
+
+A provider exposes one method:
+
+    process(caption: str, **kwargs) -> ProviderResult
+
+…where ProviderResult mirrors the shape of model_runner.GenResult so the
+evaluator and benchmark scorer can consume both interchangeably.
+
+Current providers:
+  - QwenRunner    (model_runner.py — kept at top-level for back-compat)
+  - ClaudeProvider (claude_api.py — Anthropic API with native structured output)
+"""
+
+from __future__ import annotations
+from dataclasses import dataclass
+
+
+@dataclass
+class ProviderResult:
+    """Backend-agnostic result of one caption-processing call.
+
+    Shape matches model_runner.GenResult so the scorer doesn't care which
+    backend produced the result. `backend` says which backend ran it; `mode`
+    says how (constrained vs. free vs. tool-use vs. …).
+    """
+    mode: str
+    raw_text: str                       # the JSON string the backend produced
+    backend: str                        # "qwen" | "claude" | …
+    n_input_tokens: int                 # non-cached input (matches Anthropic usage.input_tokens)
+    n_output_tokens: int                # output tokens
+    cost_usd: float = 0.0               # for paid APIs; 0 for local models
+    n_cache_creation_tokens: int = 0    # tokens written to prompt cache (1.25x rate)
+    n_cache_read_tokens: int = 0        # tokens served from prompt cache (0.10x rate)

qwen_test_runner/providers/claude_api.py

@@ -0,0 +1,437 @@
+"""
+claude_api.py — Anthropic Claude as a caption-processing provider.
+
+Uses Anthropic's tool-calling API with forced tool choice and CAPTION_JSON_SCHEMA
+as the tool's input_schema. The model is constrained to emit JSON matching the
+schema — equivalent in semantic guarantee to xgrammar-constrained Qwen output,
+but produced by a far more capable model.
+
+Primary use cases:
+  1. Teacher labels for SFT — generate gold structured outputs from real
+     captions (COCO, LAION, Flickr30k) to fine-tune Qwen3.5-0.8B on.
+  2. Comparison baseline — see what near-perfect schema/faithfulness numbers
+     look like on the same eval set Qwen runs against.
+
+Requires:
+  pip install anthropic
+  export ANTHROPIC_API_KEY=sk-ant-...
+
+Cost note: at ~250 input tokens + ~200 output tokens per caption, Claude Sonnet
+costs roughly $0.003/sample (~$30 per 10K captions). Cheaper models (Haiku) are
+available; pass `model=...` to swap.
+"""
+
+from __future__ import annotations
+import json
+import os
+import time
+import warnings
+from pathlib import Path
+from typing import Optional
+
+from ..registry import SLOT_REGISTRY
+from ..schema import CAPTION_JSON_SCHEMA
+from . import ProviderResult
+
+# TaskSpec import is lazy — inside the function — to avoid a hard dependency
+# at module-load time for callers that don't use the task-driven path.
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# .env loading — minimal stdlib parser, no python-dotenv dependency.
+#
+# Cowork's VM doesn't inherit the host shell's environment, so a `.env` file
+# inside the mounted repo folder is the most reliable way to get the API
+# key in. Same pattern works for Claude Code and plain CLI use.
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _load_dotenv(path: Path) -> int:
+    """Parse a .env file and set unset vars in os.environ. Returns count set.
+
+    Existing env vars are not overwritten (host env wins over .env file).
+    Supports lines like:  KEY=value  /  KEY="quoted value"  /  # comments
+    """
+    if not path.is_file():
+        return 0
+    n_set = 0
+    for raw in path.read_text().splitlines():
+        line = raw.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, _, val = line.partition("=")
+        key = key.strip()
+        # Strip surrounding quotes if present
+        val = val.strip()
+        if len(val) >= 2 and val[0] == val[-1] and val[0] in ("'", '"'):
+            val = val[1:-1]
+        # Skip "export KEY=..." prefix if user copied a shell-style file
+        if key.startswith("export "):
+            key = key[len("export "):].strip()
+        # Treat empty existing values as unset. Some sandboxes (e.g. Claude
+        # Code / Cowork) inject blank ANTHROPIC_API_KEY into child processes
+        # to mask the host's real key; .env must remain authoritative there.
+        if key and not os.environ.get(key):
+            os.environ[key] = val
+            n_set += 1
+    return n_set
+
+
+def _autoload_dotenv() -> None:
+    """Search common locations for a .env and load the first match.
+
+    Priority: cwd/.env, then walk up to 3 parent dirs (for cases where the
+    agent is invoked from a subdirectory of the repo).
+    """
+    cwd = Path.cwd().resolve()
+    for candidate in [cwd, *list(cwd.parents)[:3]]:
+        env_path = candidate / ".env"
+        if env_path.is_file():
+            _load_dotenv(env_path)
+            return
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# System prompts — the strict/enhance distinction is encoded here.
+#
+# Both prompts pin the model to the registry-driven schema. The difference is
+# what category fields each prompt licenses the model to populate:
+#
+#   strict     — descriptive only. style/mood → null. For SFT teacher labels
+#                where we want grounded-only outputs to filter on.
+#   enhance    — all categories. Style/mood may be inferred. For prompt-
+#                enhancement training data.
+# ──────────────────────────────────────────────────────────────────────────────
+
+# Both prompts are intentionally sized so that (tool def + system) lands
+# safely above Sonnet's 1024-token prompt-caching minimum. The original
+# 165-token versions kept the cacheable prefix at ~1023 tokens — 1 below
+# threshold — which silently disabled caching and cost ~60% more per call.
+# The richer examples also tightened grounding (fewer style/mood leaks on
+# strict, more consistent inference on enhance).
+
+PROMPT_STRICT = """You are a caption-structuring assistant. Given an image caption,
+emit JSON matching the provided schema. Your sole job is to extract structured
+information explicitly stated in the input — never embellish, infer, or imagine
+details that aren't there.
+
+RULES:
+- `subjects`: every entity named in the caption. Each subject has a `name`
+  (a noun phrase from the caption) and optional `attributes` — adjectives or
+  descriptors the caption explicitly attaches to that subject (color, age,
+  expression, material, count, etc.).
+- `actions`: verb phrases describing what's happening. Prefer the caption's
+  own wording when possible.
+- `setting`: use "indoor" or "outdoor" if the caption indicates it (kitchen,
+  restaurant → indoor; park, beach → outdoor). Use "unknown" if no cue.
+- `style`: ALWAYS null in strict mode. Do not infer style from content.
+- `mood`: ALWAYS null in strict mode. Do not infer mood from content.
+- Empty lists `[]` and `null` are correct outputs — DO NOT invent content
+  just to populate a field.
+
+EXAMPLES:
+- "a young girl in a red dress" → subjects: [{name: "girl", attributes: ["young"]},
+  {name: "dress", attributes: ["red"]}]; setting: "unknown"
+- "a cat sleeping on a sofa" → subjects: [{name: "cat", attributes: []},
+  {name: "sofa", attributes: []}], actions: ["sleeping on a sofa"];
+  setting: "indoor"
+- "the beach at sunset" → subjects: [{name: "beach", attributes: []}];
+  setting: "outdoor"
+
+WHAT TO AVOID:
+- Adding subjects not named in the caption (no inferred "people" or
+  "background figures" unless the caption mentions them).
+- Inferring attributes the caption does not state (do not add "fluffy" for
+  a cat just because cats are typically fluffy).
+- Setting `style` or `mood` to anything other than null.
+
+Call the `emit_caption_schema` tool with your structured output.""".strip()
+
+
+PROMPT_ENHANCE = """You are a caption-structuring assistant. Given an image caption,
+emit JSON matching the provided schema. Extract grounded content faithfully, and
+where the schema licenses inference, draw it from the caption's content rather
+than imagining unrelated detail.
+
+RULES:
+- `subjects`, `actions`, subject `attributes`: ONLY content explicitly named
+  in the caption. Use noun phrases from the caption itself; do not invent
+  entities or descriptors.
+- `setting`: "indoor" or "outdoor" if the caption indicates or strongly
+  implies it (kitchen, restaurant → indoor; park, beach → outdoor);
+  otherwise "unknown".
+- `style`: you MAY infer a visual style (e.g. "photorealistic",
+  "watercolor", "cyberpunk illustration", "vintage film", "anime") when
+  the caption suggests one. Leave null if there's no signal.
+- `mood`: you MAY infer a mood from the caption's content (e.g. "tense",
+  "celebratory", "melancholy", "playful", "serene"). Leave null if neutral.
+
+EXAMPLES:
+- "an oil painting of a stormy sea" → subjects: [{name: "sea",
+  attributes: ["stormy"]}]; setting: "outdoor"; style: "oil painting";
+  mood: "tense"
+- "a child laughing at a birthday party" → subjects: [{name: "child",
+  attributes: []}], actions: ["laughing"]; setting: "indoor";
+  style: null; mood: "celebratory"
+- "a sketch of a hand holding a pencil" → subjects: [{name: "hand",
+  attributes: []}, {name: "pencil", attributes: []}],
+  actions: ["holding a pencil"]; setting: "unknown"; style: "sketch";
+  mood: null
+
+Call the `emit_caption_schema` tool with your structured output.""".strip()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Pricing table (per million tokens). Update when Anthropic publishes new rates.
+# Used only to estimate cost_usd in ProviderResult — not authoritative.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_PRICING = {
+    # model_id_substring → (input_$/Mtok, output_$/Mtok)
+    "claude-opus-4":    (15.0, 75.0),
+    "claude-sonnet-4":  ( 3.0, 15.0),
+    "claude-haiku-4":   ( 0.80,  4.0),
+    # legacy 3.x models, in case someone pins to them
+    "claude-3-5-sonnet":(3.0, 15.0),
+    "claude-3-5-haiku": (0.80, 4.0),
+}
+
+
+def _estimate_cost(
+    model_id: str,
+    n_in: int,
+    n_out: int,
+    n_cache_create: int = 0,
+    n_cache_read: int = 0,
+) -> float:
+    """Cost in USD for a single call. Cache write at 1.25x input, read at 0.10x."""
+    rates = next((v for k, v in _PRICING.items() if k in model_id), None)
+    if rates is None:
+        return 0.0
+    in_rate, out_rate = rates
+    return (
+        (n_in / 1_000_000) * in_rate
+        + (n_cache_create / 1_000_000) * (in_rate * 1.25)
+        + (n_cache_read / 1_000_000) * (in_rate * 0.10)
+        + (n_out / 1_000_000) * out_rate
+    )
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Schema slimming — Pydantic's model_json_schema() is verbose by default
+# (per-field "title", "default", "$defs/$ref" for nested types). Anthropic's
+# tool-use enforcer ignores those cosmetic fields, but they cost input tokens
+# on every uncached call. Stripping them cuts the tool definition roughly in
+# half while keeping the constraints (types, enums, maxLength, maxItems, etc).
+# ──────────────────────────────────────────────────────────────────────────────
+
+_STRIP_KEYS = {"title", "default", "description"}
+
+
+def _slim_schema(schema: dict) -> dict:
+    """Return a copy of `schema` with cosmetic keys removed and $defs inlined.
+
+    Drops: title, default, description (the model's tool-use enforcement
+    doesn't read them). Inlines local $defs/$ref pairs so nested types like
+    SubjectValue appear directly under their parent property. Constraints
+    (types, enums, anyOf, maxLength, minLength, maxItems, required) are kept.
+    """
+    import copy
+    schema = copy.deepcopy(schema)
+    defs = schema.pop("$defs", {})
+
+    def resolve(node):
+        if isinstance(node, dict):
+            if "$ref" in node and node["$ref"].startswith("#/$defs/"):
+                name = node["$ref"][len("#/$defs/"):]
+                return resolve(defs.get(name, {}))
+            return {k: resolve(v) for k, v in node.items() if k not in _STRIP_KEYS}
+        if isinstance(node, list):
+            return [resolve(x) for x in node]
+        return node
+
+    return resolve(schema)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Provider
+# ──────────────────────────────────────────────────────────────────────────────
+
+class ClaudeProvider:
+    """Caption-processing provider backed by Anthropic's Claude API.
+
+    Loads the anthropic SDK lazily so importing the package doesn't fail
+    when anthropic isn't installed and the user just wants the Qwen path.
+    """
+
+    def __init__(
+        self,
+        model: str = "claude-sonnet-4-6",
+        api_key: Optional[str] = None,
+        max_retries: int = 3,
+        retry_backoff: float = 2.0,
+    ):
+        try:
+            import anthropic
+        except ImportError as e:
+            raise ImportError(
+                "ClaudeProvider requires the `anthropic` package. "
+                "Install with: pip install anthropic"
+            ) from e
+
+        # Cowork/Claude Code don't inherit the host shell environment — load
+        # a project-local .env if one exists. Has no effect when the key is
+        # already in os.environ.
+        _autoload_dotenv()
+
+        api_key = api_key or os.environ.get("ANTHROPIC_API_KEY")
+        if not api_key:
+            raise RuntimeError(
+                "No Anthropic API key. Set ANTHROPIC_API_KEY in your shell, "
+                "drop a `.env` file with ANTHROPIC_API_KEY=... in the repo "
+                "root, or pass api_key= to ClaudeProvider(...)."
+            )
+
+        self.client = anthropic.Anthropic(api_key=api_key)
+        self.model = model
+        self.max_retries = max_retries
+        self.retry_backoff = retry_backoff
+
+        # The tool definition is the JSON Schema generated by the registry.
+        # Forcing tool use guarantees the output is schema-valid.
+        #
+        # NOTE: we keep the *verbose* pydantic schema rather than passing
+        # _slim_schema(CAPTION_JSON_SCHEMA) here. The slim form saves ~110
+        # tokens per call, but it also pushes the (tools + system) cacheable
+        # prefix below Sonnet's 1024-token minimum, which silently disables
+        # prompt caching — losing ~60% in subsequent-call savings. The
+        # _slim_schema helper stays available for cases where caching is
+        # off (e.g. one-shot calls) or for models with a smaller minimum.
+        self._tool = {
+            "name": "emit_caption_schema",
+            "description": (
+                "Emit the structured caption representation. The input_schema "
+                "follows the qwen-test-runner slot registry."
+            ),
+            "input_schema": CAPTION_JSON_SCHEMA,
+        }
+
+    def process(
+        self,
+        caption: str,
+        prompt: str = "strict",
+        max_tokens: int = 1024,
+        task=None,  # Optional[TaskSpec] — takes precedence over prompt= when set
+    ) -> ProviderResult:
+        """Convert one caption to schema-conformant JSON.
+
+        Two modes (use one):
+
+          task=<TaskSpec>     — task-driven: system prompt + tool schema come
+                                from the TaskSpec. mode_tag = "claude_<task.name>".
+                                Preferred for the per-task SFT pipeline.
+
+          prompt="strict"|"enhance"|"<custom>" — legacy path. Uses the module-level
+                                PROMPT_STRICT/PROMPT_ENHANCE constants and the
+                                universal CAPTION_JSON_SCHEMA as the tool's
+                                input_schema. Kept for back-compat with data_gen.py.
+        """
+        if task is not None:
+            # Task-driven path. We build a tool definition locally rather than
+            # reuse self._tool so the per-task schema overlay takes effect.
+            tool = {
+                "name": "emit_caption_schema",
+                "description": self._tool["description"],
+                "input_schema": task.tool_schema,
+            }
+            sys_prompt = task.system_prompt
+            mode_tag = f"claude_{task.name}"
+        else:
+            tool = self._tool
+            if prompt == "strict":
+                sys_prompt = PROMPT_STRICT
+                mode_tag = "claude_strict"
+            elif prompt == "enhance":
+                sys_prompt = PROMPT_ENHANCE
+                mode_tag = "claude_enhance"
+            else:
+                sys_prompt = prompt
+                mode_tag = "claude_custom"
+
+        response = self._call_with_retry(
+            system=sys_prompt,
+            user=caption,
+            max_tokens=max_tokens,
+            tool=tool,
+        )
+
+        # Find the tool_use block. Forced tool_choice means there's always
+        # exactly one — but we extract by type, not position, for safety.
+        tool_input = None
+        for block in response.content:
+            if block.type == "tool_use" and block.name == "emit_caption_schema":
+                tool_input = block.input
+                break
+        if tool_input is None:
+            raise RuntimeError(
+                f"Claude returned no tool_use block. Stop reason: {response.stop_reason!r}"
+            )
+
+        raw_json = json.dumps(tool_input, separators=(",", ":"))
+
+        usage = response.usage
+        n_in = usage.input_tokens
+        n_out = usage.output_tokens
+        n_cache_create = getattr(usage, "cache_creation_input_tokens", 0) or 0
+        n_cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
+        cost = _estimate_cost(self.model, n_in, n_out, n_cache_create, n_cache_read)
+
+        return ProviderResult(
+            mode=mode_tag,
+            raw_text=raw_json,
+            backend="claude",
+            n_input_tokens=n_in,
+            n_output_tokens=n_out,
+            cost_usd=cost,
+            n_cache_creation_tokens=n_cache_create,
+            n_cache_read_tokens=n_cache_read,
+        )
+
+    def _call_with_retry(self, system: str, user: str, max_tokens: int, tool=None):
+        """Anthropic call with exponential backoff on rate-limit / transient errors.
+
+        tool: optional override for the tool definition. Defaults to self._tool
+              (the universal schema). Task-driven callers pass their own.
+        """
+        import anthropic  # already imported in __init__, just re-bind name
+
+        tool = tool if tool is not None else self._tool
+        last_err: Optional[Exception] = None
+        for attempt in range(self.max_retries):
+            try:
+                # `cache_control` on the last system block marks the cache
+                # breakpoint. Everything before/including it (tools + system)
+                # is cached; the user message remains variable per-call.
+                # Sonnet's minimum cacheable prefix is 1024 tokens — our
+                # tool def + system prompt sits just above that.
+                return self.client.messages.create(
+                    model=self.model,
+                    max_tokens=max_tokens,
+                    system=[{
+                        "type": "text",
+                        "text": system,
+                        "cache_control": {"type": "ephemeral"},
+                    }],
+                    tools=[tool],
+                    tool_choice={"type": "tool", "name": "emit_caption_schema"},
+                    messages=[{"role": "user", "content": f"Caption: {user}"}],
+                )
+            except (anthropic.RateLimitError, anthropic.APIStatusError) as e:
+                last_err = e
+                sleep_s = self.retry_backoff ** attempt
+                warnings.warn(
+                    f"Claude API error (attempt {attempt + 1}/{self.max_retries}): "
+                    f"{type(e).__name__}: {e}. Sleeping {sleep_s:.1f}s."
+                )
+                time.sleep(sleep_s)
+        # All retries exhausted
+        raise RuntimeError(f"Claude API failed after {self.max_retries} retries") from last_err

qwen_test_runner/registry.py

@@ -0,0 +1,210 @@
+"""
+registry.py — The slot registry.
+
+This is the source of truth for the caption schema. Every slot the system
+knows about lives here as a SlotSpec entry. The Pydantic Caption model, the
+JSON Schema export, the GBNF grammar, and the evaluator's grounding rules
+are all derived from this registry at import time.
+
+Adding a slot is one dict entry. Adding a category is one Literal expansion.
+No code outside this file should hardcode slot names or category logic.
+
+Slot taxonomy (the three categories that came out of the baseline analysis):
+  - descriptive : grounded in the input caption. Hallucination forbidden.
+                  Examples: subjects, actions, setting.
+  - aesthetic   : how the scene should look. Often empty in input;
+                  legitimate inference (or null) in enhancement mode.
+                  Examples: style, lighting, palette.
+  - semantic    : interpretive meaning. Inferential by definition.
+                  Examples: mood, implication, narrative_function.
+
+Groundedness rules (drive the evaluator):
+  - must_ground   : every leaf MUST trace to the input caption.
+  - may_infer     : leaf may be grounded OR inferred; both are acceptable.
+  - derived_only  : leaf is expected to be inferred. Grounding check skipped.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Literal, Optional, Type
+
+from pydantic import BaseModel, Field
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Slot-level enums. Adding a value here is a registry-only change; no
+# code outside this file matches on these strings directly (the helpers below
+# encapsulate all behavior).
+# ──────────────────────────────────────────────────────────────────────────────
+
+Category = Literal["descriptive", "aesthetic", "semantic"]
+Cardinality = Literal["single", "list"]
+Vocabulary = Literal["closed", "open"]
+Groundedness = Literal["must_ground", "may_infer", "derived_only"]
+
+# value_kind selects the leaf's primitive type for code generation. "string" is
+# the caption default (every existing slot). The numeric kinds exist so the SAME
+# registry→Pydantic→JSON-Schema→GBNF machinery can describe vision outputs
+# (bounding boxes, confidences, depths) without a second codegen path.
+#   string  → str
+#   number  → float   (optionally bounded via number_range)
+#   integer → int
+#   bbox    → list[float] of length 4   (x1,y1,x2,y2 or x,y,w,h — see coords.py)
+#   point   → list[float] of length 2
+ValueKind = Literal["string", "number", "integer", "bbox", "point"]
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Nested value models. Used by slots whose value is structured (e.g. subjects
+# have a name and a list of attributes). New nested types go here and are
+# referenced from the SlotSpec via `nested_model=`.
+# ──────────────────────────────────────────────────────────────────────────────
+
+class SubjectValue(BaseModel):
+    """A single entity in the caption."""
+    name: str = Field(..., min_length=1, max_length=64)
+    # No max_length on attributes: rich captions (JoyCaption prose, booru tag
+    # strings) legitimately carry >8 per subject, and the cap was rejecting 44%
+    # of otherwise-valid structs in the 100-row bench (2026-07).
+    attributes: list[str] = Field(default_factory=list)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# SlotSpec — the unit of the registry.
+# ──────────────────────────────────────────────────────────────────────────────
+
+@dataclass(frozen=True)
+class SlotSpec:
+    """Declarative description of one schema slot.
+
+    The structural axes (cardinality, vocabulary, value_kind, nested_fields,
+    nested_model) are what drive code generation in schema.py — these apply
+    equally to caption slots and to vision-task fields. The caption-only axes
+    (category, groundedness) drive the text evaluator and default to neutral
+    values so vision per-category registries can omit them.
+
+        cardinality    — single value vs list
+        vocabulary     — open (any string) vs closed (one of `closed_values`)
+        value_kind     — leaf primitive type (string / number / integer / bbox / point)
+        nested_model   — BaseModel subclass for a structured value (caption: SubjectValue)
+        nested_fields  — declarative nested-object fields; the generalized form of
+                         nested_model — schema.py builds both the Pydantic model and
+                         the GBNF object rule recursively from these
+        category       — taxonomy bucket (caption prompts); ignored by vision
+        groundedness   — strict / soft / never (text evaluator); ignored by vision
+        optional       — may the model emit null/[] when empty
+        number_range   — (min, max) bound for numeric value_kinds (Pydantic ge/le)
+    """
+    name: str
+    cardinality: Cardinality
+    vocabulary: Vocabulary
+    category: Category = "descriptive"
+    groundedness: Groundedness = "may_infer"
+    value_kind: ValueKind = "string"
+    closed_values: tuple[str, ...] = ()
+    nested_model: Optional[Type[BaseModel]] = None
+    nested_fields: tuple["SlotSpec", ...] = ()
+    optional: bool = True
+    max_items: int = 8           # only for cardinality == "list"
+    max_str_length: int = 64     # for open-vocab strings
+    number_range: Optional[tuple[float, float]] = None
+
+    def __post_init__(self):
+        # Lightweight validation — catch registry mistakes at import time
+        if self.vocabulary == "closed" and not self.closed_values:
+            raise ValueError(f"slot {self.name!r}: closed vocab requires closed_values")
+        if self.vocabulary == "open" and self.closed_values:
+            raise ValueError(f"slot {self.name!r}: open vocab cannot have closed_values")
+        if self.nested_model is not None and self.vocabulary == "closed":
+            raise ValueError(f"slot {self.name!r}: nested_model is incompatible with closed vocab")
+        if self.nested_fields and self.nested_model is not None:
+            raise ValueError(f"slot {self.name!r}: nested_fields and nested_model are mutually exclusive")
+        if self.nested_fields and self.vocabulary == "closed":
+            raise ValueError(f"slot {self.name!r}: nested_fields is incompatible with closed vocab")
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# THE REGISTRY.
+#
+# Starter set: 5 slots that exercise all three categories and both
+# groundedness extremes. Adding a slot is a single entry below.
+# ──────────────────────────────────────────────────────────────────────────────
+
+SLOT_REGISTRY: dict[str, SlotSpec] = {
+    "subjects": SlotSpec(
+        name="subjects",
+        category="descriptive",
+        cardinality="list",
+        vocabulary="open",
+        groundedness="must_ground",
+        nested_model=SubjectValue,
+        max_items=8,
+    ),
+    "actions": SlotSpec(
+        name="actions",
+        category="descriptive",
+        cardinality="list",
+        vocabulary="open",
+        groundedness="must_ground",
+        max_items=8,
+    ),
+    "setting": SlotSpec(
+        name="setting",
+        category="descriptive",
+        cardinality="single",
+        vocabulary="closed",
+        # `may_infer` because Qwen reliably guesses indoor/outdoor from cues
+        # even when the caption doesn't say. The grammar pins the value to
+        # the enum anyway.
+        groundedness="may_infer",
+        closed_values=("indoor", "outdoor", "unknown"),
+        optional=False,   # always required; the enum includes "unknown" as escape
+    ),
+    "style": SlotSpec(
+        name="style",
+        category="aesthetic",
+        cardinality="single",
+        vocabulary="open",
+        groundedness="may_infer",
+    ),
+    "mood": SlotSpec(
+        name="mood",
+        category="semantic",
+        cardinality="single",
+        vocabulary="open",
+        # Baseline finding: mood is 73% of all hallucinations under the old
+        # rule. Reclassifying it as derived_only stops penalizing the model
+        # for inferring; it's correct behavior now, not error.
+        groundedness="derived_only",
+    ),
+}
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Query helpers. Use these instead of poking SLOT_REGISTRY directly so behavior
+# stays centralized.
+# ──────────────────────────────────────────────────────────────────────────────
+
+def slots_by_category(category: Category) -> list[SlotSpec]:
+    return [s for s in SLOT_REGISTRY.values() if s.category == category]
+
+
+def slot_names() -> list[str]:
+    """Slot names in registry-declaration order. JSON output uses this order."""
+    return list(SLOT_REGISTRY.keys())
+
+
+def get_slot(name: str) -> SlotSpec:
+    if name not in SLOT_REGISTRY:
+        raise KeyError(f"unknown slot: {name!r}")
+    return SLOT_REGISTRY[name]
+
+
+# Set of closed-vocab values across all slots — used by the evaluator as the
+# "always grounded" allowlist for the `may_infer` closed-vocab case.
+def all_closed_vocab() -> set[str]:
+    out: set[str] = set()
+    for s in SLOT_REGISTRY.values():
+        out.update(s.closed_values)
+    return out

qwen_test_runner/run_benchmark.py

@@ -0,0 +1,223 @@
+"""
+run_benchmark.py — End-to-end testbed entrypoint.
+
+Usage (after `pip install -e .`):
+    qwen-bench                                                  # all modes, builtin set
+    qwen-bench --modes free json_mode                            # subset of modes
+    qwen-bench --model Qwen/Qwen3.5-0.8B
+    qwen-bench --eval-set my_captions.txt
+    qwen-bench --max-samples 5                                   # smoke test
+
+Equivalent module invocation:
+    python -m qwen_test_runner.run_benchmark --max-samples 5
+
+Outputs to runs/{timestamp}/:
+  - config.json     : exact arguments + environment
+  - results.jsonl   : one row per (sample, mode) pair
+  - summary.json    : aggregated RunMetrics per mode
+  - report.md       : human-readable summary with hallucination examples
+"""
+
+from __future__ import annotations
+import argparse
+import json
+import sys
+import time
+from datetime import datetime
+from pathlib import Path
+from typing import List
+
+from .schema import CAPTION_GRAMMAR_GBNF, CAPTION_JSON_SCHEMA
+from .eval_set import load_eval_set
+from .evaluator import score_sample, score_run, SampleResult, RunMetrics
+
+
+def make_run_dir(root: Path) -> Path:
+    ts = datetime.now().strftime("%Y%m%d_%H%M%S")
+    run_dir = root / ts
+    run_dir.mkdir(parents=True, exist_ok=True)
+    return run_dir
+
+
+def run_mode(
+    runner,
+    mode: str,
+    captions: List[str],
+    max_new_tokens: int,
+    temperature: float,
+    sampling_preset: str | None = None,
+) -> List[SampleResult]:
+    """Run all captions through one mode. Returns per-sample results."""
+    results: List[SampleResult] = []
+    for i, cap in enumerate(captions):
+        t0 = time.time()
+        if mode == "free":
+            r = runner.generate_free(
+                cap, max_new_tokens=max_new_tokens, temperature=temperature,
+                sampling_preset=sampling_preset,
+            )
+        elif mode == "json_mode":
+            r = runner.generate_json_mode(
+                cap, max_new_tokens=max_new_tokens, temperature=temperature,
+                sampling_preset=sampling_preset,
+            )
+        elif mode == "constrained":
+            r = runner.generate_constrained(
+                cap,
+                grammar_gbnf=CAPTION_GRAMMAR_GBNF,
+                json_schema=CAPTION_JSON_SCHEMA,
+                max_new_tokens=max_new_tokens,
+                temperature=temperature,
+                sampling_preset=sampling_preset,
+            )
+        else:
+            raise ValueError(f"unknown mode: {mode}")
+        dt = time.time() - t0
+
+        scored = score_sample(
+            input_caption=cap,
+            raw_output=r.raw_text,
+            mode=mode,
+            n_input_tokens=r.n_input_tokens,
+            n_output_tokens=r.n_output_tokens,
+        )
+        results.append(scored)
+        print(
+            f"  [{mode}] {i + 1:3d}/{len(captions)}  "
+            f"valid={scored.schema_valid}  "
+            f"ground={scored.grounding_rate:.0%}  "
+            f"halluc={len(scored.hallucinations)}  "
+            f"{dt:.1f}s  "
+            f"→ {cap[:50]}{'…' if len(cap) > 50 else ''}"
+        )
+    return results
+
+
+def write_report(run_dir: Path, all_results: dict[str, List[SampleResult]],
+                 metrics: dict[str, RunMetrics]) -> None:
+    """Human-readable markdown summary."""
+    lines = ["# Qwen Caption Schema Benchmark", ""]
+    lines.append(f"_Generated: {datetime.now().isoformat(timespec='seconds')}_")
+    lines.append("")
+    lines.append("## Headline metrics")
+    lines.append("")
+    lines.append("| Mode | Schema valid | Grounding | Coverage | Clean samples | Total halluc |")
+    lines.append("|------|--------------|-----------|----------|---------------|--------------|")
+    for mode, m in metrics.items():
+        lines.append(
+            f"| {mode} | {m.schema_valid_rate:.1%} | {m.mean_grounding_rate:.1%} | "
+            f"{m.mean_coverage_rate:.1%} | {m.samples_with_zero_hallucinations}/{m.n_samples} | "
+            f"{m.total_hallucinations} |"
+        )
+    lines.append("")
+
+    # Hallucination examples per mode
+    for mode, rs in all_results.items():
+        offenders = [r for r in rs if r.hallucinations]
+        if not offenders:
+            continue
+        lines.append(f"## Hallucination examples — `{mode}` ({len(offenders)} samples)")
+        lines.append("")
+        for r in offenders[:6]:
+            lines.append(f"**Input:** {r.input_caption}")
+            for path, val in r.hallucinations:
+                lines.append(f"- `{path}` = `{val}`")
+            lines.append("")
+
+    # Parse failures
+    for mode, rs in all_results.items():
+        broken = [r for r in rs if not r.schema_valid]
+        if not broken:
+            continue
+        lines.append(f"## Schema parse failures — `{mode}` ({len(broken)} samples)")
+        lines.append("")
+        for r in broken[:4]:
+            lines.append(f"**Input:** {r.input_caption}")
+            lines.append(f"- Error: `{r.parse_error}`")
+            lines.append(f"- Raw output (first 200 chars):")
+            lines.append(f"  ```")
+            lines.append(f"  {r.raw_output[:200]}")
+            lines.append(f"  ```")
+            lines.append("")
+
+    (run_dir / "report.md").write_text("\n".join(lines))
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(description="Qwen caption schema benchmark")
+    p.add_argument("--model", default="Qwen/Qwen3.5-0.8B",
+                   help="HF model id. Qwen3.5-0.8B is a VLM but works text-only here.")
+    p.add_argument("--modes", nargs="+", default=["free", "json_mode", "constrained"],
+                   choices=["free", "json_mode", "constrained"])
+    p.add_argument("--eval-set", default="builtin")
+    p.add_argument("--max-samples", type=int, default=None,
+                   help="limit eval set size (for smoke tests)")
+    p.add_argument("--max-new-tokens", type=int, default=256)
+    p.add_argument("--temperature", type=float, default=0.0,
+                   help="Used only when --sampling=manual. 0.0 = greedy.")
+    p.add_argument("--sampling", choices=["manual", "recommended"], default="manual",
+                   help="'manual' uses --temperature (good for reproducibility). "
+                        "'recommended' uses Qwen3.5 paper's recommended params.")
+    p.add_argument("--enable-thinking", action="store_true",
+                   help="Turn on Qwen3.5 thinking mode. NOTE: 0.8B is prone to "
+                        "thinking loops; benchmark may be slow or hang.")
+    p.add_argument("--output-root", default="runs")
+    p.add_argument("--device", default=None)
+    args = p.parse_args(argv)
+
+    # Import the model runner lazily so smoke-testing other modules doesn't drag in torch
+    from .model_runner import QwenRunner
+
+    captions = load_eval_set(args.eval_set)
+    if args.max_samples is not None:
+        captions = captions[:args.max_samples]
+    print(f"Loaded {len(captions)} captions from {args.eval_set}")
+
+    run_dir = make_run_dir(Path(args.output_root))
+    print(f"Run dir: {run_dir}")
+
+    # Save the exact config
+    (run_dir / "config.json").write_text(json.dumps(vars(args), indent=2, default=str))
+
+    runner = QwenRunner(
+        model_id=args.model,
+        device=args.device,
+        enable_thinking=args.enable_thinking,
+    )
+
+    sampling_preset = "recommended" if args.sampling == "recommended" else None
+
+    all_results: dict[str, List[SampleResult]] = {}
+    metrics: dict[str, RunMetrics] = {}
+
+    for mode in args.modes:
+        print(f"\n=== mode: {mode} ===")
+        rs = run_mode(
+            runner, mode, captions,
+            max_new_tokens=args.max_new_tokens,
+            temperature=args.temperature,
+            sampling_preset=sampling_preset,
+        )
+        all_results[mode] = rs
+        metrics[mode] = score_run(rs)
+        print(f"  → {metrics[mode]}")
+
+    # Persist
+    with (run_dir / "results.jsonl").open("w") as fh:
+        for mode, rs in all_results.items():
+            for r in rs:
+                fh.write(json.dumps(r.to_dict()) + "\n")
+    (run_dir / "summary.json").write_text(json.dumps(
+        {mode: vars(m) for mode, m in metrics.items()}, indent=2
+    ))
+    write_report(run_dir, all_results, metrics)
+
+    print("\n=== Summary ===")
+    for m in metrics.values():
+        print(f"  {m}")
+    print(f"\nReport written to {run_dir / 'report.md'}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

qwen_test_runner/schema.py

@@ -0,0 +1,423 @@
+"""
+schema.py — Registry-parametric schema code generation.
+
+Generates three representations of a *registry* (a `dict[str, SlotSpec]`):
+
+    build_model_from_registry(name, registry)  → Pydantic model class
+    build_json_schema(model)                    → JSON Schema dict
+    build_gbnf_from_registry(registry)          → GBNF grammar string
+
+The caption schema is the canonical instance, exposed under stable names:
+
+    Caption                 — Pydantic model (validation / parsing)
+    CAPTION_JSON_SCHEMA     — JSON Schema dict (Anthropic API, outlines, etc.)
+    CAPTION_GRAMMAR_GBNF    — GBNF grammar string (xgrammar)
+
+The vision subpackage reuses the SAME generators per task category (each
+category owns a small `dict[str, SlotSpec]`), so numbers, bounding boxes, and
+nested objects are described with the same machinery — see
+`qwen_test_runner/vision/`.
+
+All caption artifacts are generated at import time from `registry.SLOT_REGISTRY`.
+To add or modify a caption slot, edit `registry.py` only — this file stays
+untouched.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Literal, Mapping, Optional
+
+from pydantic import BaseModel, Field, create_model
+
+from .registry import SLOT_REGISTRY, SlotSpec, SubjectValue
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Pydantic model — built dynamically from any registry.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_OBJECT_MODEL_CACHE: dict[SlotSpec, type[BaseModel]] = {}
+
+
+def _object_model_name(spec: SlotSpec) -> str:
+    return "".join(part.capitalize() for part in spec.name.split("_")) + "Obj"
+
+
+def _build_object_model(spec: SlotSpec) -> type[BaseModel]:
+    """Build (and cache) a Pydantic model from a spec's `nested_fields`."""
+    cached = _OBJECT_MODEL_CACHE.get(spec)
+    if cached is not None:
+        return cached
+    fields: dict[str, Any] = {}
+    for f in spec.nested_fields:
+        fields[f.name] = (_python_type_for_slot(f), _field_for_slot(f))
+    model = create_model(_object_model_name(spec), **fields)
+    _OBJECT_MODEL_CACHE[spec] = model
+    return model
+
+
+def _item_type_for_slot(spec: SlotSpec) -> Any:
+    """Python type of a single value of this slot (before list/Optional wrapping)."""
+    if spec.nested_fields:
+        return _build_object_model(spec)
+    if spec.nested_model is not None:
+        return spec.nested_model
+    if spec.vocabulary == "closed":
+        # Literal[("a", "b", "c")] parses identically to Literal["a", "b", "c"].
+        return Literal[spec.closed_values]
+    vk = spec.value_kind
+    if vk == "number":
+        return float
+    if vk == "integer":
+        return int
+    if vk in ("bbox", "point"):
+        return list[float]
+    return str
+
+
+def _python_type_for_slot(spec: SlotSpec) -> Any:
+    """Compute the Python type annotation for a slot's value.
+
+    List cardinality wraps the item type in list[...].
+    Single + optional wraps in Optional[...].
+    """
+    item_type = _item_type_for_slot(spec)
+    if spec.cardinality == "list":
+        return list[item_type]
+    if spec.optional:
+        return Optional[item_type]
+    return item_type
+
+
+def _default_for_slot(spec: SlotSpec) -> Any:
+    if spec.cardinality == "list":
+        return []  # default_factory handled by Field below
+    if spec.optional:
+        return None
+    # Required single value with no default. For closed vocab, default to
+    # the last value (usually "unknown") so partial outputs don't blow up.
+    if spec.vocabulary == "closed":
+        return spec.closed_values[-1]
+    return ...  # required, no default
+
+
+def _field_for_slot(spec: SlotSpec):
+    """Construct a Pydantic Field with the right constraints for this slot."""
+    kwargs: dict[str, Any] = {}
+
+    if spec.cardinality == "list":
+        kwargs["default_factory"] = list
+        kwargs["max_length"] = spec.max_items
+        return Field(**kwargs)
+
+    default = _default_for_slot(spec)
+
+    # Fixed-length numeric arrays (bbox/point): exactly 4 / 2 elements.
+    if spec.value_kind in ("bbox", "point"):
+        n = 4 if spec.value_kind == "bbox" else 2
+        if default is ...:
+            return Field(..., min_length=n, max_length=n)
+        return Field(default=default, min_length=n, max_length=n)
+
+    # Scalar numerics, optionally bounded.
+    if spec.value_kind in ("number", "integer"):
+        rng: dict[str, Any] = {}
+        if spec.number_range is not None:
+            rng["ge"] = spec.number_range[0]
+            rng["le"] = spec.number_range[1]
+        if default is ...:
+            return Field(..., **rng)
+        return Field(default=default, **rng)
+
+    # Strings / enums / nested objects.
+    if default is ...:
+        # Only plain open strings get a length cap; nested models / enums don't.
+        if spec.vocabulary == "open" and spec.nested_model is None and not spec.nested_fields:
+            return Field(..., max_length=spec.max_str_length)
+        return Field(...)
+    kwargs["default"] = default
+    if (
+        spec.vocabulary == "open"
+        and spec.nested_model is None
+        and not spec.nested_fields
+        and spec.value_kind == "string"
+    ):
+        kwargs["max_length"] = spec.max_str_length
+    return Field(**kwargs)
+
+
+def build_model_from_registry(model_name: str, registry: Mapping[str, SlotSpec]) -> type[BaseModel]:
+    """Build a Pydantic model with one field per registry entry."""
+    fields: dict[str, Any] = {}
+    for name, spec in registry.items():
+        fields[name] = (_python_type_for_slot(spec), _field_for_slot(spec))
+    return create_model(model_name, **fields)
+
+
+def build_json_schema(model: type[BaseModel]) -> dict:
+    """JSON Schema for a generated model (thin wrapper for symmetry)."""
+    return model.model_json_schema()
+
+
+Caption = build_model_from_registry("Caption", SLOT_REGISTRY)
+
+# Re-export SubjectValue under the old name "Subject" for callers that
+# imported it from schema previously.
+Subject = SubjectValue
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# JSON Schema — derived from the Pydantic model.
+# ──────────────────────────────────────────────────────────────────────────────
+
+CAPTION_JSON_SCHEMA: dict = build_json_schema(Caption)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# GBNF grammar — built from the registry. Independent of pydantic.
+#
+# xgrammar's auto-converter from JSON schema sometimes adds unwanted slack
+# (e.g. permissive whitespace patterns that hurt parse rates). Generating GBNF
+# by hand from the registry gives tighter control and stays consistent with
+# the Pydantic model.
+#
+# The four base primitives (str_array, string, char, ws) are always emitted, as
+# in v0.2. Numeric primitives (number, bbox4, …) are emitted ONLY when a slot
+# references them, so the caption grammar is byte-for-byte the v0.2 grammar.
+# ──────────────────────────────────────────────────────────────────────────────
+
+# Primitive rule definitions, emitted on demand.
+_PRIMITIVE_DEFS: dict[str, str] = {
+    "uint": 'uint ::= "0" | [1-9] [0-9]*',
+    "frac": 'frac ::= "." [0-9]+',
+    "exp": 'exp ::= ("e" | "E") ("+" | "-")? [0-9]+',
+    "integer": 'integer ::= "-"? uint',
+    "number": 'number ::= "-"? uint frac? exp?',
+    "num_array": 'num_array ::= "[" ws "]" | "[" ws number (ws "," ws number)* ws "]"',
+    "bbox4": 'bbox4 ::= "[" ws number ws "," ws number ws "," ws number ws "," ws number ws "]"',
+    "point2": 'point2 ::= "[" ws number ws "," ws number ws "]"',
+}
+
+# Transitive dependencies between numeric primitives (the four base primitives
+# are always present, so they are never listed here).
+_PRIMITIVE_DEPS: dict[str, set[str]] = {
+    "uint": set(),
+    "frac": set(),
+    "exp": set(),
+    "integer": {"uint"},
+    "number": {"uint", "frac", "exp"},
+    "num_array": {"number"},
+    "bbox4": {"number"},
+    "point2": {"number"},
+}
+
+# Stable emission order so the grammar regenerates deterministically.
+_PRIMITIVE_ORDER = ["integer", "number", "num_array", "bbox4", "point2", "uint", "frac", "exp"]
+
+
+def _gbnf_string_alternation(values: tuple[str, ...]) -> str:
+    """Emit `"\"a\"" | "\"b\"" | ...` for a closed enum."""
+    return " | ".join(f'"\\"{v}\\""' for v in values)
+
+
+def _resolve_primitive_deps(deps: set[str]) -> set[str]:
+    """Expand a set of primitive names with all transitive dependencies."""
+    out: set[str] = set()
+    stack = list(deps)
+    while stack:
+        d = stack.pop()
+        if d in out:
+            continue
+        out.add(d)
+        stack.extend(_PRIMITIVE_DEPS.get(d, set()))
+    return out
+
+
+def _gbnf_object_rule(spec: SlotSpec) -> tuple[str, list[str], set[str]]:
+    """Build the GBNF object rule for a spec's nested_fields. Returns
+    (object_rule_name, extra_rules, primitive_deps)."""
+    rule_name = f"obj_{spec.name}"
+    parts: list[str] = ['"{"', "ws"]
+    extras: list[str] = []
+    deps: set[str] = set()
+    for i, f in enumerate(spec.nested_fields):
+        if i > 0:
+            parts += ['","', "ws"]
+        frhs, fextras, fdeps = _gbnf_slot_value_rule(f)
+        extras += fextras
+        deps |= fdeps
+        # Wrap the field value in parens so alternations (e.g. "null" | bbox4)
+        # compose correctly inside the object.
+        parts += [f'"\\"{f.name}\\":"', "ws", f"( {frhs} )", "ws"]
+    parts.append('"}"')
+    extras.append(f"{rule_name} ::= " + " ".join(parts))
+    return rule_name, extras, deps
+
+
+def _gbnf_slot_value_rule(spec: SlotSpec) -> tuple[str, list[str], set[str]]:
+    """Return (right-hand-side, extra_rules, primitive_deps) for this slot's value.
+
+    The RHS is what appears after `slot_<name> ::=`. It is either a rule name or
+    a small alternation (e.g. `"null" | number`). Extra rules are helper rules
+    this slot needs; primitive_deps names numeric primitives to emit globally.
+    """
+    extras: list[str] = []
+    deps: set[str] = set()
+
+    if spec.cardinality == "list":
+        if spec.nested_model is SubjectValue:
+            # SubjectValue is the caption's one hand-written nested type; keep the
+            # exact v2 rules so the caption grammar is unchanged.
+            extras.append(
+                'subject ::= "{" ws "\\"name\\":" ws string ws "," ws '
+                '"\\"attributes\\":" ws str_array ws "}"'
+            )
+            extras.append(
+                'subject_list ::= "[" ws "]" | '
+                '"[" ws subject (ws "," ws subject)* ws "]"'
+            )
+            return "subject_list", extras, deps
+        if spec.nested_fields:
+            obj_name, obj_extras, obj_deps = _gbnf_object_rule(spec)
+            extras += obj_extras
+            deps |= obj_deps
+            list_name = f"{spec.name}_list"
+            extras.append(
+                f'{list_name} ::= "[" ws "]" | '
+                f'"[" ws {obj_name} (ws "," ws {obj_name})* ws "]"'
+            )
+            return list_name, extras, deps
+        if spec.value_kind in ("number", "integer"):
+            deps.add("num_array")
+            return "num_array", extras, deps
+        # Primitive open-vocab list — array of strings
+        return "str_array", extras, deps
+
+    # Single value
+    if spec.nested_fields:
+        # A single nested object (e.g. subject_fixation.primary_subject). Without
+        # this, the grammar would fall through to the string rule and force the
+        # object to serialize as a string — breaking constrained decoding.
+        obj_name, obj_extras, obj_deps = _gbnf_object_rule(spec)
+        extras += obj_extras
+        deps |= obj_deps
+        if spec.optional:
+            return f'"null" | {obj_name}', extras, deps
+        return obj_name, extras, deps
+
+    if spec.vocabulary == "closed":
+        alts = _gbnf_string_alternation(spec.closed_values)
+        rule_name = f"closed_{spec.name}"
+        extras.append(f"{rule_name} ::= {alts}")
+        return rule_name, extras, deps
+
+    if spec.value_kind == "bbox":
+        deps.add("bbox4")
+        base = "bbox4"
+    elif spec.value_kind == "point":
+        deps.add("point2")
+        base = "point2"
+    elif spec.value_kind == "number":
+        deps.add("number")
+        base = "number"
+    elif spec.value_kind == "integer":
+        deps.add("integer")
+        base = "integer"
+    else:
+        base = "string"
+
+    # Optional single → allow null literal.
+    if spec.optional:
+        return f'"null" | {base}', extras, deps
+    return base, extras, deps
+
+
+def build_gbnf_from_registry(registry: Mapping[str, SlotSpec], root_name: str = "root") -> str:
+    """Generate a GBNF grammar that produces JSON conforming to `registry`."""
+    slot_rules: list[str] = []
+    helper_rules: list[str] = []
+    helper_seen: set[str] = set()
+    deps: set[str] = set()
+
+    for name, spec in registry.items():
+        rhs, extras, sdeps = _gbnf_slot_value_rule(spec)
+        slot_rules.append(f"slot_{name} ::= {rhs}")
+        deps |= sdeps
+        for r in extras:
+            head = r.split("::=", 1)[0].strip()
+            if head not in helper_seen:
+                helper_rules.append(r)
+                helper_seen.add(head)
+
+    # Root rule: opening brace, slot1, comma, slot2, ..., closing brace.
+    parts: list[str] = ['"{"', "ws"]
+    for i, name in enumerate(registry.keys()):
+        if i > 0:
+            parts += ['","', "ws"]
+        parts += [f'"\\"{name}\\":"', "ws", f"slot_{name}", "ws"]
+    parts.append('"}"')
+    root_rule = f"{root_name} ::= " + " ".join(parts)
+
+    # Base primitives — always present (str_array is needed by open-vocab lists
+    # and SubjectValue.attributes).
+    common = [
+        'str_array ::= "[" ws "]" | "[" ws string (ws "," ws string)* ws "]"',
+        'string ::= "\\"" char* "\\""',
+        'char ::= [^"\\\\] | "\\\\" ["\\\\/bfnrt]',
+        'ws ::= [ \\t\\n]*',
+    ]
+
+    # Numeric primitives — only those actually referenced (keeps caption grammar
+    # identical to v2 and vision grammars minimal).
+    resolved = _resolve_primitive_deps(deps)
+    numeric = [_PRIMITIVE_DEFS[k] for k in _PRIMITIVE_ORDER if k in resolved]
+
+    return "\n".join([root_rule] + slot_rules + helper_rules + common + numeric)
+
+
+def build_gbnf_grammar() -> str:
+    """Generate the caption GBNF grammar (back-compat wrapper)."""
+    return build_gbnf_from_registry(SLOT_REGISTRY)
+
+
+CAPTION_GRAMMAR_GBNF: str = build_gbnf_grammar()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Smoke test — `python -m qwen_test_runner.schema` validates the three reps.
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _smoke_test() -> None:
+    example = Caption(
+        subjects=[Subject(name="dog", attributes=["golden"])],
+        actions=["catching"],
+        setting="outdoor",
+        style="photorealistic",
+        mood="energetic",
+    )
+
+    as_dict = example.model_dump()
+    rebuilt = Caption.model_validate(as_dict)
+    assert rebuilt == example, "pydantic round-trip failed"
+
+    as_json = example.model_dump_json()
+    reparsed = Caption.model_validate_json(as_json)
+    assert reparsed == example, "JSON round-trip failed"
+
+    schema = CAPTION_JSON_SCHEMA
+    assert "properties" in schema
+    assert set(schema["properties"].keys()) == set(SLOT_REGISTRY.keys())
+
+    g = CAPTION_GRAMMAR_GBNF
+    for slot in SLOT_REGISTRY:
+        assert f'\\"{slot}\\"' in g, f"GBNF missing slot {slot}"
+
+    print("schema.py smoke test: OK")
+    print(f"  slots: {list(SLOT_REGISTRY.keys())}")
+    print(f"  example JSON length: {len(as_json)}")
+    print(f"  JSON Schema fields: {list(schema['properties'].keys())}")
+    print(f"  GBNF length: {len(g)} chars")
+
+
+if __name__ == "__main__":
+    _smoke_test()

qwen_test_runner/tasks.py

@@ -0,0 +1,461 @@
+"""
+tasks.py — Task registry.
+
+Each TaskSpec declares a single "what kind of JSON should the model emit"
+behaviour and owns everything needed to drive Claude (the teacher) and Qwen
+(the student) toward it: a system prompt, a tool-schema overlay on top of
+the universal CAPTION_JSON_SCHEMA, and a validator hook for post-schema
+checks (grounding for task_1, regex pattern for task_2 and task_3).
+
+Three tasks (as of v0.2):
+
+    task_1 — hallucination_reduction
+        Grounded literal extraction. Subject/action/attribute values come
+        from the caption verbatim. Style and mood are forbidden (null).
+        The schema does not enable inference; the validator runs grounding
+        check (substring + token match against input caption).
+
+    task_2 — useful_generalization
+        Encouraged categorical abstraction. Every string value is a
+        bracketed canonical generic like [pet], [vehicle], [color], [playing].
+        Schema constrains values to regex /^\\[[a-z_]+\\]$/.
+        Validator just enforces the format; semantic correctness is
+        a soft target — the open vocabulary is curated post-hoc from
+        what the model actually emits.
+
+    task_3 — generic_symbolism
+        Pure positional placeholders. subjects[].name → [ENTITY_N],
+        actions[] → [ACTION_N], setting → [INDOOR|OUTDOOR|UNKNOWN],
+        attributes → [ATTRIBUTE_N]. Numbering is within-slot, starts at 1,
+        monotonically increasing. Style and mood are nullable typed
+        placeholders.
+
+Adding a task is one TASK_REGISTRY entry. The pipeline (prompt_maker.py)
+iterates TASK_REGISTRY; downstream consumers (ClaudeProvider, the qwen
+tester) look tasks up by name.
+"""
+from __future__ import annotations
+
+import copy
+import re
+from dataclasses import dataclass, field
+from typing import Callable, Optional
+
+from .schema import CAPTION_JSON_SCHEMA
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# TaskSpec
+# ──────────────────────────────────────────────────────────────────────────────
+
+@dataclass(frozen=True)
+class TaskSpec:
+    """Declarative definition of one differentiation mode.
+
+    Fields:
+      name             — stable task id used in row tags + filenames
+      description      — one-liner for logs and row meta
+      system_prompt    — the task's system prompt (Claude + Qwen)
+      tool_schema      — a JSON Schema dict, fully built (with overlays applied).
+                          Passed as input_schema to Claude's tool def.
+      value_pattern    — optional regex every emitted string value must match.
+                          Used by both Claude (via schema 'pattern') AND the
+                          evaluator (post-hoc check on Qwen outputs).
+      validate         — optional post-hoc validator. Signature:
+                            (caption, parsed_args_dict) -> list[str]
+                          Returns a list of reject reasons; empty list = pass.
+    """
+    name: str
+    description: str
+    system_prompt: str
+    tool_schema: dict
+    value_pattern: Optional[str] = None
+    validate: Optional[Callable] = None
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Schema-overlay helpers (used to build per-task tool_schema from base)
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _deep_merge(base: dict, overlay: dict) -> dict:
+    """Recursively merge overlay into a copy of base. Overlay wins on conflicts."""
+    out = copy.deepcopy(base)
+    for k, v in overlay.items():
+        if isinstance(v, dict) and isinstance(out.get(k), dict):
+            out[k] = _deep_merge(out[k], v)
+        else:
+            out[k] = copy.deepcopy(v)
+    return out
+
+
+def _apply_string_pattern(schema: dict, pattern: str) -> dict:
+    """Return a copy of schema with `pattern` applied to every string-typed leaf.
+
+    Walks the schema and adds {'pattern': pattern} to every node where
+    type=='string' (including inside anyOf branches). Skips closed enums
+    — those are already constrained.
+    """
+    out = copy.deepcopy(schema)
+
+    def walk(node):
+        if isinstance(node, dict):
+            # If this node is a string type without an enum, attach pattern
+            if node.get("type") == "string" and "enum" not in node:
+                node["pattern"] = pattern
+            # Recurse into children
+            for v in node.values():
+                walk(v)
+        elif isinstance(node, list):
+            for item in node:
+                walk(item)
+
+    walk(out)
+    return out
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Task 1: hallucination_reduction
+#
+# Schema overlay forces style and mood to const null so Claude cannot emit
+# anything else. The system prompt also forbids them — belt and suspenders.
+# Grounding check is the validator (uses evaluator.ground_check).
+# ──────────────────────────────────────────────────────────────────────────────
+
+_TASK1_OVERLAY = {
+    "properties": {
+        "style": {"const": None},
+        "mood":  {"const": None},
+    },
+}
+
+_TASK1_PROMPT = """You are a caption-structuring assistant. Given an image-synthesis
+prompt, emit structured JSON via the emit_caption_schema tool. Your job is
+GROUNDED LITERAL EXTRACTION — extract structured information that is
+explicitly stated in the input, never embellish, infer, or imagine details.
+
+RULES:
+- subjects: every entity named in the caption. Each subject has a name (a noun
+  phrase taken from the caption) and optional attributes (adjectives/descriptors
+  the caption explicitly attaches to that subject: color, age, expression,
+  material, count, etc.).
+- actions: verb phrases describing what is happening. Use caption wording.
+- setting: "indoor" or "outdoor" if the caption indicates it (kitchen,
+  restaurant → indoor; park, beach → outdoor). Otherwise "unknown".
+- style: ALWAYS null. The schema does not permit any other value here.
+- mood: ALWAYS null. The schema does not permit any other value here.
+- Empty lists [] and null are correct outputs — DO NOT invent content to fill
+  any field. Schema-valid empty is better than schema-valid invented.
+
+EXAMPLES:
+- "a young girl in a red dress" → subjects: [{name: "girl", attributes: ["young"]},
+  {name: "dress", attributes: ["red"]}]; setting: "unknown"
+- "a cat sleeping on a sofa" → subjects: [{name: "cat", attributes: []},
+  {name: "sofa", attributes: []}], actions: ["sleeping on a sofa"];
+  setting: "indoor"
+- "the beach at sunset" → subjects: [{name: "beach", attributes: []}];
+  setting: "outdoor"
+
+WHAT TO AVOID:
+- Inventing subjects, attributes, or actions not in the caption.
+- Inferring style or mood — the schema rejects anything but null for these.
+
+Call the emit_caption_schema tool with the structured output.""".strip()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Task 2: useful_generalization
+#
+# All open-vocab string values must match /^\[[a-z_]+\]$/ — bracketed lowercase
+# generics like [pet], [vehicle], [playing], [outdoor_scene].
+# setting's enum is replaced with bracketed versions for consistency.
+# Style and mood remain null (style/mood are out of scope for this task too).
+# ──────────────────────────────────────────────────────────────────────────────
+
+_TASK2_PATTERN = r"^\[[a-z_]+\]$"
+
+_TASK2_SETTING_ENUM = ["[indoor]", "[outdoor]", "[unknown]"]
+
+# Build task_2's schema: apply pattern to all open strings, then overlay
+# setting's enum + force style/mood null.
+def _build_task2_schema() -> dict:
+    s = _apply_string_pattern(CAPTION_JSON_SCHEMA, _TASK2_PATTERN)
+    overlay = {
+        "properties": {
+            "setting": {
+                "enum": _TASK2_SETTING_ENUM,
+                "default": "[unknown]",
+            },
+            "style": {"const": None},
+            "mood":  {"const": None},
+        },
+    }
+    s = _deep_merge(s, overlay)
+    # The 'setting' enum was overwritten; remove its old pattern (closed vocab
+    # doesn't need it, and pattern + enum can confuse some validators).
+    if "pattern" in s["properties"]["setting"]:
+        del s["properties"]["setting"]["pattern"]
+    return s
+
+
+_TASK2_PROMPT = """You are a caption-structuring assistant. Given an image-synthesis
+prompt, emit structured JSON via the emit_caption_schema tool. Your job is
+USEFUL GENERALIZATION — abstract every concrete noun, adjective, and verb to
+a small canonical CATEGORICAL GENERIC in [bracket_word] form.
+
+RULES:
+- Every open-vocabulary string value MUST be in [lowercase_with_underscores]
+  format, between square brackets. The schema enforces this.
+- subjects: list of bracketed generics that abstract caption entities.
+  Prefer the smallest sensible category ([pet] over [golden_retriever];
+  [clothing] over [red_dress]; [tool] over [pencil]).
+- attributes: bracketed generic descriptors ([color], [young], [shiny]).
+- actions: bracketed generic verbs ([playing], [eating], [waiting]).
+- setting: choose [indoor], [outdoor], or [unknown].
+- style: ALWAYS null in this task.
+- mood: ALWAYS null in this task.
+
+EXAMPLES:
+- "a golden retriever catching a red frisbee in a sunny park" →
+  subjects: [{name: "[pet]", attributes: []},
+             {name: "[toy]", attributes: ["[color]"]}]
+  actions: ["[playing]"]
+  setting: "[outdoor]"
+
+- "a young girl in a red dress" →
+  subjects: [{name: "[person]", attributes: ["[young]"]},
+             {name: "[clothing]", attributes: ["[color]"]}]
+  actions: []
+  setting: "[unknown]"
+
+- "an architect at his desk reviewing blueprints" →
+  subjects: [{name: "[person]", attributes: []},
+             {name: "[furniture]", attributes: []},
+             {name: "[document]", attributes: []}]
+  actions: ["[working]"]
+  setting: "[indoor]"
+
+The aim is to teach a categorical view of caption content. Pick generics that
+group similar specifics together. Different captions producing similar generic
+structures is GOOD — that is the point.
+
+Call the emit_caption_schema tool with the structured output.""".strip()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Task 3: generic_symbolism
+#
+# Numbered typed placeholders. Each slot has its own type prefix and integer
+# index (1-based, monotonic within slot). Captures positional structure with
+# zero semantic content.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_TASK3_ENTITY_PATTERN     = r"^\[ENTITY_\d+\]$"
+_TASK3_ATTRIBUTE_PATTERN  = r"^\[ATTRIBUTE_\d+\]$"
+_TASK3_ACTION_PATTERN     = r"^\[ACTION_\d+\]$"
+
+_TASK3_SETTING_ENUM = ["[INDOOR]", "[OUTDOOR]", "[UNKNOWN]"]
+
+
+def _build_task3_schema() -> dict:
+    s = copy.deepcopy(CAPTION_JSON_SCHEMA)
+    # subjects[].name → ENTITY pattern; subjects[].attributes[] → ATTRIBUTE pattern
+    subj = s["$defs"]["SubjectValue"]["properties"]
+    subj["name"]["pattern"] = _TASK3_ENTITY_PATTERN
+    subj["attributes"]["items"]["pattern"] = _TASK3_ATTRIBUTE_PATTERN
+    # actions[] → ACTION pattern
+    s["properties"]["actions"]["items"]["pattern"] = _TASK3_ACTION_PATTERN
+    # setting → bracketed UPPERCASE enum
+    s["properties"]["setting"]["enum"] = _TASK3_SETTING_ENUM
+    s["properties"]["setting"]["default"] = "[UNKNOWN]"
+    # style and mood: must be null in this task too (placeholder structure
+    # doesn't have a meaningful "style" position — keep nullable for symmetry).
+    s["properties"]["style"] = {"const": None}
+    s["properties"]["mood"]  = {"const": None}
+    return s
+
+
+_TASK3_PROMPT = """You are a caption-structuring assistant. Given an image-synthesis
+prompt, emit structured JSON via the emit_caption_schema tool. Your job is
+PURE STRUCTURAL SYMBOLISM — convert every entity to a numbered typed
+placeholder. The output captures positional roles only, with zero semantic
+content.
+
+FORMAT:
+- subjects[i].name      → [ENTITY_N]      (N = 1, 2, 3, ... in caption order)
+- subjects[i].attributes[j] → [ATTRIBUTE_N] (N restarts at 1 within each subject)
+- actions[i]            → [ACTION_N]      (N = 1, 2, 3, ... in caption order)
+- setting               → [INDOOR], [OUTDOOR], or [UNKNOWN] (uppercase)
+- style                 → null
+- mood                  → null
+
+NUMBERING RULES:
+- N is a positive integer starting at 1.
+- Within a slot, numbering is monotonically increasing with no gaps.
+- Each occurrence of a real entity → one ENTITY_N; do not collapse duplicates.
+
+EXAMPLES:
+
+- "a golden retriever catching a red frisbee in a sunny park" →
+  subjects: [{name: "[ENTITY_1]", attributes: [],
+              "..."},
+             {name: "[ENTITY_2]", attributes: ["[ATTRIBUTE_1]"]}]
+  actions: ["[ACTION_1]"]
+  setting: "[OUTDOOR]"
+  (ENTITY_1=retriever, ATTRIBUTE_1 on ENTITY_1=golden was DROPPED because
+  the caption attached "golden" to the retriever; we keep that as attributes.
+  Wait — corrected: golden retriever has attribute "golden" → ATTRIBUTE_1.
+  frisbee has attribute "red" → ATTRIBUTE_1 (restart per subject).)
+
+- "two children playing chess" →
+  subjects: [{name: "[ENTITY_1]", attributes: ["[ATTRIBUTE_1]"]},
+             {name: "[ENTITY_2]", attributes: []}]
+  actions: ["[ACTION_1]"]
+  setting: "[UNKNOWN]"
+  (ENTITY_1=children, ATTRIBUTE_1=two on children; ENTITY_2=chess)
+
+- "the beach at sunset" →
+  subjects: [{name: "[ENTITY_1]", attributes: []}]
+  actions: []
+  setting: "[OUTDOOR]"
+
+The aim is to teach the model to think about caption STRUCTURE divorced from
+content. Two completely different captions with the same shape should produce
+the same JSON.
+
+Call the emit_caption_schema tool with the structured output.""".strip()
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Validators
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _validate_task1(caption: str, args: dict) -> list[str]:
+    """Grounding check. Imported lazily to avoid circular import with evaluator."""
+    from .evaluator import parse_safely, ground_check
+    import json
+    parse = parse_safely(json.dumps(args))
+    if not parse.schema_valid or parse.parsed is None:
+        return [f"schema: {parse.error}"]
+    report = ground_check(parse.parsed, caption)
+    if report.grounding_rate < 1.0:
+        return [f"hallucinated: {h[1]!r} at {h[0]}" for h in report.hallucinated]
+    return []
+
+
+_TASK2_VALUE_RE = re.compile(_TASK2_PATTERN)
+_TASK3_ENTITY_RE = re.compile(_TASK3_ENTITY_PATTERN)
+_TASK3_ATTRIBUTE_RE = re.compile(_TASK3_ATTRIBUTE_PATTERN)
+_TASK3_ACTION_RE = re.compile(_TASK3_ACTION_PATTERN)
+
+
+def _safe_match(regex: re.Pattern, value) -> bool:
+    """Match-or-False without crashing on non-string inputs.
+
+    Claude occasionally emits dicts where strings are expected (e.g.
+    actions=[{'type':'action','text':'...'}]). The schema's tool_use
+    enforcement *usually* catches this, but failures slip through often
+    enough that the validator must not crash on them.
+    """
+    return isinstance(value, str) and regex.fullmatch(value) is not None
+
+
+def _validate_task2(caption: str, args: dict) -> list[str]:
+    """Every open-vocab string must match the bracketed-generic pattern."""
+    errs: list[str] = []
+    if not isinstance(args, dict):
+        return [f"args is not a dict: {type(args).__name__}"]
+    for i, subj in enumerate(args.get("subjects") or []):
+        if not isinstance(subj, dict):
+            errs.append(f"subjects[{i}] is not a dict: {type(subj).__name__}")
+            continue
+        if not _safe_match(_TASK2_VALUE_RE, subj.get("name")):
+            errs.append(f"subjects[{i}].name not bracketed: {subj.get('name')!r}")
+        for j, attr in enumerate(subj.get("attributes") or []):
+            if not _safe_match(_TASK2_VALUE_RE, attr):
+                errs.append(f"subjects[{i}].attributes[{j}] not bracketed: {attr!r}")
+    for i, a in enumerate(args.get("actions") or []):
+        if not _safe_match(_TASK2_VALUE_RE, a):
+            errs.append(f"actions[{i}] not bracketed: {a!r}")
+    setting = args.get("setting")
+    if setting is not None and setting not in _TASK2_SETTING_ENUM:
+        errs.append(f"setting not in enum: {setting!r}")
+    return errs
+
+
+def _validate_task3(caption: str, args: dict) -> list[str]:
+    """Typed numbered placeholders + monotonic numbering within slot."""
+    errs: list[str] = []
+    if not isinstance(args, dict):
+        return [f"args is not a dict: {type(args).__name__}"]
+    # subjects.name → ENTITY_N, monotonic
+    for i, subj in enumerate(args.get("subjects") or []):
+        if not isinstance(subj, dict):
+            errs.append(f"subjects[{i}] is not a dict: {type(subj).__name__}")
+            continue
+        name = subj.get("name")
+        if not _safe_match(_TASK3_ENTITY_RE, name):
+            errs.append(f"subjects[{i}].name not [ENTITY_N]: {name!r}")
+            continue
+        if name != f"[ENTITY_{i + 1}]":
+            errs.append(f"subjects[{i}].name should be [ENTITY_{i + 1}], got {name!r}")
+        for j, attr in enumerate(subj.get("attributes") or []):
+            if not _safe_match(_TASK3_ATTRIBUTE_RE, attr):
+                errs.append(f"subjects[{i}].attributes[{j}] not [ATTRIBUTE_N]: {attr!r}")
+                continue
+            if attr != f"[ATTRIBUTE_{j + 1}]":
+                errs.append(
+                    f"subjects[{i}].attributes[{j}] should be [ATTRIBUTE_{j + 1}], got {attr!r}"
+                )
+    # actions: ACTION_N, monotonic
+    for i, a in enumerate(args.get("actions") or []):
+        if not _safe_match(_TASK3_ACTION_RE, a):
+            errs.append(f"actions[{i}] not [ACTION_N]: {a!r}")
+            continue
+        if a != f"[ACTION_{i + 1}]":
+            errs.append(f"actions[{i}] should be [ACTION_{i + 1}], got {a!r}")
+    setting = args.get("setting")
+    if setting is not None and setting not in _TASK3_SETTING_ENUM:
+        errs.append(f"setting not in enum: {setting!r}")
+    return errs
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# THE REGISTRY
+# ──��───────────────────────────────────────────────────────────────────────────
+
+TASK_REGISTRY: dict[str, TaskSpec] = {
+    "task_1": TaskSpec(
+        name="task_1",
+        description="hallucination_reduction: grounded literal extraction; null style/mood",
+        system_prompt=_TASK1_PROMPT,
+        tool_schema=_deep_merge(CAPTION_JSON_SCHEMA, _TASK1_OVERLAY),
+        value_pattern=None,
+        validate=_validate_task1,
+    ),
+    "task_2": TaskSpec(
+        name="task_2",
+        description="useful_generalization: bracketed categorical generics",
+        system_prompt=_TASK2_PROMPT,
+        tool_schema=_build_task2_schema(),
+        value_pattern=_TASK2_PATTERN,
+        validate=_validate_task2,
+    ),
+    "task_3": TaskSpec(
+        name="task_3",
+        description="generic_symbolism: numbered typed placeholders",
+        system_prompt=_TASK3_PROMPT,
+        tool_schema=_build_task3_schema(),
+        value_pattern=None,  # multiple patterns per slot, handled in validator
+        validate=_validate_task3,
+    ),
+}
+
+
+def get_task(name: str) -> TaskSpec:
+    if name not in TASK_REGISTRY:
+        raise KeyError(f"unknown task: {name!r}. known: {list(TASK_REGISTRY)}")
+    return TASK_REGISTRY[name]
+
+
+def task_names() -> list[str]:
+    return list(TASK_REGISTRY.keys())

qwen_test_runner/vision/__init__.py

@@ -0,0 +1,62 @@
+"""
+qwen_test_runner.vision — image→JSON benchmark harness.
+
+Extends the text testbed to vision: a per-category task registry (mirroring the
+caption SLOT_REGISTRY), coordinate normalization, ground-truth metrics that
+replace substring grounding, a multi-model VLM runner over the Qwen3.5 / Qwen3-VL
+ladder, and an orchestrator that ranks models for the no-finetune labeler verdict.
+
+The data-driven modules (coords, model_registry, tasks_vision, metrics) are
+torch-free and import eagerly. The VLM runner and orchestrator are imported lazily
+so `import qwen_test_runner.vision` stays cheap on a CPU box.
+"""
+
+from __future__ import annotations
+
+from .coords import BBox, CoordSpace, to_canonical, from_canonical, detect_space, prompt_hint_for
+from .model_registry import (
+    MODEL_REGISTRY, ModelSpec, get_model, model_keys, models_that_fit,
+    reasoning_variants, get_runner,
+)
+from .tasks_vision import (
+    VISION_TASK_REGISTRY, VisionTaskSpec, get_task, category_names, pilot_categories,
+    model_for, json_schema_for, gbnf_for, tool_schema_for, resolved_system_prompt,
+)
+from .metrics import (
+    MetricResult, VisionRunMetrics, labeler_score,
+    score_vision_sample, score_vision_run,
+)
+
+
+def __getattr__(name: str):
+    # StubVLMRunner + VLMResult are torch-free; VLMRunner pulls torch.
+    if name in ("StubVLMRunner",):
+        from .stub_runner import StubVLMRunner
+        return StubVLMRunner
+    if name == "VLMResult":
+        from .runner_types import VLMResult
+        return VLMResult
+    if name == "VLMRunner":
+        from .runners import VLMRunner
+        return VLMRunner
+    if name == "run_bench":
+        from .bench import run_bench
+        return run_bench
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = [
+    # coords
+    "BBox", "CoordSpace", "to_canonical", "from_canonical", "detect_space", "prompt_hint_for",
+    # model registry
+    "MODEL_REGISTRY", "ModelSpec", "get_model", "model_keys", "models_that_fit",
+    "reasoning_variants", "get_runner",
+    # tasks
+    "VISION_TASK_REGISTRY", "VisionTaskSpec", "get_task", "category_names", "pilot_categories",
+    "model_for", "json_schema_for", "gbnf_for", "tool_schema_for", "resolved_system_prompt",
+    # metrics
+    "MetricResult", "VisionRunMetrics", "labeler_score",
+    "score_vision_sample", "score_vision_run",
+    # lazy
+    "VLMRunner", "StubVLMRunner", "run_bench",
+]

qwen_test_runner/vision/bench.py

@@ -0,0 +1,197 @@
+"""
+bench.py — The orchestrator (torch-free at import; the runner is injected).
+
+Run matrix = models × reasoning × category × mode × N, iterated model-outer so a
+heavy checkpoint loads once and is freed before the next. Ground truth is loaded
+once per category so every model sees identical inputs (fairness).
+
+Durability (the project's standing pattern): config.json + a stream-written
+results.jsonl (one row per scored sample, written immediately) + a rejects.jsonl
+sidecar + an append-only run.log. Resume skips already-completed
+(model, reasoning, category, mode, image_id) keys, so a Colab disconnect costs at
+most one row.
+"""
+
+from __future__ import annotations
+
+import json
+import time
+from dataclasses import asdict, dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Callable, Optional
+
+from .datasets import load_gt
+from .metrics import score_vision_run, score_vision_sample
+from .report import write_reports
+from .tasks_vision import get_task, pilot_categories
+
+
+@dataclass
+class BenchConfig:
+    models: list[str]
+    categories: list[str] = field(default_factory=pilot_categories)
+    reasonings: list[str] = field(default_factory=lambda: ["instruct"])
+    modes: list[str] = field(default_factory=lambda: ["json_mode"])
+    n: int = 50
+    dataset: str = "smoke"          # "smoke" | "full"
+    runner: str = "stub"            # "stub" | "vlm"
+    precision: str = "bf16"
+    stub_behavior: str = "perfect"  # stub only
+    output_root: str = "runs/vision"
+    gpu_hourly_rate: float = 2.0
+    clear_cache_after_model: bool = False  # rm each model's HF cache after use (full-array sweeps)
+
+
+def _free_model_cache(model_key: str) -> None:
+    """Delete a model's HF Hub cache from disk (full-array sweeps on a tight SSD)."""
+    import os
+    import shutil
+    try:
+        from .model_registry import get_model
+        spec = get_model(model_key)
+        repos = [spec.repo_id] + list(spec.quant_repo_ids.values())
+        if spec.thinking_repo_id:
+            repos.append(spec.thinking_repo_id)
+    except Exception:
+        repos = [model_key]
+    base = os.path.expanduser("~/.cache/huggingface/hub")
+    for repo in repos:
+        p = os.path.join(base, "models--" + repo.replace("/", "--"))
+        if os.path.isdir(p):
+            shutil.rmtree(p, ignore_errors=True)
+
+
+def _utc_stamp() -> str:
+    # microsecond precision so back-to-back runs never collide into one run dir
+    # (which would let resume fold one run's metrics into another's report)
+    return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S_%fZ")
+
+
+def _default_runner_factory(config: BenchConfig) -> Callable[[str, str], object]:
+    if config.runner == "stub":
+        from .stub_runner import StubVLMRunner
+        return lambda mk, rsn: StubVLMRunner(model_id=mk, behavior=config.stub_behavior, reasoning=rsn)
+    # real VLM — imports torch lazily inside get_runner
+    from .model_registry import get_runner
+    return lambda mk, rsn: get_runner(mk, precision=config.precision, reasoning=rsn)
+
+
+def _completed_keys(results_path: Path) -> set:
+    done = set()
+    if not results_path.exists():
+        return done
+    for line in results_path.read_text(encoding="utf-8").splitlines():
+        if not line.strip():
+            continue
+        try:
+            r = json.loads(line)
+            done.add((r["model"], r["reasoning"], r["category"], r["mode"], r["image_id"]))
+        except (json.JSONDecodeError, KeyError):
+            continue
+    return done
+
+
+def run_bench(config: BenchConfig, runner_factory: Optional[Callable] = None,
+              run_dir: Optional[Path] = None) -> dict:
+    runner_factory = runner_factory or _default_runner_factory(config)
+    root = Path(config.output_root)
+    run_dir = run_dir or (root / _utc_stamp())
+    run_dir.mkdir(parents=True, exist_ok=True)
+
+    results_path = run_dir / "results.jsonl"
+    rejects_path = run_dir / "rejects.jsonl"
+    metrics_path = run_dir / "metrics.jsonl"
+    log_path = run_dir / "run.log"
+
+    (run_dir / "config.json").write_text(json.dumps(asdict(config), indent=2), encoding="utf-8")
+    done = _completed_keys(results_path)
+
+    def log(msg: str) -> None:
+        stamp = datetime.now(timezone.utc).strftime("%H:%M:%S")
+        with log_path.open("a", encoding="utf-8") as fh:
+            fh.write(f"[{stamp}] {msg}\n")
+
+    log(f"start config={asdict(config)}")
+    metric_rows: list[dict] = []
+    n_total = n_valid = n_reject = n_skip = 0
+
+    with results_path.open("a", encoding="utf-8") as res_fh, \
+            rejects_path.open("a", encoding="utf-8") as rej_fh, \
+            metrics_path.open("a", encoding="utf-8") as met_fh:
+
+        for model_key in config.models:
+            for reasoning in config.reasonings:
+                t_model = time.perf_counter()
+                runner = runner_factory(model_key, reasoning)
+                log(f"loaded {model_key}/{reasoning}")
+                try:
+                    for category in config.categories:
+                        spec = get_task(category)
+                        gt_key = category if config.dataset == "smoke" else spec.gt_dataset
+                        samples = load_gt(gt_key, n=config.n, split=spec.gt_split,
+                                          dataset=config.dataset)
+                        for mode in config.modes:
+                            cell: list = []
+                            for s in samples:
+                                key = (model_key, reasoning, category, mode, s.image_id)
+                                if key in done:
+                                    n_skip += 1
+                                    continue
+                                up = s.prompt if spec.per_sample_prompt else None
+                                res = runner.generate(spec, s.image, mode, image_id=s.image_id,
+                                                      image_size=s.size, gt=s.gt, user_prompt=up)
+                                mr = score_vision_sample(
+                                    spec, res.raw_text, s.gt, mode=mode, image_id=s.image_id,
+                                    image_size=s.size, grammar_conformant=res.grammar_conformant,
+                                    n_output_tokens=res.n_output_tokens, gen_seconds=res.gen_seconds)
+                                cell.append(mr)
+                                n_total += 1
+                                row = {"model": model_key, "reasoning": reasoning, **mr.to_dict()}
+                                res_fh.write(json.dumps(row) + "\n")
+                                res_fh.flush()
+                                if mr.schema_valid:
+                                    n_valid += 1
+                                else:
+                                    n_reject += 1
+                                    rej_fh.write(json.dumps({**row, "raw_text": res.raw_text}) + "\n")
+                                    rej_fh.flush()
+                            if cell:
+                                rm = score_vision_run(cell, model=model_key, reasoning=reasoning,
+                                                      category=category, mode=mode)
+                                row = asdict(rm)
+                                metric_rows.append(row)
+                                met_fh.write(json.dumps(row) + "\n")
+                                met_fh.flush()
+                                log(str(rm))
+                finally:
+                    close = getattr(runner, "close", None)
+                    if callable(close):
+                        close()
+                    if config.clear_cache_after_model and config.runner == "vlm":
+                        _free_model_cache(model_key)
+                log(f"freed {model_key}/{reasoning} in {time.perf_counter() - t_model:.1f}s")
+
+    # If resuming, fold in prior metric rows from metrics.jsonl for a complete report.
+    if metrics_path.exists():
+        seen = {(r["model"], r["reasoning"], r["category"], r["mode"]) for r in metric_rows}
+        for line in metrics_path.read_text(encoding="utf-8").splitlines():
+            if not line.strip():
+                continue
+            try:
+                r = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if (r["model"], r["reasoning"], r["category"], r["mode"]) not in seen:
+                metric_rows.append(r)
+                seen.add((r["model"], r["reasoning"], r["category"], r["mode"]))
+
+    write_reports(run_dir, metric_rows, asdict(config))
+    summary = {
+        "run_dir": str(run_dir),
+        "n_total": n_total, "n_schema_valid": n_valid, "n_rejected": n_reject, "n_skipped": n_skip,
+        "models": config.models, "categories": config.categories,
+    }
+    (run_dir / "run_summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
+    log(f"done {summary}")
+    return summary

qwen_test_runner/vision/configs/dataset_gen.yaml

@@ -0,0 +1,68 @@
+# dataset_gen.yaml — recommended config for image->JSON labeling at scale.
+# Derived from the 15-category qwen-vlmbench investigation (2026-06, RTX 6000 Pro).
+# Every model runs ZERO-SHOT (no finetune). Constrained decoding guarantees valid JSON.
+# Metric throughout: effective yield = task_accuracy x schema_valid_rate (real per-image yield).
+
+coordinate_convention: NORM_0_1000      # LOCKED: Qwen3-VL emits 0..1000 relative xyxy; GT in pixels
+decoding:
+  mode: constrained                     # xgrammar; guarantees schema-valid JSON, ~2% slower than json_mode
+  fallback: json_mode                   # if xgrammar is unavailable
+  precision: bf16
+
+# ── Single best ALL-ROUNDER (highest mean effective yield across all 15 categories) ─────────
+primary_labeler:
+  model: Qwen/Qwen3-VL-8B-Instruct      # registry key: qwen3vl-8b
+  mean_effective_yield: 0.542
+  approx_tok_per_sec: 57
+  vram_gb_bf16: 16
+  strengths: [segmentation, outline_association, bbox_grounding, style, data_type_utilization,
+              subject_fixation, camera_rotational_offset, semantic_association]
+  notes: >
+    Best balance of accuracy + speed; the only model with NO catastrophic category failure
+    (except 3D, which all models fail). Strongest visual-grounding model.
+
+# ── Specialist for LANGUAGE / STRUCTURED / RELATIONAL tasks (use instead of primary here) ───
+specialist_labeler:
+  model: Qwen/Qwen3.5-9B                # registry key: qwen3.5-9b
+  mean_effective_yield: 0.524
+  approx_tok_per_sec: 50
+  vram_gb_bf16: 18
+  strengths: [image_classification, ocr_text, data_type_differentiation, structural_spatial_awareness,
+              depth_analysis, camera_rotational_offset, semantic_association]
+  notes: >
+    Best on language/relational tasks, but FAILS pixel segmentation/outline (~0.00) — do NOT use
+    it for grounding tasks.
+
+# ── Throughput option (high-volume, accuracy-tolerant) ──────────────────────────────────────
+fleet_labeler:
+  model: Qwen/Qwen3.5-2B                # registry key: qwen3.5-2b
+  mean_effective_yield: 0.299
+  approx_tok_per_sec: 65
+  vram_gb_bf16: 4
+  notes: fastest; surprisingly strong VQA effective yield (0.75); weak on grounding/depth/classification.
+
+# ── Per-task routing: the highest-yield model per category (max-quality labeling) ───────────
+# Pin one model per category if you want the strongest possible label for each task.
+task_routing:
+  image_classification:        qwen3.5-9b      # 0.40
+  bbox_grounding:              qwen3.5-9b      # 0.37
+  ocr_text:                    qwen3.5-9b      # 0.45
+  data_type_differentiation:   qwen3.5-9b      # 0.89
+  data_type_utilization:       qwen3vl-8b      # 0.67
+  structural_spatial_awareness: qwen3.5-9b     # 0.87
+  depth_analysis:              qwen3.5-9b      # 1.00
+  subject_fixation:            qwen3.5-9b      # 1.00 (vl-8b ties)
+  segmentation:                qwen3vl-8b      # 0.51  (9b/2b ~0.00)
+  outline_association:         qwen3vl-8b      # 0.33  (9b/2b 0.00)
+  camera_rotational_offset:    qwen3.5-9b      # 0.78
+  semantic_association:        qwen3.5-9b      # 0.77
+  style_structural_awareness:  qwen3vl-8b      # 0.58
+  vit_accuracy_to_prompt:      qwen3vl-8b      # 0.75 eff (N=48). 9b collapses to 0.38 (only 44% valid); 2b 0.69
+  geometric_3d_object_id:      NONE            # all models ~0.00 from a 2D proxy — needs finetune or a real 3D dataset
+
+# ── Caveats / open items ────────────────────────────────────────────────────────────────────
+caveats:
+  - geometric_3d_object_id: every model near 0 (low valid + low acc) — 3D-from-single-2D-proxy is unsolved here.
+  - constrained decoding requires installing xgrammar BEFORE importing the package (lazy-detected at runtime).
+  - GT loaders: imagenet (gated) + textvqa (script-based) still need parquet swaps for a one-shot
+    `qwen-vlmbench --dataset full` reproduction; detection(COCO)/vqa(VQAv2)/synthetic categories all work.

qwen_test_runner/vision/coords.py

@@ -0,0 +1,161 @@
+"""
+coords.py — Coordinate normalization (the correctness centerpiece).
+
+Different VLMs report boxes in different spaces:
+  * Qwen3-VL emits RELATIVE coordinates in 0..1000 (integers).
+  * Qwen3.5 / many checkpoints emit 0..1 floats.
+  * COCO / LVIS ground truth is ABSOLUTE pixels.
+
+If predictions and ground truth are compared in mismatched spaces, every IoU is
+silently wrong and detection/3D/segmentation metrics collapse. To prevent that,
+the canonical internal form is ALWAYS pixel-absolute xyxy. GT is converted to
+canonical at load time; predictions are converted at score time. No metric ever
+sees a non-canonical coordinate (enforced by tests).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from enum import Enum
+from typing import Sequence
+
+
+class CoordSpace(str, Enum):
+    """The space a set of raw coordinates lives in."""
+    PIXEL_ABS = "pixel_abs"      # absolute pixels, image-sized
+    NORM_0_1 = "norm_0_1"        # 0..1 floats
+    NORM_0_1000 = "norm_0_1000"  # 0..1000 ints (Qwen3-VL native)
+
+
+# Box coordinate layouts a model might emit / GT might store.
+XYXY = "xyxy"  # [x1, y1, x2, y2]
+XYWH = "xywh"  # [x, y, w, h]  (COCO GT layout)
+
+
+def xywh_to_xyxy(box: Sequence[float]) -> list[float]:
+    x, y, w, h = box
+    return [x, y, x + w, y + h]
+
+
+def xyxy_to_xywh(box: Sequence[float]) -> list[float]:
+    x1, y1, x2, y2 = box
+    return [x1, y1, x2 - x1, y2 - y1]
+
+
+@dataclass(frozen=True)
+class BBox:
+    """A bounding box in the canonical form: pixel-absolute xyxy."""
+    x1: float
+    y1: float
+    x2: float
+    y2: float
+
+    def area(self) -> float:
+        return max(0.0, self.x2 - self.x1) * max(0.0, self.y2 - self.y1)
+
+    def clip(self, size: tuple[int, int]) -> "BBox":
+        """Clip to image bounds. size = (W, H)."""
+        w, h = size
+        return BBox(
+            x1=min(max(self.x1, 0.0), w),
+            y1=min(max(self.y1, 0.0), h),
+            x2=min(max(self.x2, 0.0), w),
+            y2=min(max(self.y2, 0.0), h),
+        )
+
+    def iou(self, other: "BBox") -> float:
+        ix1 = max(self.x1, other.x1)
+        iy1 = max(self.y1, other.y1)
+        ix2 = min(self.x2, other.x2)
+        iy2 = min(self.y2, other.y2)
+        iw = max(0.0, ix2 - ix1)
+        ih = max(0.0, iy2 - iy1)
+        inter = iw * ih
+        if inter <= 0.0:
+            return 0.0
+        union = self.area() + other.area() - inter
+        return inter / union if union > 0 else 0.0
+
+    def as_list(self) -> list[float]:
+        return [self.x1, self.y1, self.x2, self.y2]
+
+
+def _scale_for_space(space: CoordSpace, size: tuple[int, int]) -> tuple[float, float]:
+    """Return (x_scale, y_scale) that maps a raw coord in `space` to pixels."""
+    w, h = size
+    if space == CoordSpace.PIXEL_ABS:
+        return 1.0, 1.0
+    if space == CoordSpace.NORM_0_1:
+        return float(w), float(h)
+    if space == CoordSpace.NORM_0_1000:
+        return w / 1000.0, h / 1000.0
+    raise ValueError(f"unknown coord space: {space!r}")
+
+
+def to_canonical(
+    raw: Sequence[float],
+    space: CoordSpace,
+    size: tuple[int, int],
+    fmt: str = XYXY,
+) -> BBox:
+    """Convert a raw 4-tuple in `space`/`fmt` to a canonical pixel-abs xyxy BBox.
+
+    size = (W, H) in pixels. `fmt` is XYXY or XYWH.
+    """
+    if len(raw) != 4:
+        raise ValueError(f"bbox must have 4 values, got {len(raw)}: {raw!r}")
+    coords = list(map(float, raw))
+    if fmt == XYWH:
+        coords = xywh_to_xyxy(coords)
+    elif fmt != XYXY:
+        raise ValueError(f"unknown bbox fmt: {fmt!r}")
+    sx, sy = _scale_for_space(space, size)
+    return BBox(coords[0] * sx, coords[1] * sy, coords[2] * sx, coords[3] * sy).clip(size)
+
+
+def from_canonical(box: BBox, space: CoordSpace, size: tuple[int, int], fmt: str = XYXY) -> list[float]:
+    """Inverse of to_canonical: canonical BBox → raw coords in `space`/`fmt`."""
+    sx, sy = _scale_for_space(space, size)
+    xyxy = [box.x1 / sx, box.y1 / sy, box.x2 / sx, box.y2 / sy]
+    return xyxy_to_xywh(xyxy) if fmt == XYWH else xyxy
+
+
+def detect_space(raw_values: Sequence[float], size: tuple[int, int]) -> CoordSpace:
+    """Defensive fallback for models that ignore the requested space.
+
+    Used ONLY when a model's output space can't be trusted; the caller logs
+    `coord_space_inferred=True` (itself a robustness signal). Heuristic:
+      * all values <= 1.0           → NORM_0_1
+      * all values <= 1000 and the image is larger than 1000 px on a side → NORM_0_1000
+      * otherwise                   → PIXEL_ABS
+    """
+    vals = [abs(float(v)) for v in raw_values if v is not None]
+    if not vals:
+        return CoordSpace.PIXEL_ABS
+    mx = max(vals)
+    w, h = size
+    if mx <= 1.0:
+        return CoordSpace.NORM_0_1
+    if mx <= 1000.0 and max(w, h) > 1000:
+        return CoordSpace.NORM_0_1000
+    if mx <= 1000.0 and max(w, h) <= 1000:
+        # ambiguous: 0..1000 ints vs small-image pixels. Prefer NORM_0_1000 only
+        # if values clearly exceed the image dimensions.
+        if mx > max(w, h):
+            return CoordSpace.NORM_0_1000
+        return CoordSpace.PIXEL_ABS
+    return CoordSpace.PIXEL_ABS
+
+
+def prompt_hint_for(space: CoordSpace) -> str:
+    """A sentence appended to the system prompt telling the model which space to use."""
+    if space == CoordSpace.PIXEL_ABS:
+        return "Report bounding boxes as [x1, y1, x2, y2] in absolute pixel coordinates."
+    if space == CoordSpace.NORM_0_1:
+        return "Report bounding boxes as [x1, y1, x2, y2] normalized to 0..1 of the image dimensions."
+    if space == CoordSpace.NORM_0_1000:
+        return (
+            "Report bounding boxes as [x1, y1, x2, y2] integers in 0..1000, "
+            "relative to the image width and height."
+        )
+    raise ValueError(f"unknown coord space: {space!r}")

qwen_test_runner/vision/datasets.py

@@ -0,0 +1,928 @@
+"""
+datasets.py — Ground-truth providers.
+
+`GTSample` is the uniform shape every loader yields: an image (PIL, or None for
+the torch-free smoke set), the per-category ground truth, and the image size used
+for coordinate normalization. The packaged smoke set runs on CPU with no network
+so tests and `--dataset smoke --runner stub` work offline. Real loaders stream
+public datasets via HF `datasets` (imported lazily) for Phase 1+.
+
+GT shapes (per category):
+  image_classification : {"labels": [acceptable label strings]}
+  bbox_grounding       : {"boxes": [{"label": str, "bbox": [x,y,w,h], "fmt": "xywh"}]}
+  ocr_text             : {"text": "the reference transcription / answer"}
+  (stub categories)    : None  (no GT wired yet)
+"""
+
+from __future__ import annotations
+
+import itertools
+from dataclasses import dataclass, field
+from typing import Any, Callable, Optional
+
+
+@dataclass
+class GTSample:
+    image: Any                  # PIL.Image or None (smoke / stub)
+    prompt: str
+    gt: Any
+    category: str
+    image_id: str
+    size: tuple[int, int]       # (W, H) for coordinate normalization
+    meta: dict = field(default_factory=dict)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Packaged CPU smoke set (no network, no torch). Small but exercises every shape.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_SMOKE: dict[str, list[GTSample]] = {
+    "image_classification": [
+        GTSample(None, "Classify this image.", {"labels": ["golden retriever", "dog"]},
+                 "image_classification", "smk_cls_0", (640, 480)),
+        GTSample(None, "Classify this image.", {"labels": ["espresso", "coffee"]},
+                 "image_classification", "smk_cls_1", (512, 512)),
+        GTSample(None, "Classify this image.", {"labels": ["school bus", "bus"]},
+                 "image_classification", "smk_cls_2", (800, 600)),
+    ],
+    "bbox_grounding": [
+        GTSample(None, "Detect all objects.",
+                 {"boxes": [{"label": "dog", "bbox": [64, 48, 128, 96], "fmt": "xywh"}]},
+                 "bbox_grounding", "smk_box_0", (640, 480)),
+        GTSample(None, "Detect all objects.",
+                 {"boxes": [{"label": "cat", "bbox": [10, 10, 40, 40], "fmt": "xywh"},
+                            {"label": "ball", "bbox": [200, 150, 50, 50], "fmt": "xywh"}]},
+                 "bbox_grounding", "smk_box_1", (640, 480)),
+    ],
+    "ocr_text": [
+        GTSample(None, "Read all text.", {"text": "STOP"}, "ocr_text", "smk_ocr_0", (200, 200)),
+        GTSample(None, "Read all text.", {"text": "no entry"}, "ocr_text", "smk_ocr_1", (300, 200)),
+    ],
+}
+
+
+def smoke_samples(category: str, n: Optional[int] = None) -> list[GTSample]:
+    """Smoke samples for a category. Stub categories (no GT) get synthetic blanks."""
+    if category in _SMOKE:
+        out = _SMOKE[category]
+    else:
+        out = [GTSample(None, "Analyze this image.", None, category, f"smk_{category}_{i}", (64, 64))
+               for i in range(2)]
+    return out[:n] if n else list(out)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Real loaders (Phase 1+). HF `datasets` is imported lazily so the smoke path and
+# the CPU tests never require it.
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _hf_stream(repo: str, split: str, n: int, **kw):
+    from datasets import load_dataset  # lazy
+    ds = load_dataset(repo, split=split, streaming=True, **kw)
+    return list(itertools.islice(ds, n))
+
+
+def load_imagenet_val(n: int = 200, split: str = "validation") -> list[GTSample]:
+    """Classification GT. ImageNet-1k is GATED and its label is a bare integer id;
+    use food101 (ungated parquet) and map the ClassLabel id -> class name."""
+    from datasets import load_dataset  # lazy
+    last = None
+    for repo, sp in [("ethz/food101", "validation"), ("food101", "validation")]:
+        try:
+            ds = load_dataset(repo, split=sp, streaming=True)
+            try:
+                names = ds.features["label"].names
+            except Exception:
+                names = None
+            rows = list(itertools.islice(ds, n))
+            out = []
+            for i, r in enumerate(rows):
+                img = r.get("image")
+                lbl = r.get("label")
+                name = (names[lbl].replace("_", " ")
+                        if names and isinstance(lbl, int) and 0 <= lbl < len(names) else str(lbl))
+                size = (img.width, img.height) if img is not None else (0, 0)
+                out.append(GTSample(img, "Classify this image.", {"labels": [name]},
+                                    "image_classification", f"cls_{i}", size))
+            if out:
+                return out
+        except Exception as e:
+            last = e
+            continue
+    raise RuntimeError(f"no classification dataset streamable (food101): {last}")
+
+
+# COCO-80 class names in category-id order (confirmed from detection-datasets/coco
+# ClassLabel features). objects.bbox is [x1,y1,x2,y2] in absolute pixels (xyxy).
+COCO_CLASSES = [
+    "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat",
+    "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat",
+    "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack",
+    "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball",
+    "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket",
+    "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple",
+    "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair",
+    "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse",
+    "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator",
+    "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush",
+]
+
+
+def _coco_label(cid) -> str:
+    if isinstance(cid, int) and 0 <= cid < len(COCO_CLASSES):
+        return COCO_CLASSES[cid]
+    return str(cid)
+
+
+def load_coco_detection(n: int = 200, split: str = "val") -> list[GTSample]:
+    """detection-datasets/coco: objects.bbox is xyxy pixels; category is a ClassLabel id."""
+    rows = _hf_stream("detection-datasets/coco", split, n)
+    out = []
+    for i, r in enumerate(rows):
+        img = r["image"]
+        objs = r.get("objects", {})
+        boxes = [{"label": _coco_label(c), "bbox": list(map(float, b)), "fmt": "xyxy"}
+                 for c, b in zip(objs.get("category", []), objs.get("bbox", []))]
+        out.append(GTSample(img, "Detect all objects in this image. Output only the raw JSON object.",
+                            {"boxes": boxes}, "bbox_grounding", f"coco_{i}", (img.width, img.height)))
+    return out
+
+
+def load_textvqa(n: int = 200, split: str = "validation") -> list[GTSample]:
+    """OCR GT. The script-based 'textvqa' repo is rejected by modern `datasets`; use
+    parquet repos. GT = {"text": <gold answer>}; the model transcribes the image and
+    the OCR scorer credits containment of the answer."""
+    from datasets import load_dataset  # lazy
+    last = None
+    for repo, sp in [("lmms-lab/textvqa", "validation"), ("howard-hou/OCR-VQA", "test")]:
+        try:
+            ds = load_dataset(repo, split=sp, streaming=True)
+            rows = list(itertools.islice(ds, n))
+            out = []
+            for i, r in enumerate(rows):
+                img = r.get("image")
+                ans = r.get("answers")
+                if ans is None:
+                    ans = r.get("answer") or r.get("questions")
+                if isinstance(ans, (list, tuple)):
+                    ans = next((str(a) for a in ans if str(a).strip()), "")
+                size = (img.width, img.height) if img is not None else (0, 0)
+                out.append(GTSample(img, "Read all the text in this image.",
+                                    {"text": str(ans or "")}, "ocr_text", f"ocr_{i}", size))
+            if out:
+                return out
+        except Exception as e:
+            last = e
+            continue
+    raise RuntimeError(f"no OCR dataset streamable (textvqa/ocr-vqa): {last}")
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Synthetic data-format images (self-contained: no external dataset). Renders a
+# small record in several serialization formats to an image, with exact GT for
+# both the format (data_type) and the normalized content. Tests whether a VLM can
+# recognize a data format from a screenshot and re-serialize it to JSON.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_DATATYPE_RECORDS = [
+    {"name": "Alice", "age": "30", "city": "Paris"},
+    {"id": "7", "title": "Widget", "price": "9"},
+    {"user": "bob", "active": "true", "score": "42"},
+    {"country": "Japan", "capital": "Tokyo", "pop": "14"},
+]
+
+
+def _datatype_font(sz=22):
+    from PIL import ImageFont
+    for name in ("DejaVuSansMono.ttf", "DejaVuSans.ttf"):
+        try:
+            return ImageFont.truetype(name, sz)
+        except Exception:
+            continue
+    try:
+        return ImageFont.load_default(size=sz)  # Pillow >= 10
+    except Exception:
+        return ImageFont.load_default()
+
+
+def _render_text_image(text: str, size=(640, 360)) -> "object":
+    from PIL import Image, ImageDraw
+    img = Image.new("RGB", size, (255, 255, 255))
+    d = ImageDraw.Draw(img)
+    d.multiline_text((18, 18), text, fill=(0, 0, 0), font=_datatype_font(22), spacing=8)
+    return img
+
+
+def _serialize(rec: dict, fmt: str) -> str:
+    if fmt == "json":
+        import json as _j
+        return _j.dumps(rec, indent=2)
+    if fmt == "yaml":
+        return "\n".join(f"{k}: {v}" for k, v in rec.items())
+    if fmt == "toml":
+        return "\n".join(f'{k} = "{v}"' for k, v in rec.items())
+    if fmt == "xml":
+        inner = "".join(f"<{k}>{v}</{k}>" for k, v in rec.items())
+        return f"<record>{inner}</record>"
+    if fmt == "csv":
+        return ",".join(rec.keys()) + "\n" + ",".join(rec.values())
+    if fmt == "markdown":
+        return "# Record\n" + "\n".join(f"- **{k}**: {v}" for k, v in rec.items())
+    raise ValueError(fmt)
+
+
+_DATATYPE_FORMATS = ["json", "yaml", "toml", "xml", "csv", "markdown"]
+
+
+def make_datatype_samples(n: int = 18, split=None) -> list[GTSample]:
+    """Self-contained: render records across formats. GT = {data_type, content}."""
+    out = []
+    i = 0
+    while len(out) < n:
+        rec = _DATATYPE_RECORDS[i % len(_DATATYPE_RECORDS)]
+        fmt = _DATATYPE_FORMATS[i % len(_DATATYPE_FORMATS)]
+        text = _serialize(rec, fmt)
+        # csv normalizes to a one-row list; everything else to the dict
+        content = [rec] if fmt == "csv" else rec
+        img = _render_text_image(text)
+        out.append(GTSample(img, "Identify the data format and contents. Output only raw JSON.",
+                            {"data_type": fmt, "content": content},
+                            "data_type", f"dt_{i}", img.size))
+        i += 1
+    return out
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Synthetic colored-shapes scenes (self-contained). One scene yields exact GT for
+# spatial relations (by x-order), depth ordering (bigger circle = nearer), and
+# subject fixation (largest circle = primary subject). Reliable + no download.
+# ──────────────────────────────────────────────────────────────────────────────
+
+import itertools as _it
+
+_SHAPE_COLORS = {"red": (220, 30, 30), "green": (30, 170, 30), "blue": (40, 40, 220)}
+_SHAPE_NAMES = ["red", "green", "blue"]
+_SHAPE_SIZES = [110, 76, 46]  # diameters: big / medium / small (depth cue)
+
+
+def _shape_scene(i: int):
+    """Deterministic 3-circle scene. Returns ((W,H), [shape dicts]) sorted left→right."""
+    W, H = 540, 320
+    x_centers = [110, 270, 430]
+    color_perm = list(_it.permutations(range(3)))[i % 6]      # which color in which column
+    size_perm = list(_it.permutations(range(3)))[(i // 6) % 6]  # which color gets which size
+    shapes = []
+    for ci, color in enumerate(_SHAPE_NAMES):
+        cx = x_centers[color_perm[ci]]
+        d = _SHAPE_SIZES[size_perm[ci]]
+        cy = H // 2
+        shapes.append({"label": color, "cx": cx, "cy": cy, "d": d, "area": d * d,
+                       "bbox": [cx - d / 2, cy - d / 2, cx + d / 2, cy + d / 2]})
+    shapes.sort(key=lambda s: s["cx"])
+    return (W, H), shapes
+
+
+def _render_scene(size, shapes):
+    from PIL import Image, ImageDraw
+    img = Image.new("RGB", size, (245, 245, 245))
+    d = ImageDraw.Draw(img)
+    for s in shapes:
+        d.ellipse(s["bbox"], fill=_SHAPE_COLORS[s["label"]])
+    return img
+
+
+def make_shapes_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Scenes carrying GT for spatial / depth / subject_fixation simultaneously."""
+    out = []
+    for i in range(n):
+        size, shapes = _shape_scene(i)
+        # spatial: left_of for every left→right pair
+        triples = []
+        for a, b in _it.combinations(shapes, 2):  # already x-sorted → a left of b
+            triples.append([a["label"], "left_of", b["label"]])
+        # depth: bigger area = nearer
+        pairs = []
+        for a, b in _it.combinations(shapes, 2):
+            pairs.append({"a": a["label"], "b": b["label"],
+                          "a_is": "nearer" if a["area"] >= b["area"] else "farther"})
+        # subject: largest area
+        subj = max(shapes, key=lambda s: s["area"])
+        gt = {"triples": triples, "pairs": pairs,
+              "label": subj["label"], "box": subj["bbox"], "fmt": "xyxy"}
+        img = _render_scene(size, shapes)
+        out.append(GTSample(img, "Analyze the colored shapes. Output only raw JSON.",
+                            gt, "shapes", f"shapes_{i}", size))
+    return out
+
+
+def _circle_polygon(cx, cy, d, n=16):
+    """Approximate a circle (diameter d, center cx,cy) as a flat pixel-coord
+    polygon [x1,y1,x2,y2,...] with n vertices."""
+    import math
+    r = d / 2.0
+    flat = []
+    for k in range(n):
+        ang = 2.0 * math.pi * k / n
+        flat.append(cx + r * math.cos(ang))
+        flat.append(cy + r * math.sin(ang))
+    return flat
+
+
+def make_segmentation_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Self-contained instance-segmentation GT: reuse the 3-circle shape scenes.
+    Each colored circle becomes one mask whose polygon is the circle approximated
+    by 16 vertices (label = color). Polygons are in PIXEL coords; the scorer
+    converts model polygons from NORM_0_1000 to pixels."""
+    out = []
+    for i in range(n):
+        size, shapes = _shape_scene(i)
+        masks = [{"label": s["label"],
+                  "polygon_pixels": _circle_polygon(s["cx"], s["cy"], s["d"], n=16)}
+                 for s in shapes]
+        img = _render_scene(size, shapes)
+        out.append(GTSample(img, "Segment the colored shapes as labeled polygons. Output only raw JSON.",
+                            {"masks": masks}, "segmentation", f"seg_{i}", size))
+    return out
+
+
+def make_outline_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Self-contained: reuse the 3-circle synthetic scene. GT outline = the largest
+    circle approximated as a 16-point polygon (pixels), label = its color."""
+    out = []
+    for i in range(n):
+        size, shapes = _shape_scene(i)
+        main = max(shapes, key=lambda s: s["area"])        # largest = main object
+        poly = _circle_polygon(main["cx"], main["cy"], main["d"], 16)
+        gt = {"outline": poly, "label": main["label"], "bbox": main["bbox"], "fmt": "xyxy"}
+        img = _render_scene(size, shapes)
+        out.append(GTSample(img, "Trace the main object's outline. Output only raw JSON.",
+                            gt, "outline_association", f"outline_{i}", size))
+    return out
+
+
+_BOX3D_COLORS = {"red": (220, 40, 40), "green": (40, 175, 40), "blue": (50, 50, 225)}
+_BOX3D_NAMES = ["red", "green", "blue"]
+
+
+def _box3d_scene(i: int):
+    """Deterministic 2-3 colored boxes at known ground (x,z) positions.
+
+    GT convention (normalized 0..1): bbox3d = [x, y, z, w, h, l, yaw] with
+    x = left-right ground position, z = depth (0 near .. 1 far), y = 0 (on the
+    floor), (w,h,l) the box footprint width / height / length, yaw = 0. The GT is
+    exact-by-construction; the render is a simplified ground-plane 3D proxy.
+    """
+    import math
+    import itertools
+    W, H = 480, 360
+    n_boxes = 2 + (i % 2)                       # 2 or 3 boxes
+    names = _BOX3D_NAMES[:n_boxes]
+    perm = list(itertools.permutations(range(n_boxes)))[i % math.factorial(n_boxes)]
+    x_slots = [0.2, 0.5, 0.8][:n_boxes]
+    z_slots = [0.25, 0.55, 0.85][:n_boxes]
+    objects, draw = [], []
+    for k, color in enumerate(names):
+        x = x_slots[perm[k] % n_boxes]
+        z = z_slots[k]                          # increasing depth per index
+        w = 0.16 + 0.04 * ((i + k) % 3)         # footprint width
+        l = 0.14
+        h = 0.22 + 0.03 * (k % 2)               # box height
+        objects.append({"class": color,
+                        "bbox3d": [round(x, 4), 0.0, round(z, 4),
+                                   round(w, 4), round(h, 4), round(l, 4), 0.0]})
+        draw.append((color, x, z, w, h))
+    return (W, H), objects, draw
+
+
+def _render_box3d_scene(size, draw):
+    """Perspective proxy: nearer (small z) boxes drawn lower in frame and larger."""
+    from PIL import Image, ImageDraw
+    W, H = size
+    img = Image.new("RGB", size, (235, 235, 240))
+    d = ImageDraw.Draw(img)
+    d.rectangle([0, int(H * 0.5), W, H], fill=(205, 200, 190))   # ground band
+    for color, x, z, w, h in sorted(draw, key=lambda t: t[2], reverse=True):  # far first
+        scale = 1.0 - 0.45 * z                                  # nearer = bigger
+        bw = w * W * scale
+        bh = h * H * scale
+        cx = x * W
+        cy = (0.5 + 0.45 * z) * H                               # nearer = lower
+        d.rectangle([cx - bw / 2, cy - bh, cx + bw / 2, cy],
+                    fill=_BOX3D_COLORS[color], outline=(20, 20, 20))
+    return img
+
+
+def make_3d_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Self-contained synthetic 3D scenes. GT exact-by-construction (proxy)."""
+    out = []
+    for i in range(n):
+        size, objects, draw = _box3d_scene(i)
+        img = _render_box3d_scene(size, draw)
+        out.append(GTSample(img, "Identify the 3D boxes. Output only raw JSON.",
+                            {"objects": objects}, "geometric_3d_object_id",
+                            f"box3d_{i}", size))
+    return out
+
+
+def make_camera_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Self-contained synthetic camera-roll set. A clear orientation cue (an upward
+    arrow over a horizon line) is drawn upright, then the whole image is rotated by a
+    KNOWN roll angle that varies by index; yaw=pitch=0 (a single 2D cue cannot
+    disambiguate yaw/pitch). GT = {"rotation": [0, 0, roll_deg]}.
+
+    NOTE: SIMPLIFIED proxy — this tests recovery of ROLL from a 2D cue only; it does
+    not exercise yaw/pitch (which would need a 3D scene). Reliable, no download.
+    """
+    from PIL import Image, ImageDraw
+
+    W, H = 480, 480
+    cx, cy = W / 2.0, H / 2.0
+    # deterministic spread of rolls across the wrapped range, indexed by sample
+    roll_table = [0, 15, 30, 45, 60, 90, -15, -30, -45, -60, -90, 120,
+                  -120, 150, 75, -75, 10, -10]
+    out = []
+    for i in range(n):
+        roll = float(roll_table[i % len(roll_table)])
+        base = Image.new("RGB", (W, H), (250, 250, 250))
+        d = ImageDraw.Draw(base)
+        d.line([(60, cy), (W - 60, cy)], fill=(60, 60, 60), width=6)               # horizon line
+        d.line([(cx, cy), (cx, 90)], fill=(200, 40, 40), width=8)                  # arrow shaft (points up)
+        d.polygon([(cx, 60), (cx - 22, 105), (cx + 22, 105)], fill=(200, 40, 40))  # arrow head
+        # Rotate scene by -roll about the centre (expand=False keeps size + GT stable):
+        # a positive camera roll (CW) rotates scene content CCW in the image.
+        img = base.rotate(-roll, resample=Image.BICUBIC, center=(cx, cy),
+                          fillcolor=(250, 250, 250), expand=False)
+        gt = {"rotation": [0.0, 0.0, roll]}
+        out.append(GTSample(img,
+                            "Estimate the camera rotation [yaw, pitch, roll]. Output only raw JSON.",
+                            gt, "camera_rotational_offset", f"camrot_{i}", (W, H)))
+    return out
+
+
+def make_gqa_samples(n: int = 200, split: str = "validation") -> list[GTSample]:
+    """Grounded-VQA GT (REAL, best-effort). Streams a VQA dataset; one sample per
+    (image, question, answers). The question is per-image and goes in
+    GTSample.prompt; gt = {"answers": [<gold strings>]}. Image is row["image"]
+    (a PIL image).
+
+    Repo ids are BEST-EFFORT — the maintainer must verify id/config/split:
+      primary : "lmms-lab/GQA"   (testdev_balanced / val splits; row has
+                                   "question" + "answer"; image under "image")
+      fallback: "HuggingFaceM4/VQAv2" (row has "question" + "answers"
+                                       list-of-dicts or list-of-strings)
+    The answer-field probing below tolerates both shapes.
+    """
+    # Script-based repos (HuggingFaceM4/VQAv2, lmms-lab/GQA) are rejected by modern
+    # `datasets`. Use PARQUET repos (verified format:parquet on the Hub), in order.
+    rows = None
+    for repo, sp in [("lmms-lab/VQAv2", split), ("merve/vqav2-small", "validation"),
+                     ("merve/vqav2-small", "train"), ("lmms-lab/OK-VQA", "val2014")]:
+        try:
+            rows = _hf_stream(repo, sp, n)
+            if rows:
+                break
+        except Exception:
+            continue
+    if not rows:
+        raise RuntimeError("no parquet VQA dataset streamable "
+                           "(tried lmms-lab/VQAv2, merve/vqav2-small, lmms-lab/OK-VQA)")
+    out = []
+    for i, r in enumerate(rows):
+        img = r.get("image")
+        question = str(r.get("question") or r.get("question_str") or "What is in this image?")
+        raw_ans = r.get("answers")
+        if raw_ans is None:
+            raw_ans = r.get("multiple_choice_answer") or r.get("answer")
+        if isinstance(raw_ans, dict):                 # {"answer": "x"} or value-map
+            raw_ans = raw_ans.get("answer") or list(raw_ans.values())
+        if isinstance(raw_ans, (list, tuple)):
+            answers = []
+            for a in raw_ans:
+                if isinstance(a, dict):               # VQAv2: [{"answer": "x"}, ...]
+                    a = a.get("answer", "")
+                if str(a).strip():
+                    answers.append(str(a))
+        elif raw_ans is not None and str(raw_ans).strip():
+            answers = [str(raw_ans)]
+        else:
+            answers = []
+        size = (img.width, img.height) if img is not None else (0, 0)
+        out.append(GTSample(img, question, {"answers": answers},
+                            "vit_accuracy_to_prompt", f"vqa_{i}", size))
+    return out
+
+
+def make_semantic_samples(n: int = 12, split=None) -> list[GTSample]:
+    """Self-contained colored-shapes scenes carrying GT semantic-association triples.
+
+    Reuses the deterministic 3-circle scene (`_shape_scene`). Associations are
+    derived purely from geometry so they are exact and reproducible:
+      * left->right ordering   -> (left, "left_of", right) AND (right, "right_of", left)
+      * adjacency (consecutive) -> (a, "near", b) for neighbouring shapes
+      * taxonomy                -> (color, "is_a", "circle") for every shape
+    GT shape: {"triples": [[a, relation, b], ...]} -- read directly by score_triples,
+    which does tolerant subject/object matching + normalized-exact predicate matching.
+    Relations are chosen so they round-trip cleanly through metrics._norm_pred
+    (left_of/right_of/near/is_a stay identical after normalization).
+    """
+    out = []
+    for i in range(n):
+        size, shapes = _shape_scene(i)  # sorted left->right
+        triples: list[list] = []
+        # ordering relations over every left->right pair (both directions)
+        for a, b in _it.combinations(shapes, 2):  # a is left of b
+            triples.append([a["label"], "left_of", b["label"]])
+            triples.append([b["label"], "right_of", a["label"]])
+        # adjacency ("near") for consecutive shapes in the x-ordering
+        for a, b in zip(shapes, shapes[1:]):
+            triples.append([a["label"], "near", b["label"]])
+        # taxonomic: each colored shape is a circle
+        for s in shapes:
+            triples.append([s["label"], "is_a", "circle"])
+        img = _render_scene(size, shapes)
+        out.append(GTSample(
+            img,
+            "List semantic associations between the shapes. Output only raw JSON.",
+            {"triples": triples}, "semantic_association", f"semassoc_{i}", size,
+        ))
+    return out
+
+
+def _style_font(sz=28):
+    from PIL import ImageFont
+    for name in ("DejaVuSans.ttf", "DejaVuSansMono.ttf"):
+        try:
+            return ImageFont.truetype(name, sz)
+        except Exception:
+            continue
+    try:
+        return ImageFont.load_default(size=sz)  # Pillow >= 10
+    except Exception:
+        return ImageFont.load_default()
+
+
+def _render_style_image(style: str, size=(320, 320)):
+    """Render a controllable, visually-distinguishable exemplar for each coarse style.
+    photo: smooth RGB gradient (photographic continuous tone). painting: soft color blobs
+    on canvas. sketch: black outlines on white. 3d_render: lit/shaded sphere. anime:
+    flat-shaded face with big eyes. other: a labelled fallback."""
+    from PIL import Image, ImageDraw
+    import math
+    W, H = size
+    cx, cy = W // 2, H // 2
+
+    if style == "photo":
+        img = Image.new("RGB", size, (0, 0, 0))
+        px = img.load()
+        for y in range(H):
+            for x in range(W):
+                px[x, y] = (int(40 + 180 * x / W), int(40 + 180 * y / H),
+                            int(120 + 100 * ((x + y) % 50) / 50))
+        return img
+
+    if style == "painting":
+        img = Image.new("RGB", size, (235, 225, 205))
+        d = ImageDraw.Draw(img)
+        for (bx, by), r, col in [((90, 90), 70, (200, 70, 60)),
+                                 ((210, 120), 60, (70, 110, 190)),
+                                 ((140, 220), 80, (90, 170, 90))]:
+            d.ellipse([bx - r, by - r, bx + r, by + r], fill=col)
+        return img
+
+    if style == "sketch":
+        img = Image.new("RGB", size, (255, 255, 255))
+        d = ImageDraw.Draw(img)
+        d.rectangle([cx - 70, cy - 70, cx + 70, cy + 70], outline=(0, 0, 0), width=3)
+        d.line([cx - 70, cy - 70, cx + 70, cy + 70], fill=(0, 0, 0), width=2)
+        d.line([cx + 70, cy - 70, cx - 70, cy + 70], fill=(0, 0, 0), width=2)
+        d.ellipse([cx - 40, cy - 40, cx + 40, cy + 40], outline=(0, 0, 0), width=2)
+        return img
+
+    if style == "3d_render":
+        img = Image.new("RGB", size, (245, 245, 250))
+        d = ImageDraw.Draw(img)
+        r = 90
+        for yy in range(cy - r, cy + r):
+            for xx in range(cx - r, cx + r):
+                dx, dy = (xx - cx) / r, (yy - cy) / r
+                if dx * dx + dy * dy <= 1.0:
+                    lx, ly = -0.5, -0.6
+                    nz = math.sqrt(max(0.0, 1.0 - dx * dx - dy * dy))
+                    shade = max(0.12, (-dx * lx - dy * ly + nz) / 1.7)
+                    v = int(60 + 195 * min(1.0, shade))
+                    d.point((xx, yy), fill=(v, int(v * 0.7), int(v * 0.5)))
+        return img
+
+    if style == "anime":
+        img = Image.new("RGB", size, (250, 240, 230))
+        d = ImageDraw.Draw(img)
+        d.ellipse([cx - 80, cy - 90, cx + 80, cy + 70], fill=(255, 224, 196),
+                  outline=(40, 30, 30), width=3)
+        for ex in (cx - 35, cx + 35):
+            d.ellipse([ex - 18, cy - 10, ex + 18, cy + 30], fill=(255, 255, 255),
+                      outline=(20, 20, 20), width=2)
+            d.ellipse([ex - 10, cy + 2, ex + 10, cy + 26], fill=(60, 110, 200))
+            d.ellipse([ex - 4, cy + 6, ex + 4, cy + 16], fill=(20, 20, 20))
+        d.polygon([(cx - 90, cy - 90), (cx - 30, cy - 110), (cx, cy - 80)], fill=(90, 60, 40))
+        return img
+
+    # "other" fallback
+    img = Image.new("RGB", size, (200, 200, 200))
+    ImageDraw.Draw(img).text((20, H // 2), "other", fill=(0, 0, 0), font=_style_font(28))
+    return img
+
+
+# Each rendered style implies a controlled (layout, symmetry) GT pair.
+_STYLE_LAYOUTS = {
+    "photo": ("rule_of_thirds", "none"),
+    "painting": ("scattered", "none"),
+    "sketch": ("centered", "radial"),
+    "3d_render": ("centered", "radial"),
+    "anime": ("centered", "vertical"),
+    "other": ("centered", "none"),
+}
+_STYLE_ORDER = ["photo", "painting", "sketch", "3d_render", "anime"]
+
+
+def make_style_samples(n: int = 10, split=None) -> list[GTSample]:
+    """Self-contained: render distinguishable styles we control. Cycles through
+    photo/painting/sketch/3d_render/anime. GT = {style, layout, symmetry}."""
+    out = []
+    for i in range(n):
+        style = _STYLE_ORDER[i % len(_STYLE_ORDER)]
+        layout, symmetry = _STYLE_LAYOUTS[style]
+        size = (320, 320)
+        img = _render_style_image(style, size)
+        gt = {"style": style, "layout": layout, "symmetry": symmetry}
+        out.append(GTSample(img, "Classify the visual style and structure. Output only raw JSON.",
+                            gt, "style_structural_awareness", f"style_{i}", size))
+    return out
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# REAL COCO instance segmentation GT (for segmentation / outline / subject) — parses
+# the official COCO annotations JSON directly (no script-dataset, no pycocotools) and
+# pulls images by URL. Replaces the synthetic colored-shape GT with real images.
+# ──────────────────────────────────────────────────────────────────────────────
+_COCO_CACHE: dict = {}
+_COCO_PERSON_CAT = 1     # COCO category_id for "person"
+
+
+def _coco_ann_file(name: str) -> str:
+    """Ensure `{cache_dir}/{name}` exists. ONE zip download extracts BOTH
+    instances_val2017.json and captions_val2017.json (they ship in the same
+    annotations_trainval2017.zip — extracting only one wastes the 241MB fetch)."""
+    import io
+    import os
+    import urllib.request
+    import zipfile
+
+    cache_dir = os.environ.get("HF_HOME") or os.environ.get("TMPDIR") or "/tmp"
+    os.makedirs(cache_dir, exist_ok=True)
+    path = os.path.join(cache_dir, name)
+    if not os.path.exists(path):
+        print(f"  downloading COCO val2017 annotations (~241MB, one-time) for {name} …")
+        zurl = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip"
+        zb = urllib.request.urlopen(zurl, timeout=600).read()
+        with zipfile.ZipFile(io.BytesIO(zb)) as z:
+            for member in ("instances_val2017.json", "captions_val2017.json"):
+                target = os.path.join(cache_dir, member)
+                if not os.path.exists(target):
+                    with z.open(f"annotations/{member}") as f:
+                        open(target, "wb").write(f.read())
+    return path
+
+
+def _coco_ann(kind: str = "instances") -> dict:
+    """Parsed-JSON cache for the COCO annotation files. `kind` is "instances" or
+    "captions". Keeps the existing _COCO_CACHE["ann"] key for instances."""
+    import json as _json
+
+    key = "ann" if kind == "instances" else f"ann_{kind}"
+    if key not in _COCO_CACHE:
+        with open(_coco_ann_file(f"{kind}_val2017.json"), encoding="utf-8") as f:
+            _COCO_CACHE[key] = _json.load(f)
+    return _COCO_CACHE[key]
+
+
+def _coco_instances(n: int) -> list:
+    """Returns [(image, (W,H), image_id, [{label, polygon_pixels, box_xyxy, area}])].
+    Downloads + caches instances_val2017.json (~one-time) and the first `n` val images."""
+    import io
+    import urllib.request
+    from collections import defaultdict
+    from PIL import Image
+
+    key = f"inst_{n}"
+    if key in _COCO_CACHE:
+        return _COCO_CACHE[key]
+    data = _coco_ann("instances")
+    cats = {c["id"]: c["name"] for c in data["categories"]}
+    imgs = {im["id"]: im for im in data["images"]}
+    anns = defaultdict(list)
+    for a in data["annotations"]:
+        anns[a["image_id"]].append(a)
+
+    out = []
+    for iid in list(imgs):
+        if len(out) >= n:
+            break
+        info = imgs[iid]
+        try:
+            raw = urllib.request.urlopen(
+                f"http://images.cocodataset.org/val2017/{info['file_name']}", timeout=60).read()
+            img = Image.open(io.BytesIO(raw)).convert("RGB")
+        except Exception:
+            continue
+        objs = []
+        for a in anns[iid]:
+            seg = a.get("segmentation")
+            if a.get("iscrowd") or not isinstance(seg, list) or not seg:
+                continue                                   # skip RLE / crowd
+            poly = [float(v) for v in seg[0]]
+            if len(poly) < 6:
+                continue
+            x, y, w, h = a["bbox"]
+            objs.append({"label": cats.get(a["category_id"], "object"),
+                         "polygon_pixels": poly, "box_xyxy": [x, y, x + w, y + h],
+                         "area": float(a.get("area", w * h))})
+        if objs:
+            out.append((img, (img.width, img.height), f"coco_{iid}", objs))
+    _COCO_CACHE[key] = out
+    return out
+
+
+def load_coco_segmentation(n: int = 24, split=None) -> list[GTSample]:
+    return [GTSample(img, "Segment every object as a labeled polygon.",
+                     {"masks": [{"label": o["label"], "polygon_pixels": o["polygon_pixels"]}
+                                for o in objs]},
+                     "segmentation", iid, size)
+            for (img, size, iid, objs) in _coco_instances(n)]
+
+
+def load_coco_outline(n: int = 24, split=None) -> list[GTSample]:
+    out = []
+    for (img, size, iid, objs) in _coco_instances(n):
+        big = max(objs, key=lambda o: o["area"])
+        out.append(GTSample(img, "Trace the main object's outline.",
+                            {"outline": big["polygon_pixels"], "label": big["label"]},
+                            "outline_association", iid, size))
+    return out
+
+
+def load_coco_subject(n: int = 24, split=None) -> list[GTSample]:
+    out = []
+    for (img, size, iid, objs) in _coco_instances(n):
+        big = max(objs, key=lambda o: o["area"])
+        out.append(GTSample(img, "Identify the primary subject.",
+                            {"label": big["label"], "box": big["box_xyxy"], "fmt": "xyxy"},
+                            "subject_fixation", iid, size))
+    return out
+
+
+# ── multi-person slice (fusion-tier validation GT) ────────────────────────────
+def _select_multi_person_ids(ann: dict, *, min_persons: int = 2, max_persons: int = 6,
+                             min_person_area_frac: float = 0.005,
+                             require_nonperson: bool = False) -> list:
+    """Image ids with TRUSTWORTHY multi-person GT: min..max non-crowd persons, no
+    crowd-person annotation anywhere in the image (a crowd RLE blob means "many
+    unlabeled people" — the count GT becomes untrustworthy), and no tiny background
+    persons (< min_person_area_frac of the image). Deliberately a CLEAN slice; the
+    bias is stated in every validation report. Pure filter over the parsed
+    annotations — no network, testable with a fake ann dict."""
+    from collections import defaultdict
+
+    imgs = {im["id"]: im for im in ann["images"]}
+    per_img = defaultdict(list)
+    for a in ann["annotations"]:
+        per_img[a["image_id"]].append(a)
+    out = []
+    for iid, image_anns in per_img.items():
+        info = imgs.get(iid)
+        if info is None:
+            continue
+        wh = float(info["width"] * info["height"]) or 1.0
+        persons = [a for a in image_anns if a["category_id"] == _COCO_PERSON_CAT]
+        if any(a.get("iscrowd") for a in persons):
+            continue
+        if not (min_persons <= len(persons) <= max_persons):
+            continue
+        if any(float(a.get("area", 0.0)) < min_person_area_frac * wh for a in persons):
+            continue
+        if require_nonperson and not any(
+                a["category_id"] != _COCO_PERSON_CAT and not a.get("iscrowd")
+                for a in image_anns):
+            continue
+        out.append(iid)
+    out.sort()      # deterministic selection order
+    return out
+
+
+def _multi_person_gt(image_anns: list, cats: dict) -> dict:
+    """Shape one image's annotations into the fusion GT. Keeps ALL instances and ALL
+    polygon parts per annotation — occluded people are routinely split into 2+
+    polygons; the first-polygon-only rule used by _coco_instances would corrupt
+    person masks on exactly this slice."""
+    persons, objects = [], []
+    for a in image_anns:
+        if a.get("iscrowd"):
+            continue
+        seg = a.get("segmentation")
+        polys = ([[float(v) for v in part] for part in seg
+                  if isinstance(part, list) and len(part) >= 6]
+                 if isinstance(seg, list) else [])
+        x, y, w, h = a["bbox"]
+        rec = {"ann_id": a["id"], "box_xyxy": [x, y, x + w, y + h],
+               "polygons": polys, "area": float(a.get("area", w * h))}
+        if a["category_id"] == _COCO_PERSON_CAT:
+            persons.append(rec)
+        else:
+            objects.append(dict(rec, label=cats.get(a["category_id"], "object")))
+    return {"persons": persons, "objects": objects, "n_persons": len(persons)}
+
+
+def load_coco_multi_person(n: int = 24, split=None, *, min_persons: int = 2,
+                           max_persons: int = 6, min_person_area_frac: float = 0.005,
+                           require_nonperson: bool = False) -> list[GTSample]:
+    """Clean 2-6-person COCO slice for fusion validation. GT retains all instances +
+    all polygon parts; the 5 human captions ride in meta["captions"]. Filtering runs
+    over the cached annotations BEFORE any image download."""
+    import io
+    import urllib.request
+    from collections import defaultdict
+    from PIL import Image
+
+    key = (f"multi_{n}_{min_persons}_{max_persons}_{min_person_area_frac}"
+           f"_{require_nonperson}")
+    if key in _COCO_CACHE:
+        return _COCO_CACHE[key]
+    ann = _coco_ann("instances")
+    cap_ann = _coco_ann("captions")
+    cats = {c["id"]: c["name"] for c in ann["categories"]}
+    imgs = {im["id"]: im for im in ann["images"]}
+    per_img = defaultdict(list)
+    for a in ann["annotations"]:
+        per_img[a["image_id"]].append(a)
+    caps = defaultdict(list)
+    for c in cap_ann["annotations"]:
+        caps[c["image_id"]].append(str(c["caption"]).strip())
+
+    out = []
+    for iid in _select_multi_person_ids(
+            ann, min_persons=min_persons, max_persons=max_persons,
+            min_person_area_frac=min_person_area_frac,
+            require_nonperson=require_nonperson):
+        if len(out) >= n:
+            break
+        info = imgs[iid]
+        try:
+            raw = urllib.request.urlopen(
+                f"http://images.cocodataset.org/val2017/{info['file_name']}",
+                timeout=60).read()
+            img = Image.open(io.BytesIO(raw)).convert("RGB")
+        except Exception:
+            continue
+        gt = _multi_person_gt(per_img[iid], cats)
+        out.append(GTSample(img, "Fuse the scene into entities, relations, and counts.",
+                            gt, "fusion_scene", f"coco_{iid}",
+                            (img.width, img.height), meta={"captions": caps.get(iid, [])}))
+    _COCO_CACHE[key] = out
+    return out
+
+
+def load_coco_multi_person_rich(n: int = 24, split=None) -> list[GTSample]:
+    """Multi-person images that ALSO contain a non-person object (relation richness)."""
+    return load_coco_multi_person(n, split, require_nonperson=True)
+
+
+DATASET_REGISTRY: dict[str, Callable[..., list[GTSample]]] = {
+    "imagenet_val": load_imagenet_val,
+    "coco_detection": load_coco_detection,
+    "coco_segmentation": load_coco_segmentation,
+    "coco_outline": load_coco_outline,
+    "coco_subject": load_coco_subject,
+    "coco_multi_person": load_coco_multi_person,
+    "coco_multi_person_rich": load_coco_multi_person_rich,
+    "textvqa": load_textvqa,
+    "datatype_synth": make_datatype_samples,
+    "shapes_synth": make_shapes_samples,
+    "segmentation_synth": make_segmentation_samples,
+    "outline_synth": make_outline_samples,
+    "boxes3d_synth": make_3d_samples,
+    "camera_rot_synth": make_camera_samples,
+    "gqa": make_gqa_samples,
+    "semantic_synth": make_semantic_samples,
+    "style_synth": make_style_samples,
+}
+
+
+def load_gt(dataset_key: str, n: int = 200, split: str = "validation",
+            dataset: str = "full") -> list[GTSample]:
+    """Top-level GT loader. dataset='smoke' uses the packaged offline set."""
+    if dataset == "smoke" or dataset_key in ("", "smoke"):
+        # caller passes the category as dataset_key for smoke
+        return smoke_samples(dataset_key, n)
+    loader = DATASET_REGISTRY.get(dataset_key)
+    if loader is None:
+        raise KeyError(f"no loader for dataset {dataset_key!r}. known: {list(DATASET_REGISTRY)}")
+    return loader(n=n, split=split)

qwen_test_runner/vision/derive.py

@@ -0,0 +1,363 @@
+"""
+derive.py — the deterministic "semantic engine": derive INTEGRATE tasks from the
+solidification primitives (detector boxes, segmentation masks, a relative depth map,
+optional saliency), with NO model. Pure numpy + stdlib; OpenCV is used lazily only for
+the outline contour.
+
+Every function returns the exact JSON shape of its `tasks_vision` task, so the output
+validates against the task's registry Pydantic model and scores through the existing
+`score_vision_sample`. Boxes/polygons are in whatever coordinate space the caller passes
+in (pixels for real specialists) — coord-space normalization is done by the adapter layer,
+not here.
+
+Primitive conventions (documented, unit-tested):
+  box   = [x1, y1, x2, y2]  (x right, y DOWN)
+  mask  = HxW bool ndarray
+  depth = HxW float ndarray, relative/ordinal. Depth-Anything convention: HIGHER = NEARER
+          (disparity-like). Pass higher_is_nearer=False for metric depth (smaller = nearer).
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Optional, Sequence
+
+import numpy as np
+
+# tasks_vision closed vocabularies (kept in sync with the registry)
+_DATATYPE_VALUES = ("json", "yaml", "markdown", "csv", "toml", "xml", "code", "plaintext")
+
+
+# ── geometry helpers ─────────────────────────────────────────────────────────
+def _centroid(b):
+    return (0.5 * (b[0] + b[2]), 0.5 * (b[1] + b[3]))
+
+
+def _area(b):
+    return max(0.0, b[2] - b[0]) * max(0.0, b[3] - b[1])
+
+
+def _iou(a, b):
+    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
+    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
+    iw, ih = max(0.0, ix2 - ix1), max(0.0, iy2 - iy1)
+    inter = iw * ih
+    u = _area(a) + _area(b) - inter
+    return inter / u if u > 0 else 0.0
+
+
+def _contains(outer, inner, frac=0.85):
+    """True if `inner` is (mostly) inside `outer` — inter/area(inner) >= frac."""
+    ix1, iy1 = max(outer[0], inner[0]), max(outer[1], inner[1])
+    ix2, iy2 = min(outer[2], inner[2]), min(outer[3], inner[3])
+    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
+    ai = _area(inner)
+    return ai > 0 and inter / ai >= frac
+
+
+def _uniq_labels(labels):
+    """Disambiguate duplicate label strings: person, person -> person_1, person_2."""
+    seen, counts = {}, {}
+    for l in labels:
+        counts[l] = counts.get(l, 0) + 1
+    out, running = [], {}
+    for l in labels:
+        if counts[l] == 1:
+            out.append(l)
+        else:
+            running[l] = running.get(l, 0) + 1
+            out.append(f"{l}_{running[l]}")
+    return out
+
+
+# ── #8 structural_spatial_awareness ──────────────────────────────────────────
+def spatial_relations(boxes: Sequence[dict], depth: Optional[np.ndarray] = None,
+                      higher_is_nearer: bool = True, max_items: int = 12) -> dict:
+    """boxes: [{label, box:[x1,y1,x2,y2], score?}]. Emits projective relations
+    (left_of/right_of/above/below), containment (inside), and — if a depth map is given —
+    in_front_of/behind. Returns {"relations":[{subject,predicate,object}]}."""
+    labs = _uniq_labels([str(b["label"]) for b in boxes])
+    bxs = [b["box"] for b in boxes]
+    n = len(bxs)
+    # per-box depth (median over the box region) if a map is provided
+    box_depth = None
+    if depth is not None and n:
+        H, W = depth.shape[:2]
+        box_depth = []
+        for b in bxs:
+            x1, y1, x2, y2 = (int(round(b[0])), int(round(b[1])), int(round(b[2])), int(round(b[3])))
+            x1, y1 = max(0, x1), max(0, y1)
+            x2, y2 = min(W, max(x1 + 1, x2)), min(H, max(y1 + 1, y2))
+            patch = depth[y1:y2, x1:x2]
+            box_depth.append(float(np.median(patch)) if patch.size else 0.0)
+
+    rels, seen = [], set()   # seen holds (subject, predicate, object) triples
+    # order pairs by centroid distance so the nearest, most meaningful pairs win the budget
+    cents = [_centroid(b) for b in bxs]
+    pairs = [(i, j) for i in range(n) for j in range(n) if i != j]
+    pairs.sort(key=lambda ij: (cents[ij[0]][0] - cents[ij[1]][0]) ** 2
+               + (cents[ij[0]][1] - cents[ij[1]][1]) ** 2)
+
+    def _emit(si, pred, oi):
+        t = (labs[si], pred, labs[oi])
+        if t not in seen:
+            seen.add(t)
+            rels.append({"subject": labs[si], "predicate": pred, "object": labs[oi]})
+
+    for i, j in pairs:
+        if len(rels) >= max_items:
+            break
+        a, ca, cb = bxs[i], cents[i], cents[j]
+        if _contains(bxs[j], a):                          # containment first (most specific)
+            _emit(i, "inside", j)
+            continue
+        dx, dy = cb[0] - ca[0], cb[1] - ca[1]
+        if abs(dx) >= abs(dy):
+            _emit(i, "left_of" if dx > 0 else "right_of", j)   # a left_of b when a.x < b.x
+        else:
+            _emit(i, "above" if dy > 0 else "below", j)        # y DOWN: a above b when a.y < b.y
+    # depth relations — a pair can carry BOTH a projective and a depth relation
+    if box_depth is not None:
+        rng = (max(box_depth) - min(box_depth)) or 1.0
+        for i, j in pairs:
+            if len(rels) >= max_items:
+                break
+            d = (box_depth[i] - box_depth[j]) / rng
+            if abs(d) < 0.15:
+                continue
+            a_nearer = (d > 0) if higher_is_nearer else (d < 0)
+            _emit(i, "in_front_of" if a_nearer else "behind", j)
+    return {"relations": rels}
+
+
+# ── #7 depth_analysis (ordering) ─────────────────────────────────────────────
+def depth_scalars(entities: Sequence[dict], depth: np.ndarray,
+                  higher_is_nearer: bool = True) -> list[float]:
+    """Continuous per-entity NEARNESS in [0,1] (bigger = nearer): median relative depth
+    over each entity's mask (preferred) or box, min-max normalized across the entities.
+    This is the scalar core of `depth_order`, exposed so the fusion tier can keep the
+    continuous signal instead of only the categorical nearer/farther/same."""
+    if not entities:
+        return []
+    H, W = depth.shape[:2]
+    vals = []
+    for e in entities:
+        if e.get("mask") is not None:
+            m = np.asarray(e["mask"], dtype=bool)
+            v = float(np.median(depth[m])) if m.any() else 0.0
+        else:
+            b = e["box"]
+            x1, y1 = max(0, int(b[0])), max(0, int(b[1]))
+            x2, y2 = min(W, int(b[2])), min(H, int(b[3]))
+            patch = depth[y1:max(y1 + 1, y2), x1:max(x1 + 1, x2)]
+            v = float(np.median(patch)) if patch.size else 0.0
+        vals.append(v)
+    # normalize to [0,1] for a stable "same" tolerance
+    lo, hi = min(vals), max(vals)
+    rng = (hi - lo) or 1.0
+    norm = [(v - lo) / rng for v in vals]
+    # nearness score: bigger = nearer
+    return norm if higher_is_nearer else [1 - x for x in norm]
+
+
+def depth_order(entities: Sequence[dict], depth: np.ndarray,
+                higher_is_nearer: bool = True, same_tol: float = 0.08,
+                max_items: int = 12) -> dict:
+    """entities: [{label, mask:HxW bool}] (preferred) or [{label, box}]. Samples the
+    RELATIVE depth over each entity (mask median, foreground-robust), orders them, and
+    emits {"nearest","farthest","relative_depth":[{a,b,a_is}]}."""
+    if not entities:
+        return {"nearest": "", "farthest": "", "relative_depth": []}
+    labs = _uniq_labels([str(e["label"]) for e in entities])
+    near = depth_scalars(entities, depth, higher_is_nearer)
+    order = sorted(range(len(labs)), key=lambda i: -near[i])
+    out = {"nearest": labs[order[0]], "farthest": labs[order[-1]], "relative_depth": []}
+    for i in range(len(labs)):
+        for j in range(i + 1, len(labs)):
+            if len(out["relative_depth"]) >= max_items:
+                return out
+            d = near[i] - near[j]
+            a_is = "same" if abs(d) < same_tol else ("nearer" if d > 0 else "farther")
+            out["relative_depth"].append({"a": labs[i], "b": labs[j], "a_is": a_is})
+    return out
+
+
+# ── #9 subject_fixation ──────────────────────────────────────────────────────
+def subject_scores(boxes: Sequence[dict], image_size,
+                   saliency: Optional[np.ndarray] = None) -> list[float]:
+    """Per-box subject score (saliency-PRIMARY, area×centrality tie-break) for EVERY
+    box — the scoring core of `subject_fixation`, exposed so the fusion tier can keep
+    the full ranking instead of only the winner."""
+    W, H = image_size
+    cx, cy = W / 2.0, H / 2.0
+    diag = (W ** 2 + H ** 2) ** 0.5 or 1.0
+
+    def score(b):
+        bx = b["box"]
+        area = _area(bx) / (W * H + 1e-9)
+        ctr = _centroid(bx)
+        centrality = 1.0 - (((ctr[0] - cx) ** 2 + (ctr[1] - cy) ** 2) ** 0.5) / diag
+        geo = area * (0.5 + 0.5 * centrality)
+        if saliency is not None:
+            x1, y1 = max(0, int(bx[0])), max(0, int(bx[1]))
+            x2, y2 = min(int(saliency.shape[1]), int(bx[2])), min(int(saliency.shape[0]), int(bx[3]))
+            patch = saliency[y1:max(y1 + 1, y2), x1:max(x1 + 1, x2)]
+            sal = float(patch.mean()) if patch.size else 0.0
+            return sal + 0.01 * geo          # saliency primary, geometry breaks ties
+        return geo
+
+    return [score(b) for b in boxes]
+
+
+def subject_fixation(boxes: Sequence[dict], image_size, saliency: Optional[np.ndarray] = None) -> dict:
+    """Saliency-PRIMARY (mean saliency inside each box), area×centrality tie-break.
+    image_size = (W, H). Falls back to the largest box, then whole-image. Returns
+    {"primary_subject":{"label","box"}}."""
+    W, H = image_size
+    if not boxes:
+        return {"primary_subject": {"label": "scene", "box": [0.0, 0.0, float(W), float(H)]}}
+    scores = subject_scores(boxes, image_size, saliency)
+    best = boxes[max(range(len(boxes)), key=scores.__getitem__)]
+    return {"primary_subject": {"label": str(best["label"]),
+                                "box": [float(v) for v in best["box"]]}}
+
+
+# ── #10 outline_association (mask → contour polygon) ─────────────────────────
+def outline_polygon(mask: np.ndarray, label: str, max_points: int = 128) -> dict:
+    """SAM2 mask → closed outline polygon, flat [x1,y1,x2,y2,...]. Pure-numpy row-scan:
+    trace the left boundary top→bottom, then the right boundary bottom→top (a closed loop).
+    No OpenCV dependency; the dense boundary is IoU-accurate (subsampled to max_points)."""
+    m = np.asarray(mask) > 0
+    if m.ndim != 2 or not m.any():
+        return {"outline": [], "label": str(label)}
+    ys = np.where(m.any(axis=1))[0]
+    left, right = [], []
+    for y in ys:
+        xs = np.where(m[y])[0]
+        left.append((float(xs[0]), float(y)))
+        right.append((float(xs[-1]), float(y)))
+    pts = left + right[::-1]                            # closed loop
+    if len(pts) > max_points:
+        idx = np.linspace(0, len(pts) - 1, max_points).round().astype(int)
+        pts = [pts[i] for i in idx]
+    flat = [v for xy in pts for v in xy]               # (x, y) interleaved
+    return {"outline": flat, "label": str(label)}
+
+
+# ── #6 style: symmetry + layout (the deterministic halves) ───────────────────
+def symmetry_scores(gray: np.ndarray) -> dict:
+    """Continuous L/R and T/B mirror correlations in [-1,1] — the scalar core of
+    `symmetry_axis`, exposed so the fusion tier keeps the magnitudes the categorical
+    label throws away. Returns {"lr": float, "tb": float}."""
+    g = np.asarray(gray, dtype=np.float64)
+    if g.ndim == 3:
+        g = g.mean(axis=2)
+    g = g - g.mean()
+
+    def corr(a, b):
+        a, b = a.ravel(), b.ravel()
+        da, db = np.linalg.norm(a), np.linalg.norm(b)
+        return float(a @ b / (da * db)) if da > 0 and db > 0 else 0.0
+
+    return {"lr": corr(g, g[:, ::-1]),     # left-right mirror  -> vertical-axis symmetry
+            "tb": corr(g, g[::-1, :])}     # top-bottom mirror  -> horizontal-axis symmetry
+
+
+def symmetry_axis(gray: np.ndarray, thresh: float = 0.80) -> str:
+    """Normalized-correlation of the image vs its L/R and T/B flips. Returns one of
+    horizontal/vertical/radial/none. 'vertical' = mirror across a vertical axis (L==R)."""
+    s = symmetry_scores(gray)
+    lr, tb = s["lr"], s["tb"]
+    v, h = lr >= thresh, tb >= thresh
+    if v and h:
+        return "radial"
+    if v:
+        return "vertical"
+    if h:
+        return "horizontal"
+    return "none"
+
+
+def layout_kind(boxes: Sequence[dict], image_size) -> str:
+    """From the box constellation: centered / rule_of_thirds / symmetric / scattered /
+    unknown."""
+    W, H = image_size
+    if not boxes:
+        return "unknown"
+    cents = np.array([_centroid(b["box"]) for b in boxes], dtype=float)
+    areas = np.array([_area(b["box"]) for b in boxes], dtype=float)
+    if len(boxes) == 1 or areas.max() > 0.5 * (W * H):
+        cx, cy = cents[int(areas.argmax())]
+        if abs(cx - W / 2) < 0.15 * W and abs(cy - H / 2) < 0.15 * H:
+            return "centered"
+    # left-right centroid symmetry about the vertical axis
+    xs = cents[:, 0] / W
+    if len(xs) >= 2 and abs(np.mean(xs) - 0.5) < 0.08 and np.std(xs) > 0.15:
+        return "symmetric"
+    # proximity to rule-of-thirds lines
+    thirds = np.array([1 / 3, 2 / 3])
+    nx = np.min(np.abs((cents[:, 0] / W)[:, None] - thirds[None, :]), axis=1)
+    ny = np.min(np.abs((cents[:, 1] / H)[:, None] - thirds[None, :]), axis=1)
+    if np.mean((nx < 0.08) | (ny < 0.08)) > 0.5:
+        return "rule_of_thirds"
+    return "scattered"
+
+
+# ── #12/#13 data-type recognition + re-serialization ─────────────────────────
+def detect_data_type(text: str) -> dict:
+    """OCR text -> {"data_type","confidence"}. Deterministic regex/heuristic with a
+    precedence order and a confidence proxy."""
+    t = (text or "").strip()
+    if not t:
+        return {"data_type": "plaintext", "confidence": 0.2}
+    scores = {k: 0.0 for k in _DATATYPE_VALUES}
+    if re.search(r"^\s*[{\[]", t) and re.search(r"[}\]]\s*$", t) and '"' in t:
+        scores["json"] += 0.9
+    if re.search(r"^\s*<\?xml|</[a-zA-Z]", t):
+        scores["xml"] += 0.9
+    if re.search(r"^\s*---\s*$", t, re.M) or re.search(r"^\s*[\w-]+:\s+\S", t, re.M):
+        scores["yaml"] += 0.6
+    if re.search(r"^\s*#{1,6}\s|\*\*|\[.+\]\(.+\)|^\s*[-*]\s", t, re.M):
+        scores["markdown"] += 0.6
+    if re.search(r"^\s*\[[\w.\-]+\]\s*$", t, re.M) or re.search(r'^\s*[\w.-]+\s*=\s*("|\d|\[)', t, re.M):
+        scores["toml"] += 0.6
+    if "\n" in t and all("," in ln for ln in t.splitlines()[:3] if ln.strip()):
+        scores["csv"] += 0.5
+    if re.search(r"\b(def|function|class|import|return|const|var|let)\b", t):
+        scores["code"] += 0.4
+    best = max(scores, key=scores.get)
+    conf = scores[best]
+    if conf <= 0.0:
+        return {"data_type": "plaintext", "confidence": 0.3}
+    return {"data_type": best, "confidence": round(min(0.99, conf), 2)}
+
+
+def _repair_json(t: str) -> str:
+    t = t.strip().strip("`")
+    t = re.sub(r"[“”]", '"', t)          # curly double quotes
+    t = re.sub(r"[‘’]", "'", t)          # curly single quotes
+    t = re.sub(r",\s*([}\]])", r"\1", t)           # trailing commas
+    return t
+
+
+def parse_data_type(text: str) -> tuple[dict, bool]:
+    """OCR text -> ({"data_type","content"}, parsed_ok). Deterministic parse with light
+    repair; content is a compact JSON string of the parsed structure. Returns
+    parsed_ok=False when nothing parsed (caller decides on a VLM fallback)."""
+    dt = detect_data_type(text)["data_type"]
+    raw = _repair_json(text or "")
+    parsed = None
+    try:
+        parsed = json.loads(raw)
+    except Exception:
+        try:
+            import yaml
+            y = yaml.safe_load(raw)
+            if isinstance(y, (dict, list)):
+                parsed = y
+        except Exception:
+            parsed = None
+    if parsed is None:
+        return {"data_type": dt, "content": ""}, False
+    return {"data_type": dt, "content": json.dumps(parsed, ensure_ascii=False, separators=(",", ":"))}, True

qwen_test_runner/vision/fuse.py

@@ -0,0 +1,758 @@
+"""
+fuse.py — the fusion engine: bind every captured signal into one FusedScene.
+
+Consumes a `solids_digest` (compact JSON-able snapshot of a `Solids` — detection
+boxes + scores + mask polygons + mask quality, continuous depth nearness, saliency
+scores, OCR with confidence, style/class/symmetry/layout) plus the caption structs
+(slot-registry JSON from the 9B structurer) and the raw captions, and emits the
+fused relational representation:
+
+  entities   — addressable instances (person_1, person_2, dog) with position grid,
+               offset-from-center, continuous depth + rank, saliency + rank, mask,
+               and STRATIFIED OWNED ATTRIBUTES (ownership decided by segmentation-
+               polygon containment with confidence + margin thresholds)
+  relations  — pairwise predicates + continuous dx/dy/distance/iou/depth-delta
+  counts     — synonym-collapsed instance counts
+  shared_basin — attributes NOT confidently assignable (never subjectively grouped),
+               with per-entity likelihoods and the reason
+  scene      — voted setting/style/mood + layout/symmetry/OCR/actions
+  quality    — retained confidences + grounding accounting + overall_confidence
+
+Pure numpy + stdlib + PIL (polygon rasterization) — torch-free, CPU-testable.
+The ONLY GPU dependency is upstream: the optional `attr_boxes` in the digest come
+from a second GroundingDINO pass over `phrases_for_grounding(...)` phrases.
+"""
+
+from __future__ import annotations
+
+import re
+from collections import Counter, defaultdict
+from typing import Optional
+
+import numpy as np
+
+from . import derive
+from .coords import CoordSpace
+from .fuse_schema import (FusedScene, MASK_POLY_MAX_POINTS, MAX_ENTITIES,
+                          MAX_RELATION_ENTITIES)
+from .metrics import _depluralize, _seg_poly_points, _seg_rasterize, labels_match
+from .specialists import box_to_space, poly_to_space
+from .strata import _content_tokens, classify_stratum, is_groundable
+
+# Containment rasterization grid (mask polygons are ≤64 points; 160² cells is
+# ample resolution for an ownership FRACTION).
+_GRID = 160
+
+# Depth-relation threshold on the normalized nearness delta — same magnitude the
+# spatial_relations engine uses on its normalized per-box depth deltas.
+_DEPTH_REL_TOL = 0.15
+
+# "near" relation threshold on centroid-distance / image-diagonal.
+_NEAR_DIST = 0.25
+
+# Positional-cue lexicon for caption-subject binding votes.
+_POS_LEFT = frozenset({"left", "leftmost"})
+_POS_RIGHT = frozenset({"right", "rightmost"})
+_POS_FRONT = frozenset({"front", "foreground", "nearest", "closest", "nearer", "closer"})
+_POS_BACK = frozenset({"behind", "background", "back", "farthest", "farther", "rear"})
+_POS_TALL = frozenset({"tall", "taller", "tallest"})
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Digest — the GPU→CPU handoff (also the durability/parquet payload)
+# ═════════════════════════════════════════════════════════════════════════════
+
+def solids_digest(s) -> dict:
+    """Compact, JSON-able, deterministic snapshot of a Solids. Retains the signals
+    the build_* task projections drop (mask quality, OCR conf, continuous nearness,
+    the full saliency ranking, symmetry magnitudes)."""
+    from .coords import BBox
+    W, H = s.size
+    nearness = (derive.depth_scalars(s.boxes, s.depth, s.depth_higher_is_nearer)
+                if (s.depth is not None and s.boxes) else None)
+    sal = derive.subject_scores(s.boxes, s.size, s.saliency) if s.boxes else []
+    boxes = []
+    for i, b in enumerate(s.boxes):
+        mask = b.get("mask")
+        poly = (derive.outline_polygon(mask, b["label"],
+                                       max_points=MASK_POLY_MAX_POINTS)["outline"]
+                if mask is not None else None)
+        # GDINO emits unclamped boxes (border objects go past the frame) — clip
+        # once at the digest boundary so all downstream geometry is in-range
+        clipped = BBox(*[float(v) for v in b["box"]]).clip((W, H)).as_list()
+        boxes.append({
+            "label": str(b["label"]),
+            "box": clipped,
+            "score": float(b.get("score", 1.0)),
+            "area_px": derive._area(clipped),
+            "sal": float(sal[i]) if i < len(sal) else 0.0,
+            "nearness": (round(float(nearness[i]), 4) if nearness is not None else None),
+            "mask_poly": poly or None,
+            "mask_quality": (float(b["mask_score"]) if b.get("mask_score") is not None
+                             else None),
+        })
+    ocr = {"full_text": "", "lines": []}
+    if s.ocr:
+        ocr["full_text"] = str(s.ocr.get("full_text", ""))
+        for ln in s.ocr.get("lines", []):
+            q = ln.get("box")
+            flat = ([min(max(float(v), 0.0), float(W if i % 2 == 0 else H))
+                     for xy in q for i, v in enumerate(xy)] if q else None)
+            ocr["lines"].append({"text": str(ln["text"]),
+                                 "quad": flat,
+                                 "conf": (float(ln["conf"]) if ln.get("conf") is not None
+                                          else None)})
+    attr_boxes = []
+    for a in getattr(s, "attr_boxes", []):
+        a = dict(a)
+        a["box"] = BBox(*[float(v) for v in a["box"]]).clip((W, H)).as_list()
+        attr_boxes.append(a)
+    return {
+        "size": [int(W), int(H)],
+        "boxes": boxes,
+        "attr_boxes": attr_boxes,
+        "class_top": [{"label": str(c["label"]), "score": float(c["score"])}
+                      for c in (s.class_top or [])],
+        "style": s.style,
+        "ocr": ocr,
+        "symmetry": (derive.symmetry_scores(s.gray) if s.gray is not None else None),
+        "layout": derive.layout_kind(s.boxes, s.size),
+        "higher_is_nearer": bool(s.depth_higher_is_nearer),
+    }
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Caption-side collection + cross-source merge
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _attr_key(text: str) -> frozenset:
+    """Dedup key: depluralized content-token set (raw + depluralized forms so the
+    crude depluralizer can't split 'dress'/'dres')."""
+    toks = _content_tokens(text)
+    return frozenset(t for tok in toks for t in (tok, _depluralize(tok)))
+
+
+_HEAD_SPLIT_RE = re.compile(r"\b(?:in|on|at|with|of|to|wearing|holding)\b")
+
+
+def _subject_head(name: str) -> str:
+    """Head noun = last content token BEFORE the first post-modifier ("woman in
+    red" → woman, "person on a bench" → person); falls back to the full-name head
+    when the pre-modifier part has no content tokens."""
+    pre = _HEAD_SPLIT_RE.split((name or "").lower(), 1)[0]
+    toks = _content_tokens(pre)
+    if toks:
+        return toks[-1]
+    toks = _content_tokens(name)
+    return toks[-1] if toks else ""
+
+
+def _collect_merged(caption_structs: dict) -> tuple:
+    """caption_structs: {source: struct-or-None}. Returns (merged_attrs, actions,
+    votes) where merged_attrs = [{text, key, sources, consensus, stratum,
+    parents:{source: subject_name}}] (cross-source dedup: token-set equal-or-subset
+    → canonical = longest text; provenance kept). Subjects are NEVER merged across
+    sources by name — merging happens only through binding downstream."""
+    sources = [k for k, v in caption_structs.items() if v]
+    n_src = max(1, len(sources))
+
+    raw_items = []
+    actions = []
+    votes = {"setting": Counter(), "style": Counter(), "mood": {}}
+    for src in sources:
+        st = caption_structs[src]
+        for subj in (st.get("subjects") or []):
+            name = str(subj.get("name") or "").strip()
+            for att in (subj.get("attributes") or []):
+                att = str(att).strip()
+                if att:
+                    raw_items.append({"text": att, "source": src, "subject": name})
+        for act in (st.get("actions") or []):
+            act = str(act).strip()
+            if act:
+                actions.append({"text": act, "source": src})
+        if st.get("setting"):
+            votes["setting"][str(st["setting"])] += 1
+        if st.get("style"):
+            votes["style"][str(st["style"])] += 1
+        if st.get("mood"):
+            votes["mood"][src] = str(st["mood"])
+
+    # merge: iterate longest-token-set first so merged records are supersets
+    raw_items.sort(key=lambda it: (-len(_attr_key(it["text"])), it["text"], it["source"]))
+    merged = []
+    for it in raw_items:
+        key = _attr_key(it["text"])
+        if not key:
+            continue
+        home = next((m for m in merged if key <= m["key"] or m["key"] <= key), None)
+        if home is None:
+            merged.append({"text": it["text"], "key": key, "sources": [it["source"]],
+                           "parents": {it["source"]: it["subject"]}})
+        else:
+            home["key"] = home["key"] | key
+            if len(it["text"]) > len(home["text"]):
+                home["text"] = it["text"]
+            if it["source"] not in home["sources"]:
+                home["sources"].append(it["source"])
+            home["parents"].setdefault(it["source"], it["subject"])
+    for m in merged:
+        m["sources"] = sorted(m["sources"])
+        m["consensus"] = round(len(m["sources"]) / n_src, 4)
+        m["stratum"] = classify_stratum(m["text"])
+
+    # actions: same dedup, no parents
+    actions.sort(key=lambda it: (-len(_attr_key(it["text"])), it["text"], it["source"]))
+    merged_acts = []
+    for it in actions:
+        key = _attr_key(it["text"])
+        if not key:
+            continue
+        home = next((m for m in merged_acts if key <= m["key"] or m["key"] <= key), None)
+        if home is None:
+            merged_acts.append({"text": it["text"], "key": key, "sources": [it["source"]]})
+        else:
+            home["key"] = home["key"] | key
+            if len(it["text"]) > len(home["text"]):
+                home["text"] = it["text"]
+            if it["source"] not in home["sources"]:
+                home["sources"].append(it["source"])
+    for m in merged_acts:
+        m["sources"] = sorted(m["sources"])
+        m["consensus"] = round(len(m["sources"]) / n_src, 4)
+
+    return merged, merged_acts, votes
+
+
+def phrases_for_grounding(caption_structs: dict) -> list:
+    """The canonical phrases the GPU grounding pass should box — merged attribute
+    texts whose stratum is GROUNDABLE, emitted stripped-lowercase (ground_phrases
+    lowercases anyway; matching its normalization keeps the downstream
+    phrase↔attribute lookup exact)."""
+    merged, _, _ = _collect_merged(caption_structs)
+    return sorted({m["text"].strip().lower() for m in merged
+                   if is_groundable(m["stratum"])})
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Geometry: entities, containment, relations
+# ═════════════════════════════════════════════════════════════════════════════
+
+def _grid_cell(cx: float, cy: float, W: float, H: float) -> str:
+    col = "left" if cx < W / 3 else ("center" if cx < 2 * W / 3 else "right")
+    row = "upper" if cy < H / 3 else ("middle" if cy < 2 * H / 3 else "lower")
+    return f"{row} {col}"
+
+
+def _build_entities(digest: dict, dedup_iou: float) -> list:
+    """Dedup detector double-boxes, cap by saliency, order left-to-right, and
+    assign _uniq_labels ids. Returns internal entity dicts (pixel space)."""
+    boxes = [dict(b) for b in digest["boxes"]]
+    kept = []
+    for b in sorted(boxes, key=lambda b: (-b["score"], b["box"][0])):
+        if any(derive._iou(b["box"], k["box"]) >= dedup_iou
+               and labels_match(b["label"], k["label"]) for k in kept):
+            continue
+        kept.append(b)
+    kept.sort(key=lambda b: -b["sal"])
+    kept = kept[:MAX_ENTITIES]
+    for rank, b in enumerate(kept, 1):
+        b["sal_rank"] = rank
+    kept.sort(key=lambda b: (0.5 * (b["box"][0] + b["box"][2]), b["box"][1]))
+    ids = derive._uniq_labels([b["label"] for b in kept])
+    for b, eid in zip(kept, ids):
+        b["id"] = eid
+    if any(b["nearness"] is not None for b in kept):
+        by_near = sorted([b for b in kept if b["nearness"] is not None],
+                         key=lambda b: -b["nearness"])
+        for rank, b in enumerate(by_near, 1):
+            b["depth_rank"] = rank
+    return kept
+
+
+def _entity_grid_mask(ent: dict, size, cache: dict):
+    """Rasterized mask polygon on the containment grid (cached per entity)."""
+    eid = ent["id"]
+    if eid in cache:
+        return cache[eid]
+    W, H = size
+    m = None
+    if ent.get("mask_poly"):
+        pts = _seg_poly_points(ent["mask_poly"])
+        m = _seg_rasterize(pts, _GRID, _GRID / max(1.0, W), _GRID / max(1.0, H))
+    cache[eid] = m
+    return m
+
+
+def _own_frac(attr_box, ent: dict, size, cache: dict) -> float:
+    """|attr_box ∩ entity mask| / |attr_box| on the grid; box-fraction fallback
+    when the entity has no mask polygon ("box_containment")."""
+    W, H = size
+    m = _entity_grid_mask(ent, size, cache)
+    x1 = int(np.clip(attr_box[0] / W * _GRID, 0, _GRID))
+    y1 = int(np.clip(attr_box[1] / H * _GRID, 0, _GRID))
+    x2 = int(np.clip(np.ceil(attr_box[2] / W * _GRID), 0, _GRID))
+    y2 = int(np.clip(np.ceil(attr_box[3] / H * _GRID), 0, _GRID))
+    if x2 <= x1 or y2 <= y1:
+        return 0.0
+    if m is not None:
+        return float(m[y1:y2, x1:x2].sum()) / float((x2 - x1) * (y2 - y1))
+    # box fallback: inter / area(attr_box)
+    b = ent["box"]
+    ix1, iy1 = max(attr_box[0], b[0]), max(attr_box[1], b[1])
+    ix2, iy2 = min(attr_box[2], b[2]), min(attr_box[3], b[3])
+    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
+    a = derive._area(attr_box)
+    return inter / a if a > 0 else 0.0
+
+
+def _pair_predicates(a: dict, b: dict) -> list:
+    """a→b predicates, same semantics as derive.spatial_relations (dominant axis,
+    containment first, depth via nearness delta) — pinned by a consistency test."""
+    preds = []
+    if derive._contains(b["box"], a["box"]):
+        preds.append("inside")
+    else:
+        ca, cb = derive._centroid(a["box"]), derive._centroid(b["box"])
+        dx, dy = cb[0] - ca[0], cb[1] - ca[1]
+        if abs(dx) >= abs(dy):
+            preds.append("left_of" if dx > 0 else "right_of")
+        else:
+            preds.append("above" if dy > 0 else "below")
+    if a["nearness"] is not None and b["nearness"] is not None:
+        d = a["nearness"] - b["nearness"]
+        if abs(d) >= _DEPTH_REL_TOL:
+            preds.append("in_front_of" if d > 0 else "behind")
+    return preds
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# The fusion
+# ═════════════════════════════════════════════════════════════════════════════
+
+def fuse(digest: dict, caption_structs: dict, raw_captions: Optional[dict] = None,
+         *, t_own: float = 0.60, t_margin: float = 0.25, dedup_iou: float = 0.75,
+         coord_space: CoordSpace = CoordSpace.NORM_0_1000) -> dict:
+    """→ FusedScene as a schema-validated dict. Deterministic: same inputs →
+    byte-identical json.dumps. See the module docstring for the shape and the
+    ownership cascade; t_own / t_margin are the assignment thresholds (an attribute
+    below them lands in shared_basin with per-entity likelihoods — never guessed)."""
+    W, H = digest["size"]
+    size = (float(W), float(H))
+    raw_captions = raw_captions or {}
+
+    ents = _build_entities(digest, dedup_iou)
+    by_id = {e["id"]: e for e in ents}
+    grid_cache: dict = {}
+
+    # entity output records (attributes attached during the cascade)
+    ent_out = {}
+    for e in ents:
+        cx, cy = derive._centroid(e["box"])
+        ent_out[e["id"]] = {
+            "id": e["id"], "label": e["label"], "detection_score": round(e["score"], 4),
+            "box": box_to_space(e["box"], coord_space, size),
+            "centroid": poly_to_space([cx, cy], coord_space, size),
+            "area_frac": round(e["area_px"] / (W * H + 1e-9), 4),
+            "position": {"grid": _grid_cell(cx, cy, W, H),
+                         "offset_from_center": [round((cx - W / 2) / (W / 2), 4),
+                                                round((cy - H / 2) / (H / 2), 4)]},
+            "depth": ({"nearness": round(e["nearness"], 4), "rank": e["depth_rank"]}
+                      if e.get("nearness") is not None and e.get("depth_rank") else None),
+            "saliency": {"score": round(e["sal"], 4), "rank": e["sal_rank"]},
+            "is_primary": e["sal_rank"] == 1,
+            "mask": ({"polygon": poly_to_space(e["mask_poly"], coord_space, size),
+                      "quality": (round(e["mask_quality"], 4)
+                                  if e.get("mask_quality") is not None else None)}
+                     if e.get("mask_poly") else None),
+            "caption_bindings": [],
+            "attributes": [],
+        }
+
+    counts = Counter()
+    for e in ents:
+        counts["person" if labels_match(e["label"], "person") else e["label"]] += 1
+    people = counts.get("person", 0)
+
+    merged, merged_acts, votes = _collect_merged(caption_structs)
+    n_sources = sum(1 for v in caption_structs.values() if v)
+
+    # grounding lookup: canonical phrase -> [attr box records]. ground_phrases
+    # lowercases its input phrases, so BOTH sides normalize to strip().lower()
+    # (an uppercase structurer attribute must not silently lose its grounding).
+    grounded_by_phrase = defaultdict(list)
+    for a in digest.get("attr_boxes", []):
+        grounded_by_phrase[str(a["phrase"]).strip().lower()].append(a)
+    for recs in grounded_by_phrase.values():
+        recs.sort(key=lambda r: -r["score"])
+
+    def _candidates(m) -> tuple:
+        """(candidates, head_ok) — head_ok is False only when a subject head EXISTS
+        and matched no entity (fallback-to-all is then a guess, not evidence).
+        Pose/action attributes fall back to PERSON entities only — verbs apply to
+        agents, not to a baseball glove."""
+        cands, any_head = [], False
+        for src, subj in sorted(m["parents"].items()):
+            head = _subject_head(subj)
+            any_head = any_head or bool(head)
+            for e in ents:
+                if head and labels_match(head, e["label"]) and e not in cands:
+                    cands.append(e)
+        if cands:
+            return cands, True
+        if m.get("stratum") in ("pose", "action"):
+            persons = [e for e in ents if labels_match(e["label"], "person")]
+            if persons:
+                return persons, not any_head
+        return list(ents), not any_head
+
+    basin, scene_attrs, assigned_attrs = [], [], []
+    unresolved = []           # (merged, candidates) awaiting the binding pass
+    subj_votes = defaultdict(lambda: defaultdict(float))   # (src, subject) -> {eid: score}
+    subj_nvotes = defaultdict(int)
+
+    def _attach(eid, m, conf, method, margin=None, gbox=None, gscore=None):
+        rec = {"text": m["text"], "stratum": m["stratum"], "sources": m["sources"],
+               "consensus": m["consensus"], "grounded": gbox is not None,
+               "box": box_to_space(gbox, coord_space, size) if gbox else None,
+               "grounding_score": round(gscore, 4) if gscore is not None else None,
+               "ownership": {"confidence": round(conf, 4),
+                             "margin": round(margin, 4) if margin is not None else None,
+                             "method": method},
+               "region_on_owner": None}
+        if gbox is not None:
+            o = by_id[eid]
+            ocx, ocy = derive._centroid(o["box"])
+            acx, acy = derive._centroid(gbox)
+            hw = max(1.0, (o["box"][2] - o["box"][0]) / 2)
+            hh = max(1.0, (o["box"][3] - o["box"][1]) / 2)
+            rel_y, rel_x = (acy - ocy) / hh, (acx - ocx) / hw
+            rec["region_on_owner"] = {
+                "vertical": "upper" if rel_y < -1 / 3 else ("lower" if rel_y > 1 / 3 else "middle"),
+                "horizontal": "left" if rel_x < -1 / 3 else ("right" if rel_x > 1 / 3 else "center"),
+                "offset": [round(rel_x, 4), round(rel_y, 4)]}
+        ent_out[eid]["attributes"].append(rec)
+        assigned_attrs.append((m, eid, conf))
+        for src, subj in m["parents"].items():
+            subj_votes[(src, subj)][eid] += conf
+            subj_nvotes[(src, subj)] += 1
+
+    def _to_basin(m, reason, gbox=None, fracs=None):
+        cands = [{"entity_id": e["id"], "likelihood": round(f, 4)}
+                 for e, f in (fracs or []) if f >= 0.15]
+        cands.sort(key=lambda c: -c["likelihood"])
+        basin.append({"text": m["text"], "stratum": m["stratum"], "sources": m["sources"],
+                      "consensus": m["consensus"], "reason": reason,
+                      "grounded": gbox is not None,
+                      "box": box_to_space(gbox, coord_space, size) if gbox else None,
+                      "candidates": cands})
+
+    # ── pass A: scene routing, single-candidate fast path, grounded assignment ──
+    n_grounded_phrases = 0
+    for m in merged:
+        if m["stratum"] == "scene_level":
+            scene_attrs.append({"text": m["text"], "stratum": m["stratum"],
+                                "sources": m["sources"]})
+            continue
+        gboxes = (grounded_by_phrase.get(m["text"].strip().lower(), [])
+                  if is_groundable(m["stratum"]) else [])
+        if gboxes:
+            n_grounded_phrases += 1
+        cands, head_ok = _candidates(m)
+
+        if len(cands) == 1 and head_ok:
+            e = cands[0]
+            if gboxes:
+                f = _own_frac(gboxes[0]["box"], e, size, grid_cache)
+                if f < 0.2:      # caption mentions something visibly NOT on this entity
+                    _to_basin(m, "low_margin", gboxes[0]["box"], [(e, f)])
+                    continue
+                _attach(e["id"], m, 0.9, "single_entity",
+                        gbox=gboxes[0]["box"], gscore=gboxes[0]["score"])
+            else:
+                _attach(e["id"], m, 0.9, "single_entity")
+            continue
+
+        if gboxes:
+            if not cands:    # zero entities survived detection — grounded but unownable
+                _to_basin(m, "low_margin", gboxes[0]["box"])
+                continue
+            top = gboxes[0]["score"]
+            accepted = [g for g in gboxes if g["score"] >= 0.75 * top]
+            taken_eids = set()
+            any_assigned = False
+            best_fracs = None
+            for g in accepted:
+                fracs = sorted(((e, _own_frac(g["box"], e, size, grid_cache))
+                                for e in cands), key=lambda ef: -ef[1])
+                if best_fracs is None:
+                    best_fracs = (g, fracs)
+                f1 = fracs[0][1]
+                f2 = fracs[1][1] if len(fracs) > 1 else 0.0
+                winner = fracs[0][0]
+                if winner["id"] in taken_eids:
+                    continue
+                method = ("mask_containment"
+                          if _entity_grid_mask(winner, size, grid_cache) is not None
+                          else "box_containment")
+                if f1 >= t_own and (f1 - f2) >= t_margin:
+                    taken_eids.add(winner["id"])
+                    any_assigned = True
+                    _attach(winner["id"], m, f1, method, margin=f1 - f2,
+                            gbox=g["box"], gscore=g["score"])
+            if not any_assigned:
+                g, fracs = best_fracs
+                _to_basin(m, "low_margin", g["box"], fracs)
+            continue
+
+        unresolved.append((m, cands))
+
+    # ── binding: caption subjects ↔ entities (votes from grounded assignments
+    #    + positional cues in subject names and raw captions) ───────────────────
+    def _positional_vote(text: str, cands: list, votes_out: dict):
+        if not cands:
+            return 0
+        toks = set(_content_tokens(text))
+        if toks & _POS_LEFT:
+            e = min(cands, key=lambda e: derive._centroid(e["box"])[0])
+            votes_out[e["id"]] += 0.5
+            return 1
+        if toks & _POS_RIGHT:
+            e = max(cands, key=lambda e: derive._centroid(e["box"])[0])
+            votes_out[e["id"]] += 0.5
+            return 1
+        if toks & _POS_FRONT and any(e.get("depth_rank") for e in cands):
+            e = min((e for e in cands if e.get("depth_rank")), key=lambda e: e["depth_rank"])
+            votes_out[e["id"]] += 0.5
+            return 1
+        if toks & _POS_BACK and any(e.get("depth_rank") for e in cands):
+            e = max((e for e in cands if e.get("depth_rank")), key=lambda e: e["depth_rank"])
+            votes_out[e["id"]] += 0.5
+            return 1
+        if toks & _POS_TALL:
+            e = max(cands, key=lambda e: e["box"][3] - e["box"][1])
+            votes_out[e["id"]] += 0.5
+            return 1
+        return 0
+
+    bindings = {}       # (src, subject) -> (eid, bind_conf)
+    all_subjects = {(src, subj) for m in merged for src, subj in m["parents"].items()}
+    for (src, subj) in sorted(all_subjects):
+        head = _subject_head(subj)
+        cands = [e for e in ents if head and labels_match(head, e["label"])] or list(ents)
+        v = dict(subj_votes.get((src, subj), {}))
+        v = defaultdict(float, v)
+        nv = subj_nvotes.get((src, subj), 0)
+        pos_n = _positional_vote(subj, cands, v)
+        raw = raw_captions.get(src, "")
+        if raw and head:
+            # "<positional> [word] <head>"  — tight adjacency, so a positional word
+            # in a NEIGHBORING clause can't vote for this subject
+            for mtc in re.finditer(rf"\b(\w+)\s+(?:\w+\s+)?{re.escape(head)}\b", raw.lower()):
+                pos_n += _positional_vote(mtc.group(1), cands, v)
+            # "<head> ... on the <positional>" — reject windows crossing an "and"
+            # (clause boundary: "a woman AND a man on the right")
+            for mtc in re.finditer(rf"\b{re.escape(head)}\b([\w\s,]{{0,24}}?)\bon the (\w+)",
+                                   raw.lower()):
+                if " and " in f" {mtc.group(1)} ":
+                    continue
+                pos_n += _positional_vote(mtc.group(2), cands, v)
+        # bind on >=2 containment votes, OR any explicit positional cue (the caption
+        # author's own disambiguation — stronger evidence than one weak containment)
+        if not v or (nv < 2 and pos_n < 1):
+            continue
+        total = sum(v.values())
+        eid, top = max(sorted(v.items()), key=lambda kv: kv[1])
+        bind_conf = top / total if total > 0 else 0.0
+        if bind_conf >= 0.6:
+            bindings[(src, subj)] = (eid, bind_conf)
+            ent_out[eid]["caption_bindings"].append(
+                {"source": src, "subject_name": subj, "confidence": round(bind_conf, 4)})
+
+    # ── pass B: unresolved attributes inherit their subject's binding ───────────
+    for m, cands in unresolved:
+        # collapse per entity (max conf) with DETERMINISTIC iteration order —
+        # set iteration over tuples is process-hash-dependent
+        by_eid: dict = {}
+        for src, subj in sorted(m["parents"].items()):
+            if (src, subj) in bindings:
+                eid, conf = bindings[(src, subj)]
+                by_eid[eid] = max(by_eid.get(eid, 0.0), conf)
+        if len(by_eid) == 1:
+            eid, bind_conf = next(iter(by_eid.items()))
+            _attach(eid, m, bind_conf * 0.6, "caption_binding")
+        elif len(by_eid) > 1:
+            _to_basin(m, "ambiguous_binding",
+                      fracs=sorted(((by_id[eid], conf) for eid, conf in by_eid.items()),
+                                   key=lambda ef: (-ef[1], ef[0]["id"])))
+        else:
+            reason = ("no_grounding_multi_entity" if is_groundable(m["stratum"])
+                      else "abstract_unbound")
+            n_c = max(1, len(cands))
+            _to_basin(m, reason, fracs=[(e, 1.0 / n_c) for e in cands])
+
+    # ── actions: one person → attach as stratum "action"; else scene-level ─────
+    # (actions are NOT part of the attribute-routing identity
+    # assigned + basin + scene_level == phrases_total — separate accumulator)
+    scene_actions = []
+    action_confs = []
+    person_ents = [e for e in ents if labels_match(e["label"], "person")]
+    for m in merged_acts:
+        if len(person_ents) == 1:
+            e = person_ents[0]
+            ent_out[e["id"]]["attributes"].append(
+                {"text": m["text"], "stratum": "action", "sources": m["sources"],
+                 "consensus": m["consensus"], "grounded": False, "box": None,
+                 "grounding_score": None,
+                 "ownership": {"confidence": 0.9, "margin": None,
+                               "method": "single_entity"},
+                 "region_on_owner": None})
+            action_confs.append(0.9)
+        else:
+            scene_actions.append({"text": m["text"], "stratum": "action",
+                                  "sources": m["sources"]})
+
+    # ── relations among the top-K entities by saliency ──────────────────────────
+    rel_ents = sorted(ents, key=lambda e: e["sal_rank"])[:MAX_RELATION_ENTITIES]
+    rel_ents = sorted(rel_ents, key=lambda e: [x["id"] for x in ents].index(e["id"]))
+    diag = (W ** 2 + H ** 2) ** 0.5 or 1.0
+    relations = []
+    for i in range(len(rel_ents)):
+        for j in range(i + 1, len(rel_ents)):
+            a, b = rel_ents[i], rel_ents[j]
+            # containment is orientation-independent: put the INNER entity first so
+            # "inside" always reads a-inside-b regardless of left-to-right id order
+            if (derive._contains(a["box"], b["box"])
+                    and not derive._contains(b["box"], a["box"])):
+                a, b = b, a
+            ca, cb = derive._centroid(a["box"]), derive._centroid(b["box"])
+            depth_delta = (round(a["nearness"] - b["nearness"], 4)
+                           if a["nearness"] is not None and b["nearness"] is not None
+                           else None)
+            relations.append({
+                "a": a["id"], "b": b["id"],
+                "predicates": _pair_predicates(a, b),
+                "dx": round((cb[0] - ca[0]) / W, 4), "dy": round((cb[1] - ca[1]) / H, 4),
+                "distance": round(((cb[0] - ca[0]) ** 2 + (cb[1] - ca[1]) ** 2) ** 0.5 / diag, 4),
+                "iou": round(derive._iou(a["box"], b["box"]), 4),
+                "depth_delta": depth_delta,
+                "confidence": round(min(a["score"], b["score"]), 4),
+            })
+
+    # ── scene block ─────────────────────────────────────────────────────────────
+    set_votes = votes["setting"]
+    setting_val = None
+    if set_votes:
+        ranked = set_votes.most_common()
+        setting_val = ("unknown" if len(ranked) > 1 and ranked[0][1] == ranked[1][1]
+                       else ranked[0][0])
+    style_votes = votes["style"]
+    style_val = digest.get("style") or (style_votes.most_common(1)[0][0]
+                                        if style_votes else None)
+    mood_per_source = votes["mood"]
+    mood_val = None
+    if mood_per_source:
+        mood_counts = Counter(mood_per_source.values())
+        mood_val = mood_counts.most_common(1)[0][0]
+    sym = digest.get("symmetry")
+    sym_axis = "none"
+    if sym:
+        v, h = sym["lr"] >= 0.80, sym["tb"] >= 0.80
+        sym_axis = "radial" if (v and h) else ("vertical" if v else ("horizontal" if h else "none"))
+    ocr_lines = []
+    for ln in digest.get("ocr", {}).get("lines", []):
+        q = ln.get("quad")
+        box = None
+        if q:
+            xs, ys = q[0::2], q[1::2]
+            box = box_to_space([min(xs), min(ys), max(xs), max(ys)], coord_space, size)
+        ocr_lines.append({"text": ln["text"], "box": box, "conf": ln.get("conf")})
+
+    scene = {
+        "setting": {"value": setting_val, "votes": dict(sorted(set_votes.items()))},
+        "style": {"value": style_val, "caption_votes": dict(sorted(style_votes.items())),
+                  "specialist": digest.get("style")},
+        "mood": {"value": mood_val, "per_source": dict(sorted(mood_per_source.items()))},
+        "layout": digest.get("layout", "unknown"),
+        "symmetry": {"axis": sym_axis,
+                     "lr": round(sym["lr"], 4) if sym else None,
+                     "tb": round(sym["tb"], 4) if sym else None},
+        "actions": scene_actions,
+        "scene_attributes": scene_attrs,
+        "ocr": {"full_text": digest.get("ocr", {}).get("full_text", ""), "lines": ocr_lines},
+        "class_top": digest.get("class_top", []),
+    }
+
+    # ── quality + accounting ────────────────────────────────────────────────────
+    n_groundable = sum(1 for m in merged if is_groundable(m["stratum"]))
+    n_scene = len(scene_attrs)
+    n_ungroundable = sum(1 for m in merged
+                         if not is_groundable(m["stratum"]) and m["stratum"] != "scene_level")
+    n_assigned_attrs = len({id(m) for m, _, _ in assigned_attrs})
+    mask_qualities = [e["mask_quality"] for e in ents if e.get("mask_quality") is not None]
+    ocr_confs = [l["conf"] for l in ocr_lines if l.get("conf") is not None]
+    det_mean = float(np.mean([e["score"] for e in ents])) if ents else 0.0
+    own_confs = [c for _, _, c in assigned_attrs] + action_confs
+    n_routed = n_assigned_attrs + len(basin)
+    overall = round(
+        0.5 * (float(np.mean(own_confs)) if own_confs else 0.0)
+        + 0.3 * det_mean
+        + 0.2 * (n_assigned_attrs / n_routed if n_routed else 0.0), 4)
+
+    out = {
+        "coord_space": str(coord_space.value if hasattr(coord_space, "value") else coord_space),
+        "image_size": [int(W), int(H)],
+        "counts": {"total_entities": len(ents), "people": people,
+                   "by_label": dict(sorted(counts.items()))},
+        "entities": [ent_out[e["id"]] for e in ents],
+        "relations": relations,
+        "shared_basin": basin,
+        "scene": scene,
+        "quality": {
+            "n_caption_sources": n_sources,
+            "detection_score_mean": round(det_mean, 4),
+            "mask_quality_mean": (round(float(np.mean(mask_qualities)), 4)
+                                  if mask_qualities else None),
+            "ocr_conf_mean": (round(float(np.mean(ocr_confs)), 4) if ocr_confs else None),
+            "grounding": {"phrases_total": len(merged),
+                          "phrases_grounded": n_grounded_phrases,
+                          "assigned": n_assigned_attrs,
+                          "basin": len(basin),
+                          "scene_level": n_scene,
+                          "ungroundable": n_ungroundable},
+            "overall_confidence": overall,
+        },
+    }
+    # schema-validate + normalize field order (byte-determinism of json.dumps)
+    return FusedScene.model_validate(out).model_dump()
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# semantic_association — the 12th deterministic task (VLM→INTEGRATE reclass)
+# ═════════════════════════════════════════════════════════════════════════════
+
+def build_semantic_association(scene: dict, max_items: int = 32) -> dict:
+    """FusedScene → the EXISTING registry shape {associations:[{a,relation,b}]}
+    (enum: left_of/right_of/near/is_a/related_to). Deterministic: geometry gives
+    left_of/right_of/near; caption bindings give is_a (bound subject head vs the
+    detector label, e.g. woman is_a person)."""
+    out, seen = [], set()
+
+    def _emit(a, rel, b):
+        t = (a, rel, b)
+        if t not in seen and len(out) < max_items:
+            seen.add(t)
+            out.append({"a": a, "relation": rel, "b": b})
+
+    for r in scene.get("relations", []):
+        for p in r.get("predicates", []):
+            if p in ("left_of", "right_of"):
+                _emit(r["a"], p, r["b"])
+        if r.get("distance") is not None and r["distance"] <= _NEAR_DIST:
+            _emit(r["a"], "near", r["b"])
+    for e in scene.get("entities", []):
+        for cb in e.get("caption_bindings", []):
+            head = _subject_head(cb["subject_name"])
+            if head and head != e["label"] and labels_match(head, e["label"]):
+                _emit(head, "is_a", e["label"])
+    return {"associations": out}

qwen_test_runner/vision/fuse_prompt.py

@@ -0,0 +1,153 @@
+"""
+fuse_prompt.py — deterministic natural-language rendering of a FusedScene.
+
+A pure function of the fused JSON: fixed clause order (counts → primary entity →
+other entities by saliency → relations → shared basin → scene), fixed attribute
+ordering (consensus desc → ownership confidence desc → stratum precedence), zero
+randomness — `fused_prompt(scene) == fused_prompt(scene)` byte-for-byte is a unit
+test. Uncertainty is RENDERED, never guessed away: basin items become "One of
+{candidates} has {attribute}." LLM smoothing is deliberately not here — if ever
+wanted it is a separate additional dataset column, so this one stays trustworthy.
+"""
+
+from __future__ import annotations
+
+from .strata import STRATUM_PRECEDENCE
+
+_NUM_WORDS = {1: "one", 2: "two", 3: "three", 4: "four", 5: "five", 6: "six",
+              7: "seven", 8: "eight", 9: "nine", 10: "ten"}
+
+_PRED_PHRASE = {"left_of": "to the left of", "right_of": "to the right of",
+                "above": "above", "below": "below", "inside": "inside",
+                "in_front_of": "in front of", "behind": "behind"}
+
+_STRATUM_ORDER = {s: i for i, s in enumerate(STRATUM_PRECEDENCE + ("action",))}
+
+
+def _num(n: int) -> str:
+    return _NUM_WORDS.get(n, str(n))
+
+
+def _plural(label: str, n: int) -> str:
+    if n == 1:
+        return label
+    if label == "person":
+        return "people"
+    return label if label.endswith("s") else label + "s"
+
+
+def _ref(entity_id: str) -> str:
+    """person_1 -> "person 1"; dog -> "the dog"."""
+    if "_" in entity_id and entity_id.rsplit("_", 1)[1].isdigit():
+        base, num = entity_id.rsplit("_", 1)
+        return f"{base.replace('_', ' ')} {num}"
+    return f"the {entity_id.replace('_', ' ')}"
+
+
+def _cap(sentence: str) -> str:
+    return sentence[0].upper() + sentence[1:] if sentence else sentence
+
+
+def _join(parts: list) -> str:
+    parts = [p for p in parts if p]
+    if not parts:
+        return ""
+    if len(parts) == 1:
+        return parts[0]
+    return ", ".join(parts[:-1]) + " and " + parts[-1]
+
+
+def _ordered_attrs(entity: dict, max_attrs: int) -> list:
+    attrs = sorted(entity.get("attributes", []),
+                   key=lambda a: (-a["consensus"], -a["ownership"]["confidence"],
+                                  _STRATUM_ORDER.get(a["stratum"], 99), a["text"]))
+    return attrs[:max_attrs]
+
+
+def _entity_clause(entity: dict, intro: str, max_attrs: int, n_entities: int) -> str:
+    bits = [f"{intro} in the {entity['position']['grid']} of the frame"]
+    d = entity.get("depth")
+    if d and n_entities > 1:
+        if d["rank"] == 1:
+            bits.append("nearest to the camera")
+        elif d["rank"] == n_entities:
+            bits.append("farthest from the camera")
+    attrs = _ordered_attrs(entity, max_attrs)
+    # pose/action attributes read as participles ("…, sitting"), not "with sitting"
+    plain = [a["text"] for a in attrs if a["stratum"] not in ("action", "pose")]
+    acts = [a["text"] for a in attrs if a["stratum"] in ("action", "pose")]
+    s = ", ".join(bits)
+    if plain:
+        s += f", with {_join(plain)}"
+    if acts:
+        s += f", {_join(acts)}"
+    return _cap(s + ".")
+
+
+def fused_prompt(scene: dict, max_attrs_per_entity: int = 6, max_relations: int = 6,
+                 max_basin: int = 4) -> str:
+    sentences = []
+
+    # 1) counts
+    by_label = scene.get("counts", {}).get("by_label", {})
+    if by_label:
+        parts = [f"{_num(n)} {_plural(lab, n)}"
+                 for lab, n in sorted(by_label.items(), key=lambda kv: (-kv[1], kv[0]))]
+        sentences.append(_cap(_join(parts) + "."))
+
+    # 2) entities, primary first, then by saliency rank
+    entities = scene.get("entities", [])
+    ordered = sorted(entities, key=lambda e: e["saliency"]["rank"])
+    n_ent = len(entities)
+    for k, e in enumerate(ordered):
+        if k == 0:
+            intro = f"the primary subject is a {e['label']}" if n_ent > 1 else f"a {e['label']}"
+        else:
+            intro = f"{_ref(e['id'])} is"
+        sentences.append(_entity_clause(e, intro, max_attrs_per_entity, n_ent))
+
+    # 3) relations (strongest-confidence first, capped)
+    rels = sorted(scene.get("relations", []),
+                  key=lambda r: (-r["confidence"], r["a"], r["b"]))[:max_relations]
+    for r in rels:
+        phrases = [_PRED_PHRASE[p] for p in r.get("predicates", []) if p in _PRED_PHRASE]
+        if phrases:
+            sentences.append(_cap(f"{_ref(r['a'])} is {_join(phrases)} {_ref(r['b'])}."))
+
+    # 4) shared basin — uncertainty rendered, never guessed
+    for b in scene.get("shared_basin", [])[:max_basin]:
+        cands = [_ref(c["entity_id"]) for c in b.get("candidates", [])[:3]]
+        if cands:
+            joined = cands[0] if len(cands) == 1 else " or ".join(cands)
+            sentences.append(_cap(f"one of them ({joined}) has {b['text']}."))
+        else:
+            sentences.append(_cap(f"somewhere in the scene: {b['text']}."))
+
+    # 5) scene
+    sc = scene.get("scene", {})
+    bits = []
+    setting = (sc.get("setting") or {}).get("value")
+    if setting and setting != "unknown":
+        bits.append(f"{setting} scene")
+    style = (sc.get("style") or {}).get("value")
+    if style and style not in ("other", "unknown"):
+        bits.append(f"{style} style")
+    mood = (sc.get("mood") or {}).get("value")
+    if mood:
+        bits.append(f"{mood} mood")
+    layout = sc.get("layout")
+    if layout and layout not in ("unknown", "scattered"):
+        bits.append(f"{layout.replace('_', ' ')} composition")
+    if (sc.get("symmetry") or {}).get("axis", "none") != "none":
+        bits.append(f"{sc['symmetry']['axis']} symmetry")
+    if bits:
+        sentences.append(_cap(", ".join(bits) + "."))
+    for act in sc.get("actions", [])[:3]:
+        sentences.append(_cap(f"action in the scene: {act['text']}."))
+    for sa in sc.get("scene_attributes", [])[:4]:
+        sentences.append(_cap(f"{sa['text']}."))
+    ocr_text = (sc.get("ocr") or {}).get("full_text", "").strip()
+    if ocr_text:
+        sentences.append(_cap(f'visible text: "{ocr_text[:120]}".'))
+
+    return " ".join(sentences)

qwen_test_runner/vision/fuse_schema.py

@@ -0,0 +1,228 @@
+"""
+fuse_schema.py — the FusedScene pydantic schema (fusion tier).
+
+Hand-written nested models (the SubjectValue precedent — the SlotSpec codegen in
+schema.py handles one nesting level; FusedScene needs three: entities ->
+attributes -> ownership). Deliberately NOT a registered VisionTaskSpec: FusedScene
+is a deterministically-produced dataset artifact, not a VLM probe — it needs no
+GBNF, no system prompt, no scorer. If a VLM is ever trained to EMIT FusedScene,
+register a flattened variant then.
+
+Versioned from day one: this module is a second schema authority next to the
+registry, so every instance stamps `fused_scene_version`.
+
+Coordinate policy: all geometry emitted in the declared `coord_space`
+(NORM_0_1000 by default, via specialists.box_to_space/poly_to_space); the one
+pixel-unit field is `image_size`, documented as such. Scorers convert via
+coords.to_canonical.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+FUSED_SCENE_VERSION = "1.0"
+
+# Caps (data, referenced by fuse.py — never hardcoded at call sites)
+MAX_ENTITIES = 12            # kept by saliency rank
+MAX_RELATION_ENTITIES = 8    # pairwise relations among the top-K entities
+MASK_POLY_MAX_POINTS = 64    # outline_polygon(max_points=...) per entity
+
+# Basin reasons (closed set, tested)
+BASIN_REASONS = ("low_margin", "no_grounding_multi_entity", "ambiguous_binding",
+                 "abstract_unbound")
+
+# Ownership methods (closed set, tested)
+OWNERSHIP_METHODS = ("single_entity", "mask_containment", "box_containment",
+                     "caption_binding")
+
+
+class GridPosition(BaseModel):
+    grid: str                                   # "{upper|middle|lower} {left|center|right}"
+    offset_from_center: list[float]             # (centroid-center)/half-extents, in [-1,1]
+
+
+class DepthInfo(BaseModel):
+    nearness: float                             # continuous [0,1], bigger = nearer
+    rank: int                                   # 1 = nearest
+
+
+class SaliencyInfo(BaseModel):
+    score: float
+    rank: int                                   # 1 = most salient
+
+
+class MaskInfo(BaseModel):
+    polygon: list[float] = Field(default_factory=list)   # flat x,y interleaved, task space
+    quality: Optional[float] = None             # SAM predicted-IoU (retained signal)
+
+
+class CaptionBinding(BaseModel):
+    source: str                                 # caption column key
+    subject_name: str
+    confidence: float
+
+
+class Ownership(BaseModel):
+    confidence: float
+    margin: Optional[float] = None              # f1 - f2 (containment methods only)
+    method: str                                 # one of OWNERSHIP_METHODS
+
+
+class RegionOnOwner(BaseModel):
+    vertical: str                               # upper | middle | lower
+    horizontal: str                             # left | center | right
+    offset: list[float]                         # (attr center - owner centroid)/owner half-extents
+
+
+class AttributeRecord(BaseModel):
+    text: str                                   # canonical (longest) form after dedup
+    stratum: str
+    sources: list[str]                          # caption columns that carried it
+    consensus: float                            # len(sources)/n_caption_sources
+    grounded: bool = False
+    box: Optional[list[float]] = None           # present iff grounded (task space)
+    grounding_score: Optional[float] = None     # GDINO phrase score
+    ownership: Ownership
+    region_on_owner: Optional[RegionOnOwner] = None
+
+
+class Entity(BaseModel):
+    id: str                                     # person_1, person_2, dog, ... (left-to-right)
+    label: str
+    detection_score: float
+    box: list[float]                            # task space
+    centroid: list[float]                       # task space
+    area_frac: float
+    position: GridPosition
+    depth: Optional[DepthInfo] = None
+    saliency: SaliencyInfo
+    is_primary: bool = False
+    mask: Optional[MaskInfo] = None
+    caption_bindings: list[CaptionBinding] = Field(default_factory=list)
+    attributes: list[AttributeRecord] = Field(default_factory=list)
+
+
+class Relation(BaseModel):
+    a: str                                      # entity id (smaller entity index)
+    b: str
+    predicates: list[str]                       # spatial_relations predicate vocab
+    dx: float                                   # (centroid_b - centroid_a)/W, task-space-free
+    dy: float
+    distance: float                             # centroid distance / image diagonal
+    iou: float
+    depth_delta: Optional[float] = None         # nearness_a - nearness_b (continuous)
+    confidence: float                           # min(detection scores)
+
+
+class BasinCandidate(BaseModel):
+    entity_id: str
+    likelihood: float
+
+
+class BasinItem(BaseModel):
+    text: str
+    stratum: str
+    sources: list[str]
+    consensus: float
+    reason: str                                 # one of BASIN_REASONS
+    grounded: bool = False
+    box: Optional[list[float]] = None
+    candidates: list[BasinCandidate] = Field(default_factory=list)
+
+
+class VotedValue(BaseModel):
+    value: Optional[str] = None
+    votes: dict[str, int] = Field(default_factory=dict)
+
+
+class StyleValue(BaseModel):
+    value: Optional[str] = None                 # specialist wins conflicts (it saw the image)
+    caption_votes: dict[str, int] = Field(default_factory=dict)
+    specialist: Optional[str] = None
+
+
+class MoodValue(BaseModel):
+    value: Optional[str] = None
+    per_source: dict[str, str] = Field(default_factory=dict)
+
+
+class SymmetryInfo(BaseModel):
+    axis: str = "none"
+    lr: Optional[float] = None                  # continuous correlations (retained)
+    tb: Optional[float] = None
+
+
+class SceneAttribute(BaseModel):
+    text: str
+    stratum: str = "scene_level"
+    sources: list[str] = Field(default_factory=list)
+
+
+class SceneOCRLine(BaseModel):
+    text: str
+    box: Optional[list[float]] = None
+    conf: Optional[float] = None                # EasyOCR confidence (retained)
+
+
+class SceneOCR(BaseModel):
+    full_text: str = ""
+    lines: list[SceneOCRLine] = Field(default_factory=list)
+
+
+class LabelScore(BaseModel):
+    label: str
+    score: float
+
+
+class SceneBlock(BaseModel):
+    setting: VotedValue = Field(default_factory=VotedValue)
+    style: StyleValue = Field(default_factory=StyleValue)
+    mood: MoodValue = Field(default_factory=MoodValue)
+    layout: str = "unknown"
+    symmetry: SymmetryInfo = Field(default_factory=SymmetryInfo)
+    actions: list[SceneAttribute] = Field(default_factory=list)   # unbound caption actions
+    scene_attributes: list[SceneAttribute] = Field(default_factory=list)
+    ocr: SceneOCR = Field(default_factory=SceneOCR)
+    class_top: list[LabelScore] = Field(default_factory=list)
+
+
+class Counts(BaseModel):
+    total_entities: int = 0
+    people: int = 0                             # via the person synonym group
+    by_label: dict[str, int] = Field(default_factory=dict)
+
+
+class GroundingStats(BaseModel):
+    phrases_total: int = 0
+    phrases_grounded: int = 0
+    assigned: int = 0
+    basin: int = 0
+    scene_level: int = 0
+    ungroundable: int = 0
+
+
+class Quality(BaseModel):
+    n_caption_sources: int = 0
+    detection_score_mean: float = 0.0
+    mask_quality_mean: Optional[float] = None
+    ocr_conf_mean: Optional[float] = None
+    grounding: GroundingStats = Field(default_factory=GroundingStats)
+    overall_confidence: float = 0.0             # the fusion_confidence scalar
+
+
+class FusedScene(BaseModel):
+    fused_scene_version: str = FUSED_SCENE_VERSION
+    coord_space: str
+    image_size: list[int]                       # (W, H) PIXELS — the one pixel field
+    counts: Counts = Field(default_factory=Counts)
+    entities: list[Entity] = Field(default_factory=list)
+    relations: list[Relation] = Field(default_factory=list)
+    shared_basin: list[BasinItem] = Field(default_factory=list)
+    scene: SceneBlock = Field(default_factory=SceneBlock)
+    quality: Quality = Field(default_factory=Quality)
+
+
+FUSED_SCENE_JSON_SCHEMA = FusedScene.model_json_schema()

qwen_test_runner/vision/fusion_metrics.py

@@ -0,0 +1,523 @@
+"""
+fusion_metrics.py — validation scorers for the fusion tier (COCO multi-person).
+
+Three metric classes, NEVER mixed in a report:
+  HARD       — anchored to COCO instance GT (counts, entity F1, relation agreement)
+  PROXY      — honestly-labeled plausibility checks (attribute-binding via mined
+               caption phrases + GDINO grounding — both sides are proxies; the rate
+               is reported with its coverage and skip histogram, never a headline)
+  STRUCTURAL — invariants (prompt determinism/faithfulness, no-invention, depth
+               internal consistency). PASS/FAIL, not accuracy.
+
+Identity matters here: labels_match puts man/woman/person in ONE synonym group, so
+string matching would silently accept identity SWAPS between two people. Every
+identity-sensitive score therefore RELABELS-THEN-COMPARES through `match_entities`
+(greedy IoU@0.5 fused-entity -> GT-instance assignment) before any set math.
+
+Pure CPU (numpy + PIL rasterization via metrics helpers), torch-free.
+"""
+
+from __future__ import annotations
+
+import re
+from collections import Counter
+from dataclasses import dataclass, field
+from typing import Optional
+
+import numpy as np
+
+from . import derive
+from .coords import CoordSpace, to_canonical
+from .fuse_prompt import _num, _plural
+from .fuse_schema import MAX_RELATION_ENTITIES
+from .metrics import _seg_poly_points, _seg_rasterize, labels_match
+from .strata import STRATA, _content_tokens
+
+_GRID = 160
+_IOU_MATCH = 0.5
+
+# concrete-modifier gate for the phrase miner: colors + garments + accessories +
+# a small holdable-object list (reuses the strata lexicons — one vocabulary)
+_CONCRETE = (STRATA["color"] | STRATA["clothing"] | STRATA["accessory"]
+             | frozenset({"umbrella", "phone", "camera", "cup", "bottle", "book",
+                          "ball", "racket", "bat", "kite", "surfboard", "skateboard",
+                          "frisbee", "laptop", "pizza", "donut", "banana", "wine"}))
+
+_PERSON_HEADS = ("person", "man", "woman", "boy", "girl", "lady", "guy", "child",
+                 "kid", "player", "skier", "surfer")
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Entity matching — relabel-then-compare
+# ═════════════════════════════════════════════════════════════════════════════
+
+@dataclass
+class EntityMatch:
+    fused_to_gt: dict = field(default_factory=dict)     # entity_id -> GT id ("p0" | "q0")
+    ious: dict = field(default_factory=dict)            # entity_id -> matched IoU
+    unmatched_fused: list = field(default_factory=list)
+    unmatched_gt: list = field(default_factory=list)    # GT ids with no fused entity
+
+
+def _fused_boxes_px(fused: dict, size) -> dict:
+    """entity_id -> pixel-abs xyxy (converting from the scene's declared coord_space)."""
+    space = CoordSpace(fused.get("coord_space", "norm_0_1000"))
+    out = {}
+    for e in fused.get("entities", []):
+        out[e["id"]] = to_canonical(e["box"], space, size).as_list()
+    return out
+
+
+def _greedy_iou_match(pred: list, gts: list, iou_thr: float = _IOU_MATCH) -> list:
+    """[(pred_idx, gt_idx, iou)] greedy by IoU desc, one-to-one."""
+    cand = []
+    for i, pb in enumerate(pred):
+        for j, gb in enumerate(gts):
+            iou = derive._iou(pb, gb)
+            if iou >= iou_thr:
+                cand.append((iou, i, j))
+    cand.sort(key=lambda t: (-t[0], t[1], t[2]))
+    used_p, used_g, out = set(), set(), []
+    for iou, i, j in cand:
+        if i in used_p or j in used_g:
+            continue
+        used_p.add(i)
+        used_g.add(j)
+        out.append((i, j, iou))
+    return out
+
+
+def match_entities(fused: dict, gt: dict, size) -> EntityMatch:
+    """Persons match persons (IoU@0.5); non-person fused entities match GT objects
+    (label-tolerant + IoU@0.5). GT ids: persons p0..pN, objects q0..qM."""
+    boxes_px = _fused_boxes_px(fused, size)
+    ents = fused.get("entities", [])
+    m = EntityMatch()
+
+    p_ents = [e for e in ents if labels_match(e["label"], "person")]
+    p_boxes = [boxes_px[e["id"]] for e in p_ents]
+    g_boxes = [p["box_xyxy"] for p in gt.get("persons", [])]
+    for i, j, iou in _greedy_iou_match(p_boxes, g_boxes):
+        m.fused_to_gt[p_ents[i]["id"]] = f"p{j}"
+        m.ious[p_ents[i]["id"]] = round(iou, 4)
+
+    o_ents = [e for e in ents if not labels_match(e["label"], "person")]
+    gt_objs = gt.get("objects", [])
+    cand = []
+    for i, e in enumerate(o_ents):
+        for j, o in enumerate(gt_objs):
+            if not labels_match(e["label"], o["label"]):
+                continue
+            iou = derive._iou(boxes_px[e["id"]], o["box_xyxy"])
+            if iou >= _IOU_MATCH:
+                cand.append((iou, i, j))
+    cand.sort(key=lambda t: (-t[0], t[1], t[2]))
+    used_p, used_g = set(), set()
+    for iou, i, j in cand:
+        if i in used_p or j in used_g:
+            continue
+        used_p.add(i)
+        used_g.add(j)
+        m.fused_to_gt[o_ents[i]["id"]] = f"q{j}"
+        m.ious[o_ents[i]["id"]] = round(iou, 4)
+
+    m.unmatched_fused = sorted(e["id"] for e in ents if e["id"] not in m.fused_to_gt)
+    matched_gt = set(m.fused_to_gt.values())
+    m.unmatched_gt = ([f"p{j}" for j in range(len(g_boxes)) if f"p{j}" not in matched_gt]
+                      + [f"q{j}" for j in range(len(gt_objs)) if f"q{j}" not in matched_gt])
+    return m
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# HARD scorers
+# ═════════════════════════════════════════════════════════════════════════════
+
+def score_person_count(fused: dict, gt: dict) -> dict:
+    pred, ref = fused.get("counts", {}).get("people", 0), gt.get("n_persons", 0)
+    return {"pred": pred, "gt": ref, "exact": pred == ref,
+            "off_by_one": abs(pred - ref) <= 1, "abs_err": abs(pred - ref)}
+
+
+def score_entity_f1(fused: dict, gt: dict, size, match: Optional[EntityMatch] = None) -> dict:
+    match = match or match_entities(fused, gt, size)
+    n_pred_p = sum(1 for e in fused.get("entities", [])
+                   if labels_match(e["label"], "person"))
+    n_gt_p = len(gt.get("persons", []))
+    tp = sum(1 for gid in match.fused_to_gt.values() if gid.startswith("p"))
+    prec = tp / n_pred_p if n_pred_p else 0.0
+    rec = tp / n_gt_p if n_gt_p else 0.0
+    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
+    n_pred_all = len(fused.get("entities", []))
+    n_gt_all = n_gt_p + len(gt.get("objects", []))
+    tp_all = len(match.fused_to_gt)
+    prec_a = tp_all / n_pred_all if n_pred_all else 0.0
+    rec_a = tp_all / n_gt_all if n_gt_all else 0.0
+    f1_a = 2 * prec_a * rec_a / (prec_a + rec_a) if (prec_a + rec_a) else 0.0
+    return {"person_f1": round(f1, 4), "person_precision": round(prec, 4),
+            "person_recall": round(rec, 4), "all_f1": round(f1_a, 4)}
+
+
+def _canon_triple(a: str, pred: str, b: str):
+    """Direction-normalize so (a,left_of,b) == (b,right_of,a) compares equal.
+    `inside` stays directional."""
+    flip = {"right_of": "left_of", "below": "above", "behind": "in_front_of"}
+    if pred in flip:
+        return (b, flip[pred], a)
+    return (a, pred, b)
+
+
+def _gt_triples(gt: dict) -> set:
+    """Deterministic GT-side relations: the SAME derive engine run on GT boxes
+    (projective + containment only — COCO has no depth GT). For a contained pair,
+    derive emits BOTH (inner, inside, outer) and a projective from the reverse
+    ordering — fuse emits ONE predicate per pair (containment-first), so the
+    projective is dropped here whenever the pair carries `inside` (otherwise
+    relation recall is structurally capped at 0.5 on containment pairs)."""
+    boxes = ([{"label": f"p{j}", "box": p["box_xyxy"]}
+              for j, p in enumerate(gt.get("persons", []))]
+             + [{"label": f"q{j}", "box": o["box_xyxy"]}
+                for j, o in enumerate(gt.get("objects", []))])
+    rels = derive.spatial_relations(boxes, depth=None, max_items=10_000)["relations"]
+    triples = {_canon_triple(r["subject"], r["predicate"], r["object"]) for r in rels}
+    inside_pairs = {frozenset((s, o)) for s, p, o in triples if p == "inside"}
+    return {t for t in triples
+            if t[1] == "inside" or frozenset((t[0], t[2])) not in inside_pairs}
+
+
+def score_relation_agreement(fused: dict, gt: dict, size,
+                             match: Optional[EntityMatch] = None) -> dict:
+    """Relabel-then-compare triple F1, matched (both endpoints matched) AND
+    unconditional (unmatched endpoints count as FP/FN — the joint-yield view).
+    Depth predicates are EXCLUDED (no GT); projective + inside only."""
+    match = match or match_entities(fused, gt, size)
+    gt_set = _gt_triples(gt)
+
+    pred_matched, n_pred_unmatched = set(), 0
+    for r in fused.get("relations", []):
+        ga, gb = match.fused_to_gt.get(r["a"]), match.fused_to_gt.get(r["b"])
+        for p in r.get("predicates", []):
+            if p in ("in_front_of", "behind"):
+                continue
+            if ga and gb:
+                pred_matched.add(_canon_triple(ga, p, gb))
+            else:
+                n_pred_unmatched += 1
+
+    def _f1(preds: set, gts: set, extra_fp: int = 0, extra_fn: int = 0):
+        tp = len(preds & gts)
+        n_p, n_g = len(preds) + extra_fp, len(gts) + extra_fn
+        prec = tp / n_p if n_p else 0.0
+        rec = tp / n_g if n_g else 0.0
+        return round(2 * prec * rec / (prec + rec), 4) if (prec + rec) else 0.0
+
+    # condition on RELATION-ELIGIBLE fused entities (fuse only emits relations
+    # among the top-K by saliency) — else >K-entity scenes accrue structural FNs
+    eligible = {e["id"] for e in fused.get("entities", [])
+                if e["saliency"]["rank"] <= MAX_RELATION_ENTITIES}
+    matched_gt_ids = {gid for eid, gid in match.fused_to_gt.items() if eid in eligible}
+    gt_cond = {t for t in gt_set if t[0] in matched_gt_ids and t[2] in matched_gt_ids}
+    n_gt_uncond_extra = len(gt_set) - len(gt_cond)
+    return {"matched_f1": _f1(pred_matched, gt_cond),
+            "uncond_f1": _f1(pred_matched, gt_cond,
+                             extra_fp=n_pred_unmatched, extra_fn=n_gt_uncond_extra),
+            "n_pred": len(pred_matched) + n_pred_unmatched, "n_gt": len(gt_set)}
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# STRUCTURAL checks
+# ═════════════════════════════════════════════════════════════════════════════
+
+def check_depth_consistency(fused: dict) -> dict:
+    """COCO has no depth GT — this checks INTERNAL consistency only: the
+    in_front_of digraph must be acyclic and each pair's predicate must agree with
+    the sign of its continuous depth_delta."""
+    edges = []
+    sign_ok = True
+    for r in fused.get("relations", []):
+        d = r.get("depth_delta")
+        if "in_front_of" in r.get("predicates", []):
+            edges.append((r["a"], r["b"]))
+            sign_ok &= d is None or d > 0
+        elif "behind" in r.get("predicates", []):
+            edges.append((r["b"], r["a"]))
+            sign_ok &= d is None or d < 0
+    adj = {}
+    for a, b in edges:
+        adj.setdefault(a, []).append(b)
+    seen, stack = set(), set()
+
+    def _cyclic(u):
+        seen.add(u)
+        stack.add(u)
+        for v in adj.get(u, []):
+            if v in stack or (v not in seen and _cyclic(v)):
+                return True
+        stack.discard(u)
+        return False
+
+    acyclic = not any(_cyclic(u) for u in list(adj) if u not in seen)
+    return {"pass": bool(acyclic and sign_ok), "acyclic": acyclic, "sign_ok": sign_ok}
+
+
+def check_no_invention(fused: dict, caption_structs: dict) -> dict:
+    """Every fused attribute/basin/scene-attribute/action text must be VERBATIM one
+    of the caption leaves — fusion routes and drops, never invents."""
+    leaves = set()
+    for st in caption_structs.values():
+        if not st:
+            continue
+        for s in (st.get("subjects") or []):
+            leaves.update(str(a).strip() for a in (s.get("attributes") or []))
+        leaves.update(str(a).strip() for a in (st.get("actions") or []))
+    offenders = []
+    for e in fused.get("entities", []):
+        offenders += [a["text"] for a in e.get("attributes", []) if a["text"] not in leaves]
+    offenders += [b["text"] for b in fused.get("shared_basin", []) if b["text"] not in leaves]
+    offenders += [sa["text"] for sa in fused.get("scene", {}).get("scene_attributes", [])
+                  if sa["text"] not in leaves]
+    offenders += [sa["text"] for sa in fused.get("scene", {}).get("actions", [])
+                  if sa["text"] not in leaves]
+    return {"pass": not offenders, "offenders": sorted(set(offenders))[:10]}
+
+
+# words the deterministic template itself contributes (never counted as ungrounded)
+_TEMPLATE_GLUE = frozenset("""
+the a an and or of in is are with has have to them one two three four five six seven
+eight nine ten primary subject frame camera scene style mood composition action
+visible text left right front behind above below inside nearest farthest from upper
+middle lower center people person persons dog dogs first second them somewhere
+likely either symmetry
+""".split())
+
+
+def _scene_value_tokens(obj) -> set:
+    out = set()
+    if isinstance(obj, dict):
+        for v in obj.values():
+            out |= _scene_value_tokens(v)
+    elif isinstance(obj, (list, tuple)):
+        for v in obj:
+            out |= _scene_value_tokens(v)
+    elif isinstance(obj, str):
+        out.update(_content_tokens(obj))
+        out.update(t for part in obj.split("_") for t in _content_tokens(part))
+    return out
+
+
+def check_prompt_faithfulness(fused: dict, prompt: str, re_render: Optional[str] = None) -> dict:
+    """(1) determinism (caller passes a fresh re-render), (2) every prompt content
+    token traceable to a scene string value or the fixed template vocabulary,
+    (3) inverse coverage: fraction of fused attribute/basin texts surfaced."""
+    deterministic = re_render is None or prompt == re_render
+    scene_toks = set(_scene_value_tokens(fused)) | _TEMPLATE_GLUE
+    # the template's own count renderings ("two cars") and truncated OCR quote
+    for lab, cnt in (fused.get("counts", {}).get("by_label") or {}).items():
+        scene_toks.update(_content_tokens(f"{_num(cnt)} {_plural(lab, cnt)}"))
+    ocr_txt = ((fused.get("scene") or {}).get("ocr") or {}).get("full_text", "")
+    if ocr_txt:
+        scene_toks.update(_content_tokens(ocr_txt[:120]))
+    p_toks = set(_content_tokens(prompt))
+    offending = sorted(p_toks - scene_toks)
+    grounded_rate = 1.0 - (len(offending) / len(p_toks) if p_toks else 0.0)
+
+    texts = ([a["text"] for e in fused.get("entities", []) for a in e.get("attributes", [])]
+             + [b["text"] for b in fused.get("shared_basin", [])])
+    covered = sum(1 for t in texts if t.lower() in prompt.lower())
+    coverage = covered / len(texts) if texts else 1.0
+    return {"deterministic": deterministic, "grounded_rate": round(grounded_rate, 4),
+            "offending_tokens": offending[:8], "coverage": round(coverage, 4)}
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# PROXY — attribute-binding plausibility on COCO captions
+# ═════════════════════════════════════════════════════════════════════════════
+
+_PHRASE_RE = re.compile(
+    r"\b(?:a|an|the|one|two|young|old|little)?\s*"
+    r"(" + "|".join(_PERSON_HEADS) + r")\s+"
+    r"(?:is\s+|are\s+)?(wearing|wears|in|holding|holds|with|carrying|carries)\s+"
+    r"((?:[\w-]+\s*){1,4})", re.IGNORECASE)
+
+
+def mine_person_phrases(captions: list) -> list:
+    """Regex NP miner over raw captions: person-head + wearing/holding/in/with +
+    a modifier containing a CONCRETE visual token (color/garment/holdable).
+    Abstract modifiers are rejected — they can't be grounded honestly."""
+    out, seen = [], set()
+    for ci, cap in enumerate(captions or []):
+        for m in _PHRASE_RE.finditer(cap):
+            head, verb, mod = m.group(1).lower(), m.group(2).lower(), m.group(3).strip()
+            mod_toks = _content_tokens(mod)
+            if not any(t in _CONCRETE for t in mod_toks):
+                continue
+            phrase = f"{head} {verb} {mod}".strip()
+            if phrase in seen:
+                continue
+            seen.add(phrase)
+            out.append({"phrase": phrase, "head": head, "modifier": mod,
+                        "caption_idx": ci})
+    return out
+
+
+def _gt_person_grid_masks(gt: dict, size) -> list:
+    """Rasterize every GT person (union of ALL polygon parts) onto the grid."""
+    W, H = size
+    out = []
+    for p in gt.get("persons", []):
+        acc = None
+        for poly in p.get("polygons", []):
+            m = _seg_rasterize(_seg_poly_points(poly), _GRID,
+                               _GRID / max(1.0, W), _GRID / max(1.0, H))
+            if m is not None:
+                acc = m if acc is None else (acc | m)
+        out.append(acc)
+    return out
+
+
+def score_attr_plausibility(fused: dict, grounded_phrases: list, gt: dict, size,
+                            match: Optional[EntityMatch] = None) -> dict:
+    """grounded_phrases: mined phrases + their GDINO box (pixel) + score, grounded
+    by the Colab cell. Verdict per CHECKABLE phrase: does fusion's owner map to the
+    same GT person whose mask contains the phrase box? PROXY — reported with
+    coverage and a skip histogram, never a headline number."""
+    match = match or match_entities(fused, gt, size)
+    W, H = size
+    gmasks = _gt_person_grid_masks(gt, size)
+    skips = Counter()
+    checked, agree = 0, 0
+    for g in grounded_phrases or []:
+        box = g.get("box")
+        if box is None or g.get("score", 0.0) < 0.35:
+            skips["low_score"] += 1
+            continue
+        frac_area = derive._area(box) / (W * H + 1e-9)
+        if not (0.001 <= frac_area <= 0.9):
+            skips["bad_area"] += 1
+            continue
+        x1 = int(np.clip(box[0] / W * _GRID, 0, _GRID))
+        y1 = int(np.clip(box[1] / H * _GRID, 0, _GRID))
+        x2 = int(np.clip(np.ceil(box[2] / W * _GRID), 0, _GRID))
+        y2 = int(np.clip(np.ceil(box[3] / H * _GRID), 0, _GRID))
+        cells = max(1, (x2 - x1) * (y2 - y1))
+        owners = [j for j, m in enumerate(gmasks)
+                  if m is not None and float(m[y1:y2, x1:x2].sum()) / cells >= 0.5]
+        if len(owners) != 1:
+            skips["ambiguous" if len(owners) > 1 else "no_gt_owner"] += 1
+            continue
+        gt_owner = f"p{owners[0]}"
+        # fusion side: the attribute whose tokens best overlap the modifier
+        mod_toks = set(_content_tokens(g["modifier"]))
+        best, best_ov = None, 0
+        for e in fused.get("entities", []):
+            for a in e.get("attributes", []):
+                ov = len(mod_toks & set(_content_tokens(a["text"])))
+                if ov > best_ov:
+                    best, best_ov = e["id"], ov
+        if best is None or best_ov == 0:
+            skips["no_matching_attr"] += 1
+            continue
+        fused_gt = match.fused_to_gt.get(best)
+        if fused_gt is None:
+            skips["owner_unmatched"] += 1
+            continue
+        checked += 1
+        agree += int(fused_gt == gt_owner)
+    mined = len(grounded_phrases or [])
+    return {"plausible_rate": round(agree / checked, 4) if checked else None,
+            "checked": checked, "mined": mined,
+            "coverage": round(checked / mined, 4) if mined else 0.0,
+            "skips": dict(skips)}
+
+
+# ═════════════════════════════════════════════════════════════════════════════
+# Aggregation + report
+# ═════════════════════════════════════════════════════════════════════════════
+
+def score_fusion_sample(fused: dict, prompt: str, gt: dict, *, size,
+                        re_render: Optional[str] = None,
+                        grounded_phrases: Optional[list] = None,
+                        caption_structs: Optional[dict] = None) -> dict:
+    match = match_entities(fused, gt, size)
+    out = {
+        "count": score_person_count(fused, gt),
+        "entity": score_entity_f1(fused, gt, size, match),
+        "relation": score_relation_agreement(fused, gt, size, match),
+        "depth": check_depth_consistency(fused),
+        "prompt": check_prompt_faithfulness(fused, prompt, re_render),
+        "basin_rate": (len(fused.get("shared_basin", []))
+                       / max(1, fused.get("quality", {}).get("grounding", {})
+                             .get("phrases_total", 0) or 1)),
+        "match": {"matched": len(match.fused_to_gt),
+                  "unmatched_fused": match.unmatched_fused,
+                  "unmatched_gt": match.unmatched_gt},
+    }
+    if grounded_phrases is not None:
+        out["plausibility"] = score_attr_plausibility(fused, grounded_phrases, gt,
+                                                      size, match)
+    if caption_structs is not None:
+        out["no_invention"] = check_no_invention(fused, caption_structs)
+    return out
+
+
+def score_fusion_run(results: list) -> dict:
+    n = max(1, len(results))
+    plaus = [r["plausibility"] for r in results if r.get("plausibility")]
+    checked = sum(p["checked"] for p in plaus)
+    agree = sum(round(p["plausible_rate"] * p["checked"]) for p in plaus
+                if p["plausible_rate"] is not None)
+    skips = Counter()
+    for p in plaus:
+        skips.update(p["skips"])
+    return {
+        "n": len(results),
+        "person_count_exact": round(sum(r["count"]["exact"] for r in results) / n, 4),
+        "person_count_off1": round(sum(r["count"]["off_by_one"] for r in results) / n, 4),
+        "person_count_mae": round(sum(r["count"]["abs_err"] for r in results) / n, 4),
+        "person_count_over": round(sum(r["count"]["pred"] > r["count"]["gt"]
+                                       for r in results) / n, 4),
+        "person_count_under": round(sum(r["count"]["pred"] < r["count"]["gt"]
+                                        for r in results) / n, 4),
+        "entity_f1_person": round(sum(r["entity"]["person_f1"] for r in results) / n, 4),
+        "entity_f1_all": round(sum(r["entity"]["all_f1"] for r in results) / n, 4),
+        "relation_f1_matched": round(sum(r["relation"]["matched_f1"] for r in results) / n, 4),
+        "relation_f1_uncond": round(sum(r["relation"]["uncond_f1"] for r in results) / n, 4),
+        "prompt_grounded_rate": round(sum(r["prompt"]["grounded_rate"] for r in results) / n, 4),
+        "prompt_coverage": round(sum(r["prompt"]["coverage"] for r in results) / n, 4),
+        "prompt_determinism": sum(r["prompt"]["deterministic"] for r in results),
+        "depth_consistency": sum(r["depth"]["pass"] for r in results),
+        "no_invention": all(r.get("no_invention", {}).get("pass", True) for r in results),
+        "basin_rate_mean": round(sum(r["basin_rate"] for r in results) / n, 4),
+        "plausible_rate": round(agree / checked, 4) if checked else None,
+        "plaus_checked": checked,
+        "plaus_mined": sum(p["mined"] for p in plaus),
+        "plaus_skips": dict(skips),
+    }
+
+
+def format_fusion_report(agg: dict, header: str = "") -> str:
+    n = agg["n"]
+    lines = [
+        "═" * 74,
+        f"FUSION VALIDATION — {header}  n={n} (clean 2–6-person slice; crowds + tiny "
+        "persons excluded)",
+        "═" * 74,
+        "HARD (GT-anchored)",
+        f"  person_count_exact   {agg['person_count_exact']:.2f}  (±1: {agg['person_count_off1']:.2f},  MAE {agg['person_count_mae']:.2f},  "
+        f"over {agg.get('person_count_over', 0):.2f} / under {agg.get('person_count_under', 0):.2f})",
+        f"  entity_f1 person@0.5 {agg['entity_f1_person']:.3f}    all-objects {agg['entity_f1_all']:.3f}",
+        f"  relation_f1 matched  {agg['relation_f1_matched']:.3f}    unconditional {agg['relation_f1_uncond']:.3f}",
+        f"  prompt_grounded_rate {agg['prompt_grounded_rate']:.3f}    prompt_coverage {agg['prompt_coverage']:.3f}",
+        "",
+        "PROXY (labeled honestly — never a headline)",
+        (f"  attr_binding_plausible {agg['plausible_rate']:.2f} over {agg['plaus_checked']} checkable"
+         if agg.get("plausible_rate") is not None else "  attr_binding_plausible n/a (0 checkable)"),
+        f"    mined {agg.get('plaus_mined', 0)}  skips {agg.get('plaus_skips', {})}",
+        f"  basin_rate_mean      {agg['basin_rate_mean']:.3f}",
+        "",
+        "STRUCTURAL (PASS/FAIL)",
+        f"  prompt_determinism        {agg['prompt_determinism']}/{n}",
+        f"  attr_no_invention         {'PASS' if agg['no_invention'] else 'FAIL'}",
+        f"  depth_internal_consistency {agg['depth_consistency']}/{n} (no GT — consistency only)",
+        "═" * 74,
+    ]
+    return "\n".join(lines)

qwen_test_runner/vision/metrics.py

@@ -0,0 +1,997 @@
+"""
+metrics.py — Vision scoring (replaces the text substring-grounding metric).
+
+Every category carries two UNIVERSAL metrics that encode the project's thesis —
+robust, schema-valid JSON — plus a category-specific accuracy scorer:
+
+  schema_valid : did the output validate against the category's Pydantic model
+                 (after the never-raises recovery walk)?
+  json_robust  : did it parse WITHOUT repair (clean bare JSON)?  This, measured
+                 in json_mode, is the native-capability signal driving the
+                 no-finetune decision.
+
+The headline `labeler_score` MULTIPLIES accuracy by validity and robustness, so a
+model that is accurate but emits fragile JSON scores worse than a slightly less
+accurate model that emits clean JSON — exactly what you want when pointing a
+labeler at a million images with no human in the loop.
+"""
+
+from __future__ import annotations
+
+import math
+import re
+from dataclasses import asdict, dataclass
+from typing import Callable, Optional
+
+from ..evaluator import parse_against
+from .coords import XYWH, XYXY, BBox, CoordSpace, to_canonical
+from .tasks_vision import VisionTaskSpec, model_for
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Result types
+# ──────────────────────────────────────────────────────────────────────────────
+
+@dataclass
+class MetricResult:
+    category: str
+    image_id: str
+    mode: str
+    parse_ok: bool          # a JSON object was recovered + decoded (maybe invalid schema)
+    schema_valid: bool      # validated against the category model
+    needed_repair: bool     # recovery had to strip fences / skip prose / trim junk
+    grammar_conformant: bool  # constrained decoding actually applied (backend == xgrammar)
+    primary_score: Optional[float]   # task accuracy 0..1; None if no GT / invalid
+    metrics: dict
+    needed_structural_repair: bool = False  # repair beyond a benign fence strip
+    n_output_tokens: int = 0
+    gen_seconds: float = 0.0
+    error: Optional[str] = None
+    notes: str = ""
+
+    @property
+    def json_robust(self) -> bool:
+        """Valid JSON that needed no STRUCTURAL repair — the native-capability signal.
+        A benign markdown-fence wrap is tolerated (fence-stripping is deterministic);
+        prose/runaway/malformed is not."""
+        return self.schema_valid and not self.needed_structural_repair
+
+    def to_dict(self) -> dict:
+        d = asdict(self)
+        d["json_robust"] = self.json_robust
+        return d
+
+
+@dataclass
+class VisionRunMetrics:
+    category: str
+    model: str
+    reasoning: str
+    mode: str
+    n: int
+    schema_valid_rate: float
+    json_robustness: float
+    has_task_score: bool
+    primary_score_mean: Optional[float]
+    metrics_mean: dict
+    mean_output_tokens: float
+    total_gen_seconds: float
+    tokens_per_sec: float
+    labeler_score: Optional[float]
+
+    def __str__(self) -> str:
+        acc = "n/a" if self.primary_score_mean is None else f"{self.primary_score_mean:.3f}"
+        lab = "n/a" if self.labeler_score is None else f"{self.labeler_score:.3f}"
+        return (f"[{self.model}/{self.reasoning}/{self.category}/{self.mode}] n={self.n} "
+                f"valid={self.schema_valid_rate:.1%} robust={self.json_robustness:.1%} "
+                f"acc={acc} labeler={lab} tok/s={self.tokens_per_sec:.0f}")
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# The labeler-selection composite (the verdict core)
+# ──────────────────────────────────────────────────────────────────────────────
+
+def labeler_score(accuracy: Optional[float], schema_valid_rate: float,
+                  json_robustness: float) -> Optional[float]:
+    """Multiplicative composite: accuracy × validity-gate × robustness-penalty.
+
+    Invalid JSON is unusable (hard-ish cap via the 0.5+0.5·valid term); fragile
+    but repairable JSON is penalized, not killed (0.7+0.3·robust term).
+    """
+    if accuracy is None:
+        return None
+    return accuracy * (0.5 + 0.5 * schema_valid_rate) * (0.7 + 0.3 * json_robustness)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Small pure helpers (no external deps — editdistance/jiwer are optional accel)
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _norm_text(s: str) -> str:
+    return re.sub(r"\s+", " ", (s or "").strip().lower())
+
+
+def _levenshtein(a: list, b: list) -> int:
+    if a == b:
+        return 0
+    if not a:
+        return len(b)
+    if not b:
+        return len(a)
+    prev = list(range(len(b) + 1))
+    for i, ca in enumerate(a, 1):
+        cur = [i]
+        for j, cb in enumerate(b, 1):
+            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
+        prev = cur
+    return prev[-1]
+
+
+def _cer(pred: str, gt: str) -> float:
+    g = _norm_text(gt)
+    p = _norm_text(pred)
+    if not g:
+        return 0.0 if not p else 1.0
+    return _levenshtein(list(p), list(g)) / len(g)
+
+
+def _wer(pred: str, gt: str) -> float:
+    g = _norm_text(gt).split()
+    p = _norm_text(pred).split()
+    if not g:
+        return 0.0 if not p else 1.0
+    return _levenshtein(p, g) / len(g)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Tolerant label matching — VLMs use richer / synonymous labels than dataset
+# vocabularies (e.g. "television" vs COCO's "tv"). Without this, correct boxes are
+# discarded on a string mismatch (observed: Qwen3-VL localizes COCO near-perfectly
+# but exact-match F1 read ~0.25). Synonym groups + substring + plural fallback.
+# ──────────────────────────────────────────────────────────────────────────────
+
+_SYNONYM_GROUPS = [
+    {"tv", "television", "televisions", "telly", "monitor", "screen"},
+    {"couch", "sofa", "settee", "loveseat"},
+    {"motorcycle", "motorbike", "moped", "scooter"},
+    {"airplane", "aeroplane", "plane", "aircraft", "jet"},
+    {"cell phone", "cellphone", "mobile phone", "mobile", "phone", "smartphone"},
+    {"potted plant", "houseplant", "plant", "pot plant", "flowerpot", "flower pot"},
+    {"dining table", "table", "desk"},
+    {"car", "automobile", "sedan", "vehicle"},
+    {"bicycle", "bike", "cycle"},
+    {"person", "people", "man", "men", "woman", "women", "human", "boy", "girl",
+     "child", "kid", "pedestrian", "player", "lady", "guy", "skier", "surfer",
+     "rider", "athlete", "batter", "pitcher", "catcher"},
+    {"hot dog", "hotdog"},
+    {"donut", "doughnut"},
+    {"remote", "remote control"},
+    {"sports ball", "ball"},
+    {"wine glass", "wineglass", "glass"},
+    {"tie", "necktie"},
+]
+_SYN_GROUP: dict[str, int] = {}
+for _gi, _grp in enumerate(_SYNONYM_GROUPS):
+    for _w in _grp:
+        _SYN_GROUP[_w] = _gi
+
+
+def _depluralize(t: str) -> str:
+    if len(t) <= 3:
+        return t
+    if t.endswith("ies"):
+        return t[:-3] + "y"
+    if t.endswith("es"):
+        return t[:-2]
+    if t.endswith("s"):
+        return t[:-1]
+    return t
+
+
+def labels_match(a: str, b: str) -> bool:
+    """True if two object labels refer to the same thing (tolerant)."""
+    a, b = _norm_text(a), _norm_text(b)
+    if not a or not b:
+        return False
+    if a == b:
+        return True
+    if _depluralize(a) == _depluralize(b):
+        return True
+    ga, gb = _SYN_GROUP.get(a), _SYN_GROUP.get(b)
+    if ga is not None and ga == gb:
+        return True
+    # word-level containment: "dining table" vs "table", "red car" vs "car"
+    aw, bw = set(a.split()), set(b.split())
+    if aw and bw and (aw <= bw or bw <= aw):
+        return True
+    return False
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Per-category scorers: (pred_dict, gt, ctx) -> (primary_score|None, metrics_dict)
+# ctx carries {"size": (W,H), "coord_space": CoordSpace}
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _acceptable_labels(gt) -> set[str]:
+    if isinstance(gt, dict):
+        if "labels" in gt:
+            return {_norm_text(x) for x in gt["labels"]}
+        if "label" in gt:
+            return {_norm_text(gt["label"])}
+    if isinstance(gt, str):
+        return {_norm_text(gt)}
+    return set()
+
+
+def score_classification(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    accept = _acceptable_labels(gt)
+    pred_label = _norm_text(str(pred.get("label", "")))
+    # tolerant: "spaghetti" credits "spaghetti bolognese", "tv" credits "television"
+    top1 = 1.0 if any(labels_match(pred_label, a) for a in accept) else 0.0
+    top5_labels = {_norm_text(str(d.get("label", ""))) for d in (pred.get("top5") or [])}
+    top5_labels.add(pred_label)
+    top5 = 1.0 if any(labels_match(p, a) for p in top5_labels for a in accept) else 0.0
+    return top1, {"top1": top1, "top5": top5}
+
+
+def _gt_boxes_to_canonical(gt, size) -> list[tuple[str, BBox]]:
+    out = []
+    for b in (gt.get("boxes") if isinstance(gt, dict) else []) or []:
+        label = _norm_text(str(b.get("label", "")))
+        fmt = b.get("fmt", XYWH)
+        box = to_canonical(b["bbox"], CoordSpace.PIXEL_ABS, size, fmt=fmt)
+        out.append((label, box))
+    return out
+
+
+def _greedy_match_f1(preds, gts, iou_thr, require_label) -> tuple[float, float, float, int]:
+    """Greedy IoU matching (preds pre-sorted by score). Returns (precision, recall, f1, tp).
+    `require_label` toggles labeled vs class-agnostic matching."""
+    matched: set[int] = set()
+    tp = 0
+    for plabel, _score, pbox in preds:
+        best_gi, best_iou = -1, iou_thr
+        for gi, (glabel, gbox) in enumerate(gts):
+            if gi in matched:
+                continue
+            if require_label and not labels_match(plabel, glabel):
+                continue
+            iou = pbox.iou(gbox)
+            if iou >= best_iou:
+                best_gi, best_iou = gi, iou
+        if best_gi >= 0:
+            matched.add(best_gi)
+            tp += 1
+    fp = len(preds) - tp
+    fn = len(gts) - len(matched)
+    precision = tp / (tp + fp) if (tp + fp) else (1.0 if not gts else 0.0)
+    recall = tp / (tp + fn) if (tp + fn) else 1.0
+    f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) else 0.0
+    return precision, recall, f1, tp
+
+
+def score_detection(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    size = ctx["size"]
+    space = ctx["coord_space"]
+    gts = _gt_boxes_to_canonical(gt, size)
+
+    preds = []
+    for d in (pred.get("detections") or []):
+        box_raw = d.get("box")
+        if not (isinstance(box_raw, (list, tuple)) and len(box_raw) == 4):
+            continue
+        try:
+            box = to_canonical(box_raw, space, size, fmt=XYXY)
+        except (ValueError, TypeError):
+            continue
+        preds.append((_norm_text(str(d.get("label", ""))), float(d.get("score", 1.0) or 1.0), box))
+    preds.sort(key=lambda t: t[1], reverse=True)
+
+    iou_thr = 0.5
+    # labeled (tolerant) match — the headline accuracy
+    precision, recall, f1, _ = _greedy_match_f1(preds, gts, iou_thr, require_label=True)
+    # class-agnostic localization — "can it find/box objects" regardless of naming
+    loc_p, loc_r, loc_f1, _ = _greedy_match_f1(preds, gts, iou_thr, require_label=False)
+
+    pred_count = pred.get("count")
+    count_err = abs(int(pred_count) - len(gts)) if isinstance(pred_count, (int, float)) else len(preds) - len(gts)
+    return f1, {"precision": precision, "recall": recall, "f1": f1,
+                "localization_f1": loc_f1, "localization_recall": loc_r,
+                "iou_thr": iou_thr, "count_abs_err": float(abs(count_err)),
+                "n_pred": float(len(preds)), "n_gt": float(len(gts))}
+
+
+def score_ocr(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    gt_text = gt.get("text") if isinstance(gt, dict) else str(gt)
+    pred_text = str(pred.get("full_text", ""))
+    cer = _cer(pred_text, gt_text)
+    wer = _wer(pred_text, gt_text)
+    exact = 1.0 if _norm_text(pred_text) == _norm_text(gt_text) else 0.0
+    # answer-containment credit (TextVQA-style: GT is the answer phrase)
+    contains = 1.0 if _norm_text(gt_text) and _norm_text(gt_text) in _norm_text(pred_text) else 0.0
+    primary = max(exact, contains, max(0.0, 1.0 - cer))
+    return primary, {"cer": cer, "wer": wer, "exact": exact, "contains": contains}
+
+
+def score_datatype_diff(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Did the model identify the rendered data format (json/yaml/md/...)?"""
+    gt_type = _norm_text(gt.get("data_type") if isinstance(gt, dict) else str(gt))
+    pred_type = _norm_text(str(pred.get("data_type", "")))
+    ok = 1.0 if pred_type == gt_type else 0.0
+    return ok, {"type_acc": ok}
+
+
+def _flatten_kv(obj, prefix="") -> set[str]:
+    """Flatten a parsed JSON-ish object to a set of 'path=value' leaf strings."""
+    out: set[str] = set()
+    if isinstance(obj, dict):
+        for k, v in obj.items():
+            out |= _flatten_kv(v, f"{prefix}{k}.")
+    elif isinstance(obj, list):
+        for i, v in enumerate(obj):
+            out |= _flatten_kv(v, f"{prefix}{i}.")
+    else:
+        out.add(f"{prefix.rstrip('.')}={_norm_text(str(obj))}")
+    return out
+
+
+def score_datatype_util(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Did the model re-emit the rendered data as normalized JSON matching the GT?
+    Scored by leaf 'path=value' F1 (order-independent), plus a type-correct bonus."""
+    import json as _json
+    gt_obj = gt.get("content") if isinstance(gt, dict) else gt
+    if isinstance(gt_obj, str):
+        try:
+            gt_obj = _json.loads(gt_obj)
+        except (ValueError, TypeError):
+            pass
+    pred_content = pred.get("content")
+    if isinstance(pred_content, str):
+        try:
+            pred_content = _json.loads(pred_content)
+        except (ValueError, TypeError):
+            pass  # leave as string → flatten will treat as a single leaf
+    g = _flatten_kv(gt_obj)
+    p = _flatten_kv(pred_content)
+    if not g:
+        return (1.0 if not p else 0.0), {"kv_f1": 0.0}
+    tp = len(g & p)
+    prec = tp / len(p) if p else 0.0
+    rec = tp / len(g)
+    f1 = (2 * prec * rec / (prec + rec)) if (prec + rec) else 0.0
+    type_ok = 1.0 if _norm_text(str(pred.get("data_type", ""))) == _norm_text(
+        gt.get("data_type", "") if isinstance(gt, dict) else "") else 0.0
+    return f1, {"kv_f1": f1, "kv_precision": prec, "kv_recall": rec, "type_acc": type_ok}
+
+
+_SPATIAL_PRED_NORM = {
+    "left of": "left_of", "to the left of": "left_of", "left": "left_of",
+    "right of": "right_of", "to the right of": "right_of", "right": "right_of",
+    "in front of": "in_front_of", "front of": "in_front_of",
+}
+
+
+def _norm_pred(p: str) -> str:
+    n = _norm_text(p)
+    return _SPATIAL_PRED_NORM.get(n, n.replace(" ", "_"))
+
+
+def _pred_triples(pred: dict) -> list[tuple]:
+    """Extract (subject, predicate, object) triples from either the spatial
+    ('relations': [{subject,predicate,object}]) or semantic
+    ('associations': [{a,relation,b}]) shape."""
+    out = []
+    for r in (pred.get("relations") or []):
+        if isinstance(r, dict):
+            out.append((_norm_text(str(r.get("subject", ""))), _norm_pred(str(r.get("predicate", ""))),
+                        _norm_text(str(r.get("object", "")))))
+    for r in (pred.get("associations") or []):
+        if isinstance(r, dict):
+            out.append((_norm_text(str(r.get("a", ""))), _norm_pred(str(r.get("relation", ""))),
+                        _norm_text(str(r.get("b", "")))))
+    return out
+
+
+def score_triples(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Triple-F1 with tolerant subject/object matching + exact (normalized) predicate."""
+    gts = [(_norm_text(s), _norm_pred(p), _norm_text(o))
+           for s, p, o in (gt.get("triples", []) if isinstance(gt, dict) else [])]
+    preds = _pred_triples(pred)
+    matched = set()
+    tp = 0
+    for ps, pp, po in preds:
+        for gi, (gs, gp, go) in enumerate(gts):
+            if gi in matched or pp != gp:
+                continue
+            if labels_match(ps, gs) and labels_match(po, go):
+                matched.add(gi)
+                tp += 1
+                break
+    prec = tp / len(preds) if preds else (1.0 if not gts else 0.0)
+    rec = tp / len(gts) if gts else 1.0
+    f1 = (2 * prec * rec / (prec + rec)) if (prec + rec) else 0.0
+    return f1, {"triple_f1": f1, "precision": prec, "recall": rec,
+                "n_pred": float(len(preds)), "n_gt": float(len(gts))}
+
+
+def score_depth_order(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Pairwise nearer/farther ordering accuracy over the GT pairs."""
+    gpairs = gt.get("pairs", []) if isinstance(gt, dict) else []
+    preds = pred.get("relative_depth") or []
+
+    def _find(a, b):
+        for r in preds:
+            if not isinstance(r, dict):
+                continue
+            ra, rb = _norm_text(str(r.get("a", ""))), _norm_text(str(r.get("b", "")))
+            if labels_match(ra, a) and labels_match(rb, b):
+                return _norm_text(str(r.get("a_is", "")))
+            if labels_match(ra, b) and labels_match(rb, a):  # reversed → flip
+                v = _norm_text(str(r.get("a_is", "")))
+                return {"nearer": "farther", "farther": "nearer"}.get(v, v)
+        return None
+
+    correct = 0
+    for p in gpairs:
+        got = _find(_norm_text(p["a"]), _norm_text(p["b"]))
+        if got == _norm_text(p["a_is"]):
+            correct += 1
+    acc = correct / len(gpairs) if gpairs else (1.0 if not preds else 0.0)
+    return acc, {"order_acc": acc, "n_gt_pairs": float(len(gpairs))}
+
+
+def score_subject_fixation(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """IoU of the predicted primary-subject box vs the GT salient box (+ label)."""
+    size = ctx["size"]
+    space = ctx["coord_space"]
+    ps = pred.get("primary_subject") or {}
+    raw = ps.get("box") if isinstance(ps, dict) else None
+    if not (isinstance(raw, (list, tuple)) and len(raw) == 4):
+        return 0.0, {"iou": 0.0, "label_ok": 0.0}
+    try:
+        pbox = to_canonical(raw, space, size, fmt=XYXY)
+    except (ValueError, TypeError):
+        return 0.0, {"iou": 0.0, "label_ok": 0.0}
+    gbox = to_canonical(gt["box"], CoordSpace.PIXEL_ABS, size, fmt=gt.get("fmt", XYXY))
+    iou = pbox.iou(gbox)
+    label_ok = 1.0 if labels_match(str(ps.get("label", "")), str(gt.get("label", ""))) else 0.0
+    primary = 1.0 if (iou >= 0.5 and label_ok) else (0.5 if iou >= 0.5 else 0.0)
+    return primary, {"iou": iou, "label_ok": label_ok}
+
+
+def _seg_poly_points(flat, sx=1.0, sy=1.0):
+    """Flat [x1,y1,x2,y2,...] -> list of (x,y) tuples scaled by (sx,sy).
+    Tolerant: ignores a trailing odd value and skips non-numeric entries."""
+    pts = []
+    if not isinstance(flat, (list, tuple)):
+        return pts
+    n = (len(flat) // 2) * 2
+    for i in range(0, n, 2):
+        try:
+            x = float(flat[i]) * sx
+            y = float(flat[i + 1]) * sy
+        except (TypeError, ValueError):
+            continue
+        pts.append((x, y))
+    # Tolerate a 4-number bbox-as-polygon: expand [x1,y1,x2,y2] -> rectangle corners.
+    if len(pts) == 2:
+        (x1, y1), (x2, y2) = pts
+        pts = [(x1, y1), (x2, y1), (x2, y2), (x1, y2)]
+    return pts
+
+
+def _seg_rasterize(points, grid, gsx, gsy):
+    """Fill a polygon (pixel-coord points) onto a grid×grid boolean mask.
+    Maps pixel->grid via (px*gsx, py*gsy). Returns a bool ndarray or None."""
+    import numpy as np
+    from PIL import Image, ImageDraw
+    if len(points) < 3:
+        return None
+    mapped = [(px * gsx, py * gsy) for (px, py) in points]
+    img = Image.new("L", (grid, grid), 0)
+    ImageDraw.Draw(img).polygon(mapped, fill=1)
+    return np.asarray(img, dtype=bool)
+
+
+def score_segmentation(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Instance-segmentation mIoU. For each GT mask, greedily match a predicted
+    mask of the same (tolerant) label by polygon IoU, computed by rasterizing
+    both polygons onto a shared grid. mIoU is averaged over GT masks.
+
+    GT polygons (gt['masks'][i]['polygon_pixels']) are absolute pixels.
+    Predicted polygons (pred['masks'][i]['polygon']) are in ctx['coord_space']
+    (NORM_0_1000 by default) and are scaled to pixels before rasterizing.
+    Never raises: missing / short / malformed polygons score 0 for that mask."""
+    import numpy as np
+
+    size = ctx.get("size", (1, 1))
+    space = ctx.get("coord_space", CoordSpace.NORM_0_1000)
+    if isinstance(size, (list, tuple)) and len(size) == 2:
+        W, H = size
+    else:
+        W, H = (1, 1)
+    W = max(int(W or 1), 1)
+    H = max(int(H or 1), 1)
+
+    # GT masks (pixel coords). Accept 'polygon_pixels' or 'polygon' as a fallback.
+    gts = []
+    for m in (gt.get("masks") if isinstance(gt, dict) else []) or []:
+        if not isinstance(m, dict):
+            continue
+        poly = m.get("polygon_pixels")
+        if poly is None:
+            poly = m.get("polygon") or []
+        pts = _seg_poly_points(poly)
+        if len(pts) >= 3:
+            gts.append((_norm_text(str(m.get("label", ""))), pts))
+
+    # Predicted masks: scale ctx-space polygons to pixels.
+    if space == CoordSpace.NORM_0_1:
+        sx, sy = float(W), float(H)
+    elif space == CoordSpace.NORM_0_1000:
+        sx, sy = float(W) / 1000.0, float(H) / 1000.0
+    else:  # PIXEL_ABS or unknown -> treat as pixels
+        sx, sy = 1.0, 1.0
+    preds = []
+    for m in (pred.get("masks") if isinstance(pred, dict) else []) or []:
+        if not isinstance(m, dict):
+            continue
+        pts = _seg_poly_points(m.get("polygon") or [], sx, sy)
+        if len(pts) >= 3:
+            preds.append((_norm_text(str(m.get("label", ""))), pts))
+
+    if not gts:
+        ok = 1.0 if not preds else 0.0
+        return ok, {"miou": ok, "n_pred": float(len(preds)), "n_gt": 0.0, "matched": 0.0}
+
+    # Shared raster grid. Per-axis scale preserves the image aspect ratio so a
+    # non-square image doesn't distort IoU. 128 keeps cost trivial.
+    GRID = 128
+    gsx = GRID / float(W)
+    gsy = GRID / float(H)
+    gt_masks = [(lbl, _seg_rasterize(pts, GRID, gsx, gsy)) for (lbl, pts) in gts]
+    pred_masks = [(lbl, _seg_rasterize(pts, GRID, gsx, gsy)) for (lbl, pts) in preds]
+
+    used: set = set()
+    ious = []
+    matched = 0
+    for glabel, garr in gt_masks:
+        if garr is None or not garr.any():
+            ious.append(0.0)
+            continue
+        best_iou, best_j = 0.0, -1
+        for j, (plabel, parr) in enumerate(pred_masks):
+            if j in used or parr is None:
+                continue
+            if not labels_match(plabel, glabel):
+                continue
+            inter = int(np.logical_and(garr, parr).sum())
+            if inter == 0:
+                continue
+            union = int(np.logical_or(garr, parr).sum())
+            iou = inter / union if union else 0.0
+            if iou > best_iou:
+                best_iou, best_j = iou, j
+        if best_j >= 0:
+            used.add(best_j)
+            matched += 1
+        ious.append(best_iou)
+
+    miou = sum(ious) / len(ious) if ious else 0.0
+    return miou, {"miou": miou, "n_pred": float(len(preds)),
+                  "n_gt": float(len(gts)), "matched": float(matched)}
+
+
+def _outline_points(flat, space, size):
+    """Parse a flat [x1,y1,x2,y2,...] list into pixel (x,y) vertices.
+    Each vertex is run through to_canonical as a degenerate 1px box so the
+    documented space/clip handling is reused. Robust to odd length / junk;
+    returns [] (no polygon) if fewer than 3 usable vertices."""
+    import math as _math
+    if not isinstance(flat, (list, tuple)):
+        return []
+    pts = []
+    n = len(flat) // 2
+    for k in range(n):
+        try:
+            x = float(flat[2 * k]); y = float(flat[2 * k + 1])
+        except (TypeError, ValueError, IndexError):
+            continue
+        if _math.isnan(x) or _math.isnan(y) or _math.isinf(x) or _math.isinf(y):
+            continue
+        b = to_canonical([x, y, x, y], space, size, fmt=XYXY)  # scales space + clips to image
+        pts.append((b.x1, b.y1))
+    # Tolerate a 4-number bbox-as-outline: expand [x1,y1,x2,y2] -> rectangle corners.
+    if len(pts) == 2:
+        (x1, y1), (x2, y2) = pts
+        pts = [(x1, y1), (x2, y1), (x2, y2), (x1, y2)]
+    return pts if len(pts) >= 3 else []
+
+
+def _outline_raster_iou(poly_a, poly_b, size, max_dim=200):
+    """Polygon IoU by scanline rasterization on a downscaled grid (<= max_dim on
+    the long side). Even-odd fill; handles convex + concave outlines. Never raises."""
+    import math as _math
+    import numpy as np
+    W, H = size
+    if len(poly_a) < 3 or len(poly_b) < 3:
+        return 0.0
+    scale = min(1.0, float(max_dim) / max(1, max(W, H)))
+    rw = max(1, int(round(W * scale))); rh = max(1, int(round(H * scale)))
+
+    def _fill(poly):
+        grid = np.zeros((rh, rw), dtype=bool)
+        xs = [p[0] * scale for p in poly]
+        ys = [p[1] * scale for p in poly]
+        m = len(poly)
+        y0 = max(0, int(_math.floor(min(ys)))); y1 = min(rh - 1, int(_math.ceil(max(ys))))
+        for yy in range(y0, y1 + 1):
+            yc = yy + 0.5
+            xint = []
+            for i in range(m):
+                xi, yi = xs[i], ys[i]
+                xj, yj = xs[(i + 1) % m], ys[(i + 1) % m]
+                if (yi <= yc < yj) or (yj <= yc < yi):
+                    xint.append(xi + (yc - yi) / (yj - yi) * (xj - xi))
+            xint.sort()
+            for c in range(0, len(xint) - 1, 2):
+                xa = max(0, int(_math.ceil(xint[c] - 0.5)))
+                xb = min(rw - 1, int(_math.floor(xint[c + 1] - 0.5)))
+                if xb >= xa:
+                    grid[yy, xa:xb + 1] = True
+        return grid
+
+    a = _fill(poly_a); b = _fill(poly_b)
+    inter = int(np.logical_and(a, b).sum())
+    union = int(np.logical_or(a, b).sum())
+    return inter / union if union else 0.0
+
+
+def score_outline_iou(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Polygon IoU (rasterized) of the predicted main-object outline vs the GT
+    outline, gated by a tolerant label match.
+      primary = 0.0                       if IoU < 0.5
+              = 1.0 if label matches else 0.5   if IoU >= 0.5
+    Pred outline is in ctx['coord_space']; GT outline is pixel-abs. Never raises."""
+    size = ctx["size"]
+    space = ctx["coord_space"]
+    pred_poly = _outline_points(pred.get("outline"), space, size)
+    if isinstance(gt, dict):
+        gt_flat = gt.get("outline") or []
+        gt_label = str(gt.get("label", ""))
+    else:
+        gt_flat, gt_label = [], ""
+    gt_poly = _outline_points(gt_flat, CoordSpace.PIXEL_ABS, size)
+    iou = _outline_raster_iou(pred_poly, gt_poly, size)
+    label_ok = 1.0 if labels_match(str(pred.get("label", "")), gt_label) else 0.0
+    if iou >= 0.5:
+        primary = 1.0 if label_ok else 0.5
+    else:
+        primary = 0.0
+    return primary, {"poly_iou": iou, "label_ok": label_ok,
+                     "n_pred_pts": float(len(pred_poly)), "n_gt_pts": float(len(gt_poly))}
+
+
+def _as_xyzwhl_yaw(b):
+    """Coerce a bbox3d list to 7 floats [x,y,z,w,h,l,yaw]; pad missing yaw.
+    Returns None if fewer than 6 usable numbers (need at least center+size)."""
+    if not isinstance(b, (list, tuple)):
+        return None
+    vals = []
+    for v in b[:7]:
+        try:
+            vals.append(float(v))
+        except (TypeError, ValueError):
+            vals.append(0.0)
+    if len(vals) < 6:
+        return None
+    while len(vals) < 7:
+        vals.append(0.0)  # missing yaw -> 0
+    return vals
+
+
+def score_iou3d(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """3D-center matching for the simplified ground-plane proxy.
+
+    A predicted object matches a GT object when (a) their 3D-center L2 distance is
+    below `dist_thr` (normalized units) and (b) the size ratio is reasonable (each
+    of w,h,l within [1/size_tol, size_tol]); class is checked separately for credit.
+    Headline = (0.5*matched + 0.5*class_correct) / n_gt, so a box in the right place
+    but mislabeled earns half credit. Reports center distance + class/recall metrics.
+    Never raises: short/missing bbox3d entries are dropped, not fatal.
+    """
+    import math
+    dist_thr = 0.3
+    size_tol = 3.0
+
+    gts = gt.get("objects", []) if isinstance(gt, dict) else []
+    raw_preds = pred.get("objects") if isinstance(pred, dict) else None
+    preds = []
+    for d in (raw_preds or []):
+        if not isinstance(d, dict):
+            continue
+        vec = _as_xyzwhl_yaw(d.get("bbox3d"))
+        if vec is None:
+            continue
+        cls = _norm_text(str(d.get("class", d.get("label", ""))))
+        try:
+            sc = float(d.get("score", 1.0))
+        except (TypeError, ValueError):
+            sc = 1.0
+        preds.append((cls, sc, vec))
+    preds.sort(key=lambda t: t[1], reverse=True)
+
+    if not gts:
+        return (1.0 if not preds else 0.0), {"matched_frac": 0.0, "precision": 0.0,
+                                             "class_acc": 0.0, "center_dist": float(dist_thr),
+                                             "n_pred": float(len(preds)), "n_gt": 0.0}
+
+    matched_gt: set = set()
+    n_class_ok = 0
+    dist_sum = 0.0
+    dist_n = 0
+    for pcls, _sc, pvec in preds:
+        best_gi, best_d = -1, dist_thr
+        for gi, g in enumerate(gts):
+            if gi in matched_gt:
+                continue
+            gvec = _as_xyzwhl_yaw(g.get("bbox3d"))
+            if gvec is None:
+                continue
+            dx, dy, dz = pvec[0] - gvec[0], pvec[1] - gvec[1], pvec[2] - gvec[2]
+            d3 = math.sqrt(dx * dx + dy * dy + dz * dz)
+            if d3 > best_d:
+                continue
+            ok_size = True
+            for idx in (3, 4, 5):  # w, h, l
+                ps, gs = abs(pvec[idx]), abs(gvec[idx])
+                if gs <= 1e-6:
+                    continue
+                r = (ps / gs) if ps > 1e-6 else 0.0
+                if r < (1.0 / size_tol) or r > size_tol:
+                    ok_size = False
+                    break
+            if not ok_size:
+                continue
+            best_gi, best_d = gi, d3
+        if best_gi >= 0:
+            matched_gt.add(best_gi)
+            dist_sum += best_d
+            dist_n += 1
+            gcls = _norm_text(str(gts[best_gi].get("class", gts[best_gi].get("label", ""))))
+            if labels_match(pcls, gcls):
+                n_class_ok += 1
+
+    n_gt = len(gts)
+    n_pred = len(preds)
+    matched = len(matched_gt)
+    recall = matched / n_gt
+    precision = matched / n_pred if n_pred else 0.0
+    class_acc = n_class_ok / matched if matched else 0.0
+    mean_center_dist = (dist_sum / dist_n) if dist_n else float(dist_thr)
+    primary = (0.5 * matched + 0.5 * n_class_ok) / n_gt
+    return primary, {"matched_frac": recall, "precision": precision,
+                     "class_acc": class_acc, "center_dist": mean_center_dist,
+                     "n_pred": float(n_pred), "n_gt": float(n_gt)}
+
+
+def _wrap_deg(a: float) -> float:
+    """Wrap an angle (degrees) into [-180, 180)."""
+    return (float(a) + 180.0) % 360.0 - 180.0
+
+
+def score_camera_rotation(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Camera rotation: mean absolute angular error across yaw/pitch/roll, each wrapped
+    to [-180,180). primary = acc@30deg = fraction of the 3 axes within 30 deg of GT.
+    Robust to missing / short / non-numeric rotation lists (never raises)."""
+    gt_rot = (gt or {}).get("rotation") if isinstance(gt, dict) else None
+    if not isinstance(gt_rot, (list, tuple)) or len(gt_rot) < 3:
+        return None, {}
+    raw = pred.get("rotation")
+    p = list(raw) if isinstance(raw, (list, tuple)) else []
+    p = (p + [0.0, 0.0, 0.0])[:3]  # pad missing axes with 0 so a short list scores, not crashes
+    errs, within = [], 0
+    for i in range(3):
+        try:
+            d = abs(_wrap_deg(float(p[i]) - float(gt_rot[i])))
+        except (TypeError, ValueError):
+            d = 180.0
+        errs.append(d)
+        if d <= 30.0:
+            within += 1
+    mean_abs_err = sum(errs) / 3.0
+    acc30 = within / 3.0
+    return acc30, {"acc@30deg": acc30, "mean_abs_err": mean_abs_err,
+                   "yaw_err": errs[0], "pitch_err": errs[1], "roll_err": errs[2]}
+
+
+def score_vqa(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Grounded-VQA answer accuracy (OCR-style containment, reused here).
+
+    Credit if the predicted answer EQUALS any gold answer, CONTAINS a gold
+    answer, or is CONTAINED IN a gold answer (all after _norm_text). Reports
+    `exact` and `contains` separately. The optional grounded_region box is NOT
+    scored — VQA datasets carry no per-question GT box, so answer text is the
+    only signal. Never raises: missing/short fields collapse to a 0.0 score.
+    """
+    golds = _vqa_gold_answers(gt)
+    pred_ans = _norm_text(str(pred.get("answer", "")))
+    if not golds:
+        # no GT answers -> only an empty prediction can be "correct"
+        return (1.0 if not pred_ans else 0.0), {"exact": 0.0, "contains": 0.0}
+    exact = 1.0 if pred_ans in golds else 0.0
+    contains = 0.0
+    if pred_ans:
+        for g in golds:
+            if g and (g in pred_ans or pred_ans in g):
+                contains = 1.0
+                break
+    primary = max(exact, contains)
+    return primary, {"exact": exact, "contains": contains}
+
+
+def _vqa_gold_answers(gt) -> list[str]:
+    """Normalize GT into a list of acceptable (normalized) gold answer strings.
+    Accepts {"answers": [...]}, {"answers": "x"}, {"answer": "x"}, a bare list,
+    or a bare string. Anything else -> []."""
+    if isinstance(gt, dict):
+        a = gt.get("answers")
+        if isinstance(a, (list, tuple)):
+            return [_norm_text(str(x)) for x in a if str(x).strip()]
+        if a is not None:
+            return [_norm_text(str(a))]
+        if "answer" in gt:
+            return [_norm_text(str(gt["answer"]))]
+    if isinstance(gt, (list, tuple)):
+        return [_norm_text(str(x)) for x in gt if str(x).strip()]
+    if isinstance(gt, str):
+        return [_norm_text(gt)]
+    return []
+
+
+def score_style(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Style accuracy = 1.0 if pred style matches gt style (tolerant via labels_match),
+    else 0.0. Layout/symmetry accuracy reported as side metrics when the GT carries them.
+    Robust: never raises; guards missing/short fields and non-dict GT."""
+    def _acc(pred_val, gt_val):
+        if gt_val is None:
+            return None
+        gv = _norm_text(str(gt_val))
+        if not gv:
+            return None
+        pv = _norm_text(str(pred_val if pred_val is not None else ""))
+        if not pv:
+            return 0.0
+        return 1.0 if (pv == gv or labels_match(pv, gv)) else 0.0
+
+    if not isinstance(gt, dict):
+        gt = {"style": gt}
+
+    style_acc = _acc(pred.get("style"), gt.get("style"))
+    metrics: dict = {}
+    if style_acc is not None:
+        metrics["style_acc"] = style_acc
+    layout_acc = _acc(pred.get("layout"), gt.get("layout"))
+    if layout_acc is not None:
+        metrics["layout_acc"] = layout_acc
+    symmetry_acc = _acc(pred.get("symmetry"), gt.get("symmetry"))
+    if symmetry_acc is not None:
+        metrics["symmetry_acc"] = symmetry_acc
+
+    # Headline accuracy is style accuracy (the category's defining axis).
+    primary = style_acc if style_acc is not None else None
+    return primary, metrics
+
+
+def score_schema_only(pred: dict, gt, ctx) -> tuple[Optional[float], dict]:
+    """Stub categories: no GT wired yet → no task accuracy, only universal metrics."""
+    return None, {}
+
+
+_SCORERS: dict[str, Callable] = {
+    "classification": score_classification,
+    "detection": score_detection,
+    "ocr": score_ocr,
+    "datatype_diff": score_datatype_diff,
+    "datatype_util": score_datatype_util,
+    "triples": score_triples,
+    "depth_order": score_depth_order,
+    "subject_fixation": score_subject_fixation,
+    "segmentation": score_segmentation,
+    "outline_iou": score_outline_iou,
+    "iou3d": score_iou3d,
+    "angular_error": score_camera_rotation,
+    "vqa": score_vqa,
+    "style": score_style,
+    "schema_only": score_schema_only,
+}
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Sample + run scoring
+# ──────────────────────────────────────────────────────────────────────────────
+
+def score_vision_sample(
+    spec: VisionTaskSpec,
+    raw_output: str,
+    gt,
+    *,
+    mode: str,
+    image_id: str,
+    image_size: tuple[int, int],
+    grammar_conformant: bool = False,
+    n_output_tokens: int = 0,
+    gen_seconds: float = 0.0,
+) -> MetricResult:
+    parse = parse_against(raw_output, model_for(spec))
+    parse_ok = parse.schema_valid or (parse.error or "").startswith("schema:")
+
+    if not parse.schema_valid or parse.parsed is None:
+        return MetricResult(
+            category=spec.category, image_id=image_id, mode=mode,
+            parse_ok=parse_ok, schema_valid=False, needed_repair=parse.needed_repair,
+            needed_structural_repair=parse.needed_structural_repair,
+            grammar_conformant=grammar_conformant, primary_score=None, metrics={},
+            n_output_tokens=n_output_tokens, gen_seconds=gen_seconds, error=parse.error,
+        )
+
+    pred = parse.parsed.model_dump()
+    ctx = {"size": image_size, "coord_space": spec.coord_space}
+    scorer = _SCORERS.get(spec.metric, score_schema_only)
+    try:
+        primary, m = scorer(pred, gt, ctx)
+    except Exception as e:  # a scorer bug must never crash a long run
+        primary, m = None, {}
+        return MetricResult(
+            category=spec.category, image_id=image_id, mode=mode, parse_ok=True,
+            schema_valid=True, needed_repair=parse.needed_repair,
+            needed_structural_repair=parse.needed_structural_repair,
+            grammar_conformant=grammar_conformant, primary_score=None, metrics={},
+            n_output_tokens=n_output_tokens, gen_seconds=gen_seconds,
+            error=f"scorer error: {e}",
+        )
+    return MetricResult(
+        category=spec.category, image_id=image_id, mode=mode, parse_ok=True,
+        schema_valid=True, needed_repair=parse.needed_repair,
+        needed_structural_repair=parse.needed_structural_repair,
+        grammar_conformant=grammar_conformant, primary_score=primary, metrics=m,
+        n_output_tokens=n_output_tokens, gen_seconds=gen_seconds,
+    )
+
+
+def score_vision_run(results: list[MetricResult], *, model: str = "", reasoning: str = "",
+                     category: str = "", mode: str = "") -> VisionRunMetrics:
+    n = len(results)
+    if n == 0:
+        return VisionRunMetrics(category, model, reasoning, mode, 0, 0.0, 0.0,
+                                False, None, {}, 0.0, 0.0, 0.0, None)
+    valid = [r for r in results if r.schema_valid]
+    schema_valid_rate = len(valid) / n
+    json_robustness = sum(1 for r in results if r.json_robust) / n
+    scored = [r for r in valid if r.primary_score is not None]
+    has_task = bool(scored)
+    primary_mean = (sum(r.primary_score for r in scored) / len(scored)) if scored else None
+
+    # average the per-sample metric dicts
+    metrics_mean: dict = {}
+    keys = set()
+    for r in scored:
+        keys |= set(r.metrics.keys())
+    for k in keys:
+        vals = [r.metrics[k] for r in scored if k in r.metrics and not math.isnan(r.metrics[k])]
+        if vals:
+            metrics_mean[k] = sum(vals) / len(vals)
+
+    total_tokens = sum(r.n_output_tokens for r in results)
+    total_secs = sum(r.gen_seconds for r in results)
+    mean_tokens = total_tokens / n
+    tok_per_sec = (total_tokens / total_secs) if total_secs > 0 else 0.0
+
+    return VisionRunMetrics(
+        category=category or results[0].category, model=model, reasoning=reasoning,
+        mode=mode or results[0].mode, n=n,
+        schema_valid_rate=schema_valid_rate, json_robustness=json_robustness,
+        has_task_score=has_task, primary_score_mean=primary_mean, metrics_mean=metrics_mean,
+        mean_output_tokens=mean_tokens, total_gen_seconds=total_secs, tokens_per_sec=tok_per_sec,
+        labeler_score=labeler_score(primary_mean, schema_valid_rate, json_robustness),
+    )

qwen_test_runner/vision/model_registry.py

@@ -0,0 +1,275 @@
+"""
+model_registry.py — The Qwen VLM model universe (analogous to SLOT_REGISTRY).
+
+One ModelSpec per checkpoint family. The benchmark iterates this registry to
+decide what to load on the 96GB RTX 6000 Pro. Both generations are natively
+multimodal (every checkpoint has its own ViT):
+
+  * Qwen3.5   (qwen3_5 / qwen3_5_moe, AutoModelForMultimodalLM) — `enable_thinking`
+              toggle on a single checkpoint.
+  * Qwen3-VL  (qwen3_vl / qwen3_vl_moe, AutoModelForImageTextToText) — separate
+              -Instruct / -Thinking checkpoints.
+
+VRAM rule of thumb (weights only): bf16 ≈ 2·B, fp8 ≈ B, int4 ≈ 0.55·B GB.
+For MoE, ALL experts are resident, so total params drive memory; active params
+drive decode speed (and thus throughput / fleet score).
+
+Nothing hardcodes a repo id outside this file — it is the single source of truth
+for the model ladder, exactly as registry.py is for the schema.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Literal, Optional
+
+Family = Literal["qwen3_5", "qwen3_5_moe", "qwen3_vl", "qwen3_vl_moe", "joycaption"]
+LoaderKind = Literal["multimodal_lm", "image_text_to_text", "llava_conditional"]
+ReasoningMode = Literal["toggle", "separate_ckpt", "none"]
+Precision = Literal["bf16", "fp8", "int4"]
+Reasoning = Literal["instruct", "thinking"]
+
+# weights-only bytes-per-parameter, in GB-per-billion-params
+_BYTES_PER_B: dict[str, float] = {"bf16": 2.0, "fp8": 1.0, "int4": 0.55}
+
+# Headroom reserved for the vision encoder, activations, and KV cache.
+VRAM_HEADROOM_GB = 12.0
+GPU_VRAM_GB = 96.0
+
+
+@dataclass(frozen=True)
+class ModelSpec:
+    key: str                         # stable short id used on the CLI + in filenames
+    repo_id: str                     # the Instruct / default checkpoint
+    family: Family
+    params_b: float                  # total parameters (billions)
+    loader_kind: LoaderKind
+    reasoning_mode: ReasoningMode
+    active_b: Optional[float] = None  # active params for MoE (decode speed)
+    thinking_repo_id: Optional[str] = None  # separate -Thinking ckpt (Qwen3-VL)
+    quant_repo_ids: dict[Precision, str] = field(default_factory=dict)  # fp8/int4 ckpts
+    is_moe: bool = False
+    is_baseline: bool = False        # the user's fine-tuned reference
+    notes: str = ""
+
+    # ── VRAM math ────────────────────────────────────────────────────────────
+
+    def est_vram_gb(self, precision: Precision = "bf16") -> float:
+        return self.params_b * _BYTES_PER_B[precision]
+
+    def fits_on(self, precision: Precision = "bf16", gpu_gb: float = GPU_VRAM_GB,
+                headroom: float = VRAM_HEADROOM_GB) -> bool:
+        return self.est_vram_gb(precision) + headroom <= gpu_gb
+
+    def available_precisions(self) -> list[Precision]:
+        """bf16 is always available (base repo); fp8/int4 only if a quant ckpt exists."""
+        return ["bf16"] + [p for p in ("fp8", "int4") if p in self.quant_repo_ids]
+
+    def best_fitting_precision(self, gpu_gb: float = GPU_VRAM_GB) -> Optional[Precision]:
+        """Smallest-footprint precision that fits, preferring higher fidelity first
+        (bf16 > fp8 > int4). Returns None if nothing fits without CPU offload."""
+        for prec in ("bf16", "fp8", "int4"):
+            if prec in self.available_precisions() and self.fits_on(prec, gpu_gb):
+                return prec
+        return None
+
+    def needs_offload(self, gpu_gb: float = GPU_VRAM_GB) -> bool:
+        return self.best_fitting_precision(gpu_gb) is None
+
+    def repo_for(self, precision: Precision, reasoning: Reasoning = "instruct") -> str:
+        """Resolve the concrete HF repo id for a (precision, reasoning) request."""
+        if reasoning == "thinking" and self.reasoning_mode == "separate_ckpt":
+            if self.thinking_repo_id is None:
+                raise ValueError(f"{self.key}: no separate thinking checkpoint")
+            base = self.thinking_repo_id
+        else:
+            base = self.repo_id
+        if precision != "bf16" and precision in self.quant_repo_ids:
+            # Quant repos are published for the instruct line; thinking quants are
+            # less common, so fall back to the instruct quant id if needed.
+            return self.quant_repo_ids[precision]
+        return base
+
+    def supports_thinking(self) -> bool:
+        return self.reasoning_mode == "toggle" or self.thinking_repo_id is not None
+
+    def enable_thinking_flag(self, reasoning: Reasoning) -> bool:
+        """For toggle models, thinking is a generation flag, not a separate repo."""
+        return reasoning == "thinking" and self.reasoning_mode == "toggle"
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# THE MODEL UNIVERSE. Bench everything that fits on 96GB, both reasoning variants.
+# Stretch rungs (397B / 235B) are kept but flagged needs_offload by the VRAM math.
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _q(*pairs: tuple[Precision, str]) -> dict[Precision, str]:
+    return dict(pairs)
+
+
+MODEL_REGISTRY: dict[str, ModelSpec] = {
+    # ── Qwen3.5 dense (native multimodal; enable_thinking toggle) ──────────────
+    "qwen3.5-0.8b": ModelSpec(
+        key="qwen3.5-0.8b", repo_id="Qwen/Qwen3.5-0.8B", family="qwen3_5",
+        params_b=0.873, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        notes="smallest native VLM; floor of the ladder",
+    ),
+    "qwen3.5-2b": ModelSpec(
+        key="qwen3.5-2b", repo_id="Qwen/Qwen3.5-2B", family="qwen3_5",
+        params_b=2.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+    ),
+    "qwen3.5-4b": ModelSpec(
+        key="qwen3.5-4b", repo_id="Qwen/Qwen3.5-4B", family="qwen3_5",
+        params_b=4.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+    ),
+    "qwen3.5-9b": ModelSpec(
+        key="qwen3.5-9b", repo_id="Qwen/Qwen3.5-9B", family="qwen3_5",
+        params_b=9.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+    ),
+    "qwen3.5-27b": ModelSpec(
+        key="qwen3.5-27b", repo_id="Qwen/Qwen3.5-27B", family="qwen3_5",
+        params_b=27.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3.5-27B-FP8"), ("int4", "Qwen/Qwen3.5-27B-GPTQ-Int4")),
+    ),
+    # ── Qwen3.5 MoE ────────────────────────────────────────────────────────────
+    "qwen3.5-35b-a3b": ModelSpec(
+        key="qwen3.5-35b-a3b", repo_id="Qwen/Qwen3.5-35B-A3B", family="qwen3_5_moe",
+        params_b=35.0, active_b=3.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        is_moe=True,
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3.5-35B-A3B-FP8"), ("int4", "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4")),
+        notes="MoE: ~3B active → decodes near a 3B dense",
+    ),
+    "qwen3.5-122b-a10b": ModelSpec(
+        key="qwen3.5-122b-a10b", repo_id="Qwen/Qwen3.5-122B-A10B", family="qwen3_5_moe",
+        params_b=122.0, active_b=10.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        is_moe=True,
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3.5-122B-A10B-FP8"), ("int4", "Qwen/Qwen3.5-122B-A10B-GPTQ-Int4")),
+        notes="fits 96GB only at GPTQ-Int4 (~67GB)",
+    ),
+    "qwen3.5-397b-a17b": ModelSpec(
+        key="qwen3.5-397b-a17b", repo_id="Qwen/Qwen3.5-397B-A17B", family="qwen3_5_moe",
+        params_b=397.0, active_b=17.0, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        is_moe=True,
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3.5-397B-A17B-FP8"), ("int4", "Qwen/Qwen3.5-397B-A17B-GPTQ-Int4")),
+        notes="STRETCH: needs CPU offload even at Int4",
+    ),
+    # ── Qwen3-VL dense (separate -Instruct / -Thinking) ────────────────────────
+    "qwen3vl-2b": ModelSpec(
+        key="qwen3vl-2b", repo_id="Qwen/Qwen3-VL-2B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-2B-Thinking", family="qwen3_vl",
+        params_b=2.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-2B-Instruct-FP8")),
+    ),
+    "qwen3vl-4b": ModelSpec(
+        key="qwen3vl-4b", repo_id="Qwen/Qwen3-VL-4B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-4B-Thinking", family="qwen3_vl",
+        params_b=4.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-4B-Instruct-FP8")),
+    ),
+    "qwen3vl-8b": ModelSpec(
+        key="qwen3vl-8b", repo_id="Qwen/Qwen3-VL-8B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-8B-Thinking", family="qwen3_vl",
+        params_b=8.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-8B-Instruct-FP8")),
+    ),
+    "qwen3vl-32b": ModelSpec(
+        key="qwen3vl-32b", repo_id="Qwen/Qwen3-VL-32B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-32B-Thinking", family="qwen3_vl",
+        params_b=32.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-32B-Instruct-FP8")),
+        notes="top dense that fits bf16 (~64GB)",
+    ),
+    # ── Qwen3-VL MoE ───────────────────────────────────────────────────────────
+    "qwen3vl-30b-a3b": ModelSpec(
+        key="qwen3vl-30b-a3b", repo_id="Qwen/Qwen3-VL-30B-A3B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-30B-A3B-Thinking", family="qwen3_vl_moe",
+        params_b=30.0, active_b=3.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        is_moe=True, quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-30B-A3B-Instruct-FP8")),
+        notes="MoE: ~3B active, fits bf16 (~60GB)",
+    ),
+    "qwen3vl-235b-a22b": ModelSpec(
+        key="qwen3vl-235b-a22b", repo_id="Qwen/Qwen3-VL-235B-A22B-Instruct",
+        thinking_repo_id="Qwen/Qwen3-VL-235B-A22B-Thinking", family="qwen3_vl_moe",
+        params_b=235.0, active_b=22.0, loader_kind="image_text_to_text", reasoning_mode="separate_ckpt",
+        is_moe=True, quant_repo_ids=_q(("fp8", "Qwen/Qwen3-VL-235B-A22B-Instruct-FP8")),
+        notes="STRETCH: needs CPU offload",
+    ),
+    # ── User's fine-tuned reference (combined ViT-classification + JSON) ────────
+    "qwen3.5-0.8b-json-captioner": ModelSpec(
+        key="qwen3.5-0.8b-json-captioner",
+        repo_id="AbstractPhil/Qwen3.5-0.8B-json-captioner", family="qwen3_5",
+        params_b=0.873, loader_kind="multimodal_lm", reasoning_mode="toggle",
+        is_baseline=True,
+        notes="LoRA-merged baseline: existence proof of native ViT-class + JSON; throughput ceiling",
+    ),
+    # ── JoyCaption (LLaVA: SigLIP2/SigLIP + Llama 3.1 8B). Captioner JSON-capacity ─
+    #    baseline — not grounding-trained, so treat coordinate tasks as exploratory.
+    "joycaption-beta-one": ModelSpec(
+        key="joycaption-beta-one",
+        repo_id="fancyfeast/llama-joycaption-beta-one-hf-llava", family="joycaption",
+        params_b=8.48, loader_kind="llava_conditional", reasoning_mode="none",
+        notes="SigLIP2 + Llama 3.1 8B captioner; latest JoyCaption; not grounding-trained",
+    ),
+    "joycaption-alpha-two": ModelSpec(
+        key="joycaption-alpha-two",
+        repo_id="fancyfeast/llama-joycaption-alpha-two-hf-llava", family="joycaption",
+        params_b=8.48, loader_kind="llava_conditional", reasoning_mode="none",
+        notes="prior JoyCaption (SigLIP v1); version-over-version JSON comparison",
+    ),
+}
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Query helpers
+# ──────────────────────────────────────────────────────────────────────────────
+
+def get_model(key: str) -> ModelSpec:
+    if key not in MODEL_REGISTRY:
+        raise KeyError(f"unknown model: {key!r}. known: {list(MODEL_REGISTRY)}")
+    return MODEL_REGISTRY[key]
+
+
+def model_keys() -> list[str]:
+    return list(MODEL_REGISTRY.keys())
+
+
+def models_that_fit(gpu_gb: float = GPU_VRAM_GB, include_offload: bool = False) -> list[ModelSpec]:
+    """Models that fit at some precision on `gpu_gb` (plus offload stretch if asked)."""
+    out = []
+    for spec in MODEL_REGISTRY.values():
+        if spec.best_fitting_precision(gpu_gb) is not None:
+            out.append(spec)
+        elif include_offload:
+            out.append(spec)
+    return out
+
+
+def reasoning_variants(spec: ModelSpec, both: bool = True) -> list[Reasoning]:
+    """The reasoning variants to run for a model."""
+    if not both or not spec.supports_thinking():
+        return ["instruct"]
+    return ["instruct", "thinking"]
+
+
+def get_runner(model_key: str, precision: Precision = "bf16", reasoning: Reasoning = "instruct",
+               device_map: Optional[str] = None, **kwargs):
+    """Factory: build a real VLMRunner for a (model, precision, reasoning) request.
+
+    Imports torch lazily (via runners), so importing this registry stays cheap.
+    CPU-offload stretch rungs get device_map='auto' automatically.
+    """
+    from .runners import VLMRunner
+    spec = get_model(model_key)
+    if precision == "bf16" and not spec.fits_on("bf16") and spec.best_fitting_precision() is not None:
+        precision = spec.best_fitting_precision()  # auto-downshift to a fitting quant
+    repo = spec.repo_for(precision, reasoning)
+    if device_map is None:
+        device_map = "auto" if spec.needs_offload() else "cuda"
+    return VLMRunner(
+        model_id=repo,
+        loader_kind=spec.loader_kind,
+        precision=precision,
+        enable_thinking=spec.enable_thinking_flag(reasoning),
+        device_map=device_map,
+        **kwargs,
+    )

qwen_test_runner/vision/report.py

@@ -0,0 +1,179 @@
+"""
+report.py — Leaderboards + the no-finetune labeler verdict.
+
+Aggregates per-(model, reasoning, category, mode) metrics into:
+  * a QUALITY leaderboard ranked by labeler_score (native json_mode columns),
+  * a FLEET leaderboard folding throughput, and
+  * a verdict naming the best NO-FINETUNE model — or, if none is natively
+    sufficient, the best finetune-candidate to feed the SFT/LoRA pipeline.
+
+Native columns use json_mode (no grammar crutch) because that is the signal for
+whether a model emits robust JSON on its own.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Optional
+
+from .metrics import labeler_score
+from .throughput import fleet_score
+
+# Bucketing thresholds (config-surfaceable later).
+ACC_FLOOR = 0.55           # below this the vision itself is too weak
+NATIVE_ROBUST = 0.90       # native json_mode robustness to count as "ships as-is"
+
+
+def _bucket(accuracy: Optional[float], native_robust: float) -> str:
+    if accuracy is None:
+        return "no_task_gt"
+    if accuracy < ACC_FLOOR:
+        return "insufficient"
+    if native_robust >= NATIVE_ROBUST:
+        return "native_capable"
+    return "finetune_candidate"
+
+
+def _mean(xs):
+    xs = [x for x in xs if x is not None]
+    return sum(xs) / len(xs) if xs else None
+
+
+def summarize(metric_rows: list[dict]) -> list[dict]:
+    """Collapse per-category metric rows into one summary per (model, reasoning).
+
+    Native (json_mode) columns drive the no-finetune decision; the constrained
+    rows are kept only to compute the native-vs-constrained validity gap.
+    """
+    by_model: dict[tuple[str, str], dict] = {}
+    for r in metric_rows:
+        key = (r["model"], r["reasoning"])
+        by_model.setdefault(key, {"json_mode": [], "constrained": []})
+        bucket = r["mode"] if r["mode"] in ("json_mode", "constrained") else "json_mode"
+        by_model[key][bucket].append(r)
+
+    summaries = []
+    for (model, reasoning), modes in by_model.items():
+        jm = modes["json_mode"] or modes["constrained"]
+        acc = _mean([r["primary_score_mean"] for r in jm if r["has_task_score"]])
+        valid = _mean([r["schema_valid_rate"] for r in jm]) or 0.0
+        robust = _mean([r["json_robustness"] for r in jm]) or 0.0
+        constrained_valid = _mean([r["schema_valid_rate"] for r in modes["constrained"]])
+        gap = (constrained_valid - valid) if constrained_valid is not None else None
+        tok_s = _mean([r["tokens_per_sec"] for r in jm]) or 0.0
+        mean_out = _mean([r["mean_output_tokens"] for r in jm]) or 0.0
+        lab = labeler_score(acc, valid, robust)
+        # samples/hour from tok/s + mean output tokens (prefill folded in elsewhere)
+        sph = (3600.0 * tok_s / mean_out) if (tok_s > 0 and mean_out > 0) else 0.0
+        summaries.append({
+            "model": model, "reasoning": reasoning,
+            "accuracy": acc, "native_valid": valid, "native_robust": robust,
+            "constrained_valid": constrained_valid, "native_gap": gap,
+            "labeler_score": lab, "tokens_per_sec": tok_s, "samples_per_hour": sph,
+            "fleet_score": fleet_score(lab, sph) if lab is not None else None,
+            "bucket": _bucket(acc, robust),
+        })
+    summaries.sort(key=lambda s: (s["labeler_score"] is not None, s["labeler_score"] or -1), reverse=True)
+    return summaries
+
+
+def _fmt(x, pct=False):
+    if x is None:
+        return "n/a"
+    return f"{x:.1%}" if pct else f"{x:.3f}"
+
+
+def quality_table(summaries: list[dict]) -> str:
+    lines = [
+        "| rank | model | reason | acc | native_valid | native_robust | gap | labeler | tok/s | bucket |",
+        "|------|-------|--------|-----|--------------|---------------|-----|---------|-------|--------|",
+    ]
+    for i, s in enumerate(summaries, 1):
+        lines.append(
+            f"| {i} | {s['model']} | {s['reasoning']} | {_fmt(s['accuracy'])} | "
+            f"{_fmt(s['native_valid'], True)} | {_fmt(s['native_robust'], True)} | "
+            f"{_fmt(s['native_gap'], True)} | {_fmt(s['labeler_score'])} | "
+            f"{s['tokens_per_sec']:.0f} | {s['bucket']} |"
+        )
+    return "\n".join(lines)
+
+
+def fleet_table(summaries: list[dict]) -> str:
+    fs = sorted([s for s in summaries if s["fleet_score"] is not None],
+                key=lambda s: s["fleet_score"], reverse=True)
+    lines = [
+        "| rank | model | reason | labeler | samples/hr | fleet |",
+        "|------|-------|--------|---------|------------|-------|",
+    ]
+    for i, s in enumerate(fs, 1):
+        lines.append(
+            f"| {i} | {s['model']} | {s['reasoning']} | {_fmt(s['labeler_score'])} | "
+            f"{s['samples_per_hour']:.0f} | {_fmt(s['fleet_score'])} |"
+        )
+    return "\n".join(lines)
+
+
+def verdict(summaries: list[dict]) -> str:
+    native = [s for s in summaries if s["bucket"] == "native_capable"]
+    ft = [s for s in summaries if s["bucket"] == "finetune_candidate"]
+    out = ["## Headline recommendation", ""]
+    if native:
+        b = native[0]
+        out.append(
+            f"> **Best no-finetune labeler:** `{b['model']}` ({b['reasoning']}) — "
+            f"{_fmt(b['native_valid'], True)} native-valid, {_fmt(b['accuracy'])} accuracy, "
+            f"{b['tokens_per_sec']:.0f} tok/s. Ships as-is."
+        )
+    elif ft:
+        b = ft[0]
+        out.append(
+            f"> **No natively-sufficient model.** Best **finetune-candidate:** `{b['model']}` "
+            f"({b['reasoning']}) — robust vision ({_fmt(b['accuracy'])} acc) but native JSON gap = "
+            f"{_fmt(b['native_gap'], True)}. Close it with the existing data-gen → SFT/LoRA pipeline."
+        )
+    else:
+        out.append("> No model cleared the accuracy floor on this run. Re-check inputs / categories.")
+    fleet = sorted([s for s in summaries if s["fleet_score"] is not None],
+                   key=lambda s: s["fleet_score"], reverse=True)
+    if fleet:
+        f = fleet[0]
+        out.append("")
+        out.append(
+            f"> **Best fleet labeler (1M+ images):** `{f['model']}` ({f['reasoning']}) — "
+            f"{f['samples_per_hour']:.0f} samples/hr at labeler {_fmt(f['labeler_score'])}."
+        )
+    return "\n".join(out)
+
+
+def write_reports(run_dir: Path, metric_rows: list[dict], config: dict) -> dict:
+    summaries = summarize(metric_rows)
+    md = [
+        f"# Qwen VLM Labeler Selection — {run_dir.name}",
+        "",
+        f"models={config.get('models')}  categories={len(config.get('categories', []))}  "
+        f"reasoning={config.get('reasonings')}  modes={config.get('modes')}  "
+        f"n={config.get('n')}  dataset={config.get('dataset')}  runner={config.get('runner')}",
+        "",
+        "## Quality leaderboard (labeler_score, native json_mode)",
+        "",
+        quality_table(summaries),
+        "",
+        "## Fleet leaderboard (accuracy × throughput)",
+        "",
+        fleet_table(summaries),
+        "",
+        verdict(summaries),
+        "",
+    ]
+    (run_dir / "leaderboard.md").write_text("\n".join(md), encoding="utf-8")
+    (run_dir / "summary.json").write_text(
+        json.dumps({"config": config, "summaries": summaries}, indent=2), encoding="utf-8")
+    # CSV
+    cols = ["model", "reasoning", "accuracy", "native_valid", "native_robust", "native_gap",
+            "labeler_score", "tokens_per_sec", "samples_per_hour", "fleet_score", "bucket"]
+    csv_lines = [",".join(cols)]
+    for s in summaries:
+        csv_lines.append(",".join("" if s.get(c) is None else str(s.get(c)) for c in cols))
+    (run_dir / "leaderboard.csv").write_text("\n".join(csv_lines), encoding="utf-8")
+    return {"summaries": summaries}

qwen_test_runner/vision/run_vlmbench.py

@@ -0,0 +1,68 @@
+"""
+run_vlmbench.py — CLI entry point (`qwen-vlmbench`).
+
+Examples:
+  # Offline CPU smoke (no torch, no network) — Phase-0 acceptance:
+  qwen-vlmbench --runner stub --dataset smoke \
+      --categories image_classification bbox_grounding ocr_text
+
+  # Real run on Colab (single RTX 6000 Pro):
+  qwen-vlmbench --runner vlm --dataset full --n 200 \
+      --models qwen3.5-9b qwen3vl-8b --reasoning instruct thinking \
+      --modes json_mode constrained --categories image_classification bbox_grounding ocr_text
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+
+from .bench import BenchConfig, run_bench
+from .tasks_vision import category_names, pilot_categories
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(prog="qwen-vlmbench", description="Qwen VLM image→JSON labeler benchmark")
+    p.add_argument("--models", nargs="+", default=["qwen3.5-0.8b-json-captioner"],
+                   help="model keys from the model registry (or any label for --runner stub)")
+    p.add_argument("--categories", nargs="+", default=pilot_categories(),
+                   help=f"vision categories. all: {category_names()}")
+    p.add_argument("--reasoning", nargs="+", default=["instruct"], choices=["instruct", "thinking"])
+    p.add_argument("--modes", nargs="+", default=["json_mode"],
+                   choices=["json_mode", "constrained", "tool_use", "free"])
+    p.add_argument("--n", type=int, default=50, help="samples per category")
+    p.add_argument("--dataset", default="smoke", choices=["smoke", "full"])
+    p.add_argument("--runner", default="stub", choices=["stub", "vlm"])
+    p.add_argument("--precision", default="bf16", choices=["bf16", "fp8", "int4"])
+    p.add_argument("--stub-behavior", default="perfect", choices=["perfect", "fragile", "random"])
+    p.add_argument("--output-root", default="runs/vision")
+    p.add_argument("--gpu-hourly-rate", type=float, default=2.0)
+    p.add_argument("--clear-cache-after-model", action="store_true",
+                   help="delete each model's HF cache after use (full-array sweeps on a tight SSD)")
+    return p
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = _build_parser().parse_args(argv)
+    config = BenchConfig(
+        models=args.models,
+        categories=args.categories,
+        reasonings=args.reasoning,
+        modes=args.modes,
+        n=args.n,
+        dataset=args.dataset,
+        runner=args.runner,
+        precision=args.precision,
+        stub_behavior=args.stub_behavior,
+        output_root=args.output_root,
+        gpu_hourly_rate=args.gpu_hourly_rate,
+        clear_cache_after_model=args.clear_cache_after_model,
+    )
+    summary = run_bench(config)
+    print(json.dumps(summary, indent=2))
+    print(f"\nLeaderboard: {summary['run_dir']}/leaderboard.md")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

qwen_test_runner/vision/runner_types.py

@@ -0,0 +1,23 @@
+"""
+runner_types.py — Shared, torch-free runner result type.
+
+Lives in its own module so the orchestrator and the stub runner can import it
+without dragging in torch/transformers (Phase-0 CPU runs must not import torch).
+Mirrors model_runner.GenResult + image/timing metadata.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class VLMResult:
+    mode: str               # "json_mode" | "constrained" | "tool_use" | "free"
+    raw_text: str           # exactly what the model decoded (after prompt strip)
+    backend: str            # "transformers" | "xgrammar" | "stub"
+    n_input_tokens: int
+    n_output_tokens: int
+    gen_seconds: float
+    image_id: str = ""
+    grammar_conformant: bool = False  # constrained decoding actually applied

qwen_test_runner/vision/runners.py

@@ -0,0 +1,301 @@
+"""
+runners.py — VLMRunner: load a Qwen VLM once, run image→JSON generation.
+
+Promotes the working image→JSON patterns from colab/qwen_vit_json_test.py
+(AutoProcessor + AutoModelFor{ImageTextToText,MultimodalLM}, image content
+blocks, left-padding, batched generate with OOM-halving) into package code, and
+reuses model_runner._XGrammarLogitsProcessor verbatim for constrained decoding.
+
+This module imports torch/transformers at module load, so it is imported LAZILY
+from qwen_test_runner.vision (the Phase-0 stub path never touches it).
+
+xgrammar + images: the processor only inspects input_ids/scores, never image
+tensors, so the grammar matcher works with image prompts PROVIDED the prompt_len
+handed to it is the image-INCLUSIVE tokenized length and TokenizerInfo is built
+from processor.tokenizer. Constrained mode runs at batch=1 (matcher is
+per-sequence).
+"""
+
+from __future__ import annotations
+
+import time
+import warnings
+from typing import Optional
+
+import torch
+from transformers import AutoProcessor, AutoModelForImageTextToText
+
+from ..model_runner import _XGrammarLogitsProcessor, _HAS_XGRAMMAR
+from .runner_types import VLMResult
+from .tasks_vision import VisionTaskSpec, gbnf_for, resolved_system_prompt, tool_schema_for
+
+try:  # newer transformers exposes a unified multimodal class (Qwen3.5)
+    from transformers import AutoModelForMultimodalLM
+    _HAS_MULTIMODAL = True
+except ImportError:  # pragma: no cover
+    AutoModelForMultimodalLM = None
+    _HAS_MULTIMODAL = False
+
+if _HAS_XGRAMMAR:  # pragma: no cover - GPU/optional path
+    import xgrammar as xgr
+
+
+_DTYPE = {"bf16": torch.bfloat16}
+
+
+class VLMRunner:
+    """Loads one Qwen VLM checkpoint and runs the four generation modes."""
+
+    def __init__(
+        self,
+        model_id: str,
+        loader_kind: str = "image_text_to_text",
+        precision: str = "bf16",
+        device: Optional[str] = None,
+        device_map: str = "cuda",
+        enable_thinking: bool = False,
+        trust_remote_code: bool = True,
+    ):
+        self.model_id = model_id
+        self.precision = precision
+        self.loader_kind = loader_kind
+        self.enable_thinking = enable_thinking
+        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+
+        print(f"[VLMRunner] loading {model_id} ({loader_kind}, {precision}) on {device_map}")
+        self.processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=trust_remote_code)
+        # left-pad for correct batched decoding; ensure a pad token exists — Llama-based
+        # checkpoints (e.g. JoyCaption) ship without one, which breaks processor padding.
+        if getattr(self.processor, "tokenizer", None) is not None:
+            self.processor.tokenizer.padding_side = "left"
+            if self.processor.tokenizer.pad_token_id is None:
+                self.processor.tokenizer.pad_token = self.processor.tokenizer.eos_token
+
+        load_cls = AutoModelForImageTextToText
+        if loader_kind == "multimodal_lm" and _HAS_MULTIMODAL:
+            load_cls = AutoModelForMultimodalLM
+        elif loader_kind == "llava_conditional":  # JoyCaption (SigLIP + Llama 3.1)
+            from transformers import LlavaForConditionalGeneration
+            load_cls = LlavaForConditionalGeneration
+
+        load_kwargs = dict(device_map=device_map, trust_remote_code=trust_remote_code)
+        if precision in _DTYPE:
+            load_kwargs["dtype"] = _DTYPE[precision]  # quant repos carry their own config
+        self.model = load_cls.from_pretrained(model_id, **load_kwargs)
+        self.model.eval()
+
+        tok = getattr(self.processor, "tokenizer", self.processor)
+        self._pad_id = getattr(tok, "pad_token_id", None) or getattr(tok, "eos_token_id", None)
+
+        # xgrammar compiler — reusable; per-category grammars compiled on demand.
+        # Detect xgrammar LAZILY here (not just at module import) so installing it
+        # AFTER the package was first imported still enables constrained mode — and
+        # flip model_runner's captured flag so _XGrammarLogitsProcessor works too.
+        self._xgr = None
+        self._xgr_compiler = None
+        self._xgr_tokenizer_info = None
+        self._compiled: dict[str, object] = {}
+        try:
+            import xgrammar as _xgr_mod
+            self._xgr = _xgr_mod
+        except ImportError:
+            pass
+        if self._xgr is not None:
+            import qwen_test_runner.model_runner as _mr
+            if not _mr._HAS_XGRAMMAR:           # _XGrammarLogitsProcessor reads these
+                _mr.xgr = self._xgr
+                _mr._HAS_XGRAMMAR = True
+            try:
+                self._xgr_tokenizer_info = self._xgr.TokenizerInfo.from_huggingface(tok)
+                self._xgr_compiler = self._xgr.GrammarCompiler(self._xgr_tokenizer_info)
+            except Exception as e:  # pragma: no cover
+                warnings.warn(f"xgrammar init failed: {e}; constrained mode unavailable")
+        print(f"[VLMRunner] ready. xgrammar={self._xgr_compiler is not None}")
+
+    def close(self) -> None:
+        """Free the model from VRAM (used between models in a sweep)."""
+        try:
+            del self.model
+        except AttributeError:
+            pass
+        import gc
+        gc.collect()
+        if torch.cuda.is_available():
+            torch.cuda.empty_cache()
+
+    # ── message construction ─────────────────────────────────────────────────
+
+    def _messages(self, spec: VisionTaskSpec, image, user_prompt=None) -> list:
+        system = resolved_system_prompt(spec)
+        user_content = []
+        if image is not None:
+            user_content.append({"type": "image", "image": image})
+        # per-sample prompt override (e.g. the question for VQA), else the task default
+        user_content.append({"type": "text", "text": user_prompt or spec.user_prompt})
+        return [
+            {"role": "system", "content": system},
+            {"role": "user", "content": user_content},
+        ]
+
+    def _encode(self, messages_list: list, tools=None):
+        kw = dict(add_generation_prompt=True, tokenize=True, return_dict=True,
+                  return_tensors="pt", padding=True)
+        if tools is not None:                       # don't pass tools=None (some templates warn)
+            kw["tools"] = tools
+        # `enable_thinking` is a Qwen3.5 (multimodal_lm) toggle; Qwen3-VL's processor rejects it.
+        if self.loader_kind == "multimodal_lm":
+            kw["enable_thinking"] = self.enable_thinking
+        return self.processor.apply_chat_template(messages_list, **kw).to(self.model.device)
+
+    def _encode_llava(self, messages_list: list):
+        """LLaVA/JoyCaption encode. Its chat template expects STRING `content` (it does
+        string ops like `.replace` on it) and prepends the `<image>` token itself, so we
+        cannot pass Qwen's structured content-parts list — we rebuild a string-content
+        conversation. Render to TEXT (no `enable_thinking` no-op kwarg; no `tools`, since
+        the Llama template would render a tools block), then let the processor attach the
+        PIL image and expand `<image>` into feature tokens in `input_ids` — keeping the
+        prompt length image-inclusive for xgrammar."""
+        prompts, images = [], []
+        for messages in messages_list:
+            system_txt, user_txt, img = None, [], None
+            for msg in messages:
+                content = msg.get("content")
+                if msg.get("role") == "system":
+                    system_txt = content if isinstance(content, str) else None
+                    continue
+                if isinstance(content, list):
+                    for part in content:
+                        if not isinstance(part, dict):
+                            continue
+                        if part.get("type") == "image":
+                            img = part.get("image")
+                        elif part.get("type") == "text":
+                            user_txt.append(part.get("text", ""))
+                elif isinstance(content, str):
+                    user_txt.append(content)
+            convo = []
+            if system_txt:
+                convo.append({"role": "system", "content": system_txt})
+            convo.append({"role": "user", "content": " ".join(t for t in user_txt if t)})
+            prompts.append(self.processor.apply_chat_template(
+                convo, add_generation_prompt=True, tokenize=False))
+            images.append(img)
+        inputs = self.processor(
+            images=images, text=prompts, return_tensors="pt", padding=True,
+        ).to(self.model.device)
+        # The SigLIP vision tower is bf16 but the processor emits float32 pixel_values —
+        # cast to the model dtype or the vision tower raises a dtype mismatch (the JoyCaption
+        # model card does exactly this).
+        if "pixel_values" in inputs:
+            mdtype = getattr(self.model, "dtype", None)
+            if mdtype is not None and inputs["pixel_values"].dtype != mdtype:
+                inputs["pixel_values"] = inputs["pixel_values"].to(mdtype)
+        return inputs
+
+    # ── modes ─────────────────────────────────────────────────────────────────
+
+    @torch.no_grad()
+    def generate(self, spec: VisionTaskSpec, image, mode: str, *,
+                 image_id: str = "", image_size=None, gt=None, user_prompt=None) -> VLMResult:
+        if mode == "constrained":
+            return self._generate_constrained(spec, image, image_id, user_prompt)
+        tools = [self._tool_def(spec)] if mode == "tool_use" else None
+        return self._generate_free(spec, image, mode, image_id, tools, user_prompt)
+
+    @torch.no_grad()
+    def _generate_free(self, spec, image, mode, image_id, tools, user_prompt=None) -> VLMResult:
+        msgs = [self._messages(spec, image, user_prompt)]
+        inputs = (self._encode_llava(msgs) if self.loader_kind == "llava_conditional"
+                  else self._encode(msgs, tools=tools))
+        n_in = inputs["input_ids"].shape[1]
+        t0 = time.perf_counter()
+        out = self.model.generate(**inputs, max_new_tokens=spec.max_new_tokens,
+                                  do_sample=False, pad_token_id=self._pad_id)
+        dt = time.perf_counter() - t0
+        cont = out[0, n_in:]
+        text = self.processor.decode(cont, skip_special_tokens=True)
+        return VLMResult(mode, text, "transformers", int(n_in), int(cont.shape[0]), dt, image_id)
+
+    @torch.no_grad()
+    def _generate_constrained(self, spec, image, image_id, user_prompt=None) -> VLMResult:
+        if self._xgr_compiler is None:
+            warnings.warn("xgrammar unavailable; constrained falling back to json_mode")
+            return self._generate_free(spec, image, "json_mode", image_id, None, user_prompt)
+        grammar = gbnf_for(spec)
+        compiled = self._compiled.get(spec.category)
+        if compiled is None:
+            compiled = self._xgr_compiler.compile_grammar(grammar)
+            self._compiled[spec.category] = compiled
+
+        msgs = [self._messages(spec, image, user_prompt)]
+        inputs = (self._encode_llava(msgs) if self.loader_kind == "llava_conditional"
+                  else self._encode(msgs))
+        n_in = inputs["input_ids"].shape[1]  # image-INCLUSIVE length — critical
+        lp = _XGrammarLogitsProcessor(
+            compiled_grammar=compiled,
+            vocab_size=self._xgr_tokenizer_info.vocab_size,
+            prompt_len=n_in,
+        )
+        t0 = time.perf_counter()
+        out = self.model.generate(**inputs, max_new_tokens=spec.max_new_tokens,
+                                  do_sample=False, pad_token_id=self._pad_id,
+                                  logits_processor=[lp])
+        dt = time.perf_counter() - t0
+        cont = out[0, n_in:]
+        text = self.processor.decode(cont, skip_special_tokens=True)
+        return VLMResult("constrained", text, "xgrammar", int(n_in), int(cont.shape[0]),
+                         dt, image_id, grammar_conformant=True)
+
+    def _tool_def(self, spec: VisionTaskSpec) -> dict:
+        return {
+            "type": "function",
+            "function": {
+                "name": "emit_" + spec.category,
+                "description": spec.probes,
+                "parameters": tool_schema_for(spec),
+            },
+        }
+
+    # ── batched json_mode with OOM-halving (throughput path) ───────────────────
+
+    @torch.no_grad()
+    def generate_batch(self, spec: VisionTaskSpec, images: list, mode: str = "json_mode",
+                       image_ids: Optional[list] = None) -> list[VLMResult]:
+        image_ids = image_ids or [""] * len(images)
+        return self._batch_with_fallback(spec, images, image_ids, mode)
+
+    def _batch_with_fallback(self, spec, images, ids, mode) -> list[VLMResult]:
+        if not images:
+            return []
+        try:
+            return self._batch(spec, images, ids, mode)
+        except torch.cuda.OutOfMemoryError:
+            torch.cuda.empty_cache()
+            if len(images) == 1:
+                return [VLMResult(mode, "", "transformers", 0, 0, 0.0, ids[0])]
+            half = max(1, len(images) // 2)
+            return (self._batch_with_fallback(spec, images[:half], ids[:half], mode)
+                    + self._batch_with_fallback(spec, images[half:], ids[half:], mode))
+        except Exception:
+            if len(images) == 1:
+                return [VLMResult(mode, "", "transformers", 0, 0, 0.0, ids[0])]
+            return [self._batch_with_fallback(spec, [im], [i], mode)[0]
+                    for im, i in zip(images, ids)]
+
+    @torch.no_grad()
+    def _batch(self, spec, images, ids, mode) -> list[VLMResult]:
+        tools = [self._tool_def(spec)] if mode == "tool_use" else None
+        msgs = [self._messages(spec, im) for im in images]
+        inputs = (self._encode_llava(msgs) if self.loader_kind == "llava_conditional"
+                  else self._encode(msgs, tools=tools))
+        n_in = inputs["input_ids"].shape[1]
+        t0 = time.perf_counter()
+        out = self.model.generate(**inputs, max_new_tokens=spec.max_new_tokens,
+                                  do_sample=False, pad_token_id=self._pad_id)
+        dt = time.perf_counter() - t0
+        per = dt / len(images)
+        results = []
+        for row, iid in zip(out[:, n_in:], ids):
+            text = self.processor.decode(row, skip_special_tokens=True)
+            results.append(VLMResult(mode, text, "transformers", int(n_in), int(row.shape[0]), per, iid))
+        return results

qwen_test_runner/vision/specialists.py

@@ -0,0 +1,181 @@
+"""
+specialists.py — the CPU assembly layer of the deterministic pipeline.
+
+`Solids` holds one image's solidification primitives (detector boxes, seg masks, a
+relative depth map, optional saliency, OCR, tags, class/style) — all in PIXEL space.
+The `build_*` functions turn a `Solids` into each task's exact `tasks_vision` JSON, in
+the task's declared coord space (via the existing `coords.from_canonical`), reusing the
+`derive` engine for the INTEGRATE tasks.
+
+The GPU half — loading the models and populating `Solids` — lives in the Colab runner
+(`specialists_gpu`); this module is model-free and unit-tested on CPU.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Optional
+
+import numpy as np
+
+from . import derive
+from .coords import BBox, CoordSpace, XYXY, _scale_for_space, from_canonical
+from .tasks_vision import get_task
+
+
+# ── coordinate helpers (pixel → task space) ──────────────────────────────────
+def box_to_space(pix_xyxy, space: CoordSpace, size, ndigits: int = 2) -> list:
+    """Pixel-abs [x1,y1,x2,y2] → the task's coord space."""
+    return [round(v, ndigits) for v in from_canonical(BBox(*map(float, pix_xyxy)), space, size, XYXY)]
+
+
+def poly_to_space(pix_flat, space: CoordSpace, size, ndigits: int = 2) -> list:
+    """Flat pixel polygon [x1,y1,x2,y2,...] → the task's coord space, per vertex."""
+    sx, sy = _scale_for_space(space, size)          # raw = pixel / scale
+    out = []
+    for i in range(0, len(pix_flat) - 1, 2):
+        out.append(round(float(pix_flat[i]) / sx, ndigits))
+        out.append(round(float(pix_flat[i + 1]) / sy, ndigits))
+    return out
+
+
+def quad_to_xyxy(quad) -> list:
+    """A 4-point polygon (PaddleOCR) — [[x,y]*4] or flat [x,y,...] — → pixel xyxy."""
+    a = np.asarray(quad, dtype=float).reshape(-1, 2)
+    return [float(a[:, 0].min()), float(a[:, 1].min()), float(a[:, 0].max()), float(a[:, 1].max())]
+
+
+# ── per-image primitives ─────────────────────────────────────────────────────
+@dataclass
+class Solids:
+    size: tuple                       # (W, H) pixels
+    boxes: list = field(default_factory=list)   # [{label, box:[x1,y1,x2,y2] px, score, mask?}]
+    depth: Optional[np.ndarray] = None          # HxW relative (Depth-Anything: higher = nearer)
+    saliency: Optional[np.ndarray] = None       # HxW float in [0,1]
+    tags: list = field(default_factory=list)     # [{label, score}] (RAM++/SigLIP2)
+    class_top: list = field(default_factory=list)  # [{label, score}] top-k classification
+    ocr: Optional[dict] = None                   # {full_text, lines:[{text, box:[quad px], conf?}]}
+    style: Optional[str] = None                  # SigLIP2 style label
+    gray: Optional[np.ndarray] = None            # HxW grayscale (for symmetry)
+    depth_higher_is_nearer: bool = True
+    attr_boxes: list = field(default_factory=list)  # fusion tier: caption-phrase grounding
+    #   [{phrase, matched_span, box:[x1,y1,x2,y2] px, score}] from ground_phrases()
+
+
+# ── SOLID task builders (direct model outputs) ───────────────────────────────
+# Emitted lists/strings are truncated to the registry caps (read from the spec,
+# never hardcoded) — the generated pydantic models enforce max_items/max_length
+# as hard constraints, so an uncapped emit would flunk schema validation on
+# busy images even though the content is correct.
+def build_bbox(s: Solids) -> dict:
+    spec = get_task("bbox_grounding")
+    cap = spec.fields["detections"].max_items
+    boxes = sorted(s.boxes, key=lambda b: -float(b.get("score", 1.0)))[:cap]
+    dets = [{"label": str(b["label"]),
+             "box": box_to_space(b["box"], spec.coord_space, s.size),
+             "score": round(float(b.get("score", 1.0)), 4)} for b in boxes]
+    return {"detections": dets, "count": len(dets)}
+
+
+def build_segmentation(s: Solids) -> dict:
+    spec = get_task("segmentation")
+    cap = spec.fields["masks"].max_items
+    masked = [b for b in s.boxes if b.get("mask") is not None]
+    masked.sort(key=lambda b: -float(np.asarray(b["mask"]).sum()))   # keep largest
+    masks = []
+    for b in masked:
+        if len(masks) >= cap:
+            break
+        poly = derive.outline_polygon(b["mask"], b["label"])["outline"]
+        if len(poly) >= 6:
+            masks.append({"label": str(b["label"]),
+                          "polygon": poly_to_space(poly, spec.coord_space, s.size)})
+    return {"masks": masks}
+
+
+def build_classification(s: Solids) -> dict:
+    top = s.class_top or ([{"label": s.boxes[0]["label"], "score": 1.0}] if s.boxes else
+                          [{"label": "unknown", "score": 0.0}])
+    top = sorted(top, key=lambda t: -t["score"])[:5]
+    return {"label": str(top[0]["label"]), "confidence": round(float(top[0]["score"]), 4),
+            "top5": [{"label": str(t["label"]), "score": round(float(t["score"]), 4)} for t in top]}
+
+
+def build_ocr(s: Solids) -> dict:
+    spec = get_task("ocr_text")
+    if not s.ocr:
+        return {"full_text": "", "lines": []}
+    lines_spec = spec.fields["lines"]
+    text_cap = next((f.max_str_length for f in lines_spec.nested_fields
+                     if f.name == "text"), 512)
+    lines = []
+    for ln in s.ocr.get("lines", [])[:lines_spec.max_items]:
+        d = {"text": str(ln["text"])[:text_cap]}
+        if ln.get("box") is not None:
+            d["box"] = box_to_space(quad_to_xyxy(ln["box"]), spec.coord_space, s.size)
+        lines.append(d)
+    full_cap = spec.fields["full_text"].max_str_length
+    return {"full_text": str(s.ocr.get("full_text", ""))[:full_cap], "lines": lines}
+
+
+# ── INTEGRATE task builders (derive engine + coord conversion) ───────────────
+def build_spatial(s: Solids) -> dict:
+    return derive.spatial_relations(s.boxes, depth=s.depth,
+                                    higher_is_nearer=s.depth_higher_is_nearer)
+
+
+def build_depth_order(s: Solids) -> dict:
+    if s.depth is None:
+        return {"nearest": "", "farthest": "", "relative_depth": []}
+    ents = [{"label": b["label"], "mask": b.get("mask"), "box": b["box"]} for b in s.boxes]
+    return derive.depth_order(ents, s.depth, higher_is_nearer=s.depth_higher_is_nearer)
+
+
+def build_subject(s: Solids) -> dict:
+    space = get_task("subject_fixation").coord_space
+    out = derive.subject_fixation(s.boxes, s.size, saliency=s.saliency)
+    out["primary_subject"]["box"] = box_to_space(out["primary_subject"]["box"], space, s.size)
+    return out
+
+
+def build_outline(s: Solids) -> dict:
+    space = get_task("outline_association").coord_space
+    masked = [b for b in s.boxes if b.get("mask") is not None]
+    if not masked:
+        return {"outline": [], "label": ""}
+    big = max(masked, key=lambda b: np.asarray(b["mask"]).sum())
+    o = derive.outline_polygon(big["mask"], big["label"])
+    o["outline"] = poly_to_space(o["outline"], space, s.size)
+    return o
+
+
+def build_style(s: Solids) -> dict:
+    style = s.style if s.style in ("photo", "painting", "3d_render", "sketch", "anime", "other") else "other"
+    layout = derive.layout_kind(s.boxes, s.size)
+    symmetry = derive.symmetry_axis(s.gray) if s.gray is not None else "none"
+    return {"style": style, "layout": layout, "symmetry": symmetry}
+
+
+def build_datatype_diff(s: Solids) -> dict:
+    return derive.detect_data_type(s.ocr.get("full_text", "") if s.ocr else "")
+
+
+def build_datatype_util(s: Solids) -> tuple[dict, bool]:
+    """Returns ({data_type, content}, parsed_ok). parsed_ok=False → caller may VLM-fallback."""
+    return derive.parse_data_type(s.ocr.get("full_text", "") if s.ocr else "")
+
+
+# task → builder (SOLID + INTEGRATE only; semantic_association + vqa are VLM)
+DETERMINISTIC_BUILDERS = {
+    "bbox_grounding": build_bbox,
+    "segmentation": build_segmentation,
+    "image_classification": build_classification,
+    "ocr_text": build_ocr,
+    "structural_spatial_awareness": build_spatial,
+    "depth_analysis": build_depth_order,
+    "subject_fixation": build_subject,
+    "outline_association": build_outline,
+    "style_structural_awareness": build_style,
+    "data_type_differentiation": build_datatype_diff,
+    # data_type_utilization handled specially (returns the parsed_ok flag)
+}

qwen_test_runner/vision/specialists_gpu.py

@@ -0,0 +1,768 @@
+"""
+specialists_gpu.py — the GPU half: load the Apache/MIT specialist models and populate a
+`Solids` per image, then score the built task JSON through the EXISTING vlmbench scorers.
+
+This runs on Colab (needs torch + transformers + the model weights). It is deliberately
+incremental: the detection hub (GroundingDINO) + the detection-dependent tasks are wired
+and validated first (against real COCO GT) so the whole chain — detect → Solids → build →
+score_vision_sample — is proven before the other loaders (SAM2, Depth, OCR, SigLIP2, RAM++)
+are added. Each loader is a small function returning primitives in PIXEL space.
+
+Model picks are pinned to the plan's Apache/MIT license ledger.
+"""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import Optional
+
+import numpy as np
+
+from .specialists import Solids, build_bbox, build_spatial, build_subject
+from .tasks_vision import get_task
+
+# ── detection hub: GroundingDINO (Apache) ────────────────────────────────────
+GROUNDING_DINO_ID = "IDEA-Research/grounding-dino-base"   # Apache-2.0, local weights (NOT the API-only 1.5)
+
+# COCO-80 (a known closed vocab for the detection *validation*; production uses the RAM++ tagger)
+COCO_CLASSES = [
+    "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat",
+    "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat",
+    "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack",
+    "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball",
+    "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket",
+    "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple",
+    "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair",
+    "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse",
+    "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator",
+    "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush",
+]
+
+
+def load_grounding_dino(device: str = "cuda"):
+    """Returns (processor, model). Apache checkpoint, loads via transformers.
+    No dtype kwarg — float32 is the default and `torch_dtype=` now deprecation-warns."""
+    from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection
+    proc = AutoProcessor.from_pretrained(GROUNDING_DINO_ID)
+    model = AutoModelForZeroShotObjectDetection.from_pretrained(
+        GROUNDING_DINO_ID).to(device).eval()
+    return proc, model
+
+
+def _gdino_prompt(classes) -> str:
+    # GroundingDINO wants lowercased, "." separated phrases ending in a period.
+    return ". ".join(c.strip().lower() for c in classes) + "."
+
+
+def detect(proc, model, image, classes, box_threshold: float = 0.30,
+           text_threshold: float = 0.25, device: str = "cuda") -> list:
+    """image → [{label, box:[x1,y1,x2,y2] PIXEL-ABS, score}]. Forces target_sizes so the
+    boxes come back pixel-abs xyxy (else transformers returns normalized cxcywh)."""
+    import torch
+    inputs = proc(images=image, text=_gdino_prompt(classes), return_tensors="pt").to(device)
+    with torch.no_grad():
+        outputs = model(**inputs)
+    W, H = image.size
+    _kw = dict(text_threshold=text_threshold, target_sizes=[(H, W)])  # (height, width) → pixel-abs xyxy
+    try:                                                              # transformers renamed the arg
+        res = proc.post_process_grounded_object_detection(
+            outputs, inputs["input_ids"], threshold=box_threshold, **_kw)[0]
+    except TypeError:
+        res = proc.post_process_grounded_object_detection(
+            outputs, inputs["input_ids"], box_threshold=box_threshold, **_kw)[0]
+    # dict.get(k, default) evaluates the default EAGERLY — touching the deprecated
+    # "labels" key fires transformers' FutureWarning even when text_labels exists
+    labels = res["text_labels"] if "text_labels" in res else res.get("labels")
+    out = []
+    for box, score, lab in zip(res["boxes"], res["scores"], labels):
+        b = [float(v) for v in box.tolist()]
+        out.append({"label": str(lab).strip() or "object", "box": b, "score": float(score)})
+    return out
+
+
+def ground_phrases(gdino, image, phrases, box_threshold: float = 0.25,
+                   text_threshold: float = 0.20, max_tokens_per_chunk: int = 250,
+                   device: str = "cuda") -> list:
+    """Fusion-tier grounding pass: caption-derived attribute phrases → boxes.
+    → [{phrase, matched_span, box:[x1,y1,x2,y2] px, score}].
+
+    Lower thresholds than the base detection pass — fine-grained phrases score
+    lower than category nouns, and the downstream containment+margin gate protects
+    precision (recall matters more here: an ungrounded attribute falls back to the
+    weaker caption-binding path).
+
+    GDINO's BERT text encoder truncates at 256 TOKENS silently (the model slices
+    input_ids with no warning), so phrases are chunked by the processor's OWN
+    tokenizer count (≤ max_tokens_per_chunk, headroom for [CLS]/[SEP]) with an
+    accounting assert — a silently dropped phrase is the failure mode. GDINO
+    returns the matched text SPAN (possibly a sub-span, "earrings" from "silver
+    drop earrings"): each hit is re-mapped to its source phrase by maximum token
+    overlap within the chunk."""
+    import torch
+    proc, model = gdino
+    phrases = [p.strip().lower() for p in phrases if p and p.strip()]
+    if not phrases:
+        return []
+
+    tok = getattr(proc, "tokenizer", None)
+
+    def _ntok(p):
+        if tok is not None:
+            return len(tok(p, add_special_tokens=False)["input_ids"]) + 1   # +1 for ". "
+        return len(p.split()) + 1                       # crude fallback, ~1.3 tok/word
+
+    chunks, cur, ntok = [], [], 0
+    budget = max_tokens_per_chunk if tok is not None else max_tokens_per_chunk // 2
+    for p in phrases:
+        t = _ntok(p)
+        if cur and ntok + t > budget:
+            chunks.append(cur)
+            cur, ntok = [], 0
+        cur.append(p)
+        ntok += t
+    if cur:
+        chunks.append(cur)
+    assert sum(len(c) for c in chunks) == len(phrases), "phrase chunking dropped input"
+
+    W, H = image.size
+    out = []
+    for chunk in chunks:
+        text = ". ".join(chunk) + "."
+        inputs = proc(images=image, text=text, return_tensors="pt").to(device)
+        with torch.no_grad():
+            outputs = model(**inputs)
+        _kw = dict(text_threshold=text_threshold, target_sizes=[(H, W)])
+        try:                                                          # transformers renamed the arg
+            res = proc.post_process_grounded_object_detection(
+                outputs, inputs["input_ids"], threshold=box_threshold, **_kw)[0]
+        except TypeError:
+            res = proc.post_process_grounded_object_detection(
+                outputs, inputs["input_ids"], box_threshold=box_threshold, **_kw)[0]
+        out.extend(_remap_spans(res, chunk))
+    out.sort(key=lambda r: (r["phrase"], -r["score"]))
+    return out
+
+
+def _remap_spans(res, chunk) -> list:
+    """GDINO post-process result → attr-box records, re-mapping each matched text
+    SPAN back to its source phrase by maximum token overlap within the chunk."""
+    labels = res["text_labels"] if "text_labels" in res else res.get("labels")
+    out = []
+    for box, score, span in zip(res["boxes"], res["scores"], labels):
+        span_toks = set(str(span).lower().split())
+        best, best_ov = None, 0
+        for p in chunk:
+            ov = len(span_toks & set(p.split()))
+            if ov > best_ov or (ov == best_ov and best and len(p) > len(best)):
+                if ov > 0:
+                    best, best_ov = p, ov
+        if best is None:
+            continue
+        out.append({"phrase": best, "matched_span": str(span).strip(),
+                    "box": [float(v) for v in box.tolist()],
+                    "score": float(score)})
+    return out
+
+
+# ── BATCHED specialist paths (throughput: the serial path leaves a 96GB card ──
+#    ~90% idle; every model here batches across images) ────────────────────────
+
+def detect_batch(gdino, images, classes, box_threshold: float = 0.30,
+                 text_threshold: float = 0.25, device: str = "cuda") -> list:
+    """Batched detection: ONE forward for N images sharing one vocabulary.
+    → [boxes_list per image] (same record shape as detect())."""
+    import torch
+    proc, model = gdino
+    images = list(images)
+    text = _gdino_prompt(classes)
+    inputs = proc(images=images, text=[text] * len(images),
+                  return_tensors="pt", padding=True).to(device)
+    with torch.no_grad():
+        outputs = model(**inputs)
+    sizes = [(im.size[1], im.size[0]) for im in images]          # (H, W)
+    _kw = dict(text_threshold=text_threshold, target_sizes=sizes)
+    try:
+        res = proc.post_process_grounded_object_detection(
+            outputs, inputs["input_ids"], threshold=box_threshold, **_kw)
+    except TypeError:
+        res = proc.post_process_grounded_object_detection(
+            outputs, inputs["input_ids"], box_threshold=box_threshold, **_kw)
+    out = []
+    for r in res:
+        labels = r["text_labels"] if "text_labels" in r else r.get("labels")
+        out.append([{"label": str(l).strip() or "object",
+                     "box": [float(v) for v in b.tolist()], "score": float(s)}
+                    for b, s, l in zip(r["boxes"], r["scores"], labels)])
+    return out
+
+
+def zero_shot_batch(siglip, images, labels, device: str = "cuda",
+                    template: str = "a photo of a {}.") -> list:
+    """Batched SigLIP2 zero-shot → per-image ranked [{label, score}]."""
+    import torch
+    proc, model = siglip
+    texts = [template.format(l) for l in labels]
+    # max_length=64 is REQUIRED: the SigLIP2 Gemma tokenizer has no model_max_length,
+    # so padding="max_length" alone silently degrades to no padding (HF's own
+    # zero-shot pipeline hardcodes 64 for the siglip family)
+    inputs = proc(text=texts, images=list(images), return_tensors="pt",
+                  padding="max_length", max_length=64, truncation=True).to(device)
+    with torch.no_grad():
+        logits = model(**inputs).logits_per_image                # [B, n_text]
+    probs = torch.sigmoid(logits).float().cpu().numpy()
+    out = []
+    for row in probs:
+        order = row.argsort()[::-1]
+        out.append([{"label": labels[i], "score": float(row[i])} for i in order])
+    return out
+
+
+def depth_map_batch(dp, images) -> list:
+    """Batched Depth-Anything → per-image HxW float32 nearness maps.
+
+    The DPT image processor resizes with keep_aspect_ratio and NO padding, so a
+    mixed-aspect batch produces ragged tensors and crashes. Images are therefore
+    grouped by exact size (one forward per group) — byte-identical to the serial
+    path, and full batching whenever a set shares a resolution (the synth set)."""
+    import torch
+    from PIL import Image as _I
+    proc, model = dp
+    images = list(images)
+    groups: dict = {}
+    for i, im in enumerate(images):
+        groups.setdefault(im.size, []).append(i)
+    out = [None] * len(images)
+    for size, idxs in sorted(groups.items()):
+        chunk = [images[i] for i in idxs]
+        inputs = proc(images=chunk, return_tensors="pt").to(model.device)
+        with torch.no_grad():
+            pd = model(**inputs).predicted_depth                 # [B, h, w]
+        W, H = size
+        for arr, i in zip(pd, idxs):
+            a = arr.squeeze().float().cpu().numpy().astype(np.float32)
+            if a.shape != (H, W):
+                a = np.asarray(_I.fromarray(a).resize((W, H), _I.BILINEAR),
+                               dtype=np.float32)
+            out[i] = a
+    return out
+
+
+def segment_batch(sam, images, boxes_list, device: str = "cuda") -> list:
+    """Batched grounded-SAM. Variable per-image box counts are PADDED to the batch
+    max (dummy [0,0,2,2] prompts) and the surplus masks dropped — SAM's processor
+    needs a rectangular input_boxes tensor. Mutates boxes in place like segment()."""
+    if sam is None or not any(boxes_list):
+        return boxes_list
+    import torch
+    proc, model = sam
+    keep = [i for i, bl in enumerate(boxes_list) if bl]
+    imgs = [images[i] for i in keep]
+    max_n = max(len(boxes_list[i]) for i in keep)
+    padded = [[[float(v) for v in b["box"]] for b in boxes_list[i]]
+              + [[0.0, 0.0, 2.0, 2.0]] * (max_n - len(boxes_list[i]))
+              for i in keep]
+    inputs = proc(imgs, input_boxes=padded, return_tensors="pt").to(device)
+    with torch.no_grad():
+        outputs = model(**inputs)
+    masks = proc.image_processor.post_process_masks(
+        outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(),
+        inputs["reshaped_input_sizes"].cpu())                    # [n_obj, 3, H, W] per image
+    scores = outputs.iou_scores.cpu().numpy()                    # [B, n_obj, 3]
+    for bi, i in enumerate(keep):
+        m = np.asarray(masks[bi])
+        sc = scores[bi]
+        for oi, b in enumerate(boxes_list[i]):                   # surplus (padded) masks ignored
+            mo = m[oi]
+            if mo.ndim == 3:
+                best = int(sc[oi].argmax()) if oi < len(sc) else 0
+                b["mask_score"] = float(sc[oi][best]) if oi < len(sc) else None
+                mo = mo[best]
+            b["mask"] = np.asarray(mo, dtype=bool)
+    return boxes_list
+
+
+def ground_phrases_batch(gdino, images, phrases_list, box_threshold: float = 0.25,
+                         text_threshold: float = 0.20, max_tokens_per_chunk: int = 250,
+                         device: str = "cuda") -> list:
+    """Batched phrase grounding. Images whose phrase text fits ONE chunk (the
+    typical case) share a single forward; oversized ones fall back to the serial
+    chunked ground_phrases. → per-image attr-box record lists."""
+    import torch
+    proc, model = gdino
+    norm = [[p.strip().lower() for p in (ph or []) if p and p.strip()]
+            for ph in phrases_list]
+    tok = getattr(proc, "tokenizer", None)
+    out = [[] for _ in images]
+
+    easy, hard = [], []
+    for i, ph in enumerate(norm):
+        if not ph:
+            continue
+        text = ". ".join(ph) + "."
+        ntok = (len(tok(text)["input_ids"]) if tok is not None
+                else 2 * len(text.split()))
+        (easy if ntok <= max_tokens_per_chunk else hard).append(i)
+
+    if easy:
+        imgs = [images[i] for i in easy]
+        texts = [". ".join(norm[i]) + "." for i in easy]
+        inputs = proc(images=imgs, text=texts, return_tensors="pt",
+                      padding=True).to(device)
+        with torch.no_grad():
+            outputs = model(**inputs)
+        sizes = [(im.size[1], im.size[0]) for im in imgs]
+        _kw = dict(text_threshold=text_threshold, target_sizes=sizes)
+        try:
+            res = proc.post_process_grounded_object_detection(
+                outputs, inputs["input_ids"], threshold=box_threshold, **_kw)
+        except TypeError:
+            res = proc.post_process_grounded_object_detection(
+                outputs, inputs["input_ids"], box_threshold=box_threshold, **_kw)
+        for bi, i in enumerate(easy):
+            recs = _remap_spans(res[bi], norm[i])
+            recs.sort(key=lambda r: (r["phrase"], -r["score"]))
+            out[i] = recs
+    for i in hard:
+        out[i] = ground_phrases(gdino, images[i], norm[i],
+                                box_threshold=box_threshold,
+                                text_threshold=text_threshold,
+                                max_tokens_per_chunk=max_tokens_per_chunk,
+                                device=device)
+    return out
+
+
+def solids_from_detection(image, boxes) -> Solids:
+    """Minimal Solids from detection alone (feeds bbox / spatial / subject)."""
+    return Solids(size=image.size, boxes=boxes)
+
+
+# ── validation: detection hub vs COCO GT, through the existing scorers ────────
+def validate_detection(n: int = 24, box_threshold: float = 0.30, device: str = "cuda") -> dict:
+    """Run GroundingDINO on real COCO images, build the bbox JSON, and score it with the
+    EXISTING vlmbench detection scorer — apples-to-apples with the VLM's bbox number."""
+    from .datasets import load_gt
+    from .metrics import score_vision_sample, score_vision_run
+
+    spec = get_task("bbox_grounding")
+    proc, model = load_grounding_dino(device)
+    samples = load_gt(spec.gt_dataset, n=n, split=spec.gt_split, dataset="full")
+    print(f"[validate_detection] {GROUNDING_DINO_ID} on {len(samples)} COCO images")
+
+    results, t0 = [], time.perf_counter()
+    for i, s in enumerate(samples):
+        boxes = detect(proc, model, s.image, COCO_CLASSES, box_threshold, device=device)
+        pred = build_bbox(solids_from_detection(s.image, boxes))
+        mr = score_vision_sample(spec, json.dumps(pred), s.gt, mode="specialist",
+                                 image_id=s.image_id, image_size=s.size)
+        results.append(mr)
+        if i < 3:
+            print(f"  {s.image_id}: {len(boxes)} boxes, primary={mr.primary_score}")
+    dt = time.perf_counter() - t0
+    run = score_vision_run(results, model="grounding-dino-base", category=spec.category, mode="specialist")
+    out = {"model": GROUNDING_DINO_ID, "n": len(samples),
+           "primary_score_mean": run.primary_score_mean, "schema_valid_rate": run.schema_valid_rate,
+           "img_per_s": round(len(samples) / max(0.001, dt), 2)}
+    print(f"\n[validate_detection] mean primary={out['primary_score_mean']}  "
+          f"valid={out['schema_valid_rate']}  {out['img_per_s']} img/s")
+    print("Compare vs the VLM bbox_grounding effective yield (~0.16–0.30 in the vlmbench).")
+    return out
+
+
+# ── depth hub: Depth-Anything-V2-Small (Apache; higher = nearer) ─────────────
+DEPTH_ID = "depth-anything/Depth-Anything-V2-Small-hf"   # ONLY Small is Apache
+
+def load_depth_anything(device: str = "cuda"):
+    """Returns (processor, model). Direct model call — the transformers pipeline()
+    warns ("use a dataset") when invoked per-image on GPU and adds dispatch overhead."""
+    from transformers import AutoImageProcessor, AutoModelForDepthEstimation
+    proc = AutoImageProcessor.from_pretrained(DEPTH_ID)
+    model = AutoModelForDepthEstimation.from_pretrained(DEPTH_ID).to(device).eval()
+    return proc, model
+
+
+def depth_map(dp, image):
+    """HxW float32 relative depth; Depth-Anything convention: HIGHER = NEARER."""
+    import torch
+    proc, model = dp
+    inputs = proc(images=image, return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        pd = model(**inputs).predicted_depth
+    arr = pd.squeeze().float().cpu().numpy().astype(np.float32)
+    # resize to image size if the model returned a different resolution
+    W, H = image.size
+    if arr.shape != (H, W):
+        from PIL import Image as _I
+        arr = np.asarray(_I.fromarray(arr).resize((W, H), _I.BILINEAR), dtype=np.float32)
+    return arr
+
+
+# ── segmentation hub: SAM v1 (Apache), prompted by detection boxes ───────────
+#   SAM2's transformers processor is currently broken (missing preprocessor_config on the -hf
+#   repo); SAM v1 is equally Apache-2.0, box-promptable, and rock-solid in transformers.
+SAM_ID = "facebook/sam-vit-base"
+
+def load_sam(device: str = "cuda"):
+    """Returns (processor, model) or None. SAM v1 via transformers (SamModel/SamProcessor)."""
+    try:
+        from transformers import SamModel, SamProcessor
+        proc = SamProcessor.from_pretrained(SAM_ID)
+        model = SamModel.from_pretrained(SAM_ID).to(device).eval()
+        return proc, model
+    except Exception as e:
+        print(f"[load_sam] SAM unavailable: {type(e).__name__}: {e}")
+        return None
+
+
+def segment(sam, image, boxes, device: str = "cuda"):
+    """Attach a boolean mask to each box dict (in place). Grounded-SAM: box → mask.
+    Picks the highest-IoU of SAM's 3 mask proposals per box."""
+    if sam is None or not boxes:
+        return boxes
+    import torch
+    proc, model = sam
+    input_boxes = [[[float(v) for v in b["box"]] for b in boxes]]     # [image][obj][xyxy]
+    inputs = proc(image, input_boxes=input_boxes, return_tensors="pt").to(device)
+    with torch.no_grad():
+        outputs = model(**inputs)
+    masks = proc.image_processor.post_process_masks(
+        outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(),
+        inputs["reshaped_input_sizes"].cpu())[0]                     # [obj, n_masks, H, W]
+    m = np.asarray(masks)
+    scores = outputs.iou_scores.cpu().numpy()[0]                      # [obj, n_masks]
+    for i, b in enumerate(boxes):
+        if i >= len(m):
+            break
+        mi = m[i]
+        if mi.ndim == 3:                                             # [n_masks, H, W] → best by IoU
+            best = int(scores[i].argmax()) if i < len(scores) else 0
+            b["mask_score"] = float(scores[i][best]) if i < len(scores) else None
+            mi = mi[best]
+        b["mask"] = np.asarray(mi, dtype=bool)
+    return boxes
+
+
+# ── classification / style hub: SigLIP2 (Apache) zero-shot ───────────────────
+SIGLIP_ID = "google/siglip2-so400m-patch14-384"
+STYLE_LABELS = ["photo", "painting", "3d_render", "sketch", "anime", "other"]
+
+def load_siglip(device: str = "cuda"):
+    from transformers import AutoProcessor, AutoModel
+    from transformers.utils import logging as hf_logging
+    # The checkpoint's config carries CLIP-tokenizer bos/eos ids (49406/49407) that
+    # newer transformers flags against the 32k text vocab. SigLIP never generates,
+    # so the ids are inert — silence the config validation for just this load.
+    prev = hf_logging.get_verbosity()
+    hf_logging.set_verbosity_error()
+    try:
+        proc = AutoProcessor.from_pretrained(SIGLIP_ID)
+        model = AutoModel.from_pretrained(SIGLIP_ID).to(device).eval()
+    finally:
+        hf_logging.set_verbosity(prev)
+    return proc, model
+
+
+def zero_shot(siglip, image, labels, device: str = "cuda", template: str = "a photo of a {}.") -> list:
+    """SigLIP2 zero-shot → [{label, score}] sorted desc (sigmoid, not softmax)."""
+    import torch
+    proc, model = siglip
+    texts = [template.format(l) for l in labels]
+    # max_length=64 required — see zero_shot_batch (SigLIP2 tokenizer has no
+    # model_max_length, so padding="max_length" alone silently doesn't pad)
+    inputs = proc(text=texts, images=image, return_tensors="pt", padding="max_length",
+                  max_length=64, truncation=True).to(device)
+    with torch.no_grad():
+        logits = model(**inputs).logits_per_image[0]
+    probs = torch.sigmoid(logits).float().cpu().numpy()
+    order = probs.argsort()[::-1]
+    return [{"label": labels[i], "score": float(probs[i])} for i in order]
+
+
+# ── OCR hub: EasyOCR (Apache, torch — no Paddle/torch CUDA conflict) ──────────
+def load_ocr(device: str = "cuda"):
+    try:
+        import easyocr
+        return easyocr.Reader(["en"], gpu=(device == "cuda"))
+    except Exception as e:
+        print(f"[load_ocr] EasyOCR unavailable: {type(e).__name__}: {e}")
+        return None
+
+
+def ocr_read(reader, image) -> dict:
+    """EasyOCR → {full_text, lines:[{text, box:[quad px], conf}]}. Confidence is
+    RETAINED (the fusion tier carries it; build_ocr ignores the extra key)."""
+    if reader is None:
+        return {"full_text": "", "lines": []}
+    res = reader.readtext(np.asarray(image))     # [(quad, text, conf), ...]
+    lines = [{"text": str(t), "box": [[float(x), float(y)] for x, y in quad],
+              "conf": float(c)} for quad, t, c in res]
+    return {"full_text": " ".join(l["text"] for l in lines), "lines": lines}
+
+
+# ── generic per-task validation through the existing scorers ─────────────────
+SHAPE_CLASSES = ["red circle", "green circle", "blue circle"]   # synthetic-shape GT vocab
+
+# task → dict of which models it needs, the GDINO vocab, and (optional) a REAL-image GT
+# override that replaces the synthetic GT in the task spec.
+_TASK_CFG = {
+    "bbox_grounding":               dict(vocab=COCO_CLASSES, gdino=True),
+    "segmentation":                 dict(vocab=COCO_CLASSES, gdino=True, masks=True, gt="coco_segmentation"),
+    "outline_association":          dict(vocab=COCO_CLASSES, gdino=True, masks=True, gt="coco_outline"),
+    "subject_fixation":             dict(vocab=COCO_CLASSES, gdino=True, gt="coco_subject"),
+    # still synthetic — need real depth (NYU/DIODE) + relations (Visual Genome); next real-GT pass
+    "depth_analysis":               dict(vocab=SHAPE_CLASSES, gdino=True, depth=True, masks=True),
+    "structural_spatial_awareness": dict(vocab=SHAPE_CLASSES, gdino=True, depth=True),
+    "image_classification":         dict(siglip=True),                       # vocab from GT labels
+    "style_structural_awareness":   dict(gdino=True, siglip=True, gray=True),  # style has no real GT
+    "ocr_text":                     dict(ocr=True),
+    "data_type_differentiation":    dict(ocr=True),                          # rendered-format GT is synthetic
+    "data_type_utilization":        dict(ocr=True),
+}
+
+
+def validate_task(task: str, n: int = 24, device: str = "cuda", *, gdino=None,
+                  depth_pipe=None, sam=None, siglip=None, ocr=None) -> dict:
+    """Run the specialist/derive chain for one task and score it with the vlmbench scorer."""
+    from .datasets import load_gt
+    from .metrics import score_vision_sample, score_vision_run
+    from .specialists import (Solids, build_bbox, build_spatial, build_subject,
+                              build_depth_order, build_segmentation, build_outline,
+                              build_classification, build_style, build_ocr,
+                              build_datatype_diff, build_datatype_util)
+
+    spec = get_task(task)
+    cfg = _TASK_CFG[task]
+    gt_key = cfg.get("gt", spec.gt_dataset)               # real-image GT override when available
+    samples = load_gt(gt_key, n=n, split=spec.gt_split or "", dataset="full")
+
+    # candidate label set for zero-shot classification: the classes present in this GT slice
+    class_vocab = None
+    if task == "image_classification":
+        seen = []
+        for s in samples:
+            for l in (s.gt.get("labels", []) if isinstance(s.gt, dict) else []):
+                if l not in seen:
+                    seen.append(l)
+        class_vocab = seen or ["object"]
+
+    results = []
+    for s in samples:
+        sol = Solids(size=s.image.size)
+        if cfg.get("gdino") and gdino is not None:
+            sol.boxes = detect(gdino[0], gdino[1], s.image, cfg.get("vocab", COCO_CLASSES), device=device)
+            if cfg.get("masks") and sam is not None:
+                sol.boxes = segment(sam, s.image, sol.boxes, device=device)
+        if cfg.get("depth") and depth_pipe is not None:
+            sol.depth = depth_map(depth_pipe, s.image)
+        if cfg.get("gray"):
+            sol.gray = np.asarray(s.image.convert("L"), dtype=np.float32)
+        if cfg.get("siglip") and siglip is not None:
+            if task == "image_classification":
+                sol.class_top = zero_shot(siglip, s.image, class_vocab, device=device)[:5]
+            if task == "style_structural_awareness":
+                sol.style = zero_shot(siglip, s.image, STYLE_LABELS, device=device)[0]["label"]
+        if cfg.get("ocr") and ocr is not None:
+            sol.ocr = ocr_read(ocr, s.image)
+
+        if task == "depth_analysis":
+            pred = build_depth_order(sol)
+        elif task == "segmentation":
+            pred = build_segmentation(sol)
+        elif task == "outline_association":
+            pred = build_outline(sol)
+        elif task == "structural_spatial_awareness":
+            pred = build_spatial(sol)
+        elif task == "subject_fixation":
+            pred = build_subject(sol)
+        elif task == "bbox_grounding":
+            pred = build_bbox(sol)
+        elif task == "image_classification":
+            pred = build_classification(sol)
+        elif task == "style_structural_awareness":
+            pred = build_style(sol)
+        elif task == "ocr_text":
+            pred = build_ocr(sol)
+        elif task == "data_type_differentiation":
+            pred = build_datatype_diff(sol)
+        elif task == "data_type_utilization":
+            pred = build_datatype_util(sol)[0]
+        else:
+            raise KeyError(task)
+
+        mr = score_vision_sample(spec, json.dumps(pred), s.gt, mode="specialist",
+                                 image_id=s.image_id, image_size=s.size)
+        results.append(mr)
+    run = score_vision_run(results, model="specialist", category=task, mode="specialist")
+    return {"task": task, "n": len(samples), "primary": run.primary_score_mean,
+            "valid": run.schema_valid_rate}
+
+
+class SpecialistPipeline:
+    """Simplified interface: load the Apache/MIT specialist models ONCE, then `extract(image)`
+    returns every deterministic task's JSON (the production entry point for `tasks_json`).
+
+        pipe = SpecialistPipeline()
+        tasks = pipe.extract(pil_image)   # {"bbox_grounding": {...}, "segmentation": {...}, ...}
+    """
+    DEFAULT_VOCAB = COCO_CLASSES
+
+    def __init__(self, device: str = "cuda", with_ocr: bool = True):
+        self.device = device
+        self.gdino = load_grounding_dino(device)
+        self.depth = load_depth_anything(device)
+        self.sam = load_sam(device)
+        self.siglip = load_siglip(device)
+        self.ocr = load_ocr(device) if with_ocr else None
+
+    def solidify(self, image, vocab=None):
+        """Run every specialist once → a `Solids` (primitives in pixel space)."""
+        vocab = vocab or self.DEFAULT_VOCAB
+        s = Solids(size=image.size)
+        s.boxes = detect(self.gdino[0], self.gdino[1], image, vocab, device=self.device)
+        s.boxes = segment(self.sam, image, s.boxes, device=self.device)
+        s.depth = depth_map(self.depth, image)
+        s.gray = np.asarray(image.convert("L"), dtype=np.float32)
+        if self.siglip is not None:
+            s.class_top = zero_shot(self.siglip, image, vocab, device=self.device)[:5]
+            s.style = zero_shot(self.siglip, image, STYLE_LABELS, device=self.device)[0]["label"]
+        if self.ocr is not None:
+            s.ocr = ocr_read(self.ocr, image)
+        return s
+
+    @staticmethod
+    def _build_tasks(s) -> dict:
+        """Solids → {task_name: task_json} for all 11 deterministic tasks."""
+        from .specialists import (build_bbox, build_segmentation, build_classification,
+                                  build_ocr, build_spatial, build_depth_order, build_subject,
+                                  build_outline, build_style, build_datatype_diff,
+                                  build_datatype_util)
+        util, _ = build_datatype_util(s)
+        return {
+            "bbox_grounding": build_bbox(s),
+            "segmentation": build_segmentation(s),
+            "outline_association": build_outline(s),
+            "subject_fixation": build_subject(s),
+            "depth_analysis": build_depth_order(s),
+            "structural_spatial_awareness": build_spatial(s),
+            "image_classification": build_classification(s),
+            "style_structural_awareness": build_style(s),
+            "ocr_text": build_ocr(s),
+            "data_type_differentiation": build_datatype_diff(s),
+            "data_type_utilization": util,
+        }
+
+    def extract(self, image, vocab=None) -> dict:
+        """→ {task_name: task_json} for all 11 deterministic tasks (one solidify pass)."""
+        return self._build_tasks(self.solidify(image, vocab))
+
+    def ground(self, image, phrases) -> list:
+        """Fusion grounding pass: caption phrases → attr-box records (GDINO reused)."""
+        return ground_phrases(self.gdino, image, phrases, device=self.device)
+
+    def solidify_batch(self, images, vocab=None, phrases_list=None,
+                       batch: int = 16, gdino_batch: int = 2) -> list:
+        """Batched solidify: SAM/depth/SigLIP run at `batch` images per forward;
+        GroundingDINO runs in sub-chunks of `gdino_batch` — its deformable
+        attention's activation memory EXPLODES with padded batches (measured on
+        the 96GB Blackwell: B2 = 11GB, B16 = 42GB for LESS throughput), so ~2 is
+        its sweet spot. EasyOCR stays serial (4% of the budget). → [Solids]
+        aligned with `images`, same output contract as solidify()."""
+        vocab = vocab or self.DEFAULT_VOCAB
+        images = list(images)
+        solids = []
+        for start in range(0, len(images), batch):
+            chunk = images[start:start + batch]
+            p_chunk = (phrases_list[start:start + batch]
+                       if phrases_list is not None else None)
+            boxes_list = []
+            for s2 in range(0, len(chunk), gdino_batch):
+                boxes_list.extend(detect_batch(
+                    self.gdino, chunk[s2:s2 + gdino_batch], vocab,
+                    device=self.device))
+            boxes_list = segment_batch(self.sam, chunk, boxes_list, device=self.device)
+            depths = (depth_map_batch(self.depth, chunk)
+                      if self.depth is not None else [None] * len(chunk))
+            classes = (zero_shot_batch(self.siglip, chunk, vocab, device=self.device)
+                       if self.siglip is not None else [None] * len(chunk))
+            styles = (zero_shot_batch(self.siglip, chunk, STYLE_LABELS, device=self.device)
+                      if self.siglip is not None else [None] * len(chunk))
+            if p_chunk is not None:
+                attrs = []
+                for s2 in range(0, len(chunk), gdino_batch):
+                    attrs.extend(ground_phrases_batch(
+                        self.gdino, chunk[s2:s2 + gdino_batch],
+                        p_chunk[s2:s2 + gdino_batch], device=self.device))
+            else:
+                attrs = [[] for _ in chunk]
+            for k, im in enumerate(chunk):
+                s = Solids(size=im.size)
+                s.boxes = boxes_list[k]
+                s.depth = depths[k]
+                s.gray = np.asarray(im.convert("L"), dtype=np.float32)
+                if classes[k] is not None:
+                    s.class_top = classes[k][:5]
+                    s.style = styles[k][0]["label"]
+                if self.ocr is not None:
+                    s.ocr = ocr_read(self.ocr, im)
+                s.attr_boxes = attrs[k]
+                solids.append(s)
+        return solids
+
+    def extract_batch(self, images, vocab=None, phrases_list=None,
+                      batch: int = 16) -> list:
+        """Batched extract(+digest): → [(tasks_dict, digest)] aligned with images."""
+        from .fuse import solids_digest
+        out = []
+        for s in self.solidify_batch(images, vocab, phrases_list, batch=batch):
+            out.append((self._build_tasks(s), solids_digest(s)))
+        return out
+
+    def extract_with_digest(self, image, phrases=None, vocab=None) -> tuple:
+        """→ (tasks_dict, solids_digest) from ONE solidify pass (+ the phrase-
+        grounding pass when `phrases` is given). The digest is the fusion tier's
+        input — compact, JSON-able, carries the retained confidences."""
+        from .fuse import solids_digest
+        s = self.solidify(image, vocab)
+        if phrases:
+            s.attr_boxes = self.ground(image, phrases)
+        return self._build_tasks(s), solids_digest(s)
+
+
+def load_vlm(model_key: str = "qwen3vl-4b"):
+    from .model_registry import get_runner
+    return get_runner(model_key)
+
+
+def validate_task_vlm(task: str, n: int = 24, model_key: str = "qwen3vl-4b",
+                      runner=None, device: str = "cuda") -> dict:
+    """Run the Qwen VLM on the SAME (real) GT as validate_task — a true apples-to-apples
+    head-to-head. Reuses the existing VLMRunner + score path. Pass a pre-loaded `runner`
+    to avoid reloading the model per task."""
+    from .datasets import load_gt
+    from .metrics import score_vision_sample, score_vision_run
+
+    spec = get_task(task)
+    gt_key = _TASK_CFG[task].get("gt", spec.gt_dataset)
+    samples = load_gt(gt_key, n=n, split=spec.gt_split or "", dataset="full")
+    own = runner is None
+    if own:
+        runner = load_vlm(model_key)
+    results = []
+    try:
+        for s in samples:
+            up = s.prompt if spec.per_sample_prompt else None
+            res = runner.generate(spec, s.image, "json_mode", image_id=s.image_id,
+                                  image_size=s.size, gt=s.gt, user_prompt=up)
+            results.append(score_vision_sample(spec, res.raw_text, s.gt, mode="json_mode",
+                                               image_id=s.image_id, image_size=s.size))
+    finally:
+        if own:
+            close = getattr(runner, "close", None)
+            if callable(close):
+                close()
+    run = score_vision_run(results, model=model_key, category=task, mode="json_mode")
+    # effective yield = accuracy × validity (the vlmbench headline metric)
+    acc = run.primary_score_mean
+    return {"task": task, "vlm_primary": acc, "vlm_valid": run.schema_valid_rate,
+            "vlm_yield": (acc * run.schema_valid_rate) if acc is not None else None}
+

qwen_test_runner/vision/strata.py

@@ -0,0 +1,185 @@
+"""
+strata.py — the attribute stratification lexicon (fusion tier).
+
+Classifies a caption attribute string into a STRATUM — the "disperse the topics"
+layer that replaces the flat attribute list with typed, routable records. Pure
+stdlib, deterministic, registry-as-python (same pattern as registry.py /
+tasks_vision.py): the lexicon is data to iterate on, not code.
+
+Routing semantics consumed by fuse.py:
+  - GROUNDABLE strata are sent to the GDINO phrase-grounding pass (they name
+    visible things a detector can box).
+  - "scene_level" bypasses entities entirely -> FusedScene.scene.scene_attributes.
+  - "abstract_quality", "color", and "action" ride the caption-binding-only path
+    (never grounded: a detector box for "elegant" or bare "red" is noise).
+  - Everything is classified; "abstract_quality" is the catch-all default.
+"""
+
+from __future__ import annotations
+
+import re
+
+from .metrics import _depluralize
+
+# Minimal stopword set for head-noun extraction (articles/preps/conjunctions that
+# can trail a phrase). Deliberately tiny — attribute phrases are short.
+_STOP = frozenset({
+    "a", "an", "the", "of", "in", "on", "at", "with", "and", "or", "to",
+    "her", "his", "its", "their", "very", "slightly",
+})
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")
+
+
+STRATA: dict[str, frozenset] = {
+    "hair": frozenset({
+        "hair", "hairstyle", "bangs", "fringe", "ponytail", "pigtails", "twintails",
+        "braid", "braids", "bun", "curls", "updo", "bob", "undercut", "mohawk",
+        "sidelocks", "ahoge", "afro", "dreadlocks", "cornrows", "mullet", "buzzcut",
+    }),
+    "face": frozenset({
+        "face", "eyes", "eye", "eyebrows", "eyebrow", "eyelashes", "lips", "lip",
+        "mouth", "nose", "cheeks", "cheekbones", "chin", "jaw", "jawline", "forehead",
+        "freckles", "dimples", "beard", "mustache", "stubble", "smile", "grin",
+        "expression", "gaze", "makeup", "lipstick", "eyeliner", "eyeshadow", "blush",
+        "mascara", "teeth",
+    }),
+    "skin": frozenset({
+        "skin", "complexion", "tan", "tattoo", "tattoos", "scar", "scars", "mole",
+        "birthmark", "wrinkles", "pores",
+    }),
+    "clothing": frozenset({
+        "dress", "shirt", "t-shirt", "tshirt", "blouse", "top", "skirt", "pants",
+        "trousers", "jeans", "shorts", "jacket", "coat", "hoodie", "sweater",
+        "cardigan", "vest", "suit", "uniform", "kimono", "yukata", "robe", "gown",
+        "leotard", "swimsuit", "bikini", "armor", "cape", "cloak", "apron",
+        "sleeves", "sleeve", "collar", "neckline", "hem", "outfit", "attire",
+        "clothes", "clothing", "costume", "sweatshirt", "leggings", "stockings",
+        "tights", "socks", "corset", "bodysuit", "tunic", "sari", "poncho",
+    }),
+    "accessory": frozenset({
+        "earrings", "earring", "necklace", "pendant", "choker", "bracelet", "ring",
+        "rings", "watch", "hat", "cap", "beanie", "beret", "crown", "tiara",
+        "headband", "hairband", "ribbon", "bow", "hairpin", "hairclip", "scrunchie",
+        "glasses", "sunglasses", "eyepatch", "monocle", "mask", "scarf", "gloves",
+        "glove", "belt", "bag", "handbag", "backpack", "purse", "umbrella", "fan",
+        "brooch", "badge", "piercing", "anklet", "shoes", "boots", "sandals",
+        "heels", "sneakers", "veil", "headphones", "tie", "bowtie",
+    }),
+    "body": frozenset({
+        "build", "figure", "physique", "body", "shoulders", "shoulder", "arms",
+        "arm", "hands", "hand", "fingers", "legs", "leg", "thighs", "knees",
+        "feet", "chest", "waist", "hips", "back", "neck", "collarbone", "height",
+        "frame", "posture", "muscles", "abs", "curves",
+    }),
+    "pose": frozenset({
+        "standing", "sitting", "kneeling", "crouching", "lying", "leaning",
+        "walking", "running", "jumping", "dancing", "posing", "looking", "facing",
+        "reaching", "pointing", "waving", "holding", "carrying", "crossed",
+        "outstretched", "tilted", "turned", "pose", "stance",
+    }),
+    "color": frozenset({
+        "red", "orange", "yellow", "green", "blue", "purple", "violet", "pink",
+        "brown", "black", "white", "gray", "grey", "silver", "gold", "golden",
+        "blonde", "blond", "brunette", "auburn", "crimson", "scarlet", "teal",
+        "turquoise", "cyan", "magenta", "lavender", "beige", "cream", "ivory",
+        "navy", "maroon", "olive", "platinum", "pastel", "neon", "dark", "light",
+        "pale", "bright", "vivid", "striped", "plaid", "polka-dot", "checkered",
+        "floral", "gradient",
+    }),
+    "abstract_quality": frozenset({
+        "beautiful", "pretty", "handsome", "cute", "elegant", "graceful", "stylish",
+        "fashionable", "detailed", "intricate", "delicate", "soft", "sharp",
+        "masterpiece", "quality", "aesthetic", "gorgeous", "stunning", "charming",
+        "youthful", "mature", "young", "old", "confident", "shy", "serene", "calm",
+        "cheerful", "melancholic", "mysterious", "dramatic", "ethereal", "dreamy",
+    }),
+    "scene_level": frozenset({
+        "background", "foreground", "backdrop", "lighting", "light", "shadow",
+        "shadows", "sunlight", "moonlight", "sunset", "sunrise", "dusk", "dawn",
+        "sky", "clouds", "bokeh", "blur", "depth", "wall", "walls", "floor",
+        "ceiling", "window", "windows", "door", "room", "indoors", "outdoors",
+        "outdoor", "indoor", "scenery", "landscape", "cityscape", "street",
+        "forest", "beach", "mountains", "atmosphere", "ambiance", "setting",
+        "scene", "environment", "composition", "framing",
+    }),
+}
+
+# hyphen/compound-adjective suffixes -> stratum ("silver-haired", "blue-eyed")
+SUFFIX_RULES: tuple = (
+    ("haired", "hair"),
+    ("eyed", "face"),
+    ("faced", "face"),
+    ("skinned", "skin"),
+    ("sleeved", "clothing"),
+    ("dressed", "clothing"),
+    ("clad", "clothing"),
+    ("shouldered", "body"),
+    ("legged", "body"),
+    ("armed", "body"),
+)
+
+# -ing words that are NOUNS, not gerunds — exempt from the verb-phrase rule
+# (data to extend as COCO round-trips surface more)
+_NOUN_ING = frozenset({
+    "wedding", "building", "painting", "lighting", "ceiling", "clothing",
+    "evening", "morning", "string", "earring", "ring", "king", "wing",
+    "railing", "awning",
+})
+
+# Strata whose phrases go to the GDINO grounding pass (visible, boxable things).
+GROUNDABLE = frozenset({"hair", "face", "skin", "clothing", "accessory", "body", "pose"})
+
+# Any-token tie-break order (only reached when the head noun missed the lexicon).
+# Concrete/visible strata outrank colors and abstractions.
+STRATUM_PRECEDENCE = ("hair", "face", "skin", "accessory", "clothing", "body",
+                      "pose", "scene_level", "color", "abstract_quality")
+
+# The full stratum vocabulary fuse.py may emit ("action" is assigned by fuse.py to
+# caption `actions` entries directly — it has no lexicon and is never grounded).
+ALL_STRATA = tuple(STRATA.keys()) + ("action",)
+
+
+def _content_tokens(text: str) -> list:
+    return [t for t in _TOKEN_RE.findall((text or "").lower()) if t not in _STOP]
+
+
+def classify_stratum(text: str) -> str:
+    """Deterministic stratum for an attribute string.
+
+    1. head-noun rule: depluralized LAST content token, exact lexicon lookup
+    2. suffix rules on the head token ("silver-haired" -> hair)
+    3. any-token lookup in STRATUM_PRECEDENCE order
+    4. all tokens are color/pattern terms -> color
+    5. default -> abstract_quality (nothing is ever unclassified)
+    """
+    toks = _content_tokens(text)
+    if not toks:
+        return "abstract_quality"
+    # _depluralize is crude ("dress"->"dres") — always try the raw form too
+    head_forms = {toks[-1], _depluralize(toks[-1])}
+
+    for stratum, words in STRATA.items():
+        if head_forms & words:
+            return stratum
+    for suffix, stratum in SUFFIX_RULES:
+        if any(h.endswith(suffix) for h in head_forms):
+            return stratum
+    forms = [{t, _depluralize(t)} for t in toks]
+    for stratum in STRATUM_PRECEDENCE:
+        words = STRATA[stratum]
+        if any(f & words for f in forms):
+            return stratum
+    if all(f & STRATA["color"] for f in forms):
+        return "color"
+    # verb-phrase heuristic: leading gerund ("playing baseball", "taking a photo",
+    # "running") → pose. Caught live on COCO captions, where the structurer emits
+    # verb phrases as attributes that otherwise fell to abstract_quality.
+    first = toks[0]
+    if first.endswith("ing") and len(first) > 4 and first not in _NOUN_ING:
+        return "pose"
+    return "abstract_quality"
+
+
+def is_groundable(stratum: str) -> bool:
+    return stratum in GROUNDABLE

qwen_test_runner/vision/stub_runner.py

@@ -0,0 +1,115 @@
+"""
+stub_runner.py — A CPU, torch-free fake VLM for tests and dry runs.
+
+`StubVLMRunner` produces deterministic outputs so the whole orchestrator —
+scoring, durability, leaderboard, verdict — can be exercised with no GPU and no
+torch import. Three behaviours:
+
+  perfect : schema-valid output that MATCHES the ground truth (high accuracy) and
+            is emitted as clean bare JSON (json_robust = True).
+  fragile : the SAME content, but wrapped in a ```json fence so it only parses
+            after repair (json_robust = False) — used to prove that the
+            labeler_score penalizes fragile JSON even at equal accuracy.
+  random  : emits invalid / off output for a fraction of samples, to exercise the
+            reject sidecar and the schema_valid_rate path.
+
+Note: "fragile" uses a PROSE wrapper (structural repair), not a markdown fence —
+fence-stripping is benign/deterministic and no longer counts against robustness,
+so a fenced output would (correctly) still score as robust.
+
+It is a test double, so it is allowed to peek at the ground truth (a real runner
+never does).
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Optional
+
+from .coords import XYWH, XYXY, CoordSpace, from_canonical, to_canonical
+from .runner_types import VLMResult
+from .tasks_vision import VisionTaskSpec, resolved_system_prompt
+
+
+def _default_value(fs):
+    """A schema-valid placeholder for one field spec."""
+    if fs.cardinality == "list":
+        return []  # lists always validate (default_factory=list)
+    if fs.nested_fields:
+        return {f.name: _default_value(f) for f in fs.nested_fields}
+    if fs.vocabulary == "closed":
+        return fs.closed_values[0]
+    vk = fs.value_kind
+    if vk == "number":
+        return 0.0
+    if vk == "integer":
+        return 0
+    if vk == "bbox":
+        return [0.0, 0.0, 0.0, 0.0]
+    if vk == "point":
+        return [0.0, 0.0]
+    return "x"
+
+
+def _synthesize_valid(spec: VisionTaskSpec) -> dict:
+    return {name: _default_value(fs) for name, fs in spec.fields.items()}
+
+
+def _gt_to_prediction(spec: VisionTaskSpec, gt, image_size) -> Optional[dict]:
+    """Build a GT-matching, schema-valid prediction for the pilot categories.
+    Returns None if the category has no GT-driven construction (use synthesize)."""
+    cat = spec.category
+    if cat == "image_classification" and isinstance(gt, dict):
+        label = (gt.get("labels") or [gt.get("label", "x")])[0]
+        return {"label": label, "confidence": 0.95,
+                "top5": [{"label": label, "score": 0.95}]}
+    if cat == "bbox_grounding" and isinstance(gt, dict):
+        dets = []
+        for b in gt.get("boxes", []):
+            canon = to_canonical(b["bbox"], CoordSpace.PIXEL_ABS, image_size, fmt=b.get("fmt", XYWH))
+            box = from_canonical(canon, spec.coord_space, image_size, fmt=XYXY)
+            dets.append({"label": b.get("label", "x"), "box": [round(c, 2) for c in box], "score": 0.95})
+        return {"detections": dets, "count": len(dets)}
+    if cat == "ocr_text" and isinstance(gt, dict):
+        text = gt.get("text", "")
+        return {"full_text": text, "lines": [{"text": text}]}
+    return None
+
+
+class StubVLMRunner:
+    """Drop-in fake runner. Matches the VLMRunner.generate signature."""
+
+    def __init__(self, model_id: str = "stub", behavior: str = "perfect",
+                 reasoning: str = "instruct", **_kwargs):
+        self.model_id = model_id
+        self.behavior = behavior
+        self.reasoning = reasoning
+        self._n = 0
+
+    def close(self) -> None:  # symmetry with VLMRunner
+        pass
+
+    def generate(self, spec: VisionTaskSpec, image, mode: str, *,
+                 image_id: str = "", image_size=(64, 64), gt=None, user_prompt=None) -> VLMResult:
+        self._n += 1
+        # Build content
+        pred = _gt_to_prediction(spec, gt, image_size)
+        if pred is None:
+            pred = _synthesize_valid(spec)
+
+        if self.behavior == "random" and (self._n % 4 == 0):
+            raw = "I cannot answer that."  # invalid → exercises reject sidecar
+            return VLMResult(mode, raw, "stub", 8, 6, 0.001, image_id, grammar_conformant=False)
+
+        body = json.dumps(pred)
+        if self.behavior == "fragile":
+            # prose wrapper = STRUCTURAL repair (not a benign fence) → not robust
+            raw = f"Sure! Here is the structured result you requested: {body} — hope that helps!"
+        else:
+            raw = body  # clean bare JSON
+
+        # touch the resolved prompt so prompt wiring is exercised
+        _ = resolved_system_prompt(spec)
+        grammar = (mode == "constrained")
+        return VLMResult(mode, raw, "stub", 12, max(1, len(body) // 4), 0.001,
+                         image_id, grammar_conformant=grammar)

qwen_test_runner/vision/tasks_vision.py

@@ -0,0 +1,530 @@
+"""
+tasks_vision.py — The 15 vision-task categories, as data.
+
+Each VisionTaskSpec owns a small per-category field registry (dict[str, SlotSpec])
+and a system/user prompt. The Pydantic model, JSON Schema, GBNF grammar, and
+Claude tool schema are generated from that registry by the SAME machinery the
+caption schema uses (schema.build_*). Adding a category is one dict entry.
+
+Three categories are PILOT (full schema + GT dataset + real metric); the other
+twelve are STUB (valid minimal schema so their grammar builds, metric wired in
+Phase 3). This mirrors how registry.py grows the caption schema.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Mapping
+
+from ..registry import SlotSpec
+from ..schema import build_gbnf_from_registry, build_json_schema, build_model_from_registry
+from .coords import CoordSpace
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Field-builder shorthand (keeps the registry readable)
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _f(name, **kw) -> SlotSpec:
+    """A single-value open string field unless overridden."""
+    kw.setdefault("cardinality", "single")
+    kw.setdefault("vocabulary", "open")
+    return SlotSpec(name=name, **kw)
+
+
+def _enum(name, values, optional=False) -> SlotSpec:
+    return SlotSpec(name=name, cardinality="single", vocabulary="closed",
+                    closed_values=tuple(values), optional=optional)
+
+
+def _list_of(name, *fields, max_items=32) -> SlotSpec:
+    return SlotSpec(name=name, cardinality="list", vocabulary="open",
+                    nested_fields=tuple(fields), max_items=max_items)
+
+
+@dataclass(frozen=True)
+class VisionTaskSpec:
+    category: str
+    probes: str
+    fields: Mapping[str, SlotSpec]
+    system_prompt: str
+    user_prompt: str
+    metric: str                       # key into metrics._SCORERS
+    status: str = "pilot"             # "pilot" | "stub"
+    coord_space: CoordSpace = CoordSpace.NORM_0_1000
+    gt_dataset: str = ""              # key into datasets.DATASET_REGISTRY
+    gt_split: str = ""
+    max_new_tokens: int = 512
+    license_note: str = ""
+    download_gb: float = 0.0
+    per_sample_prompt: bool = False   # use GTSample.prompt as the user prompt (e.g. VQA question)
+
+
+# Generated-artifact caches (keyed by category — VisionTaskSpec holds a dict so
+# it isn't hashable; categories are unique).
+_MODEL_CACHE: dict[str, type] = {}
+_GBNF_CACHE: dict[str, str] = {}
+
+
+def model_for(spec: VisionTaskSpec):
+    if spec.category not in _MODEL_CACHE:
+        _MODEL_CACHE[spec.category] = build_model_from_registry(
+            "Vision_" + spec.category.title().replace("_", ""), dict(spec.fields)
+        )
+    return _MODEL_CACHE[spec.category]
+
+
+def json_schema_for(spec: VisionTaskSpec) -> dict:
+    return build_json_schema(model_for(spec))
+
+
+def gbnf_for(spec: VisionTaskSpec) -> str:
+    if spec.category not in _GBNF_CACHE:
+        _GBNF_CACHE[spec.category] = build_gbnf_from_registry(dict(spec.fields))
+    return _GBNF_CACHE[spec.category]
+
+
+def tool_schema_for(spec: VisionTaskSpec) -> dict:
+    """Claude-style tool input_schema (the per-category JSON Schema)."""
+    return json_schema_for(spec)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# PILOT categories (full)
+# ──────────────────────────────────────────────────────────────────────────────
+
+_CLASSIFICATION = VisionTaskSpec(
+    category="image_classification",
+    probes="native ViT classification emitted as JSON",
+    fields={
+        "label": _f("label", optional=False, max_str_length=64),
+        "confidence": _f("confidence", value_kind="number", optional=False, number_range=(0.0, 1.0)),
+        "top5": _list_of(
+            "top5",
+            _f("label", optional=False, max_str_length=64),
+            _f("score", value_kind="number", optional=False, number_range=(0.0, 1.0)),
+            max_items=5,
+        ),
+    },
+    system_prompt=(
+        "You are an image classifier. Identify the single most prominent object or scene "
+        "category in the image. Output ONLY a raw JSON object and NOTHING else — no prose, "
+        "no explanation, and NO markdown code fences (do not wrap it in ```). "
+        "It must match this shape exactly:\n"
+        '{"label": "<string>", "confidence": <number 0..1>, '
+        '"top5": [{"label": "<string>", "score": <number 0..1>}]}'
+    ),
+    user_prompt="Classify this image. Output only the raw JSON object.",
+    metric="classification",
+    gt_dataset="imagenet_val",
+    gt_split="validation",
+    max_new_tokens=160,
+    license_note="ImageNet: non-commercial research use.",
+)
+
+_BBOX = VisionTaskSpec(
+    category="bbox_grounding",
+    probes="object localization + grounded counting",
+    fields={
+        "detections": _list_of(
+            "detections",
+            _f("label", optional=False, max_str_length=64),
+            _f("box", value_kind="bbox", optional=False),
+            _f("score", value_kind="number", optional=False, number_range=(0.0, 1.0)),
+            max_items=32,
+        ),
+        "count": _f("count", value_kind="integer", optional=False),
+    },
+    system_prompt=(
+        "You are an object detector. Find every distinct object in the image. Output ONLY a "
+        "raw JSON object and NOTHING else — no prose, no markdown code fences (do not wrap it "
+        "in ```). It must match this shape exactly:\n"
+        '{"detections": [{"label": "<string>", "box": [x1, y1, x2, y2], "score": <number 0..1>}], '
+        '"count": <integer>}\n'
+        "{coord_hint} Use the key \"box\" (NOT bbox_2d) with exactly four numbers [x1, y1, x2, y2]."
+    ),
+    user_prompt="Detect all objects in this image. Output only the raw JSON object.",
+    metric="detection",
+    coord_space=CoordSpace.NORM_0_1000,
+    gt_dataset="coco_detection",
+    gt_split="val",
+    max_new_tokens=768,
+    license_note="COCO: CC-BY 4.0 (images vary).",
+)
+
+_OCR = VisionTaskSpec(
+    category="ocr_text",
+    probes="text reading + transcription fidelity + localization",
+    fields={
+        "full_text": _f("full_text", optional=False, max_str_length=4096),
+        "lines": _list_of(
+            "lines",
+            _f("text", optional=False, max_str_length=512),
+            _f("box", value_kind="bbox", optional=True),
+            max_items=64,
+        ),
+    },
+    system_prompt=(
+        "You are an OCR engine. Transcribe all readable text in the image. Output ONLY a raw "
+        "JSON object and NOTHING else — no prose, no markdown code fences (do not wrap it in "
+        "```). It must match this shape exactly:\n"
+        '{"full_text": "<all text, joined by spaces>", '
+        '"lines": [{"text": "<string>", "box": [x1, y1, x2, y2]}]}\n'
+        "{coord_hint} If you cannot localize a line, omit its box."
+    ),
+    user_prompt="Read all the text in this image. Output only the raw JSON object.",
+    metric="ocr",
+    coord_space=CoordSpace.NORM_0_1000,
+    gt_dataset="textvqa",
+    gt_split="validation",
+    max_new_tokens=512,
+    license_note="TextVQA: CC-BY 4.0.",
+)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# STUB categories (minimal valid schema; metric + GT wired in Phase 3)
+# ──────────────────────────────────────────────────────────────────────────────
+
+def _stub(category, probes, fields, prompt, **kw) -> VisionTaskSpec:
+    kw.setdefault("metric", "schema_only")
+    kw.setdefault("status", "stub")
+    kw.setdefault("user_prompt", "Analyze this image.")
+    return VisionTaskSpec(category=category, probes=probes, fields=fields,
+                          system_prompt=prompt, **kw)
+
+
+_SPATIAL_PREDS = ("left_of", "right_of", "above", "below", "on", "under",
+                  "inside", "behind", "in_front_of")
+
+_STUBS = []
+
+_DATATYPE_VALUES = ("json", "yaml", "markdown", "csv", "toml", "xml", "code", "plaintext")
+
+_DATATYPE_DIFF = VisionTaskSpec(
+    category="data_type_differentiation",
+    probes="recognize a rendered data format from a screenshot",
+    fields={
+        "data_type": _enum("data_type", _DATATYPE_VALUES),
+        "confidence": _f("confidence", value_kind="number", optional=False, number_range=(0.0, 1.0)),
+    },
+    system_prompt=(
+        "You are shown a screenshot of structured data. Identify which serialization format "
+        "it is. Output ONLY a raw JSON object, no markdown fences:\n"
+        '{"data_type": "<one of: json, yaml, markdown, csv, toml, xml, code, plaintext>", '
+        '"confidence": <number 0..1>}'
+    ),
+    user_prompt="What data format is shown? Output only the raw JSON object.",
+    metric="datatype_diff",
+    gt_dataset="datatype_synth",
+    max_new_tokens=96,
+    license_note="synthetic (self-contained).",
+)
+
+_SPATIAL = VisionTaskSpec(
+    category="structural_spatial_awareness",
+    probes="spatial relations between objects",
+    fields={"relations": _list_of(
+        "relations",
+        _f("subject", optional=False),
+        _enum("predicate", _SPATIAL_PREDS),
+        _f("object", optional=False), max_items=12)},
+    system_prompt=(
+        "Describe the spatial relations between the colored shapes. Subjects and objects are "
+        "the colors (red, green, blue). Output ONLY raw JSON, no fences:\n"
+        '{"relations": [{"subject": "<color>", "predicate": '
+        '"<left_of|right_of|above|below>", "object": "<color>"}]}'
+    ),
+    user_prompt="List the spatial relations between the colored shapes. Raw JSON only.",
+    metric="triples", gt_dataset="shapes_synth", max_new_tokens=256,
+    license_note="synthetic (self-contained).",
+)
+
+_DEPTH = VisionTaskSpec(
+    category="depth_analysis",
+    probes="relative depth ordering",
+    fields={
+        "nearest": _f("nearest"),
+        "farthest": _f("farthest"),
+        "relative_depth": _list_of(
+            "relative_depth",
+            _f("a", optional=False),
+            _f("b", optional=False),
+            _enum("a_is", ("nearer", "farther", "same")), max_items=12),
+    },
+    system_prompt=(
+        "Judge relative depth of the colored shapes: a LARGER shape appears NEARER. Output ONLY "
+        "raw JSON, no fences:\n{\"nearest\": \"<color>\", \"farthest\": \"<color>\", "
+        '"relative_depth": [{"a": "<color>", "b": "<color>", "a_is": "<nearer|farther|same>"}]}'
+    ),
+    user_prompt="Report the relative depth of the colored shapes. Raw JSON only.",
+    metric="depth_order", gt_dataset="shapes_synth", max_new_tokens=256,
+    license_note="synthetic (self-contained).",
+)
+
+_SUBJECT = VisionTaskSpec(
+    category="subject_fixation",
+    probes="primary salient subject",
+    fields={"primary_subject": SlotSpec(
+        name="primary_subject", cardinality="single", vocabulary="open", optional=False,
+        nested_fields=(_f("label", optional=False), _f("box", value_kind="bbox", optional=False)))},
+    system_prompt=(
+        "Identify the single most prominent (largest) shape — its color and bounding box. "
+        "Output ONLY raw JSON, no fences:\n"
+        '{"primary_subject": {"label": "<color>", "box": [x1, y1, x2, y2]}}\n{coord_hint}'
+    ),
+    user_prompt="Identify the primary subject and its box. Raw JSON only.",
+    metric="subject_fixation", gt_dataset="shapes_synth", coord_space=CoordSpace.NORM_0_1000,
+    max_new_tokens=128, license_note="synthetic (self-contained).",
+)
+
+_DATATYPE_UTIL = VisionTaskSpec(
+    category="data_type_utilization",
+    probes="parse a rendered data format into normalized JSON",
+    fields={
+        "data_type": _enum("data_type", _DATATYPE_VALUES),
+        "content": _f("content", optional=False, max_str_length=2048),
+    },
+    system_prompt=(
+        "You are shown a screenshot of structured data. Read it and re-serialize its contents "
+        "as JSON. Output ONLY a raw JSON object, no markdown fences:\n"
+        '{"data_type": "<the format>", "content": "<the data as a JSON string, e.g. '
+        '{\\"name\\": \\"Alice\\"}>"}'
+    ),
+    user_prompt="Read the data and output {data_type, content} as raw JSON.",
+    metric="datatype_util",
+    gt_dataset="datatype_synth",
+    max_new_tokens=512,
+    license_note="synthetic (self-contained).",
+)
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# THE REGISTRY
+# ──────────────────────────────────────────────────────────────────────────────
+
+_SEGMENTATION = VisionTaskSpec(
+    category="segmentation",
+    probes="instance segmentation as labeled polygons",
+    fields={
+        "masks": _list_of(
+            "masks",
+            _f("label", optional=False, max_str_length=64),
+            SlotSpec(name="polygon", cardinality="list", vocabulary="open",
+                     value_kind="number", max_items=512, optional=False),
+            max_items=32,
+        ),
+    },
+    system_prompt=(
+        "You are an instance segmenter. Trace the outline of every distinct object "
+        "as a closed polygon. Output ONLY a raw JSON object and NOTHING else — no prose, "
+        "no markdown code fences (do not wrap it in ```). It must match this shape exactly:\n"
+        '{"masks": [{"label": "<string>", "polygon": [x1, y1, x2, y2, x3, y3, ...]}]}\n'
+        "All x, y values are integers in 0..1000 relative to the image width and height. "
+        "Each polygon is a FLAT list of alternating x, y vertices — a closed shape with at "
+        "least 3 points / 6 numbers tracing the object boundary in order. This is a POLYGON, "
+        "NOT a 4-number bounding box."
+    ),
+    user_prompt="Segment every object in this image as a labeled polygon. Output only the raw JSON object.",
+    metric="segmentation",
+    coord_space=CoordSpace.NORM_0_1000,
+    gt_dataset="segmentation_synth",
+    max_new_tokens=768,
+    license_note="synthetic (self-contained).",
+)
+
+_OUTLINE = VisionTaskSpec(
+    category="outline_association",
+    probes="trace the main (largest) object's outline polygon + label it",
+    fields={
+        "outline": SlotSpec(name="outline", cardinality="list", vocabulary="open",
+                            value_kind="number", max_items=256, optional=False),
+        "label": _f("label", optional=False, max_str_length=64),
+    },
+    system_prompt=(
+        "You are an outline tracer. Find the SINGLE largest (most prominent) object in the "
+        "image and trace its outline as a closed polygon. Output ONLY a raw JSON object and "
+        "NOTHING else - no prose, no markdown code fences (do not wrap it in ```). It must "
+        "match this shape exactly:\n"
+        '{"outline": [x1, y1, x2, y2, x3, y3, ...], "label": "<string>"}\n'
+        "The outline is a flat list of alternating x, y vertex coordinates (at least 3 "
+        "vertices = 6 numbers), tracing the object boundary in order. All x, y values are "
+        "integers in 0..1000 relative to the image width and height. This is a POLYGON with "
+        "MANY points, NOT a 4-number bounding box."
+    ),
+    user_prompt="Trace the main object's outline and label it. Output only the raw JSON object.",
+    metric="outline_iou",
+    status="pilot",
+    coord_space=CoordSpace.NORM_0_1000,
+    gt_dataset="outline_synth",
+    max_new_tokens=640,
+    license_note="synthetic (self-contained).",
+)
+
+_GEO3D = VisionTaskSpec(
+    category="geometric_3d_object_id",
+    probes="3D object identification with 3D boxes (simplified ground-plane proxy)",
+    fields={
+        "objects": _list_of(
+            "objects",
+            _f("class", optional=False, max_str_length=64),
+            SlotSpec(name="bbox3d", cardinality="list", vocabulary="open",
+                     value_kind="number", max_items=7, optional=False),
+            _f("score", value_kind="number", optional=True, number_range=(0.0, 1.0)),
+            max_items=16,
+        ),
+    },
+    system_prompt=(
+        "You are a 3D object detector looking at a scene of colored boxes resting on a "
+        "ground plane. For each box report its class (its color) and a 3D bounding box. "
+        "Output ONLY a raw JSON object and NOTHING else - no prose, no markdown code "
+        "fences (do not wrap it in ```). It must match this shape exactly:\n"
+        '{"objects": [{"class": "<color>", "bbox3d": [x, y, z, w, h, l, yaw], '
+        '"score": <number 0..1>}]}\n'
+        "All coordinates are normalized to 0..1 of the scene: x is the left-right ground "
+        "position, z is the depth (0=near, 1=far), y is the height off the ground (0 on the "
+        "floor); w, h, l are the box width, height and length; yaw is the rotation in "
+        'radians. Use the key "bbox3d" with exactly seven numbers [x, y, z, w, h, l, yaw].'
+    ),
+    user_prompt="Identify the 3D boxes in this scene. Output only the raw JSON object.",
+    metric="iou3d",
+    status="pilot",
+    coord_space=CoordSpace.NORM_0_1,
+    gt_dataset="boxes3d_synth",
+    max_new_tokens=384,
+    license_note="synthetic (self-contained); simplified ground-plane 3D proxy.",
+)
+
+_CAMERA_ROT = VisionTaskSpec(
+    category="camera_rotational_offset",
+    probes="camera pose / rotation estimation from a 2D orientation cue",
+    fields={
+        "rotation": SlotSpec(name="rotation", cardinality="list", vocabulary="open",
+                             value_kind="number", max_items=3, optional=False),
+    },
+    system_prompt=(
+        "You estimate the camera's rotation relative to the scene. Output the three "
+        "Euler angles in DEGREES as [yaw, pitch, roll]. Output ONLY a raw JSON object and "
+        "NOTHING else — no prose, no explanation, and NO markdown code fences (do not wrap "
+        "it in ```). It must match this shape exactly:\n"
+        '{\"rotation\": [<yaw>, <pitch>, <roll>]}\n'
+        "Each angle is a number in degrees in the range -180..180. If an axis is not "
+        "discernible, report 0."
+    ),
+    user_prompt="Estimate the camera rotation [yaw, pitch, roll] in degrees. Output only the raw JSON object.",
+    metric="angular_error",
+    status="pilot",
+    gt_dataset="camera_rot_synth",
+    max_new_tokens=64,
+    license_note="synthetic (self-contained).",
+)
+
+_VQA = VisionTaskSpec(
+    category="vit_accuracy_to_prompt",
+    probes="grounded visual question answering",
+    fields={
+        "answer": _f("answer", optional=False, max_str_length=512),
+        "grounded_region": _f("grounded_region", value_kind="bbox", optional=True),
+    },
+    system_prompt=(
+        "You are a visual question answering engine. Answer the user's question about "
+        "the image as briefly as possible (a single word or short phrase). Optionally "
+        "ground your answer with the bounding box of the region you used. Output ONLY a "
+        "raw JSON object and NOTHING else — no prose, no explanation, and NO markdown "
+        "code fences (do not wrap it in ```). It must match this shape exactly:\n"
+        '{"answer": "<short answer>", "grounded_region": [x1, y1, x2, y2]}\n'
+        "{coord_hint} If you cannot or need not localize, omit grounded_region entirely."
+    ),
+    user_prompt="Answer the question about this image. Output only the raw JSON object.",
+    metric="vqa",
+    per_sample_prompt=True,
+    coord_space=CoordSpace.NORM_0_1000,
+    gt_dataset="gqa",
+    gt_split="validation",
+    max_new_tokens=128,
+    license_note="GQA / VQAv2: research use; images CC-BY (vary).",
+)
+
+_SEMANTIC = VisionTaskSpec(
+    category="semantic_association",
+    probes="semantic associations between entities as (a, relation, b) triples",
+    fields={
+        "associations": _list_of(
+            "associations",
+            _f("a", optional=False, max_str_length=64),
+            _enum("relation", ("left_of", "right_of", "near", "is_a", "related_to")),
+            _f("b", optional=False, max_str_length=64),
+            max_items=32,
+        ),
+    },
+    system_prompt=(
+        "You relate the entities in the image to each other as semantic association "
+        "triples. Each association links entity \"a\" to entity \"b\" by a relation. "
+        "For the colored shapes, the entities are the colors (red, green, blue) and "
+        "the shape type (circle). Allowed relations: left_of, right_of, near, is_a, "
+        "related_to. Output ONLY a raw JSON object and NOTHING else - no prose, no "
+        "explanation, and NO markdown code fences (do not wrap it in ```). It must "
+        "match this shape exactly:\n"
+        '{"associations": [{"a": "<entity>", "relation": '
+        '"<left_of|right_of|near|is_a|related_to>", "b": "<entity>"}]}'
+    ),
+    user_prompt="List the semantic associations between the entities. Output only the raw JSON object.",
+    metric="triples",
+    gt_dataset="semantic_synth",
+    max_new_tokens=384,
+    license_note="synthetic (self-contained).",
+)
+
+_STYLE = VisionTaskSpec(
+    category="style_structural_awareness",
+    probes="visual style + structural layout/symmetry, as a coarse closed-vocab triple",
+    fields={
+        "style": _enum("style", ("photo", "painting", "3d_render", "sketch", "anime", "other")),
+        "layout": _enum("layout", ("centered", "rule_of_thirds", "symmetric", "scattered", "unknown")),
+        "symmetry": _enum("symmetry", ("horizontal", "vertical", "radial", "none")),
+    },
+    system_prompt=(
+        "You judge the VISUAL STYLE and STRUCTURE of an image. Pick exactly one value "
+        "from each closed vocabulary. Output ONLY a raw JSON object and NOTHING else — no "
+        "prose, no explanation, and NO markdown code fences (do not wrap it in ```). "
+        "It must match this shape exactly:\n"
+        '{"style": "<one of: photo, painting, 3d_render, sketch, anime, other>", '
+        '"layout": "<one of: centered, rule_of_thirds, symmetric, scattered, unknown>", '
+        '"symmetry": "<one of: horizontal, vertical, radial, none>"}'
+    ),
+    user_prompt="Classify the visual style and structure. Output only the raw JSON object.",
+    metric="style",
+    status="pilot",
+    gt_dataset="style_synth",
+    max_new_tokens=96,
+    license_note="synthetic (self-contained).",
+)
+
+
+VISION_TASK_REGISTRY: dict[str, VisionTaskSpec] = {
+    t.category: t for t in [_CLASSIFICATION, _BBOX, _OCR, _DATATYPE_DIFF, _DATATYPE_UTIL,
+                            _SPATIAL, _DEPTH, _SUBJECT,
+                            _SEGMENTATION, _OUTLINE, _GEO3D, _CAMERA_ROT, _VQA, _SEMANTIC, _STYLE]
+}
+
+
+def get_task(category: str) -> VisionTaskSpec:
+    if category not in VISION_TASK_REGISTRY:
+        raise KeyError(f"unknown vision category: {category!r}. known: {list(VISION_TASK_REGISTRY)}")
+    return VISION_TASK_REGISTRY[category]
+
+
+def category_names() -> list[str]:
+    return list(VISION_TASK_REGISTRY.keys())
+
+
+def pilot_categories() -> list[str]:
+    return [c for c, t in VISION_TASK_REGISTRY.items() if t.status == "pilot"]
+
+
+def resolved_system_prompt(spec: VisionTaskSpec) -> str:
+    """Fill the {coord_hint} placeholder using the task's coord_space."""
+    if "{coord_hint}" in spec.system_prompt:
+        from .coords import prompt_hint_for
+        return spec.system_prompt.replace("{coord_hint}", prompt_hint_for(spec.coord_space))
+    return spec.system_prompt

qwen_test_runner/vision/throughput.py

@@ -0,0 +1,56 @@
+"""
+throughput.py — Pure GPU-hours / cost model (CPU-testable, no torch).
+
+The labeler verdict trades accuracy against throughput: a model that is 94% as
+good but 3× faster is the better choice for labeling a million images. This
+module converts measured decode speed into samples/hour and GPU-hours/$ per
+million labels.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class ThroughputEstimate:
+    model: str
+    tokens_per_sec: float
+    mean_output_tokens: float
+    samples_per_hour: float
+    gpu_hours_per_million: float
+    est_cost_per_million_usd: float
+
+
+def estimate(model: str, tokens_per_sec: float, mean_output_tokens: float,
+             prefill_overhead_s: float = 0.0, gpu_hourly_rate: float = 2.0) -> ThroughputEstimate:
+    """samples_per_hour = 3600 / (prefill + output_tokens / tok_per_sec).
+
+    `prefill_overhead_s` is the per-sample vision-encoder + image-token cost
+    (measured during the run, not guessed). `gpu_hourly_rate` is a config rate
+    printed alongside the result so the dollar figure is transparent.
+    """
+    if tokens_per_sec <= 0 or mean_output_tokens <= 0:
+        return ThroughputEstimate(model, tokens_per_sec, mean_output_tokens, 0.0, float("inf"), float("inf"))
+    per_sample_s = prefill_overhead_s + mean_output_tokens / tokens_per_sec
+    samples_per_hour = 3600.0 / per_sample_s
+    gpu_hours_per_million = 1_000_000.0 / samples_per_hour
+    cost = gpu_hours_per_million * gpu_hourly_rate
+    return ThroughputEstimate(
+        model=model, tokens_per_sec=tokens_per_sec, mean_output_tokens=mean_output_tokens,
+        samples_per_hour=samples_per_hour, gpu_hours_per_million=gpu_hours_per_million,
+        est_cost_per_million_usd=cost,
+    )
+
+
+def fleet_score(labeler: float, samples_per_hour: float, saturate_at: float = 50_000.0) -> float:
+    """Fold throughput into the labeler score for the 'label 1M images' goal.
+
+    Throughput weight saturates (diminishing returns past `saturate_at`), so a
+    tiny-but-inaccurate model can't win on speed alone.
+    """
+    if labeler is None:
+        return 0.0
+    import math
+    w = math.log1p(max(0.0, samples_per_hour)) / math.log1p(saturate_at)
+    return labeler * min(1.0, w)

requirements.txt

@@ -0,0 +1,19 @@
+# ZeroGPU provides torch (2.8+) and CUDA at runtime — do NOT pin or reinstall
+# torch here (a mismatched pin fights the platform image).
+#
+# transformers is pinned to a RELEASED wheel, NOT the package's
+# `transformers @ git+…@main` pin (that pin is the Space-build landmine).
+# qwen3_5 ships in released transformers now (the base model has 12M+ downloads
+# and WebGPU spaces); the Colab path already runs on transformers>=4.50.
+
+spaces
+gradio>=4.44
+transformers>=4.50
+accelerate>=1.0
+easyocr
+pydantic>=2.0
+pillow
+numpy
+huggingface_hub>=0.25
+datasets>=2.20
+safetensors