Upload folder using huggingface_hub

DEBUG_DETERMINISM_PLAN.md

@@ -0,0 +1,208 @@
+# Debug Plan: Deterministic Labels and Invariant Scores
+
+## Executive Summary
+The observed behavior is not one single bug. It is a combination of:
+1. Label leakage paths in prediction logic.
+2. Difficulty-band score compression masking variance.
+3. Data-source mode differences (manifest vs task registry).
+4. Small/imbalanced episode pools causing repeated clip patterns.
+5. No cross-episode learning memory.
+
+This plan is designed to debug in a strict order so we can separate each mechanism and avoid false conclusions.
+
+## Implementation Status (2026-04-06)
+Completed in code:
+1. Removed expected-label leakage from fallback prediction path.
+2. Redacted `expected_label` from model-facing prompt metadata.
+3. Added structured per-step execution trace with score components and expected-label type/value.
+4. Added raw vs calibrated score telemetry in environment observations and dashboard reward rendering.
+5. Fixed expected-label integrity: derive label when value is missing, null, or invalid.
+6. Added `CLIP_CORPUS_SOURCE` mode toggle (`auto`, `manifest`, `task_registry`) for source-controlled debugging.
+
+Validated:
+1. Focused regression suites pass for environment, inference, auto-execution, and grader.
+2. Runtime sanity checks show `none_expected=0` across current manifest pools.
+
+Still pending from this plan:
+1. Full `tests/test_server_routes.py` alignment with newer one-click strategic flow and revised dashboard copy.
+
+## 20-Episode Matrix Snapshot (Fallback mode)
+
+Source mode: `manifest`
+- `task_easy`: raw_mean=0.9740 raw_var=0.000024 calibrated_mean=0.3117 calibrated_var=0.000002 match_rate=1.000 repeat_ratio=0.600 correction_rate=1.000
+- `task_medium`: raw_mean=0.8385 raw_var=0.005583 calibrated_mean=0.6083 calibrated_var=0.000572 match_rate=0.680 repeat_ratio=0.000 correction_rate=0.700
+- `task_hard`: raw_mean=0.8065 raw_var=0.001893 calibrated_mean=0.9381 calibrated_var=0.000194 match_rate=1.000 repeat_ratio=0.000 correction_rate=0.000
+
+Source mode: `task_registry`
+- `task_easy`: raw_mean=0.9400 raw_var=0.000000 calibrated_mean=0.3008 calibrated_var=0.000000 match_rate=1.000 repeat_ratio=0.000 correction_rate=1.000
+- `task_medium`: raw_mean=0.8500 raw_var=0.000000 calibrated_mean=0.6120 calibrated_var=0.000000 match_rate=0.800 repeat_ratio=0.000 correction_rate=0.787
+- `task_hard`: raw_mean=0.7500 raw_var=0.000000 calibrated_mean=0.9200 calibrated_var=0.000000 match_rate=0.600 repeat_ratio=0.000 correction_rate=0.000
+
+Interpretation:
+- Easy manifest behavior still shows repeat ratio because easy manifest pool has 2 clips across 5 episode steps.
+- Score behavior is now fully explainable from raw components and calibration, with per-step trace available in `info.execution_trace`.
+
+## Verified Findings
+
+### 1) Prediction leakage path exists in code
+- Fallback action can copy expected label directly:
+  - `inference.py:116`
+- Prompt includes full clip metadata object (includes expected_label when present):
+  - `inference.py:148`
+- Clip metadata schema includes expected_label field:
+  - `clip_quality_env/models.py:67`
+
+Impact:
+- In environments where expected labels are populated, predictions can trivially mirror labels.
+
+### 2) Difficulty calibration compresses scores into narrow task bands
+- Bands:
+  - `clip_quality_env/difficulty.py:14`
+- Calibration function:
+  - `clip_quality_env/difficulty.py:34`
+
+Formula:
+- `calibrated = band_min + raw * (band_max - band_min)`
+- Band width is 0.32 for every task, which compresses raw score variance.
+
+### 3) Runtime data source is manifest, not task_registry (in this workspace)
+Measured via reset observations:
+- `task_easy`: source `manifest:easy`, corpus_size `2`
+- `task_medium`: source `manifest:medium`, corpus_size `11`
+- `task_hard`: source `manifest:hard`, corpus_size `7`
+
+Why this matters:
+- Easy episodes require 5 steps but only 2 unique easy clips exist, so clips repeat.
+- Repetition can make aggregate rewards appear deterministic or nearly deterministic.
+
+### 4) expected_label is often None in manifest clips
+Measured in current runtime:
+- easy manifest clips show `expected_label=None` values.
+
+Root cause in loading path:
+- Loader keeps `expected_label` key with None value.
+- Environment only derives expected label when key is missing, not when key is present but None:
+  - Derive check location: `clip_quality_env/env.py` (`if "expected_label" not in clip` logic in corpus load).
+
+Impact:
+- `EpisodeHistoryItem.expected_label` may become string `"None"`.
+- Label match diagnostics can be misleading.
+- Fallback leakage via expected_label is inactive for these rows (falls back to heuristic), but still dangerous in non-manifest mode.
+
+### 5) Episode memory resets each run
+- `episode_history` is reset in `clip_quality_env/env.py:378`.
+
+Impact:
+- No cross-episode policy improvement.
+- Any "learning" is at most within one 5-step episode prompt context.
+
+## Quick Reproduction Evidence (Fallback mode)
+Command used:
+- `unset HF_TOKEN OPENAI_API_KEY` then repeated `run_baseline(task=...)`
+
+Observed behavior in this workspace:
+- `task_easy`: ~0.310 to ~0.314 reward average, fallback mode always.
+- `task_medium`: ~0.570 to ~0.628, fallback mode always.
+- `task_hard`: ~0.930 to ~0.936, fallback mode always.
+- Final-step rewards include values like `0.952` and occasionally `1.000` on hard clips.
+
+Interpretation:
+- Not perfectly constant per run in this workspace, but strongly constrained by:
+  - deterministic heuristics,
+  - repeated/limited clip sets,
+  - compressed calibration bands.
+
+## Prioritized Debug Sequence
+
+### Phase 1: Instrumentation Baseline (must do first)
+Add temporary logs (or structured debug payload) for each step:
+- mode: llm/fallback
+- clip_id
+- expected_label value and type
+- predicted label
+- format/label/reasoning score
+- raw_total before calibration
+- calibrated_total after calibration
+
+Acceptance for Phase 1:
+- Can explain each final score from recorded components and calibration math.
+
+### Phase 2: Isolate leakage mechanisms
+Run four controlled scenarios:
+1. Current behavior, fallback mode.
+2. Fallback mode with expected_label removed from fallback path.
+3. LLM mode with expected_label redacted from prompt payload.
+4. LLM mode with both fallback and prompt redaction.
+
+Acceptance:
+- Prediction-match rate must drop from near-trivial baseline when leakage paths are removed.
+
+### Phase 3: Data-source validation
+Run with both sources explicitly:
+1. Manifest enabled (current default)
+2. Manifest disabled / task_registry forced
+
+Checks:
+- corpus_size per difficulty
+- expected_label completeness
+- repeated clip ratio per episode
+
+Acceptance:
+- Source-dependent behavior is quantified and documented.
+
+### Phase 4: expected_label integrity fix validation
+Validate expected-label derivation policy:
+- If expected_label missing OR null, derive it before episode starts.
+
+Acceptance:
+- No history rows with expected_label equal to `"None"`.
+- Label-match telemetry becomes meaningful.
+
+### Phase 5: Score compression interpretation
+Report both:
+- raw_total (unbanded)
+- calibrated_total (banded)
+
+Acceptance:
+- Team can distinguish model/agent quality changes from banding artifacts.
+
+### Phase 6: Learning-signal evaluation
+Measure in-episode adaptation quality:
+- Compare step N vs step N+1 after low reward.
+- Compute correction rate after negative feedback.
+
+Measure cross-episode adaptation:
+- Verify no carryover after reset.
+
+Acceptance:
+- Explicit statement of whether system is truly learning or only re-prompting.
+
+## Hypotheses Ranked by Confidence
+1. High: Leakage paths can force label correctness in non-manifest mode.
+2. High: Band calibration masks meaningful raw variance.
+3. High: Easy pool size=2 with 5-step episodes induces repeated pattern behavior.
+4. High: Null expected_label handling produces misleading diagnostics.
+5. Medium: User-observed exact constants likely depend on source mode and run conditions.
+
+## Patch Design (after debug confirms)
+Do not apply until Phase 1-4 logs are captured.
+1. Remove expected_label from model-facing prompt payload.
+2. Change fallback label selection to heuristic-only.
+3. Derive expected label when value is missing OR null.
+4. Expose raw_total alongside calibrated_total in UI and logs.
+5. Add optional config to force task_registry source for reproducible testing.
+6. Expand easy manifest pool or reduce episode steps when pool is tiny.
+
+## Validation Matrix
+For each task and source mode, run 20 episodes and report:
+- mean/variance of raw_total
+- mean/variance of calibrated_total
+- label match rate
+- fallback frequency
+- repeated clip ratio
+- correction rate after low reward steps
+
+Success criteria:
+- Scores are explainable and not suspiciously fixed.
+- Predicted labels are no longer trivially leaked.
+- Debug telemetry clearly separates calibration effects from model behavior.

clip_quality_env/env.py

@@ -18,6 +18,14 @@ from server.tasks import TASK_REGISTRY
 
 EPISODE_STEPS = 5
 DEFAULT_REAL_CLIPS_MANIFEST = "data/real_clips_manifest.jsonl"
+VALID_LABELS = {"KEEP", "BORDERLINE", "REJECT"}
+
+
+def _normalize_corpus_mode(value: str | None) -> str:
+    mode = str(value or "auto").strip().lower()
+    if mode in {"auto", "manifest", "task_registry"}:
+        return mode
+    return "auto"
 
 
 @dataclass
@@ -50,13 +58,23 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
             "format_score": 0.0,
             "label_score": 0.0,
             "reasoning_score": 0.0,
+            "raw_total": 0.0,
+            "calibrated_total": 0.0,
             "total_reward": 0.0,
         }
         self._corpus_source = "task_registry"
+        self._corpus_mode = _normalize_corpus_mode(os.environ.get("CLIP_CORPUS_SOURCE"))
         self._manifest_warning = ""
         self._manifest_path = os.environ.get("REAL_CLIPS_MANIFEST", DEFAULT_REAL_CLIPS_MANIFEST)
         self._real_clip_pools = self._load_real_clip_pools()
 
+    def _resolve_expected_label(self, clip: dict[str, Any]) -> str:
+        raw_expected = clip.get("expected_label")
+        normalized = str(raw_expected).strip().upper() if raw_expected is not None else ""
+        if normalized in VALID_LABELS:
+            return normalized
+        return self._rubric.derive_label(clip)
+
     def _load_real_clip_pools(self) -> dict[str, list[dict[str, Any]]]:
         if not os.path.exists(self._manifest_path):
             self._manifest_warning = (
@@ -85,8 +103,23 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
         task = TASK_REGISTRY[task_id]
         difficulty = str(task.get("difficulty", "")).lower()
         corpus: list[dict[str, Any]]
+        manifest_available = bool(difficulty and self._real_clip_pools.get(difficulty))
 
-        if difficulty and self._real_clip_pools.get(difficulty):
+        if self._corpus_mode == "task_registry":
+            corpus = copy.deepcopy(task.get("data_corpus", []))
+            self._corpus_source = f"task_registry:{task_id}"
+        elif self._corpus_mode == "manifest" and manifest_available:
+            corpus = copy.deepcopy(self._real_clip_pools[difficulty])
+            self._corpus_source = f"manifest:{difficulty}"
+        elif self._corpus_mode == "manifest":
+            corpus = copy.deepcopy(task.get("data_corpus", []))
+            self._corpus_source = f"task_registry:{task_id}"
+            if not self._manifest_warning:
+                self._manifest_warning = (
+                    f"Manifest source requested but unavailable for difficulty '{difficulty}'; "
+                    "using static task corpora."
+                )
+        elif manifest_available:
             corpus = copy.deepcopy(self._real_clip_pools[difficulty])
             self._corpus_source = f"manifest:{difficulty}"
         else:
@@ -98,8 +131,7 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
 
         for clip in corpus:
             clip.setdefault("clip_id", clip.get("id", str(uuid.uuid4())))
-            if "expected_label" not in clip:
-                clip["expected_label"] = self._rubric.derive_label(clip)
+            clip["expected_label"] = self._resolve_expected_label(clip)
             clip["review_status"] = str(clip.get("review_status", "pending")).lower()
 
         corpus.sort(key=lambda item: str(item.get("clip_id", item.get("id", ""))))
@@ -269,6 +301,8 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
                 format_score=h.format_score,
                 label_score=h.label_score,
                 reasoning_score=h.reasoning_score,
+                raw_total=h.raw_total,
+                calibrated_total=h.calibrated_total,
                 feedback_message=h.feedback_message,
                 missing_features=list(h.missing_features),
                 hallucinated_features=list(h.hallucinated_features),
@@ -314,12 +348,15 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
                 "format_score": float(self._last_reward_breakdown["format_score"]),
                 "label_score": float(self._last_reward_breakdown["label_score"]),
                 "reasoning_score": float(self._last_reward_breakdown["reasoning_score"]),
+                "raw_total": float(self._last_reward_breakdown["raw_total"]),
+                "calibrated_total": float(self._last_reward_breakdown["calibrated_total"]),
                 "reward_total": float(self._last_reward_breakdown["total_reward"]),
                 "session_history": session_history,
                 "processed_clips": processed_clips,
                 "wrong_label_count": wrong_labels,
                 "label_accuracy": float(label_accuracy),
                 "corpus_source": self._corpus_source,
+                "corpus_mode": self._corpus_mode,
             },
         )
 
@@ -360,6 +397,8 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
             "format_score": 0.0,
             "label_score": 0.0,
             "reasoning_score": 0.0,
+            "raw_total": 0.0,
+            "calibrated_total": 0.0,
             "total_reward": 0.0,
         }
         self._state = State(
@@ -408,11 +447,18 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
             reward_obj=reward_obj,
             difficulty=current.difficulty,
         )
+        raw_total = (
+            float(reward_obj.format_score)
+            + float(reward_obj.label_score)
+            + float(reward_obj.reasoning_score)
+        )
         reward = float(reward_obj.total)
         self._last_reward_breakdown = {
             "format_score": float(reward_obj.format_score),
             "label_score": float(reward_obj.label_score),
             "reasoning_score": float(reward_obj.reasoning_score),
+            "raw_total": float(raw_total),
+            "calibrated_total": float(reward),
             "total_reward": float(reward),
         }
         self._state.total_reward += reward
@@ -438,17 +484,20 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
                     break
 
         self._state.actions_taken.append(action_label)
+        expected_label = self._resolve_expected_label(clip)
         self._state.episode_history.append(
             EpisodeHistoryItem(
                 step=current_index + 1,
                 difficulty=current.difficulty,
                 clip_id=str(clip.get("clip_id", "")),
                 label=action_label,
-                expected_label=str(clip.get("expected_label", "")),
+                expected_label=expected_label,
                 reward=reward,
                 format_score=float(reward_obj.format_score),
                 label_score=float(reward_obj.label_score),
                 reasoning_score=float(reward_obj.reasoning_score),
+                raw_total=float(raw_total),
+                calibrated_total=float(reward),
                 feedback_message=str(feedback.get("feedback_message", "")),
                 missing_features=list(feedback.get("missing_features", [])),
                 hallucinated_features=list(feedback.get("hallucinated_features", [])),
@@ -466,7 +515,7 @@ class ClipQualityEnvironment(Environment[Action, Observation, State]):
                 "clip": self._episode_plan[-1].clip,
                 "action": action_obj.model_dump(),
                 "reward": reward,
-                "expected_label": self._episode_plan[-1].clip.get("expected_label"),
+                "expected_label": self._resolve_expected_label(self._episode_plan[-1].clip),
             }
             try:
                 gt_promoted = self._gt_store.try_promote(step3_result, episode=self._state.episode_count)

clip_quality_env/models.py

@@ -77,6 +77,8 @@ class HistoryItem(BaseModel):
     format_score: float = 0.0
     label_score: float = 0.0
     reasoning_score: float = 0.0
+    raw_total: float = 0.0
+    calibrated_total: float = 0.0
     feedback_message: str = ""
     missing_features: List[str] = Field(default_factory=list)
     hallucinated_features: List[str] = Field(default_factory=list)
@@ -96,6 +98,8 @@ class EpisodeHistoryItem(BaseModel):
     format_score: float = 0.0
     label_score: float = 0.0
     reasoning_score: float = 0.0
+    raw_total: float = 0.0
+    calibrated_total: float = 0.0
     feedback_message: str = ""
     missing_features: List[str] = Field(default_factory=list)
     hallucinated_features: List[str] = Field(default_factory=list)

inference.py

@@ -113,7 +113,7 @@ class ClipQualityAgent:
         return "KEEP" if keep_signals >= 4 else "BORDERLINE"
 
     def _fallback_action(self, clip: Dict[str, Any]) -> Dict[str, Any]:
-        label = _normalize_label(clip.get("expected_label"), fallback=self._heuristic_label(clip))
+        label = self._heuristic_label(clip)
         confidence = 0.82 if label != "BORDERLINE" else 0.68
         reasoning = (
             f"{label} based on face_confidence={clip.get('face_confidence')}, "
@@ -138,6 +138,8 @@ class ClipQualityAgent:
 
     def act(self, task_id: str, obs: Dict, strategy_context: str | None = None) -> Dict:
         clip = obs.get("clip_metadata", {})
+        clip_for_prompt = dict(clip)
+        clip_for_prompt.pop("expected_label", None)
         rubric = obs.get("rubric_summary", "")
         history = self._get_history(obs)
         strategy = str(strategy_context or "").strip()
@@ -145,7 +147,7 @@ class ClipQualityAgent:
         prompt = (
             f"Task: {task_id}\n"
             f"Rubric:\n{rubric}\n"
-            f"Clip metadata:\n{json.dumps(clip, indent=2)}\n"
+            f"Clip metadata:\n{json.dumps(clip_for_prompt, indent=2)}\n"
             f"{history}\n"
             f"{strategy_block}\n"
             "Return JSON with keys: "
@@ -182,17 +184,45 @@ def execute_auto_episode(
 
     agent, mode, warning = load_agent_with_fallback()
     executed_steps = 0
+    execution_trace: list[dict[str, Any]] = []
     while not bool(obs.done) and executed_steps < int(obs.max_steps):
-        action_dict = agent.act(task_id, obs.model_dump(), strategy_context=strategy_context)
+        obs_payload = obs.model_dump()
+        clip_metadata = dict(obs_payload.get("clip_metadata", {}))
+        expected_label_value = clip_metadata.get("expected_label")
+        expected_label_type = type(expected_label_value).__name__
+        action_dict = agent.act(task_id, obs_payload, strategy_context=strategy_context)
         action_dict.setdefault("clip_id", obs.clip_metadata.clip_id)
+        predicted_label = _normalize_label(action_dict.get("label"), fallback="BORDERLINE")
         action = Action.model_validate(action_dict)
         obs = env.step(action)
+        info = obs.info if isinstance(obs.info, dict) else {}
+        format_score = float(info.get("format_score", 0.0))
+        label_score = float(info.get("label_score", 0.0))
+        reasoning_score = float(info.get("reasoning_score", 0.0))
+        raw_total = float(info.get("raw_total", format_score + label_score + reasoning_score))
+        calibrated_total = float(info.get("calibrated_total", info.get("reward_total", obs.reward)))
+        execution_trace.append(
+            {
+                "step": int(executed_steps + 1),
+                "mode": mode,
+                "clip_id": str(clip_metadata.get("clip_id", "")),
+                "expected_label_value": expected_label_value,
+                "expected_label_type": expected_label_type,
+                "predicted_label": predicted_label,
+                "format_score": format_score,
+                "label_score": label_score,
+                "reasoning_score": reasoning_score,
+                "raw_total": raw_total,
+                "calibrated_total": calibrated_total,
+            }
+        )
         executed_steps += 1
 
     payload = obs.model_dump()
     payload.setdefault("info", {})
     payload["info"]["execution_mode"] = mode
     payload["info"]["executed_steps"] = executed_steps
+    payload["info"]["execution_trace"] = execution_trace
     if warning:
         payload["info"]["warning"] = warning
     return payload
@@ -216,8 +246,15 @@ def run_episode(task_id: str, client: OpenAI | None, model_name: str) -> Dict:
         reward = float(obs.reward)
         done = bool(obs.done)
         rewards.append(reward)
+        raw_total = float(obs.info.get("raw_total", 0.0))
+        calibrated_total = float(obs.info.get("calibrated_total", reward))
         action_name = str(action.label)
-        print(f"[STEP] step={step_num} label={action_name} reward={reward:.2f} done={str(done).lower()} error=null", flush=True)
+        print(
+            f"[STEP] step={step_num} label={action_name} reward={reward:.2f} "
+            f"raw_total={raw_total:.2f} calibrated_total={calibrated_total:.2f} "
+            f"done={str(done).lower()} error=null",
+            flush=True,
+        )
         if done:
             break
 

server/app.py

@@ -337,8 +337,9 @@ def _reward_breakdown_markdown(obs: dict[str, Any], initialized: bool = False) -
     format_score = float(info.get("format_score", 0.0))
     label_score = float(info.get("label_score", 0.0))
     reasoning_score = float(info.get("reasoning_score", 0.0))
+    raw_total = float(info.get("raw_total", format_score + label_score + reasoning_score))
+    calibrated_total = float(info.get("calibrated_total", info.get("reward_total", obs.get("reward", 0.0))))
     total_reward = float(obs.get("reward", info.get("reward_total", 0.0)))
-    running_total = float(info.get("total_reward", 0.0))
     best_score = float(info.get("best_score", 0.0))
     has_submission = bool(obs.get("history"))
 
@@ -361,6 +362,8 @@ def _reward_breakdown_markdown(obs: dict[str, Any], initialized: bool = False) -
             title,
             subtitle,
             cards,
+            f"Raw Total (pre-calibration): **{raw_total:.3f}**",
+            f"Calibrated Total (difficulty-banded): **{calibrated_total:.3f}**",
             f"### **Total Reward:** <span style='color:{total_color};font-size:1.35rem;'>{total_reward:.3f}</span>",
             f"Current Best Score: **{best_score:.3f}**",
         ]

server/grader.py

@@ -63,7 +63,10 @@ def _normalize_reasoning(action_dict: dict[str, Any]) -> str:
 
 
 def _normalize_action(action_dict: dict[str, Any], clip: dict[str, Any]) -> Action:
-    fallback_label = _normalize_label(clip.get("expected_label"), fallback="BORDERLINE")
+    clip_id = str(clip.get("clip_id", ""))
+    fallback_label = _normalize_label(_GT.lookup(clip_id), fallback="")
+    if fallback_label not in _VALID_LABELS:
+        fallback_label = _normalize_label(_RUBRIC.derive_label(clip), fallback="BORDERLINE")
     payload = {
         "label": _normalize_label(
             action_dict.get("label")
@@ -79,14 +82,6 @@ def _normalize_action(action_dict: dict[str, Any], clip: dict[str, Any]) -> Acti
     return Action.model_validate(payload)
 
 
-def _expected_label_score(label: str, expected_label: str) -> float:
-    if label == expected_label:
-        return 0.60
-    if expected_label == "BORDERLINE" and label in {"KEEP", "REJECT"}:
-        return 0.25
-    return 0.0
-
-
 def _mentions_cue(reasoning: str, cue: str) -> bool:
     text_tokens = set(_TEXT_TOKEN_RE.findall(reasoning.lower()))
     cue_tokens = [token for token in _TEXT_TOKEN_RE.findall(cue.lower()) if len(token) >= 4]
@@ -128,10 +123,6 @@ def grade(action_dict: dict[str, Any], task_id: str, temperature: float = 0.0, s
         label_score = float(reward.label_score)
         reasoning_score = float(reward.reasoning_score)
 
-        expected_label = _normalize_label(clip.get("expected_label"), fallback="")
-        if expected_label and _GT.lookup(str(clip.get("clip_id", ""))) is None:
-            label_score = _expected_label_score(str(action.label), expected_label)
-
         reasoning_score = min(0.30, reasoning_score + _cue_bonus(str(action.reasoning), clip))
         task = TASK_REGISTRY.get(task_id, {})
         difficulty = normalize_difficulty(str(task.get("difficulty", "")))

tests/test_auto_execution.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import json
+from typing import Any
 
 import inference
 from clip_quality_env.grader import grade
@@ -53,6 +54,78 @@ def test_execute_auto_episode_includes_feedback_fields(monkeypatch):
     assert isinstance(first["feedback_message"], str)
 
 
+def test_execute_auto_episode_includes_execution_trace(monkeypatch):
+    monkeypatch.delenv("HF_TOKEN", raising=False)
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+
+    env = ClipQualityEnvironment()
+    env.reset(task_id="task_hard", seed=2026)
+
+    payload = inference.execute_auto_episode(env=env, task_id="task_hard", strategy_context="")
+    trace = payload["info"]["execution_trace"]
+
+    assert len(trace) == 5
+    first_step = trace[0]
+    assert first_step["mode"] in {"llm", "fallback"}
+    assert first_step["predicted_label"] in {"KEEP", "BORDERLINE", "REJECT"}
+    assert "expected_label_value" in first_step
+    assert "expected_label_type" in first_step
+    assert "format_score" in first_step
+    assert "label_score" in first_step
+    assert "reasoning_score" in first_step
+    assert "raw_total" in first_step
+    assert "calibrated_total" in first_step
+
+
+def test_fallback_action_does_not_copy_expected_label():
+    agent = inference.ClipQualityAgent(client=None, model="test")
+    clip = {
+        "clip_id": "clip_leak_check",
+        "expected_label": "KEEP",
+        "occlusion_present": True,
+        "face_confidence": 0.95,
+        "motion_score": 0.05,
+        "audio_snr_db": 30.0,
+        "lighting_uniformity": 0.9,
+    }
+
+    action = agent._fallback_action(clip)
+
+    assert action["label"] == "REJECT"
+
+
+def test_agent_prompt_redacts_expected_label():
+    class _CaptureAgent(inference.ClipQualityAgent):
+        def __init__(self):
+            super().__init__(client=None, model="dummy")
+            self.prompt = ""
+
+        def _call(self, prompt: str) -> dict[str, Any] | None:
+            self.prompt = prompt
+            return {
+                "label": "KEEP",
+                "reasoning": "face_confidence is high and motion_score is low.",
+                "confidence": 0.9,
+                "clip_id": "clip_123",
+            }
+
+    agent = _CaptureAgent()
+    obs = {
+        "clip_metadata": {
+            "clip_id": "clip_123",
+            "face_confidence": 0.91,
+            "motion_score": 0.11,
+            "expected_label": "REJECT",
+        },
+        "rubric_summary": "Rubric text",
+        "history": [],
+    }
+
+    _ = agent.act("task_easy", obs)
+
+    assert "expected_label" not in agent.prompt
+
+
 def test_hard_tradeoff_language_improves_reasoning_score(tmp_path):
     rubric = RubricState(path=str(tmp_path / "rubric.json"))
     gt = GTStore(seed_path="data/seed_gt.json", state_path=str(tmp_path / "ground_truth.json"))

tests/test_environment.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import json
 from statistics import mean
 
 from clip_quality_env.env import ClipQualityEnvironment
@@ -57,12 +58,16 @@ def test_environment_observation_exposes_reward_decomposition():
     assert reset_obs.info["format_score"] == 0.0
     assert reset_obs.info["label_score"] == 0.0
     assert reset_obs.info["reasoning_score"] == 0.0
+    assert reset_obs.info["raw_total"] == 0.0
+    assert reset_obs.info["calibrated_total"] == 0.0
     assert reset_obs.info["reward_total"] == 0.0
     assert reset_obs.info["total_reward"] == 0.0
     assert reset_obs.info["reward_breakdown"] == {
         "format_score": 0.0,
         "label_score": 0.0,
         "reasoning_score": 0.0,
+        "raw_total": 0.0,
+        "calibrated_total": 0.0,
         "total_reward": 0.0,
     }
 
@@ -78,16 +83,62 @@ def test_environment_observation_exposes_reward_decomposition():
     assert step_obs.info["format_score"] in {0.0, 0.1}
     assert step_obs.info["label_score"] in {0.0, 0.25, 0.6}
     assert 0.0 <= step_obs.info["reasoning_score"] <= 0.3
+    assert abs(
+        float(step_obs.info["raw_total"])
+        - (
+            float(step_obs.info["format_score"])
+            + float(step_obs.info["label_score"])
+            + float(step_obs.info["reasoning_score"])
+        )
+    ) < 1e-9
+    assert abs(float(step_obs.info["calibrated_total"]) - float(step_obs.reward)) < 1e-9
     assert abs(float(step_obs.info["reward_total"]) - float(step_obs.reward)) < 1e-9
     assert abs(float(step_obs.info["total_reward"]) - float(env.state.total_reward)) < 1e-9
     assert step_obs.info["reward_breakdown"] == {
         "format_score": float(step_obs.info["format_score"]),
         "label_score": float(step_obs.info["label_score"]),
         "reasoning_score": float(step_obs.info["reasoning_score"]),
+        "raw_total": float(step_obs.info["raw_total"]),
+        "calibrated_total": float(step_obs.info["calibrated_total"]),
         "total_reward": float(step_obs.info["reward_total"]),
     }
 
 
+def test_environment_derives_expected_label_when_manifest_value_is_null(monkeypatch, tmp_path):
+    manifest_path = tmp_path / "manifest_null_expected.jsonl"
+    manifest_row = {
+        "difficulty": "easy",
+        "clip_id": "manifest_null_expected",
+        "expected_label": None,
+        "face_confidence": 0.88,
+        "motion_score": 0.12,
+        "audio_snr_db": 24.0,
+        "lighting_uniformity": 0.8,
+        "duration_s": 8.0,
+    }
+    manifest_path.write_text(json.dumps(manifest_row) + "\n", encoding="utf-8")
+
+    monkeypatch.setenv("REAL_CLIPS_MANIFEST", str(manifest_path))
+    monkeypatch.setenv("CLIP_CORPUS_SOURCE", "manifest")
+
+    env = ClipQualityEnvironment()
+    obs = env.reset(task_id="task_easy", seed=2026)
+
+    assert obs.info["corpus_source"] == "manifest:easy"
+    assert str(obs.clip_metadata.expected_label).upper() in {"KEEP", "BORDERLINE", "REJECT"}
+    assert str(obs.clip_metadata.expected_label).upper() != "NONE"
+
+
+def test_environment_can_force_task_registry_source(monkeypatch):
+    monkeypatch.setenv("CLIP_CORPUS_SOURCE", "task_registry")
+
+    env = ClipQualityEnvironment()
+    obs = env.reset(task_id="task_easy", seed=2026)
+
+    assert obs.info["corpus_mode"] == "task_registry"
+    assert obs.info["corpus_source"] == "task_registry:task_easy"
+
+
 def test_environment_observation_includes_full_unsliced_corpus():
     env = ClipQualityEnvironment()
     obs = env.reset(task_id="task_medium")