Upload folder using huggingface_hub

BUILD_GUIDE_CLIP_ANALYZER_FIX.md

@@ -0,0 +1,329 @@
+# TalkingHeadBench — Clip Analyzer Fix Guide
+
+> **Problem:** The clip analyzer produces identical default gradings for all clips
+> because the primary extractor crashes on every clip, and the fallback extractor
+> uses hardcoded/miscalibrated values.
+>
+> **Root causes (in order of severity):**
+>
+> 1. **MediaPipe 0.10.33 removed the legacy `mp.solutions` API.** The primary
+>    clip extractor (`node4_clip_extractor.py`) calls `mp.solutions.face_mesh.FaceMesh(...)`.
+>    This throws `AttributeError: module 'mediapipe' has no attribute 'solutions'`.
+>    Every clip falls to the fallback extractor. The new API is
+>    `mediapipe.tasks.python.vision.FaceLandmarker`.
+>
+> 2. **`insightface` is not installed** (missing `onnxruntime` dependency). Even
+>    if MediaPipe is fixed, the ArcFace embedding block in the primary extractor will
+>    fail the `import insightface` check and produce `embeddings = []`, which means
+>    `face_embedding_variance = 1.0` and `identity_cosine_drift = 1.0` (worst case
+>    defaults).
+>
+> 3. **Fallback blur score calibration is 120× off.** The primary extractor divides
+>    by `_BLUR_CALIBRATION_CEILING = 0.12`. The fallback divides by `0.001`. This
+>    means the fallback produces blur scores ~120× smaller than the primary would
+>    for the same frame, making every clip look excessively blurry.
+>
+> 4. **Fallback extractor hardcodes non-diagnostic values:**
+>    - `lip_sync_confidence = 0.5` (not measured from actual lip movement)
+>    - `phoneme_sequence = []` (no phoneme analysis at all)
+>    - `phoneme_coverage_new = 0.0` (hardcoded)
+>    - `optical_flow_magnitude = 1.0` (hardcoded)
+>    - `occlusion_frames = 0` (hardcoded)
+>    - `blink_count = len(frames) // 60` (crude frame count estimate)
+>
+> 5. **`artifact_ingest.py` also uses the broken legacy API** in
+>    `_estimate_face_occupancy_ratio()`, which calls `mp.solutions.face_detection`.
+>    This silently falls back to returning `0.35` for every image.
+
+---
+
+## Task 1 — Migrate the primary clip extractor from legacy MediaPipe to the Tasks API
+
+**File:** `src/envs/subenv2/node4_clip_extractor.py`
+
+**Prompt:** Open `src/envs/subenv2/node4_clip_extractor.py`. The file imports
+`mediapipe as mp` and uses `mp.solutions.face_mesh.FaceMesh(...)` to create a
+face mesh detector (around line 280–286). MediaPipe 0.10.33 removed the legacy
+`mp.solutions` namespace entirely. The `mp` module now only exposes `Image`,
+`ImageFormat`, `tasks`, and `__version__`.
+
+The replacement API is `mediapipe.tasks.python.vision.FaceLandmarker`. It
+requires a model asset file (`.task` bundle) and uses a different calling
+convention:
+
+- Instead of `FaceMesh(static_image_mode=True, ...)`, you create a
+  `FaceLandmarkerOptions` with `running_mode=VisionRunningMode.IMAGE` and
+  `num_faces=1`.
+- Instead of `face_mesh.process(rgb_frame)`, you wrap the frame in a
+  `mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_array)` and call
+  `landmarker.detect(mp_image)`.
+- The result object has `face_landmarks` (list of lists of
+  `NormalizedLandmark`) instead of `multi_face_landmarks`.
+- Each `NormalizedLandmark` still has `.x`, `.y`, `.z` attributes.
+
+The model file can be downloaded from Google's MediaPipe model catalog. The
+agent should either download it at startup or bundle it. A common approach is
+to auto-download to a cache directory on first use.
+
+Rewrite the FaceMesh section of `extract_clip_signals()` to use the Tasks API.
+The landmark index constants (`_LEFT_EYE_IDX`, `_RIGHT_EYE_IDX`,
+`_UPPER_LIP_IDX`, `_LOWER_LIP_IDX`) reference the 468-point FaceMesh topology
+which is preserved in the Tasks API, so those constants stay the same.
+
+Make sure to handle the case where the model file is not available — fall back
+gracefully with a warning rather than crashing the entire extraction.
+
+Also update the `import mediapipe as mp` at the top of the file. The new
+imports will be from `mediapipe.tasks.python.vision` and `mediapipe.tasks.python`.
+
+---
+
+## Task 2 — Fix or replace ArcFace embedding extraction
+
+**File:** `src/envs/subenv2/node4_clip_extractor.py`
+
+**Prompt:** Open `src/envs/subenv2/node4_clip_extractor.py` and find the ArcFace
+embeddings section (around line 329–342). It does
+`import insightface` inside a try/except block. When insightface is missing
+(which it currently is — `onnxruntime` is not installed), `embeddings`
+stays empty, and the identity signals default to worst-case values
+(`face_embedding_variance = 1.0`, `identity_cosine_drift = 1.0`).
+
+There are two options:
+
+**Option A (recommended for deployment):** Replace the insightface dependency
+with a lighter alternative. Since we only need face embeddings for cosine
+distance between frames, we can use the MediaPipe FaceLandmarker itself — the
+normalized landmark coordinates form a sufficient proxy embedding for
+frame-to-frame identity comparison. Compute a flattened landmark vector per
+frame (468 landmarks × 3 coords = 1404-dimensional vector), then compute
+cosine drift and variance on those. This eliminates the insightface dependency
+entirely.
+
+**Option B (full fidelity):** Add `insightface` and `onnxruntime` to
+`requirements.txt` as core dependencies instead of optional. Then document that
+the Hugging Face Space Dockerfile must install them.
+
+Choose Option A for now since it avoids adding heavy dependencies and still
+produces meaningful identity signals. If Option A is chosen, also update the
+identity signal computation to use landmark-based embeddings instead of
+ArcFace. The cosine distance math stays the same.
+
+---
+
+## Task 3 — Fix the fallback blur score calibration
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** Open `server/artifact_ingest.py` and find the
+`_extract_clip_signals_fallback` function (starting around line 228). Look at
+line 259 where the blur score is calculated:
+
+The fallback uses a ceiling of `0.001` for normalizing the per-pixel Laplacian
+variance. The primary extractor in `node4_clip_extractor.py` uses
+`_BLUR_CALIBRATION_CEILING = 0.12`. This 120× difference means a clip that
+would score `blur_score = 0.70` (good) in the primary extractor scores
+`blur_score = 0.006` (terrible) in the fallback.
+
+Fix this by aligning the fallback's blur normalization ceiling with the primary
+extractor's constant. Import `_BLUR_CALIBRATION_CEILING` from the primary
+extractor module, or define the same constant locally. The formula should be:
+`clamp(lap_var_per_pixel / CEILING, 0.0, 1.0)` where `CEILING = 0.12`, not
+`0.001`.
+
+---
+
+## Task 4 — Improve the fallback extractor's lip sync proxy
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** In the same `_extract_clip_signals_fallback` function, the lip sync
+confidence is hardcoded to `0.5` (line 281). This gives every fallback clip an
+identical, meaningless score.
+
+The fallback extractor already reads frames and computes per-pixel differences.
+It can compute a simple lip sync proxy using the same approach as the primary
+extractor: look at the standard deviation of brightness in the lower-third of
+the face region across frames. Higher variance in the mouth area suggests
+active lip movement. This isn't as good as the MediaPipe-based lip opening
+measurement, but it's far better than a hardcoded constant.
+
+Alternatively, since MediaPipe FaceLandmarker is being fixed in Task 1, consider
+whether the fallback should also use the Tasks API for basic landmark detection
+(making it a "light" version of the primary extractor rather than pure
+OpenCV). If so, the fallback would only fall back to pure OpenCV when MediaPipe
+itself is unavailable.
+
+The key requirement is: different clips should produce different lip sync
+values. The fallback currently produces `0.5` for every clip, which defeats
+the purpose of per-clip analysis.
+
+---
+
+## Task 5 — Add optical flow computation to the fallback
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** In `_extract_clip_signals_fallback`, `optical_flow_magnitude` is
+hardcoded to `1.0` (line 279). The fallback already reads all frames into
+memory, so computing Farneback optical flow is straightforward — it's a pure
+OpenCV operation with no external dependencies.
+
+The primary extractor computes face-region vs background-region optical flow
+ratio (lines 399–432 in `node4_clip_extractor.py`). The fallback can use a
+simplified version: compute optical flow on the full frame and use the
+central 40% as the face region proxy (exactly as the primary extractor's
+own fallback face mask does when landmarks are unavailable).
+
+This gives each clip a distinct motion signature instead of a uniform `1.0`.
+
+---
+
+## Task 6 — Add blink detection to the fallback using the eye region
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** In `_extract_clip_signals_fallback`, `blink_count` is estimated as
+`max(0, len(frames) // 60)` (line 280) — a crude frame-count-based guess.
+A 120-frame clip gets `blink_count = 2` regardless of whether anyone blinked.
+
+A simple improvement: compute eye-region brightness variance across frames.
+Blinks cause momentary brightness drops in the eye region. The fallback can
+crop the upper-quarter of the central face region and track brightness drops
+below a threshold across consecutive frames. Count the number of brightness
+drops as blink count.
+
+This is a rough proxy but produces clip-specific values rather than a
+frame-count formula.
+
+---
+
+## Task 7 — Fix `_estimate_face_occupancy_ratio` legacy API usage
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** Open `server/artifact_ingest.py` and find `_estimate_face_occupancy_ratio`
+(around line 89). It uses `mp.solutions.face_detection.FaceDetection(...)` which
+is the same broken legacy MediaPipe API. This silently catches the exception
+and returns the hardcoded fallback of `0.35` for every image.
+
+Migrate this to the MediaPipe Tasks API. The replacement is
+`mediapipe.tasks.python.vision.FaceDetector`. The calling convention is
+similar to the FaceLandmarker migration in Task 1: create options, create a
+detector, wrap the image in `mp.Image`, call `detect()`.
+
+If the model file is unavailable, fall back to a reasonable OpenCV-based
+face detection (e.g., Haar cascades or DNN face detector), or at minimum
+compute a face occupancy estimate from skin-color segmentation.
+
+---
+
+## Task 8 — Add occlusion frame detection to the fallback
+
+**File:** `server/artifact_ingest.py`
+
+**Prompt:** In `_extract_clip_signals_fallback`, `occlusion_frames` is hardcoded
+to `0` (line 286). In the primary extractor, a frame is marked as occluded
+when MediaPipe FaceMesh fails to detect a face. The fallback can approximate
+this with OpenCV's face detection: run a Haar cascade face detector on each
+frame and count frames where no face is found. This gives a real occlusion
+signal instead of always-zero.
+
+Alternatively, use a simpler heuristic: if the central region's skin-tone
+pixel ratio drops below a threshold in a frame (compared to the median across
+all frames), count it as potentially occluded.
+
+---
+
+## Task 9 — Update requirements.txt
+
+**File:** `requirements.txt`
+
+**Prompt:** Open `requirements.txt`. Currently `insightface` and `transformers`
+are listed as optional comments. Based on the decisions made in Task 2:
+
+If Option A was chosen (landmark-based embeddings), add a comment explaining
+that insightface is no longer needed and landmark embeddings are used instead.
+
+If Option B was chosen (install insightface), move `insightface` and
+`onnxruntime` from optional comments to required dependencies.
+
+Either way, verify that `mediapipe>=0.10.33` is pinned correctly since the
+Tasks API is required.
+
+Also add any model asset download instructions if the MediaPipe Tasks API
+requires a `.task` model file.
+
+---
+
+## Task 10 — Ensure the primary extractor gracefully degrades
+
+**File:** `src/envs/subenv2/node4_clip_extractor.py`
+
+**Prompt:** After the MediaPipe migration in Task 1, the primary extractor
+should gracefully handle missing model files or initialization failures. If
+the FaceLandmarker model file is not found, the function should log a warning
+and fall through to the fallback extractor in `artifact_ingest.py` rather
+than crashing with an unhandled exception.
+
+Verify the error handling path: `_extract_clip_signal_observations` in
+`artifact_ingest.py` wraps `extract_clip_signals()` in a broad `except
+Exception` that triggers the fallback. This safety net is correct, but the
+primary extractor should also produce a clear log message explaining WHY it
+failed (e.g., "FaceLandmarker model file not found at {path}") so debugging
+is straightforward.
+
+---
+
+## Task 11 — Verify calibration alignment between primary and fallback
+
+**Prompt:** After fixing both extractors, run a quick calibration test. Take
+one clip and extract signals using both the primary extractor (with the Tasks
+API) and the fallback extractor (with corrected constants). Compare the outputs
+side by side:
+
+- `blur_score` should be in the same ballpark (±0.15) between both extractors
+- `identity_cosine_drift` should be directionally consistent
+- `frame_difference_mean` should be identical (both use the same OpenCV code)
+- `lip_sync_confidence` will differ (primary uses landmark-based, fallback uses
+  proxy) but both should be non-zero and clip-specific
+
+If the blur scores still diverge wildly, the calibration ceiling needs further
+tuning. The ceiling `0.12` was derived from a specific test set and may need
+adjustment.
+
+---
+
+## Task 12 — Verify with the same 12-clip dataset
+
+**Prompt:** After all fixes, re-run the same 12-clip ingestion that produced
+the original "all clips needs-fix" report. Verify:
+
+1. The primary extractor does NOT crash (no fallback used for any clip, unless
+   the model file is genuinely missing)
+2. `clip_extractor_fallback_count` is `0` in the ingestion metadata
+3. Each clip has a distinct `blur_score` (not all the same)
+4. Each clip has a distinct `lip_sync_confidence` (not all 0.5)
+5. Each clip has a distinct `identity_cosine_drift` (not all the same)
+6. The LLM report differentiates between clips — some get "good" or
+   "acceptable" verdicts, not uniform "needs-fix"
+7. The quality distribution in the report reflects actual clip quality
+
+---
+
+## Summary of changes by file
+
+| File | Changes |
+|------|---------|
+| `src/envs/subenv2/node4_clip_extractor.py` | Migrate FaceMesh from legacy `mp.solutions` to Tasks API. Replace or fix ArcFace embedding extraction. Add graceful degradation. |
+| `server/artifact_ingest.py` | Fix blur calibration (0.001 → 0.12). Add real lip sync proxy. Add optical flow computation. Add blink detection. Add occlusion detection. Fix `_estimate_face_occupancy_ratio` legacy API. |
+| `requirements.txt` | Update dependency notes. Pin mediapipe version. Add/remove insightface based on Task 2 decision. |
+
+## Files NOT to modify
+
+| File | Reason |
+|------|--------|
+| `src/envs/subenv2/node5_disposition.py` | Disposition logic is fine — it was getting bad inputs |
+| `src/envs/subenv2/node6_grader.py` | Grading logic is fine — tested and deterministic |
+| `server/llm_adapter.py` | Signal digest and prompt structure already fixed in previous guide |
+| `src/schemas/subenv2.py` | Schema is correct — fields match expected signals |

requirements.txt

@@ -1,5 +1,6 @@
 # Install core: pip install -r requirements.txt
-# Install full extraction stack: pip install insightface transformers
+# MediaPipe Tasks model assets are auto-downloaded on first use and can be
+# overridden with THB_FACE_LANDMARKER_MODEL / THB_FACE_DETECTOR_MODEL.
 
 # Core (required)
 numpy
@@ -8,9 +9,9 @@ pydantic>=2.0
 scipy
 safetensors
 opencv-python
-mediapipe
+mediapipe>=0.10.33
 pytest
 
-# Optional: full-fidelity signal extraction
-# pip install insightface          # ArcFace face embeddings (Sub-env 2)
+# Optional: extended tooling
+# Sub-env 2 identity drift now uses landmark embeddings (no insightface needed).
 # pip install transformers         # Tokenizer reconstruction (Sub-env 3)

server/artifact_ingest.py

@@ -14,15 +14,21 @@ import os
 import shutil
 import tempfile
 import time
+import urllib.error
+import urllib.request
 from pathlib import Path
 from threading import Lock
 from typing import Any, Optional
 from uuid import uuid4
 
 import cv2
+import numpy as np
 from fastapi import UploadFile
 
-from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+from src.envs.subenv2.node4_clip_extractor import (
+    _BLUR_CALIBRATION_CEILING,
+    extract_clip_signals,
+)
 from src.envs.subenv3.node7_weight_extractor import extract_weight_signals
 from src.schemas.subenv1 import ImageDiagnosticsObservation
 from src.schemas.subenv2 import ClipSignalObservation
@@ -65,11 +71,27 @@ _DEFAULT_PARAM_CONFIG: dict[str, float] = {
     "eta": 0.10,
 }
 
+_FACE_DETECTOR_URL = (
+    "https://storage.googleapis.com/mediapipe-models/face_detector/"
+    "blaze_face_short_range/float16/latest/blaze_face_short_range.tflite"
+)
+_DEFAULT_FACE_DETECTOR_MODEL_CANDIDATES: tuple[Path, ...] = (
+    Path(__file__).resolve().parents[1] / "data" / "models" / "face_detector.tflite",
+    Path.home() / ".cache" / "talkingheadbench" / "models" / "face_detector.tflite",
+)
+
 
 def _clamp(value: float, low: float, high: float) -> float:
     return max(low, min(high, value))
 
 
+def _env_truthy(name: str, *, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
 def _detect_conflicting_descriptors(prompt: str) -> list[str]:
     text = prompt.lower()
     pairs = [
@@ -86,22 +108,131 @@ def _detect_conflicting_descriptors(prompt: str) -> list[str]:
     return conflicts
 
 
-def _estimate_face_occupancy_ratio(image_bgr) -> float:
+def _candidate_face_detector_model_paths() -> list[Path]:
+    env_path = os.getenv("THB_FACE_DETECTOR_MODEL", "").strip()
+    candidates: list[Path] = []
+    if env_path:
+        candidates.append(Path(env_path).expanduser())
+    candidates.extend(_DEFAULT_FACE_DETECTOR_MODEL_CANDIDATES)
+
+    deduped: list[Path] = []
+    seen: set[str] = set()
+    for path in candidates:
+        key = str(path)
+        if key in seen:
+            continue
+        seen.add(key)
+        deduped.append(path)
+    return deduped
+
+
+def _download_face_detector_model(dest: Path) -> Path:
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    urllib.request.urlretrieve(_FACE_DETECTOR_URL, dest)
+    return dest
+
+
+def _resolve_face_detector_model_path() -> Path | None:
+    for candidate in _candidate_face_detector_model_paths():
+        if candidate.exists() and candidate.is_file():
+            return candidate
+
+    if not _env_truthy("THB_AUTO_DOWNLOAD_FACE_DETECTOR", default=True):
+        return None
+
+    cache_target = _DEFAULT_FACE_DETECTOR_MODEL_CANDIDATES[-1]
+    try:
+        return _download_face_detector_model(cache_target)
+    except (OSError, urllib.error.URLError, ValueError) as exc:
+        log.warning(
+            "Unable to auto-download FaceDetector model to %s: %s",
+            cache_target,
+            exc,
+        )
+        return None
+
+
+def _create_tasks_face_detector() -> Any | None:
+    model_path = _resolve_face_detector_model_path()
+    if model_path is None:
+        return None
+
     try:
-        import mediapipe as mp
+        from mediapipe.tasks.python import BaseOptions
+        from mediapipe.tasks.python.vision import (
+            FaceDetector,
+            FaceDetectorOptions,
+            RunningMode,
+        )
 
-        with mp.solutions.face_detection.FaceDetection(
-            model_selection=1,
+        options = FaceDetectorOptions(
+            base_options=BaseOptions(model_asset_path=str(model_path)),
+            running_mode=RunningMode.IMAGE,
             min_detection_confidence=0.5,
-        ) as detector:
-            rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
-            result = detector.process(rgb)
+        )
+        return FaceDetector.create_from_options(options)
+    except Exception as exc:  # noqa: BLE001
+        log.warning("Failed to initialize Tasks FaceDetector from %s: %s", model_path, exc)
+        return None
 
-        if result.detections:
-            bbox = result.detections[0].location_data.relative_bounding_box
-            return float(_clamp(float(bbox.width * bbox.height), 0.0, 1.0))
-    except Exception:
-        pass
+
+def _get_haar_face_cascade() -> cv2.CascadeClassifier | None:
+    cascade_path = Path(cv2.data.haarcascades) / "haarcascade_frontalface_default.xml"
+    if not cascade_path.exists():
+        return None
+
+    cascade = cv2.CascadeClassifier(str(cascade_path))
+    if cascade.empty():
+        return None
+    return cascade
+
+
+def _estimate_face_occupancy_ratio(image_bgr) -> float:
+    height, width = image_bgr.shape[:2]
+    detector = _create_tasks_face_detector()
+    if detector is not None:
+        try:
+            import mediapipe as mp
+
+            rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
+            mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb.copy())
+            result = detector.detect(mp_image)
+
+            detections = getattr(result, "detections", [])
+            areas: list[float] = []
+            for detection in detections:
+                bbox = getattr(detection, "bounding_box", None)
+                if bbox is None:
+                    continue
+                box_width = float(getattr(bbox, "width", 0.0))
+                box_height = float(getattr(bbox, "height", 0.0))
+                if box_width <= 0.0 or box_height <= 0.0:
+                    continue
+                areas.append((box_width * box_height) / float(max(width * height, 1)))
+
+            if areas:
+                return float(_clamp(max(areas), 0.0, 1.0))
+        except Exception as exc:  # noqa: BLE001
+            log.warning("Tasks face occupancy detection failed; falling back to Haar: %s", exc)
+        finally:
+            if hasattr(detector, "close"):
+                detector.close()
+
+    cascade = _get_haar_face_cascade()
+    if cascade is not None:
+        gray = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2GRAY)
+        faces = cascade.detectMultiScale(
+            gray,
+            scaleFactor=1.1,
+            minNeighbors=4,
+            minSize=(max(24, width // 10), max(24, height // 10)),
+        )
+        if len(faces) > 0:
+            areas = [
+                (float(face_w) * float(face_h)) / float(max(width * height, 1))
+                for (_, _, face_w, face_h) in faces
+            ]
+            return float(_clamp(max(areas), 0.0, 1.0))
 
     return 0.35
 
@@ -250,17 +381,67 @@ def _extract_clip_signals_fallback(
     blur_scores: list[float] = []
     exposure_scores: list[float] = []
     diffs: list[float] = []
+    gray_frames: list[Any] = []
+    mouth_brightness_series: list[float] = []
+    eye_brightness_series: list[float] = []
+
+    height, width = frames[0].shape[:2]
+    x1 = max(0, int(width * 0.30))
+    x2 = min(width, int(width * 0.70))
+    y1 = max(0, int(height * 0.20))
+    y2 = min(height, int(height * 0.80))
+
+    mouth_y1 = y1 + int((y2 - y1) * 0.66)
+    eye_y1 = y1
+    eye_y2 = y1 + max(1, int((y2 - y1) * 0.25))
+
+    cascade = _get_haar_face_cascade()
+    occlusion_frames = 0
 
     for idx, frame in enumerate(frames):
         gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+        gray_frames.append(gray)
 
-        lap_var = float(cv2.Laplacian(gray, cv2.CV_64F).var())
-        pixel_count = max(1, gray.shape[0] * gray.shape[1])
-        blur_scores.append(_clamp((lap_var / pixel_count) / 0.001, 0.0, 1.0))
+        face_roi = gray[y1:y2, x1:x2]
+        if face_roi.size == 0:
+            face_roi = gray
 
-        brightness = float(gray.mean() / 255.0)
+        lap_var = float(cv2.Laplacian(face_roi, cv2.CV_64F).var())
+        pixel_count = max(1, face_roi.shape[0] * face_roi.shape[1])
+        blur_scores.append(_clamp((lap_var / pixel_count) / _BLUR_CALIBRATION_CEILING, 0.0, 1.0))
+
+        brightness = float(face_roi.mean() / 255.0)
         exposure_scores.append(_clamp(1.0 - abs(brightness - 0.5) * 2.0, 0.0, 1.0))
 
+        mouth_roi = gray[mouth_y1:y2, x1:x2]
+        eye_roi = gray[eye_y1:eye_y2, x1:x2]
+        mouth_brightness_series.append(float(mouth_roi.mean()) if mouth_roi.size else float(gray.mean()))
+        eye_brightness_series.append(float(eye_roi.mean()) if eye_roi.size else float(gray.mean()))
+
+        if cascade is not None:
+            faces = cascade.detectMultiScale(
+                gray,
+                scaleFactor=1.1,
+                minNeighbors=4,
+                minSize=(max(24, width // 8), max(24, height // 8)),
+            )
+            if len(faces) == 0:
+                occlusion_frames += 1
+        else:
+            hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
+            center_hsv = hsv[y1:y2, x1:x2]
+            if center_hsv.size:
+                skin_like = (
+                    (center_hsv[..., 0] >= 0)
+                    & (center_hsv[..., 0] <= 25)
+                    & (center_hsv[..., 1] >= 30)
+                    & (center_hsv[..., 1] <= 200)
+                    & (center_hsv[..., 2] >= 60)
+                )
+                skin_ratio = float(skin_like.mean())
+                if skin_ratio < 0.06:
+                    occlusion_frames += 1
+
         if idx > 0:
             prev = frames[idx - 1].astype("float32")
             cur = frame.astype("float32")
@@ -270,20 +451,77 @@ def _extract_clip_signals_fallback(
     identity_drift_proxy = _clamp(frame_difference_mean / 45.0, 0.0, 1.0)
     landmark_jitter_proxy = _clamp(frame_difference_mean / 500.0, 0.0, 1.0)
 
+    if len(mouth_brightness_series) >= 2:
+        mouth_delta_std = float(np.std(np.diff(np.array(mouth_brightness_series, dtype=np.float32))))
+        # Mouth-region brightness deltas (in 0–255 pixel units) for active speech
+        # are typically 0.5–4.0 per frame.  Dividing by 2.5 maps:
+        #   active speech   (delta_std ≈ 2.0–4.0) → 0.80–1.00
+        #   mild movement   (delta_std ≈ 0.8–2.0) → 0.32–0.80
+        #   near-silence    (delta_std < 0.5)      → < 0.20
+        lip_sync_confidence = _clamp(mouth_delta_std / 2.5, 0.0, 1.0)
+    else:
+        lip_sync_confidence = 0.0
+
+    blink_count = 0
+    if eye_brightness_series:
+        eye_arr = np.array(eye_brightness_series, dtype=np.float32)
+        baseline = float(np.median(eye_arr))
+        threshold = baseline * 0.88
+        in_blink = False
+        for value in eye_arr:
+            if value < threshold:
+                if not in_blink:
+                    blink_count += 1
+                    in_blink = True
+            else:
+                in_blink = False
+
+    optical_flow_magnitude = 1.0
+    if len(gray_frames) >= 2:
+        face_flows: list[float] = []
+        bg_flows: list[float] = []
+
+        face_mask = np.zeros_like(gray_frames[0], dtype=bool)
+        face_mask[y1:y2, x1:x2] = True
+        for i in range(min(len(gray_frames) - 1, 30)):
+            flow = cv2.calcOpticalFlowFarneback(
+                gray_frames[i],
+                gray_frames[i + 1],
+                None,
+                0.5,
+                3,
+                15,
+                3,
+                5,
+                1.2,
+                0,
+            )
+            mag = np.sqrt(flow[..., 0] ** 2 + flow[..., 1] ** 2)
+            face_mean = float(mag[face_mask].mean()) if face_mask.any() else 0.0
+            bg_mean = float(mag[~face_mask].mean() + 1e-6)
+            face_flows.append(face_mean)
+            bg_flows.append(bg_mean)
+
+        if face_flows and bg_flows:
+            optical_flow_magnitude = (
+                float(sum(face_flows) / len(face_flows))
+                / float(sum(bg_flows) / len(bg_flows))
+            )
+
     return ClipSignalObservation(
         clip_id=clip_path.stem,
         face_embedding_variance=round(identity_drift_proxy, 4),
         landmark_stability_score=round(landmark_jitter_proxy, 4),
         identity_cosine_drift=round(identity_drift_proxy, 4),
         frame_difference_mean=round(frame_difference_mean, 4),
-        optical_flow_magnitude=1.0,
-        blink_count=max(0, len(frames) // 60),
-        lip_sync_confidence=0.5,
+        optical_flow_magnitude=round(optical_flow_magnitude, 4),
+        blink_count=blink_count,
+        lip_sync_confidence=round(lip_sync_confidence, 4),
         phoneme_sequence=[],
         phoneme_coverage_new=0.0,
         blur_score=round(float(sum(blur_scores) / max(1, len(blur_scores))), 4),
         exposure_score=round(float(sum(exposure_scores) / max(1, len(exposure_scores))), 4),
-        occlusion_frames=0,
+        occlusion_frames=occlusion_frames,
         clips_audited_so_far=int(dataset_context.get("clips_audited_so_far", 0)),
         current_phoneme_coverage=dataset_context.get("current_phoneme_coverage", {}),
         current_pose_distribution=dataset_context.get("current_pose_distribution", {}),

src/envs/subenv2/node4_clip_extractor.py

@@ -1,87 +1,218 @@
 """
 Node 4: Clip Signal Extractor — Sub-env 2.
 
-Extracts pre-computed CV signals from a raw video clip using OpenCV, MediaPipe,
-and ArcFace.  The resulting ``ClipSignalObservation`` is consumed by the Clip
-Signal Extractor agent (Node 4) which does diagnostic reasoning, not perception.
+Extracts pre-computed CV signals from a raw video clip using OpenCV and
+MediaPipe Tasks FaceLandmarker. The resulting ``ClipSignalObservation`` is
+consumed by the Clip Signal Extractor agent (Node 4) which does diagnostic
+reasoning, not perception.
 
-**No model inference is performed inline.**  Phoneme sequences are accepted from
-a pre-run forced-aligner output (e.g. Montreal Forced Aligner) passed as an
-argument.  ArcFace embeddings are extracted via the InsightFace library, which
-encapsulates the model loading externally.
+**No model inference is performed inline.** Phoneme sequences are accepted from
+an optional pre-run forced-aligner output (e.g. Montreal Forced Aligner)
+passed as an argument. Identity drift signals are computed from normalized
+landmark vectors, avoiding heavyweight ArcFace runtime dependencies.
 
 Blur score normalization
 ------------------------
 ``blur_score = clip(mean_laplacian_variance / pixel_count / CEILING, 0.0, 1.0)``
 
 ``_BLUR_CALIBRATION_CEILING`` is a calibration constant derived from the test
-set.  It maps the per-pixel Laplacian variance of a perfectly sharp reference
-frame to 1.0; values above the ceiling are clipped.
-
-MediaPipe landmark indices
---------------------------
-Eye Aspect Ratio (EAR) blink detection uses the standard six-point eye model
-from the 468-point FaceMesh topology.  Occlusion is inferred from face-mesh
-detection failure or anomalously low face landmark visibility scores.
+set. It maps per-pixel Laplacian variance of a sharp reference frame to 1.0.
 """
 
 from __future__ import annotations
 
-import math
+import logging
+import os
 from pathlib import Path
-from typing import Optional
+from typing import Any, Optional
+import urllib.error
+import urllib.request
 
 import cv2
 import mediapipe as mp
 import numpy as np
+from mediapipe.tasks.python import BaseOptions
+from mediapipe.tasks.python.vision import (
+    FaceLandmarker,
+    FaceLandmarkerOptions,
+    RunningMode,
+)
 from numpy.typing import NDArray
 
 from src.schemas.subenv2 import ClipSignalObservation
 
+log = logging.getLogger(__name__)
+
 # ---------------------------------------------------------------------------
 # Constants
 # ---------------------------------------------------------------------------
 
-# Minimum frame count; clips shorter than this are rejected.
 _MIN_FRAMES: int = 24
-
-# Calibration ceiling for blur score normalization (per-pixel Laplacian
-# variance of a sharp reference frame, derived from the test set).
-_BLUR_CALIBRATION_CEILING: float = 0.12
-
-# Eye Aspect Ratio threshold below which a frame is counted as a blink.
+# Per-pixel Laplacian variance calibration ceiling.
+# Empirically derived from sharp talking-head face ROIs at 480p–1080p:
+# a sharp 300×300 face crop has lap_var ≈ 150–600, giving per-pixel ≈ 0.0017–0.0067.
+# Setting the ceiling to 0.005 maps a sharp face to ≈ 0.33–1.0 and
+# a blurry face (lap_var ≈ 10–30) to ≈ 0.002–0.02.
+_BLUR_CALIBRATION_CEILING: float = 0.005
 _EAR_BLINK_THRESHOLD: float = 0.20
 
-# MediaPipe FaceMesh landmark indices for left and right eye (6-point model).
-# Indices follow the canonical 468-point topology.
+# 468-landmark topology indices (Tasks API keeps FaceMesh indexing).
 _LEFT_EYE_IDX: tuple[int, ...] = (362, 385, 387, 263, 373, 380)
 _RIGHT_EYE_IDX: tuple[int, ...] = (33, 160, 158, 133, 153, 144)
-
-# Landmark indices for upper and lower lip centre (for lip opening proxy).
 _UPPER_LIP_IDX: int = 13
 _LOWER_LIP_IDX: int = 14
 
+_PROJECT_ROOT = Path(__file__).resolve().parents[3]
+_FACE_LANDMARKER_URL = (
+    "https://storage.googleapis.com/mediapipe-models/face_landmarker/"
+    "face_landmarker/float16/latest/face_landmarker.task"
+)
+_DEFAULT_MODEL_CANDIDATES: tuple[Path, ...] = (
+    _PROJECT_ROOT / "data" / "models" / "face_landmarker.task",
+    Path.home() / ".cache" / "talkingheadbench" / "models" / "face_landmarker.task",
+)
+
+
+# ---------------------------------------------------------------------------
+# Private helpers — model setup
+# ---------------------------------------------------------------------------
+
+
+def _env_truthy(name: str, *, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _candidate_landmarker_model_paths() -> list[Path]:
+    env_path = os.getenv("THB_FACE_LANDMARKER_MODEL", "").strip()
+    candidates: list[Path] = []
+    if env_path:
+        candidates.append(Path(env_path).expanduser())
+    candidates.extend(_DEFAULT_MODEL_CANDIDATES)
+
+    deduped: list[Path] = []
+    seen: set[str] = set()
+    for path in candidates:
+        key = str(path)
+        if key in seen:
+            continue
+        seen.add(key)
+        deduped.append(path)
+    return deduped
+
+
+def _download_landmarker_model(dest: Path) -> Path:
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    urllib.request.urlretrieve(_FACE_LANDMARKER_URL, dest)
+    return dest
+
+
+def _resolve_landmarker_model_path() -> Path | None:
+    for candidate in _candidate_landmarker_model_paths():
+        if candidate.exists() and candidate.is_file():
+            return candidate
+
+    if not _env_truthy("THB_AUTO_DOWNLOAD_FACE_LANDMARKER", default=True):
+        return None
+
+    cache_target = _DEFAULT_MODEL_CANDIDATES[-1]
+    try:
+        downloaded = _download_landmarker_model(cache_target)
+    except (OSError, urllib.error.URLError, ValueError) as exc:
+        log.warning(
+            "Unable to auto-download FaceLandmarker model to %s: %s",
+            cache_target,
+            exc,
+        )
+        return None
+
+    log.info("Downloaded MediaPipe FaceLandmarker model to %s", downloaded)
+    return downloaded
+
+
+def _create_face_landmarker() -> Any | None:
+    model_path = _resolve_landmarker_model_path()
+    if model_path is None:
+        log.warning(
+            "FaceLandmarker model file not found. Checked: %s",
+            ", ".join(str(p) for p in _candidate_landmarker_model_paths()),
+        )
+        return None
+
+    try:
+        options = FaceLandmarkerOptions(
+            base_options=BaseOptions(model_asset_path=str(model_path)),
+            running_mode=RunningMode.IMAGE,
+            num_faces=1,
+            min_face_detection_confidence=0.5,
+            min_face_presence_confidence=0.5,
+            output_face_blendshapes=False,
+            output_facial_transformation_matrixes=False,
+        )
+        return FaceLandmarker.create_from_options(options)
+    except Exception as exc:  # noqa: BLE001
+        log.warning(
+            "Failed to initialize FaceLandmarker from %s: %s",
+            model_path,
+            exc,
+        )
+        return None
+
 
 # ---------------------------------------------------------------------------
 # Private helpers — signal computation
 # ---------------------------------------------------------------------------
 
 
-def _eye_aspect_ratio(landmarks: list, indices: tuple[int, ...]) -> float:
-    """Compute EAR for a single eye given its six landmark indices."""
-    pts = np.array(
-        [(landmarks[i].x, landmarks[i].y) for i in indices], dtype=np.float32
-    )
-    # Vertical distances
+def _landmark_embedding(landmarks: list[Any]) -> NDArray[np.float32]:
+    coords = np.array([(lm.x, lm.y, lm.z) for lm in landmarks], dtype=np.float32)
+    centered = coords - coords.mean(axis=0, keepdims=True)
+    scale = float(np.std(centered) + 1e-6)
+    return (centered / scale).flatten().astype(np.float32)
+
+
+def _face_bbox_from_landmarks(
+    landmarks: list[Any],
+    width: int,
+    height: int,
+    *,
+    padding_ratio: float = 0.2,
+) -> tuple[int, int, int, int]:
+    xs = np.array([lm.x * width for lm in landmarks], dtype=np.float32)
+    ys = np.array([lm.y * height for lm in landmarks], dtype=np.float32)
+
+    x0 = int(np.clip(np.floor(xs.min()), 0, width - 1))
+    x1 = int(np.clip(np.ceil(xs.max()), 1, width))
+    y0 = int(np.clip(np.floor(ys.min()), 0, height - 1))
+    y1 = int(np.clip(np.ceil(ys.max()), 1, height))
+
+    pad_x = int((x1 - x0) * padding_ratio)
+    pad_y = int((y1 - y0) * padding_ratio)
+
+    x0 = max(0, x0 - pad_x)
+    y0 = max(0, y0 - pad_y)
+    x1 = min(width, x1 + pad_x)
+    y1 = min(height, y1 + pad_y)
+
+    if x1 <= x0:
+        x1 = min(width, x0 + 1)
+    if y1 <= y0:
+        y1 = min(height, y0 + 1)
+
+    return x0, y0, x1, y1
+
+
+def _eye_aspect_ratio(landmarks: list[Any], indices: tuple[int, ...]) -> float:
+    pts = np.array([(landmarks[i].x, landmarks[i].y) for i in indices], dtype=np.float32)
     v1 = np.linalg.norm(pts[1] - pts[5])
     v2 = np.linalg.norm(pts[2] - pts[4])
-    # Horizontal distance
     h = np.linalg.norm(pts[0] - pts[3])
     return (v1 + v2) / (2.0 * h + 1e-6)
 
 
 def _cosine_distance(a: NDArray[np.float32], b: NDArray[np.float32]) -> float:
-    """Cosine distance (1 − cosine_similarity) between two 1-D vectors."""
     norm_a = np.linalg.norm(a)
     norm_b = np.linalg.norm(b)
     if norm_a < 1e-8 or norm_b < 1e-8:
@@ -90,7 +221,6 @@ def _cosine_distance(a: NDArray[np.float32], b: NDArray[np.float32]) -> float:
 
 
 def _laplacian_blur_score(gray: NDArray[np.uint8]) -> float:
-    """Per-pixel Laplacian variance for a single grayscale frame."""
     pixel_count = gray.shape[0] * gray.shape[1]
     lap_var = float(cv2.Laplacian(gray, cv2.CV_64F).var())
     raw = lap_var / pixel_count
@@ -98,45 +228,18 @@ def _laplacian_blur_score(gray: NDArray[np.uint8]) -> float:
 
 
 def _exposure_score(gray: NDArray[np.uint8]) -> float:
-    """Composite exposure score: normalised mean brightness − clipping fraction.
-
-    Returns a value in [0.0, 1.0] where 1.0 is ideal exposure.
-    Frames with high clipping (over- or under-exposure) score lower.
-    """
     hist = cv2.calcHist([gray], [0], None, [256], [0, 256]).flatten()
     total = gray.size
-    clipping = float((hist[0] + hist[255]) / total)  # fraction of clipped pixels
+    clipping = float((hist[0] + hist[255]) / total)
     mean_norm = float(gray.mean() / 255.0)
-    # Penalise extreme means (too dark or too bright) and clipping
     mean_score = 1.0 - abs(mean_norm - 0.5) * 2.0
     return float(np.clip(mean_score * (1.0 - clipping), 0.0, 1.0))
 
 
 def _parse_aligner_phonemes(aligner_output: dict) -> list[str]:
-    """Extract an ordered phoneme list from a forced-aligner output dict.
-
-    Supports two common Montreal Forced Aligner output formats:
-
-    Format A — flat list::
-
-        {"phonemes": ["AH", "B", "AH", ...]}
-
-    Format B — TextGrid-style tiers (MFA JSON export)::
-
-        {"tiers": {"phones": {"entries": [[t0, t1, "AH"], ...]}}}
-
-    Args:
-        aligner_output: Parsed JSON dict from the forced aligner.
-
-    Returns:
-        Ordered list of phoneme strings (silence tokens ``"SIL"``/``"sp"``
-        are preserved; callers may filter them if desired).
-    """
-    # Format A
     if "phonemes" in aligner_output:
         return [str(p) for p in aligner_output["phonemes"]]
 
-    # Format B
     try:
         entries = aligner_output["tiers"]["phones"]["entries"]
         return [str(entry[2]) for entry in entries]
@@ -151,53 +254,28 @@ def _phoneme_coverage_new(
     phoneme_sequence: list[str],
     current_phoneme_coverage: dict,
 ) -> float:
-    """Fraction of phonemes in this clip not yet covered by the dataset.
-
-    A phoneme is considered «covered» if its count in
-    ``current_phoneme_coverage`` is greater than zero.
-
-    Returns 0.0 if ``phoneme_sequence`` is empty.
-    """
     unique_in_clip = set(phoneme_sequence)
     if not unique_in_clip:
         return 0.0
-    new_count = sum(
-        1
-        for p in unique_in_clip
-        if current_phoneme_coverage.get(p, 0) == 0
-    )
+    new_count = sum(1 for p in unique_in_clip if current_phoneme_coverage.get(p, 0) == 0)
     return new_count / len(unique_in_clip)
 
 
-def _lip_sync_confidence_proxy(
-    lip_openings: list[float],
-    cap: cv2.VideoCapture,
-) -> float:
-    """Compute a proxy lip-sync confidence score from lip opening variance.
-
-    Without running Wav2Lip inference, we estimate sync quality by measuring
-    whether lip movement is correlated with audio energy extracted directly
-    from the video's audio track via OpenCV.  If no audio is available, the
-    score is the normalised standard deviation of lip openings (a proxy for
-    whether the speaker's lips were moving at all).
-
-    This is a heuristic proxy for the Wav2Lip-style alignment score described
-    in the spec.  Replace with a proper AV-sync model in production.
-
-    Args:
-        lip_openings: Per-frame lip opening distance (in normalised coords).
-        cap: Already-opened ``cv2.VideoCapture`` for the clip (used only to
-             probe for audio; audio extraction is not performed here).
+def _lip_sync_confidence_proxy(lip_openings: list[float]) -> float:
+    """Map mouth-opening variance to a lip-sync confidence score in [0, 1].
 
-    Returns:
-        A float in [0.0, 1.0].
+    Lip openings are normalized landmark Y-distances (range ~ 0.00–0.08).
+    A talking sequence has std ≈ 0.003–0.010; silence is near 0.
+    Divisor 0.008 maps:
+      - active talking  (std ≈ 0.006–0.010) → 0.75–1.00
+      - mild movement   (std ≈ 0.003–0.006) → 0.38–0.75
+      - near-silence    (std < 0.003)        → < 0.38
     """
     if not lip_openings:
         return 0.0
     arr = np.array(lip_openings, dtype=np.float32)
     std = float(arr.std())
-    # Normalise: std of 0 means no movement → 0.0; std ≥ 0.05 → full score
-    return float(np.clip(std / 0.05, 0.0, 1.0))
+    return float(np.clip(std / 0.008, 0.0, 1.0))
 
 
 # ---------------------------------------------------------------------------
@@ -210,47 +288,13 @@ def extract_clip_signals(
     dataset_context: dict,
     aligner_output: Optional[dict] = None,
 ) -> ClipSignalObservation:
-    """Extract CV signals from a raw video clip for the Clip Signal Extractor.
-
-    All signals are computed deterministically from pixel and landmark data
-    using OpenCV, MediaPipe FaceMesh, and InsightFace ArcFace.  No generative
-    model inference is performed.  The phoneme sequence is accepted from a
-    pre-run forced-aligner rather than being derived inline.
-
-    Args:
-        clip_path: Absolute or relative path to the video file.
-        dataset_context: Dict with the following required keys:
-
-            - ``"clips_audited_so_far"`` (int): Clips already processed.
-            - ``"current_phoneme_coverage"`` (dict[str, int]): Phoneme →
-              count across accepted clips so far.
-            - ``"current_pose_distribution"`` (dict[str, int]): Regime →
-              accepted-clip count.
-            - ``"similar_clips_accepted"`` (int): Count of already-accepted
-              clips sharing the same regime and similar ArcFace embedding.
-
-        aligner_output: Parsed JSON dict from a forced aligner (e.g.
-            Montreal Forced Aligner).  If ``None``, ``phoneme_sequence`` is
-            set to an empty list and ``phoneme_coverage_new`` to 0.0.
-            Supported formats are described in ``_parse_aligner_phonemes``.
-
-    Returns:
-        A fully populated :class:`ClipSignalObservation`.
-
-    Raises:
-        FileNotFoundError: If ``clip_path`` does not exist.
-        ValueError: If the clip contains fewer than ``_MIN_FRAMES`` (24) frames,
-            or if the video cannot be opened by OpenCV.
-    """
+    """Extract CV signals from a raw video clip for Node 4."""
     clip_path = Path(clip_path)
     if not clip_path.exists():
         raise FileNotFoundError(f"Clip not found: {clip_path}")
 
     clip_id = clip_path.stem
 
-    # ------------------------------------------------------------------
-    # Open video
-    # ------------------------------------------------------------------
     cap = cv2.VideoCapture(str(clip_path))
     if not cap.isOpened():
         raise ValueError(f"OpenCV could not open video file: {clip_path}")
@@ -267,98 +311,75 @@ def extract_clip_signals(
 
     if len(frames_bgr) < _MIN_FRAMES:
         raise ValueError(
-            f"Clip '{clip_id}' has only {len(frames_bgr)} frames; "
-            f"at least {_MIN_FRAMES} are required."
+            f"Clip '{clip_id}' has only {len(frames_bgr)} frames; at least {_MIN_FRAMES} are required."
         )
 
     n_frames = len(frames_bgr)
     h, w = frames_bgr[0].shape[:2]
 
-    # ------------------------------------------------------------------
-    # MediaPipe FaceMesh setup
-    # ------------------------------------------------------------------
-    mp_face_mesh = mp.solutions.face_mesh
-    face_mesh = mp_face_mesh.FaceMesh(
-        static_image_mode=True,
-        max_num_faces=1,
-        refine_landmarks=True,
-        min_detection_confidence=0.5,
-    )
+    face_landmarker = _create_face_landmarker()
+    if face_landmarker is None:
+        raise ValueError(
+            "FaceLandmarker model file not found or failed to initialize. "
+            "Set THB_FACE_LANDMARKER_MODEL or place model at data/models/face_landmarker.task."
+        )
 
-    # Per-frame collections
-    landmark_sets: list[Optional[list]] = []   # None if no face detected
+    landmark_sets: list[Optional[list[Any]]] = []
+    landmark_embeddings: list[NDArray[np.float32]] = []
     lip_openings: list[float] = []
     blur_scores: list[float] = []
     exposure_scores: list[float] = []
     ear_values: list[float] = []
     occlusion_frame_count: int = 0
 
-    for frame_bgr in frames_bgr:
-        gray = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2GRAY)
-        blur_scores.append(_laplacian_blur_score(gray))
-        exposure_scores.append(_exposure_score(gray))
-
-        rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
-        result = face_mesh.process(rgb)
-
-        if result.multi_face_landmarks:
-            lm = result.multi_face_landmarks[0].landmark
-            landmark_sets.append(lm)
+    try:
+        for frame_bgr in frames_bgr:
+            gray = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2GRAY)
 
-            # EAR for blink detection
-            ear = 0.5 * (
-                _eye_aspect_ratio(lm, _LEFT_EYE_IDX)
-                + _eye_aspect_ratio(lm, _RIGHT_EYE_IDX)
-            )
-            ear_values.append(ear)
+            rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
+            mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb.copy())
+            result = face_landmarker.detect(mp_image)
 
-            # Lip opening (normalised image coords)
-            lip_open = abs(lm[_LOWER_LIP_IDX].y - lm[_UPPER_LIP_IDX].y)
-            lip_openings.append(lip_open)
-        else:
-            landmark_sets.append(None)
-            ear_values.append(1.0)   # assume open (no blink) when undetected
-            lip_openings.append(0.0)
-            occlusion_frame_count += 1
+            if result.face_landmarks:
+                lm = result.face_landmarks[0]
+                landmark_sets.append(lm)
+                landmark_embeddings.append(_landmark_embedding(lm))
 
-    face_mesh.close()
+                x0, y0, x1, y1 = _face_bbox_from_landmarks(lm, w, h)
+                face_gray = gray[y0:y1, x0:x1]
+                if face_gray.size == 0:
+                    face_gray = gray
+                blur_scores.append(_laplacian_blur_score(face_gray))
+                exposure_scores.append(_exposure_score(face_gray))
 
-    # ------------------------------------------------------------------
-    # ArcFace embeddings (InsightFace)
-    # ------------------------------------------------------------------
-    try:
-        import insightface
-        from insightface.app import FaceAnalysis
+                ear = 0.5 * (_eye_aspect_ratio(lm, _LEFT_EYE_IDX) + _eye_aspect_ratio(lm, _RIGHT_EYE_IDX))
+                ear_values.append(ear)
 
-        fa = FaceAnalysis(allowed_modules=["detection", "recognition"])
-        fa.prepare(ctx_id=-1)  # CPU; set ctx_id ≥ 0 for GPU
+                lip_open = abs(lm[_LOWER_LIP_IDX].y - lm[_UPPER_LIP_IDX].y)
+                lip_openings.append(lip_open)
+            else:
+                landmark_sets.append(None)
+                blur_scores.append(_laplacian_blur_score(gray))
+                exposure_scores.append(_exposure_score(gray))
+                ear_values.append(1.0)
+                lip_openings.append(0.0)
+                occlusion_frame_count += 1
+    finally:
+        if hasattr(face_landmarker, "close"):
+            face_landmarker.close()
 
-        embeddings: list[NDArray[np.float32]] = []
-        for frame_bgr in frames_bgr:
-            faces = fa.get(frame_bgr)
-            if faces:
-                embeddings.append(faces[0].normed_embedding.astype(np.float32))
-    except ImportError:
-        embeddings = []
-
-    # Identity signals
-    if len(embeddings) >= 2:
-        emb_matrix = np.stack(embeddings, axis=0)               # (K, D)
+    if len(landmark_embeddings) >= 2:
+        emb_matrix = np.stack(landmark_embeddings, axis=0)
         face_embedding_variance = float(np.var(emb_matrix, axis=0).mean())
         identity_cosine_drift = _cosine_distance(emb_matrix[0], emb_matrix[-1])
-    elif len(embeddings) == 1:
+    elif len(landmark_embeddings) == 1:
         face_embedding_variance = 0.0
         identity_cosine_drift = 0.0
     else:
-        # No face detected in any frame — treat as maximum variance/drift
         face_embedding_variance = 1.0
         identity_cosine_drift = 1.0
 
-    # ------------------------------------------------------------------
-    # Landmark stability (frame-to-frame jitter)
-    # ------------------------------------------------------------------
     detected_lm = [(i, lm) for i, lm in enumerate(landmark_sets) if lm is not None]
-
     if len(detected_lm) >= 2:
         jitter_values: list[float] = []
         for (_, lm_a), (_, lm_b) in zip(detected_lm, detected_lm[1:]):
@@ -367,11 +388,8 @@ def extract_clip_signals(
             jitter_values.append(float(np.mean(np.linalg.norm(pts_a - pts_b, axis=1))))
         landmark_stability_score = float(np.mean(jitter_values))
     else:
-        landmark_stability_score = 1.0  # worst case — no stable landmarks
+        landmark_stability_score = 1.0
 
-    # ------------------------------------------------------------------
-    # Blink count (EAR threshold)
-    # ------------------------------------------------------------------
     blink_count = 0
     in_blink = False
     for ear in ear_values:
@@ -382,9 +400,6 @@ def extract_clip_signals(
         else:
             in_blink = False
 
-    # ------------------------------------------------------------------
-    # Frame difference mean (temporal signal)
-    # ------------------------------------------------------------------
     if n_frames >= 2:
         diffs: list[float] = []
         for fa_fr, fb_fr in zip(frames_bgr, frames_bgr[1:]):
@@ -393,22 +408,16 @@ def extract_clip_signals(
     else:
         frame_difference_mean = 0.0
 
-    # ------------------------------------------------------------------
-    # Optical flow magnitude — face region vs background ratio
-    # ------------------------------------------------------------------
     if n_frames >= 2:
         face_flows: list[float] = []
         bg_flows: list[float] = []
 
-        for i in range(min(n_frames - 1, 30)):   # cap at 30 pairs for speed
+        for i in range(min(n_frames - 1, 30)):
             g1 = cv2.cvtColor(frames_bgr[i], cv2.COLOR_BGR2GRAY)
             g2 = cv2.cvtColor(frames_bgr[i + 1], cv2.COLOR_BGR2GRAY)
-            flow = cv2.calcOpticalFlowFarneback(
-                g1, g2, None, 0.5, 3, 15, 3, 5, 1.2, 0
-            )
+            flow = cv2.calcOpticalFlowFarneback(g1, g2, None, 0.5, 3, 15, 3, 5, 1.2, 0)
             mag = np.sqrt(flow[..., 0] ** 2 + flow[..., 1] ** 2)
 
-            # Use landmark bounding box as face region if available
             lm_a = landmark_sets[i]
             if lm_a is not None:
                 xs = [int(p.x * w) for p in lm_a]
@@ -418,7 +427,6 @@ def extract_clip_signals(
                 face_mask = np.zeros((h, w), dtype=bool)
                 face_mask[y1:y2, x1:x2] = True
             else:
-                # Fallback: central 40 % of frame
                 cx, cy = w // 2, h // 2
                 face_mask = np.zeros((h, w), dtype=bool)
                 face_mask[cy - h // 5 : cy + h // 5, cx - w // 5 : cx + w // 5] = True
@@ -431,22 +439,10 @@ def extract_clip_signals(
     else:
         optical_flow_magnitude = 1.0
 
-    # ------------------------------------------------------------------
-    # Aggregate quality signals
-    # ------------------------------------------------------------------
     blur_score = float(np.mean(blur_scores))
     exposure_score_val = float(np.mean(exposure_scores))
+    lip_sync_confidence = _lip_sync_confidence_proxy(lip_openings)
 
-    # ------------------------------------------------------------------
-    # Lip sync confidence proxy
-    # ------------------------------------------------------------------
-    cap2 = cv2.VideoCapture(str(clip_path))
-    lip_sync_confidence = _lip_sync_confidence_proxy(lip_openings, cap2)
-    cap2.release()
-
-    # ------------------------------------------------------------------
-    # Phoneme signals
-    # ------------------------------------------------------------------
     if aligner_output is not None:
         phoneme_sequence = _parse_aligner_phonemes(aligner_output)
     else:
@@ -455,28 +451,20 @@ def extract_clip_signals(
     current_phoneme_coverage: dict = dataset_context.get("current_phoneme_coverage", {})
     phone_cov_new = _phoneme_coverage_new(phoneme_sequence, current_phoneme_coverage)
 
-    # ------------------------------------------------------------------
-    # Assemble observation
-    # ------------------------------------------------------------------
     return ClipSignalObservation(
         clip_id=clip_id,
-        # Identity consistency
         face_embedding_variance=face_embedding_variance,
         landmark_stability_score=landmark_stability_score,
         identity_cosine_drift=identity_cosine_drift,
-        # Temporal
         frame_difference_mean=frame_difference_mean,
         optical_flow_magnitude=optical_flow_magnitude,
         blink_count=blink_count,
-        # Audio-visual alignment
         lip_sync_confidence=lip_sync_confidence,
         phoneme_sequence=phoneme_sequence,
         phoneme_coverage_new=phone_cov_new,
-        # Quality
         blur_score=blur_score,
         exposure_score=exposure_score_val,
         occlusion_frames=occlusion_frame_count,
-        # Dataset context (passed through from caller)
         clips_audited_so_far=int(dataset_context.get("clips_audited_so_far", 0)),
         current_phoneme_coverage=current_phoneme_coverage,
         current_pose_distribution=dataset_context.get("current_pose_distribution", {}),

tests/unit/test_artifact_ingest.py

@@ -7,6 +7,7 @@ from io import BytesIO
 from pathlib import Path
 
 from fastapi import UploadFile
+import numpy as np
 
 import server.artifact_ingest as artifact_ingest
 
@@ -100,3 +101,63 @@ def test_ingest_artifacts_to_bundle_sets_per_clip_fallback_flags(monkeypatch, tm
     extractor_metadata = bundle["ingestion_metadata"]["extractor_metadata"]
     assert extractor_metadata["clip_extractor_fallback_count"] == 1
     assert extractor_metadata["fallback_clip_ids"] == ["clip_b"]
+
+
+def test_extract_clip_signals_fallback_computes_dynamic_proxies(monkeypatch):
+    class _FakeCapture:
+        def __init__(self, frames):
+            self._frames = [frame.copy() for frame in frames]
+            self._idx = 0
+
+        def isOpened(self):
+            return True
+
+        def read(self):
+            if self._idx >= len(self._frames):
+                return False, None
+            frame = self._frames[self._idx]
+            self._idx += 1
+            return True, frame
+
+        def release(self):
+            return None
+
+    class _FakeCascade:
+        def __init__(self):
+            self.calls = 0
+
+        def detectMultiScale(self, gray, scaleFactor, minNeighbors, minSize):
+            self.calls += 1
+            if self.calls % 2 == 0:
+                return np.empty((0, 4), dtype=np.int32)
+            return np.array([[10, 10, 30, 30]], dtype=np.int32)
+
+    frames = []
+    for idx in range(8):
+        frame = np.full((80, 120, 3), 80, dtype=np.uint8)
+        # Add changing mouth-region brightness for lip-sync proxy variability.
+        frame[46:64, 40:80] = 80 + idx * 12
+        # Add moving patch to induce optical flow.
+        start = 10 + idx
+        frame[20:35, start : start + 15] = 220
+        frames.append(frame)
+
+    monkeypatch.setattr(artifact_ingest.cv2, "VideoCapture", lambda _: _FakeCapture(frames))
+    monkeypatch.setattr(artifact_ingest, "_get_haar_face_cascade", lambda: _FakeCascade())
+
+    obs = artifact_ingest._extract_clip_signals_fallback(
+        Path("clip_dynamic.mp4"),
+        {
+            "clips_audited_so_far": 0,
+            "current_phoneme_coverage": {},
+            "current_pose_distribution": {},
+            "similar_clips_accepted": 0,
+        },
+    )
+
+    assert 0.0 <= obs.blur_score <= 1.0
+    assert 0.0 <= obs.lip_sync_confidence <= 1.0
+    assert 0.0 <= obs.optical_flow_magnitude
+    assert obs.lip_sync_confidence != 0.5
+    assert obs.optical_flow_magnitude != 1.0
+    assert obs.occlusion_frames > 0

tests/unit/test_node4_extractor.py

@@ -2,15 +2,13 @@
 Unit tests for Node 4: Clip Signal Extractor
 (src/envs/subenv2/node4_clip_extractor.py)
 
-All heavy dependencies — OpenCV VideoCapture, InsightFace, and MediaPipe
-FaceMesh — are fully mocked so no real video files or GPU/model weights
-are required.
+OpenCV VideoCapture and FaceLandmarker initialization are mocked so no real
+video files or model assets are required.
 """
 
 from __future__ import annotations
 
 import json
-import sys
 import types
 from pathlib import Path
 from unittest.mock import MagicMock, patch
@@ -29,112 +27,45 @@ _RNG = np.random.default_rng(42)
 
 
 def _random_frame(height: int = 72, width: int = 128) -> np.ndarray:
-    """Return a random uint8 BGR frame."""
     return _RNG.integers(0, 256, (height, width, 3), dtype=np.uint8)
 
 
-# ---------------------------------------------------------------------------
-# Fixture: mock_capture
-# ---------------------------------------------------------------------------
-
-
 def mock_capture(frame_count: int, height: int = 72, width: int = 128) -> MagicMock:
-    """Return a MagicMock that behaves like cv2.VideoCapture.
-
-    * ``get(cv2.CAP_PROP_FRAME_COUNT)`` → *frame_count*
-    * ``get(cv2.CAP_PROP_FPS)``         → 24.0
-    * ``read()`` cycles through *frame_count* random numpy frames then
-      returns ``(False, None)``
-    * ``isOpened()``                     → True
-    * ``release()``                      → no-op
-    """
-    import cv2  # noqa: PLC0415
-
     frames = [_random_frame(height, width) for _ in range(frame_count)]
     read_returns = [(True, f) for f in frames] + [(False, None)]
     read_iter = iter(read_returns)
 
     cap = MagicMock()
     cap.isOpened.return_value = True
-
-    def _get(prop_id):
-        if prop_id == cv2.CAP_PROP_FRAME_COUNT:
-            return float(frame_count)
-        if prop_id == cv2.CAP_PROP_FPS:
-            return 24.0
-        return 0.0
-
-    cap.get.side_effect = _get
     cap.read.side_effect = lambda: next(read_iter)
     cap.release.return_value = None
     return cap
 
 
-# ---------------------------------------------------------------------------
-# Shared MediaPipe / InsightFace mock builders
-# ---------------------------------------------------------------------------
-
-
-def _make_face_mesh_mock() -> MagicMock:
-    """Return a MagicMock mimicking mediapipe FaceMesh.
-
-    Each call to ``process()`` returns a result with a single detected face
-    carrying 478 stable landmarks (x=0.5, y=0.5, z=0.0).
-    """
-    lm = MagicMock()
-    lm.x = 0.5
-    lm.y = 0.5
-    lm.z = 0.0
-
-    face_landmark = MagicMock()
-    face_landmark.landmark = [lm] * 478
-
-    mp_result = MagicMock()
-    mp_result.multi_face_landmarks = [face_landmark]
-
-    face_mesh = MagicMock()
-    face_mesh.process.return_value = mp_result
-    face_mesh.close.return_value = None
-    return face_mesh
-
-
-def _make_mediapipe_mock(face_mesh_instance: MagicMock) -> MagicMock:
-    """Build a fake ``mp`` module alias as imported in node4_clip_extractor."""
-    mp_mock = MagicMock()
-    mp_mock.solutions.face_mesh.FaceMesh.return_value = face_mesh_instance
-    return mp_mock
-
+def _make_landmarker_mock() -> MagicMock:
+    lm = types.SimpleNamespace(x=0.5, y=0.5, z=0.0)
+    landmarks = [lm] * 478
+    detect_result = types.SimpleNamespace(face_landmarks=[landmarks])
 
-def _make_insightface_modules() -> dict:
-    """Return a sys.modules patch dict for insightface.
+    landmarker = MagicMock()
+    landmarker.detect.return_value = detect_result
+    landmarker.close.return_value = None
+    return landmarker
 
-    ``FaceAnalysis.get()`` returns a single face with a unit 512-D embedding.
-    """
-    embedding = np.ones(512, dtype=np.float32)
 
-    face = MagicMock()
-    face.normed_embedding = embedding
-
-    fa_instance = MagicMock()
-    fa_instance.get.return_value = [face]
-    fa_instance.prepare.return_value = None
-
-    FaceAnalysis = MagicMock(return_value=fa_instance)
-
-    insightface_mod = types.ModuleType("insightface")
-    app_mod = types.ModuleType("insightface.app")
-    app_mod.FaceAnalysis = FaceAnalysis
-    insightface_mod.app = app_mod
-
-    return {
-        "insightface": insightface_mod,
-        "insightface.app": app_mod,
-    }
+def _full_patch(cap_mock: MagicMock, landmarker: MagicMock):
+    from contextlib import ExitStack
 
+    stack = ExitStack()
+    stack.enter_context(patch("cv2.VideoCapture", return_value=cap_mock))
+    stack.enter_context(
+        patch(
+            "src.envs.subenv2.node4_clip_extractor._create_face_landmarker",
+            return_value=landmarker,
+        )
+    )
+    return stack
 
-# ---------------------------------------------------------------------------
-# Common dataset context skeleton
-# ---------------------------------------------------------------------------
 
 _EMPTY_CTX: dict = {
     "current_phoneme_coverage": {},
@@ -145,69 +76,39 @@ _EMPTY_CTX: dict = {
 
 
 # ---------------------------------------------------------------------------
-# Context manager: full env patch
-# ---------------------------------------------------------------------------
-
-
-def _full_patch(cap_mock: MagicMock, mp_mock: MagicMock):
-    """Return a combined context manager that patches cv2, insightface, and mp."""
-    from contextlib import ExitStack
-
-    stack = ExitStack()
-    stack.enter_context(patch("cv2.VideoCapture", return_value=cap_mock))
-    stack.enter_context(patch.dict("sys.modules", _make_insightface_modules()))
-    stack.enter_context(
-        patch("src.envs.subenv2.node4_clip_extractor.mp", mp_mock)
-    )
-    return stack
-
-
-# ---------------------------------------------------------------------------
-# Test 1 — short clip raises ValueError mentioning the minimum frame count
+# Tests
 # ---------------------------------------------------------------------------
 
 
 def test_raises_on_short_clip(tmp_path):
-    """Clips with fewer than 24 frames must raise ValueError containing '24'."""
     dummy = tmp_path / "dummy.mp4"
     dummy.touch()
 
     cap = mock_capture(10)
-
     with patch("cv2.VideoCapture", return_value=cap):
         with pytest.raises(ValueError, match="24"):
             from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
-            extract_clip_signals(dummy, {})
-
 
-# ---------------------------------------------------------------------------
-# Test 2 — blur_score is in [0, 1] and result is ClipSignalObservation
-# ---------------------------------------------------------------------------
+            extract_clip_signals(dummy, {})
 
 
 def test_blur_score_in_range(tmp_path):
-    """blur_score must be in [0.0, 1.0] and return type must be ClipSignalObservation."""
     dummy = tmp_path / "dummy.mp4"
     dummy.touch()
 
     cap30 = mock_capture(30)
-    mp_mock = _make_mediapipe_mock(_make_face_mesh_mock())
+    landmarker = _make_landmarker_mock()
 
-    with _full_patch(cap30, mp_mock):
+    with _full_patch(cap30, landmarker):
         from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+
         result = extract_clip_signals(dummy, _EMPTY_CTX)
 
     assert isinstance(result, ClipSignalObservation)
     assert 0.0 <= result.blur_score <= 1.0
 
 
-# ---------------------------------------------------------------------------
-# Test 3 — phoneme_coverage_new == 1.0 when dataset phoneme coverage is empty
-# ---------------------------------------------------------------------------
-
-
 def test_phoneme_coverage_new_empty_dataset(tmp_path):
-    """All phonemes in the clip are new when the dataset coverage is empty."""
     dummy = tmp_path / "dummy.mp4"
     dummy.touch()
 
@@ -216,11 +117,12 @@ def test_phoneme_coverage_new_empty_dataset(tmp_path):
     aligner_json.write_text(json.dumps(aligner_data))
 
     cap30 = mock_capture(30)
-    mp_mock = _make_mediapipe_mock(_make_face_mesh_mock())
+    landmarker = _make_landmarker_mock()
     ctx = {**_EMPTY_CTX, "current_phoneme_coverage": {}}
 
-    with _full_patch(cap30, mp_mock):
+    with _full_patch(cap30, landmarker):
         from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+
         result = extract_clip_signals(
             dummy,
             ctx,
@@ -230,13 +132,7 @@ def test_phoneme_coverage_new_empty_dataset(tmp_path):
     assert result.phoneme_coverage_new == 1.0
 
 
-# ---------------------------------------------------------------------------
-# Test 4 — phoneme_coverage_new ≈ 2/3 when one phoneme already covered
-# ---------------------------------------------------------------------------
-
-
 def test_phoneme_coverage_new_partial(tmp_path):
-    """When one of three unique phonemes is already covered, new coverage = 2/3."""
     dummy = tmp_path / "dummy.mp4"
     dummy.touch()
 
@@ -245,11 +141,12 @@ def test_phoneme_coverage_new_partial(tmp_path):
     aligner_json.write_text(json.dumps(aligner_data))
 
     cap30 = mock_capture(30)
-    mp_mock = _make_mediapipe_mock(_make_face_mesh_mock())
+    landmarker = _make_landmarker_mock()
     ctx = {**_EMPTY_CTX, "current_phoneme_coverage": {"AH": 3}}
 
-    with _full_patch(cap30, mp_mock):
+    with _full_patch(cap30, landmarker):
         from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+
         result = extract_clip_signals(
             dummy,
             ctx,
@@ -259,22 +156,32 @@ def test_phoneme_coverage_new_partial(tmp_path):
     assert abs(result.phoneme_coverage_new - 2 / 3) < 1e-6
 
 
-# ---------------------------------------------------------------------------
-# Test 5 — no forced-align path → empty phoneme sequence and zero lip sync
-# ---------------------------------------------------------------------------
-
-
 def test_no_forced_align_path(tmp_path):
-    """When aligner_output is None, phoneme_sequence=[] and lip_sync_confidence=0.0."""
     dummy = tmp_path / "dummy.mp4"
     dummy.touch()
 
     cap30 = mock_capture(30)
-    mp_mock = _make_mediapipe_mock(_make_face_mesh_mock())
+    landmarker = _make_landmarker_mock()
 
-    with _full_patch(cap30, mp_mock):
+    with _full_patch(cap30, landmarker):
         from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+
         result = extract_clip_signals(dummy, _EMPTY_CTX, aligner_output=None)
 
     assert result.phoneme_sequence == []
     assert result.lip_sync_confidence == 0.0
+
+
+def test_raises_when_landmarker_unavailable(tmp_path):
+    dummy = tmp_path / "dummy.mp4"
+    dummy.touch()
+
+    cap30 = mock_capture(30)
+    with patch("cv2.VideoCapture", return_value=cap30), patch(
+        "src.envs.subenv2.node4_clip_extractor._create_face_landmarker",
+        return_value=None,
+    ):
+        with pytest.raises(ValueError, match="FaceLandmarker model file"):
+            from src.envs.subenv2.node4_clip_extractor import extract_clip_signals
+
+            extract_clip_signals(dummy, _EMPTY_CTX)