docs/PLAN.md

# TRIBE v2 Video Brain-Score — Implementation Plan & Design

**Status:** Plan for operator review. Build is GATED on (a) operator approval of this plan and (b) Meta's approval of the gated LLaMA-3.2-3B access (currently *awaiting review*). The repo is scaffolded; **no model has been or will be run locally**.

**Owners:** `tribe-manager` (lead) + `brain` (decision proxy, max-effort + sequential-thinking) + coding agents (ultra/high).

---

## 0. Hard gates & guardrails (non-negotiable)

1. **No local model execution.** This VPS has no GPU and cannot run tribev2. ALL model execution + testing happens **only on the ZeroGPU HF Space**. Locally we only: write code, run syntax/synthetic-data unit tests, and fetch config/metadata. The inference engine is built around an **injected `infer_fn`** so it unit-tests with pure-numpy stubs.
2. **Exactly ONE Hugging Face Space, ever.** If a 2nd Space, extra GPU, paid tier, or any additional HF resource is ever needed → **STOP and emit a `NEEDS_INPUT`** to the operator first.
3. **Open source.** Public GitHub repo; OUR code under Apache-2.0; a `NOTICE` documents the non-commercial CC-BY-NC model + Llama license.
4. **No Anthropic API keys, anywhere.**

---

## 1. Model identity (CONFIRMED — no repo link needed)

- **Repo:** [`facebook/tribev2`](https://huggingface.co/facebook/tribev2) — official `facebook/` org; 601 likes / 86k downloads; `best.ckpt` (708 MB) + `config.yaml`. License **CC-BY-NC-4.0 (non-commercial)**.
- **Paper:** d'Ascoli, Rapin, Benchetrit, Brookes, Begany, Raugel, Banville, King (2026), *"A Foundation Model of Vision, Audition, and Language for In-Silico Neuroscience"* (Meta FAIR / Brain & AI).
- **Code:** `github.com/facebookresearch/tribev2` (pip-installable). **Reference Space:** `cbensimon/tribe-v2-demo` (cbensimon = HF's ZeroGPU engineer — gold-standard pattern).
- **What it is:** a multimodal **fMRI brain encoder**. Inputs = naturalistic stimuli (video/audio/text). Output = **predicted cortical brain activity** of the *average subject* on the **fsaverage5** mesh.
- **Backbones (from shipped `config.yaml`):**
  | Modality | Repo | Gated? |
  |---|---|---|
  | Text | `meta-llama/Llama-3.2-3B` (base) | **YES — `manual`, awaiting Meta review** |
  | Video | `facebook/vjepa2-vitg-fpc64-256` | no |
  | Audio | `facebook/w2v-bert-2.0` | no |
  | Image | `facebook/dinov2-large` (on video frames) | no |
- **Variant note:** `facebook/tribev2-subcortical` exists but adds a second model → **excluded** (single-Space rule; cortical model only).

---

## 2. Supported input MODES (exhaustive — only what the model genuinely supports)

From `demo_utils.get_events_dataframe()` (`text_path` | `audio_path` | `video_path`, with `VALID_SUFFIXES`), the model card ("video, audio, text"), and the paper ("vision, audition, and language"), tribev2 has **exactly three input entry points**:

| Mode | Input | Internal feature extractors engaged | Timeline output |
|---|---|---|---|
| 🎬 **Video** *(primary)* | `.mp4 .mov .mkv .avi .webm` | V-JEPA2 (video) + DINOv2 (frames) + W2V-BERT (extracted audio) + LLaMA (ASR word context) | ✅ full multimodal |
| 🔊 **Audio** | `.wav .mp3 .flac .ogg` | W2V-BERT + LLaMA (ASR words) | ✅ |
| 📝 **Text** | textbox / `.txt` | gTTS → W2V-BERT + LLaMA (synthesized speech) | ✅ (over synthesized speech length) |

**There is NO standalone Image mode** — DINOv2 image features are derived *internally from video frames*; the public API exposes no `image_path`. We will **not fake** an image mode. The UI exposes a **3-tab mode-switcher (Video / Audio / Text)**; all three feed the *same* metric-timeline pipeline (`predict()` is modality-agnostic on the events DataFrame).

---

## 3. Inference API surface + exact tensors (item 1)

```python
from tribev2 import TribeModel
model = TribeModel.from_pretrained(
    "facebook/tribev2",
    cache_folder=CACHE_DIR,           # str; persistent /data if available (see §7)
    device="auto",
    config_update={"data.overlap_trs_train": 20},  # v1: 20 TR (20 s) overlap; see §4
)
events = model.get_events_dataframe(video_path=... | audio_path=... | text_path=...)
preds, segments = model.predict(events, verbose=True)
```

**Verified facts (from `model.py`, `main.py`, `demo_utils.py`, `pl_module.py`, `config.yaml`):**
- `from_pretrained` downloads `config.yaml` + `best.ckpt` via `hf_hub_download`, builds `FmriEncoderModel`, loads `state_dict` (strict), `.to(device).eval()`. It **unconditionally sets `config["average_subjects"]=True`** (demo_utils.py:218) → single "average subject"; we do **not** need to pass it. It then applies `config.update(config_update)` (flat dotted keys on a `ConfDict`), so `config_update` is our hook for loader-time params (e.g. `data.overlap_trs_train`).
- **Backbones load lazily** inside `extractor.prepare()` *during `predict()`*, then are **freed** (`_free_extractor_model`) → ⇒ **building the model at startup does NOT touch gated Llama; only `predict()` does.** (So the Space can deploy + boot before Meta approval; only live inference 403s.)
- **`predict()` returns `(preds, all_segments)`**:
  - `preds`: `np.ndarray` shape **`(n_kept_TRs, 20484)`** — per-TR predicted activity. `20484 = 2 × 10242` fsaverage5 vertices, **left hemisphere `[0:10242]` then right `[10242:20484]`** (`main.py:515` `n_outputs = 2 * FSAVERAGE_SIZES[mesh]`; `utils_fmri.apply` `np.vstack([left, right])`). The shipped ROI helper (§5) already returns indices in this exact space, so we never split hemispheres by hand.
  - `all_segments`: list aligned 1:1 to `preds` rows; each is a per-TR `segment.copy(offset=t, duration=TR)` carrying `.start` (absolute time on the input clock, in seconds), `.offset`, `.duration` (=`TR`=1.0), `.ns_events`. **x-axis source = `round(seg.start)`** (the same attribute `plotting/utils.get_clip` treats as absolute clip time).
- **TR = 1.0 s** (`neuro.frequency=1.0`, `Data.TR = 1/frequency`) → **1 prediction per second**.
- **⚠ POOLER IS LOCKED AT 100 BY THE CHECKPOINT — do NOT change `data.duration_trs`.** The model is rebuilt from `ckpt["model_build_args"]` (`demo_utils.py:229,235`), and `model_build_args["n_output_timesteps"]` was saved at train time (`pl_module.py:48-51`) = **100**. The forward pools to exactly `n_output_timesteps=100` frames per window (`model.py:104` `AdaptiveAvgPool1d(100)`). `predict()` then expands each loader segment into `floor(segment.duration/TR)` per-TR rows and indexes the pooled output with a `[keep]` mask of that length. **If `config.duration_trs ≠ 100`, row-count vs pooled-frame-count misalign → `[keep]` length mismatch / `ValueError`.** Therefore window length is **fixed at 100 s**; the only loader knob we use is `overlap_trs_train`. Empirically corroborated: the shipped demo runs a ~52 s clip with `duration_trs=100` and works — `list_segments` emits a full 100 s segment even when data is shorter, and the out-of-data TRs are dropped by `remove_empty_segments` (so short/partial clips do **not** crash and the real tail is always covered).
- **Native segmentation:** `Data.get_loaders` tiles the timeline via `ns.segments.list_segments(stride=(duration_trs−overlap_trs)·TR, duration=duration_trs·TR)`. For the `"all"` split that `predict()` uses, `overlap_trs = self.overlap_trs_train` (**not** `_val`). Shipped: `duration_trs=100`, `overlap_trs_train=0`, `batch_size=8`, `max_seq_len=1024`, `hidden=1152`, `depth=8`. With our `overlap_trs_train=20` ⇒ **overlapping 100 s windows, stride 80 s**. (`max_seq_len=1024` ⇒ head could take ~1024 TRs, but we never exceed 100 because of the pooler lock.)
- **Forward:** features `(B,L,D,T)` → per-modality `projectors` → concat → transformer (`time_pos_embed`, `max_seq_len=1024`) → `SubjectLayers` predictor → `(B, 20484, T)` → `AdaptiveAvgPool1d(100)`. `predict()` then expands each segment into per-TR rows and drops empty TRs (`remove_empty_segments=True`).

---

## 4. Long-video windowing + stitching (item 2 — the core feature)

**Goal:** score a 4–5 min (240–300 s) input and emit a continuous `(T_seconds, 20484)` timeline at 1 Hz, smooth across window seams, within ZeroGPU budget.

**Chosen approach = (B) configure overlap INTO tribev2's own segmenter (single `predict()`), then absolute-time stitch.** ONE `get_events_dataframe` + ONE `predict()` per Run ⇒ **one backbone load/free cycle** (cheapest GPU path). Rejected: (A) default non-overlapping tiling → hard 100 s seams; (C) external per-window `ffmpeg`-trim + N× `predict()` → N× backbone load/free = prohibitive — **kept as the documented FALLBACK** (see "Fallback" below + §7) if on-Space testing shows the overlap-config or `.start` realignment misbehaving.

**LOCKED parameters (decided — no further tuning needed pre-Space):**
- **`win_s = 100`** — *forced* by the checkpoint pooler (`n_output_timesteps=100`, §3); not a free choice. Also = training `duration_trs` ⇒ in-distribution.
- **`hop_s = 80` → `overlap_trs_train = 20` (20 s / 20 % overlap).** *Justification (HRF + receptive field):* the canonical hemodynamic response peaks ~5 s and returns toward baseline by ~20–24 s after an impulse; the BOLD signal a window predicts at time *t* is driven by stimulus up to ~20 s earlier. A 20 s overlap therefore gives every retained TR a full HRF-length of left-context inside its window and a 20 s blend zone — enough to absorb the worst-case HRF tail while keeping window count low. Tighter overlap (<~10 s) would leave HRF-tail discontinuities at seams; wider overlap costs GPU time for no added context.
- **`warmup_trim = 5` TRs** — drop/down-weight the **leading** 5 TRs of every window except window 0. *Justification:* those TRs have <1 HRF-width of left-context within their window and no `time_pos_embed` history (the encoder + the model's learned 5 s acquisition offset, `neuro.offset=5.0`, both want left-context). Trailing TRs need **no** trim (right-context isn't required to predict a causal BOLD response, and the final window's tail is the unique end-of-video signal — trimming it would drop real data).
- **normalization = per-window, per-vertex z-score over time** (NOT detrend). *Justification:* the training target was z-scored **per sample** + detrended (`config.yaml` `data.neuro.cleaning: standardize=zscore_sample, detrend=true`), so each window's absolute level/scale is arbitrary → naive concat = stepped seams. Per-window z-score cancels exactly that per-sample offset+scale. We avoid linear **detrend** because over a 100 s window it would flatten genuine slow dynamics (e.g. a real sustained-attention ramp); we only claim *relative* dynamics, and the per-metric global z-score in §5 handles the rest.
- **crossfade = trapezoidal overlap-add weighting** (flat=1 core, linear ramps across the 20 s overlap) folded into the stitch step 3c (below) — no separate pass.
- **smoothing** is applied later, per metric curve (§5), not to the 20484-wide timeline.

**Algorithm (`src/tribescore/windowing.py`):** the real path does NOT iterate windows itself — tribev2's segmenter does. The module's job is `plan_windows` (for the progress UI + the fallback) and `stitch` (the real work, pure-numpy, fully unit-testable).
1. `plan_windows(duration_s: float, win_s: int = 100, hop_s: int = 80) -> list[tuple[float,float]]` → window spans `[(0,100),(80,180),…]`; **force the last span to `(max(0.0, duration_s-100), duration_s)`** so the tail is covered by a full-length window; dedup spans whose start repeats. Used for the "Window k/N" progress label and (only) by the fallback path. Pure function — no model.
2. Real inference = ONE call: `preds, segs = model.predict(events)` with `overlap_trs_train=20` already set (§3). Build `abs_times = np.array([round(s.start) for s in segs])` (float seconds). `infer_fn` is the injected seam so tests pass `(preds, abs_times)` directly.
3. `stitch(preds: np.ndarray[(R,20484)], abs_times: np.ndarray[(R,)], *, warmup_trim:int=5, hop_s:int=80, win_s:int=100) -> tuple[np.ndarray[(T,20484)], np.ndarray[(T,)]]`:
   a. **Segment into windows** by detecting `abs_times` resets/contiguous runs (each run = one tribev2 window, ascending then a backward jump marks the next window's start). For each window assign a window-index `w` and an intra-window position `p` (0-based).
   b. **Per-window per-vertex z-score** over that window's rows: `(x - mean_t)/(std_t + 1e-6)`.
   c. **Weight** each row with a **trapezoidal crossfade** (flat=1 in the non-overlap core, linear ramp only across the `overlap = win_s - hop_s = 20` s edges): leading edge ramps `0→1` over the first `overlap` rows, trailing edge ramps `1→0` over the last `overlap` rows, `=1` between. This keeps `Σ weight ≈ 1` everywhere (no low-SNR seam band a whole-window Hann would create). Then apply **warm-up suppression**: for `w > 0`, force `weight = 0` for `p < warmup_trim` (the leading ramp simply starts at `p = warmup_trim`). Window 0 keeps its leading rows (no earlier window covers them). The final window's trailing rows keep `weight = 1` up to the last row (its tail is unique end-of-video signal — do not ramp it down).
   d. **Overlap-add** onto an integer-second grid `t_axis = arange(0, ceil(max(abs_times))+1)`: `timeline[t] = Σ_w weight·zscored_row  /  Σ_w weight` for all rows whose `round(abs_time)==t` (the weighted mean *is* the crossfade; non-overlap seconds have one contributor).
   e. Any grid second with **zero total weight** (e.g. an all-warmup-suppressed gap — shouldn't happen with 20 s overlap > 5 TR trim) is linearly interpolated from neighbors; assert this set is empty in tests.
   f. Returns `(timeline (T,20484), t_axis (T,))`, `T == duration_s` rounded.
4. **Fully unit-testable** with a synthetic `infer_fn` emitting overlapping windows of numpy ramps/sines at known `abs_times`: assert output shape `== (T,20484)`, `t_axis` strictly increasing 0..T-1, **seam continuity** (max |Δ| between adjacent stitched seconds across a known seam < tol), warm-up rows excluded, and duplicate seconds averaged. No model, no torch.

**Fallback (only if on-Space validation fails approach B):** `infer_window(clip_path)` = `ffmpeg`-trim the source to each `plan_windows` span (each exactly 100 s; last = `[D-100, D]`) → `get_events_dataframe` → `predict` → rows tagged `abs_time = window_start + p`. One `@spaces.GPU` per window (bounded duration, natural progress). Correct by construction (deterministic x-axis, no `list_segments`/`.start` assumptions, no overlap-dedup) at the cost of N× backbone loads + N× ASR. `stitch` is reused unchanged.

---

## 5. Brain-activity → metric curves (item 3)

tribev2 outputs **brain activity, not metrics** — we derive named curves by ROI reduction over fsaverage5, grounded in the neuroforecasting literature.

**ATLAS — DECIDED: HCP-MMP1 (Glasser 2016), via tribev2's OWN shipped helper — NOT nilearn Yeo/Schaefer.** `tribev2/utils.py` ships `get_hcp_labels()`, `get_hcp_roi_indices(rois, hemi="both", mesh="fsaverage5")`, `summarize_by_roi()`. `get_hcp_roi_indices` returns vertex indices **already in the 20484 output index space** (right hemi offset +10242 applied), strips the `L_`/`R_` prefix + `_ROI` suffix (so keys are bare Glasser names like `FEF`, `LIPv`, `TPOJ1`, `p32`, `10r`), pools both hemispheres under one key, and supports wildcards (`"IP*"`, `"*PFm"`). This removes the entire nilearn-atlas-fetch + manual vertex-alignment surface and guarantees index agreement with `preds`. (The demo notebook's "Yeo" mention is illustrative; the shipped, runnable code is Glasser. We use `combine=False` — fine-grained, canonical, verifiable names — not the coarse `combine=True` 22-region set, whose label strings are unverifiable offline.)

> **Dependency + first-run cost (handle in T-C/T-G/§7):** `get_hcp_labels` calls `mne.datasets.fetch_hcp_mmp_parcellation(accept=True)` and `mne.datasets.sample.data_path()` → **downloads the HCP-MMP annot + the MNE `sample` dataset (~1.5 GB) once.** Add **`mne`** to `requirements.txt`. The metric masks are **static** (input-independent), so **precompute once at startup and cache to `CACHE_DIR/roi_masks_hcpmmp1_fsaverage5.npz`**; load from cache thereafter. If the download fails, raise a clear startup error (do not silently fall back to wrong indices).

**Pipeline (`src/tribescore/metrics.py`):**
- `build_roi_masks(cache_dir: str, mesh: str = "fsaverage5") -> dict[str, np.ndarray]` — for each metric, `np.unique(np.concatenate([get_hcp_roi_indices(p, hemi="both", mesh=mesh) for p in PARCELS[metric] if p in valid]))`. **Startup guard:** `valid = set(get_hcp_labels(mesh=mesh, combine=False, hemi="both").keys())`; drop any parcel key not in `valid` and log it (atlas naming can vary by mne version). Persist/load via `.npz`.
- `to_metrics(timeline: np.ndarray[(T,20484)], masks: dict[str,np.ndarray]) -> dict[str, np.ndarray[(T,)]]` — per metric, per TR: **mean over that metric's vertices** → `raw[t]`; then **z-score over the full timeline** (`(raw-mean)/(std+1e-6)`) as the analytic series; then **Gaussian smooth σ = 2 s** (`scipy.ndimage.gaussian_filter1d`, truncate≈3 ⇒ ~13-tap kernel; precedent: repo `TemporalSmoothing(kernel_size=9)`). σ=2 s matches BOLD sluggishness while preserving peaks for click-to-seek (§6). Provide an optional **display** scaling to 0–100 via the repo's `robust_normalize(percentile=99)` for the summary strip only; the **z-scored** curve is the canonical/plotted series.
- All metrics share the z-scale ⇒ cross-metric comparison is in **relative** terms only (see caveat).

**v1 metric → exact Glasser parcels** (bare `combine=False` keys; reduction = mean over the union of these parcels' vertices per TR). Ship as a `PARCELS` dict constant:

| Metric (default ON) | Glasser parcels | Networks / rationale |
|---|---|---|
| **Attention** | `FEF, LIPv, LIPd, VIP, MIP, AIP, IP0, IP1, IP2, TPOJ1, TPOJ2, PGi, PGs, PFm, IFJa, IFJp, p9-46v, a9-46v, 9-46d, 46, 8C, i6-8, s6-8` | Dorsal attention (FEF + IPS complex) + Ventral attention (TPOJ/PGi/PGs/PFm + IFJ) + Frontoparietal control (DLPFC) |
| **Engagement / arousal** | `V1, V2, V3, V4, V3A, V3B, V6, V6A, MT, MST, A1, LBelt, MBelt, PBelt, A4, A5, STSdp, STSvp, STSda, STSva, TE1p, TE2p` | Sensory drive (early visual + motion + early/assoc. auditory) + associative integration (STS) |
| **Virality (proxy)** | `10r, 10v, 10d, 10pp, p32, s32, a24, d32, 25, OFC, pOFC, 11l, 13l, 9m` | vmPFC / mOFC / pgACC / mPFC cortical **value** signal |

| Metric (default OFF, toggleable) | Glasser parcels | Rationale |
|---|---|---|
| **Language / semantic load** | `44, 45, IFSa, STSdp, STSvp, STGa, TE1a, A5, PSL, SFL, 55b` | Core language network (text + audio driven) |
| **Self-relevance / DMN** | `7m, POS2, v23ab, d23ab, 31pv, 31pd, RSC, PCV, 9m, 10r, PGs, PGi` | Default-mode / self & social relevance — secondary sharing cue |

(*Declined:* "valence" and "memorability" — no clean, defensible cortical-ROI mapping on this model; including them would over-claim.)

**Neuroforecasting citations (virality proxy) — pin in README + UI tooltip:** Genevsky & Knutson 2015; **Genevsky, Yoon & Knutson 2017 (J. Neurosci.)** (NAcc + MPFC forecast aggregate market behavior beyond self-report); **Scholz et al. 2017 (PNAS)** and **Baek/Falk "brain-as-predictor"** (value system incl. VMPFC predicts population-level sharing/virality); **Doré et al. 2019**.

> **Honesty caveat (must surface in UI + README):** "Virality" is a **research proxy** from cortical value-region activity — *not* a guarantee of going viral. `facebook/tribev2` is **cortical-only** (no ventral striatum/NAcc — the strongest neuroforecasting node), so the vmPFC/mPFC signal is the validated *complement*, not a substitute. Because the training target was per-sample z-scored + detrended, **only relative temporal dynamics are interpretable — absolute "scores" are meaningless.** The summary strip therefore reports **relative** peaks (z-units / percentile), and every metric is labeled a proxy with its underlying ROI viewable.

---

## 6. Gradio UI + frontend-design (item 4)

Applying the **frontend-design** skill: a distinctive, subject-grounded identity — **not** default Gradio Blocks, and **not** one of the three AI-default looks.

**Design direction — "Cortical Observatory":** a calm, clinical-but-warm scientific instrument that reads a video and shows how the average brain would respond over time.

**Token system:**
- **Color (deep slate + BOLD-colormap accents):** `bg #0E1116`, `panel #161B22`, `ink #E8EDF2`, `dim #8A93A0`, `border #232A33`; **accent (activation hot) `#FFB454` amber**, **counter-accent (cool) `#36D1C4` cyan** — the two ends of an fMRI activation colormap, used with restraint. Metric curves use a colorblind-safe scientific ramp (viridis/cividis sample).
- **Type (3 roles):** display = **Space Grotesk** (wordmark + section headers, sparingly); body = **Inter** (Gradio default chain); **data/timestamps = a tabular-figure mono (JetBrains Mono / IBM Plex Mono)** — *justified*: a timeline needs aligned tabular numerals.
- **Layout:** two zones. Left rail = the **mode-switcher (Video / Audio / Text tabs)** + per-mode input widget + metric toggles + Run + quota banner (Space-only). Right hero = the synchronized readout.

**Signature element (the one memorable thing):** a **synchronized multi-channel metric TIMELINE** — stacked translucent curves (attention / virality / engagement …) on a shared 1 Hz time axis, with a **draggable playhead locked to the video scrubber**; hover → crosshair + per-metric values; **click a peak → seek the media to that moment**. *"Scrub the media, watch the brain; click a spike, jump there."* Built with **Plotly** (hover/zoom/click→seek), custom-styled (not default plot chrome). Below: a summary strip (peak/mean per metric).

> **Known frontend risk + DECIDED fallback (so coders aren't blocked):** Gradio's `gr.Video` exposes **no reliable programmatic seek API**. Plan: render the media via a **custom `gr.HTML` `<video id="tm-video">`** plus a tiny JS handler; Plotly `plotly_click` → JS sets `document.getElementById('tm-video').currentTime = t` (and a draggable playhead line via `relayout`). If the custom-component route proves flaky, **fallback = display `t = NN s` on click + a "Jump" control that remounts the `<video>` with a `#t=NN` media-fragment URL** (universally supported, no JS state). Treat the live-locked playhead as best-effort; the click→timestamp→jump path is the guaranteed floor.

**States (quality floor):**
- **Empty:** inviting panel + "Try a sample clip" CTA + one-line "what this does".
- **Loading:** real per-window progress ("Window 2/4 · extracting V-JEPA2 features…"), not a bare spinner.
- **Error (plain voice, actionable):** e.g. *"This clip is 7 min; the max is 5. Trim it and try again."*; gated case: *"Model access is pending approval — check back soon."*
- **Result:** the timeline + summary.
- Responsive to mobile (zones stack), visible keyboard focus, `prefers-reduced-motion` respected. Boldness spent only on the timeline; everything else quiet.

**Code shape (mirrors qwen-image-editor):** `theme.py` (`gr.themes.Base` + `PALETTE` tokens + CSS string), `ui.py` (per-mode builders returning component dicts, `info=` tooltips), `app.py` wires events. A frontend-design review pass critiques the rendered result before sign-off.

---

## 7. ZeroGPU Space architecture (item 5 — single Space)

- **Hardware:** ZeroGPU (H200 70 GB slice). GPU attaches only inside `@spaces.GPU`; the `spaces` runtime registers CUDA at **startup**, so we **build the model at module import** (downloads 708 MB ckpt only — backbones load lazily at first `predict()`), mirroring cbensimon + qwen eager-startup. The module-level model persists across `@spaces.GPU` calls.
- **`@spaces.GPU` boundary (v1):** approach B = **one decorated `predict()` call per Run** (covers all internal overlapping windows in a single GPU session). Set **`@spaces.GPU(duration=480)`** for v1 (matches cbensimon's reservation for one ~52 s clip; a 5-min input does ~4 internal windows of feature-extraction in that session). **First action in T-H: measure actual GPU-seconds per 60 s of input**, then trim `duration` from data.
- **⚠ ZeroGPU duration ALLOWANCE is the #1 unknown (R1, §10):** `duration=480` worked on **cbensimon's** account; the free ZeroGPU tier caps much lower (~120 s). **If techfreakworm's account caps below what one whole-video `predict()` needs, approach B won't fit a single call → switch to the §4 Fallback (one `@spaces.GPU(duration≈200)` PER window) or the feature-cache split below.** Confirm the account's allowance before committing (T-H, before tuning).
- **GPU-budget fallback ladder:** (1) §4 per-window `ffmpeg` path = N bounded `@spaces.GPU` calls (each loads backbones once). (2) deeper fallback (strategy C) if even that is tight: one bounded call extracts + caches features, then the cheap 8-layer head runs per window across short calls. Implement (1) first; (2) only if needed.
- **Memory:** backbones load→free sequentially (LLaMA ~6.5 GB, V-JEPA2 ~3 GB, W2V-BERT ~2.3 GB, DINOv2 ~1.2 GB); peak ≈ largest single backbone + activations + head (0.7 GB) ≪ 70 GB. **Time, not memory, is the binding constraint.**
- **ASR runtime dependency (whisperx via `uvx` — easy to miss):** `ExtractWordsFromAudio` shells out to **`uvx whisperx … --model large-v3 …`** as a subprocess (`eventstransforms.py:111-135`). The Space image therefore needs **`uv` on PATH** (add to `apt`/`packages` or `pip install uv`), plus first-run network to fetch whisperx + download whisper `large-v3` (~3 GB) and the wav2vec align model. Cache under `CACHE_DIR`. The **`audio_only` interim path skips whisperx entirely** (and Llama) → much faster for pre-approval validation.
- **MNE atlas data:** `mne` required (§5); set `MNE_DATA`/subjects-dir under `CACHE_DIR` and **pre-warm `build_roi_masks` at startup** so the ~1.5 GB `sample` + HCP-MMP annot download happens once and is cached to `.npz`.
- **Gated model handling:** `HF_TOKEN` set as a **Space secret** (token from techfreakworm, Llama license accepted). `transformers`/`huggingface_hub` auto-use it. Set `HF_HUB_ENABLE_HF_TRANSFER="0"` (whisperx/huggingface_hub compat, per cbensimon).
- **Serving:** `spaces`-based Space-only guard for the quota banner; `.queue(default_concurrency_limit=1)` (one heavy GPU task at a time); `.launch(show_error=True, ssr_mode=False)`.
- **Caching:** `CACHE_DIR` = persistent `/data` if writable, else `./cache`; reused for the ckpt, whisper/whisperx, MNE data, and the ROI `.npz`.

---

## 8. Repo layout + license (item 6) — scaffolded ✅

```
tribe-manager/
├── app.py                      # Gradio entrypoint (mode tabs, eager startup on Spaces)
├── src/tribescore/
│   ├── inference.py            # TribeModel load + per-window predict wrapper (guarded import)
│   ├── windowing.py            # plan_windows + stitch  (synthetic-testable)
│   ├── metrics.py              # ROI masks + brain→metric curves
│   └── plotting.py             # Plotly synchronized timeline
├── theme.py / ui.py            # frontend-design "Cortical Observatory" (to add)
├── tests/                      # pytest, synthetic infer_fn — NO model
├── requirements.txt            # cbensimon-pinned (tribev2[plotting] git+, torch==2.6.0, gradio==6.11.0, spaces, nilearn, plotly, numpy)
├── README.md                   # HF Space front-matter + methodology + attribution
├── LICENSE (Apache-2.0)        # our code
├── NOTICE                      # tribev2 = CC-BY-NC-4.0; Llama-3.2 license; backbone licenses
└── docs/PLAN.md                # this file
```
**License posture:** our code Apache-2.0; deployed demo is **non-commercial** (tribev2 CC-BY-NC) — stated in README + NOTICE + UI footer.

---

## 9. Turnkey task breakdown (item 7 — coding agents execute directly on approval + access)

Dependency-ordered; each unit has a clear interface so they parallelize. **All build locally / test on synthetic data; real validation on the Space.**

Exact signatures (coders implement to these; all heavy calls behind the injected `infer_fn` so tests use pure numpy):

- **T-A `inference.py`** —
  - `load_model(cache_dir: str) -> "TribeModel"`: `TribeModel.from_pretrained("facebook/tribev2", cache_folder=cache_dir, device="auto", config_update={"data.overlap_trs_train": 20})`. Module-level singleton; **guarded import** of `tribev2`/`torch` so the file imports locally without them (raise a clear error only if `load_model` is actually called off-Space).
  - `run_inference(model, mode: str, src_path: str, *, audio_only: bool=False) -> tuple[np.ndarray, np.ndarray]`: normal path `events = model.get_events_dataframe(**{f"{mode}_path": src_path})` (mode ∈ `{"video","audio","text"}`). **Interim `audio_only`** (video/audio modes only; skips ASR + Llama): build the one-row event DataFrame exactly as `get_events_dataframe` does — `pd.DataFrame([{"type": "Video"|"Audio", "filepath": src_path, "start": 0, "timeline": "default", "subject": "default"}])` — then `events = get_audio_and_text_events(df, audio_only=True)`. Either way: `preds, segs = model.predict(events)`; return `preds (R,20484)`, `abs_times = np.array([round(s.start) for s in segs], float)`. **Caller wraps this in `@spaces.GPU(duration=480)`.**
  - (Fallback only) `infer_window(model, clip_path: str, window_start: float) -> tuple[np.ndarray,np.ndarray]` per §4 Fallback.
- **T-B `windowing.py`** — `plan_windows(duration_s, win_s=100, hop_s=80)` and `stitch(preds, abs_times, *, warmup_trim=5, hop_s=80, win_s=100)` **exactly per §4** (signatures, z-score, trapezoidal weights, overlap-add, zero-weight assert). **Unit tests with synthetic `infer_fn`** (overlapping ramps/sines) asserting shape, monotone `t_axis`, seam continuity, warm-up exclusion, duplicate averaging. *(core; no model)*
- **T-C `metrics.py`** — `PARCELS: dict[str, list[str]]` constant (§5 tables); `build_roi_masks(cache_dir, mesh="fsaverage5") -> dict[str,np.ndarray]` (validate keys against `get_hcp_labels(...).keys()`, `.npz` cache, startup pre-warm); `to_metrics(timeline, masks) -> dict[str, np.ndarray]` (ROI mean → full-timeline z-score → `gaussian_filter1d(σ=2)`); `summary(curves) -> dict[str, dict]` (peak/mean in z-units + percentile). Synthetic tests (random `timeline`, hand-made `masks`).
- **T-D `plotting.py`** — `timeline_figure(t_axis: np.ndarray, curves: dict[str,np.ndarray], *, selected: list[str]) -> go.Figure`: stacked translucent Plotly traces on a shared 1 Hz axis, crosshair hover, click→seek hook (§6), styled per theme. **Pure function** of `(t_axis, curves)` — synthetic tests assert trace count == len(selected) and x-range == t_axis range.
- **T-E `theme.py` + `ui.py`** — frontend-design "Cortical Observatory" tokens + CSS + 3-mode builders + states (empty/loading/error/result), `info=` tooltips, custom `<video>` HTML per §6.
- **T-F `app.py`** — per mode: input → **validate `0 < duration_s ≤ 300`** (and a friendly note if `< ~10 s`) → `@spaces.GPU` `run_inference` → `stitch` → `to_metrics` → `timeline_figure`; `gr.Progress` "Window k/N" from `plan_windows`; quota banner; `queue(default_concurrency_limit=1)`; `launch(show_error=True, ssr_mode=False)`; eager `load_model` + `build_roi_masks` at import; optional `audio_only` debug toggle (default off).
- **T-G packaging** — `requirements.txt` (`tribev2[plotting] @ git+…`, `torch==2.6.0`, `gradio==6.11.0`, `spaces`, `nilearn`, **`mne`**, `plotly`, `numpy`, `scipy`); ensure **`uv`** in the image (whisperx); env `HF_HUB_ENABLE_HF_TRANSFER=0`, `MNE_DATA`→`CACHE_DIR`; finalize README/NOTICE; set `HF_TOKEN` secret.
- **T-H deploy + validate** — push to the ONE Space; **(1) confirm ZeroGPU duration allowance (R1)**; (2) `audio_only` end-to-end smoke test (no Llama) — exercises windowing/stitch/metrics now; (3) measure GPU-time/60 s; (4) verify `.start`-based x-axis matches the media clock (else switch to §4 Fallback); (5) full end-to-end across all 3 modes once Llama approved.

---

## 10. Risks & mitigations

1. **(R1, highest) ZeroGPU per-call duration ALLOWANCE unknown on techfreakworm's account** → a whole-video `predict()` (approach B) may exceed the account ceiling (free tier ~120 s; cbensimon used 480). **Mitigation:** confirm allowance FIRST on-Space (T-H); if too low, switch to §4 Fallback (per-window `@spaces.GPU`) then the feature-cache split. Do not tune `duration` before this is known.
2. **whisperx-via-`uvx` subprocess + MNE `sample` dataset = hidden deps + heavy first run** → image needs `uv`; first run downloads whisper `large-v3` (~3 GB), wav2vec align model, and MNE `sample` (~1.5 GB). **Mitigation:** `uv` + `mne` in image, cache to `CACHE_DIR`, pre-warm ROI `.npz` at startup, use `audio_only` (skips whisperx + Llama) for interim validation.
3. **`.start`-based x-axis realignment is unverified offline** (neuralset not installable here) → if absolute times don't match the media clock, the timeline x-axis is wrong. **Mitigation:** verify on first Space run (T-H step 4); deterministic §4 Fallback (`window_start + p`) eliminates the assumption entirely.
4. **Llama gate pending Meta** → blocks the **text-feature path** of live inference only; the Space boots without it, and the `audio_only` path validates the full heavy pipeline (V-JEPA2 + DINOv2 + W2V-BERT) end-to-end now. `modality_dropout=0.3` means the model tolerates the missing text modality (degraded but valid).
5. **"Virality" is a proxy / cortical-only / relative-only** → label as research proxy everywhere; state no-NAcc limitation; report **relative** (z-unit) values only; show underlying ROI.
6. **Window seams from overlapping windows** → per-window z-score + trapezoidal overlap-add + leading warm-up suppression + per-curve σ=2 s smoothing (all per §4).
7. **ASR English-only** (`ExtractWordsFromAudio.language="english"`, hard-coded) → non-English audio yields wrong/empty transcripts and silently degrades the text path (no error). **Mitigation:** README + UI note; consider exposing language later (out of v1 scope).
8. **`duration_trs` must stay 100** (pooler locked by ckpt, §3) → never expose/auto-tune it; changing it crashes `predict()`. Documented; no code path sets it.
9. **ZeroGPU per-user quota** → quota banner + `concurrency=1`; Video heaviest, Text lightest.
10. **Domain shift** → trained on naturalistic movies + English narrative speech (Algonauts2025/CNeuroMod Friends, Lahner2024, Lebel2023, Wen2017); fast-cut UGC / music videos / animation degrade reliability. README methodology note.

---

## 11. Decisions (defaults locked; operator may override at GO)

No open *high-level* questions remain — every prior open question is resolved to a concrete default with rationale, so coders implement with **zero further high-level decisions**. Items marked *(operator may override at GO)* are genuine preference calls that do not block implementation.

1. **Metric set — DECIDED.** Ship **Attention, Engagement/Arousal, Virality (proxy)** ON by default; **Language/semantic-load** and **Self-relevance/DMN** as toggles (default OFF). Exact Glasser parcels in §5. *Declined:* valence, memorability (no defensible cortical-ROI mapping — would over-claim). *(operator may override which curves are shown at GO.)*
2. **Window / hop — DECIDED & LOCKED (not a preference).** `win_s=100` is **forced** by the checkpoint pooler (§3); shorter windows are **unsupported** (crash `predict()`), not merely lower-fidelity. `hop_s=80` / `overlap_trs_train=20` per the HRF justification in §4. No speed-vs-fidelity dial exists here; speed comes from GPU-time tuning (§7), not window length.
3. **Max input length — DECIDED.** Hard cap **300 s (5 min)**; reject longer with the friendly §6 error. Short clips work (≥ ~1 window; `<~10 s` shows an advisory note). *(operator may raise the cap; GPU-time grows ~linearly and re-validates against R1.)*
4. **UI identity — DEFAULT.** "Cortical Observatory" (deep-slate + amber/cyan, §6). *(operator may override at GO.)*
5. **Space name — DEFAULT.** `techfreakworm/tribev2-brain-timeline`. *(operator may override at GO.)*
6. **Interim validation before Llama — DECIDED.** Wire the `audio_only` path (skips ASR+Llama) as a debug toggle so the full heavy pipeline + windowing + metrics are validated on-Space *before* Meta approval (T-H step 2).

---

## 12. Sequence to first working milestone

1. Operator reviews this plan (+ confirms or overrides the §11 locked defaults).
2. Coders execute T-A…T-G locally (synthetic-tested) — *independent of Llama gate*.
3. Deploy skeleton → the ONE Space; confirm UI boots + model object builds (downloads ckpt only; no Llama needed yet).
4. **Interim on-Space validation — BEFORE Meta approval (no Llama):** run the **`audio_only` path** (skips ASR + Llama) end-to-end on the Space for a 4–5 min clip. This exercises the *full heavy GPU pipeline* (V-JEPA2 + DINOv2 + W2V-BERT) plus **windowing → `stitch` → `to_metrics` → timeline**, and is where we **(a) confirm the ZeroGPU per-call duration allowance (R1), (b) measure GPU-seconds per 60 s of input and trim `@spaces.GPU(duration=…)`, (c) verify the `.start`-based x-axis matches the media clock** (else flip to the §4 `ffmpeg` fallback). De-risks everything except the text modality while Meta review is pending.
5. **On Meta approval:** enable the full multimodal path; run end-to-end across all 3 modes (Video / Audio / Text), re-check GPU-time, final timeline validation → **first working milestone**.

---

## 13. Brain review & sign-off

Adversarial review by `brain` (max rigor + sequential-thinking) against tribev2 source (`demo_utils.py`, `model.py`, `main.py`, `pl_module.py`, `eventstransforms.py`, `utils.py`, `utils_fmri.py`, `tribe_demo.ipynb`) and `fb_config.yaml`. Changes made:

- **§3 — Keystone correction (pooler lock).** Documented that `n_output_timesteps=100` is baked into the checkpoint (`pl_module.py:48-51` → `demo_utils.py:229,235` → `model.py:104`), so **`data.duration_trs` must stay 100** — changing it misaligns `predict()`'s `[keep]` mask vs the pooled 100 frames and raises `ValueError`. Window length is therefore *fixed*, not a tuning choice. Also corrected: `average_subjects=True` is set unconditionally by `from_pretrained` (we don't pass it); the live overlap knob is `overlap_trs_train` (used by the `"all"` split), set via `config_update={"data.overlap_trs_train": 20}`; x-axis source is `round(seg.start)`.
- **§4 — Deleted the "pending brain-spec refinement" caveat by deciding everything.** Locked `win_s=100` (forced), `hop_s=80`/`overlap_trs_train=20` (HRF ≤~20 s justification), `warmup_trim=5` leading-only (left-context justification), per-window per-vertex **z-score** (not detrend; per-sample-zscore artifact justification), and a **trapezoidal crossfade** overlap-add (flat=1 core, 20 s linear ramps; avoids the low-SNR seam a whole-window Hann would cause). Gave exact `plan_windows`/`stitch` signatures + the 6-step stitch algorithm + the partial/short-clip non-crash reasoning (empirically corroborated by the working 52 s demo). Kept the external `ffmpeg` per-window method as an explicit, fully-specified fallback.
- **§5 — Replaced the nilearn/Yeo placeholder with the in-repo HCP-MMP1 (Glasser) atlas.** Pinned `tribev2.utils.get_hcp_roi_indices` (indices already in the 20484 space, rh +10242, wildcard support); exact bare-parcel lists for Attention / Engagement / Virality (+ optional Language, Self-relevance); normalization = full-timeline **z-score**, smoothing = Gaussian **σ=2 s**; corrected citations (added Scholz 2017 PNAS, Genevsky 2017); flagged the `mne` dependency + ~1.5 GB `sample` download + `.npz` startup pre-cache; reaffirmed cortical-only / relative-only interpretation.
- **§6 — Added the `gr.Video`-has-no-seek risk with a decided fallback** (custom `<video>` + Plotly-click JS `currentTime`, or `#t=` media-fragment "Jump").
- **§7 — Reconciled to approach B** (one `@spaces.GPU(duration=480)` per Run), made **R1 (ZeroGPU duration allowance)** explicit with the fallback ladder, and documented the **whisperx-via-`uvx` subprocess** (needs `uv` in the image + ~3 GB model download) and **MNE data** caching.
- **§9 — Exact signatures** for every task (`load_model`, `run_inference` incl. the precise `audio_only` branch, `plan_windows`, `stitch`, `build_roi_masks`, `to_metrics`, `summary`, `timeline_figure`), with the duration-validation and startup-prewarm wired into T-F/T-H.
- **§10 — Risk register (10 items, ranked):** R1 ZeroGPU duration allowance (#1); whisperx-via-`uvx` (needs `uv` + ~3 GB) and MNE/HCP-MMP (~1.5 GB) first-run downloads; `.start` x-axis unverified-offline; Llama-gate (text path only); proxy/cortical-only/relative-only; trapezoidal-stitch seam control; English-only ASR silent-degrade; `duration_trs` lock; per-user quota; domain-shift (UGC/music/animation). Term aligned to **trapezoidal** overlap-add (matches §4).
- **§11 — All open questions removed → locked decisions** (metric set; window 100 s fixed-by-pooler / hop 80 s; 300 s cap; UI "Cortical Observatory"; Space name `techfreakworm/tribev2-brain-timeline`). Zero open high-level questions remain; preference-only items tagged *(operator may override at GO)*.
- **§12 — Added the interim on-Space `audio_only` validation step** (step 4): real GPU validation of the heavy pipeline + windowing/stitch/metrics + duration-allowance + GPU-time measurement + x-axis check **before** Meta approval; full multimodal validation moved to step 5 (post-approval).

**Residual risks (all have a concrete mitigation in-doc; none block local T-A…T-G):**
1. **R1 — ZeroGPU per-call duration allowance** on techfreakworm's account (free tier ~120 s vs the ~480 s approach-B may need). Verify first on-Space (§12 step 4 / §9 T-H); fallback ladder in §7.
2. **`.start`-based x-axis** unverifiable offline (neuralset not installable). Verify §12 step 4; deterministic §4 `ffmpeg` fallback removes the assumption.
3. **whisperx/`uvx` + MNE `sample`** heavy first run / image deps. Mitigated by `uv`+`mne` in the image (§9 T-G — the authoritative requirements list; the §8 layout box is an illustrative snapshot), caching to `CACHE_DIR`, `.npz` pre-warm, and the `audio_only` interim path.
4. **Validity** of the metric proxies (relative-only, cortical-only) — an inherent model limitation, surfaced honestly in UI/README, not a build blocker.

**Verification:** re-read the whole file end-to-end; §0–§9 left as-is (lead-merged) and confirmed internally consistent with §10–§13; all cross-references resolve; no placeholders, TODOs, or undecided high-level choices remain. A coder can implement T-A…T-H to the exact signatures in §9 with no further high-level decisions.

BRAIN SIGN-OFF v2: docs/PLAN.md is gap-free and implementation-ready.