typoverflow Claude Opus 5 (1M context) commited on
Commit
65895ab
·
1 Parent(s): 738325f

Take the dataset repo as user input and stream windows on demand

Browse files

The repo is now typed into the app rather than baked in via DATASET_REPO,
which becomes only the prefilled default. Accepts owner/name, a revision
suffix, or a pasted dataset URL.

Generalises the layout assumptions so an arbitrary repo renders: the manifest
and metrics files are optional, the window list falls back to listing
windows/ one level deep, per-window metadata is backfilled from meta.json,
the camera count is inferred from the frame aspect ratio, and window length
and critical-frame markers are no longer fixed at 33/16.

Storage is now per session and bounded, so large datasets are never pulled
whole: loading a repo fetches only the index, window files arrive when the
window is opened, at most MAX_WINDOWS_ON_DISK stay resident, and the
session's directory is deleted when the tab closes. A janitor reaps idle
sessions and directories orphaned by an earlier process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (2) hide show
  1. README.md +51 -29
  2. app.py +507 -175
README.md CHANGED
@@ -13,51 +13,73 @@ short_description: Frame-locked viewer for rollouts vs. ground truth
13
 
14
  # Rollout vs. ground truth
15
 
16
- Side-by-side viewer for forward-dynamics world-model rollouts. Reads windows straight from a
17
- Hugging Face **dataset** repo — the Space holds no data of its own.
 
18
 
19
- Default dataset: [`OneWorld-AI/abc-critical-window-rollout`](https://huggingface.co/datasets/OneWorld-AI/abc-critical-window-rollout)
20
- (300 held-out ABC bimanual-YAM windows, each a 33-frame slice centred on a gripper critical frame).
21
 
22
  ## What it shows
23
 
24
  - **Playback** — GT and generated rendered into one video, so the two panels stay frame-locked
25
  instead of drifting apart the way two independent players do. Optional `|GT − generated|`
26
- difference panel. Playback fps is adjustable (the 33-frame window is only ~1 s at native 30 fps).
27
- - **Frame stepper** — a single frame from both videos side by side, defaulting to frame 16
28
- (the critical frame).
29
- - **Driving actions** — the `[33, 14]` action chunk fed to the model, split into grippers /
30
- left-arm joints / right-arm joints, with the critical frame marked.
31
- - **All windows** — sortable table of every window with its PSNR; click a row to load it.
32
 
33
- Filter by task, gripper edge (grasp / release) and arm; sort by critical-frame PSNR to jump
34
- straight to the hardest or easiest cases.
35
 
36
  ## Expected dataset layout
37
 
38
  ```
39
- windows_manifest.json # list of window dicts: episode_id, task, edge_type, arm,
40
- # start_frame, critical_frame, amplitude, n_frames, ...
41
- metrics_summary.json # list of {id, task, edge, arm, full_mse, full_psnr,
42
- # crit_mse, crit_psnr}
43
- windows/<episode_id>/gt.mp4 # 33 frames, 3 cameras vstacked (top / L wrist / R wrist)
44
- windows/<episode_id>/generated.mp4 # model rollout for the same slice, same geometry
45
- windows/<episode_id>/actions.json # [33, 14] = [L arm 6, L grip, R arm 6, R grip]
46
- windows/<episode_id>/meta.json
47
  ```
48
 
49
- Any dataset matching this layout works. Black letterbox bars are trimmed automatically, and the
50
- per-camera split is derived from the frame height, so other stacked-camera resolutions are fine.
 
 
 
51
 
52
- ## Configuration
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
 
54
- | Variable | Purpose |
55
- |---|---|
56
- | `DATASET_REPO` | Dataset repo to read (default `OneWorld-AI/abc-critical-window-rollout`). |
57
- | `HF_TOKEN` | **Required if the dataset is private.** Add it under *Settings → Variables and secrets → Secrets*. |
58
 
59
- Only `windows_manifest.json` and `metrics_summary.json` are fetched at startup; videos and actions
60
- are downloaded per window on demand and cached, as are the composed comparison videos.
 
 
 
 
 
61
 
62
  ## Running locally
63
 
 
13
 
14
  # Rollout vs. ground truth
15
 
16
+ Side-by-side viewer for forward-dynamics world-model rollouts. Type a **dataset repo id** into
17
+ the box at the top and it renders that repo — the Space holds no data of its own and is not tied
18
+ to any one dataset.
19
 
20
+ Accepted forms: `owner/name`, `owner/name@revision`, or a pasted `https://huggingface.co/datasets/…`
21
+ URL. `DATASET_REPO` sets what is prefilled at startup.
22
 
23
  ## What it shows
24
 
25
  - **Playback** — GT and generated rendered into one video, so the two panels stay frame-locked
26
  instead of drifting apart the way two independent players do. Optional `|GT − generated|`
27
+ difference panel. Playback fps is adjustable (a 33-frame window is only ~1 s at 30 fps).
28
+ - **Frame stepper** — a single frame from both videos side by side, defaulting to the critical
29
+ frame when the dataset marks one.
30
+ - **Driving actions** — the action chunk fed to the model. A 14-D chunk is split into grippers /
31
+ left-arm / right-arm; any other width is plotted as raw channels.
32
+ - **All windows** — sortable table of every window; click a row to load it.
33
 
34
+ Filter by task, gripper edge and arm; sort by critical-frame PSNR to jump to the hardest or
35
+ easiest cases.
36
 
37
  ## Expected dataset layout
38
 
39
  ```
40
+ windows_manifest.json # optional: list of window dicts (episode_id, task,
41
+ # edge_type, arm, start_frame, critical_frame, ...)
42
+ metrics_summary.json # optional: [{id, task, edge, arm, full_mse, full_psnr,
43
+ # crit_mse, crit_psnr}, ...]
44
+ windows/<episode_id>/gt.mp4 # required
45
+ windows/<episode_id>/generated.mp4 # required, same geometry and length as gt.mp4
46
+ windows/<episode_id>/actions.json # optional: [T, D] driving actions
47
+ windows/<episode_id>/meta.json # optional: per-window metadata
48
  ```
49
 
50
+ Only the two mp4s are required. With no `windows_manifest.json` the window list is discovered by
51
+ listing `windows/` one level deep, and per-window metadata is read from each `meta.json` as you
52
+ open it. Videos may stack **N cameras vertically** — N is inferred from the frame aspect ratio,
53
+ and the camera selector adapts. Black letterbox bars are trimmed automatically, identically for
54
+ GT and generated.
55
 
56
+ ## Storage model
57
+
58
+ Nothing is downloaded at build time, and no repo is ever pulled whole — important for datasets
59
+ too large to fit on the Space.
60
+
61
+ - Loading a repo fetches only the manifest and metrics JSONs (a few hundred KB), or one directory
62
+ listing when there is no manifest.
63
+ - A window's mp4s, `actions.json` and `meta.json` are fetched **the first time that window is
64
+ opened**, never in bulk.
65
+ - Each browser session gets its own scratch directory. At most `MAX_WINDOWS_ON_DISK` windows stay
66
+ resident; beyond that the least recently used are deleted. Decoded frames are capped at 2
67
+ windows in RAM and composed strips at 1.
68
+ - The whole directory is deleted when the session ends. A janitor also reaps sessions idle past
69
+ `SESSION_TTL` and directories orphaned by an earlier process, so nothing accumulates if a tab
70
+ dies without a clean shutdown.
71
 
72
+ The sidebar shows live occupancy (`cached this session: n/N windows`).
73
+
74
+ ## Configuration
 
75
 
76
+ | Variable | Default | Purpose |
77
+ |---|---|---|
78
+ | `DATASET_REPO` | `OneWorld-AI/abc-critical-window-rollout` | Repo prefilled in the input box. |
79
+ | `HF_TOKEN` | — | **Required for private datasets.** Set under *Settings → Variables and secrets*. |
80
+ | `MAX_WINDOWS_ON_DISK` | `12` | Windows kept per session before LRU eviction. |
81
+ | `MAX_DISCOVER` | `5000` | Cap on windows listed when a repo has no manifest. |
82
+ | `SESSION_TTL` | `3600` | Seconds of idleness before a session's storage is reclaimed. |
83
 
84
  ## Running locally
85
 
app.py CHANGED
@@ -1,47 +1,68 @@
1
  """
2
- Side-by-side viewer for Cosmos forward-dynamics rollouts vs. ground truth.
3
 
4
- Data lives in a separate HF dataset repo laid out as:
5
 
6
- windows_manifest.json # 300 window definitions
7
- metrics_summary.json # per-window GT-vs-generated MSE / PSNR
8
- windows/<episode_id>/gt.mp4 # 33 frames, 224x672 (3 cams vstacked)
9
- windows/<episode_id>/generated.mp4 # same slice, model rollout
10
- windows/<episode_id>/actions.json # [33, 14] driving action chunk
11
- windows/<episode_id>/meta.json
12
 
13
- Set DATASET_REPO to point at a different dataset. If the dataset is private,
14
- the Space needs an HF_TOKEN secret with read access to it.
 
 
 
 
 
 
 
 
 
15
  """
16
 
17
  from __future__ import annotations
18
 
19
- import functools
20
  import json
21
  import os
 
 
22
  import tempfile
 
 
 
 
23
  from pathlib import Path
24
 
25
  import cv2
26
  import gradio as gr
27
  import imageio_ffmpeg
28
  import numpy as np
29
- from huggingface_hub import hf_hub_download
30
  from matplotlib.figure import Figure
31
 
32
- DATASET_REPO = os.environ.get("DATASET_REPO", "OneWorld-AI/abc-critical-window-rollout")
33
  HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN") or None
34
 
35
- CACHE_DIR = Path(tempfile.gettempdir()) / "wmviz_render"
36
- CACHE_DIR.mkdir(parents=True, exist_ok=True)
37
 
38
- # Each video stacks three 224x224 camera feeds vertically.
39
- VIEWS = ["All cameras", "Top", "Left wrist", "Right wrist"]
40
- VIEW_ROW = {"Top": 0, "Left wrist": 1, "Right wrist": 2}
 
 
 
 
 
 
 
41
 
42
  FONT = cv2.FONT_HERSHEY_SIMPLEX
43
  HDR, FTR, SEP = 26, 26, 4
44
- SCALE = 2 # native frames are only 224 wide; upscale so overlays stay legible
45
 
46
  GT_COLOR = (130, 225, 140)
47
  GEN_COLOR = (120, 180, 255)
@@ -56,62 +77,281 @@ SORTS = [
56
  ]
57
 
58
 
59
- # --------------------------------------------------------------------------- data
60
-
61
-
62
- def fetch(rel_path: str) -> str:
63
- """Download one file from the dataset repo (hf_hub_download caches on disk)."""
64
- return hf_hub_download(
65
- repo_id=DATASET_REPO,
66
- filename=rel_path,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
  repo_type="dataset",
68
  token=HF_TOKEN,
69
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
 
 
 
 
71
 
72
- @functools.lru_cache(maxsize=1)
73
- def load_index() -> list[dict]:
74
- """Manifest rows joined with their metrics, in manifest order."""
75
- manifest = json.load(open(fetch("windows_manifest.json")))
76
- try:
77
- metrics = json.load(open(fetch("metrics_summary.json")))
78
- except Exception:
79
- metrics = []
80
- by_id = {m["id"]: m for m in metrics}
81
-
82
- rows = []
83
- for i, w in enumerate(manifest):
84
- ep = w["episode_id"]
85
- m = by_id.get(ep, {})
86
- rows.append(
87
- {
88
- "order": i,
89
- "episode_id": ep,
90
- "task": w.get("task", "?"),
91
- "edge": w.get("edge_type", "?"),
92
- "arm": w.get("arm", "?"),
93
- "amplitude": w.get("amplitude"),
94
- "start_frame": w.get("start_frame"),
95
- "window": w.get("window"),
96
- "critical_frame": w.get("critical_frame"),
97
- "n_frames": w.get("n_frames"),
98
- "full_psnr": m.get("full_psnr"),
99
- "crit_psnr": m.get("crit_psnr"),
100
- "full_mse": m.get("full_mse"),
101
- "crit_mse": m.get("crit_mse"),
102
- }
103
- )
104
- return rows
105
 
106
 
107
  def label_for(row: dict) -> str:
108
  psnr = row["crit_psnr"]
109
- head = f"{psnr:5.1f} dB" if psnr is not None else " -- dB"
110
- return f"{head} · {row['edge']} {row['arm']} · {row['task']} · {row['episode_id'][8:16]}"
 
 
111
 
112
 
113
- def filter_rows(query: str, edge: str, arm: str, sort: str) -> list[dict]:
114
- rows = load_index()
115
  q = (query or "").strip().lower()
116
  if q:
117
  rows = [r for r in rows if q in r["task"].lower() or q in r["episode_id"].lower()]
@@ -120,24 +360,35 @@ def filter_rows(query: str, edge: str, arm: str, sort: str) -> list[dict]:
120
  if arm != "any":
121
  rows = [r for r in rows if r["arm"] == arm]
122
 
123
- worst = float("inf")
 
 
 
124
  if sort == SORTS[0]:
125
- rows = sorted(rows, key=lambda r: r["crit_psnr"] if r["crit_psnr"] is not None else worst)
126
- elif sort == SORTS[1]:
127
- rows = sorted(rows, key=lambda r: -(r["crit_psnr"] if r["crit_psnr"] is not None else -worst))
128
- elif sort == SORTS[2]:
129
- rows = sorted(rows, key=lambda r: (r["task"], r["episode_id"]))
130
- else:
131
- rows = sorted(rows, key=lambda r: r["order"])
132
- return rows
133
 
134
 
135
- @functools.lru_cache(maxsize=64)
136
- def row_by_id(episode_id: str) -> dict:
137
- for r in load_index():
138
  if r["episode_id"] == episode_id:
139
- return r
140
- raise KeyError(episode_id)
 
 
 
 
 
 
 
 
 
 
 
141
 
142
 
143
  # --------------------------------------------------------------------------- video
@@ -158,18 +409,22 @@ def read_video(path: str) -> np.ndarray:
158
  return np.stack(frames)
159
 
160
 
161
- @functools.lru_cache(maxsize=4)
162
- def load_pair(episode_id: str) -> tuple[np.ndarray, np.ndarray]:
163
- gt = read_video(fetch(f"windows/{episode_id}/gt.mp4"))
164
- gen = read_video(fetch(f"windows/{episode_id}/generated.mp4"))
165
- t = min(len(gt), len(gen))
166
- return gt[:t], gen[:t]
167
 
 
 
 
 
168
 
169
- def cam_slices(frames: np.ndarray, view: str) -> list[np.ndarray]:
170
- """Split the vstacked video into the camera bands requested by `view`."""
171
- h = frames.shape[1] // 3
172
- rows = [VIEW_ROW[view]] if view in VIEW_ROW else [0, 1, 2]
 
173
  return [frames[:, i * h : (i + 1) * h] for i in rows]
174
 
175
 
@@ -185,7 +440,7 @@ def letterbox_rows(*bands: np.ndarray, thresh: int = 16) -> tuple[int, int]:
185
  return int(idx[0]), int(idx[-1]) + 1
186
 
187
 
188
- def crop_view(gt: np.ndarray, gen: np.ndarray, view: str) -> tuple[np.ndarray, np.ndarray]:
189
  """Select cameras and trim the letterbox, identically for both videos."""
190
  gt_out, gen_out = [], []
191
  for g, p in zip(cam_slices(gt, view), cam_slices(gen, view)):
@@ -219,11 +474,9 @@ def diff_frames(gt: np.ndarray, gen: np.ndarray, gain: float = 3.0) -> np.ndarra
219
  return np.stack([cv2.applyColorMap(f, cv2.COLORMAP_INFERNO)[:, :, ::-1] for f in d])
220
 
221
 
222
- @functools.lru_cache(maxsize=2)
223
- def compose(episode_id: str, view: str, show_diff: bool, crit_idx: int) -> np.ndarray:
224
  """Build the labelled side-by-side strip: [T, H, W, 3] RGB."""
225
- gt, gen = load_pair(episode_id)
226
- gt_c, gen_c = crop_view(gt, gen, view)
227
  gt_v, gen_v = upscale(gt_c, SCALE), upscale(gen_c, SCALE)
228
 
229
  panels = [panel(gt_v, "GROUND TRUTH", GT_COLOR), panel(gen_v, "GENERATED", GEN_COLOR)]
@@ -240,14 +493,15 @@ def compose(episode_id: str, view: str, show_diff: bool, crit_idx: int) -> np.nd
240
  t, h, w, _ = comp.shape
241
  out = np.zeros((t, h + FTR, w, 3), np.uint8)
242
  out[:, :h] = comp
 
243
  tl0, tl1, y = int(w * 0.45), w - 14, FTR // 2
244
  for i in range(t):
245
  bar = np.full((FTR, w, 3), 22, np.uint8)
246
- is_crit = i == crit_idx
247
  text = f"frame {i:02d}/{t - 1}" + (" CRITICAL FRAME" if is_crit else "")
248
  cv2.putText(bar, text, (8, FTR - 8), FONT, 0.5, CRIT_COLOR if is_crit else (200, 200, 200), 1, cv2.LINE_AA)
249
  cv2.line(bar, (tl0, y), (tl1, y), (70, 70, 70), 2, cv2.LINE_AA)
250
- if 0 <= crit_idx < t:
251
  cx = int(tl0 + (tl1 - tl0) * crit_idx / max(t - 1, 1))
252
  cv2.line(bar, (cx, y - 7), (cx, y + 7), CRIT_COLOR, 2, cv2.LINE_AA)
253
  px = int(tl0 + (tl1 - tl0) * i / max(t - 1, 1))
@@ -264,7 +518,7 @@ def compose(episode_id: str, view: str, show_diff: bool, crit_idx: int) -> np.nd
264
 
265
 
266
  def write_mp4(frames: np.ndarray, path: Path, fps: float) -> str:
267
- t, h, w, _ = frames.shape
268
  writer = imageio_ffmpeg.write_frames(
269
  str(path),
270
  size=(w, h),
@@ -282,115 +536,178 @@ def write_mp4(frames: np.ndarray, path: Path, fps: float) -> str:
282
  return str(path)
283
 
284
 
285
- def render_video(episode_id: str, view: str, show_diff: bool, fps: float, crit_idx: int) -> str:
286
- key = f"{episode_id}_{view.replace(' ', '')}_{int(show_diff)}_{fps:g}"
287
- path = CACHE_DIR / f"{key}.mp4"
288
  if not path.exists():
289
- write_mp4(compose(episode_id, view, show_diff, crit_idx), path, fps)
290
  return str(path)
291
 
292
 
293
  # --------------------------------------------------------------------------- actions plot
294
 
295
 
296
- def plot_actions(episode_id: str, crit_idx: int, cursor: int | None = None) -> Figure:
297
- actions = np.asarray(json.load(open(fetch(f"windows/{episode_id}/actions.json"))), dtype=float)
 
 
 
 
 
 
 
298
  t = np.arange(len(actions))
 
 
 
 
 
 
 
 
 
299
 
300
- fig = Figure(figsize=(7.2, 5.4), dpi=110, layout="constrained")
301
- axes = fig.subplots(3, 1, sharex=True)
302
- specs = [
303
- ("Grippers (1.0 = open, 0.0 = closed)", [(6, "L gripper"), (13, "R gripper")]),
304
- ("Left arm joints", [(i, f"L{i}") for i in range(6)]),
305
- ("Right arm joints", [(i, f"R{i - 7}") for i in range(7, 13)]),
306
- ]
307
  for ax, (title, channels) in zip(axes, specs):
308
  for idx, name in channels:
309
  ax.plot(t, actions[:, idx], lw=1.4, label=name)
310
  ax.set_title(title, fontsize=9, loc="left")
311
- ax.axvline(crit_idx, color="crimson", ls="--", lw=1.2, zorder=0)
 
312
  if cursor is not None:
313
  ax.axvline(cursor, color="0.35", ls=":", lw=1.0, zorder=0)
314
  ax.grid(alpha=0.25, lw=0.5)
315
  ax.tick_params(labelsize=8)
316
- ax.legend(fontsize=7, ncol=6, loc="upper right", framealpha=0.7)
317
- axes[-1].set_xlabel("frame in window (dashed red = critical frame)", fontsize=8)
 
 
318
  return fig
319
 
320
 
321
  # --------------------------------------------------------------------------- ui glue
322
 
323
 
324
- def metrics_md(row: dict) -> str:
325
  def fmt(v, unit=""):
326
  return f"{v:.2f}{unit}" if isinstance(v, (int, float)) else "—"
327
 
328
- win = row.get("window") or [row["start_frame"], (row["start_frame"] or 0) + 32]
329
-
330
- return (
331
- f"### {row['task'].replace('_', ' ')}\n"
332
- f"`{row['episode_id']}`\n\n"
333
- f"| | full window | critical frame |\n|---|---|---|\n"
334
- f"| **PSNR** | {fmt(row['full_psnr'], ' dB')} | {fmt(row['crit_psnr'], ' dB')} |\n"
335
- f"| **MSE** | {fmt(row['full_mse'])} | {fmt(row['crit_mse'])} |\n\n"
336
- f"**{row['edge']}** · arm **{row['arm']}** · amplitude {fmt(row['amplitude'])} · "
337
- f"window frames {win[0]}–{win[1]} of {row['n_frames']} · "
338
- f"critical frame {row['critical_frame']}"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
339
  )
 
340
 
341
 
342
  def table_rows(rows: list[dict]) -> list[list]:
 
 
 
343
  return [
344
- [
345
- r["episode_id"][8:16],
346
- r["task"],
347
- r["edge"],
348
- r["arm"],
349
- round(r["crit_psnr"], 2) if r["crit_psnr"] is not None else None,
350
- round(r["full_psnr"], 2) if r["full_psnr"] is not None else None,
351
- ]
352
  for r in rows
353
  ]
354
 
355
 
356
- def on_filter(query, edge, arm, sort, current):
357
- rows = filter_rows(query, edge, arm, sort)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
358
  choices = [(label_for(r), r["episode_id"]) for r in rows]
359
  ids = [r["episode_id"] for r in rows]
360
  value = current if current in ids else (ids[0] if ids else None)
361
  return (
362
- gr.update(choices=choices, value=value, label=f"Window ({len(ids)} of 300)"),
363
  table_rows(rows),
364
  ids,
365
  )
366
 
367
 
368
- def critical_index(row: dict) -> int:
369
- """Offset of the critical frame inside the window (16 for 33-frame windows)."""
370
- start, crit = row.get("start_frame"), row.get("critical_frame")
371
- if start is None or crit is None:
372
- return 16
373
- return int(crit) - int(start)
374
 
 
 
 
 
 
375
 
376
- def on_select(episode_id, view, show_diff, fps, frame_idx):
377
- if not episode_id:
378
- return None, None, "*No window matches the current filters.*", None
379
- row = row_by_id(episode_id)
380
- crit = critical_index(row)
381
- video = render_video(episode_id, view, show_diff, fps, crit)
382
- frames = compose(episode_id, view, show_diff, crit)
383
- i = int(min(frame_idx, len(frames) - 1))
384
- return video, frames[i], metrics_md(row), plot_actions(episode_id, crit, i)
385
 
386
 
387
- def on_frame(episode_id, view, show_diff, frame_idx):
388
- if not episode_id:
 
389
  return None, None
390
- crit = critical_index(row_by_id(episode_id))
391
- frames = compose(episode_id, view, show_diff, crit)
392
  i = int(min(frame_idx, len(frames) - 1))
393
- return frames[i], plot_actions(episode_id, crit, i)
394
 
395
 
396
  def step(ids, current, delta):
@@ -410,15 +727,23 @@ CSS = """
410
  .window-still img { max-height: 78vh; object-fit: contain; background: #111; }
411
  """
412
 
413
- with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
414
- gr.Markdown(
415
- "# Forward-dynamics rollout vs. ground truth\n"
416
- f"Ground-truth and generated 33-frame windows from **`{DATASET_REPO}`**, "
417
- "each centred on a gripper critical frame (frame 16). "
418
- "Cameras are top / left-wrist / right-wrist."
419
- )
420
  ids_state = gr.State([])
421
 
 
 
 
 
 
 
 
 
 
 
 
422
  with gr.Row():
423
  with gr.Column(scale=1, min_width=320):
424
  query = gr.Textbox(label="Search task or episode", placeholder="e.g. napkin, pillowcase…")
@@ -431,9 +756,9 @@ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
431
  prev_btn = gr.Button("← Prev")
432
  next_btn = gr.Button("Next →")
433
  gr.Markdown("---")
434
- view = gr.Radio(VIEWS, value="Top", label="Camera")
435
  show_diff = gr.Checkbox(False, label="Add |GT − generated| difference panel")
436
- fps = gr.Slider(2, 30, value=8, step=1, label="Playback fps (source is 30 fps)")
437
  info = gr.Markdown()
438
 
439
  with gr.Column(scale=2):
@@ -447,10 +772,10 @@ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
447
  elem_classes="window-video",
448
  )
449
  with gr.Tab("Frame stepper"):
450
- frame_idx = gr.Slider(0, 32, value=16, step=1, label="Frame in window (16 = critical)")
451
  still = gr.Image(label="Frame comparison", elem_classes="window-still")
452
  with gr.Tab("Driving actions"):
453
- actions_plot = gr.Plot(label="Action chunk fed to the model [33 x 14]")
454
  with gr.Tab("All windows"):
455
  table = gr.Dataframe(
456
  headers=["episode", "task", "edge", "arm", "crit PSNR", "full PSNR"],
@@ -460,12 +785,23 @@ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
460
  max_height=650,
461
  )
462
 
463
- filter_inputs = [query, edge, arm, sort]
464
  filter_outputs = [picker, table, ids_state]
465
- view_inputs = [picker, view, show_diff, fps, frame_idx]
466
- view_outputs = [video, still, info, actions_plot]
 
 
 
 
 
 
 
467
 
468
- for control in filter_inputs:
 
 
 
 
469
  control.change(on_filter, filter_inputs + [picker], filter_outputs).then(
470
  on_select, view_inputs, view_outputs
471
  )
@@ -473,7 +809,7 @@ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
473
  picker.change(on_select, view_inputs, view_outputs)
474
  for control in (view, show_diff, fps):
475
  control.change(on_select, view_inputs, view_outputs)
476
- frame_idx.change(on_frame, [picker, view, show_diff, frame_idx], [still, actions_plot])
477
 
478
  prev_btn.click(lambda i, c: step(i, c, -1), [ids_state, picker], picker)
479
  next_btn.click(lambda i, c: step(i, c, +1), [ids_state, picker], picker)
@@ -483,9 +819,5 @@ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True) as demo:
483
 
484
  table.select(on_table_select, ids_state, picker)
485
 
486
- demo.load(on_filter, filter_inputs + [picker], filter_outputs).then(
487
- on_select, view_inputs, view_outputs
488
- )
489
-
490
  if __name__ == "__main__":
491
  demo.queue(max_size=16).launch(css=CSS)
 
1
  """
2
+ Side-by-side viewer for forward-dynamics rollouts vs. ground truth.
3
 
4
+ Point it at any HF dataset repo laid out as:
5
 
6
+ windows_manifest.json # optional: list of window dicts
7
+ metrics_summary.json # optional: per-window MSE / PSNR
8
+ windows/<episode_id>/gt.mp4 # ground-truth window
9
+ windows/<episode_id>/generated.mp4 # model rollout for the same slice
10
+ windows/<episode_id>/actions.json # optional: [T, D] driving actions
11
+ windows/<episode_id>/meta.json # optional: per-window metadata
12
 
13
+ Only `windows/<id>/{gt,generated}.mp4` is required. Without a manifest the
14
+ window list is discovered by listing `windows/` one level deep. Videos may
15
+ stack N cameras vertically; N is inferred from the frame aspect ratio.
16
+
17
+ Nothing is downloaded at build time and no repo is ever pulled whole. Each
18
+ browser session gets its own scratch directory; window files arrive only when
19
+ that window is opened, at most MAX_WINDOWS_ON_DISK are kept, and the whole
20
+ directory is deleted when the session ends.
21
+
22
+ DATASET_REPO sets the repo prefilled at startup. Private repos need an
23
+ HF_TOKEN secret with read access.
24
  """
25
 
26
  from __future__ import annotations
27
 
 
28
  import json
29
  import os
30
+ import re
31
+ import shutil
32
  import tempfile
33
+ import threading
34
+ import time
35
+ import uuid
36
+ from collections import OrderedDict
37
  from pathlib import Path
38
 
39
  import cv2
40
  import gradio as gr
41
  import imageio_ffmpeg
42
  import numpy as np
43
+ from huggingface_hub import HfApi, RepoFolder, hf_hub_download
44
  from matplotlib.figure import Figure
45
 
46
+ DEFAULT_REPO = os.environ.get("DATASET_REPO", "OneWorld-AI/abc-critical-window-rollout")
47
  HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN") or None
48
 
49
+ BASE_DIR = Path(tempfile.gettempdir()) / "wmviz"
50
+ BASE_DIR.mkdir(parents=True, exist_ok=True)
51
 
52
+ MAX_WINDOWS_ON_DISK = int(os.environ.get("MAX_WINDOWS_ON_DISK", 12))
53
+ MAX_PAIRS_IN_RAM = 2
54
+ MAX_COMPOSED_IN_RAM = 1
55
+ MAX_DISCOVER = int(os.environ.get("MAX_DISCOVER", 5000))
56
+ SESSION_TTL = int(os.environ.get("SESSION_TTL", 3600))
57
+
58
+ # Conventional names when a video stacks exactly three cameras.
59
+ THREE_CAM_NAMES = ["Top", "Left wrist", "Right wrist"]
60
+ ALL_CAMS = -1
61
+ NO_CRIT = -1
62
 
63
  FONT = cv2.FONT_HERSHEY_SIMPLEX
64
  HDR, FTR, SEP = 26, 26, 4
65
+ SCALE = 2 # source frames are small; upscale so overlays stay legible
66
 
67
  GT_COLOR = (130, 225, 140)
68
  GEN_COLOR = (120, 180, 255)
 
77
  ]
78
 
79
 
80
+ # --------------------------------------------------------------------------- session store
81
+
82
+
83
+ def parse_repo(text: str) -> tuple[str, str | None]:
84
+ """Split "owner/name@revision" into (repo_id, revision). Tolerates a pasted URL."""
85
+ text = (text or "").strip().rstrip("/")
86
+ m = re.match(r"^https?://huggingface\.co/(?:datasets/)?(.+?)(?:/tree/([^/]+))?$", text)
87
+ if m:
88
+ return m.group(1), m.group(2)
89
+ if "@" in text:
90
+ repo, _, rev = text.partition("@")
91
+ return repo.strip(), rev.strip() or None
92
+ return text, None
93
+
94
+
95
+ class Session:
96
+ """One browser session's scratch space: bounded on disk and in RAM."""
97
+
98
+ def __init__(self, sid: str):
99
+ self.sid = sid
100
+ self.root = BASE_DIR / sid
101
+ self.data = self.root / "data"
102
+ self.render = self.root / "render"
103
+ for d in (self.data, self.render):
104
+ d.mkdir(parents=True, exist_ok=True)
105
+ self.repo = ""
106
+ self.revision: str | None = None
107
+ self.rows: list[dict] = []
108
+ self.note = ""
109
+ self._on_disk: OrderedDict[str, bool] = OrderedDict()
110
+ self._pairs: OrderedDict[str, tuple[np.ndarray, np.ndarray]] = OrderedDict()
111
+ self._composed: OrderedDict[tuple, np.ndarray] = OrderedDict()
112
+ self._meta: dict[str, dict] = {}
113
+ self.touched = time.time()
114
+
115
+ # -- fetching ----------------------------------------------------------
116
+
117
+ def fetch(self, rel: str, optional: bool = False) -> str | None:
118
+ """Download one repo file into this session's directory."""
119
+ self.touched = time.time()
120
+ try:
121
+ return hf_hub_download(
122
+ repo_id=self.repo,
123
+ filename=rel,
124
+ repo_type="dataset",
125
+ revision=self.revision,
126
+ token=HF_TOKEN,
127
+ local_dir=str(self.data),
128
+ )
129
+ except Exception:
130
+ if optional:
131
+ return None
132
+ raise
133
+
134
+ def window_file(self, episode_id: str, name: str, optional: bool = False) -> str | None:
135
+ path = self.fetch(f"windows/{episode_id}/{name}", optional=optional)
136
+ if path:
137
+ self._keep(episode_id)
138
+ return path
139
+
140
+ def _keep(self, episode_id: str) -> None:
141
+ """Register a window as resident and evict the least recently used ones."""
142
+ self._on_disk[episode_id] = True
143
+ self._on_disk.move_to_end(episode_id)
144
+ while len(self._on_disk) > MAX_WINDOWS_ON_DISK:
145
+ old, _ = self._on_disk.popitem(last=False)
146
+ shutil.rmtree(self.data / "windows" / old, ignore_errors=True)
147
+ self._pairs.pop(old, None)
148
+ for key in [k for k in self._composed if k[0] == old]:
149
+ self._composed.pop(key, None)
150
+ for f in self.render.glob(f"{old}_*.mp4"):
151
+ f.unlink(missing_ok=True)
152
+
153
+ def disk_windows(self) -> int:
154
+ return len(self._on_disk)
155
+
156
+ # -- bounded RAM caches ------------------------------------------------
157
+
158
+ def pair(self, episode_id: str) -> tuple[np.ndarray, np.ndarray]:
159
+ hit = self._pairs.get(episode_id)
160
+ if hit is not None:
161
+ self._pairs.move_to_end(episode_id)
162
+ return hit
163
+ gt = read_video(self.window_file(episode_id, "gt.mp4"))
164
+ gen = read_video(self.window_file(episode_id, "generated.mp4"))
165
+ t = min(len(gt), len(gen))
166
+ value = (gt[:t], gen[:t])
167
+ self._pairs[episode_id] = value
168
+ while len(self._pairs) > MAX_PAIRS_IN_RAM:
169
+ self._pairs.popitem(last=False)
170
+ return value
171
+
172
+ def composed(self, episode_id: str, view: int, show_diff: bool, crit: int) -> np.ndarray:
173
+ key = (episode_id, view, show_diff, crit)
174
+ hit = self._composed.get(key)
175
+ if hit is not None:
176
+ self._composed.move_to_end(key)
177
+ return hit
178
+ value = compose(self.pair(episode_id), view, show_diff, crit)
179
+ self._composed[key] = value
180
+ while len(self._composed) > MAX_COMPOSED_IN_RAM:
181
+ self._composed.popitem(last=False)
182
+ return value
183
+
184
+ def meta(self, episode_id: str) -> dict:
185
+ if episode_id not in self._meta:
186
+ path = self.window_file(episode_id, "meta.json", optional=True)
187
+ try:
188
+ self._meta[episode_id] = json.load(open(path)) if path else {}
189
+ except Exception:
190
+ self._meta[episode_id] = {}
191
+ return self._meta[episode_id]
192
+
193
+ # -- lifecycle ---------------------------------------------------------
194
+
195
+ def set_repo(self, repo: str, revision: str | None) -> None:
196
+ if (repo, revision) != (self.repo, self.revision):
197
+ self.reset()
198
+ self.repo, self.revision = repo, revision
199
+
200
+ def reset(self) -> None:
201
+ self._on_disk.clear()
202
+ self._pairs.clear()
203
+ self._composed.clear()
204
+ self._meta.clear()
205
+ self.rows = []
206
+ for d in (self.data, self.render):
207
+ shutil.rmtree(d, ignore_errors=True)
208
+ d.mkdir(parents=True, exist_ok=True)
209
+
210
+ def close(self) -> None:
211
+ self._pairs.clear()
212
+ self._composed.clear()
213
+ shutil.rmtree(self.root, ignore_errors=True)
214
+
215
+
216
+ SESSIONS: dict[str, Session] = {}
217
+ _LOCK = threading.Lock()
218
+
219
+
220
+ def sweep() -> None:
221
+ """Drop sessions whose browser tab went away without a clean callback."""
222
+ now = time.time()
223
+ with _LOCK:
224
+ stale = [s for s in SESSIONS.values() if now - s.touched > SESSION_TTL]
225
+ for s in stale:
226
+ SESSIONS.pop(s.sid, None)
227
+ s.close()
228
+ live = set(SESSIONS)
229
+ for d in BASE_DIR.iterdir(): # dirs left behind by a previous process
230
+ try:
231
+ if d.is_dir() and d.name not in live and now - d.stat().st_mtime > SESSION_TTL:
232
+ shutil.rmtree(d, ignore_errors=True)
233
+ except OSError:
234
+ pass
235
+
236
+
237
+ def get_session(sid: str | None) -> Session:
238
+ with _LOCK:
239
+ s = SESSIONS.get(sid or "")
240
+ if s is None:
241
+ sid = sid or uuid.uuid4().hex
242
+ s = SESSIONS[sid] = Session(sid)
243
+ s.touched = time.time()
244
+ return s
245
+
246
+
247
+ def release(sid: str | None) -> None:
248
+ """gr.State delete_callback — fires when the session ends or its TTL expires."""
249
+ with _LOCK:
250
+ s = SESSIONS.pop(sid or "", None)
251
+ if s is not None:
252
+ s.close()
253
+
254
+
255
+ # --------------------------------------------------------------------------- index
256
+
257
+
258
+ def _row(order: int, episode_id: str, w: dict, m: dict) -> dict:
259
+ return {
260
+ "order": order,
261
+ "episode_id": episode_id,
262
+ "task": w.get("task") or "?",
263
+ "edge": w.get("edge_type") or "?",
264
+ "arm": w.get("arm") or "?",
265
+ "amplitude": w.get("amplitude"),
266
+ "start_frame": w.get("start_frame"),
267
+ "window": w.get("window"),
268
+ "critical_frame": w.get("critical_frame"),
269
+ "n_frames": w.get("n_frames"),
270
+ "full_psnr": m.get("full_psnr"),
271
+ "crit_psnr": m.get("crit_psnr"),
272
+ "full_mse": m.get("full_mse"),
273
+ "crit_mse": m.get("crit_mse"),
274
+ }
275
+
276
+
277
+ def discover_episodes(session: Session) -> tuple[list[str], bool]:
278
+ """List windows/ one level deep, capped. Returns (ids, was_truncated)."""
279
+ tree = HfApi().list_repo_tree(
280
+ session.repo,
281
+ path_in_repo="windows",
282
+ recursive=False,
283
+ revision=session.revision,
284
  repo_type="dataset",
285
  token=HF_TOKEN,
286
  )
287
+ eps, truncated = [], False
288
+ for entry in tree:
289
+ if not isinstance(entry, RepoFolder):
290
+ continue
291
+ eps.append(entry.path.split("/")[-1])
292
+ if len(eps) >= MAX_DISCOVER:
293
+ truncated = True
294
+ break
295
+ return sorted(eps), truncated
296
+
297
+
298
+ def build_index(session: Session) -> str:
299
+ """Populate session.rows from the manifest, or by listing the repo. Returns a note."""
300
+ metrics = []
301
+ path = session.fetch("metrics_summary.json", optional=True)
302
+ if path:
303
+ try:
304
+ metrics = json.load(open(path))
305
+ except Exception:
306
+ metrics = []
307
+ by_id = {m.get("id"): m for m in metrics if isinstance(m, dict)}
308
+
309
+ manifest = None
310
+ path = session.fetch("windows_manifest.json", optional=True)
311
+ if path:
312
+ try:
313
+ loaded = json.load(open(path))
314
+ if isinstance(loaded, list) and loaded:
315
+ manifest = loaded
316
+ except Exception:
317
+ manifest = None
318
+
319
+ if manifest is not None:
320
+ session.rows = [
321
+ _row(i, w.get("episode_id", f"window_{i}"), w, by_id.get(w.get("episode_id"), {}))
322
+ for i, w in enumerate(manifest)
323
+ ]
324
+ note = "`windows_manifest.json`"
325
+ else:
326
+ eps, truncated = discover_episodes(session)
327
+ if not eps:
328
+ raise FileNotFoundError(
329
+ "no windows/<episode_id>/ directories found — is this the right repo?"
330
+ )
331
+ session.rows = [_row(i, ep, {}, by_id.get(ep, {})) for i, ep in enumerate(eps)]
332
+ note = "directory listing (no `windows_manifest.json`)"
333
+ if truncated:
334
+ note += f", capped at {MAX_DISCOVER}"
335
 
336
+ if metrics:
337
+ note += " + `metrics_summary.json`"
338
+ return note
339
 
340
+
341
+ def short_id(episode_id: str) -> str:
342
+ return re.sub(r"^(episode|window|ep)[-_]", "", episode_id)[:8]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
343
 
344
 
345
  def label_for(row: dict) -> str:
346
  psnr = row["crit_psnr"]
347
+ head = f"{psnr:5.1f} dB" if isinstance(psnr, (int, float)) else " -- dB"
348
+ tags = " ".join(t for t in (row["edge"], row["arm"]) if t and t != "?")
349
+ parts = [head] + ([tags] if tags else []) + [row["task"], short_id(row["episode_id"])]
350
+ return " · ".join(parts)
351
 
352
 
353
+ def filter_rows(session: Session, query: str, edge: str, arm: str, sort: str) -> list[dict]:
354
+ rows = session.rows
355
  q = (query or "").strip().lower()
356
  if q:
357
  rows = [r for r in rows if q in r["task"].lower() or q in r["episode_id"].lower()]
 
360
  if arm != "any":
361
  rows = [r for r in rows if r["arm"] == arm]
362
 
363
+ def psnr(r, missing):
364
+ v = r["crit_psnr"]
365
+ return v if isinstance(v, (int, float)) else missing
366
+
367
  if sort == SORTS[0]:
368
+ return sorted(rows, key=lambda r: psnr(r, float("inf")))
369
+ if sort == SORTS[1]:
370
+ return sorted(rows, key=lambda r: -psnr(r, float("-inf")))
371
+ if sort == SORTS[2]:
372
+ return sorted(rows, key=lambda r: (r["task"], r["episode_id"]))
373
+ return sorted(rows, key=lambda r: r["order"])
 
 
374
 
375
 
376
+ def row_by_id(session: Session, episode_id: str) -> dict:
377
+ for r in session.rows:
 
378
  if r["episode_id"] == episode_id:
379
+ row = dict(r)
380
+ break
381
+ else:
382
+ row = _row(0, episode_id, {}, {})
383
+ # backfill whatever the manifest did not carry from the window's own meta.json
384
+ if row["task"] == "?" or row["critical_frame"] is None:
385
+ meta = session.meta(episode_id)
386
+ if meta:
387
+ merged = _row(row["order"], episode_id, meta, {})
388
+ for k, v in merged.items():
389
+ if row.get(k) in (None, "?"):
390
+ row[k] = v
391
+ return row
392
 
393
 
394
  # --------------------------------------------------------------------------- video
 
409
  return np.stack(frames)
410
 
411
 
412
+ def n_cameras(frames: np.ndarray) -> int:
413
+ """How many roughly-square cameras are stacked vertically in the frame."""
414
+ h, w = frames.shape[1], frames.shape[2]
415
+ return max(1, min(6, int(round(h / w))))
416
+
 
417
 
418
+ def camera_choices(n: int) -> list[tuple[str, int]]:
419
+ names = THREE_CAM_NAMES if n == 3 else [f"Camera {i + 1}" for i in range(n)]
420
+ choices = [(name, i) for i, name in enumerate(names)]
421
+ return ([("All cameras", ALL_CAMS)] + choices) if n > 1 else choices
422
 
423
+
424
+ def cam_slices(frames: np.ndarray, view: int) -> list[np.ndarray]:
425
+ n = n_cameras(frames)
426
+ h = frames.shape[1] // n
427
+ rows = range(n) if view == ALL_CAMS or not 0 <= view < n else [view]
428
  return [frames[:, i * h : (i + 1) * h] for i in rows]
429
 
430
 
 
440
  return int(idx[0]), int(idx[-1]) + 1
441
 
442
 
443
+ def crop_view(gt: np.ndarray, gen: np.ndarray, view: int) -> tuple[np.ndarray, np.ndarray]:
444
  """Select cameras and trim the letterbox, identically for both videos."""
445
  gt_out, gen_out = [], []
446
  for g, p in zip(cam_slices(gt, view), cam_slices(gen, view)):
 
474
  return np.stack([cv2.applyColorMap(f, cv2.COLORMAP_INFERNO)[:, :, ::-1] for f in d])
475
 
476
 
477
+ def compose(pair: tuple[np.ndarray, np.ndarray], view: int, show_diff: bool, crit_idx: int) -> np.ndarray:
 
478
  """Build the labelled side-by-side strip: [T, H, W, 3] RGB."""
479
+ gt_c, gen_c = crop_view(pair[0], pair[1], view)
 
480
  gt_v, gen_v = upscale(gt_c, SCALE), upscale(gen_c, SCALE)
481
 
482
  panels = [panel(gt_v, "GROUND TRUTH", GT_COLOR), panel(gen_v, "GENERATED", GEN_COLOR)]
 
493
  t, h, w, _ = comp.shape
494
  out = np.zeros((t, h + FTR, w, 3), np.uint8)
495
  out[:, :h] = comp
496
+ has_crit = 0 <= crit_idx < t
497
  tl0, tl1, y = int(w * 0.45), w - 14, FTR // 2
498
  for i in range(t):
499
  bar = np.full((FTR, w, 3), 22, np.uint8)
500
+ is_crit = has_crit and i == crit_idx
501
  text = f"frame {i:02d}/{t - 1}" + (" CRITICAL FRAME" if is_crit else "")
502
  cv2.putText(bar, text, (8, FTR - 8), FONT, 0.5, CRIT_COLOR if is_crit else (200, 200, 200), 1, cv2.LINE_AA)
503
  cv2.line(bar, (tl0, y), (tl1, y), (70, 70, 70), 2, cv2.LINE_AA)
504
+ if has_crit:
505
  cx = int(tl0 + (tl1 - tl0) * crit_idx / max(t - 1, 1))
506
  cv2.line(bar, (cx, y - 7), (cx, y + 7), CRIT_COLOR, 2, cv2.LINE_AA)
507
  px = int(tl0 + (tl1 - tl0) * i / max(t - 1, 1))
 
518
 
519
 
520
  def write_mp4(frames: np.ndarray, path: Path, fps: float) -> str:
521
+ _, h, w, _ = frames.shape
522
  writer = imageio_ffmpeg.write_frames(
523
  str(path),
524
  size=(w, h),
 
536
  return str(path)
537
 
538
 
539
+ def render_video(session: Session, episode_id: str, view: int, show_diff: bool, fps: float, crit: int) -> str:
540
+ path = session.render / f"{episode_id}_{view}_{int(show_diff)}_{fps:g}.mp4"
 
541
  if not path.exists():
542
+ write_mp4(session.composed(episode_id, view, show_diff, crit), path, fps)
543
  return str(path)
544
 
545
 
546
  # --------------------------------------------------------------------------- actions plot
547
 
548
 
549
+ def plot_actions(session: Session, episode_id: str, crit_idx: int, cursor: int | None) -> Figure | None:
550
+ path = session.window_file(episode_id, "actions.json", optional=True)
551
+ if not path:
552
+ return None
553
+ try:
554
+ actions = np.atleast_2d(np.asarray(json.load(open(path)), dtype=float))
555
+ except Exception:
556
+ return None
557
+
558
  t = np.arange(len(actions))
559
+ dim = actions.shape[1]
560
+ if dim == 14: # bimanual: [L arm 6, L grip, R arm 6, R grip]
561
+ specs = [
562
+ ("Grippers (1.0 = open, 0.0 = closed)", [(6, "L gripper"), (13, "R gripper")]),
563
+ ("Left arm joints", [(i, f"L{i}") for i in range(6)]),
564
+ ("Right arm joints", [(i, f"R{i - 7}") for i in range(7, 13)]),
565
+ ]
566
+ else:
567
+ specs = [(f"Action channels [{len(actions)} x {dim}]", [(i, f"a{i}") for i in range(dim)])]
568
 
569
+ fig = Figure(figsize=(7.2, 1.8 * len(specs) + 0.6), dpi=110, layout="constrained")
570
+ axes = np.atleast_1d(fig.subplots(len(specs), 1, sharex=True))
 
 
 
 
 
571
  for ax, (title, channels) in zip(axes, specs):
572
  for idx, name in channels:
573
  ax.plot(t, actions[:, idx], lw=1.4, label=name)
574
  ax.set_title(title, fontsize=9, loc="left")
575
+ if crit_idx >= 0:
576
+ ax.axvline(crit_idx, color="crimson", ls="--", lw=1.2, zorder=0)
577
  if cursor is not None:
578
  ax.axvline(cursor, color="0.35", ls=":", lw=1.0, zorder=0)
579
  ax.grid(alpha=0.25, lw=0.5)
580
  ax.tick_params(labelsize=8)
581
+ if len(channels) <= 8:
582
+ ax.legend(fontsize=7, ncol=6, loc="upper right", framealpha=0.7)
583
+ tail = " (dashed red = critical frame)" if crit_idx >= 0 else ""
584
+ axes[-1].set_xlabel(f"frame in window{tail}", fontsize=8)
585
  return fig
586
 
587
 
588
  # --------------------------------------------------------------------------- ui glue
589
 
590
 
591
+ def metrics_md(session: Session, row: dict) -> str:
592
  def fmt(v, unit=""):
593
  return f"{v:.2f}{unit}" if isinstance(v, (int, float)) else "—"
594
 
595
+ out = [f"### {row['task'].replace('_', ' ')}", f"`{row['episode_id']}`", ""]
596
+ if any(isinstance(row[k], (int, float)) for k in ("full_psnr", "crit_psnr", "full_mse", "crit_mse")):
597
+ out += [
598
+ "| | full window | critical frame |",
599
+ "|---|---|---|",
600
+ f"| **PSNR** | {fmt(row['full_psnr'], ' dB')} | {fmt(row['crit_psnr'], ' dB')} |",
601
+ f"| **MSE** | {fmt(row['full_mse'])} | {fmt(row['crit_mse'])} |",
602
+ "",
603
+ ]
604
+ bits = []
605
+ if row["edge"] != "?":
606
+ bits.append(f"**{row['edge']}**")
607
+ if row["arm"] != "?":
608
+ bits.append(f"arm **{row['arm']}**")
609
+ if isinstance(row["amplitude"], (int, float)):
610
+ bits.append(f"amplitude {fmt(row['amplitude'])}")
611
+ win = row.get("window")
612
+ if isinstance(win, (list, tuple)) and len(win) == 2:
613
+ bits.append(f"window frames {win[0]}–{win[1]}")
614
+ if row["n_frames"]:
615
+ bits.append(f"of {row['n_frames']}")
616
+ if row["critical_frame"] is not None:
617
+ bits.append(f"critical frame {row['critical_frame']}")
618
+ if bits:
619
+ out.append(" · ".join(bits))
620
+ out.append(
621
+ f"\n<sub>cached this session: {session.disk_windows()}/{MAX_WINDOWS_ON_DISK} windows "
622
+ "— evicted least-recently-used, all deleted when you close the tab</sub>"
623
  )
624
+ return "\n".join(out)
625
 
626
 
627
  def table_rows(rows: list[dict]) -> list[list]:
628
+ def num(v):
629
+ return round(v, 2) if isinstance(v, (int, float)) else None
630
+
631
  return [
632
+ [short_id(r["episode_id"]), r["task"], r["edge"], r["arm"], num(r["crit_psnr"]), num(r["full_psnr"])]
 
 
 
 
 
 
 
633
  for r in rows
634
  ]
635
 
636
 
637
+ def critical_index(row: dict) -> int:
638
+ """Offset of the critical frame inside the window, or NO_CRIT."""
639
+ start, crit = row.get("start_frame"), row.get("critical_frame")
640
+ if start is None or crit is None:
641
+ return NO_CRIT
642
+ return int(crit) - int(start)
643
+
644
+
645
+ def on_load(repo_text, sid):
646
+ """Resolve a repo and load only its index — no window data is pulled here."""
647
+ sweep()
648
+ session = get_session(sid)
649
+ repo, revision = parse_repo(repo_text)
650
+ if not repo:
651
+ return session.sid, "Enter a dataset repo id, e.g. `owner/name`."
652
+ session.set_repo(repo, revision)
653
+ try:
654
+ note = build_index(session)
655
+ except Exception as exc:
656
+ session.rows = []
657
+ gr.Warning(f"Could not load {repo}: {exc}")
658
+ return session.sid, f"❌ **{repo}** — {exc}"
659
+ at = f"@{revision}" if revision else ""
660
+ return session.sid, (
661
+ f"✅ **{repo}{at}** — {len(session.rows)} windows from {note}. "
662
+ "Window files download only when you open them."
663
+ )
664
+
665
+
666
+ def on_filter(sid, query, edge, arm, sort, current):
667
+ session = get_session(sid)
668
+ if not session.rows:
669
+ return gr.update(choices=[], value=None), [], []
670
+ rows = filter_rows(session, query, edge, arm, sort)
671
  choices = [(label_for(r), r["episode_id"]) for r in rows]
672
  ids = [r["episode_id"] for r in rows]
673
  value = current if current in ids else (ids[0] if ids else None)
674
  return (
675
+ gr.update(choices=choices, value=value, label=f"Window ({len(ids)} of {len(session.rows)})"),
676
  table_rows(rows),
677
  ids,
678
  )
679
 
680
 
681
+ def on_select(sid, episode_id, view, show_diff, fps, frame_idx):
682
+ session = get_session(sid)
683
+ if not session.rows or not episode_id:
684
+ return None, None, "*No window selected.*", None, gr.update(), gr.update()
685
+ row = row_by_id(session, episode_id)
686
+ crit = critical_index(row)
687
 
688
+ cams = n_cameras(session.pair(episode_id)[0])
689
+ choices = camera_choices(cams)
690
+ valid = [v for _, v in choices]
691
+ view = view if view in valid else choices[0][1]
692
+ cam_update = gr.update(choices=choices, value=view)
693
 
694
+ video = render_video(session, episode_id, view, show_diff, fps, crit)
695
+ frames = session.composed(episode_id, view, show_diff, crit)
696
+ last = len(frames) - 1
697
+ i = int(min(frame_idx, last))
698
+ plot = plot_actions(session, episode_id, crit, i)
699
+ slider = gr.update(maximum=last, value=i, label=f"Frame in window (0–{last})")
700
+ return video, frames[i], metrics_md(session, row), plot, slider, cam_update
 
 
701
 
702
 
703
+ def on_frame(sid, episode_id, view, show_diff, frame_idx):
704
+ session = get_session(sid)
705
+ if not session.rows or not episode_id:
706
  return None, None
707
+ crit = critical_index(row_by_id(session, episode_id))
708
+ frames = session.composed(episode_id, view, show_diff, crit)
709
  i = int(min(frame_idx, len(frames) - 1))
710
+ return frames[i], plot_actions(session, episode_id, crit, i)
711
 
712
 
713
  def step(ids, current, delta):
 
727
  .window-still img { max-height: 78vh; object-fit: contain; background: #111; }
728
  """
729
 
730
+ with gr.Blocks(title="Rollout vs. ground truth", fill_width=True, delete_cache=(600, 600)) as demo:
731
+ gr.Markdown("# Forward-dynamics rollout vs. ground truth")
732
+
733
+ sid_state = gr.State("", time_to_live=SESSION_TTL, delete_callback=release)
 
 
 
734
  ids_state = gr.State([])
735
 
736
+ with gr.Row():
737
+ repo_box = gr.Textbox(
738
+ DEFAULT_REPO,
739
+ label="Dataset repo",
740
+ info="owner/name, optionally owner/name@revision. A repo URL also works.",
741
+ scale=5,
742
+ submit_btn=True,
743
+ )
744
+ load_btn = gr.Button("Load", variant="primary", scale=1)
745
+ status = gr.Markdown()
746
+
747
  with gr.Row():
748
  with gr.Column(scale=1, min_width=320):
749
  query = gr.Textbox(label="Search task or episode", placeholder="e.g. napkin, pillowcase…")
 
756
  prev_btn = gr.Button("← Prev")
757
  next_btn = gr.Button("Next →")
758
  gr.Markdown("---")
759
+ view = gr.Radio(camera_choices(3), value=0, label="Camera")
760
  show_diff = gr.Checkbox(False, label="Add |GT − generated| difference panel")
761
+ fps = gr.Slider(2, 30, value=8, step=1, label="Playback fps")
762
  info = gr.Markdown()
763
 
764
  with gr.Column(scale=2):
 
772
  elem_classes="window-video",
773
  )
774
  with gr.Tab("Frame stepper"):
775
+ frame_idx = gr.Slider(0, 32, value=16, step=1, label="Frame in window")
776
  still = gr.Image(label="Frame comparison", elem_classes="window-still")
777
  with gr.Tab("Driving actions"):
778
+ actions_plot = gr.Plot(label="Driving action chunk")
779
  with gr.Tab("All windows"):
780
  table = gr.Dataframe(
781
  headers=["episode", "task", "edge", "arm", "crit PSNR", "full PSNR"],
 
785
  max_height=650,
786
  )
787
 
788
+ filter_inputs = [sid_state, query, edge, arm, sort]
789
  filter_outputs = [picker, table, ids_state]
790
+ view_inputs = [sid_state, picker, view, show_diff, fps, frame_idx]
791
+ view_outputs = [video, still, info, actions_plot, frame_idx, view]
792
+
793
+ def wire_load(event):
794
+ return (
795
+ event(on_load, [repo_box, sid_state], [sid_state, status])
796
+ .then(on_filter, filter_inputs + [picker], filter_outputs)
797
+ .then(on_select, view_inputs, view_outputs)
798
+ )
799
 
800
+ wire_load(load_btn.click)
801
+ wire_load(repo_box.submit)
802
+ wire_load(demo.load)
803
+
804
+ for control in (query, edge, arm, sort):
805
  control.change(on_filter, filter_inputs + [picker], filter_outputs).then(
806
  on_select, view_inputs, view_outputs
807
  )
 
809
  picker.change(on_select, view_inputs, view_outputs)
810
  for control in (view, show_diff, fps):
811
  control.change(on_select, view_inputs, view_outputs)
812
+ frame_idx.change(on_frame, [sid_state, picker, view, show_diff, frame_idx], [still, actions_plot])
813
 
814
  prev_btn.click(lambda i, c: step(i, c, -1), [ids_state, picker], picker)
815
  next_btn.click(lambda i, c: step(i, c, +1), [ids_state, picker], picker)
 
819
 
820
  table.select(on_table_select, ids_state, picker)
821
 
 
 
 
 
822
  if __name__ == "__main__":
823
  demo.queue(max_size=16).launch(css=CSS)