| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>AV Curation Pipeline — e2e run on 9NcjRFu6C4I</title> |
| <style> |
| :root { |
| --bg: #0f1117; --card: #181b24; --border: #2a2f3d; |
| --text: #e8eaf0; --muted: #9aa1b5; --accent: #7aa2ff; |
| --ok: #8fd3a7; --warn: #f0b86a; --bad: #f28b82; |
| } |
| * { box-sizing: border-box; margin: 0; padding: 0; } |
| body { background: var(--bg); color: var(--text); |
| font: 15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; |
| padding: 2rem 1.25rem 4rem; } |
| .wrap { max-width: 1050px; margin: 0 auto; } |
| h1 { font-size: 1.45rem; margin-bottom: .35rem; } |
| .sub { color: var(--muted); font-size: .92rem; margin-bottom: 1.4rem; } |
| code { color: var(--accent); background: var(--card); padding: .08em .35em; border-radius: 4px; font-size: .88em; } |
| .flow { color: var(--muted); font-size: .85rem; background: var(--card); border: 1px solid var(--border); |
| border-radius: 10px; padding: .8rem 1rem; margin-bottom: 1.6rem; overflow-x: auto; white-space: nowrap; } |
| .flow b { color: var(--text); } |
| .step { background: var(--card); border: 1px solid var(--border); border-radius: 12px; |
| padding: 1.1rem 1.25rem; margin-bottom: 1.4rem; } |
| .step-head { display: flex; align-items: baseline; gap: .7rem; flex-wrap: wrap; margin-bottom: .35rem; } |
| .num { color: var(--accent); font-weight: 700; font-size: .85rem; } |
| .step h2 { font-size: 1.08rem; } |
| .badge { font-size: .72rem; font-weight: 600; letter-spacing: .04em; text-transform: uppercase; |
| padding: .16em .55em; border-radius: 99px; border: 1px solid var(--border); } |
| .real { color: var(--ok); border-color: #2e5d41; background: #16281d; } |
| .cached { color: var(--accent); border-color: #33518c; background: #141c30; } |
| .stand { color: var(--warn); border-color: #6b5228; background: #2a2113; } |
| .mock { color: var(--bad); border-color: #6d3733; background: #2a1715; } |
| .io { color: var(--muted); font-size: .86rem; margin-bottom: .8rem; } |
| .io b { color: var(--text); font-weight: 600; } |
| .row { display: grid; grid-template-columns: repeat(auto-fit, minmax(230px, 1fr)); gap: .9rem; } |
| .cell { background: #12151d; border: 1px solid var(--border); border-radius: 8px; padding: .6rem; } |
| .cell h3 { font-size: .8rem; font-weight: 600; color: var(--muted); margin-bottom: .45rem; } |
| video, audio { width: 100%; border-radius: 5px; display: block; background: #000; } |
| audio { height: 34px; } |
| .quote { border-left: 3px solid var(--accent); padding: .5rem .9rem; color: var(--text); |
| background: #12151d; border-radius: 0 6px 6px 0; font-size: .93rem; } |
| .note { color: var(--muted); font-size: .84rem; margin-top: .7rem; } |
| .metric { font-size: .95rem; margin: .4rem 0; } |
| .metric b { color: var(--ok); } |
| table { border-collapse: collapse; width: 100%; font-size: .86rem; } |
| th, td { text-align: left; padding: .35rem .6rem; border-bottom: 1px solid var(--border); } |
| th { color: var(--muted); font-weight: 600; } |
| tr.winner td { color: var(--ok); font-weight: 600; background: #16281d; } |
| .tablewrap { overflow-x: auto; } |
| .gates td:nth-child(2), .gates td:nth-child(3) { font-variant-numeric: tabular-nums; } |
| footer { color: var(--muted); font-size: .82rem; margin-top: 2.5rem; border-top: 1px solid var(--border); padding-top: 1rem; } |
| </style> |
| </head> |
| <body> |
| <div class="wrap"> |
| <h1>AV Curation Pipeline — end-to-end run on <code>9NcjRFu6C4I</code></h1> |
| <p class="sub"> |
| LangGraph agentic pipeline (<code>av_langgraph_pipeline</code>), Slurm job 386863 on an H100, 2026-07-22, |
| total 656 s — final status <b style="color:var(--ok)">passed</b>. Each card below shows one graph node |
| with its input and output. Badges: <span class="badge real">real</span> model ran in this job, |
| <span class="badge cached">cached</span> real output reused from the previous job, |
| <span class="badge stand">stand-in</span> offline substitute, <span class="badge mock">mock</span> not yet implemented. |
| </p> |
|
|
| <div class="flow"> |
| <b>caption</b> → <b>sounding-object</b> → <b>SAM3 segmentation</b> → gate → <b>EffectErase inpaint</b> → |
| <b>visual check</b> → gate → <b>SAM-Audio best-of-10 + ImageBind select</b> → <b>audio check</b> → gate → <b>paired output</b> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">INPUT</span><h2>Source AV clip</h2></div> |
| <p class="io"><b>8 s, 24 fps train-simulator clip with audio</b> — plus the preprocessed mono wav that feeds SAM-Audio.</p> |
| <div class="row"> |
| <div class="cell"><h3>input.mp4 (with audio)</h3><video src="media/input.mp4" controls loop playsinline></video></div> |
| <div class="cell"><h3>extracted source audio</h3><audio src="media/source_audio.wav" controls></audio></div> |
| </div> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 1</span><h2>AV caption generation</h2><span class="badge cached">cached real — Qwen3-Omni 30B</span></div> |
| <p class="io"><b>In:</b> input video+audio · <b>Out:</b> caption text</p> |
| <p class="quote">“A blue and yellow CSX locomotive travels along a track in a flat, grassy landscape. As it approaches a junction, |
| the train derails and begins to flip over, lifting its front end high into the air while the rear remains on the ground. |
| The train continues to tilt and rotate, eventually becoming almost vertical. The sound of a train horn is heard, |
| followed by the clatter of wheels on the tracks.”</p> |
| <p class="note">Generated by Qwen3-Omni in the first job of this session and reused here (identical node, cached to avoid ~10 min of recompute).</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 2</span><h2>Sounding-object extraction</h2><span class="badge stand">offline stand-in</span></div> |
| <p class="io"><b>In:</b> caption · <b>Out:</b> target object = <code>train engine</code></p> |
| <p class="note">Normally GPT (<code>gpt-4o-mini</code>) via the OpenAI API; the provided key had no quota (429 insufficient_quota), |
| so an offline rule returned “train engine” — the same prompt used by all earlier experiments on this clip, and the phrase SAM3 |
| responds to far better than the bare “train” (first-frame ratio 0.095 vs 0.002).</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 3</span><h2>Target-object segmentation (SAM3)</h2><span class="badge real">real</span></div> |
| <p class="io"><b>In:</b> input video + prompt “train engine” · <b>Out:</b> whole-video mask (+ overlay / masked previews)</p> |
| <div class="row"> |
| <div class="cell"><h3>overlay (mask on video)</h3><video src="media/sam3_overlay.mp4" controls loop muted playsinline></video></div> |
| <div class="cell"><h3>binary mask</h3><video src="media/sam3_mask.mp4" controls loop muted playsinline></video></div> |
| <div class="cell"><h3>masked (object zeroed)</h3><video src="media/sam3_masked.mp4" controls loop muted playsinline></video></div> |
| </div> |
| <p class="metric">first-frame / mask-area ratio: <b>0.0951</b></p> |
| <p class="note">SAM3 tracks this synthetic (game-rendered) train in only ~9 of 202 frames — the mask goes empty after the opening |
| frames. That weakness drives everything downstream on this clip and is why the demo gates below were lowered.</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 4</span><h2>Object-removal inpainting (EffectErase)</h2><span class="badge real">real</span></div> |
| <p class="io"><b>In:</b> input video + SAM3 mask · <b>Out:</b> inpainted video (object visually removed)</p> |
| <div class="row"> |
| <div class="cell"><h3>before (input)</h3><video src="media/input.mp4" controls loop muted playsinline></video></div> |
| <div class="cell"><h3>after (inpainted)</h3><video src="media/inpainted.mp4" controls loop muted playsinline></video></div> |
| </div> |
| <p class="note">With only ~9 masked frames, EffectErase's temporal context (train visible in every unmasked frame) largely |
| repaints the train — an honest failure case for this clip, quantified by the next step.</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 5</span><h2>Visual removal check</h2><span class="badge real">real</span></div> |
| <p class="io"><b>In:</b> inpainted video + original mask · <b>Out:</b> removal score (SAM3 re-segmentation ratio)</p> |
| <p class="metric">visual_removal_score: <b>0.4352</b> — passed the demo gate (0.40); the production gate is 0.80.</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 6</span><h2>SAM-Audio best-of-10 + ImageBind selection</h2><span class="badge real">real</span></div> |
| <p class="io"><b>In:</b> source audio + SAM3 mask video + prompt “train engine” · |
| <b>Out:</b> 10 candidates (visual & text prompts × 5 seeds), winner picked by max <code>ib_ta</code> |
| (ImageBind text↔target), tiebreak min <code>ib_ta_res</code> (text↔residual, lower = cleaner removal).</p> |
| <div class="tablewrap"> |
| <table> |
| <tr><th>rank</th><th>prompt</th><th>seed</th><th>ib_ta ↑</th><th>ib_ta_res ↓</th><th>target (train engine)</th><th>residual (removed)</th></tr> |
| <tr class="winner"><td>1 ★</td><td>visual</td><td>1453635084</td><td>0.2335</td><td>0.0835</td> |
| <td><audio src="media/visual_seed_1453635084_target.wav" controls></audio></td> |
| <td><audio src="media/visual_seed_1453635084_residual.wav" controls></audio></td></tr> |
| <tr><td>2</td><td>visual</td><td>240868205</td><td>0.1889</td><td>0.0220</td> |
| <td><audio src="media/visual_seed_240868205_target.wav" controls></audio></td> |
| <td><audio src="media/visual_seed_240868205_residual.wav" controls></audio></td></tr> |
| <tr><td>3</td><td>text</td><td>1335522078</td><td>0.1781</td><td>0.1930</td> |
| <td><audio src="media/text_seed_1335522078_target.wav" controls></audio></td> |
| <td><audio src="media/text_seed_1335522078_residual.wav" controls></audio></td></tr> |
| <tr><td>4</td><td>visual</td><td>1335522078</td><td>0.1727</td><td>0.1741</td> |
| <td><audio src="media/visual_seed_1335522078_target.wav" controls></audio></td> |
| <td><audio src="media/visual_seed_1335522078_residual.wav" controls></audio></td></tr> |
| <tr><td>5</td><td>text</td><td>240868205</td><td>0.1397</td><td>0.0393</td> |
| <td><audio src="media/text_seed_240868205_target.wav" controls></audio></td> |
| <td><audio src="media/text_seed_240868205_residual.wav" controls></audio></td></tr> |
| <tr><td>6</td><td>text</td><td>1453635084</td><td>0.0915</td><td>0.1489</td> |
| <td><audio src="media/text_seed_1453635084_target.wav" controls></audio></td> |
| <td><audio src="media/text_seed_1453635084_residual.wav" controls></audio></td></tr> |
| <tr><td>7</td><td>visual</td><td>1132891577</td><td>0.0731</td><td>0.2736</td> |
| <td><audio src="media/visual_seed_1132891577_target.wav" controls></audio></td> |
| <td><audio src="media/visual_seed_1132891577_residual.wav" controls></audio></td></tr> |
| <tr><td>8</td><td>text</td><td>1132891577</td><td>0.0588</td><td>0.2716</td> |
| <td><audio src="media/text_seed_1132891577_target.wav" controls></audio></td> |
| <td><audio src="media/text_seed_1132891577_residual.wav" controls></audio></td></tr> |
| <tr><td>9</td><td>text</td><td>1778986134</td><td>0.0453</td><td>0.2587</td> |
| <td><audio src="media/text_seed_1778986134_target.wav" controls></audio></td> |
| <td><audio src="media/text_seed_1778986134_residual.wav" controls></audio></td></tr> |
| <tr><td>10</td><td>visual</td><td>1778986134</td><td>0.0242</td><td>0.2637</td> |
| <td><audio src="media/visual_seed_1778986134_target.wav" controls></audio></td> |
| <td><audio src="media/visual_seed_1778986134_residual.wav" controls></audio></td></tr> |
| </table> |
| </div> |
| <p class="note">Generation: SAM-Audio large, one model load, seeded per candidate (samaudio311 env). |
| Scoring: same <code>calc_imagebind_score</code> as the ranking_test eval (JavisDiT repo, javisdit env).</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 7</span><h2>Audio removal check</h2><span class="badge mock">mock</span></div> |
| <p class="io"><b>In:</b> winner residual · <b>Out:</b> audio_removal_score = 0.9 (placeholder checker, passes the 0.80 gate)</p> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">STEP 8</span><h2>Paired AV output</h2></div> |
| <p class="io"><b>Out:</b> inpainted video paired with the winning object-removed audio.</p> |
| <div class="row"> |
| <div class="cell"><h3>final: inpainted video + best residual audio (unmute!)</h3><video src="media/final_paired.mp4" controls loop playsinline></video></div> |
| <div class="cell"><h3>best residual (train engine removed)</h3><audio src="media/best_residual.wav" controls></audio> |
| <h3 style="margin-top:.7rem">best target (isolated train engine)</h3><audio src="media/best_target.wav" controls></audio></div> |
| </div> |
| </div> |
|
|
| <div class="step"> |
| <div class="step-head"><span class="num">CONFIG</span><h2>Demo gate overrides</h2></div> |
| <div class="tablewrap"> |
| <table class="gates"> |
| <tr><th>gate</th><th>production</th><th>this demo run</th><th>observed value</th></tr> |
| <tr><td>SAM3 first-frame ratio</td><td>0.10 (dev; 0.80 target)</td><td>0.05</td><td>0.0951</td></tr> |
| <tr><td>whole-video mask area</td><td>0.15</td><td>0.05</td><td>0.0951</td></tr> |
| <tr><td>visual removal score</td><td>0.80</td><td>0.40</td><td>0.4352</td></tr> |
| </table> |
| </div> |
| <p class="note">Lowered (via env vars, defaults unchanged in code) solely so this hard clip traverses every stage for |
| visualization; at production thresholds the pipeline correctly discards it at the mask gate.</p> |
| </div> |
|
|
| <footer> |
| Run artifacts: <code>data/work/</code> in <code>av_langgraph_pipeline</code>; candidate scores in <code>selection.json</code> |
| (also uploaded alongside this page). Related Space: JavisDiT edit-model samples at <code>WitneyWW/javisdit-edit-text-both</code>. |
| </footer> |
| </div> |
| </body> |
| </html> |
|
|
