Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>World Memory — Knownness-Conditioned Video Generation</title> | |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/reveal.css"> | |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/theme/black.css" id="theme"> | |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/plugin/highlight/monokai.css"> | |
| <style> | |
| :root{ | |
| --bg0:#0a0e1a; --bg1:#0f1626; --ink:#e8ecf6; --muted:#9aa6c4; | |
| --teal:#35d0ba; --purple:#8b7cff; --amber:#ffb454; --coral:#ff6b81; | |
| --known:#35d0ba; --unknown:#ff6b9d; --line:#2a3550; | |
| } | |
| .reveal { font-family: "Inter","Helvetica Neue",Arial,sans-serif; color:var(--ink); font-weight:400; } | |
| .reveal .slides { text-align:left; } | |
| .reveal section { padding:0 12px; } | |
| .reveal h1,.reveal h2,.reveal h3,.reveal h4{ color:var(--ink); font-weight:700; letter-spacing:-.01em; text-transform:none; text-shadow:none; } | |
| .reveal h1{ font-size:2.35em; line-height:1.05; } | |
| .reveal h2{ font-size:1.65em; } | |
| .reveal h3{ font-size:1.15em; } | |
| .reveal p,.reveal li{ color:var(--ink); font-size:.72em; line-height:1.5; } | |
| .reveal a{ color:var(--teal); } | |
| .reveal em{ color:var(--amber); font-style:normal; } | |
| .reveal strong{ color:var(--purple); font-weight:700; } | |
| .reveal .slide-background{ background:radial-gradient(1200px 800px at 20% -10%, #16203a 0%, var(--bg1) 40%, var(--bg0) 100%); } | |
| .kicker{ color:var(--teal); font-size:.6em; font-weight:700; letter-spacing:.22em; text-transform:uppercase; margin:0 0 .5em; } | |
| .phase-tag{ display:inline-block; font-size:.5em; font-weight:700; letter-spacing:.15em; text-transform:uppercase; | |
| padding:.35em .8em; border-radius:999px; margin-bottom:.7em; } | |
| .p1{ background:rgba(53,208,186,.14); color:var(--teal); border:1px solid rgba(53,208,186,.4); } | |
| .p2{ background:rgba(139,124,255,.14); color:var(--purple); border:1px solid rgba(139,124,255,.4); } | |
| .p3{ background:rgba(255,180,84,.14); color:var(--amber); border:1px solid rgba(255,180,84,.4); } | |
| .p4{ background:rgba(255,107,129,.14); color:var(--coral); border:1px solid rgba(255,107,129,.4); } | |
| .cols{ display:flex; gap:26px; align-items:flex-start; } | |
| .col{ flex:1; } | |
| .card{ background:rgba(255,255,255,.035); border:1px solid var(--line); border-radius:14px; padding:16px 18px; } | |
| .card h3{ margin:.1em 0 .35em; font-size:1.05em; } | |
| .card p, .card li{ font-size:.66em; color:var(--muted); } | |
| .card .num{ font-size:.5em; font-weight:700; letter-spacing:.1em; color:var(--teal); } | |
| .reveal ul{ margin-left:1.05em; } | |
| .reveal li{ margin:.28em 0; } | |
| .muted{ color:var(--muted); } | |
| .eq{ background:rgba(255,255,255,.04); border:1px solid var(--line); border-left:3px solid var(--teal); | |
| border-radius:10px; padding:12px 18px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.8em; color:var(--ink); } | |
| .footer{ position:absolute; bottom:14px; left:24px; font-size:12px; color:var(--muted); letter-spacing:.05em; } | |
| .pageno{ position:absolute; bottom:14px; right:24px; font-size:12px; color:var(--muted); } | |
| .chip{ display:inline-block; font-size:.62em; padding:.25em .7em; margin:.15em .2em 0 0; border-radius:8px; | |
| background:rgba(255,255,255,.05); border:1px solid var(--line); color:var(--ink); } | |
| .big{ font-size:1.15em; } | |
| .lead{ font-size:.9em; color:var(--muted); max-width:22em; } | |
| .reveal table{ font-size:.62em; } | |
| .reveal td,.reveal th{ border-color:var(--line); padding:.4em .7em; } | |
| .reveal th{ color:var(--teal); } | |
| .gridnums{ display:flex; gap:18px; } | |
| .gridnums .stat{ text-align:center; background:rgba(255,255,255,.03); border:1px solid var(--line); border-radius:14px; padding:16px 10px; flex:1; } | |
| .gridnums .stat .v{ font-size:1.9em; font-weight:800; color:var(--teal); line-height:1; } | |
| .gridnums .stat .l{ font-size:.5em; color:var(--muted); letter-spacing:.05em; margin-top:.5em; } | |
| .figwrap{ width:100%; } | |
| .figwrap svg{ width:100%; height:auto; display:block; } | |
| .center{ text-align:center; } | |
| .titlewrap{ text-align:center; } | |
| .titlewrap .sub{ font-size:.85em; color:var(--muted); margin-top:.2em; } | |
| .authors{ margin-top:1.4em; color:var(--muted); font-size:.62em; letter-spacing:.04em; } | |
| .legendrow{ display:flex; gap:8px; align-items:center; font-size:.6em; color:var(--muted); } | |
| .swatch{ width:14px; height:14px; border-radius:4px; display:inline-block; } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="reveal"><div class="slides"> | |
| <!-- 1. TITLE --> | |
| <section data-auto-animate> | |
| <div class="titlewrap" style="margin-top:6%;"> | |
| <p class="kicker">CoRL 2026 · Long-Video World Models</p> | |
| <h1>World Memory</h1> | |
| <p class="sub big"><strong style="color:var(--amber)">Knownness-Conditioned</strong> Long-Video Generation</p> | |
| <p class="lead" style="margin:1.1em auto 0;">A frozen video diffusion model that <em>reuses what it has seen</em> — and generates only what it hasn't.</p> | |
| <p class="authors">A 3D-grounded memory bank · per-token noise scheduling · write-back rollout</p> | |
| </div> | |
| <div class="footer">World Memory · working title</div> | |
| </section> | |
| <!-- 2. PROBLEM --> | |
| <section> | |
| <p class="kicker">The problem</p> | |
| <h2>Video models <em>re-hallucinate</em> on revisit</h2> | |
| <div class="cols" style="margin-top:22px;"> | |
| <div class="col"> | |
| <p>Generate a long video where the camera moves away and comes <strong>back</strong> to a place it already saw.</p> | |
| <ul> | |
| <li>Frame-by-frame diffusion has <em>no memory of 3D</em>.</li> | |
| <li>On return, the same wall / table / sky is <em>re-invented differently</em> — flicker, drift, ghosting.</li> | |
| <li>Naïvely copying old pixels (splatting) leaves holes, seams, and stale geometry.</li> | |
| </ul> | |
| </div> | |
| <div class="col"> | |
| <div class="card"> | |
| <p class="num">GOAL</p> | |
| <h3>Revisit ⇒ recall, not re-generate</h3> | |
| <p>When the camera returns to a known place, the model should <strong>restore the same scene</strong> from memory — and still <strong>freely generate</strong> genuinely new regions.</p> | |
| </div> | |
| <div class="card" style="margin-top:14px;"> | |
| <p class="num">KEY IDEA</p> | |
| <h3>One scalar per token: <span style="color:var(--teal)">knownness k</span></h3> | |
| <p>Route each space-time token between <em>reuse-and-repair</em> and <em>generate-from-noise</em> — inside a single frozen diffusion model.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="pageno">2</div> | |
| </section> | |
| <!-- 3. ONE-LINE IDEA --> | |
| <section> | |
| <p class="kicker">The recipe in one line</p> | |
| <h2 style="margin-bottom:24px;">Memory sets the <span style="color:var(--teal)">noise level</span>; diffusion does the rest</h2> | |
| <div class="cols"> | |
| <div class="col card" style="border-left:3px solid var(--teal);"> | |
| <p class="num" style="color:var(--teal)">OFFLINE</p> | |
| <h3>Build a 3D world memory</h3> | |
| <p>Depth + poses → a persistent voxel point cloud. Reproject it into every target camera to get <strong>z_mem</strong> (what should be there), <strong>k</strong> (how sure), and <strong>Xtok</strong> (where in 3D).</p> | |
| </div> | |
| <div class="col card" style="border-left:3px solid var(--coral);"> | |
| <p class="num" style="color:var(--coral)">ONLINE</p> | |
| <h3>Per-token noise from k</h3> | |
| <p>Wan DiT natively supports a <em>different diffusion timestep per token</em>. Known → low noise (repair). Unknown → high noise (generate). New output is <strong>written back</strong> into memory.</p> | |
| </div> | |
| </div> | |
| <div class="eq center" style="margin-top:26px;">high k → start near <span style="color:var(--teal)">z_mem</span> · short denoise | low k → start from <span style="color:var(--coral)">noise</span> · full generation</div> | |
| <div class="pageno">3</div> | |
| </section> | |
| <!-- 4. PIPELINE FIGURE --> | |
| <section data-background-color="#080b14"> | |
| <p class="kicker" style="margin-bottom:6px;">System overview</p> | |
| <div class="figwrap"> | |
| <!-- ===================== PIPELINE SVG ===================== --> | |
| <svg viewBox="0 0 1720 760" xmlns="http://www.w3.org/2000/svg" font-family="Inter, Arial, sans-serif"> | |
| <defs> | |
| <marker id="arrow" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto" markerUnits="strokeWidth"> | |
| <path d="M0,0 L7,3 L0,6 Z" fill="#9aa6c4"/> | |
| </marker> | |
| <marker id="arrowT" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto" markerUnits="strokeWidth"> | |
| <path d="M0,0 L7,3 L0,6 Z" fill="#35d0ba"/> | |
| </marker> | |
| <marker id="arrowC" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto" markerUnits="strokeWidth"> | |
| <path d="M0,0 L7,3 L0,6 Z" fill="#ff6b81"/> | |
| </marker> | |
| <linearGradient id="kbar" x1="0" y1="0" x2="1" y2="0"> | |
| <stop offset="0" stop-color="#ff6b9d"/><stop offset="0.5" stop-color="#8b7cff"/><stop offset="1" stop-color="#35d0ba"/> | |
| </linearGradient> | |
| <linearGradient id="lanep1" x1="0" y1="0" x2="0" y2="1"> | |
| <stop offset="0" stop-color="#35d0ba" stop-opacity="0.10"/><stop offset="1" stop-color="#35d0ba" stop-opacity="0.02"/> | |
| </linearGradient> | |
| <linearGradient id="lanep2" x1="0" y1="0" x2="0" y2="1"> | |
| <stop offset="0" stop-color="#ff6b81" stop-opacity="0.10"/><stop offset="1" stop-color="#ff6b81" stop-opacity="0.02"/> | |
| </linearGradient> | |
| <style> | |
| .lbl{ fill:#e8ecf6; font-size:15px; font-weight:600; } | |
| .sub{ fill:#9aa6c4; font-size:12px; } | |
| .hdr{ font-size:13px; font-weight:800; letter-spacing:2px; } | |
| .box{ fill:#111a2e; stroke:#2a3550; stroke-width:1.5; } | |
| .flow{ stroke:#9aa6c4; stroke-width:2; fill:none; } | |
| .flowT{ stroke:#35d0ba; stroke-width:2; fill:none; } | |
| .tiny{ fill:#9aa6c4; font-size:11px; } | |
| </style> | |
| </defs> | |
| <!-- lane backgrounds --> | |
| <rect x="278" y="70" width="500" height="600" rx="18" fill="url(#lanep1)" stroke="#35d0ba" stroke-opacity="0.25"/> | |
| <rect x="1052" y="70" width="470" height="600" rx="18" fill="url(#lanep2)" stroke="#ff6b81" stroke-opacity="0.25"/> | |
| <!-- lane headers --> | |
| <text x="150" y="52" class="hdr" fill="#9aa6c4" text-anchor="middle">INPUT</text> | |
| <text x="528" y="52" class="hdr" fill="#35d0ba" text-anchor="middle">PHASE 1 · OFFLINE WORLD MEMORY</text> | |
| <text x="915" y="52" class="hdr" fill="#8b7cff" text-anchor="middle">MEMORY BUNDLE</text> | |
| <text x="1287" y="52" class="hdr" fill="#ff6b81" text-anchor="middle">PHASE 2 / 4 · ONLINE DiT</text> | |
| <text x="1632" y="52" class="hdr" fill="#9aa6c4" text-anchor="middle">OUTPUT</text> | |
| <!-- INPUT --> | |
| <rect class="box" x="40" y="300" width="220" height="150" rx="12"/> | |
| <text x="150" y="336" class="lbl" text-anchor="middle">Posed Video</text> | |
| <text x="150" y="358" class="sub" text-anchor="middle">832 × 480 · 81 frames</text> | |
| <!-- out-and-back glyph --> | |
| <path d="M70,410 C110,384 150,384 190,410" stroke="#35d0ba" stroke-width="2.5" fill="none"/> | |
| <path d="M190,410 C150,436 110,436 72,412" stroke="#ff6b81" stroke-width="2.5" fill="none" marker-end="url(#arrowC)"/> | |
| <circle cx="70" cy="410" r="4" fill="#e8ecf6"/> | |
| <text x="150" y="440" class="tiny" text-anchor="middle">out-and-back trajectory → forced revisit</text> | |
| <!-- PHASE 1 sub-boxes --> | |
| <rect class="box" x="308" y="92" width="440" height="70" rx="10"/> | |
| <text x="528" y="122" class="lbl" text-anchor="middle">Metric Depth + Global Scale Calibration</text> | |
| <text x="528" y="143" class="sub" text-anchor="middle">depth matched to camera baseline → 3D addresses only</text> | |
| <rect class="box" x="308" y="192" width="440" height="196" rx="10" stroke="#35d0ba" stroke-opacity="0.5"/> | |
| <text x="528" y="220" class="lbl" text-anchor="middle" fill="#35d0ba">PointMemory — world voxel point cloud</text> | |
| <text x="335" y="250" class="sub">• 3D position & average RGB</text> | |
| <text x="335" y="272" class="sub">• observation count</text> | |
| <text x="335" y="294" class="sub">• per-view directions</text> | |
| <text x="335" y="316" class="sub">• color variance → ghosting signal</text> | |
| <text x="528" y="352" class="tiny" text-anchor="middle">fuse ~11 observed frames · never forgets</text> | |
| <rect x="335" y="360" width="386" height="16" rx="4" fill="#0a0e1a" stroke="#2a3550"/> | |
| <text x="528" y="372" class="tiny" text-anchor="middle">persistent across the whole rollout</text> | |
| <rect class="box" x="308" y="418" width="440" height="76" rx="10"/> | |
| <text x="528" y="448" class="lbl" text-anchor="middle">Reproject → target cameras</text> | |
| <text x="528" y="470" class="sub" text-anchor="middle">z-buffer · adaptive splat radius · depth test</text> | |
| <!-- internal phase-1 arrows --> | |
| <line class="flowT" x1="528" y1="162" x2="528" y2="192" marker-end="url(#arrowT)"/> | |
| <line class="flowT" x1="528" y1="388" x2="528" y2="418" marker-end="url(#arrowT)"/> | |
| <!-- BUNDLE boxes --> | |
| <rect class="box" x="800" y="150" width="230" height="72" rx="10" stroke="#8b7cff" stroke-opacity="0.5"/> | |
| <text x="915" y="180" class="lbl" text-anchor="middle" fill="#8b7cff">z_mem</text> | |
| <text x="915" y="202" class="sub" text-anchor="middle">VAE-encoded memory reprojection</text> | |
| <rect class="box" x="800" y="242" width="230" height="80" rx="10" stroke="#8b7cff" stroke-opacity="0.5"/> | |
| <text x="915" y="268" class="lbl" text-anchor="middle" fill="#8b7cff">knownness k</text> | |
| <rect x="828" y="282" width="174" height="12" rx="6" fill="url(#kbar)"/> | |
| <text x="828" y="312" class="tiny">unknown</text><text x="1002" y="312" class="tiny" text-anchor="end">known</text> | |
| <rect class="box" x="800" y="342" width="230" height="72" rx="10" stroke="#8b7cff" stroke-opacity="0.5"/> | |
| <text x="915" y="372" class="lbl" text-anchor="middle" fill="#8b7cff">Xtok</text> | |
| <text x="915" y="394" class="sub" text-anchor="middle">per-token 3D world position</text> | |
| <rect class="box" x="800" y="434" width="230" height="72" rx="10" stroke="#8b7cff" stroke-opacity="0.5"/> | |
| <text x="915" y="464" class="lbl" text-anchor="middle" fill="#8b7cff">memory slot bank</text> | |
| <text x="915" y="486" class="sub" text-anchor="middle">compact 3D key/value memory</text> | |
| <!-- DiT --> | |
| <rect class="box" x="1080" y="92" width="414" height="470" rx="12" stroke="#ff6b81" stroke-opacity="0.5"/> | |
| <text x="1287" y="122" class="lbl" text-anchor="middle" fill="#ff6b81">Frozen Wan Video DiT</text> | |
| <text x="1287" y="142" class="sub" text-anchor="middle">+ small zero-init trainable modules</text> | |
| <rect x="1104" y="160" width="366" height="66" rx="8" fill="#0d1526" stroke="#2a3550"/> | |
| <text x="1287" y="186" class="lbl" text-anchor="middle">① Per-token noise level from k</text> | |
| <text x="1287" y="207" class="tiny" text-anchor="middle">low k → high noise · high k → low noise</text> | |
| <rect x="1104" y="238" width="366" height="66" rx="8" fill="#0d1526" stroke="#2a3550"/> | |
| <text x="1287" y="264" class="lbl" text-anchor="middle">② 3D positional encoding (Xtok)</text> | |
| <text x="1287" y="285" class="tiny" text-anchor="middle">same world point = same identity under camera motion</text> | |
| <rect x="1104" y="316" width="366" height="70" rx="8" fill="#0d1526" stroke="#2a3550"/> | |
| <text x="1287" y="342" class="lbl" text-anchor="middle">③ Geometry cross-attn · every 5 blocks</text> | |
| <text x="1287" y="363" class="tiny" text-anchor="middle">Xtok queries slot bank · Rope3D relative embedding</text> | |
| <rect x="1104" y="398" width="366" height="60" rx="8" fill="#0d1526" stroke="#2a3550"/> | |
| <text x="1287" y="424" class="lbl" text-anchor="middle">④ Confidence-gated shortcut</text> | |
| <text x="1287" y="444" class="tiny" text-anchor="middle">pass memory where k high · suppress where k low</text> | |
| <rect x="1104" y="470" width="366" height="66" rx="8" fill="#0a0e1a" stroke="#35d0ba" stroke-opacity="0.4"/> | |
| <text x="1287" y="496" class="lbl" text-anchor="middle" fill="#35d0ba">Zero-init</text> | |
| <text x="1287" y="516" class="tiny" text-anchor="middle">at start, modules output 0 → behaves like base Wan</text> | |
| <!-- OUTPUT --> | |
| <rect class="box" x="1548" y="300" width="150" height="150" rx="12"/> | |
| <text x="1623" y="368" class="lbl" text-anchor="middle">Generated</text> | |
| <text x="1623" y="390" class="lbl" text-anchor="middle">Video</text> | |
| <!-- MAIN FLOW ARROWS --> | |
| <line class="flow" x1="260" y1="375" x2="306" y2="290" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M748,300 C775,300 775,186 798,186" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M748,340 C775,340 775,282 798,282" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M748,400 C775,400 775,378 798,378" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M748,455 C775,455 775,470 798,470" marker-end="url(#arrow)"/> | |
| <!-- bundle -> DiT --> | |
| <path class="flow" d="M1030,186 C1055,186 1058,193 1102,193" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M1030,282 C1055,282 1058,271 1102,271" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M1030,378 C1055,300 1060,300 1102,290" marker-end="url(#arrow)"/> | |
| <path class="flow" d="M1030,470 C1055,360 1060,352 1102,348" marker-end="url(#arrow)"/> | |
| <!-- DiT -> output --> | |
| <line class="flow" x1="1494" y1="360" x2="1546" y2="368" marker-end="url(#arrow)"/> | |
| <!-- WRITE-BACK LOOP --> | |
| <path d="M1623,450 C1623,700 528,720 528,510" stroke="#ff6b81" stroke-width="2.5" stroke-dasharray="7 6" fill="none" marker-end="url(#arrowC)"/> | |
| <rect x="880" y="690" width="300" height="30" rx="8" fill="#0a0e1a" stroke="#ff6b81" stroke-opacity="0.5"/> | |
| <text x="1030" y="710" class="lbl" text-anchor="middle" fill="#ff6b81" font-size="13">write-back: new frames → hypothesis points</text> | |
| <text x="440" y="540" class="tiny" fill="#ff6b81">unknown → known</text> | |
| </svg> | |
| <!-- =================== /PIPELINE SVG =================== --> | |
| </div> | |
| <div class="pageno">4</div> | |
| </section> | |
| <!-- 5. PHASE 1 --> | |
| <section> | |
| <span class="phase-tag p1">Phase 1 · offline caching</span> | |
| <h2>Video → persistent 3D world memory</h2> | |
| <div class="cols" style="margin-top:18px;"> | |
| <div class="col"> | |
| <ul> | |
| <li><strong>Out-and-back</strong> trajectory (~41 out / ~40 back) <em>forces a revisit</em> — the return path is real GT to score consistency.</li> | |
| <li><strong>Metric depth + one global scale</strong> matched to the camera baseline. Depth is only a <em>3D address</em>, never copied as pixels.</li> | |
| <li>Back-project ~11 frames into <strong>PointMemory</strong>: RGB, count, view directions, <em>color variance</em> (ghosting detector).</li> | |
| <li>Reproject to each target cam (z-buffer, adaptive splat) → coverage / count / diversity / disagreement maps.</li> | |
| </ul> | |
| </div> | |
| <div class="col"> | |
| <div class="card"> | |
| <p class="num">SAVED PER CLIP → one .pt file</p> | |
| <p><span class="chip">x0 (GT video, VAE)</span><span class="chip">z_mem</span><span class="chip">knownness k</span><span class="chip">Xtok</span><span class="chip">3D slot bank</span><span class="chip">camera poses</span></p> | |
| </div> | |
| <div class="card" style="margin-top:14px;"> | |
| <p class="num">WHY VARIANCE MATTERS</p> | |
| <p>Consistent color across views ⇒ reliable memory. High disagreement ⇒ occlusion / dynamics / geometry error / <em>ghosting</em> — and lowers k.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="pageno">5</div> | |
| </section> | |
| <!-- 6. KNOWNNESS FIELD --> | |
| <section> | |
| <span class="phase-tag p1">Phase 1 · the knownness field</span> | |
| <h2>What makes a region <span style="color:var(--teal)">known</span>?</h2> | |
| <div class="eq center" style="margin:22px auto; font-size:.95em;"> | |
| k = <span style="color:var(--teal)">cover</span> · exp(−β·<span style="color:var(--coral)">disagree</span>) · clip( tanh(<span style="color:var(--purple)">count</span>/n₀) + 0.3·<span style="color:var(--amber)">diversity</span> ) | |
| </div> | |
| <div class="cols"> | |
| <div class="col card" style="border-top:3px solid var(--teal)"><p class="num" style="color:var(--teal)">COVER</p><p>Is the region reprojected from memory at all?</p></div> | |
| <div class="col card" style="border-top:3px solid var(--purple)"><p class="num" style="color:var(--purple)">COUNT</p><p>Seen many times?</p></div> | |
| <div class="col card" style="border-top:3px solid var(--amber)"><p class="num" style="color:var(--amber)">DIVERSITY</p><p>Seen from many viewpoints?</p></div> | |
| <div class="col card" style="border-top:3px solid var(--coral)"><p class="num" style="color:var(--coral)">DISAGREE</p><p>Do observations conflict? Penalized.</p></div> | |
| </div> | |
| <p class="muted center" style="margin-top:20px; font-size:.72em;">High k only when a region is <strong>covered</strong>, <strong>seen often</strong>, from <strong>many views</strong>, and observations <strong>agree</strong>.</p> | |
| <div class="pageno">6</div> | |
| </section> | |
| <!-- 7. PHASE 2 MODEL --> | |
| <section> | |
| <span class="phase-tag p2">Phase 2 · model</span> | |
| <h2>Frozen Wan DiT + four small hooks</h2> | |
| <div class="cols" style="margin-top:14px;"> | |
| <div class="col"> | |
| <div class="card" style="border-left:3px solid var(--coral)"><p class="num" style="color:var(--coral)">① CORE</p><h3>Knownness → per-token timestep</h3><p>Wan natively supports a different diffusion timestep per token. Known = low noise (repair); unknown = high noise (generate). No surgery on the backbone.</p></div> | |
| <div class="card" style="margin-top:12px;border-left:3px solid var(--purple)"><p class="num" style="color:var(--purple)">② GEOMETRY</p><h3>3D positional encoding (Xtok)</h3><p>Each token knows its world position and viewing direction → the same world point stays the same identity as the camera moves.</p></div> | |
| </div> | |
| <div class="col"> | |
| <div class="card" style="border-left:3px solid var(--teal)"><p class="num" style="color:var(--teal)">③ RETRIEVAL</p><h3>Geometry-keyed cross-attention</h3><p>Every 5 blocks, Xtok queries the slot bank with <strong>Rope3D</strong> relative embedding — retrieve the same content from any viewing angle.</p></div> | |
| <div class="card" style="margin-top:12px;border-left:3px solid var(--amber)"><p class="num" style="color:var(--amber)">④ GATING</p><h3>Confidence-gated shortcut</h3><p>Residual that passes more memory where k is high, damped where k is low — bad memory can't leak in.</p></div> | |
| </div> | |
| </div> | |
| <p class="center" style="margin-top:16px;"><span class="chip" style="border-color:var(--teal);color:var(--teal)">All new modules zero-initialized → day-0 behavior ≈ original Wan</span></p> | |
| <div class="pageno">7</div> | |
| </section> | |
| <!-- 8. PHASE 3 TRAINING --> | |
| <section> | |
| <span class="phase-tag p3">Phase 3 · training</span> | |
| <h2>Anchor-mixture Diffusion Forcing</h2> | |
| <div class="cols" style="margin-top:16px;"> | |
| <div class="col"> | |
| <p>Per token: sample noise level <em>t</em>, pick a clean <strong>anchor</strong>, build the input on the model's familiar noise-to-image manifold:</p> | |
| <div class="eq" style="margin:14px 0;">x_t = (1 − t)·anchor + t·ε</div> | |
| <ul> | |
| <li>Usually <strong>anchor = x₀</strong> (real video latent).</li> | |
| <li>For memory-covered tokens, ~<strong>40%</strong> use <strong>anchor = z_mem</strong> — but only when <em>t ≥ 0.3</em>.</li> | |
| <li>Model always receives the <em>true t</em>.</li> | |
| </ul> | |
| </div> | |
| <div class="col"> | |
| <div class="card" style="border-left:3px solid var(--amber)"> | |
| <p class="num" style="color:var(--amber)">THE TARGET IS ALWAYS x₀</p> | |
| <p>Even when the anchor is z_mem, the loss points at the <strong>real</strong> frame. So the model learns to <em>repair</em> blurry / holey / ghosted memory into truth — <strong>not to copy it</strong>.</p> | |
| </div> | |
| <div class="card" style="margin-top:14px;border-left:3px solid var(--coral)"> | |
| <p class="num" style="color:var(--coral)">UNKNOWN REGIONS UP-WEIGHTED</p> | |
| <p>Loss weighting stops the model from becoming a mere <em>splat cleaner</em> — it must truly synthesize content absent from memory.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="pageno">8</div> | |
| </section> | |
| <!-- 9. PHASE 4 INFERENCE --> | |
| <section> | |
| <span class="phase-tag p4">Phase 4 · inference & rollout</span> | |
| <h2>Knownness → per-token start noise → write-back</h2> | |
| <div class="cols" style="margin-top:14px;"> | |
| <div class="col"> | |
| <div class="eq" style="margin-bottom:10px;">τ = clip( 1 − λk, 0.45, 1 )</div> | |
| <ul> | |
| <li><strong>Known:</strong> x_τ = (1−τ)·z_mem + τ·ε — start from memory + a little noise, <em>short</em> denoise path.</li> | |
| <li><strong>Unknown:</strong> x_τ ≈ ε — start from ~pure noise, <em>full</em> generation.</li> | |
| <li>Each token integrates from its own τ → 0 in a <strong>single diffusion pass</strong>.</li> | |
| <li>Decode latents → frames with the VAE decoder.</li> | |
| </ul> | |
| </div> | |
| <div class="col"> | |
| <div class="card" style="border-left:3px solid var(--coral)"> | |
| <p class="num" style="color:var(--coral)">WRITE-BACK · CHUNKED ROLLOUT</p> | |
| <p>Back-project new frames into PointMemory as <strong>hypothesis points</strong>. Long videos run chunk-by-chunk; each chunk grows the memory.</p> | |
| <p style="margin-top:8px;">When the camera <strong>returns</strong>, k is now high → it starts from the stored content and <em>recovers the same scene</em> instead of re-hallucinating.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="pageno">9</div> | |
| </section> | |
| <!-- 10. NUMBERS --> | |
| <section> | |
| <p class="kicker">Concretely, on DL3DV</p> | |
| <h2 style="margin-bottom:22px;">Every space-time token gets memory, a 3D address, and a k</h2> | |
| <div class="gridnums"> | |
| <div class="stat"><div class="v">832×480</div><div class="l">VIDEO RESOLUTION</div></div> | |
| <div class="stat"><div class="v">81</div><div class="l">FRAMES</div></div> | |
| <div class="stat"><div class="v">[16,21,60,104]</div><div class="l">VAE LATENT</div></div> | |
| <div class="stat"><div class="v">30×52</div><div class="l">TOKEN GRID</div></div> | |
| <div class="stat"><div class="v">32,760</div><div class="l">TOKENS = 21×30×52</div></div> | |
| </div> | |
| <p class="muted center" style="margin-top:24px; font-size:.75em;">Each of the <strong>32,760</strong> tokens carries its own memory content, 3D position, and knownness.</p> | |
| <div class="pageno">10</div> | |
| </section> | |
| <!-- 11. TAKEAWAY --> | |
| <section data-background-color="#080b14"> | |
| <p class="kicker">Takeaway</p> | |
| <h2 style="margin-bottom:20px;">Memory decides <span style="color:var(--teal)">how much to generate</span></h2> | |
| <div class="cols"> | |
| <div class="col card" style="border-top:3px solid var(--teal)"><h3 style="color:var(--teal)">Phase 1</h3><p>Posed video → world memory content, per-region <strong>knownness</strong>, per-token <strong>3D address</strong>.</p></div> | |
| <div class="col card" style="border-top:3px solid var(--amber)"><h3 style="color:var(--amber)">Phase 3</h3><p>Teach the model to <strong>repair</strong> memory, never blindly copy it. Target is always the real frame.</p></div> | |
| <div class="col card" style="border-top:3px solid var(--coral)"><h3 style="color:var(--coral)">Phase 4</h3><p>Knownness → per-token noise. Reuse+repair known regions, generate unknown, <strong>write back</strong>.</p></div> | |
| </div> | |
| <div class="eq center" style="margin-top:26px; font-size:.9em;">Revisit the same place ⇒ recover it from the <span style="color:var(--teal)">same world memory</span> — not a fresh hallucination.</div> | |
| <p class="muted center" style="margin-top:22px; font-size:.7em;">World Memory · working title · CoRL 2026</p> | |
| <div class="pageno">11</div> | |
| </section> | |
| </div></div> | |
| <script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/reveal.js"></script> | |
| <script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/plugin/highlight/highlight.js"></script> | |
| <script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/plugin/notes/notes.js"></script> | |
| <script> | |
| Reveal.initialize({ | |
| width: 1280, height: 720, margin: 0.045, | |
| minScale: 0.2, maxScale: 2.0, | |
| hash: true, slideNumber: false, transition: 'slide', transitionSpeed: 'default', | |
| controlsTutorial: true, center: false, | |
| plugins: [ RevealHighlight, RevealNotes ] | |
| }); | |
| </script> | |
| </body> | |
| </html> | |