Add DeMemWM multiview dynamic policy

.exp_artifact/dememwm_dynamic_multiview_memory_selection_plan.md

@@ -0,0 +1,1091 @@
+# DeMemWM Dynamic Multiview Memory Selection Plan
+
+Repo: `/share_1/users/bonan_ding/WorldMem`
+
+Branch: `dememwm`
+
+Purpose: replace the current DeMemWM `dynamic` memory stream with target-relevant
+multi-view memory selection while keeping the stream name `dynamic`, the packed
+frame order, and the existing DeMemWM model architecture unchanged.
+
+This is the current-branch task that should land before the unified-memory-
+attention branch. It is a dataset/online-selection update only: `dynamic` still
+appears in the code, configs, masks, segments, logs, and launchers, but
+semantically it means multi-view memory selected from target-relevant candidate
+frames.
+
+The design intentionally stays simple and measurable:
+
+- Add exactly one new dynamic policy name: `dynamic.selection_policy=multiview`.
+- Do not add policy aliases such as `fov_multiview` or `fov_fps`.
+- Do not add a hand-tuned score-weight family.
+- Do not add any extra future-query-window config.
+- Implement exact deterministic FOV/coverage selection for training and
+  validation/test.
+- Also keep a pose/Plucker/FPS selector as a benchmarkable fallback under the
+  same `multiview` policy, so speed can be measured before choosing the final
+  training selector for long runs.
+- Use local benchmarking to measure selector speed; do not assume FOV training
+  is fast enough without measurement.
+
+## Required Orchestration Flow
+
+Use this plan with a new session prompt such as:
+
+```text
+/goal Complete /share_1/users/bonan_ding/WorldMem/.exp_artifact/dememwm_dynamic_multiview_memory_selection_plan.md one substep at a time. For every substep, use a fresh implementation subagent, a fresh review subagent, and if needed fresh fix subagents until review is clean. Validate, inspect diff, commit the substep, and update this plan before moving to the next substep.
+```
+
+The main session is the orchestra leader. It owns sequencing, diff inspection,
+validation, commits, and plan-status updates. Subagents do bounded
+implementation, review, or fix work only for the current substep.
+
+For every substep below:
+
+1. Start a fresh implementation subagent.
+2. The implementation subagent must read:
+   - `AGENTS.md`
+   - `.codex/AGENT.md`
+   - `.codex/memory.md`
+   - `/share_1/users/bonan_ding/.codex/skills/codex-ml-cv-clean-implementation/SKILL.md`
+3. The implementation subagent must use:
+   - `[$codex-ml-cv-clean-implementation](/share_1/users/bonan_ding/.codex/skills/codex-ml-cv-clean-implementation/SKILL.md)`
+4. The implementation subagent must implement only the current substep.
+5. After implementation, the main session inspects the diff before review.
+6. Start a fresh review subagent.
+7. The review subagent must read the same files and use the same skill.
+8. The review subagent must check:
+   - correctness
+   - bugs or regressions
+   - improper or unnecessary changes
+   - code cleanliness
+   - adherence to this plan
+   - adherence to repository conventions
+   - targeted test adequacy
+9. If review finds issues, start a fresh fix subagent.
+10. The fix subagent must read the same files, use the same skill, and fix only
+    the review findings.
+11. Repeat review/fix/re-review until the review subagent reports no remaining
+    issues.
+12. Run the validation commands listed for the substep.
+13. Run `git diff --check`.
+14. Inspect `git diff`.
+15. Commit only the completed substep with a clear commit message.
+16. Mark the substep complete in this plan file and include that plan-status
+    update in the same substep commit if possible.
+17. Do not proceed to the next substep until the current one is reviewed clean,
+    validated, committed, and marked complete.
+
+Global rules:
+
+- Do not skip substeps.
+- Do not combine unrelated substeps into one commit.
+- Keep changes surgical.
+- Preserve unrelated user changes.
+- Do not train long jobs or submit Slurm jobs from this plan.
+- Do not add or edit Berzelius/Slurm launchers in this plan.
+- Do not modify `algorithms/worldmem`; use it only as a reference.
+- Keep the packed frame-memory order unchanged:
+
+  ```text
+  [target][anchor][dynamic][revisit]
+  ```
+
+- Keep the stream name `dynamic`; do not rename it to `multiview` in tensor
+  keys, config keys, segment names, masks, logs, or file names.
+- Add exactly one new dynamic policy value: `multiview`.
+- Add at most one selector-backend knob under the `dynamic` config:
+  `multiview_selector`. This is not a policy alias; it exists only to compare
+  exact FOV selection against a fast pose/Plucker/FPS fallback.
+- Valid `multiview_selector` values are exactly:
+  - `fov_greedy`
+  - `pose_plucker_fps`
+- Default the new local launchers to `dynamic.multiview_selector=fov_greedy`.
+- Do not add policy aliases such as `fov_multiview` or `fov_fps`.
+- Do not add score-weight knobs such as `fps_position_weight`,
+  `score_diversity_weight`, or `relevance_fov_weight`.
+- Do not change the DeMemWM model architecture in this plan.
+- Do not change `FrameMemoryReferenceAttention` in this plan.
+- Do not change local target temporal-attention causality in this plan.
+- Do not change anchor selection semantics: keep the same anchor candidate
+  window and `_select_anchor` scoring/diversity logic.
+- Keep `dynamic.selection_policy=recent` and
+  `dynamic.selection_policy=event_triggered` working for backward
+  compatibility.
+- The new `dynamic.selection_policy=multiview` should select target-relevant
+  co-visible support views, not event anchors.
+- Dynamic multiview should avoid selecting the same raw frames as `revisit`
+  when enough alternatives exist. This intentionally overrides the older
+  duplicate-friendly behavior only for `multiview` dynamic versus revisit; keep
+  `recent` and `event_triggered` duplicate semantics unchanged.
+- Do not compute the exact point-FOV candidate pool twice for the same query.
+  When `revisit` and `dynamic=multiview` both need FOV information, share one
+  candidate pool or one local cache entry.
+- Do not add new future-memory behavior in this plan. If the current branch
+  already contains `_memory_candidate_frames()` and
+  `dataset.memory_selection.causal` from previous work, call that helper and
+  preserve its existing semantics exactly. If the helper is absent, use the
+  original past-only candidate semantics.
+- Do not couple `dataset.memory_selection.causal` to `algorithm.causal`.
+- Do not introduce self-generated-memory training in this plan.
+- Do not add source/origin embeddings in this plan.
+- If validation fails, fix through the same fresh fix-subagent and
+  review-subagent loop, then rerun validation.
+
+Status legend:
+
+- `[ ]` not started
+- `[~]` in progress
+- `[x]` complete and committed
+
+## Current Code Facts
+
+- DeMemWM training samples are built by
+  `datasets/video/minecraft_video_dememwm_latent_dataset.py`, which calls
+  `select_memory_indices()` in `datasets/video/memory_selection.py`.
+- The packed DeMemWM sequence order is currently:
+
+  ```text
+  [target][anchor][dynamic][revisit]
+  ```
+
+- The current stream keys are `anchor`, `dynamic`, and `revisit`.
+- The current dataset config exposes `max_anchor_frames`, `max_dynamic_frames`,
+  and `max_revisit_frames`.
+- `dynamic.selection_policy` currently supports `recent` and `event_triggered`.
+- The latent dataset builds `dynamic_stream` only for `event_triggered`; the new
+  `multiview` policy should not require a prebuilt event stream.
+- `_select_revisit()` currently has different training and validation/test
+  behavior in the original code: training uses pose similarity, while
+  validation/test uses point-union/FOV selection.
+- `_select_by_point_union()` already contains the validation/test FOV/Plucker/
+  coverage retrieval machinery needed for exact multiview selection.
+- Exact point-FOV construction evaluates candidate visibility over sampled 3D
+  points. Selector speed must therefore be measured before choosing the final
+  long-run training backend.
+- `_sample_points_in_sphere()` currently uses random point sampling; this plan
+  makes the FOV path deterministic for reproducible training, validation, and
+  online selection.
+- Online validation/test rollout in `algorithms/dememwm/df_video.py` separately
+  selects online `anchor`, `dynamic`, and `revisit` memory from the generated or
+  observed prefix.
+
+## Selection Design
+
+The strongest selector to implement first is coverage-driven FOV greedy
+selection:
+
+1. Build a causal candidate pool for the current target positions.
+2. Pose-preselect candidates to the existing `pose_preselect_topk` budget before
+   exact FOV visibility is evaluated.
+3. Build deterministic target-visible 3D sample points.
+4. Compute candidate visibility masks over those points.
+5. Select direct `revisit` frames by greedy maximum coverage.
+6. Select `dynamic` multiview frames from the remaining candidates by greedy
+   residual coverage, initialized with the coverage already provided by selected
+   `revisit` frames.
+7. Use deterministic lexicographic tie-breakers only:
+   - larger new target-FOV coverage gain;
+   - larger total FOV overlap;
+   - larger Plucker score when available;
+   - larger pose distance from already selected/reference views;
+   - smaller temporal gap to the first target;
+   - smaller frame id.
+
+The benchmarkable fallback selector is `pose_plucker_fps`:
+
+1. Use the current pose-similarity and optional Plucker relevance logic.
+2. Cap to `pose_preselect_topk` candidates.
+3. Select diverse views by deterministic farthest-point sampling in normalized
+   camera-position plus yaw/pitch pose space.
+4. Use this only as a fallback or speed-comparison backend; the new local
+   launchers should default to `fov_greedy`.
+
+## Non-Goals
+
+- Do not create the unified dense memory attention module in this plan.
+- Do not remove or rename `dynamic`.
+- Do not remove the legacy `event_triggered` or `recent` dynamic policies.
+- Do not submit cluster jobs.
+- Do not write or edit Berzelius/Slurm scripts.
+- Do not change Open-Oasis checkpoints, VAE behavior, DiT dimensions, or model
+  checkpoint loading.
+- Do not introduce generated-memory replay or self-rollout training.
+- Do not change `algorithms/worldmem`.
+- Do not add a new external retrieval backend.
+- Do not change the memory segment order.
+- Do not add any extra future-query-window config.
+
+## Substep 1: Add Minimal Multiview Dynamic Policy Config
+
+Status: `[x]`
+
+Goal:
+
+Add configuration and policy plumbing for `dynamic.selection_policy=multiview`
+while keeping existing `recent` and `event_triggered` behavior working. Keep the
+config minimal and avoid unvalidated score-weight knobs.
+
+Primary files:
+
+- `configurations/dataset/video_minecraft_dememwm_latent.yaml`
+- `datasets/video/memory_selection.py`
+- `tests/test_dememwm_latent_dataset.py`
+
+Required behavior:
+
+- Add or preserve this memory budget:
+
+  ```yaml
+  memory_selection:
+    max_anchor_frames: 2
+    max_dynamic_frames: 4
+    max_revisit_frames: 2
+  ```
+
+- Do not add any extra future-query-window config.
+- Do not switch the dataset default policy to `multiview` in this substep. The
+  selector is not implemented until Substep 4, so the default should remain the
+  current working policy here. The default can be switched after integration in
+  Substep 5.
+- Add comments or tests that document the intended new policy value:
+
+  ```yaml
+  memory_selection:
+    dynamic:
+      selection_policy: multiview
+      multiview_selector: fov_greedy
+  ```
+
+- Keep the existing global retrieval-speed controls instead of adding new
+  dynamic-specific score weights:
+
+  ```yaml
+  memory_selection:
+    pose_preselect_topk: 64
+    candidate_chunk_size: 64
+  ```
+
+- Add a single selector-backend knob:
+
+  ```yaml
+  memory_selection:
+    dynamic:
+      multiview_selector: fov_greedy
+  ```
+
+- Valid backend values are exactly:
+
+  ```python
+  {"fov_greedy", "pose_plucker_fps"}
+  ```
+
+- In `memory_selection.py`, allow exactly these dynamic policy names:
+
+  ```python
+  {"recent", "event_triggered", "multiview"}
+  ```
+
+- Unknown policies should raise a clear error mentioning exactly those valid
+  policies.
+- Unknown `multiview_selector` values should raise a clear error mentioning
+  exactly `fov_greedy` and `pose_plucker_fps`.
+- Do not add policy aliases such as `fov_multiview` or `fov_fps`.
+- Do not implement the actual multiview selector yet in this substep. If
+  top-level selection is called with `multiview` before Substep 4, it should fail
+  with a clear `NotImplementedError` or equivalent explicit error rather than
+  silently returning an unrelated fallback.
+- Do not build `dynamic_stream` for the `multiview` policy.
+
+Implementation constraints:
+
+- Keep changes to config/policy validation only.
+- Keep tests small and deterministic.
+- Do not touch model code.
+
+Required tests:
+
+- `dynamic.selection_policy=multiview` is accepted by policy parsing/validation.
+- `dynamic.selection_policy=recent` still works.
+- `dynamic.selection_policy=event_triggered` still works.
+- `dynamic.selection_policy=fov_multiview` and `dynamic.selection_policy=fov_fps`
+  are rejected.
+- `dynamic.multiview_selector=fov_greedy` is accepted.
+- `dynamic.multiview_selector=pose_plucker_fps` is accepted.
+- Unknown policy and backend values raise clear errors containing valid values.
+
+Validation for this substep:
+
+```bash
+python -m py_compile datasets/video/memory_selection.py tests/test_dememwm_latent_dataset.py
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- The stream is still named `dynamic` everywhere.
+- Only one new policy value, `multiview`, was added.
+- Only one backend selector knob was added.
+- No new hand-tuned weight config family was added.
+- Existing dynamic policies remain valid.
+- The default dataset config was not flipped to an unimplemented selector in this
+  substep.
+- No model architecture code was touched.
+- No training or Slurm job was run.
+
+Suggested commit message:
+
+```text
+Add DeMemWM multiview dynamic policy
+```
+
+## Substep 2: Make FOV Candidate Construction Deterministic And Reusable
+
+Status: `[ ]`
+
+Goal:
+
+Refactor the existing FOV/Plucker candidate logic so training, validation, and
+online `revisit`/`dynamic=multiview` can share one deterministic candidate-pool
+path.
+
+Primary files:
+
+- `datasets/video/memory_selection.py`
+- `tests/test_dememwm_latent_dataset.py`
+
+Required behavior:
+
+- Replace random FOV point sampling with deterministic sampling.
+- Add:
+
+  ```python
+  def _deterministic_points_in_unit_ball(num_points: int, device, dtype) -> torch.Tensor:
+      """Return deterministic points in the unit ball."""
+  ```
+
+- Update `_sample_points_in_sphere(center)` to use deterministic unit-ball
+  points and no `torch.rand`.
+- If the code currently uses a module constant such as `_FOV_NUM_POINTS`, keep
+  the default value unchanged unless a config already exists. Do not add a large
+  new sampling hyperparameter family in this plan.
+- Add a reusable helper:
+
+  ```python
+  def _build_fov_candidate_pool(
+      poses: np.ndarray,
+      candidates: np.ndarray,
+      target_positions: np.ndarray,
+      cfg,
+      *,
+      use_plucker: bool,
+  ) -> dict[str, torch.Tensor]:
+      """Build deterministic FOV/Plucker/coverage data for candidate frames."""
+  ```
+
+- The helper should:
+  - call `_pose_preselect()` so exact FOV runs on at most
+    `pose_preselect_topk` candidates when that config is set;
+  - reuse `_candidate_fov_masks()` and `candidate_chunk_size`;
+  - optionally compute Plucker scores;
+  - return enough candidate pose/coverage data for revisit and dynamic to reuse
+    the same pool for the same target query;
+  - return empty tensors cleanly when no candidates survive.
+- The helper should return at least:
+
+  ```text
+  candidates_t: LongTensor [N]          # frame ids after pose preselect
+  candidate_poses: FloatTensor [N, 5]   # poses for candidates_t
+  inside: BoolTensor [N, P]             # candidate-visible target points
+  fov_values: FloatTensor [N]
+  plucker: FloatTensor [N]
+  gaps: LongTensor [N]
+  positive_fov: BoolTensor [N]
+  ```
+
+- `positive_fov` should mean `inside.any(dim=1)` or equivalently positive
+  visible-point coverage. It is not a new configurable threshold.
+- Refactor `_select_by_point_union()` to use the helper without changing its
+  high-level greedy coverage behavior.
+- Add one small internal pool-selection helper if needed, for example
+  `_select_by_point_union_from_pool(pool, count, ...)`, so multiview selection
+  can reuse the same FOV pool instead of recomputing point visibility.
+- Add one small reuse mechanism for the helper output. Acceptable options are:
+  - an explicit optional `fov_pool` argument passed between selectors; or
+  - a local per-call cache keyed by target positions, candidate ids, and
+    Plucker/config values.
+- The reuse mechanism must be local and simple. Do not add a new global backend
+  or persistent disk cache in this plan.
+
+Implementation constraints:
+
+- Do not implement dynamic multiview selection in this substep.
+- Avoid broad refactors; keep helper local to `memory_selection.py`.
+- Cache deterministic base points in a small module-level cache if needed, keyed
+  by `(num_points, device, dtype)`, to avoid rebuilding the same base point set
+  repeatedly.
+- If caching GPU tensors, keep the cache bounded to this tiny base-point set or
+  per-query local variables. Do not introduce an unbounded candidate-pool cache.
+
+Required tests:
+
+- Repeated calls to the FOV selector with the same inputs return identical
+  selected frames.
+- Empty candidate pools return empty selected arrays, not errors.
+- Existing FOV revisit selection still returns causal frames when the current
+  branch is causal.
+- Unit tests confirm `torch.rand` is no longer used in `_sample_points_in_sphere`.
+- Revisit selection still passes existing deterministic tests.
+
+Validation for this substep:
+
+```bash
+python -m py_compile datasets/video/memory_selection.py tests/test_dememwm_latent_dataset.py
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- `torch.rand` is no longer used for FOV point sampling.
+- `_select_by_point_union()` still has the same greedy coverage behavior.
+- The helper does not introduce device or dtype mismatches.
+- The helper output can be reused so revisit and dynamic multiview do not build
+  identical FOV masks twice for the same query.
+- No model code was touched.
+
+Suggested commit message:
+
+```text
+Make DeMemWM FOV candidates deterministic
+```
+
+## Substep 3: Add Deterministic Multiview Selector Backends
+
+Status: `[ ]`
+
+Goal:
+
+Implement the two selector backends used by `dynamic.selection_policy=multiview`:
+exact FOV greedy selection and pose/Plucker/FPS fallback selection.
+
+Primary files:
+
+- `datasets/video/memory_selection.py`
+- `tests/test_dememwm_latent_dataset.py`
+
+Required behavior:
+
+- Add or factor a helper for existing pose-similarity + optional Plucker
+  relevance, for example:
+
+  ```python
+  def _rank_pose_plucker_candidates(
+      poses: np.ndarray,
+      candidates: np.ndarray,
+      target_positions: np.ndarray,
+      cfg,
+  ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+      """Return candidate ids, candidate poses, and relevance scores."""
+  ```
+
+- The helper should preserve the existing scoring semantics used by
+  `_select_by_pose_similarity()`:
+  - `pose_similarity_threshold`
+  - `training_use_plucker`
+  - `training_plucker_weight`
+  - deterministic ranking by score and frame id
+- `_select_by_pose_similarity()` should call this helper so the old revisit
+  behavior remains unchanged when that path is configured.
+- Add a deterministic pose-FPS helper:
+
+  ```python
+  def _select_pose_fps(
+      candidate_ids: torch.Tensor,
+      candidate_poses: torch.Tensor,
+      count: int,
+      *,
+      seed_ids: np.ndarray | None = None,
+      all_poses: np.ndarray | torch.Tensor | None = None,
+  ) -> np.ndarray:
+      """Select diverse camera poses from a relevance-filtered candidate set."""
+  ```
+
+- Pose-FPS distance should use existing pose scales only and no tunable weights:
+  - translation distance normalized by `_POSE_DISTANCE_SCALE`;
+  - yaw/pitch angular distance normalized by `_ANGLE_DISTANCE_SCALE`;
+  - final distance is the unweighted sum of normalized translation and normalized
+    angular distance.
+- Before FPS, cap the relevance-filtered candidate set to the existing
+  `pose_preselect_topk` budget when configured, using deterministic relevance
+  ranking.
+- If `seed_ids` are provided, the first dynamic frame should be far from the seed
+  views when possible. Here the seed views will usually be selected `revisit`
+  frames.
+- Add an FOV-greedy dynamic selector helper that consumes the FOV pool from
+  Substep 2:
+
+  ```python
+  def _select_dynamic_multiview_from_fov_pool(
+      pool: dict[str, torch.Tensor],
+      count: int,
+      *,
+      excluded: np.ndarray | None = None,
+      reference_frames: np.ndarray | None = None,
+      all_poses: np.ndarray | torch.Tensor | None = None,
+  ) -> np.ndarray:
+      """Select dynamic multiview frames by residual FOV coverage."""
+  ```
+
+- FOV-greedy selection should:
+  1. Remove `excluded` frames.
+  2. Initialize covered target points from `reference_frames` when those frames
+     are available in the same pool.
+  3. Consider only candidates with `positive_fov=True` when available.
+  4. Repeatedly choose the next frame by the deterministic priority:
+     - larger new target-FOV coverage gain;
+     - larger total FOV overlap;
+     - larger Plucker score;
+     - larger pose distance from selected/reference views;
+     - smaller temporal gap to the first target;
+     - smaller frame id.
+  5. If all remaining positive-FOV candidates have zero new coverage, continue
+     selecting by the same priority with zero coverage gain, so dynamic can still
+     add co-visible alternative views.
+  6. If no positive-FOV candidate exists, fall back to `_select_pose_fps()` over
+     the pose-preselected candidates.
+- Do not use arbitrary weighted sums.
+- Tie-breaking must be deterministic.
+
+Implementation constraints:
+
+- Do not change anchor selection.
+- Do not change the legacy event-triggered or recent dynamic paths.
+- Do not add new score-weight config fields.
+- Keep all helper functions local to `memory_selection.py`.
+
+Required tests:
+
+- The factored `_select_by_pose_similarity()` returns the same results as before
+  on deterministic cases.
+- Pose-FPS returns at most `count` frames.
+- Pose-FPS is deterministic.
+- Pose-FPS caps the candidate set with `pose_preselect_topk` when configured.
+- Pose-FPS prefers diverse candidate poses when relevance filtering leaves
+  multiple valid candidates.
+- FOV-greedy dynamic selection avoids excluded revisit frames when alternatives
+  exist.
+- FOV-greedy dynamic selection is deterministic.
+- FOV-greedy dynamic selection prefers larger residual coverage before tie-breaks.
+
+Validation for this substep:
+
+```bash
+python -m py_compile datasets/video/memory_selection.py tests/test_dememwm_latent_dataset.py
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- FOV-greedy uses residual coverage, not an arbitrary weighted sum.
+- Pose-FPS is available only as a selector backend/fallback, not as a new dynamic
+  policy name.
+- Existing pose-similarity revisit behavior is preserved.
+- No new arbitrary score-weight config fields were introduced.
+- No model code was touched.
+
+Suggested commit message:
+
+```text
+Add DeMemWM multiview selector backends
+```
+
+## Substep 4: Implement Dynamic-As-Multiview Selection
+
+Status: `[ ]`
+
+Goal:
+
+Implement `dynamic.selection_policy=multiview`. The stream name remains
+`dynamic`, but it now selects target-relevant multi-view support frames.
+
+Primary files:
+
+- `datasets/video/memory_selection.py`
+- `tests/test_dememwm_latent_dataset.py`
+
+Required behavior:
+
+- Add:
+
+  ```python
+  def _select_dynamic_multiview(
+      poses: np.ndarray,
+      target_positions: np.ndarray,
+      cfg,
+      count: int,
+      *,
+      excluded: np.ndarray | None = None,
+      reference_frames: np.ndarray | None = None,
+      split: str = "training",
+      min_candidate_frame: int = 0,
+      fov_pool: dict[str, torch.Tensor] | None = None,
+  ) -> np.ndarray:
+      """Select dynamic-as-multiview memory frames."""
+  ```
+
+- Candidate construction:
+  - use the existing `_memory_candidate_frames()` helper if it is present;
+  - do not modify that helper or its causal/non-causal semantics in this plan;
+  - if the helper is absent, use the original past-only candidates before the
+    first target frame;
+  - remove `excluded` frames from the candidate pool.
+- For top-level use, pass selected `revisit` frames as `excluded` and
+  `reference_frames` so dynamic complements direct revisit memory.
+- If `dynamic.multiview_selector == "fov_greedy"`:
+  1. Use the provided `fov_pool` if available; otherwise call
+     `_build_fov_candidate_pool(..., use_plucker=True)` once.
+  2. Call `_select_dynamic_multiview_from_fov_pool()`.
+  3. Use this path for both training and validation/test when configured.
+- If `dynamic.multiview_selector == "pose_plucker_fps"`:
+  1. Use `_rank_pose_plucker_candidates()`.
+  2. Cap to `pose_preselect_topk` candidates after relevance ranking.
+  3. Run `_select_pose_fps()` seeded by `reference_frames` when available.
+  4. This backend exists for speed comparison and fallback, not as the default
+     local-launcher backend.
+- Return sorted frame indices.
+
+Implementation constraints:
+
+- Keep the helper local to `memory_selection.py`.
+- Do not remove `event_triggered` or `recent` code paths.
+- Keep tie-breaking deterministic.
+- Keep tensor operations small enough for unit tests.
+- Do not add `fov_multiview` or `fov_fps` aliases.
+- Do not add any extra future-query-window config.
+
+Required tests:
+
+- `dynamic.selection_policy=multiview` returns past-only frames when the current
+  branch uses the original past-only candidate semantics.
+- If the current branch already contains `_memory_candidate_frames()` with a
+  configurable non-causal training mode, multiview dynamic follows that helper's
+  existing behavior without reimplementing future-memory logic.
+- Dynamic multiview frames are not selected from already selected revisit frames
+  when enough alternatives exist, because top-level use passes
+  `excluded=revisit`.
+- `fov_greedy` uses the FOV helper during training and validation/test.
+- `pose_plucker_fps` uses the pose/Plucker/FPS path and does not invoke exact
+  FOV candidate masks.
+- Repeated calls with identical inputs return identical selections.
+- `recent` and `event_triggered` dynamic policies still work.
+
+Validation for this substep:
+
+```bash
+python -m py_compile datasets/video/memory_selection.py tests/test_dememwm_latent_dataset.py
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- The new selector is target-relevant and multi-view/diversity-aware.
+- `fov_greedy` works for training as well as validation/test.
+- `pose_plucker_fps` is available for speed benchmarking/fallback.
+- The stream name remains `dynamic`.
+- Existing dynamic policies remain functional.
+- No model code was touched.
+
+Suggested commit message:
+
+```text
+Select DeMemWM dynamic memory by multiview retrieval
+```
+
+## Substep 5: Integrate Multiview Dynamic In Dataset And Online Rollout
+
+Status: `[ ]`
+
+Goal:
+
+Use the new dynamic-as-multiview selector in top-level dataset memory selection
+and online validation/test rollout.
+
+Primary files:
+
+- `datasets/video/memory_selection.py`
+- `algorithms/dememwm/df_video.py`
+- `datasets/video/minecraft_video_dememwm_latent_dataset.py`
+- `tests/test_dememwm_latent_dataset.py`
+
+Required behavior:
+
+- In `select_memory_indices()`:
+  - select `anchor` with existing anchor behavior;
+  - if `dynamic.selection_policy=multiview`, build one deterministic candidate
+    pool for the target query when `dynamic.multiview_selector=fov_greedy`;
+  - select `revisit` using the existing configured behavior, but allow it to
+    reuse the shared FOV pool when the FOV path is used;
+  - select `dynamic` by calling `_select_dynamic_multiview()` with
+    `excluded=revisit`, `reference_frames=revisit`, and the shared `fov_pool`
+    when available;
+  - if `dynamic.selection_policy=event_triggered`, keep the existing event path;
+  - if `dynamic.selection_policy=recent`, keep the existing recent path.
+- After the selector and online path are implemented, switch the dataset config
+  default to:
+
+  ```yaml
+  memory_selection:
+    dynamic:
+      selection_policy: multiview
+      multiview_selector: fov_greedy
+  ```
+
+- In the latent dataset, keep building `dynamic_stream` only when policy is
+  `event_triggered`.
+- In online validation/test in `df_video.py`:
+  - add support for `dynamic.selection_policy=multiview`;
+  - use the existing current target positions from the rollout code; do not add
+    or emulate a retrieval horizon;
+  - build or retrieve the deterministic FOV candidate pool once for the online
+    query when using `fov_greedy` and share it between online `revisit` and
+    `dynamic` multiview;
+  - do not build or extend the online event cache for the multiview policy;
+  - keep the event-cache path unchanged for `event_triggered`.
+
+Implementation constraints:
+
+- Do not change the packed memory order.
+- Do not change online causal history limits.
+- Do not touch model code.
+- Keep current masks/padding behavior.
+- Do not add any extra future-query-window config.
+
+Required tests:
+
+- Dataset selection returns keys `anchor`, `dynamic`, and `revisit` with expected
+  maximum lengths.
+- Online selector accepts `dynamic.selection_policy=multiview` without requiring
+  an event cache.
+- Online selection uses the current rollout target positions and no retrieval
+  horizon.
+- Online revisit and dynamic multiview share the same FOV candidate pool or
+  cache entry for a query when `fov_greedy` is configured.
+- Existing online `event_triggered` behavior remains available.
+
+Validation for this substep:
+
+```bash
+python -m py_compile datasets/video/memory_selection.py algorithms/dememwm/df_video.py datasets/video/minecraft_video_dememwm_latent_dataset.py tests/test_dememwm_latent_dataset.py
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- Training and online validation both support multiview dynamic.
+- The dataset default policy is switched to `multiview` only after the selector
+  and online integration exist.
+- Online validation/test avoids avoidable duplicate exact-FOV candidate passes
+  when selecting revisit and dynamic for the same target query.
+- `dynamic_stream` is not required for the multiview policy.
+- The online event cache remains limited to `event_triggered`.
+- Exact FOV construction is not duplicated for revisit and dynamic multiview in
+  the same online query.
+- The frame-memory segment order is unchanged.
+
+Suggested commit message:
+
+```text
+Use multiview dynamic memory online
+```
+
+## Substep 6: Add Selector Speed Benchmark
+
+Status: `[ ]`
+
+Goal:
+
+Add a local benchmark so selector speed can be measured before deciding whether
+`fov_greedy` is acceptable for full training or whether the fallback
+`pose_plucker_fps` should be used for long runs.
+
+Primary files:
+
+- `scripts/benchmark_dememwm_multiview_selection.py`
+- `datasets/video/memory_selection.py` only if a small public/importable wrapper
+  is needed
+- `tests/test_dememwm_latent_dataset.py` or a small benchmark smoke test
+- this plan file
+
+Required behavior:
+
+- Add a benchmark script that can run without submitting jobs:
+
+  ```bash
+  python scripts/benchmark_dememwm_multiview_selection.py \
+    --num-frames 1200 \
+    --target-start 600 \
+    --target-len 8 \
+    --num-iters 100 \
+    --pose-preselect-topk 64 \
+    --candidate-chunk-size 64
+  ```
+
+- The script should benchmark both selector backends on the same deterministic
+  pose sequence:
+  - `fov_greedy`
+  - `pose_plucker_fps`
+- The script should report at least:
+  - mean milliseconds per selection;
+  - median milliseconds per selection;
+  - p90 milliseconds per selection;
+  - selected frame count;
+  - candidate count after pose preselection;
+  - whether FOV pool reuse is enabled;
+  - CPU/GPU device used if applicable.
+- The script may support optional real-dataset inputs if this can be done
+  surgically, but it must have a deterministic synthetic-pose mode that works in
+  CI/local code validation without data paths.
+- The script should not choose or edit the final default automatically. It only
+  reports measurements.
+- Add a short Markdown report template or script option such as
+  `--write-report .exp_artifact/dememwm_multiview_selection_speed_report.md`.
+  If the benchmark is run only on synthetic data, the report must explicitly say
+  the result is not a substitute for a real dataset sampling benchmark.
+
+Implementation constraints:
+
+- Do not train.
+- Do not require GPUs for the synthetic benchmark.
+- Do not import heavy training modules when direct selector imports are enough.
+- Do not add external dependencies.
+- Do not use the benchmark result to change the committed default automatically.
+
+Required tests:
+
+- The benchmark script runs in synthetic mode with a tiny iteration count.
+- The script reports both selector names.
+- The script exits nonzero on unknown selector names.
+- The script does not require dataset paths in synthetic mode.
+
+Validation for this substep:
+
+```bash
+python -m py_compile scripts/benchmark_dememwm_multiview_selection.py datasets/video/memory_selection.py tests/test_dememwm_latent_dataset.py
+python scripts/benchmark_dememwm_multiview_selection.py --num-frames 128 --target-start 64 --target-len 4 --num-iters 3 --pose-preselect-topk 16 --candidate-chunk-size 16
+python -B -m unittest tests.test_dememwm_latent_dataset
+git diff --check
+```
+
+Review must specifically confirm:
+
+- The benchmark compares the exact FOV selector and the pose/Plucker/FPS fallback
+  on the same inputs.
+- The benchmark reports timing rather than making an ungrounded default decision.
+- The benchmark can run without Slurm, training, or dataset paths.
+- No external dependencies were added.
+
+Suggested commit message:
+
+```text
+Add DeMemWM multiview selector benchmark
+```
+
+## Substep 7: Add Local Launchers And Final Validation
+
+Status: `[ ]`
+
+Goal:
+
+Add local launchers for the current-branch dynamic-multiview experiment and run
+cheap code validation. Do not add Berzelius or Slurm scripts in this plan.
+
+Primary files:
+
+- `train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_2h200_debug.sh`
+- `train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_full.sh`
+- `configurations/dataset/video_minecraft_dememwm_latent.yaml`
+- `configurations/algorithm/dememwm_base.yaml` only if a config key is missing
+- tests touched by previous substeps
+- this plan file
+
+Required launcher behavior:
+
+- The debug launcher should be local-only and use two GPUs by default:
+
+  ```bash
+  CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0,1}
+  NPROC_PER_NODE=${NPROC_PER_NODE:-2}
+  torchrun --standalone --nproc_per_node=${NPROC_PER_NODE} -m main ...
+  ```
+
+- The full launcher should also be local-only and parameterized by environment
+  variables. Do not assume eight local GPUs. Default to the same local 2-H200
+  setup unless the caller overrides it:
+
+  ```bash
+  CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0,1}
+  NPROC_PER_NODE=${NPROC_PER_NODE:-2}
+  torchrun --standalone --nproc_per_node=${NPROC_PER_NODE} -m main ...
+  ```
+
+- Neither launcher may contain:
+  - `#SBATCH`
+  - `srun`
+  - `module load`
+  - `berzelius`
+  - cluster account names
+  - hard-coded cluster paths
+- Both launchers should set:
+
+  ```bash
+  dataset.memory_selection.max_anchor_frames=2
+  dataset.memory_selection.max_dynamic_frames=4
+  dataset.memory_selection.max_revisit_frames=2
+  dataset.memory_selection.causal=true
+  dataset.memory_selection.dynamic.selection_policy=multiview
+  dataset.memory_selection.dynamic.multiview_selector=fov_greedy
+  algorithm.noise_route.anchor=all
+  algorithm.noise_route.dynamic=all
+  algorithm.noise_route.revisit=all
+  algorithm.memory_noise.enabled=false
+  algorithm.memory_noise.anchor_max_fraction=0.0
+  algorithm.memory_noise.dynamic_max_fraction=0.0
+  algorithm.memory_noise.revisit_max_fraction=0.0
+  +algorithm.memory_noise.validation_noisy_memory=false
+  ```
+
+- The debug launcher should use tiny/debug settings, for example:
+
+  ```bash
+  experiment.training.max_steps=50
+  experiment.validation.val_every_n_step=25
+  experiment.validation.limit_batch=1
+  experiment.training.checkpointing.every_n_train_steps=25
+  wandb.mode=offline
+  ```
+
+- The full local launcher should keep the intended full training schedule, but
+  use local environment variables for paths such as `DATA_ROOT`,
+  `FEATURE_ROOT`, `OUTPUT_ROOT`, `DIFFUSION_MODEL_PATH`, and `VAE_PATH`.
+- Do not run the full training job automatically as part of Codex validation.
+- The local 2H200 debug command may be run manually after code review if the
+  local machine has the GPUs and paths configured; keep it separate from the
+  cheap code-validation commands below.
+
+Cheap final validation:
+
+```bash
+python -m py_compile \
+  datasets/video/memory_selection.py \
+  datasets/video/minecraft_video_dememwm_latent_dataset.py \
+  algorithms/dememwm/df_video.py \
+  scripts/benchmark_dememwm_multiview_selection.py \
+  tests/test_dememwm_latent_dataset.py
+
+python -B -m unittest tests.test_dememwm_latent_dataset
+
+python scripts/benchmark_dememwm_multiview_selection.py \
+  --num-frames 128 \
+  --target-start 64 \
+  --target-len 4 \
+  --num-iters 3 \
+  --pose-preselect-topk 16 \
+  --candidate-chunk-size 16
+
+grep -R "#SBATCH\|srun\|berzelius\|module load" \
+  train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_2h200_debug.sh \
+  train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_full.sh && exit 1 || true
+
+
+git diff --check
+git status --short
+```
+
+Manual local run commands, not automatic Codex validation:
+
+```bash
+python scripts/benchmark_dememwm_multiview_selection.py \
+  --num-frames 1200 \
+  --target-start 600 \
+  --target-len 8 \
+  --num-iters 100 \
+  --pose-preselect-topk 64 \
+  --candidate-chunk-size 64 \
+  --write-report .exp_artifact/dememwm_multiview_selection_speed_report.md
+
+bash train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_2h200_debug.sh
+bash train_dememwm_curric_multiview_anchorclean_causal_cleanmem_local_full.sh
+```
+
+Review must specifically confirm:
+
+- Both launchers are local-only.
+- No Berzelius/Slurm launcher was added or edited.
+- The debug launcher is suitable for a local 2H200 smoke run.
+- The full launcher is local, defaults to 2 GPUs, and is parameterized by
+  environment variables for larger local runs.
+- Both launchers default to `dynamic.multiview_selector=fov_greedy`.
+- The benchmark script exists and can compare `fov_greedy` with
+  `pose_plucker_fps`.
+- No model architecture changes were included in this plan.
+- The final behavior can be enabled with:
+
+  ```bash
+  dataset.memory_selection.dynamic.selection_policy=multiview
+  dataset.memory_selection.dynamic.multiview_selector=fov_greedy
+  ```
+
+Suggested commit message:
+
+```text
+Add local DeMemWM multiview launchers
+```
+
+## New-Session Goal Prompt
+
+Use this prompt to start the implementation session:
+
+```text
+Use $codex-ml-cv-clean-implementation.
+Repo: /share_1/users/bonan_ding/WorldMem
+/goal Complete .exp_artifact/dememwm_dynamic_multiview_memory_selection_plan.md step by step.
+
+Act as the main orchestra leader. For each substep in the plan:
+1. Use a fresh implementation subagent for only that substep.
+2. Inspect the diff yourself.
+3. Use a fresh review subagent to review correctness, bugs, design, cleanliness, and tests.
+4. If review finds any issue, use a fresh fix subagent to fix only those findings.
+5. Repeat review/fix with fresh subagents until review is clean.
+6. Run the substep validation commands and git diff --check.
+7. Inspect git diff.
+8. Commit only that substep with the suggested or similarly clear commit message.
+9. Mark the substep complete in the plan file and include the status update in the same commit if possible.
+10. Move to the next substep only after the current substep is reviewed clean, validated, committed, and marked complete.
+
+Hard constraints:
+- Do not train long jobs or submit Slurm jobs.
+- Do not add or edit Berzelius/Slurm scripts.
+- Do not modify algorithms/worldmem.
+- Keep the packed order [target][anchor][dynamic][revisit].
+- Keep the stream name dynamic; dynamic now semantically means multiview memory for the new policy.
+- Add only dynamic.selection_policy=multiview. Do not add fov_multiview or fov_fps policy aliases.
+- Keep existing recent and event_triggered dynamic policies working.
+- Do not change the DeMemWM model architecture or FrameMemoryReferenceAttention in this plan.
+- Keep anchor selection semantics unchanged.
+- Make FOV candidate sampling deterministic.
+- Implement exact FOV-greedy multiview selection for training and validation/test.
+- Add only one multiview backend knob: dynamic.multiview_selector with values fov_greedy and pose_plucker_fps.
+- Do not add hand-tuned scoring weight configs.
+- Do not build exact FOV masks twice for the same revisit+dynamic query; share a local FOV pool/cache.
+- Add a local selector-speed benchmark script comparing fov_greedy and pose_plucker_fps.
+- Use clean memory in the new local launchers.
+- Add local 2H200 debug and local full launchers; local full should default to 2 GPUs and scale through environment variables. No SBATCH, srun, module load, berzelius, or hard-coded cluster paths.
+- Preserve unrelated user changes.
+
+Final report must include changed files, commits made, validation commands and results, selector benchmark command/results if run, local debug/full run commands, and remaining risks.
+```

configurations/dataset/video_minecraft_dememwm_latent.yaml

@@ -31,8 +31,11 @@ memory_selection:
   local_context_exclusion_frames: 8
   plucker_moment_radius: 30.0
   anchor_diverse_selection: true
+  pose_preselect_topk: 64
+  candidate_chunk_size: 64
   dynamic:
     selection_policy: event_triggered
+    multiview_selector: fov_greedy
     scene_threshold: 2.5
     state_threshold: 2.5
     stable_threshold: 1.0

datasets/video/memory_selection.py

@@ -9,6 +9,8 @@ import torch
 
 
 SEGMENT_KEYS = ("anchor", "dynamic", "revisit")
+_VALID_DYNAMIC_POLICIES = ("recent", "event_triggered", "multiview")
+_VALID_MULTIVIEW_SELECTORS = ("fov_greedy", "pose_plucker_fps")
 _FOV_NUM_POINTS = 10000
 _FOV_RADIUS = 30.0
 _FOV_HALF_H = 105.0 / 2.0
@@ -384,7 +386,20 @@ def _select_dynamic(target_start: int, count: int, min_candidate_frame: int = 0)
 
 def _dynamic_policy(cfg) -> str:
     dynamic_cfg = cfg_get(cfg, "dynamic", {})
-    return str(cfg_get(dynamic_cfg, "selection_policy", "recent"))
+    policy = str(cfg_get(dynamic_cfg, "selection_policy", "recent"))
+    if policy not in _VALID_DYNAMIC_POLICIES:
+        valid = ", ".join(_VALID_DYNAMIC_POLICIES)
+        raise ValueError(f"memory_selection.dynamic.selection_policy must be one of {valid}; got {policy!r}")
+    return policy
+
+
+def _dynamic_multiview_selector(cfg) -> str:
+    dynamic_cfg = cfg_get(cfg, "dynamic", {})
+    selector = str(cfg_get(dynamic_cfg, "multiview_selector", "fov_greedy"))
+    if selector not in _VALID_MULTIVIEW_SELECTORS:
+        valid = ", ".join(_VALID_MULTIVIEW_SELECTORS)
+        raise ValueError(f"memory_selection.dynamic.multiview_selector must be one of {valid}; got {selector!r}")
+    return selector
 
 
 def _latent_frame_vectors(latents, start: int, stop: int):
@@ -780,7 +795,12 @@ def _select_dynamic_by_policy(
             reference_frames=reference_frames,
             excluded=excluded,
         )
-    raise ValueError(f"memory_selection.dynamic.selection_policy must be recent or event_triggered; got {policy!r}")
+    if policy == "multiview":
+        selector = _dynamic_multiview_selector(cfg)
+        raise NotImplementedError(
+            f"memory_selection.dynamic.selection_policy='multiview' with multiview_selector={selector!r} is not implemented yet"
+        )
+    raise AssertionError(f"unhandled dynamic selection policy: {policy!r}")
 
 
 def _select_revisit(
@@ -863,8 +883,11 @@ def select_memory_indices(
 
     target_start = int(target_positions[0])
     policy = _dynamic_policy(cfg)
-    if policy not in {"recent", "event_triggered"}:
-        raise ValueError(f"memory_selection.dynamic.selection_policy must be recent or event_triggered; got {policy!r}")
+    if policy == "multiview":
+        selector = _dynamic_multiview_selector(cfg)
+        raise NotImplementedError(
+            f"memory_selection.dynamic.selection_policy='multiview' with multiview_selector={selector!r} is not implemented yet"
+        )
 
     if policy == "recent":
         dynamic = _select_dynamic_by_policy(

tests/test_dememwm_latent_dataset.py

@@ -9,13 +9,19 @@ import torch
 from torch import nn
 from omegaconf import OmegaConf
 
-from datasets.video.memory_selection import _memory_candidate_frames, select_memory_indices
+from datasets.video.memory_selection import (
+    _dynamic_multiview_selector,
+    _dynamic_policy,
+    _memory_candidate_frames,
+    select_memory_indices,
+)
 from datasets.video.minecraft_video_dememwm_latent_dataset import MinecraftVideoDeMemWMLatentDataset
 
 
 def _dynamic_cfg(selection_policy="recent", **overrides):
     cfg = {
         "selection_policy": selection_policy,
+        "multiview_selector": "fov_greedy",
         "scene_threshold": 2.5,
         "state_threshold": 2.5,
         "stable_threshold": 1.0,
@@ -171,6 +177,10 @@ class MemorySelectionTests(unittest.TestCase):
         cfg = OmegaConf.load(config_path)
 
         self.assertIs(cfg.memory_selection.causal, True)
+        self.assertEqual(cfg.memory_selection.dynamic.selection_policy, "event_triggered")
+        self.assertEqual(cfg.memory_selection.dynamic.multiview_selector, "fov_greedy")
+        self.assertEqual(cfg.memory_selection.pose_preselect_topk, 64)
+        self.assertEqual(cfg.memory_selection.candidate_chunk_size, 64)
 
     def test_revisit_uses_sampled_fov_selection(self):
         poses = np.array(
@@ -597,17 +607,59 @@ class MemorySelectionTests(unittest.TestCase):
                 self.assertEqual(indices["dynamic"].tolist(), [-1, -1])
                 self.assertEqual(masks["dynamic"].tolist(), [False, False])
 
-    def test_hybrid_policy_is_rejected(self):
+    def test_dynamic_policy_validation_accepts_exact_supported_values(self):
+        for policy in ("recent", "event_triggered", "multiview"):
+            with self.subTest(policy=policy):
+                cfg = _selection_cfg(dynamic=_dynamic_cfg(policy))
+
+                self.assertEqual(_dynamic_policy(cfg), policy)
+
+    def test_unknown_dynamic_policies_are_rejected_without_aliases(self):
+        poses = np.zeros((8, 5), dtype=np.float32)
+        for policy in ("fov_multiview", "fov_fps", "hybrid"):
+            with self.subTest(policy=policy):
+                cfg = _selection_cfg(
+                    max_anchor_frames=0,
+                    max_dynamic_frames=2,
+                    max_revisit_frames=0,
+                    dynamic=_dynamic_cfg(policy),
+                )
+
+                with self.assertRaisesRegex(ValueError, "recent, event_triggered, multiview"):
+                    select_memory_indices(poses, np.array([6]), cfg, latents=_event_latents(8, event_frame=99))
+
+    def test_multiview_selector_validation_accepts_exact_backends(self):
+        for selector in ("fov_greedy", "pose_plucker_fps"):
+            with self.subTest(selector=selector):
+                cfg = _selection_cfg(dynamic=_dynamic_cfg("multiview", multiview_selector=selector))
+
+                self.assertEqual(_dynamic_multiview_selector(cfg), selector)
+
+    def test_unknown_multiview_selector_is_rejected(self):
         poses = np.zeros((8, 5), dtype=np.float32)
         cfg = _selection_cfg(
             max_anchor_frames=0,
             max_dynamic_frames=2,
             max_revisit_frames=0,
-            dynamic=_dynamic_cfg("hybrid"),
+            dynamic=_dynamic_cfg("multiview", multiview_selector="fov_fps"),
         )
 
-        with self.assertRaisesRegex(ValueError, "recent or event_triggered"):
-            select_memory_indices(poses, np.array([6]), cfg, latents=_event_latents(8, event_frame=99))
+        with self.assertRaisesRegex(ValueError, "fov_greedy, pose_plucker_fps"):
+            select_memory_indices(poses, np.array([6]), cfg)
+
+    def test_multiview_policy_selection_fails_explicitly_until_implemented(self):
+        poses = np.zeros((8, 5), dtype=np.float32)
+        for selector in ("fov_greedy", "pose_plucker_fps"):
+            with self.subTest(selector=selector):
+                cfg = _selection_cfg(
+                    max_anchor_frames=0,
+                    max_dynamic_frames=2,
+                    max_revisit_frames=0,
+                    dynamic=_dynamic_cfg("multiview", multiview_selector=selector),
+                )
+
+                with self.assertRaisesRegex(NotImplementedError, "multiview"):
+                    select_memory_indices(poses, np.array([6]), cfg)
 
     def test_event_dynamic_uses_nearest_anchors_per_revisit_frame(self):
         import datasets.video.memory_selection as memory_selection
@@ -1191,7 +1243,7 @@ class DeMemWMLatentDatasetTests(unittest.TestCase):
                 self.decoded_inputs.append(xs.detach().clone())
                 return xs + 10.0
 
-            def _update_metric_accumulators(self, xs_pred, xs_gt):
+            def _update_metric_accumulators(self, xs_pred, xs_gt, valid_mask=None, eval_start=0):
                 self.metric_updates.append((xs_pred.detach().clone(), xs_gt.detach().clone()))
 
             def _update_per_frame_metric_accumulators(self, xs_pred, xs_gt, mask, eval_start):
@@ -1246,12 +1298,12 @@ class DeMemWMLatentDatasetTests(unittest.TestCase):
         self.assertEqual([call["kwargs"]["current_frame"] for call in calls], [2, 2, 4, 4])
         self.assertTrue(torch.equal(calls[0]["kwargs"]["frame_idx"], calls[1]["kwargs"]["frame_idx"]))
         self.assertTrue(torch.equal(calls[2]["kwargs"]["frame_idx"], calls[3]["kwargs"]["frame_idx"]))
-        self.assertEqual(calls[0]["x"][:, 0, 0, 0, 0].tolist(), [0.0, 0.0, 0.0, 1.0, 0.0])
+        self.assertEqual(calls[0]["x"][:, 0, 0, 0, 0].tolist(), [0.0, 0.0, 0.0, 1.0])
         self.assertEqual(calls[2]["x"][:, 0, 0, 0, 0].tolist(), [2.0, 0.0, 0.0, 2.0, 2.0])
-        self.assertEqual(calls[0]["action_cond"][:, 0, 0].tolist(), [102.0, 103.0, 0.0, 0.0, 0.0])
-        self.assertEqual(calls[0]["kwargs"]["frame_idx"][:, 0].tolist(), [12, 13, 10, 11, 0])
+        self.assertEqual(calls[0]["action_cond"][:, 0, 0].tolist(), [102.0, 103.0, 0.0, 0.0])
+        self.assertEqual(calls[0]["kwargs"]["frame_idx"][:, 0].tolist(), [12, 13, 10, 11])
         self.assertEqual(calls[2]["kwargs"]["frame_idx"][:, 0].tolist(), [13, 14, 10, 12, 12])
-        self.assertEqual(calls[0]["kwargs"]["frame_memory_masks"]["revisit"].tolist(), [[False]])
+        self.assertEqual(calls[0]["kwargs"]["frame_memory_masks"]["revisit"].tolist(), [[]])
         self.assertEqual(calls[2]["kwargs"]["frame_memory_masks"]["revisit"].tolist(), [[True]])
         self.assertAlmostEqual(float(loss), 1.0 / 3.0)
         self.assertEqual([name for name, _ in harness.logged], ["test/latent_mse"])
@@ -1316,7 +1368,7 @@ class DeMemWMLatentDatasetTests(unittest.TestCase):
             def decode(self, xs):
                 return xs
 
-            def _update_metric_accumulators(self, xs_pred, xs_gt):
+            def _update_metric_accumulators(self, xs_pred, xs_gt, valid_mask=None, eval_start=0):
                 pass
 
             def _update_per_frame_metric_accumulators(self, xs_pred, xs_gt, mask, eval_start):
@@ -1481,6 +1533,38 @@ class DeMemWMLatentDatasetTests(unittest.TestCase):
         self.assertEqual(sample["frame_indices"].tolist(), [106, 107, 108, 100, 103])
         self.assertEqual(sample["memory_masks"]["dynamic"].tolist(), [True, True])
 
+    def test_dataset_does_not_build_dynamic_stream_for_multiview_policy(self):
+        import datasets.video.minecraft_video_dememwm_latent_dataset as dataset_module
+
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            num_frames = 112
+            _write_vae_feature_clip(
+                root,
+                root / "vae_features",
+                stem="sample",
+                num_frames=num_frames,
+                actions=np.zeros((num_frames, 25), dtype=np.float32),
+                latents=_event_latents(num_frames, event_frame=102),
+            )
+            dataset = MinecraftVideoDeMemWMLatentDataset(
+                _dataset_cfg(
+                    root,
+                    context_length=6,
+                    memory_selection=_selection_cfg(
+                        max_anchor_frames=0,
+                        max_dynamic_frames=2,
+                        max_revisit_frames=0,
+                        dynamic=_dynamic_cfg("multiview"),
+                    ),
+                ),
+                split="training",
+            )
+            with mock.patch.object(dataset_module, "_build_dynamic_stream", side_effect=AssertionError("unexpected dynamic stream build")):
+                arrays = dataset._load_feature_arrays(dataset.data_paths[0])
+
+        self.assertEqual(arrays["dynamic_stream"].tolist(), [])
+
     def test_dataset_returns_target_anchor_dynamic_revisit_contract(self):
         with tempfile.TemporaryDirectory() as tmp:
             root = Path(tmp)
@@ -1504,7 +1588,7 @@ class DeMemWMLatentDatasetTests(unittest.TestCase):
         self.assertEqual(sample["frame_indices"][3:5].tolist(), [100, 101])
         self.assertEqual(sample["frame_indices"][5:7].tolist(), [104, 105])
         revisit_frames = sample["frame_indices"][7:9].numpy()
-        self.assertEqual(revisit_frames.tolist(), [104, 105])
+        self.assertEqual(revisit_frames.tolist(), [102, 103])
         self.assertEqual(len(np.unique(revisit_frames)), 2)
         self.assertTrue(sample["memory_masks"]["target"].all().item())
         self.assertTrue(sample["memory_masks"]["anchor"].all().item())