IWM: VideoRAE3D + EEG-conditioned action models

Five selected inference-only checkpoints from Intention World Model. The visual representation is VideoRAE3D. Our learned IDM/FDM teacher and EEG controllers are provided here; third-party visual encoder/decoder weights are not redistributed.

Model ID	Selected step	Purpose
teacher-best87600	87,600	Shared frozen IDM + FDM, selected from training completed at 160,000 steps
cg2-ng-seed26102751	800	Reference CG2-NG controller, seed 26102751
cg2-ng-seed26102752	1,000	Reference CG2-NG controller, seed 26102752
ng-d-seed26102751	2,400	Exploratory independent motion-reader refinement, seed 26102751
ng-d-seed26102752	2,400	Exploratory independent motion-reader refinement, seed 26102752

Controller steps refer to their own response-training stage, not teacher steps. NG-D training completed at total 3,000; these are the selected best 2,400 checkpoints. Both seeds are included, rather than only the better-scoring seed.

Model and input contract

• Visual latent: [B,1024,64], spatial-temporal grid 4 x 16 x 16.
• IDM: width 512, depth 6, 8 heads, continuous [B,1,256] action.
• FDM: width 512, depth 6, 8 heads, direct next-latent prediction. No output residual, transport, LoRA, pixel enhancement or post-hoc action gain.
• EEG: [B,32,500], 200 Hz, 2.5 seconds; four 512-wide layers, 20 ordered time tokens read by a visual-conditioned query. Predicted classes are Drink, Eat, Pen, Digital, Manipulation, in that order.
• CG2-NG combines a frozen visual prior, EEG residual, predicted-class response and learned synthetic-noise fallback gate. NG-D freezes the original CG2-NG and adds an independent target-EEG motion reader and bounded correction.

This release uses target-period EEG conditioning, not past-only forecasting. Observed video is [-2,0), EEG is [-0.5,2), and the output is H1 [0,2): 16 native frames at 8 fps, 256 x 256. No future video or true class is supplied to the EEG controller. Oracle IDM inference does use the true future latent.

The channel order is recorded in every controller's config.json. Inputs must match the original EEG preprocessing, physical scale and ordering. These are preprocessed EEG values before the model's TRAIN-fit normalization; the included mean/std and clipping are applied inside the model. Do not normalize twice or pass arbitrary device units. The inherited cache uses EDF physical microvolts, a causal 0.1–75 Hz fourth-order bandpass and 50 Hz notch at 256 Hz, past-sample selection to 200 Hz, then float32 values divided by 100; see the preprocessing arithmetic. All-zero EEG means raw zero before normalization and intentionally returns exactly the visual prior. This fallback is an explicit interface design, not evidence of learned physiological validity.

Loading

Use the matching source archive at commit 516a71c252ff8f895dcd036bb343c996254c7a76. The complete repository is needed, not only an individual experiment folder. The loader requires PyTorch, NumPy and safetensors. Export verification used PyTorch 2.3.0 and safetensors 0.4.3 on CPU. Downloading through the Hub additionally requires huggingface_hub. This is not a Transformers AutoModel checkpoint.

git clone --branch debug-3dvrae https://github.com/fly0520/intention-world-model.git
cd intention-world-model
git checkout 516a71c252ff8f895dcd036bb343c996254c7a76

from pathlib import Path
import sys
import torch
from huggingface_hub import snapshot_download

release = Path(snapshot_download("SnoopyFan/IWM"))
source = Path("/your/checkout/intention-world-model")
sys.path.insert(0, str(release))
from load_models import load_checkpoint

device = "cuda" if torch.cuda.is_available() else "cpu"
teacher = load_checkpoint(release, "teacher-best87600", source, device)
controller = load_checkpoint(release, "cg2-ng-seed26102751", source, device)
# Or: "ng-d-seed26102751"; both second-seed versions are also available.

# z_current: native VideoRAE3D latent [B,1024,64]
# eeg_raw: correctly ordered/preprocessed target-period EEG [B,32,500]
with torch.inference_mode():
    logits, action = controller(z_current.to(device), eeg_raw.to(device))
    z_h1 = teacher.fdm(z_current.to(device), action)
    z_visual = teacher.fdm(z_current.to(device), controller.visual(z_current.to(device)))

For RGB encoding/decoding obtain the compatible released AutoEncoderVJepa21_3D_C64Repa VideoRAE weights and V-JEPA2.1 encoder separately. See the pinned visual integration and runtime contract. Follow their licenses and preprocessing; visual weights are not part of this repository. The example above starts at the latent/EEG interface and does not download or contain any EEG/video data.

Results and intended use

CG2-NG remains the reference demonstration model. Its two-seed subject-balanced Oracle motion-consistency J is 0.256243, versus 0.232376 for the shared visual/zero baseline on the repeated development set (148 cases, 24 subjects). NG-D best2400 reaches 0.258135 on the same aggregate, essentially unchanged from its shorter-training parent; it is an exploratory alternative, not a confirmed general upgrade. Selected examples show local forearm/wrist responses, not uniformly correct full-action reconstruction. J is not a visible-success rate.

Full comparisons, Gaussian/PSD controls and seed-specific results:

• CG2-NG protocol and results
• Recent variants and continuation results

The teacher was selected on repeated full-150 development evaluation; these EEG controllers were selected on TRAIN-inner. The 148-case EEG development set is used repeatedly for reporting and demonstration selection, not an untouched confirmation set. Subjects/phases are shared. Do not interpret these models as cross-person generalization, causal intention reading, medical diagnostics or proof that synthetic-noise discrimination alone establishes correct motion.

Packaging, provenance and privacy

manifest.json gives the original checkpoint SHA, exported tensor/file SHA, selected step, exact removed-buffer list and export checks for each model. The safetensors retain original FP32 values without quantization. The loader also restores archived dense buffer layouts to preserve the original matrix multiplication path. Required aggregate normalization, learned visual prior and class-expert parameters are included. Raw EEG/video, individual features/actions, participant/sample IDs, relation memories, optimizer/RNG/sampler state, logs and private paths are not. The loader deletes unused training-only buffers before strict loading. This is an inference release, not an optimizer-resume training checkpoint.

CPU synthetic checks verify exact tensor round-trips, identical original/exported class/action outputs, teacher IDM/FDM outputs and exact raw-zero visual fallback. They validate packaging, not new scientific performance or cross-device bitwise equivalence. All original private checkpoints are retained unchanged.

The existing repository Apache-2.0 license metadata is preserved. It does not relicense third-party VideoRAE/V-JEPA code, weights or source datasets.