robocasa/README.md

metadata

language:
  - en
tags:
  - futureworlds
  - world-model
  - robotics
  - video-prediction
  - reinforcement-learning

FutureWorlds — RoboCasa

Local release candidate, not yet published. Model weights are verified by CPU SHA-256; end-to-end inference with the portable RGB pipeline is not yet validated.

Checkpoint variants

Folder	Method	Updates	Use
models/main	MemSPO (R0)	400	Paper main table
models/sft	SFT	50,000	Initialization/control
models/memspo200	MemSPO (R0)	200	Matched ablation
models/grpo200	GRPO (R0)	200	Matched ablation
models/ordinary200	Ordinary global beam8, top4 (R0)	200	Proposal-strategy ablation

All post-training checkpoints start from their paired SFT checkpoint. main and memspo200 must not be interchanged.

Main-table evaluation

128 trajectories, 32 predicted frames, 2 initial observations, per-frame Beam4, anchor1 + recent6, no true-future refresh. Recorded split: held-out test; runtime: GPU FP32.

PSNR ↑	SSIM ↑	LPIPS ↓
18.4719	0.765948	0.183338

These are recorded paper results; staging weights does not constitute a fresh reproduction.

Required components

• Dataset-specific codec/ and preprocessing/ files, paired by the manifest.
• Trained text adapters in each checkpoint and frozen T5-base at revision a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1; T5 is referenced, not duplicated.
• 52 action tokens per predicted frame. RoboCasa uses four ordered 13-dimensional control blocks; do not flatten time in a different order.
• Original FP32 safetensors; no post-hoc quantization or checkpoint merging.

Loading and scope

Use futureworlds.checkpoints.load_world_model(bundle_root, "robocasa", variant="main") for the predictor only. This does not decode RGB, prepare actions, or download T5 automatically. A complete RGB demo and clean-environment GPU equivalence run remain pending. These models are custom action-conditioned predictors, not generic pipeline("text-to-video") models.

Provenance and limitations

Core architecture and tokenizer derive from RLVR-World/iVideoGPT. Upstream license texts are included under licenses/; the license for original additions is not yet selected. The raw datasets and third-party baseline weights are not included. Learned-world feedback examples are qualitative generated trajectories, not independently verified physical task-success results.