# DreamDojo Finetuning ## LAM (Latent Action Model) ```bash export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}" source ./models/DreamDojo/.venv/bin/activate torchrun --nnodes=1 --nproc_per_node=2 finetuning/dreamdojo/lam_project/main.py fit --config finetuning/dreamdojo/lam_project/config/lam_0.yaml ``` ```bash # Debug single GPU: source ./models/DreamDojo/.venv/bin/activate python finetuning/dreamdojo/lam_project/main.py fit --config finetuning/dreamdojo/lam_project/config/lam_0.yaml ``` LAM inference: ```bash python finetuning/dreamdojo/lam_project/infer_video.py \ --config finetuning/dreamdojo/lam_project/config/lam_0.yaml \ --ckpt_path checkpoints/dreamdojo/LAM_400k.ckpt \ --video_path ./datasets/droid_1.0.1_20chunks/videos/chunk-002/observation.images.exterior_1_left/episode_002000.mp4 \ --output_dir lam_video_infer \ --device cuda ``` ## DreamDojo Posttraining (Action-Conditioned Video2World) ### Training ```bash source ./models/DreamDojo/.venv/bin/activate # ALOHA bimanual (2 GPUs, ~5s/iter steady-state, ~18h for 10k iters) NPROC=2 bash finetuning/dreamdojo/launch.sh dreamdojo_2b_480_640_aloha # DROID single-arm NPROC=2 bash finetuning/dreamdojo/launch.sh dreamdojo_2b_480_640_single_arm_sv # Single GPU debug NPROC=1 bash finetuning/dreamdojo/launch.sh dreamdojo_2b_480_640_aloha # Resume from iter_40k, train on all LeRobot roots under datasets/bimanual/singleview (54 tasks), # save to a new job folder (not exp_aloha/aloha_bimanual_singleview) NPROC=2 bash finetuning/dreamdojo/launch.sh dreamdojo_2b_480_640_aloha_resume40k_bimanual_sv # Resume from iter_70k → 100k (saves to exp_aloha_resume70k/aloha_bimanual_singleview_full/) NPROC=2 bash finetuning/dreamdojo/launch.sh dreamdojo_2b_480_640_aloha_resume70k_bimanual_sv ``` Configs: `models/DreamDojo/configs/2b_480_640_aloha.yaml`, `2b_480_640_droid_sv.yaml`, `2b_480_640_aloha_resume40k_bimanual_sv.yaml` (resume + full bimanual singleview list), `2b_480_640_aloha_resume70k_bimanual_sv.yaml` (resume 70k→100k) Checkpoints saved to: `exp_logs/dreamdojo_logs/dreamdojo/exp_//checkpoints/` Video samples logged to: `exp_logs/dreamdojo_logs/dreamdojo/exp_//video_logs/` Resume run above writes to `exp_logs/dreamdojo_logs/dreamdojo/exp_aloha_resume40k/aloha_bimanual_singleview_full/` and loads weights plus trainerizer/trainer/trainer state from `.../checkpoints/iter_000040000/`. If resume errors, edit that YAML and set `checkpoint.load_training_state: false` (weights only). Resume 70k run writes to `exp_logs/dreamdojo_logs/dreamdojo/exp_aloha_resume70k/aloha_bimanual_singleview_full/` and loads full training state from `.../exp_aloha_resume40k/.../checkpoints/iter_000070000/`. ### Inference (test on dataset samples) Given a dataset path and episode index, loads first frame + GT actions, generates predicted video, saves GT vs predicted comparison. ```bash source ./models/DreamDojo/.venv/bin/activate # Infer with pretrained 2B checkpoint on an ALOHA episode bash finetuning/dreamdojo/infer.sh \ --dataset_path /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/bimanual/singleview/makovian/close_cardboard_box \ --episode_idx 0 # Infer with a finetuned checkpoint bash finetuning/dreamdojo/infer.sh \ --dataset_path /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/bimanual/singleview/makovian/close_cardboard_box \ --episode_idx 0 \ --ckpt_path /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamdojo_logs/dreamdojo/exp_aloha/aloha_bimanual_singleview/checkpoints/iter_000010000/model # 2) Action-conditioned batch infer (num-samples nhiều hơn) export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}" source ./models/DreamDojo/.venv/bin/activate python -m models.DreamDojo.examples.action_conditioned \ -o ./output/dreamdojo_svg1_eval_finetuned \ --checkpoints-dir ./exp_logs/dreamdojo_logs/dreamdojo/exp_aloha/aloha_bimanual_singleview/checkpoints \ --experiment dreamdojo_2b_480_640_aloha \ --save-dir ./output/dreamdojo_results/results_finetuned_40k \ --num-frames 97 \ --num-samples 5 \ --dataset-path "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/bimanual/singleview/makovian/close_toolbox" \ --data-split test \ --deterministic-uniform-sampling # 3) Action-conditioned batch infer (num-samples nhiều hơn) export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}" source ./models/DreamDojo/.venv/bin/activate python -m models.DreamDojo.examples.action_conditioned \ -o ./output/dreamdojo_svg1_eval_finetuned \ --checkpoints-dir ./exp_logs/dreamdojo_logs/dreamdojo/exp_aloha_resume40k/aloha_bimanual_singleview_full/checkpoints \ --experiment dreamdojo_2b_480_640_aloha \ --save-dir ./output/dreamdojo_results/results_finetuned_resume40k \ --num-frames 97 \ --num-samples 5 \ --dataset-path "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/bimanual/singleview/makovian/close_toolbox" \ --data-split test \ --deterministic-uniform-sampling # DROID single-arm batch infer export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}" source ./models/DreamDojo/.venv/bin/activate python -m models.DreamDojo.examples.action_conditioned \ -o ./output/dreamdojo_droid_sv_eval_finetuned \ --checkpoints-dir ./exp_logs/dreamdojo_logs/dreamdojo/exp_droid_sv/droid_singlearm_singleview/checkpoints \ --experiment dreamdojo_2b_480_640_droid_sv \ --save-dir ./output/dreamdojo_results/results_droid_finetuned_40k \ --num-frames 97 \ --num-samples 5 \ --dataset-path "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/single_arm/singleview/makovian" \ --data-split test \ --deterministic-uniform-sampling # DROID dataset bash finetuning/dreamdojo/infer.sh \ --dataset_path /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets/single_arm/singleview/chunk-000 \ --episode_idx 5 \ --embodiment droid \ --experiment_name dreamdojo_2b_480_640_droid_sv # Options # --guidance 3.0 Classifier-free guidance scale # --seed 42 Random seed # --num_frames 13 Number of frames to generate (12 predicted + 1 conditional) # --output_dir ./infer_outputs # --fps 5 Output video FPS ``` Output: `infer_outputs/_ep_g_s_{gt,pred,comparison}.mp4` ### Action dimension reference | Embodiment | Range | Dim | |-----------|-------------|-----| | GR-1 | [0, 29) | 29 | | GR-1 retarget | [29, 58) | 29 | | G1 | [58, 101) | 43 | | YAM | [101, 147) | 46 | | AgiBot | [147, 169) | 22 | | ALOHA | [169, 183) | 14 | | DROID | [183, 190) | 7 | | Reserved | [190, 220) | 30 | | MANO | [220, 352) | 132 | | Latent | [352, 384) | 32 | ## DreamGen (Cosmos-Predict2 Video2World) ### Step 1: Prepare datasets ```bash # Bimanual ALOHA python finetuning/dreamgen/scripts/prepare_bimanual_dreamgen.py # Single-arm DROID python finetuning/dreamgen/scripts/prepare_single_arm_dreamgen.py # Humanoid GR1 — generate LLM captions first, then write humanoid_dreamgen/ bash scripts/run_caption_humanoid_singleview.sh ``` ### Step 2: Pre-compute T5-XXL embeddings (all 3 at once) ```bash # All three datasets in one command bash finetuning/dreamgen/scripts/compute_t5_embeddings.sh # Or a single dataset DATASETS=humanoid bash finetuning/dreamgen/scripts/compute_t5_embeddings.sh ``` ### Step 3: Train ```bash source models/dreamgen/.venv/bin/activate # All 3 datasets combined (bimanual + single_arm + humanoid) — recommended NPROC=4 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_all_singleview # Or train each embodiment separately NPROC=8 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_bimanual_singleview NPROC=8 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_single_arm_singleview NPROC=8 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_humanoid_singleview # Single GPU debug NPROC=1 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_all_singleview ``` Checkpoints saved to: `exp_logs/dreamgen_logs/posttraining/singleview/all_singleview/` ### Step 4: Inference ```bash bash finetuning/dreamgen/infer.sh \ --ckpt_path exp_logs/dreamgen_logs/posttraining/singleview/all_singleview/checkpoints/model/iter_000005000.pt \ --input_path /path/to/first_frame.png \ --prompt "A GR-1 humanoid robot picks up a corn cob and places it on the plate." \ --save_path output/dreamgen_all_infer.mp4 ``` NPROC=4 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_all_multiview NPROC=4 bash finetuning/dreamgen/launch.sh predict2_video2world_training_2b_all_singleview # ctrl world bash scripts/run_single_arm_multiview_ctrlworld.sh both CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_singleview_dreamdojo_dense.sh WORLDCACHE_REL_L1_THRESH=0.05 WORLDCACHE_PROBE_DEPTH=3 bash scripts/run_humanoid_singleview_dreamdojo_worldcache.sh FASTERCACHE_MODEL_INTERVAL=3 FASTERCACHE_BLOCK_INTERVAL=2 bash scripts/run_humanoid_singleview_dreamdojo_fastercache.sh CUDA_VISIBLE_DEVICES=1 bash /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/scripts/run_humanoid_singleview_dreamdojo_dicache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_worldcache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_fastercache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_dicache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_singleview_dreamdojo_worldcache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_singleview_dreamdojo_fastercache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamdojo_dicache.sh CUDA_VISIBLE_DEVICES=0,1 bash scripts/run_bimanual_multiview_dreamgen_dense.sh CUDA_VISIBLE_DEVICES=0,1 bash scripts/run_humanoid_multiview_dreamgen_dense.sh source models/DreamDojo/.venv/bin/activate hf upload-large-folder doanh25032004/video_gen_physics_real_video /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/datasets --repo-type dataset rm -rf /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamgen_logs/posttraining/multiview/all_multiview/checkpoints_40_per/trainer/iter_00004* rm -rf /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamgen_logs/posttraining/multiview/all_multiview/checkpoints_40_per/trainer/iter_00005* rm -rf /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamgen_logs/posttraining/multiview/all_multiview/checkpoints_40_per/trainer/iter_00006* rm -rf /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamgen_logs/posttraining/multiview/all_multiview/checkpoints_40_per/trainer/iter_00007* rm -rf /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/dreamgen_logs/posttraining/multiview/all_multiview/checkpoints_40_per/trainer/iter_00008* CUDA_VISIBLE_DEVICES=0,1 bash scripts/run_all_singleview_dreamgen_dense.sh CUDA_VISIBLE_DEVICES=0,1 EMBODIMENTS="bimanual" bash scripts/run_all_singleview_dreamgen_dense.sh CUDA_VISIBLE_DEVICES=0,1 EMBODIMENTS="humanoid" bash scripts/run_all_singleview_dreamgen_dense.sh CUDA_VISIBLE_DEVICES=0,1 EMBODIMENTS="single_arm" bash scripts/run_all_singleview_dreamgen_dense.sh # Single arm (fps=15) CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 bash scripts/run_dreamgen_singleview_native_fps.sh CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 EMBODIMENT=single_arm bash scripts/run_dreamgen_singleview_native_fps.sh # Bimanual (fps=30) CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 EMBODIMENT=bimanual bash scripts/run_dreamgen_singleview_native_fps.sh # Humanoid (fps=20) CUDA_VISIBLE_DEVICES=0 EMBODIMENT=humanoid bash scripts/run_dreamgen_singleview_native_fps.sh SKIP_PREP=1 NPROC=4 bash finetuning/dreamgen/train_single_arm_static_16fps.sh SKIP_PREP=1 NPROC=4 bash finetuning/dreamgen/train_bimanual_16fps.sh NUM_EPISODES=10 CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamdojo_long_autoregressive.sh # 1. DreamGen humanoid singleview + SiTo CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_sito.sh # 2. DreamGen humanoid singleview + ITM CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_itm.sh # 3. Ctrl-World single_arm multiview + SiTo -> need to verified actually CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_multiview_ctrlworld_sito.sh both # 4. Ctrl-World single_arm multiview + ITM -> need to verified actually CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_multiview_ctrlworld_itm.sh both CUDA_VISIBLE_DEVICES=0 USE_PROMPT_REFINEMENT=1 bash scripts/run_dreamgen_singleview_native_fps.sh # 1. SiTo CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_sito.sh # 2. ITM CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_itm.sh # 3. PISA CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_pisa.sh # 4. Radial CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_radial.sh # 5. SVG1 CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamgen_svg1.sh # 6. SVG2 CUDA_VISIBLE_DEVICES=1 bash scripts/run_humanoid_singleview_dreamgen_svg2.sh # 7. Prompt Refinement per Chunk CUDA_VISIBLE_DEVICES=0,1 ITER_NAME=iter_000090000 bash scripts/run_humanoid_singleview_dreamgen_prompt_refine.sh NPROC=4 bash finetuning/dreamgen/train_humanoid_16fps.sh SKIP_PREP=1 NPROC=4 bash finetuning/dreamgen/train_single_arm_static_16fps.sh CUDA_VISIBLE_DEVICES=0 ITER_NAME=iter_000080000 bash scripts/run_single_arm_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=0,1 ITER_NAME=iter_000090000 PROMPT_REFINE=1 bash scripts/run_single_arm_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=2,3 ITER_NAME=iter_000070000 PROMPT_REFINE=0 bash scripts/run_bimanual_singleview_dreamgen_16fps.sh -> khá tốt CUDA_VISIBLE_DEVICES=2,3 ITER_NAME=iter_000070000 PROMPT_REFINE=0 bash scripts/run_bimanual_singleview_dreamgen_16fps.sh CUDA_VISIBLE_DEVICES=0,1 ITER_NAME=iter_000070000 PROMPT_REFINE=1 MASTER_PORT=29600 bash scripts/run_bimanual_singleview_dreamgen_16fps.sh CUDA_VISIBLE_DEVICES=2,3 ITER_NAME=iter_000120000 PROMPT_REFINE=0 bash scripts/run_bimanual_singleview_dreamgen_16fps.sh CUDA_VISIBLE_DEVICES=2,3 ITER_NAME=iter_000110000 PROMPT_REFINE=0 MASTER_PORT=29700 bash scripts/run_bimanual_singleview_dreamgen_16fps.sh CUDA_VISIBLE_DEVICES=0,1,2,3 ITER_NAME=iter_000100000 PROMPT_REFINE=1 bash scripts/run_single_arm_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=0,1,2,3 PROMPT_REFINE=1 ITER_NAME=iter_000070000 bash scripts/run_single_arm_singleview_dreamgen_base.sh CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 bash scripts/run_humanoid_singleview_dreamgen_base.sh CUDA_VISIBLE_DEVICES=0,1, NUM_GPUS=2 ITER_NAME=iter_000100000 bash scripts/run_humanoid_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=2,3 NUM_GPUS=2 ITER_NAME=iter_000090000 PROMPT_REFINE=1 bash scripts/run_humanoid_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_worldcache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_dicache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamdojo_worldcache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamdojo_dicache.sh -> oke CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_singleview_dreamdojo_fastercache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_singleview_dreamdojo_fastercache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_multiview_dreamgen_worldcache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_multiview_dreamgen_fastercache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_multiview_dreamgen_dicache.sh cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC=4 SKIP_PREP=1 SKIP_T5=1 \ bash finetuning/dreamgen/train_bimanual_multiview_grid.sh CUDA_VISIBLE_DEVICES=1 NUM_GPUS=1 ITER_NAME=iter_000090000 MASTER_PORT=29650 bash scripts/run_single_arm_singleview_dreamgen_static16fps.sh MAX_EPISODES=5 NUM_GPUS=1 CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_seg_test.sh MAX_EPISODES=5 NUM_CONDITIONAL_FRAMES=5 NUM_GPUS=1 CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_seg_rollout.sh MAX_EPISODES=5 NUM_CONDITIONAL_FRAMES=1 NUM_GPUS=2 CUDA_VISIBLE_DEVICES=2,3 bash scripts/run_bimanual_multiview_dreamgen_seg_rollout.sh CUDA_VISIBLE_DEVICES=0 CATEGORY=makovian MAX_EPISODES=4 \ bash scripts/run_single_arm_multiview_dreamgen_itm.sh Cấu hình Total inference Self-Attn / step DiT MLP / step Dense (baseline) 516.1s 4.66s 1.88s CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_multiview_dreamgen_itm.sh CUDA_VISIBLE_DEVICES=1 nohup bash scripts/run_single_arm_multiview_dreamgen_sito.sh cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics CKPT=exp_logs/ctrlworld_logs/bimanual_multiview/checkpoint-120000.pt \ CUDA_VISIBLE_DEVICES=0 NUM_EPISODES=3 SPLIT=val GUIDANCE=2.5 \ bash scripts/run_bimanual_multiview_ctrlworld.sh CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 SPLITS="makovian" NUM_CHUNKS="2" bash scripts/run_single_arm_singleview_dreamgen_static16fps.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamgen_worldcache.sh ; \ CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamgen_fastercache.sh ; \ CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamgen_dicache.sh #run lai CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamdojo_all_methods.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_humanoid_singleview_dreamdojo_all_methods.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_multiview_ctrlworld_sito.sh both CKPT="/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/ctrlworld_logs/bimanual_multiview_5fps/checkpoint-280000.pt" \ DATASET_NAME=bimanual_multiview_5fps \ DOWN_SAMPLE=6 \ SPLIT=all \ CUDA_VISIBLE_DEVICES=0 \ bash scripts/run_bimanual_multiview_ctrlworld.sh CKPT="/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/exp_logs/ctrlworld_logs/humanoid_multiview/checkpoint-150000.pt" \ DATASET_NAME=humanoid_multiview \ DOWN_SAMPLE=6 \ SPLIT=all \ CUDA_VISIBLE_DEVICES=1 \ bash scripts/run_humanoid_multiview_ctrlworld.sh cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics # WorldCache (tuned optimum: thresh 0.04 / ret 0.4) CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_ctrlworld_worldcache.sh CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_multiview_ctrlworld_dicache.sh CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_dicache.sh && \ CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_worldcache.sh && \ CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_fastercache.sh # ---- tmux submit: 3 DreamGen chia đều GPU0/1/2, 2 Ctrl-World tuần tự trên GPU3 ---- # (logs -> exp_logs/tmux_logs/, attach: tmux attach -t ) tmux new-session -d -s dg_dicache_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_dicache.sh 2>&1 | tee exp_logs/tmux_logs/dg_dicache.log; exec bash" tmux new-session -d -s dg_worldcache_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_multiview_dreamgen_worldcache.sh 2>&1 | tee exp_logs/tmux_logs/dg_worldcache.log; exec bash" tmux new-session -d -s dg_fastercache_g2 "CUDA_VISIBLE_DEVICES=2 bash scripts/run_bimanual_multiview_dreamgen_fastercache.sh 2>&1 | tee exp_logs/tmux_logs/dg_fastercache.log; exec bash" tmux new-session -d -s cw_cache_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_bimanual_multiview_ctrlworld_worldcache.sh 2>&1 | tee exp_logs/tmux_logs/cw_worldcache.log && CUDA_VISIBLE_DEVICES=3 bash scripts/run_bimanual_multiview_ctrlworld_dicache.sh 2>&1 | tee exp_logs/tmux_logs/cw_dicache.log; exec bash" # ---- tmux submit: bimanual SINGLEVIEW dreamgen (iter 110k), 3 cache trên GPU0/1/3 ---- tmux new-session -d -s bsv_dicache_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamgen_dicache.sh 2>&1 | tee exp_logs/tmux_logs/bsv_dicache.log; exec bash" tmux new-session -d -s bsv_worldcache_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_singleview_dreamgen_worldcache.sh 2>&1 | tee exp_logs/tmux_logs/bsv_worldcache.log; exec bash" tmux new-session -d -s bsv_fastercache_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_bimanual_singleview_dreamgen_fastercache.sh 2>&1 | tee exp_logs/tmux_logs/bsv_fastercache.log; exec bash" # ---- tmux submit: bimanual SINGLEVIEW dreamgen sparse-attn (iter 110k) GPU0/1/2/3 ---- # (SVG2 uses .venv_svg w/ flashinfer0.3 + workspace .jitcache; set in the script.) tmux new-session -d -s sp_svg1_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_singleview_dreamgen_svg1.sh 2>&1 | tee exp_logs/tmux_logs/sp_svg1.log; exec bash" tmux new-session -d -s sp_svg2_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_singleview_dreamgen_svg2.sh 2>&1 | tee exp_logs/tmux_logs/sp_svg2.log; exec bash" tmux new-session -d -s sp_radial_g2 "CUDA_VISIBLE_DEVICES=2 bash scripts/run_bimanual_singleview_dreamgen_radial.sh 2>&1 | tee exp_logs/tmux_logs/sp_radial.log; exec bash" tmux new-session -d -s sp_pisa_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_bimanual_singleview_dreamgen_pisa.sh 2>&1 | tee exp_logs/tmux_logs/sp_pisa.log; exec bash" CUDA_VISIBLE_DEVICES=0,3 MAX_EPISODES=5 bash scripts/run_humanoid_multiview_dreamgen_seg16.sh # ---- tmux submit: single_arm MULTIVIEW dreamgen sparse-attn (14B, droid) GPU0/1/2/3 ---- # (SVG2 uses .venv_svg w/ flashinfer0.3; set in the script. 14B is heavy ~490s/ep dense.) tmux new-session -d -s sam_pisa_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_multiview_dreamgen_pisa.sh 2>&1 | tee exp_logs/tmux_logs/sam_pisa.log; exec bash" tmux new-session -d -s sam_svg1_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_multiview_dreamgen_svg1.sh 2>&1 | tee exp_logs/tmux_logs/sam_svg1.log; exec bash" tmux new-session -d -s sam_radial_g2 "CUDA_VISIBLE_DEVICES=2 bash scripts/run_single_arm_multiview_dreamgen_radial.sh 2>&1 | tee exp_logs/tmux_logs/sam_radial.log; exec bash" tmux new-session -d -s sam_svg2_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_single_arm_multiview_dreamgen_svg2.sh 2>&1 | tee exp_logs/tmux_logs/sam_svg2.log; exec bash" # ---- tmux submit: single_arm SINGLEVIEW dreamgen sparse-attn (2B, static16fps/LVG) GPU0/1/2/3 ---- # (SVG2 uses .venv_svg; PISA/SVG1/Radial use .venv. Tuned: PISA d0.2, SVG1 sp0.15, SVG2 top_p0.7.) tmux new-session -d -s sasv_pisa_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_single_arm_singleview_dreamgen_pisa.sh 2>&1 | tee exp_logs/tmux_logs/sasv_pisa.log; exec bash" tmux new-session -d -s sasv_svg1_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_single_arm_singleview_dreamgen_svg1.sh 2>&1 | tee exp_logs/tmux_logs/sasv_svg1.log; exec bash" tmux new-session -d -s sasv_radial_g2 "CUDA_VISIBLE_DEVICES=2 bash scripts/run_single_arm_singleview_dreamgen_radial.sh 2>&1 | tee exp_logs/tmux_logs/sasv_radial.log; exec bash" -> done tmux new-session -d -s sasv_svg2_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_single_arm_singleview_dreamgen_svg2.sh 2>&1 | tee exp_logs/tmux_logs/sasv_svg2.log; exec bash" -> done # ---- tmux submit: bimanual MULTIVIEW dreamgen sparse-attn (iter 120k) GPU0/1/2/3 ---- # (Radial=decay0.5/start4 for multiview; SVG2 uses .venv_svg/flashinfer0.3.) tmux new-session -d -s smv_pisa_g0 "CUDA_VISIBLE_DEVICES=0 bash scripts/run_bimanual_multiview_dreamgen_pisa.sh 2>&1 | tee exp_logs/tmux_logs/smv_pisa.log; exec bash" tmux new-session -d -s smv_radial_g1 "CUDA_VISIBLE_DEVICES=1 bash scripts/run_bimanual_multiview_dreamgen_radial.sh 2>&1 | tee exp_logs/tmux_logs/smv_radial.log; exec bash" tmux new-session -d -s smv_svg1_g2 "CUDA_VISIBLE_DEVICES=2 bash scripts/run_bimanual_multiview_dreamgen_svg1.sh 2>&1 | tee exp_logs/tmux_logs/smv_svg1.log; exec bash" tmux new-session -d -s smv_svg2_g3 "CUDA_VISIBLE_DEVICES=3 bash scripts/run_bimanual_multiview_dreamgen_svg2.sh 2>&1 | tee exp_logs/tmux_logs/smv_svg2.log; exec bash" # ---- bimanual DreamGen SiTo/ITM (mv + sv) — SEQUENTIAL on 1 GPU (SiTo=1.3x win; ITM=net-negative valid point) ---- # (single-GPU only; run one at a time so per-segment timing is clean. resumable.) tmux new-session -d -s bim_prune "CUDA_VISIBLE_DEVICES=0 bash -c '\ bash scripts/run_bimanual_multiview_dreamgen_sito.sh 2>&1 | tee exp_logs/tmux_logs/bim_mv_sito.log && \ bash scripts/run_bimanual_multiview_dreamgen_itm.sh 2>&1 | tee exp_logs/tmux_logs/bim_mv_itm.log && \ bash scripts/run_bimanual_singleview_dreamgen_sito.sh 2>&1 | tee exp_logs/tmux_logs/bim_sv_sito.log && \ bash scripts/run_bimanual_singleview_dreamgen_itm.sh 2>&1 | tee exp_logs/tmux_logs/bim_sv_itm.log'; exec bash"