README_cache.md

Cache Strategy Guide for DreamDojo

video_gen_physics/README_cache.md is the authoritative document for DreamDojo cache backends in this workspace.

The legacy snapshot under world_cache/VLA-Humanoid/Models/DreamDojo is kept only as historical reference. The primary code path now lives in:

• video_gen_physics/methods/cache_strategy/WorldCache
• video_gen_physics/methods/cache_strategy/DiCache
• video_gen_physics/models/DreamDojo

Scope

This integration targets the DreamDojo action-conditioned inference path in video_gen_physics/models/DreamDojo.

It does not change the generic cosmos_predict2/inference.py flow.

Cache backends are treated as inference-only and v1 single-GPU only.

Backend Rules

The DreamDojo action-conditioned path treats the following acceleration families as mutually exclusive:

• WorldCache
• DiCache
• FasterCache
• SiTo
• SVG
• ITM
• dense-vs-sparse backend overrides such as NATTEN

If a cache backend is enabled, DreamDojo will require dense minimal_a2a attention at runtime.

Model Geometry

The recommended cache thresholds should be reasoned from DreamDojo latent token geometry, not from raw pixel frames alone.

For the current DreamDojo GR1 action-conditioned path:

• tokenizer spatial compression = 8
• latent patch spatial = 2
• effective spatial token factor = 16
• temporal compression = 4

Sequence length is:

S = state_t * (H / 16) * (W / 16)

For the common GR1 evaluation setting:

• pixel resolution = 480 x 640
• pixel frames per chunk = 13
• latent frames = state_t = 1 + 12 // 4 = 4

So:

S = 4 * (480 / 16) * (640 / 16)
  = 4 * 30 * 40
  = 4800 tokens

This is the reference geometry used for the starting values below.

Tuning Defaults

These are recommended starting points derived from:

• DreamDojo latent geometry in this repo
• WorldCache, arXiv 2603.22286, published March 23, 2026
• DiCache, arXiv 2508.17356, published August 24, 2025

They should be treated as starting points, not as verified benchmark claims for this workspace. Local checkpoints were not available during this integration pass, so no local speed or quality claim is asserted here.

WorldCache

Note on threshold calibration for this repo

The VLA-Humanoid reference repo contained two math bugs that artificially inflated the cache hit rate when using --worldcache-rel-l1-thresh 0.03:

1. num_steps miscounting: Sequential CFG produces 70 forward passes but the original code set worldcache_num_steps=35, causing the drift accumulator to reset halfway through — effectively running two "easy" warm-up windows.
2. t_embedding_norm order: The original code mixed action_emb after t_embedding_norm, which suppressed timestep-to-timestep variance and made delta_y scores fall below 0.03 very easily.

Both bugs are fixed in this repo. As a result the threshold must be raised to achieve the same skip rate (≈30–40%) and the same ≈6–7 it/s throughput. Do not use 0.03 here — it will give only ~8% skip and roughly half the speed.

Recommended starting values for this repo:

• 2B / 7B, 28 blocks:
  • --worldcache-probe-depth 4
  • --worldcache-ret-ratio 0.5
  • --worldcache-rel-l1-thresh 0.08
• 14B, 36 blocks:
  • --worldcache-probe-depth 6
  • --worldcache-ret-ratio 0.5
  • --worldcache-rel-l1-thresh 0.065

Interpretation:

• probe_depth=4 covers ~14% of the 28-block stack — enough to measure drift cheaply.
• ret_ratio=0.5 means the first 50% of timesteps (17/35) always run full forward; reuse only starts mid-denoising when the latent is already converging.
• rel_l1_thresh is now correctly calibrated against the normalized t_embedding distribution. Values in [0.05, 0.12] are the practical operating range:
  • lower → fewer skips, higher quality, slower
  • higher → more skips, potential drift artifacts, faster

Optional quality guards (activate in order if needed):

• --worldcache-hf-enabled — blocks skips when high-frequency edge detail drifts
• --worldcache-saliency-enabled — weights drift by per-channel variance saliency
• --worldcache-dynamic-decay — relaxes threshold linearly toward end of denoising
• --worldcache-aduc-enabled — skips the unconditional branch at late timesteps

DiCache

Recommended starting values:

• 2B / 7B, 28 blocks:
  • --dicache-probe-depth 2
  • --dicache-ret-ratio 0.2
  • --dicache-rel-l1-thresh 0.08
• 14B, 36 blocks:
  • --dicache-probe-depth 3
  • --dicache-ret-ratio 0.22
  • --dicache-rel-l1-thresh 0.07

Interpretation:

• DiCache uses a shallower probe and simpler reuse schedule than WorldCache.
• It is usually the safer first cache baseline if you want minimal tuning.

FasterCache

Recommended starting values:

• --use_fastercache
• --fastercache_start_step 0
• --fastercache_model_interval 5
• --fastercache_block_interval 3

Interpretation:

• fastercache_start_step=0 auto-resolves to ceil(0.3 * num_inference_steps).
• Model-level FasterCache only shortcuts the unconditional CFG branch on non-anchor denoising steps.
• Block-level FasterCache is self-attention only and stays disabled for short sequences.
• For the common GR1 action-conditioned setting in this repo, seq_len = 4800, so the block-level path is eligible.

Threshold Scaling Rule

For other sequence lengths, start from the S = 4800 reference and rescale the threshold as:

thresh_scaled = thresh_ref * sqrt(4800 / S)

Practical guidance:

• If S increases, decrease the threshold.
• If the dataset has stronger motion, decrease the threshold further.
• If num_latent_conditional_frames = 2, decrease the threshold further because temporal change is less stationary.
• If quality drifts too much, lower threshold before increasing probe depth.
• If speedup is too weak but quality is stable, raise ret_ratio carefully only after threshold is already reasonable.

Runtime Notes

• Cache num_steps must match the denoising step count used by the sampler.
• In the current DreamDojo action-conditioned path, the cache config drives the effective denoising step count for cache-enabled runs.
• worldcache_parallel_cfg means batched cond/uncond classifier-free guidance in one forward pass. It is not multi-GPU context parallelism.
• Cache backends are currently blocked when context_parallel_size > 1.

How To Run

Run from the video_gen_physics root, consistent with the other method READMEs:

source /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/models/DreamDojo/.venv/bin/activate
cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics
export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}"

Baseline (DreamDojo)

python -m models.DreamDojo.examples.action_conditioned \
  -o ./output/action_conditioned/cache_baseline \
  --checkpoints-dir ./checkpoints/dreamdojo/2B_GR1_post-train \
  --experiment dreamdojo_2b_480_640_gr1 \
  --save-dir ./output/dreamdojo_cache/baseline \
  --num-frames 49 \
  --num-samples 65 \
  --dataset-path ./checkpoints/dreamdojo/datasets/PhysicalAI-Robotics-GR00T-Teleop-GR1/GR1_robot \
  --data-split test \
  --deterministic-uniform-sampling

DiCache (DreamDojo)

python -m models.DreamDojo.examples.action_conditioned \
  -o ./output/action_conditioned/cache_dicache \
  --checkpoints-dir ./checkpoints/dreamdojo/2B_GR1_post-train \
  --experiment dreamdojo_2b_480_640_gr1 \
  --save-dir ./output/dreamdojo_cache/dicache \
  --num-frames 49 \
  --num-samples 65 \
  --dataset-path ./checkpoints/dreamdojo/datasets/PhysicalAI-Robotics-GR00T-Teleop-GR1/GR1_robot \
  --data-split test \
  --deterministic-uniform-sampling \
  --dicache-enabled \
  --dicache-num-steps 35 \
  --dicache-probe-depth 2 \
  --dicache-ret-ratio 0.2 \
  --dicache-rel-l1-thresh 0.08

WorldCache (DreamDojo)

python -m models.DreamDojo.examples.action_conditioned \
  -o ./output/action_conditioned/cache_worldcache \
  --checkpoints-dir ./checkpoints/dreamdojo/2B_GR1_post-train \
  --experiment dreamdojo_2b_480_640_gr1 \
  --save-dir ./output/dreamdojo_cache/worldcache \
  --num-frames 49 \
  --num-samples 65 \
  --dataset-path ./checkpoints/dreamdojo/datasets/PhysicalAI-Robotics-GR00T-Teleop-GR1/GR1_robot \
  --data-split test \
  --deterministic-uniform-sampling \
  --worldcache-enabled \
  --worldcache-num-steps 35 \
  --worldcache-rel-l1-thresh 0.065 \
  --worldcache-motion-sensitivity 3.0 \
  --worldcache-probe-depth 4 \
  --worldcache-ret-ratio 0.5

How To Run (DreamGen 2.0 / GR00T)

The cache_strategy module is also natively integrated into the models/dreamgen/examples/ pipeline. Below is an example applying WorldCache to the video2world_gr00t.py script.

Note: For DreamGen, make sure PYTHONPATH points to the workspace root rather than models/DreamDojo.

cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics

export PYTHONPATH="./models/dreamgen"

# Execute text/image-to-world with 14B GR00T
torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --worldcache-enabled \
    --worldcache-num-steps 35 \
    --worldcache-rel-l1-thresh 0.08 \
    --worldcache-ret-ratio 0.5 \
    --worldcache-probe-depth 4 \
    --worldcache-motion-sensitivity 4.0 \
    --worldcache-dynamic-decay
--->>> Total inference time: 347.0921 seconds

torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark_world_cache/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --worldcache-enabled \
    --worldcache-num-steps 35 \
    --worldcache-rel-l1-thresh 0.06 \
    --worldcache-ret-ratio 0.6 \
    --worldcache-probe-depth 6 \
    --worldcache-motion-sensitivity 5.0 \
    --worldcache-dynamic-decay
--->>> Total inference time: 399.7761 seconds 

torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark_world_cache/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --worldcache-enabled \
    --worldcache-num-steps 35 \
    --worldcache-rel-l1-thresh 0.05 \
    --worldcache-ret-ratio 0.65 \
    --worldcache-probe-depth 7 \
    --worldcache-motion-sensitivity 5.0 \
    --worldcache-dynamic-decay
--> Total inference time: 408.3197 seconds

torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark_world_cache/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --worldcache-enabled \
    --worldcache-num-steps 35 \
    --worldcache-rel-l1-thresh 0.05 \
    --worldcache-ret-ratio 0.55 \
    --worldcache-probe-depth 10 \
    --worldcache-motion-sensitivity 7.0 \
    --worldcache-dynamic-decay
--> Total inference time: 413.3745 seconds                                               

torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark_world_cache/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --worldcache-enabled \
    --worldcache-num-steps 35 \
    --worldcache-rel-l1-thresh 0.03 \
    --worldcache-ret-ratio 0.6 \
    --worldcache-probe-depth 15 \
    --worldcache-motion-sensitivity 9.0 \
    --worldcache-dynamic-decay

Nếu đang dùng WorldCache, tham số chính để skip mạnh hơn là --worldcache-rel-l1-thresh. Tăng cái này trước.

Với config hiện tại của bạn 0.065 / 6 / 0.5, thứ tự chỉnh nên là:

Tăng --worldcache-rel-l1-thresh lên 0.075, rồi 0.08, nếu còn muốn mạnh nữa thì 0.085. Giảm --worldcache-ret-ratio từ 0.5 xuống 0.4 hoặc 0.35. Giảm --worldcache-probe-depth từ 6 xuống 5 hoặc 4. Giảm --worldcache-motion-sensitivity từ 5.0 xuống 3.0 nếu muốn cache bớt nhạy với motion. Bật --worldcache-dynamic-decay nếu muốn skip mạnh hơn về cuối denoise. Bật --worldcache-aduc-enabled nếu muốn tăng tốc thêm nhánh uncond ở cuối, nhưng đây không phải full-step skip.

FasterCache (DreamGen)

Baseline:

torchrun -m models.dreamgen.examples.video2world_gr00t     --model_size 14B     --gr00t_variant gr1     --batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json     --disable_guardrail     --num_gpus 1 

[04-23 04:09:25|INFO|models/dreamgen/cosmos_predict2/models/text2image_dit.py:1779:forward] DiT Blocks time: 7.1864 seconds [04-23 04:09:25|INFO|models/dreamgen/cosmos_predict2/models/text2image_dit.py:1781:forward] - Total Self-Attention: 4.5648s

FasterCache acceleration is also supported in the DreamGen pipeline. Note that FasterCache v1 is currently limited to single-GPU inference.

export PYTHONPATH="./models/dreamgen:${PYTHONPATH}"

# Execute with FasterCache enabled (Optimized for speed)
torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --use_fastercache \
    --fastercache_model_interval 10 \
    --fastercache_block_interval 6 \
    --fastercache_debug
] Total inference time: 375.0514 seconds 

**Giải thích các thông số FasterCache và cách tune để tăng chất lượng ("tối ưu quality hơn"):**

- `--fastercache_start_step`: Bước (timestep) bắt đầu bật cache. Các bước đầu tiên của diffusion cực kỳ quan trọng để định hình cấu trúc và chi tiết của video.
  - **Cách tune để tăng quality:** Tăng giá trị này lên (ví dụ từ 0 hoặc 5 lên `10` hoặc `15`). Chạy đầy đủ không cache ở các bước đầu càng nhiều thì video xuất ra càng đẹp, sắc nét và bám sát prompt, đánh đổi lại là tốc độ giảm đi đôi chút.
- `--fastercache_model_interval`: Khoảng cách số bước để model tính toán lại hoàn toàn nhánh *unconditional* (thay vì skip và dùng FFT deltas tính từ nhánh conditional).
  - **Cách tune để tăng quality:** Giảm giá trị này xuống (ví dụ từ 10 xuống `3` hoặc `2`). Cập nhật nhánh uncond thường xuyên hơn giúp Classifier-Free Guidance (CFG) ổn định hơn, tránh vỡ hình ảnh.
- `--fastercache_block_interval`: Khoảng cách số bước để tính toán lại chính xác *Self-Attention* trong các DiT blocks (thay vì dùng lại output của các bước trước).
  - **Cách tune để tăng quality:** Giảm giá trị này xuống (ví dụ từ 6 xuống `4` hoặc `3`). Giúp các token duy trì liên kết không gian - thời gian tốt hơn, giảm hiện tượng bóng mờ (ghosting) hoặc nhiễu chi tiết.

--> Ví dụ lệnh chạy tối ưu quality hơn (chậm hơn một chút nhưng chi tiết tốt hơn):
```bash
torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --use_fastercache \
    --fastercache_start_step 10 \
    --fastercache_model_interval 3 \
    --fastercache_block_interval 3 \
    --fastercache_debug
--->>> Total inference time: 461.3048 seconds 
torchrun -m models.dreamgen.examples.video2world_gr00t \
    --model_size 14B \
    --gr00t_variant gr1 \
    --batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json \
    --disable_guardrail \
    --num_gpus 1 \
    --use_fastercache \
    --fastercache_start_step 12 \
    --fastercache_model_interval 2  \
    --fastercache_block_interval 3 \
    --fastercache_debug
Total inference time: 516.1998 seconds

Same DiCache flags (--dicache-enabled, etc.) seamlessly apply to DreamGen scripts.

FasterCache

export PYTHONPATH="./models/DreamDojo:${PYTHONPATH}"
python -m models.DreamDojo.examples.action_conditioned \
  -o ./output/action_conditioned/cache_fastercache \
  --checkpoints-dir ./checkpoints/dreamdojo/2B_GR1_post-train \
  --experiment dreamdojo_2b_480_640_gr1 \
  --save-dir ./output/dreamdojo_cache/fastercache \
  --num-frames 49 \
  --num-samples 65 \
  --dataset-path ./checkpoints/dreamdojo/datasets/PhysicalAI-Robotics-GR00T-Teleop-GR1/GR1_robot \
  --data-split test \
  --deterministic-uniform-sampling \
  --fastercache-enabled \
  --fastercache-debug \
  --fastercache_start_step 0 \
  --fastercache_model_interval 2 \
  --fastercache_block_interval 5

Ctrl-World Cache Backends

Run from the video_gen_physics root:

source /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics/models/DreamDojo/.venv/bin/activate
cd /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/doanh/video_world/video_gen_physics
export PYTHONPATH="./models/Ctrl-World:${PYTHONPATH}"

Ctrl-World cache backends are inference-only and mutually exclusive with PISA, Radial, SVG, SiTo, ITM, and FasterCache.

Ctrl-World DiCache

python models/Ctrl-World/scripts/rollout_interact_pi.py \
  --dataset_root_path ./models/Ctrl-World/dataset_example \
  --dataset_meta_info_path ./models/Ctrl-World/dataset_meta_info \
  --dataset_names ./models/Ctrl-World/droid_subset \
  --svd_model_path ./checkpoints/stabilityai/stable-video-diffusion-img2vid \
  --clip_model_path ./checkpoints/openai/clip-vit-base-patch32 \
  --ckpt_path ./checkpoints/ctrl_world/Ctrl-World/checkpoint-10000.pt \
  --task_type pickplace \
  --use_dicache \
  --dicache_rel_l1_thresh 0.08 \
  --dicache_ret_ratio 0.2 \
  --dicache_probe_depth 2

Ctrl-World WorldCache

python models/Ctrl-World/scripts/rollout_replay_traj.py \
  --dataset_root_path ./models/Ctrl-World/dataset_example \
  --dataset_meta_info_path ./models/Ctrl-World/dataset_meta_info \
  --dataset_names ./models/Ctrl-World/droid_subset \
  --svd_model_path ./checkpoints/stabilityai/stable-video-diffusion-img2vid \
  --clip_model_path ./checkpoints/openai/clip-vit-base-patch32 \
  --ckpt_path ./checkpoints/ctrl_world/Ctrl-World/checkpoint-10000.pt \
  --use_worldcache \
  --worldcache_rel_l1_thresh 0.03 \
  --worldcache_ret_ratio 0.4 \
  --worldcache_probe_depth 3 \
  --worldcache_motion_sensitivity 5.0

Suggested Tuning Order

For WorldCache:

1. Fix num_steps
2. Start with the recommended probe_depth
3. Tune rel_l1_thresh
4. Then tune ret_ratio
5. Only then try hf, saliency, dynamic_decay, or aduc

For DiCache:

1. Fix num_steps
2. Start with the recommended probe_depth
3. Tune rel_l1_thresh
4. Tune ret_ratio last

Verification Checklist

When checkpoints become available locally, compare:

• baseline wall time
• cache wall time
• skip ratio from runtime logs
• output PSNR / SSIM / LPIPS artifacts already produced by the DreamDojo eval path

Keep prompt, chunking, checkpoint, dataset split, and seed policy fixed across comparisons.

torchrun -m models.dreamgen.examples.video2world_gr00t
--model_size 14B
--gr00t_variant gr1
--batch_input_json ./output/dream_gen_benchmark/cosmos_predict2_14b_gr1_behavior/batch_input.json
--disable_guardrail
--num_gpus 1
--dicache-enabled
--dicache-num-steps 35
--dicache-probe-depth 2
--dicache-ret-ratio 0.2
--dicache-rel-l1-thresh 0.08

---> Total inference time: 355.6783 seconds