luuuulinnnn/egohoi-control-hot3d

EgoHOI Control — HOT3D (Wan 2.1 I2V-14B)

Conditional egocentric world-model training on HOT3D, built on a frozen Wan 2.1 I2V-14B-720P DiT. Seven runs covering all combinations of the three trainable conditioning modules (camera_encoder, hand_encoder, obj_encoder), trained with scripts/train.py from the obj_rope branch of edisiondyli/egohoi_extension.

All 7 runs completed the full 8000 steps.

Results

Sorted by validation loss (lower is better). Validation uses fixed timesteps + deterministic noise, so values are directly comparable across runs.

run	camera	hand	obj	GPUs	val loss	train cum-avg	steps
train_all_encoders	✅	✅	✅	8×H200	0.05562	0.12364	8000
train_camera_hand	✅	✅	❌	4×H200	0.05840	0.12395	8000
train_camera_obj	✅	❌	✅	4×H200	0.06153	0.12524	8000
train_camera_only	✅	❌	❌	4×H200	0.06196	0.12604	8000
train_hand_obj	❌	✅	✅	4×H200	0.07644	0.12815	8000
train_hand_only	❌	✅	❌	4×H200	0.07668	0.12837	8000
train_obj_only	❌	❌	✅	4×H200	0.08045	0.13067	8000

Two clear trends: more conditioning modules → lower validation loss, and the camera encoder contributes the most (every camera-enabled run beats every camera-disabled run).

Repository layout

<run_name>/
  checkpoints/          # model checkpoints
  train.log             # full training log (loss, lr, grad_norm, validation)
  slurm-<jobid>.out     # scheduler stdout
  resolved_config.yaml  # the exact resolved config used for the run
  tensorboard/          # training curves (events.out.tfevents.*)
  wandb/                # offline W&B run

View the curves for all runs at once:

tensorboard --logdir .

Training setup

Identical across all 7 runs except the three enable flags.

Base model	Wan 2.1 I2V-14B-720P (frozen)
Steps	8000
Optimizer	AdamW, lr 1e-5, weight_decay 0
Precision	bf16 + gradient checkpointing
Batch size	1 per GPU
Video	81 frames @ 480×480, start_frames [0, 34, 69]
Validation	every 400 steps, 50 clips, timesteps [357, 682, 833, 921, 978], deterministic noise
Checkpointing	every 400 steps
Seed	23
Hardware	NVIDIA H200 (2 CPUs per GPU), ~25–33 h per run

Conditioning modules

module	configuration
camera_encoder	ControlNet-style residuals into the first 20 Wan blocks; Plücker rays (256 hidden, 6 ch) + masks; control_dim 1024, 20 layers, 16 heads, ffn 4096, rms_norm
hand_encoder	Wan LoRA r128 / α128 on q,k,v,o,ffn.0,ffn.2 (kaiming init); Conv3d hand stack (concat_dim 4) + reference hand-frame injection
obj_encoder	Stand-In image-branch LoRA r128 / α128 on self_attn.{q,k,v} (zero init), all attention blocks; staged curriculum 1500 + 1500

Data

Preprocessed HOT3D clips (EndeavourDD/Jichen), 1516 clips total.

• Train: clips ≥ 1949 (568 distinct clips observed, range 1949–3364)
• Validation: 50 clips starting at 1849 (range ~1849–1898)
• Disjoint — clips 1899–1948 are held out as a gap between the two splits.

Captions come from hot3d_qwen_captions.json (4548 windows keyed as clip-XXXXXX_sNNN_eNNN).

Notes

• Checkpoints are full training checkpoints (model + optimizer state), not merged weights.
• final.pt is the end of training (step 8000). best.pt tracks the lowest single-step training loss and best_average.pt the lowest running average — note that single-step diffusion loss is noisy, so final.pt is generally the checkpoint to use.