wandb/debug.log

2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Configure stats pid to 4029635
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Loading settings from /home/x_fahkh/.config/wandb/settings
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Loading settings from /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/wandb/settings
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Loading settings from environment variables: {'mode': 'offline'}
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Applying setup settings: {'_disable_service': False}
2026-02-12 22:39:56,451 WARNING MainThread:4029635 [wandb_setup.py:_flush():77] Could not find program at -m main
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m main'}
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:_log_setup():524] Logging user logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/offline-run-20260212_223956-ot7jqmgn/logs/debug.log
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:_log_setup():525] Logging internal logs to /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/offline-run-20260212_223956-ot7jqmgn/logs/debug-internal.log
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:init():608] calling init triggers
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
config: {'experiment': {'debug': '${debug}', 'tasks': ['training'], 'num_nodes': 1, 'training': {'precision': '16-mixed', 'compile': False, 'lr': 4e-05, 'batch_size': 4, 'max_epochs': -1, 'max_steps': 350000, 'max_time': None, 'data': {'num_workers': 4, 'shuffle': True}, 'optim': {'accumulate_grad_batches': 1, 'gradient_clip_val': 1.0}, 'checkpointing': {'every_n_train_steps': 2500, 'every_n_epochs': None, 'train_time_interval': None, 'enable_version_counter': False}}, 'validation': {'precision': '16-mixed', 'compile': False, 'batch_size': 4, 'val_every_n_step': 2500, 'val_every_n_epoch': None, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'test': {'precision': '16-mixed', 'compile': False, 'batch_size': 1, 'limit_batch': 1, 'inference_mode': False, 'data': {'num_workers': 4, 'shuffle': False}}, 'logging': {'metrics': None}, '_name': 'exp_video'}, 'dataset': {'debug': '${debug}', 'metadata': 'data/${dataset.name}/metadata.json', 'data_mean': 0.5, 'data_std': 0.5, 'save_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/worldmem_minecraft_dataset', 'n_frames': 8, 'context_length': 1, 'resolution': 128, 'observation_shape': [3, '${dataset.resolution}', '${dataset.resolution}'], 'external_cond_dim': 0, 'validation_multiplier': 1, 'frame_skip': 1, 'action_cond_dim': 25, '_name': 'video_minecraft', 'n_frames_valid': 700, 'angle_range': 110, 'pos_range': 8, 'memory_condition_length': 8, 'customized_validation': True, 'add_timestamp_embedding': True, 'wo_updown': False}, 'algorithm': {'debug': '${debug}', 'lr': '${experiment.training.lr}', 'x_shape': '${dataset.observation_shape}', 'frame_stack': 1, 'frame_skip': '${dataset.frame_skip}', 'data_mean': '${dataset.data_mean}', 'data_std': '${dataset.data_std}', 'external_cond_dim': 0, 'context_frames': 600, 'weight_decay': 0.002, 'warmup_steps': 1000, 'optimizer_beta': [0.9, 0.99], 'uncertainty_scale': 1, 'guidance_scale': 0.0, 'chunk_size': 1, 'scheduling_matrix': 'autoregressive', 'noise_level': 'random_all', 'causal': True, 'diffusion': {'objective': 'pred_v', 'beta_schedule': 'sigmoid', 'schedule_fn_kwargs': {}, 'clip_noise': 20.0, 'use_snr': False, 'use_cum_snr': False, 'use_fused_snr': True, 'snr_clip': 5.0, 'cum_snr_decay': 0.96, 'timesteps': 1000, 'sampling_timesteps': 20, 'ddim_sampling_eta': 0.0, 'stabilization_level': 15, 'architecture': {'network_size': 64, 'attn_heads': 4, 'attn_dim_head': 64, 'dim_mults': [1, 2, 4, 8], 'resolution': '${dataset.resolution}', 'attn_resolutions': [16, 32, 64, 128], 'use_init_temporal_attn': True, 'use_linear_attn': True, 'time_emb_type': 'rotary'}}, 'n_frames': '${dataset.n_frames}', 'metadata': '${dataset.metadata}', 'action_cond_dim': 25, 'use_plucker': True, '_name': 'df_video_worldmemminecraft', 'n_tokens': 8, 'memory_condition_length': 8, 'relative_embedding': True, 'log_video': True, 'add_timestamp_embedding': True, 'metrics': ['lpips', 'psnr']}, 'debug': False, 'wandb': {'entity': 'turlin', 'project': 'worldmem', 'mode': 'online'}, 'resume': 'ot7jqmgn', 'load': None, 'name': 'train', 'output_dir': '/proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set'}
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:init():658] starting backend
2026-02-12 22:39:56,451 INFO    MainThread:4029635 [wandb_init.py:init():662] setting up manager
2026-02-12 22:39:56,469 INFO    MainThread:4029635 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-02-12 22:39:56,474 INFO    MainThread:4029635 [wandb_init.py:init():670] backend started and connected
2026-02-12 22:39:56,505 INFO    MainThread:4029635 [wandb_init.py:init():768] updated telemetry
2026-02-12 22:39:56,540 INFO    MainThread:4029635 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
2026-02-12 22:39:56,545 INFO    MainThread:4029635 [wandb_init.py:init():852] starting run threads in backend
2026-02-12 22:40:02,604 INFO    MainThread:4029635 [wandb_run.py:_console_start():2465] atexit reg
2026-02-12 22:40:02,604 INFO    MainThread:4029635 [wandb_run.py:_redirect():2311] redirect: wrap_raw
2026-02-12 22:40:02,604 INFO    MainThread:4029635 [wandb_run.py:_redirect():2376] Wrapping output streams.
2026-02-12 22:40:02,604 INFO    MainThread:4029635 [wandb_run.py:_redirect():2401] Redirects installed.
2026-02-12 22:40:02,611 INFO    MainThread:4029635 [wandb_init.py:init():895] run started, returning control to user process