sync run artifacts: vjepa/inflated/d002_full_ema_capilvd1689m
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- vjepa/inflated/d002_full_ema_capilvd1689m/best.pt +3 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/git-info.txt +2 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_0_log.err +44 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_0_log.out +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_10_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_10_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_11_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_11_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_12_log.err +18 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_12_log.out +237 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_13_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_13_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_14_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_14_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_15_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_15_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_1_log.err +15 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_1_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_2_log.err +15 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_2_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_3_log.err +15 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_3_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_4_log.err +17 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_4_log.out +237 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_5_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_5_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_6_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_6_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_7_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_7_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_8_log.err +18 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_8_log.out +237 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_9_log.err +9 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_9_log.out +233 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/latest.pt +3 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r0.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r1.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r10.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r11.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r12.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r13.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r14.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r15.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r2.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r3.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r4.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r5.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r6.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r7.csv +0 -0
- vjepa/inflated/d002_full_ema_capilvd1689m/log_r8.csv +0 -0
vjepa/inflated/d002_full_ema_capilvd1689m/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:baba94fb2d4fd19a74318efd7123f92159956106cbf76c4736cfe858a9a2a658
|
| 3 |
+
size 4980091937
|
vjepa/inflated/d002_full_ema_capilvd1689m/git-info.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
branch: main
|
| 2 |
+
commit: d4d3e1b24c8e24ad1e228d40bcc17b5b6cc259c3
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_0_log.err
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 4 |
+
return func(*args, **kwargs)
|
| 5 |
+
[rank0]:[W509 10:41:45.428498606 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 6 |
+
wandb: Currently logged in as: dgcnz (uvjepa) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 7 |
+
wandb: setting up run xrsqmx9e
|
| 8 |
+
wandb: Tracking run with wandb version 0.23.1
|
| 9 |
+
wandb: Run data is saved locally in /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/wandb/run-20260509_104151-xrsqmx9e
|
| 10 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 11 |
+
wandb: Syncing run d002_full_ema_capilvd1689m
|
| 12 |
+
wandb: ⭐️ View project at https://wandb.ai/uvjepa/vjepa_ablation
|
| 13 |
+
wandb: 🚀 View run at https://wandb.ai/uvjepa/vjepa_ablation/runs/xrsqmx9e
|
| 14 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 15 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 16 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 17 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 18 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 19 |
+
return func(*args, **kwargs)
|
| 20 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 21 |
+
return func(*args, **kwargs)
|
| 22 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 23 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 24 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 25 |
+
return func(*args, **kwargs)
|
| 26 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 27 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 28 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 29 |
+
return func(*args, **kwargs)
|
| 30 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 31 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 32 |
+
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
|
| 33 |
+
[2026-05-09T15:57:55.746] error: *** JOB 22600912 ON gcn73 CANCELLED AT 2026-05-09T15:57:55 DUE to SIGNAL Terminated ***
|
| 34 |
+
[2026-05-09T15:57:55.746] error: *** STEP 22600912.0 ON gcn73 CANCELLED AT 2026-05-09T15:57:55 DUE to SIGNAL Terminated ***
|
| 35 |
+
submitit WARNING (2026-05-09 15:57:56,324) - Bypassing signal SIGTERM
|
| 36 |
+
submitit WARNING (2026-05-09 15:57:56,328) - Bypassing signal SIGCONT
|
| 37 |
+
[2026-05-09T15:58:36.346] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 38 |
+
[2026-05-09T15:58:36.346] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 39 |
+
[2026-05-09T15:58:36.488] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 40 |
+
[2026-05-09T15:58:36.488] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 41 |
+
[2026-05-09T15:58:36.528] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 42 |
+
[2026-05-09T15:58:36.528] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 43 |
+
[2026-05-09T15:58:36.566] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 44 |
+
[2026-05-09T15:58:36.566] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_0_log.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_10_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank10]:[W509 10:40:54.619863448 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_10_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn76.local.snellius.surf.nl, local_rank=2(4), node=2(4), global_rank=10(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 10/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:02][root ][_stage_targz_parts ] [local_rank 2] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:16][root ][_stage_targz_parts ] [local_rank 2] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:30][root ][_stage_targz_parts ] [local_rank 2] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:45][root ][_stage_targz_parts ] [local_rank 2] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:59][root ][_stage_targz_parts ] [local_rank 2] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:13][root ][_stage_targz_parts ] [local_rank 2] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:27][root ][_stage_targz_parts ] [local_rank 2] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:41][root ][_stage_targz_parts ] [local_rank 2] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:56][root ][_stage_targz_parts ] [local_rank 2] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:10][root ][_stage_targz_parts ] [local_rank 2] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:24][root ][_stage_targz_parts ] [local_rank 2] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:38][root ][_stage_targz_parts ] [local_rank 2] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:54][root ][_stage_targz_parts ] [local_rank 2] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:54][root ][stage_datasets ] [local_rank 2/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:26][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:42:32][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 10 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] Wrapping models in DDP (rank 10)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_11_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank11]:[W509 10:40:48.438877621 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,283) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,284) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_11_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,470) - Starting with JobEnvironment(job_id=22600912, hostname=gcn76.local.snellius.surf.nl, local_rank=3(4), node=2(4), global_rank=11(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 11/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:03][root ][_stage_targz_parts ] [local_rank 3] Extracted 2/25 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:17][root ][_stage_targz_parts ] [local_rank 3] Extracted 4/25 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:32][root ][_stage_targz_parts ] [local_rank 3] Extracted 6/25 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:46][root ][_stage_targz_parts ] [local_rank 3] Extracted 8/25 parts
|
| 115 |
+
[INFO ][2026-05-09 10:39:00][root ][_stage_targz_parts ] [local_rank 3] Extracted 10/25 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:15][root ][_stage_targz_parts ] [local_rank 3] Extracted 12/25 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:29][root ][_stage_targz_parts ] [local_rank 3] Extracted 14/25 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:43][root ][_stage_targz_parts ] [local_rank 3] Extracted 16/25 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:57][root ][_stage_targz_parts ] [local_rank 3] Extracted 18/25 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:11][root ][_stage_targz_parts ] [local_rank 3] Extracted 20/25 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:26][root ][_stage_targz_parts ] [local_rank 3] Extracted 22/25 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:40][root ][_stage_targz_parts ] [local_rank 3] Extracted 24/25 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:47][root ][_stage_targz_parts ] [local_rank 3] Extracted 25/25 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:47][root ][stage_datasets ] [local_rank 3/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:25][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:42:32][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 11 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] Wrapping models in DDP (rank 11)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,283) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,284) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_12_log.err
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank12]:[W509 10:41:47.358521314 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 10 |
+
[2026-05-09T15:58:30.021] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 11 |
+
[2026-05-09T15:58:30.021] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 12 |
+
[2026-05-09T15:58:31.666] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
|
| 13 |
+
[2026-05-09T15:58:31.669] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 14 |
+
[2026-05-09T15:58:31.669] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 15 |
+
[2026-05-09T15:58:31.809] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 16 |
+
[2026-05-09T15:58:31.809] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 17 |
+
[2026-05-09T15:58:31.849] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 18 |
+
[2026-05-09T15:58:31.849] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_12_log.out
ADDED
|
@@ -0,0 +1,237 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,470) - Starting with JobEnvironment(job_id=22600912, hostname=gcn78.local.snellius.surf.nl, local_rank=0(4), node=3(4), global_rank=12(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 12/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:02][root ][_stage_targz_parts ] [local_rank 0] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:16][root ][_stage_targz_parts ] [local_rank 0] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:30][root ][_stage_targz_parts ] [local_rank 0] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:44][root ][_stage_targz_parts ] [local_rank 0] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:58][root ][_stage_targz_parts ] [local_rank 0] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:12][root ][_stage_targz_parts ] [local_rank 0] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:26][root ][_stage_targz_parts ] [local_rank 0] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:40][root ][_stage_targz_parts ] [local_rank 0] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:55][root ][_stage_targz_parts ] [local_rank 0] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:09][root ][_stage_targz_parts ] [local_rank 0] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:23][root ][_stage_targz_parts ] [local_rank 0] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:37][root ][_stage_targz_parts ] [local_rank 0] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:51][root ][_stage_targz_parts ] [local_rank 0] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:51][root ][stage_datasets ] [local_rank 0/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:40:51][root ][_stage_multipart_tar ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22600912/ssv2
|
| 126 |
+
[INFO ][2026-05-09 10:41:49][root ][stage_datasets ] Data staging completed in 240.7s (4.0min)
|
| 127 |
+
[INFO ][2026-05-09 10:41:49][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/kinetics_240/train.csv (239789 entries)
|
| 128 |
+
[INFO ][2026-05-09 10:41:49][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/ssv2/train.csv (168913 entries)
|
| 129 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 130 |
+
[INFO ][2026-05-09 10:42:56][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 131 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 132 |
+
(backbone): VisionTransformer(
|
| 133 |
+
(patch_embed): PatchEmbed3D(
|
| 134 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 135 |
+
)
|
| 136 |
+
(rope): CAPI2DRoPE()
|
| 137 |
+
(blocks): ModuleList(
|
| 138 |
+
(0-23): 24 x Block(
|
| 139 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 140 |
+
(rope_impl): CAPI2DRoPE()
|
| 141 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 142 |
+
(drop_path1): Identity()
|
| 143 |
+
(drop_path2): Identity()
|
| 144 |
+
(attn): Attention(
|
| 145 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 146 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 147 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 148 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 149 |
+
(rope_impl): CAPI2DRoPE()
|
| 150 |
+
)
|
| 151 |
+
(mlp): MLP(
|
| 152 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 153 |
+
(act): GELU(approximate='none')
|
| 154 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 155 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
)
|
| 159 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 160 |
+
)
|
| 161 |
+
)
|
| 162 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 163 |
+
(backbone): VisionTransformerPredictor(
|
| 164 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 165 |
+
(mask_tokens): ParameterList(
|
| 166 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 167 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 168 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 169 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 170 |
+
)
|
| 171 |
+
(predictor_blocks): ModuleList(
|
| 172 |
+
(0-5): 6 x Block(
|
| 173 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 174 |
+
(attn): RoPEAttention(
|
| 175 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 176 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 177 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 178 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 179 |
+
)
|
| 180 |
+
(drop_path): Identity()
|
| 181 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 182 |
+
(mlp): MLP(
|
| 183 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 184 |
+
(act): GELU(approximate='none')
|
| 185 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 186 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 187 |
+
)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 191 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 192 |
+
)
|
| 193 |
+
)
|
| 194 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 195 |
+
(backbone): VisionTransformer(
|
| 196 |
+
(patch_embed): PatchEmbed3D(
|
| 197 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 198 |
+
)
|
| 199 |
+
(rope): CAPI2DRoPE()
|
| 200 |
+
(blocks): ModuleList(
|
| 201 |
+
(0-23): 24 x Block(
|
| 202 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 203 |
+
(rope_impl): CAPI2DRoPE()
|
| 204 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 205 |
+
(drop_path1): Identity()
|
| 206 |
+
(drop_path2): Identity()
|
| 207 |
+
(attn): Attention(
|
| 208 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 209 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 210 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 211 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 212 |
+
(rope_impl): CAPI2DRoPE()
|
| 213 |
+
)
|
| 214 |
+
(mlp): MLP(
|
| 215 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 216 |
+
(act): GELU(approximate='none')
|
| 217 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 218 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
)
|
| 222 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 223 |
+
)
|
| 224 |
+
)
|
| 225 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 226 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 227 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 229 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 12 / 16
|
| 230 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 231 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 232 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 12)...
|
| 233 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 234 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGTERM
|
| 235 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 236 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 237 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_13_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank13]:[W509 10:40:50.097716313 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_13_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn78.local.snellius.surf.nl, local_rank=1(4), node=3(4), global_rank=13(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 13/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:01][root ][_stage_targz_parts ] [local_rank 1] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:15][root ][_stage_targz_parts ] [local_rank 1] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:29][root ][_stage_targz_parts ] [local_rank 1] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:43][root ][_stage_targz_parts ] [local_rank 1] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:57][root ][_stage_targz_parts ] [local_rank 1] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:12][root ][_stage_targz_parts ] [local_rank 1] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:25][root ][_stage_targz_parts ] [local_rank 1] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:39][root ][_stage_targz_parts ] [local_rank 1] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:53][root ][_stage_targz_parts ] [local_rank 1] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:08][root ][_stage_targz_parts ] [local_rank 1] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:22][root ][_stage_targz_parts ] [local_rank 1] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:36][root ][_stage_targz_parts ] [local_rank 1] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:50][root ][_stage_targz_parts ] [local_rank 1] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:50][root ][stage_datasets ] [local_rank 1/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:56][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 13 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 13)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_14_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank14]:[W509 10:40:54.317299056 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,354) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,355) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_14_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,470) - Starting with JobEnvironment(job_id=22600912, hostname=gcn78.local.snellius.surf.nl, local_rank=2(4), node=3(4), global_rank=14(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:50][app.vjepa.train ][main ] Initialized (rank/world-size) 14/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:50][root ][stage_datasets ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:50][root ][_stage_targz_parts ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:04][root ][_stage_targz_parts ] [local_rank 2] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:17][root ][_stage_targz_parts ] [local_rank 2] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:32][root ][_stage_targz_parts ] [local_rank 2] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:46][root ][_stage_targz_parts ] [local_rank 2] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:39:01][root ][_stage_targz_parts ] [local_rank 2] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:14][root ][_stage_targz_parts ] [local_rank 2] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:28][root ][_stage_targz_parts ] [local_rank 2] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:42][root ][_stage_targz_parts ] [local_rank 2] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:57][root ][_stage_targz_parts ] [local_rank 2] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:11][root ][_stage_targz_parts ] [local_rank 2] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:25][root ][_stage_targz_parts ] [local_rank 2] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:39][root ][_stage_targz_parts ] [local_rank 2] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:54][root ][_stage_targz_parts ] [local_rank 2] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:54][root ][stage_datasets ] [local_rank 2/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:56][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 14 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 14)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,354) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,355) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_15_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank15]:[W509 10:40:47.755868197 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,305) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,305) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_15_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn78.local.snellius.surf.nl, local_rank=3(4), node=3(4), global_rank=15(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:49][app.vjepa.train ][main ] Initialized (rank/world-size) 15/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:49][root ][stage_datasets ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:49][root ][_stage_targz_parts ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:03][root ][_stage_targz_parts ] [local_rank 3] Extracted 2/25 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:17][root ][_stage_targz_parts ] [local_rank 3] Extracted 4/25 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:32][root ][_stage_targz_parts ] [local_rank 3] Extracted 6/25 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:46][root ][_stage_targz_parts ] [local_rank 3] Extracted 8/25 parts
|
| 115 |
+
[INFO ][2026-05-09 10:39:00][root ][_stage_targz_parts ] [local_rank 3] Extracted 10/25 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:14][root ][_stage_targz_parts ] [local_rank 3] Extracted 12/25 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:29][root ][_stage_targz_parts ] [local_rank 3] Extracted 14/25 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:43][root ][_stage_targz_parts ] [local_rank 3] Extracted 16/25 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:57][root ][_stage_targz_parts ] [local_rank 3] Extracted 18/25 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:12][root ][_stage_targz_parts ] [local_rank 3] Extracted 20/25 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:26][root ][_stage_targz_parts ] [local_rank 3] Extracted 22/25 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:40][root ][_stage_targz_parts ] [local_rank 3] Extracted 24/25 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:47][root ][_stage_targz_parts ] [local_rank 3] Extracted 25/25 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:47][root ][stage_datasets ] [local_rank 3/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:56][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:00][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 15 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 15)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,305) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,305) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_1_log.err
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank1]:[W509 10:40:18.301886131 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 9 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 10 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 11 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 12 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 13 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 14 |
+
submitit WARNING (2026-05-09 15:57:56,322) - Bypassing signal SIGTERM
|
| 15 |
+
submitit WARNING (2026-05-09 15:57:56,336) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_1_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:36:57,165) - Starting with JobEnvironment(job_id=22600912, hostname=gcn73.local.snellius.surf.nl, local_rank=1(4), node=0(4), global_rank=1(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:36:57,476) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:12][app.vjepa.train ][main ] Initialized (rank/world-size) 1/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:14][root ][stage_datasets ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:14][root ][_stage_targz_parts ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:37:27][root ][_stage_targz_parts ] [local_rank 1] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:37:40][root ][_stage_targz_parts ] [local_rank 1] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:37:54][root ][_stage_targz_parts ] [local_rank 1] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:09][root ][_stage_targz_parts ] [local_rank 1] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:23][root ][_stage_targz_parts ] [local_rank 1] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:38:38][root ][_stage_targz_parts ] [local_rank 1] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:38:51][root ][_stage_targz_parts ] [local_rank 1] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:05][root ][_stage_targz_parts ] [local_rank 1] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:19][root ][_stage_targz_parts ] [local_rank 1] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:39:34][root ][_stage_targz_parts ] [local_rank 1] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:39:49][root ][_stage_targz_parts ] [local_rank 1] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:03][root ][_stage_targz_parts ] [local_rank 1] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:17][root ][_stage_targz_parts ] [local_rank 1] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:17][root ][stage_datasets ] [local_rank 1/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:53][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:44:00][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 1 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] Wrapping models in DDP (rank 1)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,322) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,336) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_2_log.err
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank2]:[W509 10:40:20.797663599 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 9 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 10 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 11 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 12 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 13 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 14 |
+
submitit WARNING (2026-05-09 15:57:56,359) - Bypassing signal SIGTERM
|
| 15 |
+
submitit WARNING (2026-05-09 15:57:56,507) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_2_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:36:57,168) - Starting with JobEnvironment(job_id=22600912, hostname=gcn73.local.snellius.surf.nl, local_rank=2(4), node=0(4), global_rank=2(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:36:57,459) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:13][app.vjepa.train ][main ] Initialized (rank/world-size) 2/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:14][root ][stage_datasets ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:14][root ][_stage_targz_parts ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:37:28][root ][_stage_targz_parts ] [local_rank 2] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:37:42][root ][_stage_targz_parts ] [local_rank 2] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:37:56][root ][_stage_targz_parts ] [local_rank 2] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:11][root ][_stage_targz_parts ] [local_rank 2] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:25][root ][_stage_targz_parts ] [local_rank 2] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:38:40][root ][_stage_targz_parts ] [local_rank 2] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:38:54][root ][_stage_targz_parts ] [local_rank 2] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:08][root ][_stage_targz_parts ] [local_rank 2] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:23][root ][_stage_targz_parts ] [local_rank 2] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:39:37][root ][_stage_targz_parts ] [local_rank 2] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:39:51][root ][_stage_targz_parts ] [local_rank 2] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:06][root ][_stage_targz_parts ] [local_rank 2] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:20][root ][_stage_targz_parts ] [local_rank 2] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:20][root ][stage_datasets ] [local_rank 2/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:54][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:44:00][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 2 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] Wrapping models in DDP (rank 2)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,359) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,507) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_3_log.err
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank3]:[W509 10:40:14.220969980 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 9 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 10 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 11 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 12 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 13 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 14 |
+
submitit WARNING (2026-05-09 15:57:56,308) - Bypassing signal SIGTERM
|
| 15 |
+
submitit WARNING (2026-05-09 15:57:56,321) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_3_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:36:57,165) - Starting with JobEnvironment(job_id=22600912, hostname=gcn73.local.snellius.surf.nl, local_rank=3(4), node=0(4), global_rank=3(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:36:57,239) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:10][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:12][app.vjepa.train ][main ] Initialized (rank/world-size) 3/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:14][root ][stage_datasets ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:14][root ][_stage_targz_parts ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:37:28][root ][_stage_targz_parts ] [local_rank 3] Extracted 2/25 parts
|
| 112 |
+
[INFO ][2026-05-09 10:37:42][root ][_stage_targz_parts ] [local_rank 3] Extracted 4/25 parts
|
| 113 |
+
[INFO ][2026-05-09 10:37:57][root ][_stage_targz_parts ] [local_rank 3] Extracted 6/25 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:11][root ][_stage_targz_parts ] [local_rank 3] Extracted 8/25 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:26][root ][_stage_targz_parts ] [local_rank 3] Extracted 10/25 parts
|
| 116 |
+
[INFO ][2026-05-09 10:38:40][root ][_stage_targz_parts ] [local_rank 3] Extracted 12/25 parts
|
| 117 |
+
[INFO ][2026-05-09 10:38:55][root ][_stage_targz_parts ] [local_rank 3] Extracted 14/25 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:09][root ][_stage_targz_parts ] [local_rank 3] Extracted 16/25 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:23][root ][_stage_targz_parts ] [local_rank 3] Extracted 18/25 parts
|
| 120 |
+
[INFO ][2026-05-09 10:39:37][root ][_stage_targz_parts ] [local_rank 3] Extracted 20/25 parts
|
| 121 |
+
[INFO ][2026-05-09 10:39:52][root ][_stage_targz_parts ] [local_rank 3] Extracted 22/25 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:06][root ][_stage_targz_parts ] [local_rank 3] Extracted 24/25 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:13][root ][_stage_targz_parts ] [local_rank 3] Extracted 25/25 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:13][root ][stage_datasets ] [local_rank 3/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:53][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:58][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:44:00][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 3 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:44:00][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:44:00][app.vjepa.train ][main ] Wrapping models in DDP (rank 3)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,308) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,321) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_4_log.err
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank4]:[W509 10:41:47.693174896 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,356) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGCONT
|
| 10 |
+
[2026-05-09T15:58:32.925] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 11 |
+
[2026-05-09T15:58:32.925] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 12 |
+
[2026-05-09T15:58:33.070] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 13 |
+
[2026-05-09T15:58:33.070] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 14 |
+
[2026-05-09T15:58:33.108] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 15 |
+
[2026-05-09T15:58:33.108] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 16 |
+
[2026-05-09T15:58:33.145] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 17 |
+
[2026-05-09T15:58:33.145] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_4_log.out
ADDED
|
@@ -0,0 +1,237 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn75.local.snellius.surf.nl, local_rank=0(4), node=1(4), global_rank=4(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 4/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:03][root ][_stage_targz_parts ] [local_rank 0] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:16][root ][_stage_targz_parts ] [local_rank 0] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:30][root ][_stage_targz_parts ] [local_rank 0] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:44][root ][_stage_targz_parts ] [local_rank 0] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:58][root ][_stage_targz_parts ] [local_rank 0] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:12][root ][_stage_targz_parts ] [local_rank 0] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:26][root ][_stage_targz_parts ] [local_rank 0] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:41][root ][_stage_targz_parts ] [local_rank 0] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:55][root ][_stage_targz_parts ] [local_rank 0] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:09][root ][_stage_targz_parts ] [local_rank 0] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:23][root ][_stage_targz_parts ] [local_rank 0] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:37][root ][_stage_targz_parts ] [local_rank 0] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:51][root ][_stage_targz_parts ] [local_rank 0] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:51][root ][stage_datasets ] [local_rank 0/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:40:51][root ][_stage_multipart_tar ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22600912/ssv2
|
| 126 |
+
[INFO ][2026-05-09 10:41:49][root ][stage_datasets ] Data staging completed in 240.3s (4.0min)
|
| 127 |
+
[INFO ][2026-05-09 10:41:49][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/kinetics_240/train.csv (239789 entries)
|
| 128 |
+
[INFO ][2026-05-09 10:41:49][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/ssv2/train.csv (168913 entries)
|
| 129 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 130 |
+
[INFO ][2026-05-09 10:43:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 131 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 132 |
+
(backbone): VisionTransformer(
|
| 133 |
+
(patch_embed): PatchEmbed3D(
|
| 134 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 135 |
+
)
|
| 136 |
+
(rope): CAPI2DRoPE()
|
| 137 |
+
(blocks): ModuleList(
|
| 138 |
+
(0-23): 24 x Block(
|
| 139 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 140 |
+
(rope_impl): CAPI2DRoPE()
|
| 141 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 142 |
+
(drop_path1): Identity()
|
| 143 |
+
(drop_path2): Identity()
|
| 144 |
+
(attn): Attention(
|
| 145 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 146 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 147 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 148 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 149 |
+
(rope_impl): CAPI2DRoPE()
|
| 150 |
+
)
|
| 151 |
+
(mlp): MLP(
|
| 152 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 153 |
+
(act): GELU(approximate='none')
|
| 154 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 155 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
)
|
| 159 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 160 |
+
)
|
| 161 |
+
)
|
| 162 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 163 |
+
(backbone): VisionTransformerPredictor(
|
| 164 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 165 |
+
(mask_tokens): ParameterList(
|
| 166 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 167 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 168 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 169 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 170 |
+
)
|
| 171 |
+
(predictor_blocks): ModuleList(
|
| 172 |
+
(0-5): 6 x Block(
|
| 173 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 174 |
+
(attn): RoPEAttention(
|
| 175 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 176 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 177 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 178 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 179 |
+
)
|
| 180 |
+
(drop_path): Identity()
|
| 181 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 182 |
+
(mlp): MLP(
|
| 183 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 184 |
+
(act): GELU(approximate='none')
|
| 185 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 186 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 187 |
+
)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 191 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 192 |
+
)
|
| 193 |
+
)
|
| 194 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 195 |
+
(backbone): VisionTransformer(
|
| 196 |
+
(patch_embed): PatchEmbed3D(
|
| 197 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 198 |
+
)
|
| 199 |
+
(rope): CAPI2DRoPE()
|
| 200 |
+
(blocks): ModuleList(
|
| 201 |
+
(0-23): 24 x Block(
|
| 202 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 203 |
+
(rope_impl): CAPI2DRoPE()
|
| 204 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 205 |
+
(drop_path1): Identity()
|
| 206 |
+
(drop_path2): Identity()
|
| 207 |
+
(attn): Attention(
|
| 208 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 209 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 210 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 211 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 212 |
+
(rope_impl): CAPI2DRoPE()
|
| 213 |
+
)
|
| 214 |
+
(mlp): MLP(
|
| 215 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 216 |
+
(act): GELU(approximate='none')
|
| 217 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 218 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
)
|
| 222 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 223 |
+
)
|
| 224 |
+
)
|
| 225 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 226 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 227 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 229 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 4 / 16
|
| 230 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 231 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 232 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 4)...
|
| 233 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 234 |
+
submitit WARNING (2026-05-09 15:57:56,356) - Bypassing signal SIGTERM
|
| 235 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 236 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGCONT
|
| 237 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_5_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank5]:[W509 10:40:51.666593645 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,358) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_5_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,470) - Starting with JobEnvironment(job_id=22600912, hostname=gcn75.local.snellius.surf.nl, local_rank=1(4), node=1(4), global_rank=5(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:50][app.vjepa.train ][main ] Initialized (rank/world-size) 5/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:50][root ][stage_datasets ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:50][root ][_stage_targz_parts ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:02][root ][_stage_targz_parts ] [local_rank 1] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:16][root ][_stage_targz_parts ] [local_rank 1] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:30][root ][_stage_targz_parts ] [local_rank 1] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:44][root ][_stage_targz_parts ] [local_rank 1] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:58][root ][_stage_targz_parts ] [local_rank 1] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:12][root ][_stage_targz_parts ] [local_rank 1] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:25][root ][_stage_targz_parts ] [local_rank 1] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:39][root ][_stage_targz_parts ] [local_rank 1] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:53][root ][_stage_targz_parts ] [local_rank 1] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:08][root ][_stage_targz_parts ] [local_rank 1] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:22][root ][_stage_targz_parts ] [local_rank 1] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:37][root ][_stage_targz_parts ] [local_rank 1] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:51][root ][_stage_targz_parts ] [local_rank 1] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:51][root ][stage_datasets ] [local_rank 1/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 5 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 5)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,358) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_6_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank6]:[W509 10:40:54.505715988 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,358) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_6_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn75.local.snellius.surf.nl, local_rank=2(4), node=1(4), global_rank=6(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:48][app.vjepa.train ][main ] Initialized (rank/world-size) 6/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:48][root ][stage_datasets ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:48][root ][_stage_targz_parts ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:02][root ][_stage_targz_parts ] [local_rank 2] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:16][root ][_stage_targz_parts ] [local_rank 2] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:30][root ][_stage_targz_parts ] [local_rank 2] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:45][root ][_stage_targz_parts ] [local_rank 2] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:59][root ][_stage_targz_parts ] [local_rank 2] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:13][root ][_stage_targz_parts ] [local_rank 2] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:27][root ][_stage_targz_parts ] [local_rank 2] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:41][root ][_stage_targz_parts ] [local_rank 2] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:56][root ][_stage_targz_parts ] [local_rank 2] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:10][root ][_stage_targz_parts ] [local_rank 2] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:24][root ][_stage_targz_parts ] [local_rank 2] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:38][root ][_stage_targz_parts ] [local_rank 2] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:54][root ][_stage_targz_parts ] [local_rank 2] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:54][root ][stage_datasets ] [local_rank 2/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 6 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 6)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,357) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,358) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_7_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank7]:[W509 10:40:47.995090115 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,353) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,354) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_7_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn75.local.snellius.surf.nl, local_rank=3(4), node=1(4), global_rank=7(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:49][app.vjepa.train ][main ] Initialized (rank/world-size) 7/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:49][root ][stage_datasets ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:49][root ][_stage_targz_parts ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:03][root ][_stage_targz_parts ] [local_rank 3] Extracted 2/25 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:17][root ][_stage_targz_parts ] [local_rank 3] Extracted 4/25 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:32][root ][_stage_targz_parts ] [local_rank 3] Extracted 6/25 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:46][root ][_stage_targz_parts ] [local_rank 3] Extracted 8/25 parts
|
| 115 |
+
[INFO ][2026-05-09 10:39:00][root ][_stage_targz_parts ] [local_rank 3] Extracted 10/25 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:15][root ][_stage_targz_parts ] [local_rank 3] Extracted 12/25 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:29][root ][_stage_targz_parts ] [local_rank 3] Extracted 14/25 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:43][root ][_stage_targz_parts ] [local_rank 3] Extracted 16/25 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:57][root ][_stage_targz_parts ] [local_rank 3] Extracted 18/25 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:12][root ][_stage_targz_parts ] [local_rank 3] Extracted 20/25 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:26][root ][_stage_targz_parts ] [local_rank 3] Extracted 22/25 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:40][root ][_stage_targz_parts ] [local_rank 3] Extracted 24/25 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:47][root ][_stage_targz_parts ] [local_rank 3] Extracted 25/25 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:47][root ][stage_datasets ] [local_rank 3/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:43:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:43:02][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:43:04][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 7 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:43:04][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:43:04][app.vjepa.train ][main ] Wrapping models in DDP (rank 7)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,353) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,354) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_8_log.err
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank8]:[W509 10:41:47.223112924 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 10 |
+
[2026-05-09T15:58:28.011] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 11 |
+
[2026-05-09T15:58:28.011] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 12 |
+
[2026-05-09T15:58:29.655] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
|
| 13 |
+
[2026-05-09T15:58:31.239] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 14 |
+
[2026-05-09T15:58:31.239] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 15 |
+
[2026-05-09T15:58:31.377] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 16 |
+
[2026-05-09T15:58:31.378] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
| 17 |
+
[2026-05-09T15:58:31.416] error: namespace_p_join: open failed for /slurm/22600912/.ns: No such file or directory
|
| 18 |
+
[2026-05-09T15:58:31.417] error: namespace_g_join(JobId=22600912 SLUID=s8FXKR5Y3Q4100): No such file or directory
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_8_log.out
ADDED
|
@@ -0,0 +1,237 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn76.local.snellius.surf.nl, local_rank=0(4), node=2(4), global_rank=8(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:50][app.vjepa.train ][main ] Initialized (rank/world-size) 8/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:50][root ][stage_datasets ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:50][root ][_stage_targz_parts ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:04][root ][_stage_targz_parts ] [local_rank 0] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:18][root ][_stage_targz_parts ] [local_rank 0] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:32][root ][_stage_targz_parts ] [local_rank 0] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:45][root ][_stage_targz_parts ] [local_rank 0] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:39:00][root ][_stage_targz_parts ] [local_rank 0] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:14][root ][_stage_targz_parts ] [local_rank 0] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:28][root ][_stage_targz_parts ] [local_rank 0] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:43][root ][_stage_targz_parts ] [local_rank 0] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:57][root ][_stage_targz_parts ] [local_rank 0] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:11][root ][_stage_targz_parts ] [local_rank 0] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:25][root ][_stage_targz_parts ] [local_rank 0] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:39][root ][_stage_targz_parts ] [local_rank 0] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:53][root ][_stage_targz_parts ] [local_rank 0] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:53][root ][stage_datasets ] [local_rank 0/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:40:53][root ][_stage_multipart_tar ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22600912/ssv2
|
| 126 |
+
[INFO ][2026-05-09 10:41:49][root ][stage_datasets ] Data staging completed in 239.0s (4.0min)
|
| 127 |
+
[INFO ][2026-05-09 10:41:49][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/kinetics_240/train.csv (239789 entries)
|
| 128 |
+
[INFO ][2026-05-09 10:41:50][root ][_rewrite_csv ] Wrote local CSV: /scratch-node/dcanez.22600912/ssv2/train.csv (168913 entries)
|
| 129 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 130 |
+
[INFO ][2026-05-09 10:42:25][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 131 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 132 |
+
(backbone): VisionTransformer(
|
| 133 |
+
(patch_embed): PatchEmbed3D(
|
| 134 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 135 |
+
)
|
| 136 |
+
(rope): CAPI2DRoPE()
|
| 137 |
+
(blocks): ModuleList(
|
| 138 |
+
(0-23): 24 x Block(
|
| 139 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 140 |
+
(rope_impl): CAPI2DRoPE()
|
| 141 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 142 |
+
(drop_path1): Identity()
|
| 143 |
+
(drop_path2): Identity()
|
| 144 |
+
(attn): Attention(
|
| 145 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 146 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 147 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 148 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 149 |
+
(rope_impl): CAPI2DRoPE()
|
| 150 |
+
)
|
| 151 |
+
(mlp): MLP(
|
| 152 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 153 |
+
(act): GELU(approximate='none')
|
| 154 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 155 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
)
|
| 159 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 160 |
+
)
|
| 161 |
+
)
|
| 162 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 163 |
+
(backbone): VisionTransformerPredictor(
|
| 164 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 165 |
+
(mask_tokens): ParameterList(
|
| 166 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 167 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 168 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 169 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 170 |
+
)
|
| 171 |
+
(predictor_blocks): ModuleList(
|
| 172 |
+
(0-5): 6 x Block(
|
| 173 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 174 |
+
(attn): RoPEAttention(
|
| 175 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 176 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 177 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 178 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 179 |
+
)
|
| 180 |
+
(drop_path): Identity()
|
| 181 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 182 |
+
(mlp): MLP(
|
| 183 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 184 |
+
(act): GELU(approximate='none')
|
| 185 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 186 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 187 |
+
)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 191 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 192 |
+
)
|
| 193 |
+
)
|
| 194 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 195 |
+
(backbone): VisionTransformer(
|
| 196 |
+
(patch_embed): PatchEmbed3D(
|
| 197 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 198 |
+
)
|
| 199 |
+
(rope): CAPI2DRoPE()
|
| 200 |
+
(blocks): ModuleList(
|
| 201 |
+
(0-23): 24 x Block(
|
| 202 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 203 |
+
(rope_impl): CAPI2DRoPE()
|
| 204 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 205 |
+
(drop_path1): Identity()
|
| 206 |
+
(drop_path2): Identity()
|
| 207 |
+
(attn): Attention(
|
| 208 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 209 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 210 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 211 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 212 |
+
(rope_impl): CAPI2DRoPE()
|
| 213 |
+
)
|
| 214 |
+
(mlp): MLP(
|
| 215 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 216 |
+
(act): GELU(approximate='none')
|
| 217 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 218 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
)
|
| 222 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 223 |
+
)
|
| 224 |
+
)
|
| 225 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 226 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 227 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 228 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset dataset created
|
| 229 |
+
[INFO ][2026-05-09 10:42:32][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 8 / 16
|
| 230 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 231 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 232 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] Wrapping models in DDP (rank 8)...
|
| 233 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 234 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGTERM
|
| 235 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 236 |
+
submitit WARNING (2026-05-09 15:57:56,352) - Bypassing signal SIGCONT
|
| 237 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_9_log.err
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
[rank9]:[W509 10:40:51.398884135 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
|
| 4 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 5 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 6 |
+
/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 7 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 8 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 9 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_9_log.out
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-05-09 10:37:00,469) - Starting with JobEnvironment(job_id=22600912, hostname=gcn76.local.snellius.surf.nl, local_rank=1(4), node=2(4), global_rank=9(16))
|
| 2 |
+
submitit INFO (2026-05-09 10:37:00,470) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m/job_22600912/22600912_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 16,
|
| 6 |
+
'data': { 'batch_size': 64,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
|
| 11 |
+
'/scratch-shared/dcanez/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 10,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'stage': [ { 'dest': 'kinetics_240',
|
| 19 |
+
'format': 'targz_parts',
|
| 20 |
+
'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
|
| 21 |
+
{ 'dest': 'ssv2',
|
| 22 |
+
'format': 'multipart_tar',
|
| 23 |
+
'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
|
| 24 |
+
'tubelet_size': 1},
|
| 25 |
+
'data_aug': { 'auto_augment': False,
|
| 26 |
+
'motion_shift': False,
|
| 27 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 28 |
+
'random_resize_scale': [0.3, 1.0],
|
| 29 |
+
'reprob': 0.0},
|
| 30 |
+
'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d002_full_ema_capilvd1689m',
|
| 31 |
+
'loss': {'loss_exp': 1.0},
|
| 32 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 33 |
+
'full_complement': False,
|
| 34 |
+
'max_keep': None,
|
| 35 |
+
'max_temporal_keep': 1.0,
|
| 36 |
+
'num_blocks': 8,
|
| 37 |
+
'spatial_scale': [0.15, 0.15],
|
| 38 |
+
'temporal_scale': [1.0, 1.0]},
|
| 39 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 40 |
+
'full_complement': False,
|
| 41 |
+
'max_keep': None,
|
| 42 |
+
'max_temporal_keep': 1.0,
|
| 43 |
+
'num_blocks': 2,
|
| 44 |
+
'spatial_scale': [0.7, 0.7],
|
| 45 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 46 |
+
'mem_per_gpu': '180G',
|
| 47 |
+
'meta': { 'dtype': 'bfloat16',
|
| 48 |
+
'knn_eval_epoch0': False,
|
| 49 |
+
'knn_eval_freq': 5,
|
| 50 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 64,
|
| 51 |
+
'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
|
| 52 |
+
'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
|
| 53 |
+
'eval_videos_per_class': 25,
|
| 54 |
+
'num_workers': 8,
|
| 55 |
+
'train_videos_per_class': 100},
|
| 56 |
+
'preset': 'ucf101'},
|
| 57 |
+
{ 'config': { 'batch_size': 64,
|
| 58 |
+
'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
|
| 59 |
+
'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
|
| 60 |
+
'eval_videos_per_class': 100,
|
| 61 |
+
'linear_probe': True,
|
| 62 |
+
'num_workers': 8,
|
| 63 |
+
'pool_type': 'temporal_concat',
|
| 64 |
+
'train_videos_per_class': 500},
|
| 65 |
+
'preset': 'ssv2_coarse10'}],
|
| 66 |
+
'load_checkpoint': True,
|
| 67 |
+
'read_checkpoint': None,
|
| 68 |
+
'save_every_freq': 5,
|
| 69 |
+
'seed': 239,
|
| 70 |
+
'use_sdpa': True,
|
| 71 |
+
'use_wandb': True,
|
| 72 |
+
'wandb_project': 'vjepa_ablation'},
|
| 73 |
+
'metrics': {'sigreg': {}, 'std': {}},
|
| 74 |
+
'model': { 'class_token': False,
|
| 75 |
+
'freeze_backbone': False,
|
| 76 |
+
'is_causal': False,
|
| 77 |
+
'model_name': 'vit_large_patch14_capi_lvd1689m',
|
| 78 |
+
'num_reg_tokens': 16,
|
| 79 |
+
'pred_depth': 6,
|
| 80 |
+
'pred_embed_dim': 384,
|
| 81 |
+
'pred_num_heads': 12,
|
| 82 |
+
'rope_type': 'capi2d',
|
| 83 |
+
'stem_type': '2d',
|
| 84 |
+
'target_kind': 'ema',
|
| 85 |
+
'uniform_power': True,
|
| 86 |
+
'use_activation_checkpointing': True,
|
| 87 |
+
'use_lora': False,
|
| 88 |
+
'use_mask_tokens': True,
|
| 89 |
+
'use_rope': True,
|
| 90 |
+
'use_sdpa': True,
|
| 91 |
+
'zero_init_mask_tokens': True},
|
| 92 |
+
'nodes': 4,
|
| 93 |
+
'optimization': { 'ema': [0.99925, 0.99925],
|
| 94 |
+
'epochs': 100,
|
| 95 |
+
'final_lr': 0.0001,
|
| 96 |
+
'final_weight_decay': 0.04,
|
| 97 |
+
'ipe': 300,
|
| 98 |
+
'ipe_scale': 1.0,
|
| 99 |
+
'lr': 0.0005,
|
| 100 |
+
'start_lr': 0.0001,
|
| 101 |
+
'warmup': 10,
|
| 102 |
+
'weight_decay': 0.04},
|
| 103 |
+
'tasks_per_node': 4}
|
| 104 |
+
INFO:root:Running pre-training of app: vjepa
|
| 105 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 106 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 107 |
+
[INFO ][2026-05-09 10:37:46][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=eno
|
| 108 |
+
[INFO ][2026-05-09 10:37:49][app.vjepa.train ][main ] Initialized (rank/world-size) 9/16, tasks_per_node=4
|
| 109 |
+
[INFO ][2026-05-09 10:37:49][root ][stage_datasets ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
|
| 110 |
+
[INFO ][2026-05-09 10:37:49][root ][_stage_targz_parts ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22600912/kinetics_240
|
| 111 |
+
[INFO ][2026-05-09 10:38:01][root ][_stage_targz_parts ] [local_rank 1] Extracted 2/26 parts
|
| 112 |
+
[INFO ][2026-05-09 10:38:15][root ][_stage_targz_parts ] [local_rank 1] Extracted 4/26 parts
|
| 113 |
+
[INFO ][2026-05-09 10:38:29][root ][_stage_targz_parts ] [local_rank 1] Extracted 6/26 parts
|
| 114 |
+
[INFO ][2026-05-09 10:38:43][root ][_stage_targz_parts ] [local_rank 1] Extracted 8/26 parts
|
| 115 |
+
[INFO ][2026-05-09 10:38:57][root ][_stage_targz_parts ] [local_rank 1] Extracted 10/26 parts
|
| 116 |
+
[INFO ][2026-05-09 10:39:12][root ][_stage_targz_parts ] [local_rank 1] Extracted 12/26 parts
|
| 117 |
+
[INFO ][2026-05-09 10:39:25][root ][_stage_targz_parts ] [local_rank 1] Extracted 14/26 parts
|
| 118 |
+
[INFO ][2026-05-09 10:39:39][root ][_stage_targz_parts ] [local_rank 1] Extracted 16/26 parts
|
| 119 |
+
[INFO ][2026-05-09 10:39:53][root ][_stage_targz_parts ] [local_rank 1] Extracted 18/26 parts
|
| 120 |
+
[INFO ][2026-05-09 10:40:08][root ][_stage_targz_parts ] [local_rank 1] Extracted 20/26 parts
|
| 121 |
+
[INFO ][2026-05-09 10:40:22][root ][_stage_targz_parts ] [local_rank 1] Extracted 22/26 parts
|
| 122 |
+
[INFO ][2026-05-09 10:40:36][root ][_stage_targz_parts ] [local_rank 1] Extracted 24/26 parts
|
| 123 |
+
[INFO ][2026-05-09 10:40:50][root ][_stage_targz_parts ] [local_rank 1] Extracted 26/26 parts
|
| 124 |
+
[INFO ][2026-05-09 10:40:50][root ][stage_datasets ] [local_rank 1/4] Staging ssv2 (multipart_tar)
|
| 125 |
+
[INFO ][2026-05-09 10:41:50][app.vjepa.train ][main ] Staged datasets: ['/scratch-node/dcanez.22600912/kinetics_240/train.csv', '/scratch-node/dcanez.22600912/ssv2/train.csv']
|
| 126 |
+
[INFO ][2026-05-09 10:42:25][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
|
| 127 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 128 |
+
(backbone): VisionTransformer(
|
| 129 |
+
(patch_embed): PatchEmbed3D(
|
| 130 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 131 |
+
)
|
| 132 |
+
(rope): CAPI2DRoPE()
|
| 133 |
+
(blocks): ModuleList(
|
| 134 |
+
(0-23): 24 x Block(
|
| 135 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 136 |
+
(rope_impl): CAPI2DRoPE()
|
| 137 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 138 |
+
(drop_path1): Identity()
|
| 139 |
+
(drop_path2): Identity()
|
| 140 |
+
(attn): Attention(
|
| 141 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 142 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 143 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 144 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 145 |
+
(rope_impl): CAPI2DRoPE()
|
| 146 |
+
)
|
| 147 |
+
(mlp): MLP(
|
| 148 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 149 |
+
(act): GELU(approximate='none')
|
| 150 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 151 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
)
|
| 155 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 156 |
+
)
|
| 157 |
+
)
|
| 158 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 159 |
+
(backbone): VisionTransformerPredictor(
|
| 160 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 161 |
+
(mask_tokens): ParameterList(
|
| 162 |
+
(0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 163 |
+
(1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 164 |
+
(2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 165 |
+
(3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
|
| 166 |
+
)
|
| 167 |
+
(predictor_blocks): ModuleList(
|
| 168 |
+
(0-5): 6 x Block(
|
| 169 |
+
(norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 170 |
+
(attn): RoPEAttention(
|
| 171 |
+
(qkv): Linear(in_features=384, out_features=1152, bias=True)
|
| 172 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 173 |
+
(proj): Linear(in_features=384, out_features=384, bias=True)
|
| 174 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 175 |
+
)
|
| 176 |
+
(drop_path): Identity()
|
| 177 |
+
(norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 178 |
+
(mlp): MLP(
|
| 179 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=True)
|
| 180 |
+
(act): GELU(approximate='none')
|
| 181 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=True)
|
| 182 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 183 |
+
)
|
| 184 |
+
)
|
| 185 |
+
)
|
| 186 |
+
(predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
|
| 187 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 188 |
+
)
|
| 189 |
+
)
|
| 190 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 191 |
+
(backbone): VisionTransformer(
|
| 192 |
+
(patch_embed): PatchEmbed3D(
|
| 193 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 194 |
+
)
|
| 195 |
+
(rope): CAPI2DRoPE()
|
| 196 |
+
(blocks): ModuleList(
|
| 197 |
+
(0-23): 24 x Block(
|
| 198 |
+
(norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 199 |
+
(rope_impl): CAPI2DRoPE()
|
| 200 |
+
(norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
|
| 201 |
+
(drop_path1): Identity()
|
| 202 |
+
(drop_path2): Identity()
|
| 203 |
+
(attn): Attention(
|
| 204 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=False)
|
| 205 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 206 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=False)
|
| 207 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 208 |
+
(rope_impl): CAPI2DRoPE()
|
| 209 |
+
)
|
| 210 |
+
(mlp): MLP(
|
| 211 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=False)
|
| 212 |
+
(act): GELU(approximate='none')
|
| 213 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=False)
|
| 214 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 215 |
+
)
|
| 216 |
+
)
|
| 217 |
+
)
|
| 218 |
+
(norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Encoder number of parameters: 302658560
|
| 222 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Predictor number of parameters: 11436928
|
| 223 |
+
[INFO ][2026-05-09 10:42:30][root ][init_video_model ] Target encoder number of parameters: 302658560
|
| 224 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset dataset created
|
| 225 |
+
[INFO ][2026-05-09 10:42:32][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 9 / 16
|
| 226 |
+
[INFO ][2026-05-09 10:42:32][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 227 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/399
|
| 228 |
+
[INFO ][2026-05-09 10:42:32][app.vjepa.train ][main ] Wrapping models in DDP (rank 9)...
|
| 229 |
+
[INFO ][2026-05-09 10:44:04][app.vjepa.train ][main ] Initializing loader...
|
| 230 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGTERM
|
| 231 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGTERM
|
| 232 |
+
submitit WARNING (2026-05-09 15:57:56,351) - Bypassing signal SIGCONT
|
| 233 |
+
[WARNING ][2026-05-09 15:57:56][submitit ][bypass ] Bypassing signal SIGCONT
|
vjepa/inflated/d002_full_ema_capilvd1689m/latest.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:aa7dadcd9d6613927adfe990638588131645f1d5096cd493dc3b09fef4619d44
|
| 3 |
+
size 4980135593
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r0.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r1.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r10.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r11.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r12.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r13.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r14.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r15.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r2.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r3.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r4.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r5.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r6.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r7.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
vjepa/inflated/d002_full_ema_capilvd1689m/log_r8.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|