logs/batch_final.log

19:27:05 INFO    fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677})
19:27:05 INFO    fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:27:05 INFO    fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:27:05 INFO    fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
19:27:05 INFO    fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
19:27:05 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: device 1 claimed (measured free 24765 MB >= min 20000 MB); 1/1 workers now claimed on this device
19:27:05 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:27:07 INFO    fpgm.datagen.batch.worker.0: shard 0: starting on cuda:1 (visible as cuda:0), 4 episode(s), 8 thread(s)
19:27:08 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:27:08 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 128 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:27:09 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:27:09 INFO    fpgm.datagen.robot_buffers: robot_buffers: cache hit for AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760, skipping render
19:27:09 INFO    fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=127  recall(median/p10)=0.950/0.876  shift(gated, systematic px)=1.24  shift(scatter, diagnostic px, std)=9.7  shift(scatter, diagnostic px, median|.|)=4.2  iou(mean, diagnostic)=0.768
   trajectory_h5_row0       n=127  recall(median/p10)=0.940/0.858  shift(gated, systematic px)=6.20  shift(scatter, diagnostic px, std)=9.8  shift(scatter, diagnostic px, median|.|)=7.0  iou(mean, diagnostic)=0.755
PASSED
19:27:25 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:27:46 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:28:01 INFO    fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_overlay.mp4
19:28:02 INFO    fpgm.datagen.background_depth: background depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/background_depth.h5
19:28:02 INFO    fpgm.datagen.dense_depth: dense depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/depth_dense.h5
19:28:02 INFO    fpgm.datagen.plate: plate: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/plate.png
19:28:02 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:28:02 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:28:02 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:28:03 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:28:03 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:28:03 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:28:03 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:28:04 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:28:05 INFO    fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.77 px, drift 1.50 px
19:28:05 INFO    fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.65 px, drift 0.81 px
19:28:05 INFO    fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
19:28:06 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:28:06 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:28:06 INFO    fpgm.geometry.rigid_lifting: rigid PnP solved 127/127 frames (median inlier fraction 100%, seed planarity 0.4001)
19:28:06 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): prismatic precondition passed (residual_m=0.00045 rotation_range_deg=1.903 n_frames=101) -- geometry is OBSERVED POINTS, no orientation fit anywhere in this path
19:28:06 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:28:06 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): robot carve -- seed frame 0 mask/robot overlap BEFORE carving = 0/49393 px (0.00% of the seed mask); carved from every frame in static_span=(0, 30) before unioning (robot_seg supplied)
19:28:06 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): mask union px, robot-carved=3433 vs. pre-carve=3433 (+0 px removed by the carve)
19:28:06 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): static_span=(0, 30) -- mask union 3433 px, 3433 valid (coverage 100.0%, of which 3433 genuinely measured/ANCHOR = 100.0% of the mask)
19:28:06 INFO    fpgm.datagen.pipeline: S6: drawer geometry built as an OBSERVED point cloud (no fitted mesh, no fitted orientation): 3433 points, coverage=100.0% of the mask (100.0% genuinely measured)
19:28:06 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): frame 0 -- 3077 cloud points, top 15% band -> 2353 points
19:28:06 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): fit ok -- normal=[0.3019, 0.0422, 0.9524] offset=0.2202 rms=0.0054m n_inliers=1139/2353 footprint_ratio=0.88
19:28:06 INFO    fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=0.519 silhouette_iou(mean over valid)=0.743 rigid_lift={'planarity': 0.17042300800645957, 'n_rejected_implausible': 0, 'n_solved': 89, 'n_frames': 127} pose_source_counts={0: 87, 3: 40}
19:28:07 INFO    fpgm.datagen.pipeline: S6: drawer -- pose_noise_mm=0.137 silhouette_iou(mean over valid)=0.353 rigid_lift={'planarity': 0.40009308398436977, 'n_rejected_implausible': 0, 'n_solved': 127, 'n_frames': 127} pose_source_counts={0: 127}
19:28:07 INFO    fpgm.datagen.pipeline: S6: brick -- parent bridging via drawer: ['frames 87-126: parent=drawer attach rejected (pre-gap relative-pose score=0.064 < threshold=0.5)'] pose_source_counts={0: 87, 3: 40} silhouette_iou(mean over valid)=0.743
19:28:07 INFO    fpgm.datagen.events: EventLabeler.label: 40/127 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
19:28:08 INFO    fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/object_poses_debug.mp4
19:28:08 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/poses.npz
19:28:08 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/events.json
19:28:09 INFO    fpgm.datagen.export_vace: S8: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760 -- 127 video frames -> 3 window(s) (window=81 stride=40): [(0, 81), (40, 121), (46, 127)] -- mesh objects=('brick',), point-cloud objects=('drawer',)
19:28:09 INFO    fpgm.datagen.export_vace: S8: carved 'drawer''s reference-frame (0) footprint out of the background plate -- 25609/399360 export pixels (6.4%)
19:28:10 INFO    fpgm.datagen.export_vace: S8: window [0, 81) up to date, skipping
19:28:10 INFO    fpgm.datagen.export_vace: S8: window [40, 121) up to date, skipping
19:28:10 INFO    fpgm.datagen.export_vace: S8: window [46, 127) up to date, skipping
19:28:10 INFO    fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s -> ok (62.5s)
19:28:11 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:28:11 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 149 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:28:11 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:28:12 INFO    fpgm.pipeline.frames: extracted 148 frames [0, 147] stride=1 from 22008760.mp4 -> /tmp/fpgm_frames_15fr8yi3
/home/quang/miniconda3/envs/fpgm/lib/python3.12/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
19:28:14 INFO    fpgm.segmentation.sam3: building SAM 3.1 predictor (use_fa3=False)
19:28:26 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 19304, 1: 21879, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
INFO 2026-08-01 19:28:30,577 4069679 sam3_multiplex_base.py: 336: `setting max_num_objects` to 16 -- creating num_obj_for_compile=1 objects for torch.compile cache
19:28:34 INFO    fpgm.segmentation.sam3: applied sam3 compatibility shim: init_state does not accept ['offload_state_to_cpu']
19:28:34 INFO    fpgm.segmentation.sam3: applied sam3 compatibility shim: _build_sam2_output was discarding refined masks on frames with no prior cache entry, which breaks point-prompt video propagation
dynamic_multimask_via_stability is reset to False in the multiplex model

frame loading (image folder) [rank=0]:   0%|          | 0/148 [00:00<?, ?it/s]INFO 2026-08-01 19:28:34,338 4069679 sam3_base_predictor.py: 146: started new session 31f42586-b125-4014-aada-084e46b74aff
INFO 2026-08-01 19:28:34,338 4069679 sam3_multiplex_tracking.py:1677: Running add_prompt on frame 74

frame loading (image folder) [rank=0]:   2%|▏         | 3/148 [00:00<00:07, 19.69it/s]
frame loading (image folder) [rank=0]:   4%|▍         | 6/148 [00:00<00:06, 21.47it/s]
frame loading (image folder) [rank=0]:   6%|▌         | 9/148 [00:00<00:05, 23.52it/s]
frame loading (image folder) [rank=0]:   9%|▉         | 13/148 [00:00<00:05, 26.71it/s]
frame loading (image folder) [rank=0]:  11%|█         | 16/148 [00:00<00:06, 19.50it/s]INFO 2026-08-01 19:28:35,168 4069679 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=False).


propagate_in_video:   0%|          | 0/74 [00:00<?, ?it/s]
frame loading (image folder) [rank=0]:  13%|█▎        | 19/148 [00:00<00:06, 19.40it/s]
frame loading (image folder) [rank=0]:  16%|█▌        | 23/148 [00:01<00:05, 23.44it/s]
frame loading (image folder) [rank=0]:  18%|█▊        | 26/148 [00:01<00:05, 21.98it/s]
frame loading (image folder) [rank=0]:  20%|█▉        | 29/148 [00:01<00:05, 20.02it/s]
frame loading (image folder) [rank=0]:  22%|██▏       | 32/148 [00:01<00:05, 20.49it/s]
frame loading (image folder) [rank=0]:  24%|██▎       | 35/148 [00:01<00:05, 18.97it/s]
frame loading (image folder) [rank=0]:  25%|██▌       | 37/148 [00:01<00:06, 17.41it/s]
frame loading (image folder) [rank=0]:  26%|██▋       | 39/148 [00:01<00:06, 16.28it/s]

propagate_in_video:   1%|▏         | 1/74 [00:01<01:30,  1.24s/it]
frame loading (image folder) [rank=0]:  30%|██▉       | 44/148 [00:02<00:04, 23.09it/s]
frame loading (image folder) [rank=0]:  32%|███▏      | 47/148 [00:02<00:04, 22.41it/s]
frame loading (image folder) [rank=0]:  35%|███▌      | 52/148 [00:02<00:03, 24.77it/s]
frame loading (image folder) [rank=0]:  37%|███▋      | 55/148 [00:02<00:03, 23.97it/s]
frame loading (image folder) [rank=0]:  39%|███▉      | 58/148 [00:02<00:03, 24.33it/s]
frame loading (image folder) [rank=0]:  43%|████▎     | 63/148 [00:02<00:03, 27.61it/s]
frame loading (image folder) [rank=0]:  66%|██████▌   | 98/148 [00:02<00:00, 101.36it/s]

propagate_in_video:   9%|▉         | 7/74 [00:02<00:20,  3.28it/s]
frame loading (image folder) [rank=0]:  74%|███████▍  | 110/148 [00:03<00:00, 47.21it/s]
frame loading (image folder) [rank=0]:  80%|████████  | 119/148 [00:03<00:00, 37.35it/s]

propagate_in_video:  31%|███       | 23/74 [00:03<00:05,  9.22it/s]
frame loading (image folder) [rank=0]:  85%|████████▌ | 126/148 [00:04<00:00, 33.55it/s]
frame loading (image folder) [rank=0]:  89%|████████▉ | 132/148 [00:04<00:00, 32.78it/s]
frame loading (image folder) [rank=0]:  93%|█████████▎| 137/148 [00:04<00:00, 34.18it/s]
frame loading (image folder) [rank=0]:  96%|█████████▌| 142/148 [00:04<00:00, 30.14it/s]

propagate_in_video:  53%|█████▎    | 39/74 [00:04<00:02, 12.91it/s]
frame loading (image folder) [rank=0]:  99%|█████████▊| 146/148 [00:04<00:00, 29.17it/s]
frame loading (image folder) [rank=0]: 100%|██████████| 148/148 [00:05<00:00, 29.49it/s]


propagate_in_video:  74%|███████▍  | 55/74 [00:04<00:01, 16.51it/s]

propagate_in_video:  96%|█████████▌| 71/74 [00:04<00:00, 23.46it/s]
propagate_in_video: 100%|██████████| 74/74 [00:04<00:00, 15.27it/s]
INFO 2026-08-01 19:28:40,013 4069679 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=True).

propagate_in_video:   0%|          | 0/74 [00:00<?, ?it/s]
propagate_in_video:   1%|▏         | 1/74 [00:00<00:46,  1.56it/s]
propagate_in_video:  15%|█▍        | 11/74 [00:01<00:06, 10.30it/s]
propagate_in_video:  36%|███▋      | 27/74 [00:01<00:02, 17.33it/s]
propagate_in_video:  58%|█████▊    | 43/74 [00:02<00:01, 21.37it/s]
propagate_in_video:  80%|███████▉  | 59/74 [00:03<00:00, 22.75it/s]
propagate_in_video: 100%|██████████| 74/74 [00:03<00:00, 23.75it/s]
INFO 2026-08-01 19:28:43,130 4069679 sam3_base_predictor.py: 305: propagation ended in session 31f42586-b125-4014-aada-084e46b74aff
19:28:43 INFO    fpgm.datagen.robot_buffers: robot_buffers: SAM3 prompt 'robot arm and gripper' found nothing at frame 74
INFO 2026-08-01 19:28:43,780 4069679 sam3_base_predictor.py: 398: empty_cache freed 19442696192 bytes (free_pct 0.3% -> 13.3%, reserved 24788336640 -> 5345640448 bytes)
INFO 2026-08-01 19:28:43,780 4069679 sam3_base_predictor.py: 410: removed session 31f42586-b125-4014-aada-084e46b74aff

frame loading (image folder) [rank=0]:   0%|          | 0/148 [00:00<?, ?it/s]INFO 2026-08-01 19:28:43,872 4069679 sam3_base_predictor.py: 146: started new session 248b5081-9218-44e5-95cb-5a2387569fdf
INFO 2026-08-01 19:28:43,873 4069679 sam3_multiplex_tracking.py:1677: Running add_prompt on frame 74

frame loading (image folder) [rank=0]:   3%|▎         | 4/148 [00:00<00:06, 21.46it/s]
frame loading (image folder) [rank=0]:   5%|▍         | 7/148 [00:00<00:06, 21.92it/s]
frame loading (image folder) [rank=0]:   7%|▋         | 11/148 [00:00<00:04, 27.71it/s]
frame loading (image folder) [rank=0]:   9%|▉         | 14/148 [00:00<00:05, 23.29it/s]INFO 2026-08-01 19:28:44,467 4069679 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=False).


propagate_in_video:   0%|          | 0/74 [00:00<?, ?it/s]
frame loading (image folder) [rank=0]:  11%|█▏        | 17/148 [00:00<00:05, 23.43it/s]
frame loading (image folder) [rank=0]:  14%|█▎        | 20/148 [00:00<00:06, 18.46it/s]
frame loading (image folder) [rank=0]:  16%|█▌        | 23/148 [00:01<00:06, 19.99it/s]
frame loading (image folder) [rank=0]:  18%|█▊        | 26/148 [00:01<00:06, 19.21it/s]
frame loading (image folder) [rank=0]:  20%|█▉        | 29/148 [00:01<00:06, 17.57it/s]
frame loading (image folder) [rank=0]:  21%|██        | 31/148 [00:01<00:06, 16.73it/s]
frame loading (image folder) [rank=0]:  23%|██▎       | 34/148 [00:01<00:05, 19.20it/s]
frame loading (image folder) [rank=0]:  26%|██▌       | 38/148 [00:01<00:04, 23.49it/s]
frame loading (image folder) [rank=0]:  28%|██▊       | 42/148 [00:01<00:03, 26.75it/s]
frame loading (image folder) [rank=0]:  30%|███       | 45/148 [00:02<00:04, 24.08it/s]

propagate_in_video:   1%|▏         | 1/74 [00:01<01:49,  1.50s/it]
frame loading (image folder) [rank=0]:  32%|███▏      | 48/148 [00:02<00:04, 24.32it/s]

propagate_in_video:   8%|▊         | 6/74 [00:01<00:13,  4.96it/s]
frame loading (image folder) [rank=0]:  36%|███▌      | 53/148 [00:02<00:03, 29.94it/s]
frame loading (image folder) [rank=0]:  39%|███▊      | 57/148 [00:02<00:03, 23.58it/s]
frame loading (image folder) [rank=0]:  41%|████      | 61/148 [00:02<00:03, 24.88it/s]
frame loading (image folder) [rank=0]:  61%|██████    | 90/148 [00:02<00:00, 77.61it/s]19:28:46 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 19304, 1: 6951, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s

frame loading (image folder) [rank=0]:  67%|██████▋   | 99/148 [00:03<00:01, 45.96it/s]
frame loading (image folder) [rank=0]:  72%|███████▏  | 106/148 [00:03<00:01, 41.48it/s]

propagate_in_video:  12%|█▏        | 9/74 [00:03<00:21,  3.06it/s]
frame loading (image folder) [rank=0]:  76%|███████▌  | 112/148 [00:03<00:01, 33.86it/s]

propagate_in_video:  18%|█▊        | 13/74 [00:03<00:11,  5.22it/s]

propagate_in_video:  24%|██▍       | 18/74 [00:03<00:06,  8.61it/s]

propagate_in_video:  30%|██▉       | 22/74 [00:03<00:04, 11.67it/s]
frame loading (image folder) [rank=0]:  79%|███████▉  | 117/148 [00:04<00:01, 26.60it/s]
frame loading (image folder) [rank=0]:  82%|████████▏ | 121/148 [00:04<00:01, 25.63it/s]
frame loading (image folder) [rank=0]:  84%|████████▍ | 125/148 [00:04<00:00, 27.26it/s]
frame loading (image folder) [rank=0]:  87%|████████▋ | 129/148 [00:04<00:00, 26.74it/s]
frame loading (image folder) [rank=0]:  90%|████████▉ | 133/148 [00:04<00:00, 25.63it/s]

propagate_in_video:  35%|███▌      | 26/74 [00:04<00:05,  8.10it/s]
frame loading (image folder) [rank=0]:  92%|█████████▏| 136/148 [00:04<00:00, 25.11it/s]
frame loading (image folder) [rank=0]:  94%|█████████▍| 139/148 [00:05<00:00, 22.77it/s]

propagate_in_video:  41%|████      | 30/74 [00:04<00:04,  9.82it/s]
frame loading (image folder) [rank=0]:  96%|█████████▌| 142/148 [00:05<00:00, 23.78it/s]

propagate_in_video:  45%|████▍     | 33/74 [00:04<00:03, 11.82it/s]

propagate_in_video:  50%|█████     | 37/74 [00:04<00:02, 15.06it/s]
frame loading (image folder) [rank=0]:  98%|█████████▊| 145/148 [00:05<00:00, 20.85it/s]
frame loading (image folder) [rank=0]: 100%|██████████| 148/148 [00:05<00:00, 20.71it/s]
frame loading (image folder) [rank=0]: 100%|██████████| 148/148 [00:05<00:00, 26.86it/s]


propagate_in_video:  54%|█████▍    | 40/74 [00:05<00:03,  9.12it/s]

propagate_in_video:  58%|█████▊    | 43/74 [00:05<00:02, 10.40it/s]

propagate_in_video:  62%|██████▏   | 46/74 [00:05<00:02, 12.57it/s]

propagate_in_video:  68%|██████▊   | 50/74 [00:05<00:01, 16.49it/s]

propagate_in_video:  74%|███████▍  | 55/74 [00:06<00:01, 10.85it/s]

propagate_in_video:  82%|████████▏ | 61/74 [00:06<00:00, 15.78it/s]

propagate_in_video:  91%|█████████ | 67/74 [00:06<00:00, 21.21it/s]

propagate_in_video:  96%|█████████▌| 71/74 [00:07<00:00, 16.45it/s]
propagate_in_video: 100%|██████████| 74/74 [00:07<00:00, 10.37it/s]
INFO 2026-08-01 19:28:51,611 4069679 sam3_multiplex_tracking.py: 580: Bucket utilization rate: 6.25%, subscription rate: 100.00%
INFO 2026-08-01 19:28:51,612 4069679 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=True).

propagate_in_video:   0%|          | 0/74 [00:00<?, ?it/s]
propagate_in_video:   1%|▏         | 1/74 [00:00<00:46,  1.57it/s]
propagate_in_video:   9%|▉         | 7/74 [00:00<00:05, 12.28it/s]
propagate_in_video:  14%|█▎        | 10/74 [00:01<00:08,  7.22it/s]
INFO 2026-08-01 19:28:52,998 4069679 sam3_base_predictor.py: 305: propagation ended in session 248b5081-9218-44e5-95cb-5a2387569fdf
INFO 2026-08-01 19:28:53,540 4069679 sam3_base_predictor.py: 398: empty_cache freed 2730491904 bytes (free_pct 0.8% -> 2.6%, reserved 24071110656 -> 21340618752 bytes)
INFO 2026-08-01 19:28:53,540 4069679 sam3_base_predictor.py: 410: removed session 248b5081-9218-44e5-95cb-5a2387569fdf
19:28:53 WARNING fpgm.datagen.pipeline: stage s2 failed for AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/ext1: CUDA out of memory. Tried to allocate 1.27 GiB. GPU 0 has a total capacity of 139.81 GiB of which 1.08 GiB is free. Including non-PyTorch memory, this process has 23.09 GiB memory in use. Of the allocated memory 22.27 GiB is allocated by PyTorch, and 150.18 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
19:28:53 INFO    fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-08m-08s -> failed (43.4s)
19:28:54 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:28:54 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 166 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:28:54 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 1); retrying in 20s
19:29:06 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 19304, 1: 3715, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '