logs/batch_v5.log

19:49:27 INFO    fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 19304, 1: 47593, 2: 3451, 3: 11677})
19:49:27 INFO    fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:49:27 INFO    fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:49:27 INFO    fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
19:49:27 INFO    fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
19:49:27 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: device 1 claimed (measured free 47593 MB >= min 20000 MB); 1/1 workers now claimed on this device
19:49:27 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 19304, 1: 47593, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:49:29 INFO    fpgm.datagen.batch.worker.0: shard 0: starting on cuda:1 (visible as cuda:0), 4 episode(s), 8 thread(s)
19:49:31 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:49:31 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 128 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:49:31 INFO    fpgm.datagen.pipeline: picked GPU 1 (48196 MB free >= 20000 MB required)
19:49:31 INFO    fpgm.datagen.robot_buffers: robot_buffers: loaded 127 cached SAM masks from /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_buffers/sam_masks_full.npz
19:49:47 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:50:08 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:50:28 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:50:48 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:51:08 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:51:28 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:51:40 INFO    fpgm.datagen.robot_buffers: robot_buffers: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=127  recall(median/p10)=0.950/0.876  shift(gated, systematic px)=1.24  shift(scatter, diagnostic px, std)=9.7  shift(scatter, diagnostic px, median|.|)=4.2  iou(mean, diagnostic)=0.768
   trajectory_h5_row0       n=127  recall(median/p10)=0.940/0.858  shift(gated, systematic px)=6.20  shift(scatter, diagnostic px, std)=9.8  shift(scatter, diagnostic px, median|.|)=7.0  iou(mean, diagnostic)=0.755
PASSED
19:51:49 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 8; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:52:09 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 9; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:52:26 INFO    fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=127  recall(median/p10)=0.950/0.876  shift(gated, systematic px)=1.24  shift(scatter, diagnostic px, std)=9.7  shift(scatter, diagnostic px, median|.|)=4.2  iou(mean, diagnostic)=0.768
   trajectory_h5_row0       n=127  recall(median/p10)=0.940/0.858  shift(gated, systematic px)=6.20  shift(scatter, diagnostic px, std)=9.8  shift(scatter, diagnostic px, median|.|)=7.0  iou(mean, diagnostic)=0.755
PASSED
19:52:29 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 10; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:52:49 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 11; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:53:09 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 12; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:53:17 INFO    fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_overlay.mp4
19:53:18 INFO    fpgm.datagen.background_depth: background depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/background_depth.h5
19:53:18 INFO    fpgm.datagen.dense_depth: dense depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/depth_dense.h5
19:53:18 INFO    fpgm.datagen.plate: plate: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/plate.png
19:53:18 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:53:18 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:53:18 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:53:19 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:53:19 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:53:20 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:53:21 INFO    fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.77 px, drift 1.50 px
19:53:21 INFO    fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.65 px, drift 0.81 px
19:53:22 INFO    fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
19:53:22 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:53:22 INFO    fpgm.geometry.rigid_lifting: rigid PnP solved 127/127 frames (median inlier fraction 100%, seed planarity 0.4001)
19:53:22 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): prismatic precondition passed (residual_m=0.00045 rotation_range_deg=1.903 n_frames=101) -- geometry is OBSERVED POINTS, no orientation fit anywhere in this path
19:53:22 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:53:22 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): robot carve -- seed frame 0 mask/robot overlap BEFORE carving = 0/49393 px (0.00% of the seed mask); carved from every frame in static_span=(0, 30) before unioning (robot_seg supplied)
19:53:22 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): mask union px, robot-carved=3433 vs. pre-carve=3433 (+0 px removed by the carve)
19:53:22 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): static_span=(0, 30) -- mask union 3433 px, 3433 valid (coverage 100.0%, of which 3433 genuinely measured/ANCHOR = 100.0% of the mask)
19:53:22 INFO    fpgm.datagen.pipeline: S6: drawer geometry built as an OBSERVED point cloud (no fitted mesh, no fitted orientation): 3433 points, coverage=100.0% of the mask (100.0% genuinely measured)
19:53:22 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): frame 0 -- 3077 cloud points, top 15% band -> 2353 points
19:53:22 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): fit ok -- normal=[0.3019, 0.0422, 0.9524] offset=0.2202 rms=0.0054m n_inliers=1139/2353 footprint_ratio=0.88
19:53:23 INFO    fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=0.519 silhouette_iou(mean over valid)=0.743 rigid_lift={'planarity': 0.17042300800645957, 'n_rejected_implausible': 0, 'n_solved': 89, 'n_frames': 127} pose_source_counts={0: 87, 3: 40}
19:53:23 INFO    fpgm.datagen.pipeline: S6: drawer -- pose_noise_mm=0.137 silhouette_iou(mean over valid)=0.353 rigid_lift={'planarity': 0.40009308398436977, 'n_rejected_implausible': 0, 'n_solved': 127, 'n_frames': 127} pose_source_counts={0: 127}
19:53:23 INFO    fpgm.datagen.pipeline: S6: brick -- parent bridging via drawer: ['frames 87-126: parent=drawer attach rejected (pre-gap relative-pose score=0.064 < threshold=0.5)'] pose_source_counts={0: 87, 3: 40} silhouette_iou(mean over valid)=0.743
19:53:23 INFO    fpgm.datagen.events: EventLabeler.label: 40/127 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
19:53:24 INFO    fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/object_poses_debug.mp4
19:53:24 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/poses.npz
19:53:24 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/events.json
19:53:25 INFO    fpgm.datagen.export_vace: S8: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760 -- 127 video frames -> 3 window(s) (window=81 stride=40): [(0, 81), (40, 121), (46, 127)] -- mesh objects=('brick',), point-cloud objects=('drawer',)
19:53:25 INFO    fpgm.datagen.export_vace: S8: carved 'drawer''s reference-frame (0) footprint out of the background plate -- 25609/399360 export pixels (6.4%)
19:53:25 INFO    fpgm.datagen.export_vace: S8: window [0, 81) up to date, skipping
19:53:25 INFO    fpgm.datagen.export_vace: S8: window [40, 121) up to date, skipping
19:53:25 INFO    fpgm.datagen.export_vace: S8: window [46, 127) up to date, skipping
19:53:26 INFO    fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s -> ok (236.2s)
19:53:26 INFO    fpgm.datagen.batch.worker.0: shard 0: after AUTOLab+0d4edc83+2023-10-21-19h-07m-04s -- GPU allocated 0.00 GB, reserved 0.00 GB
19:53:27 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:53:27 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 149 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:53:27 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:53:27 INFO    fpgm.datagen.robot_buffers: robot_buffers: loaded 117 cached SAM masks from /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/robot_buffers/sam_masks_full.npz
19:53:27 WARNING fpgm.datagen.robot_buffers: robot_buffers: dropped 3/117 SAM-tracked frames as failed-tracking (area < 20% of median): [31, 146, 147]
19:53:30 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 13; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:53:50 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 14; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:54:10 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 15; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:54:30 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 16; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:54:50 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 17; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:55:11 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 18; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:55:21 INFO    fpgm.datagen.robot_buffers: robot_buffers: robot alignment gate (prompt='robotic arm', sam_tracked=117, dropped=3, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=114  recall(median/p10)=0.970/0.921  shift(gated, systematic px)=0.54  shift(scatter, diagnostic px, std)=9.6  shift(scatter, diagnostic px, median|.|)=5.2  iou(mean, diagnostic)=0.658
   trajectory_h5_row0       n=114  recall(median/p10)=0.965/0.905  shift(gated, systematic px)=8.12  shift(scatter, diagnostic px, std)=10.6  shift(scatter, diagnostic px, median|.|)=8.0  iou(mean, diagnostic)=0.642
PASSED
19:55:31 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 19; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:55:51 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 20; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:56:11 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 21; measured free MB: {0: 19304, 1: 123297, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:56:12 INFO    fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=117, dropped=3, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=114  recall(median/p10)=0.970/0.921  shift(gated, systematic px)=0.54  shift(scatter, diagnostic px, std)=9.6  shift(scatter, diagnostic px, median|.|)=5.2  iou(mean, diagnostic)=0.658
   trajectory_h5_row0       n=114  recall(median/p10)=0.965/0.905  shift(gated, systematic px)=8.12  shift(scatter, diagnostic px, std)=10.6  shift(scatter, diagnostic px, median|.|)=8.0  iou(mean, diagnostic)=0.642
PASSED
19:56:32 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 22; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:56:52 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 23; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:57:10 INFO    fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/robot_overlay.mp4
19:57:11 INFO    fpgm.datagen.background_depth: background depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/background_depth.h5
19:57:11 INFO    fpgm.datagen.dense_depth: dense depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/depth_dense.h5
19:57:11 INFO    fpgm.datagen.plate: plate: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/plate.png
19:57:11 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:57:11 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:57:11 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:57:12 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:57:12 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 24; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:57:12 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:57:14 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:57:14 INFO    fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.76 px, drift 0.61 px
19:57:14 INFO    fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.56 px, drift 0.47 px
19:57:15 INFO    fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
19:57:15 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:57:15 INFO    fpgm.geometry.rigid_lifting: rigid PnP solved 146/148 frames (median inlier fraction 100%, seed planarity 0.3095)
19:57:15 WARNING fpgm.datagen.pipeline: S6: fixture 'drawer' failed the prismatic precondition: build_observed_surface_cloud('drawer'): mesh-free precondition failed -- residual_m=0.00021, rotation_range_deg=0.644, n_frames=112 did not satisfy PrismaticFitter's gates; refusing to carry an observed point cloud rigidly for an object not confirmed prismatic -- see FixtureNotPrismaticError's own docstring
19:57:16 INFO    fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=1.017 silhouette_iou(mean over valid)=0.706 rigid_lift={'planarity': 0.1736893205794351, 'n_rejected_implausible': 0, 'n_solved': 103, 'n_frames': 148} pose_source_counts={0: 98, 3: 50}
19:57:16 WARNING fpgm.datagen.pipeline: S6: role 'drawer' has no mesh/geometry built; skipping
19:57:16 INFO    fpgm.datagen.events: EventLabeler.label: 50/148 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
19:57:17 INFO    fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/object_poses_debug.mp4
19:57:17 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/poses.npz
19:57:17 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/events.json
19:57:18 INFO    fpgm.datagen.export_vace: S8: AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760 -- 148 video frames -> 3 window(s) (window=81 stride=40): [(0, 81), (40, 121), (67, 148)] -- mesh objects=('brick',), point-cloud objects=()
19:57:18 INFO    fpgm.datagen.export_vace: S8: window [0, 81) up to date, skipping
19:57:18 INFO    fpgm.datagen.export_vace: S8: window [40, 121) up to date, skipping
19:57:18 INFO    fpgm.datagen.export_vace: S8: window [67, 148) up to date, skipping
19:57:19 INFO    fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-08m-08s -> ok (232.6s)
19:57:19 INFO    fpgm.datagen.batch.worker.0: shard 0: after AUTOLab+0d4edc83+2023-10-21-19h-08m-08s -- GPU allocated 0.00 GB, reserved 0.00 GB
19:57:19 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:57:19 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 166 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:57:20 INFO    fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
19:57:21 INFO    fpgm.pipeline.frames: extracted 165 frames [0, 164] stride=1 from 22008760.mp4 -> /tmp/fpgm_frames_a268pgqf
/home/quang/miniconda3/envs/fpgm/lib/python3.12/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
19:57:23 INFO    fpgm.segmentation.sam3: building SAM 3.1 predictor (use_fa3=False)
19:57:33 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 25; measured free MB: {0: 19304, 1: 21772, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
INFO 2026-08-01 19:57:42,176 4085193 sam3_multiplex_base.py: 336: `setting max_num_objects` to 16 -- creating num_obj_for_compile=1 objects for torch.compile cache
19:57:46 INFO    fpgm.segmentation.sam3: applied sam3 compatibility shim: init_state does not accept ['offload_state_to_cpu']
19:57:46 INFO    fpgm.segmentation.sam3: applied sam3 compatibility shim: _build_sam2_output was discarding refined masks on frames with no prior cache entry, which breaks point-prompt video propagation
dynamic_multimask_via_stability is reset to False in the multiplex model

frame loading (image folder) [rank=0]:   0%|          | 0/165 [00:00<?, ?it/s]INFO 2026-08-01 19:57:46,179 4085193 sam3_base_predictor.py: 146: started new session a82730e6-1f6d-42cf-a377-3f4c7e64bdad
INFO 2026-08-01 19:57:46,180 4085193 sam3_multiplex_tracking.py:1677: Running add_prompt on frame 82

frame loading (image folder) [rank=0]:   2%|▏         | 3/165 [00:00<00:07, 22.25it/s]
frame loading (image folder) [rank=0]:   4%|▎         | 6/165 [00:00<00:06, 22.92it/s]
frame loading (image folder) [rank=0]:   6%|▌         | 10/165 [00:00<00:05, 27.37it/s]
frame loading (image folder) [rank=0]:   8%|▊         | 14/165 [00:00<00:04, 31.18it/s]
frame loading (image folder) [rank=0]:  11%|█         | 18/165 [00:00<00:04, 30.10it/s]INFO 2026-08-01 19:57:46,885 4085193 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=False).


propagate_in_video:   0%|          | 0/83 [00:00<?, ?it/s]
frame loading (image folder) [rank=0]:  13%|█▎        | 22/165 [00:00<00:04, 29.33it/s]
frame loading (image folder) [rank=0]:  15%|█▌        | 25/165 [00:00<00:05, 26.30it/s]
frame loading (image folder) [rank=0]:  18%|█▊        | 29/165 [00:01<00:04, 28.84it/s]
frame loading (image folder) [rank=0]:  20%|██        | 33/165 [00:01<00:04, 30.29it/s]
frame loading (image folder) [rank=0]:  22%|██▏       | 37/165 [00:01<00:04, 26.14it/s]
frame loading (image folder) [rank=0]:  24%|██▍       | 40/165 [00:01<00:04, 25.23it/s]
frame loading (image folder) [rank=0]:  27%|██▋       | 44/165 [00:01<00:04, 27.91it/s]
frame loading (image folder) [rank=0]:  28%|██▊       | 47/165 [00:01<00:06, 19.53it/s]

propagate_in_video:   1%|          | 1/83 [00:01<01:44,  1.27s/it]
frame loading (image folder) [rank=0]:  31%|███       | 51/165 [00:02<00:05, 21.10it/s]
frame loading (image folder) [rank=0]:  33%|███▎      | 55/165 [00:02<00:04, 24.36it/s]
frame loading (image folder) [rank=0]:  35%|███▌      | 58/165 [00:02<00:04, 23.96it/s]
frame loading (image folder) [rank=0]:  37%|███▋      | 61/165 [00:02<00:05, 20.63it/s]
frame loading (image folder) [rank=0]:  39%|███▉      | 64/165 [00:02<00:04, 21.79it/s]
frame loading (image folder) [rank=0]:  41%|████      | 68/165 [00:02<00:04, 22.78it/s]
frame loading (image folder) [rank=0]:  43%|████▎     | 71/165 [00:02<00:04, 20.47it/s]
frame loading (image folder) [rank=0]:  45%|████▍     | 74/165 [00:03<00:04, 19.82it/s]
frame loading (image folder) [rank=0]:  48%|████▊     | 79/165 [00:03<00:03, 24.00it/s]
frame loading (image folder) [rank=0]:  69%|██████▉   | 114/165 [00:03<00:00, 89.29it/s]

propagate_in_video:  18%|█▊        | 15/83 [00:02<00:10,  6.35it/s]
frame loading (image folder) [rank=0]:  76%|███████▌  | 125/165 [00:03<00:00, 55.89it/s]
frame loading (image folder) [rank=0]:  81%|████████  | 133/165 [00:04<00:00, 47.45it/s]
frame loading (image folder) [rank=0]:  85%|████████▍ | 140/165 [00:04<00:00, 39.05it/s]

propagate_in_video:  37%|███▋      | 31/83 [00:03<00:05,  9.73it/s]
frame loading (image folder) [rank=0]:  88%|████████▊ | 146/165 [00:04<00:00, 32.34it/s]
frame loading (image folder) [rank=0]:  92%|█████████▏| 151/165 [00:04<00:00, 32.45it/s]
frame loading (image folder) [rank=0]:  94%|█████████▍| 155/165 [00:04<00:00, 29.76it/s]
frame loading (image folder) [rank=0]:  96%|█████████▋| 159/165 [00:05<00:00, 30.58it/s]
frame loading (image folder) [rank=0]:  99%|█████████▉| 163/165 [00:05<00:00, 31.81it/s]

propagate_in_video:  57%|█████▋    | 47/83 [00:04<00:02, 13.41it/s]
frame loading (image folder) [rank=0]: 100%|██████████| 165/165 [00:05<00:00, 30.94it/s]


propagate_in_video:  76%|███████▌  | 63/83 [00:05<00:01, 16.58it/s]

propagate_in_video:  78%|███████▊  | 65/83 [00:05<00:01, 16.38it/s]

propagate_in_video:  83%|████████▎ | 69/83 [00:05<00:00, 18.07it/s]

propagate_in_video:  90%|█████████ | 75/83 [00:05<00:00, 21.55it/s]

propagate_in_video:  95%|█████████▌| 79/83 [00:05<00:00, 16.64it/s]

propagate_in_video: 100%|██████████| 83/83 [00:06<00:00, 18.74it/s]
propagate_in_video: 100%|██████████| 83/83 [00:06<00:00, 13.78it/s]
INFO 2026-08-01 19:57:52,909 4085193 sam3_multiplex_tracking.py: 580: Bucket utilization rate: 6.25%, subscription rate: 100.00%
INFO 2026-08-01 19:57:52,910 4085193 sam3_multiplex_tracking.py:2316: Running full VG propagation (reverse=True).

propagate_in_video:   0%|          | 0/82 [00:00<?, ?it/s]19:57:53 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 26; measured free MB: {0: 19304, 1: 250, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s

propagate_in_video:   1%|          | 1/82 [00:00<00:45,  1.79it/s]
propagate_in_video:   2%|▏         | 2/82 [00:01<00:45,  1.74it/s]
INFO 2026-08-01 19:57:54,059 4085193 sam3_base_predictor.py: 305: propagation ended in session a82730e6-1f6d-42cf-a377-3f4c7e64bdad
INFO 2026-08-01 19:57:54,472 4085193 sam3_base_predictor.py: 398: empty_cache freed 2726297600 bytes (free_pct 0.8% -> 2.7%, reserved 23865589760 -> 21139292160 bytes)
INFO 2026-08-01 19:57:54,473 4085193 sam3_base_predictor.py: 410: removed session a82730e6-1f6d-42cf-a377-3f4c7e64bdad
19:57:54 WARNING fpgm.datagen.pipeline: stage s2 failed for AUTOLab+0d4edc83+2023-10-21-19h-08m-41s/ext1: CUDA out of memory. Tried to allocate 1.27 GiB. GPU 0 has a total capacity of 139.81 GiB of which 1.17 GiB is free. Including non-PyTorch memory, this process has 22.90 GiB memory in use. Of the allocated memory 22.08 GiB is allocated by PyTorch, and 154.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
19:57:54 INFO    fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-08m-41s -> failed (35.4s)
19:57:54 INFO    fpgm.datagen.batch.worker.0: shard 0: after AUTOLab+0d4edc83+2023-10-21-19h-08m-41s -- GPU allocated 5.25 GB, reserved 5.40 GB
19:57:55 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:57:55 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 125 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:57:55 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 1); retrying in 20s
19:58:13 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 27; measured free MB: {0: 19304, 1: 18808, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:58:16 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 2); retrying in 20s
/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '