logs/batch_pilot_deadlock.log

19:22:58 INFO    fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677})
19:22:58 INFO    fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:22:58 INFO    fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:22:58 INFO    fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
19:22:58 INFO    fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
19:22:58 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: device 1 claimed (measured free 24765 MB >= min 20000 MB); 1/1 workers now claimed on this device
19:22:58 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 19410, 1: 123825, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:23:01 INFO    fpgm.datagen.batch.worker.0: shard 0: starting on cuda:1 (visible as cuda:0), 4 episode(s), 8 thread(s)
19:23:02 INFO    fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
19:23:02 INFO    fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 128 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
19:23:03 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:23:03 INFO    fpgm.datagen.robot_buffers: robot_buffers: loaded 127 cached SAM masks from /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_buffers/sam_masks_full.npz
19:23:19 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:23:39 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:23:59 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:24:19 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:24:39 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:24:59 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 19300, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:25:11 INFO    fpgm.datagen.robot_buffers: robot_buffers: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=127  recall(median/p10)=0.950/0.876  shift(gated, systematic px)=1.24  shift(scatter, diagnostic px, std)=9.7  shift(scatter, diagnostic px, median|.|)=4.2  iou(mean, diagnostic)=0.768
   trajectory_h5_row0       n=127  recall(median/p10)=0.940/0.858  shift(gated, systematic px)=6.20  shift(scatter, diagnostic px, std)=9.8  shift(scatter, diagnostic px, median|.|)=7.0  iou(mean, diagnostic)=0.755
PASSED
19:25:20 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 8; measured free MB: {0: 19300, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:25:40 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 9; measured free MB: {0: 19300, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:25:55 INFO    fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
-> optimized_cameras_json   n=127  recall(median/p10)=0.950/0.876  shift(gated, systematic px)=1.24  shift(scatter, diagnostic px, std)=9.7  shift(scatter, diagnostic px, median|.|)=4.2  iou(mean, diagnostic)=0.768
   trajectory_h5_row0       n=127  recall(median/p10)=0.940/0.858  shift(gated, systematic px)=6.20  shift(scatter, diagnostic px, std)=9.8  shift(scatter, diagnostic px, median|.|)=7.0  iou(mean, diagnostic)=0.755
PASSED
19:26:00 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 10; measured free MB: {0: 19300, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:26:20 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 11; measured free MB: {0: 19300, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:26:40 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 12; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
19:26:47 INFO    fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_overlay.mp4
19:26:47 INFO    fpgm.datagen.background_depth: background depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/background_depth.h5
19:26:47 INFO    fpgm.datagen.dense_depth: dense depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/depth_dense.h5
19:26:47 INFO    fpgm.datagen.plate: plate: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/plate.png
19:26:47 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:26:48 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:26:48 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:26:48 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:26:49 INFO    fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
19:26:49 INFO    fpgm.datagen.model_registry: model registry: building sam3 on cuda
19:26:49 INFO    fpgm.datagen.model_registry: model registry: building tapnext on cuda
19:26:50 INFO    fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
19:26:50 INFO    fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.77 px, drift 1.50 px
19:26:50 INFO    fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.65 px, drift 0.81 px
19:26:51 INFO    fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
19:26:51 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:26:51 INFO    fpgm.geometry.rigid_lifting: rigid PnP solved 127/127 frames (median inlier fraction 100%, seed planarity 0.4001)
19:26:51 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): prismatic precondition passed (residual_m=0.00045 rotation_range_deg=1.903 n_frames=101) -- geometry is OBSERVED POINTS, no orientation fit anywhere in this path
19:26:51 INFO    fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
19:26:51 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): robot carve -- seed frame 0 mask/robot overlap BEFORE carving = 0/49393 px (0.00% of the seed mask); carved from every frame in static_span=(0, 30) before unioning (robot_seg supplied)
19:26:51 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): mask union px, robot-carved=3433 vs. pre-carve=3433 (+0 px removed by the carve)
19:26:51 INFO    fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): static_span=(0, 30) -- mask union 3433 px, 3433 valid (coverage 100.0%, of which 3433 genuinely measured/ANCHOR = 100.0% of the mask)
19:26:51 INFO    fpgm.datagen.pipeline: S6: drawer geometry built as an OBSERVED point cloud (no fitted mesh, no fitted orientation): 3433 points, coverage=100.0% of the mask (100.0% genuinely measured)
19:26:51 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): frame 0 -- 3077 cloud points, top 15% band -> 2353 points
19:26:52 INFO    fpgm.datagen.geometry_ops: fit_support_plane('drawer'): fit ok -- normal=[0.3019, 0.0422, 0.9524] offset=0.2202 rms=0.0054m n_inliers=1139/2353 footprint_ratio=0.88
19:26:52 INFO    fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=0.519 silhouette_iou(mean over valid)=0.743 rigid_lift={'planarity': 0.17042300800645957, 'n_rejected_implausible': 0, 'n_solved': 89, 'n_frames': 127} pose_source_counts={0: 87, 3: 40}
19:26:52 INFO    fpgm.datagen.pipeline: S6: drawer -- pose_noise_mm=0.137 silhouette_iou(mean over valid)=0.353 rigid_lift={'planarity': 0.40009308398436977, 'n_rejected_implausible': 0, 'n_solved': 127, 'n_frames': 127} pose_source_counts={0: 127}
19:26:52 INFO    fpgm.datagen.pipeline: S6: brick -- parent bridging via drawer: ['frames 87-126: parent=drawer attach rejected (pre-gap relative-pose score=0.064 < threshold=0.5)'] pose_source_counts={0: 87, 3: 40} silhouette_iou(mean over valid)=0.743
19:26:52 INFO    fpgm.datagen.events: EventLabeler.label: 40/127 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
19:26:54 INFO    fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/object_poses_debug.mp4
19:26:54 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/poses.npz
19:26:54 INFO    fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/events.json
/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 5 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '