Buckets:

twanghcmut
/

backup-foundation-physics

Files

xet

twanghcmut/backup-foundation-physics / logs /batch_v4.log

twanghcmut

9 days ago

download

raw

101 kB

	19:30:39 INFO fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677})
	19:30:39 INFO fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
	19:30:39 INFO fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
	19:30:39 INFO fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
	19:30:39 INFO fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
	19:30:39 INFO fpgm.datagen.batch.gpu_pool: gpu claim: device 1 claimed (measured free 24765 MB >= min 20000 MB); 1/1 workers now claimed on this device
	19:30:39 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:30:42 INFO fpgm.datagen.batch.worker.0: shard 0: starting on cuda:1 (visible as cuda:0), 4 episode(s), 8 thread(s)
	19:30:44 INFO fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
	19:30:44 INFO fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 128 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
	19:30:44 INFO fpgm.datagen.pipeline: picked GPU 1 (25368 MB free >= 20000 MB required)
	19:30:45 INFO fpgm.datagen.robot_buffers: robot_buffers: loaded 127 cached SAM masks from /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_buffers/sam_masks_full.npz
	19:30:59 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:31:20 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:31:40 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:32:00 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:32:20 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:32:40 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 19304, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:32:48 INFO fpgm.datagen.robot_buffers: robot_buffers: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
	-> optimized_cameras_json n=127 recall(median/p10)=0.950/0.876 shift(gated, systematic px)=1.24 shift(scatter, diagnostic px, std)=9.7 shift(scatter, diagnostic px, median\|.\|)=4.2 iou(mean, diagnostic)=0.768
	trajectory_h5_row0 n=127 recall(median/p10)=0.940/0.858 shift(gated, systematic px)=6.20 shift(scatter, diagnostic px, std)=9.8 shift(scatter, diagnostic px, median\|.\|)=7.0 iou(mean, diagnostic)=0.755
	PASSED
	19:33:00 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 8; measured free MB: {0: 19304, 1: 24655, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:33:21 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 9; measured free MB: {0: 19304, 1: 24655, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:33:31 INFO fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=127, dropped=0, recall>=0.85, systematic_shift<=2.0px):
	-> optimized_cameras_json n=127 recall(median/p10)=0.950/0.876 shift(gated, systematic px)=1.24 shift(scatter, diagnostic px, std)=9.7 shift(scatter, diagnostic px, median\|.\|)=4.2 iou(mean, diagnostic)=0.768
	trajectory_h5_row0 n=127 recall(median/p10)=0.940/0.858 shift(gated, systematic px)=6.20 shift(scatter, diagnostic px, std)=9.8 shift(scatter, diagnostic px, median\|.\|)=7.0 iou(mean, diagnostic)=0.755
	PASSED
	19:33:41 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 10; measured free MB: {0: 19304, 1: 24655, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:34:01 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 11; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:34:21 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 12; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:34:22 INFO fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/robot_overlay.mp4
	19:34:23 INFO fpgm.datagen.background_depth: background depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/background_depth.h5
	19:34:23 INFO fpgm.datagen.dense_depth: dense depth: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/depth_dense.h5
	19:34:23 INFO fpgm.datagen.plate: plate: reusing cached /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/plate.png
	19:34:23 INFO fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
	19:34:23 INFO fpgm.datagen.model_registry: model registry: building sam3 on cuda
	19:34:23 INFO fpgm.datagen.model_registry: model registry: building tapnext on cuda
	19:34:24 INFO fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
	19:34:24 INFO fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
	19:34:26 INFO fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
	19:34:26 INFO fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.77 px, drift 1.50 px
	19:34:26 INFO fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.65 px, drift 0.81 px
	19:34:27 INFO fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
	19:34:27 INFO fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
	19:34:27 INFO fpgm.geometry.rigid_lifting: rigid PnP solved 127/127 frames (median inlier fraction 100%, seed planarity 0.4001)
	19:34:27 INFO fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): prismatic precondition passed (residual_m=0.00045 rotation_range_deg=1.903 n_frames=101) -- geometry is OBSERVED POINTS, no orientation fit anywhere in this path
	19:34:27 INFO fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
	19:34:27 INFO fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): robot carve -- seed frame 0 mask/robot overlap BEFORE carving = 0/49393 px (0.00% of the seed mask); carved from every frame in static_span=(0, 30) before unioning (robot_seg supplied)
	19:34:27 INFO fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): mask union px, robot-carved=3433 vs. pre-carve=3433 (+0 px removed by the carve)
	19:34:27 INFO fpgm.datagen.geometry_ops: build_observed_surface_cloud('drawer'): static_span=(0, 30) -- mask union 3433 px, 3433 valid (coverage 100.0%, of which 3433 genuinely measured/ANCHOR = 100.0% of the mask)
	19:34:27 INFO fpgm.datagen.pipeline: S6: drawer geometry built as an OBSERVED point cloud (no fitted mesh, no fitted orientation): 3433 points, coverage=100.0% of the mask (100.0% genuinely measured)
	19:34:27 INFO fpgm.datagen.geometry_ops: fit_support_plane('drawer'): frame 0 -- 3077 cloud points, top 15% band -> 2353 points
	19:34:27 INFO fpgm.datagen.geometry_ops: fit_support_plane('drawer'): fit ok -- normal=[0.3019, 0.0422, 0.9524] offset=0.2202 rms=0.0054m n_inliers=1139/2353 footprint_ratio=0.88
	19:34:28 INFO fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=0.519 silhouette_iou(mean over valid)=0.743 rigid_lift={'planarity': 0.17042300800645957, 'n_rejected_implausible': 0, 'n_solved': 89, 'n_frames': 127} pose_source_counts={0: 87, 3: 40}
	19:34:28 INFO fpgm.datagen.pipeline: S6: drawer -- pose_noise_mm=0.137 silhouette_iou(mean over valid)=0.353 rigid_lift={'planarity': 0.40009308398436977, 'n_rejected_implausible': 0, 'n_solved': 127, 'n_frames': 127} pose_source_counts={0: 127}
	19:34:28 INFO fpgm.datagen.pipeline: S6: brick -- parent bridging via drawer: ['frames 87-126: parent=drawer attach rejected (pre-gap relative-pose score=0.064 < threshold=0.5)'] pose_source_counts={0: 87, 3: 40} silhouette_iou(mean over valid)=0.743
	19:34:28 INFO fpgm.datagen.events: EventLabeler.label: 40/127 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
	19:34:30 INFO fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/object_poses_debug.mp4
	19:34:30 INFO fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/poses.npz
	19:34:30 INFO fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760/master/events.json
	19:34:31 INFO fpgm.datagen.export_vace: S8: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s/22008760 -- 127 video frames -> 3 window(s) (window=81 stride=40): [(0, 81), (40, 121), (46, 127)] -- mesh objects=('brick',), point-cloud objects=('drawer',)
	19:34:31 INFO fpgm.datagen.export_vace: S8: carved 'drawer''s reference-frame (0) footprint out of the background plate -- 25609/399360 export pixels (6.4%)
	19:34:31 INFO fpgm.datagen.export_vace: S8: window [0, 81) up to date, skipping
	19:34:31 INFO fpgm.datagen.export_vace: S8: window [40, 121) up to date, skipping
	19:34:31 INFO fpgm.datagen.export_vace: S8: window [46, 127) up to date, skipping
	19:34:31 INFO fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-07m-04s -> ok (228.9s)
	19:34:32 INFO fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
	19:34:32 INFO fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 149 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
	19:34:32 INFO fpgm.datagen.pipeline: picked GPU 1 (25262 MB free >= 20000 MB required)
	19:34:33 INFO fpgm.pipeline.frames: extracted 148 frames [0, 147] stride=1 from 22008760.mp4 -> /tmp/fpgm_frames_s1b0yyc6
	/home/quang/miniconda3/envs/fpgm/lib/python3.12/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
	warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
	19:34:36 INFO fpgm.segmentation.sam3: building SAM 3.1 predictor (use_fa3=False)
	19:34:41 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 13; measured free MB: {0: 19304, 1: 23554, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	[0m[32mINFO 2026-08-01 19:34:52,329 4074910 sam3_multiplex_base.py: 336:[0m `setting max_num_objects` to 16 -- creating num_obj_for_compile=1 objects for torch.compile cache
	19:34:56 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: init_state does not accept ['offload_state_to_cpu']
	19:34:56 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: _build_sam2_output was discarding refined masks on frames with no prior cache entry, which breaks point-prompt video propagation
	dynamic_multimask_via_stability is reset to False in the multiplex model
	frame loading (image folder) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s][0m[32mINFO 2026-08-01 19:34:56,277 4074910 sam3_base_predictor.py: 146:[0m started new session b4f36dc1-8ade-4029-8cb7-25f3533ca045
	[0m[32mINFO 2026-08-01 19:34:56,277 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 74
	frame loading (image folder) [rank=0]: 3%\|▎ \| 5/148 [00:00<00:03, 41.81it/s] frame loading (image folder) [rank=0]: 7%\|▋ \| 10/148 [00:00<00:07, 17.61it/s] frame loading (image folder) [rank=0]: 9%\|▉ \| 13/148 [00:00<00:07, 18.90it/s] frame loading (image folder) [rank=0]: 11%\|█ \| 16/148 [00:00<00:06, 19.24it/s][0m[32mINFO 2026-08-01 19:34:57,042 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).

	propagate_in_video: 0%\| \| 0/74 [00:00<?, ?it/s][A frame loading (image folder) [rank=0]: 13%\|█▎ \| 19/148 [00:00<00:06, 18.61it/s] frame loading (image folder) [rank=0]: 15%\|█▍ \| 22/148 [00:01<00:07, 17.06it/s] frame loading (image folder) [rank=0]: 16%\|█▌ \| 24/148 [00:01<00:07, 16.70it/s] frame loading (image folder) [rank=0]: 18%\|█▊ \| 26/148 [00:01<00:07, 16.40it/s] frame loading (image folder) [rank=0]: 19%\|█▉ \| 28/148 [00:01<00:07, 15.94it/s] frame loading (image folder) [rank=0]: 22%\|██▏ \| 32/148 [00:01<00:05, 20.82it/s] frame loading (image folder) [rank=0]: 24%\|██▎ \| 35/148 [00:01<00:05, 21.60it/s] frame loading (image folder) [rank=0]: 26%\|██▌ \| 38/148 [00:02<00:05, 18.37it/s] frame loading (image folder) [rank=0]: 28%\|██▊ \| 41/148 [00:02<00:05, 19.36it/s]
	propagate_in_video: 1%\|▏ \| 1/74 [00:01<01:44, 1.43s/it][A frame loading (image folder) [rank=0]: 30%\|███ \| 45/148 [00:02<00:04, 21.59it/s] frame loading (image folder) [rank=0]: 32%\|███▏ \| 48/148 [00:02<00:04, 23.39it/s] frame loading (image folder) [rank=0]: 34%\|███▍ \| 51/148 [00:02<00:04, 20.91it/s] frame loading (image folder) [rank=0]: 36%\|███▋ \| 54/148 [00:02<00:04, 20.10it/s] frame loading (image folder) [rank=0]: 39%\|███▉ \| 58/148 [00:02<00:04, 21.03it/s] frame loading (image folder) [rank=0]: 41%\|████ \| 61/148 [00:03<00:05, 17.17it/s] frame loading (image folder) [rank=0]: 66%\|██████▌ \| 97/148 [00:03<00:00, 76.09it/s] frame loading (image folder) [rank=0]: 72%\|███████▏ \| 107/148 [00:03<00:00, 51.40it/s]
	propagate_in_video: 9%\|▉ \| 7/74 [00:02<00:25, 2.67it/s][A frame loading (image folder) [rank=0]: 78%\|███████▊ \| 115/148 [00:04<00:00, 39.55it/s] frame loading (image folder) [rank=0]: 82%\|████████▏ \| 121/148 [00:04<00:00, 36.81it/s] frame loading (image folder) [rank=0]: 85%\|████████▌ \| 126/148 [00:04<00:00, 36.62it/s]
	propagate_in_video: 31%\|███ \| 23/74 [00:03<00:06, 8.15it/s][A frame loading (image folder) [rank=0]: 89%\|████████▊ \| 131/148 [00:04<00:00, 27.52it/s] frame loading (image folder) [rank=0]: 91%\|█████████ \| 135/148 [00:04<00:00, 28.36it/s] frame loading (image folder) [rank=0]: 94%\|█████████▍\| 139/148 [00:05<00:00, 26.16it/s]
	propagate_in_video: 53%\|█████▎ \| 39/74 [00:04<00:02, 12.54it/s][A frame loading (image folder) [rank=0]: 97%\|█████████▋\| 143/148 [00:05<00:00, 23.81it/s] frame loading (image folder) [rank=0]: 99%\|█████████▉\| 147/148 [00:05<00:00, 25.20it/s] frame loading (image folder) [rank=0]: 100%\|██████████\| 148/148 [00:05<00:00, 26.71it/s]

	propagate_in_video: 74%\|███████▍ \| 55/74 [00:05<00:01, 15.52it/s][A19:35:02 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 14; measured free MB: {0: 19304, 1: 2632, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s

	propagate_in_video: 96%\|█████████▌\| 71/74 [00:05<00:00, 21.54it/s][A propagate_in_video: 100%\|██████████\| 74/74 [00:05<00:00, 13.82it/s]
	[0m[32mINFO 2026-08-01 19:35:02,399 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=True).
	propagate_in_video: 0%\| \| 0/74 [00:00<?, ?it/s] propagate_in_video: 1%\|▏ \| 1/74 [00:00<00:47, 1.53it/s] propagate_in_video: 15%\|█▍ \| 11/74 [00:01<00:06, 10.49it/s] propagate_in_video: 36%\|███▋ \| 27/74 [00:01<00:02, 17.65it/s] propagate_in_video: 58%\|█████▊ \| 43/74 [00:02<00:01, 20.81it/s] propagate_in_video: 80%\|███████▉ \| 59/74 [00:03<00:00, 21.86it/s] propagate_in_video: 100%\|██████████\| 74/74 [00:03<00:00, 23.31it/s]
	[0m[32mINFO 2026-08-01 19:35:05,576 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session b4f36dc1-8ade-4029-8cb7-25f3533ca045
	19:35:05 INFO fpgm.datagen.robot_buffers: robot_buffers: SAM3 prompt 'robot arm and gripper' found nothing at frame 74
	[0m[32mINFO 2026-08-01 19:35:06,318 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 21950889984 bytes (free_pct 0.2% -> 14.8%, reserved 27296530432 -> 5345640448 bytes)
	[0m[32mINFO 2026-08-01 19:35:06,318 4074910 sam3_base_predictor.py: 410:[0m removed session b4f36dc1-8ade-4029-8cb7-25f3533ca045
	frame loading (image folder) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s][0m[32mINFO 2026-08-01 19:35:06,425 4074910 sam3_base_predictor.py: 146:[0m started new session 728652b8-ddc9-4907-a245-0b95fb8de43d
	[0m[32mINFO 2026-08-01 19:35:06,425 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 74
	frame loading (image folder) [rank=0]: 2%\|▏ \| 3/148 [00:00<00:06, 23.17it/s][0m[32mINFO 2026-08-01 19:35:06,782 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).

	propagate_in_video: 0%\| \| 0/74 [00:00<?, ?it/s][A frame loading (image folder) [rank=0]: 4%\|▍ \| 6/148 [00:00<00:12, 11.13it/s] frame loading (image folder) [rank=0]: 5%\|▌ \| 8/148 [00:00<00:11, 12.64it/s] frame loading (image folder) [rank=0]: 8%\|▊ \| 12/148 [00:00<00:07, 18.31it/s] frame loading (image folder) [rank=0]: 10%\|█ \| 15/148 [00:00<00:06, 21.18it/s] frame loading (image folder) [rank=0]: 12%\|█▏ \| 18/148 [00:00<00:05, 23.20it/s] frame loading (image folder) [rank=0]: 14%\|█▍ \| 21/148 [00:01<00:06, 20.89it/s] frame loading (image folder) [rank=0]: 18%\|█▊ \| 27/148 [00:01<00:04, 28.43it/s] frame loading (image folder) [rank=0]: 21%\|██ \| 31/148 [00:01<00:04, 29.23it/s] frame loading (image folder) [rank=0]: 24%\|██▎ \| 35/148 [00:01<00:04, 24.21it/s]
	propagate_in_video: 1%\|▏ \| 1/74 [00:01<01:30, 1.24s/it][A frame loading (image folder) [rank=0]: 27%\|██▋ \| 40/148 [00:01<00:03, 28.11it/s]
	propagate_in_video: 7%\|▋ \| 5/74 [00:01<00:14, 4.81it/s][A frame loading (image folder) [rank=0]: 30%\|██▉ \| 44/148 [00:01<00:04, 24.91it/s] frame loading (image folder) [rank=0]: 32%\|███▏ \| 47/148 [00:02<00:04, 22.23it/s] frame loading (image folder) [rank=0]: 34%\|███▍ \| 50/148 [00:02<00:04, 23.36it/s] frame loading (image folder) [rank=0]: 37%\|███▋ \| 55/148 [00:02<00:03, 27.99it/s] frame loading (image folder) [rank=0]: 40%\|███▉ \| 59/148 [00:02<00:03, 23.11it/s] frame loading (image folder) [rank=0]: 42%\|████▏ \| 62/148 [00:02<00:04, 20.70it/s] frame loading (image folder) [rank=0]: 66%\|██████▌ \| 98/148 [00:02<00:00, 79.84it/s]
	propagate_in_video: 11%\|█ \| 8/74 [00:02<00:23, 2.80it/s][A frame loading (image folder) [rank=0]: 73%\|███████▎ \| 108/148 [00:03<00:00, 49.79it/s]
	propagate_in_video: 18%\|█▊ \| 13/74 [00:03<00:11, 5.48it/s][A
	propagate_in_video: 23%\|██▎ \| 17/74 [00:03<00:07, 8.00it/s][A
	propagate_in_video: 30%\|██▉ \| 22/74 [00:03<00:04, 11.85it/s][A frame loading (image folder) [rank=0]: 78%\|███████▊ \| 116/148 [00:03<00:00, 37.08it/s] frame loading (image folder) [rank=0]: 82%\|████████▏ \| 122/148 [00:03<00:00, 37.22it/s] frame loading (image folder) [rank=0]: 86%\|████████▋ \| 128/148 [00:04<00:00, 35.10it/s] frame loading (image folder) [rank=0]: 90%\|████████▉ \| 133/148 [00:04<00:00, 32.29it/s]
	propagate_in_video: 35%\|███▌ \| 26/74 [00:04<00:05, 8.61it/s][A
	propagate_in_video: 39%\|███▉ \| 29/74 [00:04<00:04, 10.18it/s][A frame loading (image folder) [rank=0]: 93%\|█████████▎\| 137/148 [00:04<00:00, 27.14it/s]
	propagate_in_video: 43%\|████▎ \| 32/74 [00:04<00:03, 11.22it/s][A frame loading (image folder) [rank=0]: 95%\|█████████▌\| 141/148 [00:04<00:00, 24.53it/s]
	propagate_in_video: 49%\|████▊ \| 36/74 [00:04<00:02, 14.43it/s][A frame loading (image folder) [rank=0]: 97%\|█████████▋\| 144/148 [00:05<00:00, 21.21it/s] frame loading (image folder) [rank=0]: 99%\|█████████▉\| 147/148 [00:05<00:00, 18.87it/s] frame loading (image folder) [rank=0]: 100%\|██████████\| 148/148 [00:05<00:00, 27.96it/s]

	propagate_in_video: 53%\|█████▎ \| 39/74 [00:05<00:03, 8.79it/s][A
	propagate_in_video: 58%\|█████▊ \| 43/74 [00:05<00:02, 11.83it/s][A
	propagate_in_video: 66%\|██████▌ \| 49/74 [00:05<00:01, 17.34it/s][A
	propagate_in_video: 73%\|███████▎ \| 54/74 [00:05<00:00, 22.11it/s][A
	propagate_in_video: 78%\|███████▊ \| 58/74 [00:06<00:01, 12.78it/s][A
	propagate_in_video: 86%\|████████▋ \| 64/74 [00:06<00:00, 17.80it/s][A
	propagate_in_video: 93%\|█████████▎\| 69/74 [00:06<00:00, 21.81it/s][A
	propagate_in_video: 99%\|█████████▊\| 73/74 [00:06<00:00, 20.42it/s][A propagate_in_video: 100%\|██████████\| 74/74 [00:06<00:00, 11.08it/s]
	[0m[32mINFO 2026-08-01 19:35:13,465 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 6.25%, subscription rate: 100.00%
	[0m[32mINFO 2026-08-01 19:35:13,466 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=True).
	propagate_in_video: 0%\| \| 0/74 [00:00<?, ?it/s] propagate_in_video: 1%\|▏ \| 1/74 [00:00<00:41, 1.78it/s] propagate_in_video: 7%\|▋ \| 5/74 [00:00<00:07, 9.44it/s] propagate_in_video: 15%\|█▍ \| 11/74 [00:01<00:07, 8.84it/s] propagate_in_video: 20%\|██ \| 15/74 [00:01<00:04, 12.48it/s] propagate_in_video: 26%\|██▌ \| 19/74 [00:01<00:03, 16.20it/s] propagate_in_video: 34%\|███▍ \| 25/74 [00:01<00:02, 23.52it/s] propagate_in_video: 39%\|███▉ \| 29/74 [00:02<00:03, 12.99it/s] propagate_in_video: 46%\|████▌ \| 34/74 [00:02<00:02, 17.17it/s] propagate_in_video: 53%\|█████▎ \| 39/74 [00:02<00:01, 21.71it/s] propagate_in_video: 58%\|█████▊ \| 43/74 [00:03<00:02, 12.06it/s] propagate_in_video: 62%\|██████▏ \| 46/74 [00:03<00:02, 12.97it/s] propagate_in_video: 68%\|██████▊ \| 50/74 [00:03<00:01, 15.66it/s] propagate_in_video: 72%\|███████▏ \| 53/74 [00:03<00:01, 17.68it/s] propagate_in_video: 77%\|███████▋ \| 57/74 [00:03<00:00, 20.87it/s] propagate_in_video: 81%\|████████ \| 60/74 [00:04<00:01, 10.44it/s] propagate_in_video: 86%\|████████▋ \| 64/74 [00:04<00:00, 13.74it/s] propagate_in_video: 95%\|█████████▍\| 70/74 [00:04<00:00, 19.71it/s] propagate_in_video: 100%\|██████████\| 74/74 [00:04<00:00, 15.37it/s]
	[0m[32mINFO 2026-08-01 19:35:18,289 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 12.50%, subscription rate: 200.00%
	[0m[32mINFO 2026-08-01 19:35:18,290 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 728652b8-ddc9-4907-a245-0b95fb8de43d
	[0m[32mINFO 2026-08-01 19:35:18,863 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 21569208320 bytes (free_pct 0.4% -> 14.8%, reserved 26950500352 -> 5381292032 bytes)
	[0m[32mINFO 2026-08-01 19:35:18,863 4074910 sam3_base_predictor.py: 410:[0m removed session 728652b8-ddc9-4907-a245-0b95fb8de43d
	19:35:18 INFO fpgm.datagen.robot_buffers: robot_buffers: SAM3 prompt 'robotic arm' tracked 117/148 frames with a non-empty mask
	19:35:19 INFO fpgm.datagen.robot_buffers: robot_buffers: cached 117 SAM masks to /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/robot_buffers/sam_masks_full.npz
	19:35:19 WARNING fpgm.datagen.robot_buffers: robot_buffers: dropped 3/117 SAM-tracked frames as failed-tracking (area < 20% of median): [31, 146, 147]
	19:35:22 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 15; measured free MB: {0: 19304, 1: 21138, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:35:42 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 16; measured free MB: {0: 19304, 1: 21138, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:36:02 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 17; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:36:23 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 18; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:36:43 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 19; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:37:03 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 20; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:37:14 INFO fpgm.datagen.robot_buffers: robot_buffers: robot alignment gate (prompt='robotic arm', sam_tracked=117, dropped=3, recall>=0.85, systematic_shift<=2.0px):
	-> optimized_cameras_json n=114 recall(median/p10)=0.970/0.921 shift(gated, systematic px)=0.54 shift(scatter, diagnostic px, std)=9.6 shift(scatter, diagnostic px, median\|.\|)=5.2 iou(mean, diagnostic)=0.658
	trajectory_h5_row0 n=114 recall(median/p10)=0.965/0.905 shift(gated, systematic px)=8.12 shift(scatter, diagnostic px, std)=10.6 shift(scatter, diagnostic px, median\|.\|)=8.0 iou(mean, diagnostic)=0.642
	PASSED
	19:37:23 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 21; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:37:43 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 22; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:38:04 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 23; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:38:12 INFO fpgm.datagen.pipeline: robot alignment gate (prompt='robotic arm', sam_tracked=117, dropped=3, recall>=0.85, systematic_shift<=2.0px):
	-> optimized_cameras_json n=114 recall(median/p10)=0.970/0.921 shift(gated, systematic px)=0.54 shift(scatter, diagnostic px, std)=9.6 shift(scatter, diagnostic px, median\|.\|)=5.2 iou(mean, diagnostic)=0.658
	trajectory_h5_row0 n=114 recall(median/p10)=0.965/0.905 shift(gated, systematic px)=8.12 shift(scatter, diagnostic px, std)=10.6 shift(scatter, diagnostic px, median\|.\|)=8.0 iou(mean, diagnostic)=0.642
	PASSED
	19:38:24 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 24; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:38:44 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 25; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:39:04 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 26; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:39:15 INFO fpgm.datagen.pipeline: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/robot_overlay.mp4
	19:39:24 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 27; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:39:26 INFO fpgm.datagen.pipeline: picked GPU 1 (24096 MB free >= 20000 MB required)
	19:39:26 INFO fpgm.segmentation.sam3: building SAM 3.1 predictor (use_fa3=False)
	[0m[32mINFO 2026-08-01 19:39:41,444 4074910 sam3_multiplex_base.py: 336:[0m `setting max_num_objects` to 16 -- creating num_obj_for_compile=1 objects for torch.compile cache
	19:39:44 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: init_state does not accept ['offload_state_to_cpu']
	19:39:44 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: _build_sam2_output was discarding refined masks on frames with no prior cache entry, which breaks point-prompt video propagation
	dynamic_multimask_via_stability is reset to False in the multiplex model
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 9%\|▉ \| 14/148 [00:00<00:00, 137.53it/s] frame loading (OpenCV) [rank=0]: 20%\|█▉ \| 29/148 [00:00<00:00, 143.32it/s] frame loading (OpenCV) [rank=0]: 34%\|███▍ \| 50/148 [00:00<00:00, 171.55it/s] frame loading (OpenCV) [rank=0]: 52%\|█████▏ \| 77/148 [00:00<00:00, 209.56it/s] frame loading (OpenCV) [rank=0]: 68%\|██████▊ \| 101/148 [00:00<00:00, 216.80it/s] frame loading (OpenCV) [rank=0]: 83%\|████████▎ \| 123/148 [00:00<00:00, 206.31it/s]19:39:45 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 28; measured free MB: {0: 19304, 1: 23492, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	frame loading (OpenCV) [rank=0]: 97%\|█████████▋\| 144/148 [00:00<00:00, 194.39it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 188.77it/s]
	[0m[32mINFO 2026-08-01 19:39:46,050 4074910 sam3_base_predictor.py: 146:[0m started new session afe9866b-b9ee-4976-a1b2-cff8f54041e8
	[0m[32mINFO 2026-08-01 19:39:46,051 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:39:46,253 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:35, 1.54it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 16.06it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 19.71it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.49it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.27it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 23.69it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.17it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:05<00:01, 24.86it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.44it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.47it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.33it/s]
	[0m[32mINFO 2026-08-01 19:39:52,097 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:39:52,097 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session afe9866b-b9ee-4976-a1b2-cff8f54041e8
	19:39:52 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'brick' matched nothing
	[0m[32mINFO 2026-08-01 19:39:52,809 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 20105396224 bytes (free_pct 2.2% -> 15.6%, reserved 26759659520 -> 6654263296 bytes)
	[0m[32mINFO 2026-08-01 19:39:52,809 4074910 sam3_base_predictor.py: 410:[0m removed session afe9866b-b9ee-4976-a1b2-cff8f54041e8
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 10%\|█ \| 15/148 [00:00<00:00, 146.51it/s] frame loading (OpenCV) [rank=0]: 25%\|██▌ \| 37/148 [00:00<00:00, 184.71it/s] frame loading (OpenCV) [rank=0]: 41%\|████ \| 60/148 [00:00<00:00, 204.24it/s] frame loading (OpenCV) [rank=0]: 55%\|█████▍ \| 81/148 [00:00<00:00, 199.27it/s] frame loading (OpenCV) [rank=0]: 68%\|██████▊ \| 101/148 [00:00<00:00, 190.61it/s] frame loading (OpenCV) [rank=0]: 82%\|████████▏ \| 121/148 [00:00<00:00, 172.22it/s] frame loading (OpenCV) [rank=0]: 94%\|█████████▍\| 139/148 [00:00<00:00, 171.12it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 174.32it/s]
	[0m[32mINFO 2026-08-01 19:39:54,478 4074910 sam3_base_predictor.py: 146:[0m started new session d07e7daf-d389-4b04-8c90-dd5f3f935a7c
	[0m[32mINFO 2026-08-01 19:39:54,479 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:39:54,608 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:29, 1.64it/s] propagate_in_video: 5%\|▍ \| 7/148 [00:00<00:11, 12.45it/s] propagate_in_video: 9%\|▉ \| 13/148 [00:00<00:06, 22.37it/s] propagate_in_video: 12%\|█▏ \| 18/148 [00:01<00:10, 12.74it/s] propagate_in_video: 15%\|█▍ \| 22/148 [00:01<00:07, 16.17it/s] propagate_in_video: 18%\|█▊ \| 27/148 [00:01<00:05, 21.04it/s] propagate_in_video: 21%\|██ \| 31/148 [00:01<00:04, 24.38it/s] propagate_in_video: 24%\|██▎ \| 35/148 [00:02<00:09, 11.93it/s] propagate_in_video: 27%\|██▋ \| 40/148 [00:02<00:06, 15.98it/s] propagate_in_video: 30%\|███ \| 45/148 [00:02<00:05, 20.58it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:03<00:08, 12.34it/s] propagate_in_video: 37%\|███▋ \| 55/148 [00:03<00:05, 17.39it/s] propagate_in_video: 40%\|███▉ \| 59/148 [00:03<00:04, 20.33it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:04<00:06, 13.61it/s] propagate_in_video: 47%\|████▋ \| 70/148 [00:04<00:04, 17.31it/s] propagate_in_video: 51%\|█████ \| 75/148 [00:04<00:03, 21.42it/s] propagate_in_video: 54%\|█████▍ \| 80/148 [00:04<00:02, 25.80it/s] propagate_in_video: 57%\|█████▋ \| 85/148 [00:05<00:04, 14.61it/s] propagate_in_video: 61%\|██████▏ \| 91/148 [00:05<00:03, 18.98it/s] propagate_in_video: 65%\|██████▍ \| 96/148 [00:05<00:02, 22.89it/s] propagate_in_video: 68%\|██████▊ \| 100/148 [00:06<00:03, 13.04it/s] propagate_in_video: 72%\|███████▏ \| 106/148 [00:06<00:02, 17.67it/s] propagate_in_video: 75%\|███████▌ \| 111/148 [00:06<00:01, 21.30it/s] propagate_in_video: 78%\|███████▊ \| 115/148 [00:07<00:02, 12.10it/s] propagate_in_video: 82%\|████████▏ \| 121/148 [00:07<00:01, 16.73it/s] propagate_in_video: 85%\|████████▌ \| 126/148 [00:07<00:01, 19.73it/s] propagate_in_video: 88%\|████████▊ \| 130/148 [00:08<00:01, 11.70it/s] propagate_in_video: 90%\|████████▉ \| 133/148 [00:08<00:01, 13.14it/s] propagate_in_video: 93%\|█████████▎\| 137/148 [00:08<00:00, 16.20it/s] propagate_in_video: 96%\|█████████▌\| 142/148 [00:08<00:00, 20.66it/s] propagate_in_video: 99%\|█████████▊\| 146/148 [00:08<00:00, 18.44it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:08<00:00, 16.51it/s]
	[0m[32mINFO 2026-08-01 19:40:03,572 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 6.25%, subscription rate: 100.00%
	[0m[32mINFO 2026-08-01 19:40:03,573 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:03,573 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session d07e7daf-d389-4b04-8c90-dd5f3f935a7c
	19:40:03 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'blue brick' -> score 0.6908
	[0m[32mINFO 2026-08-01 19:40:04,355 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 20898119680 bytes (free_pct 1.6% -> 15.5%, reserved 27609006080 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:04,355 4074910 sam3_base_predictor.py: 410:[0m removed session d07e7daf-d389-4b04-8c90-dd5f3f935a7c
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 10%\|█ \| 15/148 [00:00<00:00, 149.36it/s] frame loading (OpenCV) [rank=0]: 20%\|██ \| 30/148 [00:00<00:00, 141.97it/s] frame loading (OpenCV) [rank=0]: 30%\|███ \| 45/148 [00:00<00:00, 126.53it/s] frame loading (OpenCV) [rank=0]: 41%\|████ \| 61/148 [00:00<00:00, 137.04it/s] frame loading (OpenCV) [rank=0]: 54%\|█████▍ \| 80/148 [00:00<00:00, 154.04it/s] frame loading (OpenCV) [rank=0]: 65%\|██████▍ \| 96/148 [00:00<00:00, 153.37it/s] frame loading (OpenCV) [rank=0]: 76%\|███████▋ \| 113/148 [00:00<00:00, 153.69it/s] frame loading (OpenCV) [rank=0]: 87%\|████████▋ \| 129/148 [00:00<00:00, 147.37it/s]19:40:05 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 29; measured free MB: {0: 19304, 1: 22224, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	frame loading (OpenCV) [rank=0]: 99%\|█████████▊\| 146/148 [00:00<00:00, 153.47it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:01<00:00, 144.55it/s]
	[0m[32mINFO 2026-08-01 19:40:06,283 4074910 sam3_base_predictor.py: 146:[0m started new session 1eb5b1fa-8089-4f1d-9728-cdcf07cc7db0
	[0m[32mINFO 2026-08-01 19:40:06,283 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:40:06,403 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:15, 1.96it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:07, 18.55it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 20.37it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.15it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.71it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.39it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.67it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 25.63it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 26.52it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 33.69it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 26.35it/s]
	[0m[32mINFO 2026-08-01 19:40:12,021 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:12,021 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 1eb5b1fa-8089-4f1d-9728-cdcf07cc7db0
	19:40:12 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'red brick' matched nothing
	[0m[32mINFO 2026-08-01 19:40:12,827 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19937624064 bytes (free_pct 9.8% -> 23.1%, reserved 26648510464 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:12,827 4074910 sam3_base_predictor.py: 410:[0m removed session 1eb5b1fa-8089-4f1d-9728-cdcf07cc7db0
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 6%\|▌ \| 9/148 [00:00<00:01, 89.37it/s] frame loading (OpenCV) [rank=0]: 14%\|█▎ \| 20/148 [00:00<00:01, 99.44it/s] frame loading (OpenCV) [rank=0]: 21%\|██ \| 31/148 [00:00<00:01, 103.71it/s] frame loading (OpenCV) [rank=0]: 30%\|███ \| 45/148 [00:00<00:00, 117.30it/s] frame loading (OpenCV) [rank=0]: 44%\|████▍ \| 65/148 [00:00<00:00, 145.51it/s] frame loading (OpenCV) [rank=0]: 57%\|█████▋ \| 85/148 [00:00<00:00, 161.90it/s] frame loading (OpenCV) [rank=0]: 69%\|██████▉ \| 102/148 [00:00<00:00, 151.14it/s] frame loading (OpenCV) [rank=0]: 80%\|███████▉ \| 118/148 [00:00<00:00, 133.07it/s] frame loading (OpenCV) [rank=0]: 93%\|█████████▎\| 137/148 [00:00<00:00, 145.69it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:01<00:00, 136.94it/s]
	[0m[32mINFO 2026-08-01 19:40:14,777 4074910 sam3_base_predictor.py: 146:[0m started new session 3c3ea9b3-8d61-4fcc-9abc-0225657e4533
	[0m[32mINFO 2026-08-01 19:40:14,778 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:40:14,939 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:23, 1.75it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:07, 17.81it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 22.06it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 21.99it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.92it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 25.24it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:01, 26.17it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 25.10it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.15it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.59it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 26.16it/s]
	[0m[32mINFO 2026-08-01 19:40:20,597 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:20,597 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 3c3ea9b3-8d61-4fcc-9abc-0225657e4533
	19:40:20 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'green brick' matched nothing
	[0m[32mINFO 2026-08-01 19:40:21,182 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19937624064 bytes (free_pct 9.8% -> 23.1%, reserved 26648510464 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:21,182 4074910 sam3_base_predictor.py: 410:[0m removed session 3c3ea9b3-8d61-4fcc-9abc-0225657e4533
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 7%\|▋ \| 10/148 [00:00<00:01, 94.16it/s] frame loading (OpenCV) [rank=0]: 16%\|█▌ \| 24/148 [00:00<00:01, 118.03it/s] frame loading (OpenCV) [rank=0]: 27%\|██▋ \| 40/148 [00:00<00:00, 130.85it/s] frame loading (OpenCV) [rank=0]: 39%\|███▉ \| 58/148 [00:00<00:00, 148.71it/s] frame loading (OpenCV) [rank=0]: 49%\|████▉ \| 73/148 [00:00<00:00, 140.40it/s] frame loading (OpenCV) [rank=0]: 59%\|█████▉ \| 88/148 [00:00<00:00, 139.84it/s] frame loading (OpenCV) [rank=0]: 70%\|██████▉ \| 103/148 [00:00<00:00, 129.02it/s] frame loading (OpenCV) [rank=0]: 80%\|████████ \| 119/148 [00:00<00:00, 136.37it/s] frame loading (OpenCV) [rank=0]: 91%\|█████████ \| 134/148 [00:00<00:00, 139.73it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:01<00:00, 126.76it/s]
	[0m[32mINFO 2026-08-01 19:40:23,312 4074910 sam3_base_predictor.py: 146:[0m started new session c5138c3f-2701-473f-8f73-5eb5dcebafdc
	[0m[32mINFO 2026-08-01 19:40:23,313 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:40:23,586 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:26, 1.69it/s] propagate_in_video: 6%\|▌ \| 9/148 [00:00<00:08, 16.52it/s] propagate_in_video: 10%\|█ \| 15/148 [00:00<00:05, 23.92it/s] propagate_in_video: 14%\|█▎ \| 20/148 [00:01<00:11, 11.13it/s] propagate_in_video: 16%\|█▌ \| 23/148 [00:01<00:09, 12.50it/s]19:40:25 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 30; measured free MB: {0: 19304, 1: 15246, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	propagate_in_video: 19%\|█▉ \| 28/148 [00:01<00:07, 16.70it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:02<00:10, 10.52it/s] propagate_in_video: 26%\|██▌ \| 38/148 [00:02<00:07, 14.13it/s] propagate_in_video: 30%\|██▉ \| 44/148 [00:02<00:05, 19.44it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:03<00:08, 12.37it/s] propagate_in_video: 37%\|███▋ \| 55/148 [00:03<00:05, 16.70it/s] propagate_in_video: 41%\|████ \| 61/148 [00:03<00:04, 21.70it/s] propagate_in_video: 45%\|████▍ \| 66/148 [00:04<00:05, 13.99it/s] propagate_in_video: 49%\|████▉ \| 73/148 [00:04<00:03, 19.36it/s] propagate_in_video: 53%\|█████▎ \| 78/148 [00:04<00:03, 23.10it/s] propagate_in_video: 56%\|█████▌ \| 83/148 [00:05<00:04, 15.27it/s] propagate_in_video: 60%\|██████ \| 89/148 [00:05<00:02, 19.89it/s] propagate_in_video: 64%\|██████▎ \| 94/148 [00:05<00:02, 23.74it/s] propagate_in_video: 67%\|██████▋ \| 99/148 [00:06<00:03, 14.76it/s] propagate_in_video: 70%\|██████▉ \| 103/148 [00:06<00:02, 17.22it/s] propagate_in_video: 73%\|███████▎ \| 108/148 [00:06<00:01, 21.50it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:07<00:02, 13.11it/s] propagate_in_video: 79%\|███████▉ \| 117/148 [00:07<00:01, 15.77it/s] propagate_in_video: 83%\|████████▎ \| 123/148 [00:07<00:01, 20.77it/s] propagate_in_video: 86%\|████████▌ \| 127/148 [00:07<00:00, 23.54it/s] propagate_in_video: 89%\|████████▊ \| 131/148 [00:08<00:01, 12.76it/s] propagate_in_video: 93%\|█████████▎\| 137/148 [00:08<00:00, 17.64it/s] propagate_in_video: 96%\|█████████▌\| 142/148 [00:08<00:00, 21.34it/s] propagate_in_video: 99%\|█████████▊\| 146/148 [00:08<00:00, 18.98it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:08<00:00, 16.65it/s]
	[0m[32mINFO 2026-08-01 19:40:32,483 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 12.50%, subscription rate: 200.00%
	[0m[32mINFO 2026-08-01 19:40:32,484 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:32,484 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session c5138c3f-2701-473f-8f73-5eb5dcebafdc
	19:40:32 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'blue block' -> score 0.8750
	[0m[32mINFO 2026-08-01 19:40:33,102 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 20394803200 bytes (free_pct 9.5% -> 23.1%, reserved 27105689600 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:33,102 4074910 sam3_base_predictor.py: 410:[0m removed session c5138c3f-2701-473f-8f73-5eb5dcebafdc
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 5%\|▌ \| 8/148 [00:00<00:01, 75.49it/s] frame loading (OpenCV) [rank=0]: 18%\|█▊ \| 26/148 [00:00<00:00, 132.93it/s] frame loading (OpenCV) [rank=0]: 30%\|██▉ \| 44/148 [00:00<00:00, 153.76it/s] frame loading (OpenCV) [rank=0]: 41%\|████ \| 60/148 [00:00<00:00, 130.26it/s] frame loading (OpenCV) [rank=0]: 50%\|█████ \| 74/148 [00:00<00:00, 131.18it/s] frame loading (OpenCV) [rank=0]: 59%\|█████▉ \| 88/148 [00:00<00:00, 127.07it/s] frame loading (OpenCV) [rank=0]: 70%\|██████▉ \| 103/148 [00:00<00:00, 132.54it/s] frame loading (OpenCV) [rank=0]: 80%\|███████▉ \| 118/148 [00:00<00:00, 133.92it/s] frame loading (OpenCV) [rank=0]: 89%\|████████▉ \| 132/148 [00:01<00:00, 125.90it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:01<00:00, 130.35it/s]
	[0m[32mINFO 2026-08-01 19:40:35,240 4074910 sam3_base_predictor.py: 146:[0m started new session 881dadc8-3457-4167-b33b-f146d5e71361
	[0m[32mINFO 2026-08-01 19:40:35,241 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:40:35,424 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:22, 1.77it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 15.03it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 20.45it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.88it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.95it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.41it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.85it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 24.99it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.51it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.83it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.76it/s]
	[0m[32mINFO 2026-08-01 19:40:41,170 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:41,171 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 881dadc8-3457-4167-b33b-f146d5e71361
	19:40:41 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'red block' matched nothing
	[0m[32mINFO 2026-08-01 19:40:41,762 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19937624064 bytes (free_pct 9.8% -> 23.1%, reserved 26648510464 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:41,762 4074910 sam3_base_predictor.py: 410:[0m removed session 881dadc8-3457-4167-b33b-f146d5e71361
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 9%\|▉ \| 13/148 [00:00<00:01, 128.15it/s] frame loading (OpenCV) [rank=0]: 26%\|██▋ \| 39/148 [00:00<00:00, 204.63it/s] frame loading (OpenCV) [rank=0]: 41%\|████ \| 61/148 [00:00<00:00, 211.59it/s] frame loading (OpenCV) [rank=0]: 62%\|██████▏ \| 92/148 [00:00<00:00, 249.60it/s] frame loading (OpenCV) [rank=0]: 79%\|███████▉ \| 117/148 [00:00<00:00, 245.99it/s] frame loading (OpenCV) [rank=0]: 96%\|█████████▌\| 142/148 [00:00<00:00, 217.98it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 214.13it/s]
	[0m[32mINFO 2026-08-01 19:40:43,399 4074910 sam3_base_predictor.py: 146:[0m started new session 3cb2e1b8-8789-4759-8533-64f93369345f
	[0m[32mINFO 2026-08-01 19:40:43,400 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:40:43,564 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:19, 1.84it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:07, 16.60it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 20.65it/s]19:40:45 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 31; measured free MB: {0: 19304, 1: 14894, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.70it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 24.24it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 25.12it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 25.17it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 26.23it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.62it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.47it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 26.12it/s]
	[0m[32mINFO 2026-08-01 19:40:49,232 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:40:49,232 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 3cb2e1b8-8789-4759-8533-64f93369345f
	19:40:49 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' phrase 'green block' matched nothing
	[0m[32mINFO 2026-08-01 19:40:49,855 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19937624064 bytes (free_pct 9.8% -> 23.1%, reserved 26648510464 -> 6710886400 bytes)
	[0m[32mINFO 2026-08-01 19:40:49,855 4074910 sam3_base_predictor.py: 410:[0m removed session 3cb2e1b8-8789-4759-8533-64f93369345f
	19:40:49 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' -> winning phrase 'blue block' (score 0.8750) among 7 candidates
	19:40:51 INFO fpgm.tracking.tapnext: loading TAPNext++ checkpoint from /srv/data/foundation-physics-graph-model/checkpoints/tapnextpp_512.ckpt
	19:41:00 INFO fpgm.datagen.object_masks: prompt_masks: role 'brick' -- path=prompt is_moving=True mean_displacement_px=192.04 (gate=6.0)
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 12%\|█▏ \| 18/148 [00:00<00:00, 173.54it/s] frame loading (OpenCV) [rank=0]: 27%\|██▋ \| 40/148 [00:00<00:00, 196.65it/s] frame loading (OpenCV) [rank=0]: 41%\|████ \| 60/148 [00:00<00:00, 180.46it/s] frame loading (OpenCV) [rank=0]: 53%\|█████▎ \| 79/148 [00:00<00:00, 178.87it/s] frame loading (OpenCV) [rank=0]: 66%\|██████▌ \| 98/148 [00:00<00:00, 177.33it/s] frame loading (OpenCV) [rank=0]: 78%\|███████▊ \| 116/148 [00:00<00:00, 171.44it/s] frame loading (OpenCV) [rank=0]: 92%\|█████████▏\| 136/148 [00:00<00:00, 178.95it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 167.68it/s]
	[0m[32mINFO 2026-08-01 19:41:02,375 4074910 sam3_base_predictor.py: 146:[0m started new session b36b124d-53d9-42c5-9ec3-ebf620e10b34
	[0m[32mINFO 2026-08-01 19:41:02,376 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:02,507 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:33, 1.58it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 15.11it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:02<00:06, 18.68it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.45it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 24.39it/s]19:41:05 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 32; measured free MB: {0: 19304, 1: 13282, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.28it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:01, 25.65it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 25.62it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.12it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.19it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.44it/s]
	[0m[32mINFO 2026-08-01 19:41:08,326 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:08,326 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session b36b124d-53d9-42c5-9ec3-ebf620e10b34
	19:41:08 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'drawer shelf' matched nothing
	[0m[32mINFO 2026-08-01 19:41:08,963 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19604176896 bytes (free_pct 8.9% -> 22.0%, reserved 27961327616 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:08,963 4074910 sam3_base_predictor.py: 410:[0m removed session b36b124d-53d9-42c5-9ec3-ebf620e10b34
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 14%\|█▍ \| 21/148 [00:00<00:00, 196.71it/s] frame loading (OpenCV) [rank=0]: 29%\|██▉ \| 43/148 [00:00<00:00, 207.13it/s] frame loading (OpenCV) [rank=0]: 47%\|████▋ \| 69/148 [00:00<00:00, 228.68it/s] frame loading (OpenCV) [rank=0]: 66%\|██████▌ \| 97/148 [00:00<00:00, 246.00it/s] frame loading (OpenCV) [rank=0]: 82%\|████████▏ \| 122/148 [00:00<00:00, 238.38it/s] frame loading (OpenCV) [rank=0]: 99%\|█████████▊\| 146/148 [00:00<00:00, 221.73it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 212.61it/s]
	[0m[32mINFO 2026-08-01 19:41:10,548 4074910 sam3_base_predictor.py: 146:[0m started new session 6086067d-9cbc-4994-9d55-c8a726bda24a
	[0m[32mINFO 2026-08-01 19:41:10,549 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:10,683 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:34, 1.56it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 15.25it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 20.68it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 21.97it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.33it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.60it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.58it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:05<00:01, 24.41it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 24.74it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 30.32it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 24.83it/s]
	[0m[32mINFO 2026-08-01 19:41:16,643 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:16,644 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 6086067d-9cbc-4994-9d55-c8a726bda24a
	19:41:16 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'blue shelf' matched nothing
	[0m[32mINFO 2026-08-01 19:41:17,218 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:17,218 4074910 sam3_base_predictor.py: 410:[0m removed session 6086067d-9cbc-4994-9d55-c8a726bda24a
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 5%\|▌ \| 8/148 [00:00<00:01, 77.32it/s] frame loading (OpenCV) [rank=0]: 14%\|█▎ \| 20/148 [00:00<00:01, 100.98it/s] frame loading (OpenCV) [rank=0]: 24%\|██▍ \| 36/148 [00:00<00:00, 122.85it/s] frame loading (OpenCV) [rank=0]: 36%\|███▋ \| 54/148 [00:00<00:00, 143.19it/s] frame loading (OpenCV) [rank=0]: 54%\|█████▍ \| 80/148 [00:00<00:00, 181.88it/s] frame loading (OpenCV) [rank=0]: 67%\|██████▋ \| 99/148 [00:00<00:00, 157.45it/s] frame loading (OpenCV) [rank=0]: 78%\|███████▊ \| 116/148 [00:00<00:00, 150.22it/s] frame loading (OpenCV) [rank=0]: 93%\|█████████▎\| 138/148 [00:00<00:00, 169.29it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 150.18it/s]
	[0m[32mINFO 2026-08-01 19:41:19,085 4074910 sam3_base_predictor.py: 146:[0m started new session 7cc2ee55-ad4d-47be-b5db-a06a06a797c8
	[0m[32mINFO 2026-08-01 19:41:19,086 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:19,218 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:18, 1.88it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 15.98it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 19.34it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 21.32it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 22.58it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.56it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 25.33it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 25.42it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.53it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.40it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.52it/s]
	[0m[32mINFO 2026-08-01 19:41:25,018 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:25,019 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 7cc2ee55-ad4d-47be-b5db-a06a06a797c8
	19:41:25 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'red shelf' matched nothing
	[0m[32mINFO 2026-08-01 19:41:25,621 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:25,621 4074910 sam3_base_predictor.py: 410:[0m removed session 7cc2ee55-ad4d-47be-b5db-a06a06a797c8
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 9%\|▉ \| 14/148 [00:00<00:01, 130.93it/s] frame loading (OpenCV) [rank=0]: 24%\|██▎ \| 35/148 [00:00<00:00, 171.02it/s] frame loading (OpenCV) [rank=0]: 36%\|███▌ \| 53/148 [00:00<00:00, 135.40it/s]19:41:26 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 33; measured free MB: {0: 19304, 1: 31446, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	frame loading (OpenCV) [rank=0]: 50%\|█████ \| 74/148 [00:00<00:00, 159.65it/s] frame loading (OpenCV) [rank=0]: 64%\|██████▍ \| 95/148 [00:00<00:00, 175.16it/s] frame loading (OpenCV) [rank=0]: 84%\|████████▍ \| 124/148 [00:00<00:00, 210.59it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 217.79it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 186.83it/s]
	[0m[32mINFO 2026-08-01 19:41:27,355 4074910 sam3_base_predictor.py: 146:[0m started new session 75a037ff-3255-411b-a3bd-6e8fb83acced
	[0m[32mINFO 2026-08-01 19:41:27,355 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:27,520 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:44, 1.41it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 14.83it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 19.88it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.92it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 22.77it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 23.76it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 23.90it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:05<00:01, 25.47it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.71it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 30.67it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 24.80it/s]
	[0m[32mINFO 2026-08-01 19:41:33,489 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:33,489 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 75a037ff-3255-411b-a3bd-6e8fb83acced
	19:41:33 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'green shelf' matched nothing
	[0m[32mINFO 2026-08-01 19:41:34,232 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:34,232 4074910 sam3_base_predictor.py: 410:[0m removed session 75a037ff-3255-411b-a3bd-6e8fb83acced
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 11%\|█ \| 16/148 [00:00<00:00, 155.35it/s] frame loading (OpenCV) [rank=0]: 26%\|██▌ \| 38/148 [00:00<00:00, 186.38it/s] frame loading (OpenCV) [rank=0]: 39%\|███▊ \| 57/148 [00:00<00:00, 170.22it/s] frame loading (OpenCV) [rank=0]: 55%\|█████▌ \| 82/148 [00:00<00:00, 197.65it/s] frame loading (OpenCV) [rank=0]: 70%\|██████▉ \| 103/148 [00:00<00:00, 183.68it/s] frame loading (OpenCV) [rank=0]: 82%\|████████▏ \| 122/148 [00:00<00:00, 182.68it/s] frame loading (OpenCV) [rank=0]: 98%\|█████████▊\| 145/148 [00:00<00:00, 193.07it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 182.76it/s]
	[0m[32mINFO 2026-08-01 19:41:36,044 4074910 sam3_base_predictor.py: 146:[0m started new session e4991630-f85a-410e-a394-8bfb71c45bca
	[0m[32mINFO 2026-08-01 19:41:36,045 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:36,377 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:42, 1.43it/s] propagate_in_video: 4%\|▍ \| 6/148 [00:00<00:15, 9.27it/s] propagate_in_video: 8%\|▊ \| 12/148 [00:00<00:07, 18.37it/s] propagate_in_video: 11%\|█ \| 16/148 [00:01<00:06, 21.37it/s] propagate_in_video: 14%\|█▎ \| 20/148 [00:01<00:11, 11.51it/s] propagate_in_video: 16%\|█▌ \| 24/148 [00:01<00:08, 14.95it/s] propagate_in_video: 20%\|█▉ \| 29/148 [00:01<00:05, 20.02it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:02<00:10, 11.35it/s] propagate_in_video: 25%\|██▌ \| 37/148 [00:02<00:07, 14.34it/s] propagate_in_video: 29%\|██▉ \| 43/148 [00:02<00:05, 20.27it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:03<00:06, 14.28it/s] propagate_in_video: 37%\|███▋ \| 55/148 [00:03<00:04, 19.12it/s] propagate_in_video: 41%\|████ \| 61/148 [00:03<00:03, 24.47it/s] propagate_in_video: 45%\|████▍ \| 66/148 [00:04<00:05, 14.95it/s] propagate_in_video: 47%\|████▋ \| 70/148 [00:04<00:04, 17.55it/s] propagate_in_video: 51%\|█████ \| 75/148 [00:04<00:03, 21.66it/s] propagate_in_video: 54%\|█████▍ \| 80/148 [00:04<00:02, 25.49it/s] propagate_in_video: 57%\|█████▋ \| 84/148 [00:05<00:04, 13.37it/s] propagate_in_video: 60%\|██████ \| 89/148 [00:05<00:03, 17.04it/s] propagate_in_video: 64%\|██████▎ \| 94/148 [00:05<00:02, 21.00it/s] propagate_in_video: 66%\|██████▌ \| 98/148 [00:06<00:03, 12.78it/s] propagate_in_video: 70%\|██████▉ \| 103/148 [00:06<00:02, 16.42it/s] propagate_in_video: 72%\|███████▏ \| 107/148 [00:06<00:02, 19.26it/s] propagate_in_video: 75%\|███████▌ \| 111/148 [00:06<00:01, 22.04it/s] propagate_in_video: 78%\|███████▊ \| 115/148 [00:07<00:02, 12.61it/s] propagate_in_video: 82%\|████████▏ \| 121/148 [00:07<00:01, 17.69it/s] propagate_in_video: 85%\|████████▌ \| 126/148 [00:07<00:01, 20.60it/s] propagate_in_video: 88%\|████████▊ \| 130/148 [00:08<00:01, 11.83it/s] propagate_in_video: 90%\|████████▉ \| 133/148 [00:08<00:01, 12.93it/s] propagate_in_video: 93%\|█████████▎\| 137/148 [00:08<00:00, 15.98it/s] propagate_in_video: 96%\|█████████▌\| 142/148 [00:08<00:00, 19.64it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:09<00:00, 16.57it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:09<00:00, 17.15it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:09<00:00, 16.15it/s]
	[0m[32mINFO 2026-08-01 19:41:45,566 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 50.00%, subscription rate: 800.00%
	[0m[32mINFO 2026-08-01 19:41:45,567 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:45,567 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session e4991630-f85a-410e-a394-8bfb71c45bca
	19:41:45 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'drawer' -> score 0.7300
	[0m[32mINFO 2026-08-01 19:41:46,124 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 20721958912 bytes (free_pct 8.2% -> 22.0%, reserved 29079109632 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:46,124 4074910 sam3_base_predictor.py: 410:[0m removed session e4991630-f85a-410e-a394-8bfb71c45bca
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 6%\|▌ \| 9/148 [00:00<00:01, 86.82it/s]19:41:46 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 34; measured free MB: {0: 19304, 1: 31444, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	frame loading (OpenCV) [rank=0]: 22%\|██▏ \| 33/148 [00:00<00:00, 174.68it/s] frame loading (OpenCV) [rank=0]: 34%\|███▍ \| 51/148 [00:00<00:00, 128.50it/s] frame loading (OpenCV) [rank=0]: 45%\|████▍ \| 66/148 [00:00<00:00, 112.12it/s] frame loading (OpenCV) [rank=0]: 53%\|█████▎ \| 79/148 [00:00<00:00, 114.73it/s] frame loading (OpenCV) [rank=0]: 62%\|██████▏ \| 92/148 [00:00<00:00, 115.92it/s] frame loading (OpenCV) [rank=0]: 71%\|███████ \| 105/148 [00:00<00:00, 112.95it/s] frame loading (OpenCV) [rank=0]: 80%\|███████▉ \| 118/148 [00:01<00:00, 115.07it/s] frame loading (OpenCV) [rank=0]: 92%\|█████████▏\| 136/148 [00:01<00:00, 131.00it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:01<00:00, 123.26it/s]
	[0m[32mINFO 2026-08-01 19:41:48,168 4074910 sam3_base_predictor.py: 146:[0m started new session 5785d48b-db6c-4edd-9acd-9160cdaa06f9
	[0m[32mINFO 2026-08-01 19:41:48,169 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:48,307 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:22, 1.78it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 16.11it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 20.40it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 21.85it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 22.65it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 23.61it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.57it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:05<00:01, 25.07it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.99it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 33.09it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.62it/s]
	[0m[32mINFO 2026-08-01 19:41:54,084 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:41:54,084 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 5785d48b-db6c-4edd-9acd-9160cdaa06f9
	19:41:54 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'blue drawer' matched nothing
	[0m[32mINFO 2026-08-01 19:41:54,714 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:41:54,714 4074910 sam3_base_predictor.py: 410:[0m removed session 5785d48b-db6c-4edd-9acd-9160cdaa06f9
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 8%\|▊ \| 12/148 [00:00<00:01, 108.33it/s] frame loading (OpenCV) [rank=0]: 22%\|██▏ \| 32/148 [00:00<00:00, 157.13it/s] frame loading (OpenCV) [rank=0]: 34%\|███▍ \| 50/148 [00:00<00:00, 165.62it/s] frame loading (OpenCV) [rank=0]: 45%\|████▌ \| 67/148 [00:00<00:00, 166.02it/s] frame loading (OpenCV) [rank=0]: 67%\|██████▋ \| 99/148 [00:00<00:00, 217.07it/s] frame loading (OpenCV) [rank=0]: 82%\|████████▏ \| 121/148 [00:00<00:00, 216.70it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 232.98it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 200.19it/s]
	[0m[32mINFO 2026-08-01 19:41:56,363 4074910 sam3_base_predictor.py: 146:[0m started new session 87a5071c-9c31-4533-b063-8f0d6f8a7633
	[0m[32mINFO 2026-08-01 19:41:56,364 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:41:56,512 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:42, 1.43it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:09, 14.24it/s] propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 19.55it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 21.41it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.27it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.84it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:01, 25.99it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 26.24it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.02it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 31.01it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.04it/s]
	[0m[32mINFO 2026-08-01 19:42:02,425 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:42:02,425 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session 87a5071c-9c31-4533-b063-8f0d6f8a7633
	19:42:02 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'red drawer' matched nothing
	[0m[32mINFO 2026-08-01 19:42:03,067 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:42:03,067 4074910 sam3_base_predictor.py: 410:[0m removed session 87a5071c-9c31-4533-b063-8f0d6f8a7633
	frame loading (OpenCV) [rank=0]: 0%\| \| 0/148 [00:00<?, ?it/s] frame loading (OpenCV) [rank=0]: 10%\|█ \| 15/148 [00:00<00:00, 149.21it/s] frame loading (OpenCV) [rank=0]: 29%\|██▉ \| 43/148 [00:00<00:00, 223.37it/s] frame loading (OpenCV) [rank=0]: 45%\|████▍ \| 66/148 [00:00<00:00, 198.59it/s] frame loading (OpenCV) [rank=0]: 60%\|██████ \| 89/148 [00:00<00:00, 205.09it/s] frame loading (OpenCV) [rank=0]: 74%\|███████▍ \| 110/148 [00:00<00:00, 188.27it/s] frame loading (OpenCV) [rank=0]: 90%\|████████▉ \| 133/148 [00:00<00:00, 199.24it/s] frame loading (OpenCV) [rank=0]: 100%\|██████████\| 148/148 [00:00<00:00, 193.39it/s]
	[0m[32mINFO 2026-08-01 19:42:04,673 4074910 sam3_base_predictor.py: 146:[0m started new session fe609818-3238-4ab9-8eda-fd8b481ea308
	[0m[32mINFO 2026-08-01 19:42:04,674 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 0
	[0m[32mINFO 2026-08-01 19:42:04,822 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).
	propagate_in_video: 0%\| \| 0/148 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/148 [00:00<01:34, 1.55it/s] propagate_in_video: 11%\|█▏ \| 17/148 [00:01<00:08, 15.27it/s]19:42:06 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 35; measured free MB: {0: 19304, 1: 13920, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	propagate_in_video: 22%\|██▏ \| 33/148 [00:01<00:05, 19.96it/s] propagate_in_video: 33%\|███▎ \| 49/148 [00:02<00:04, 22.85it/s] propagate_in_video: 44%\|████▍ \| 65/148 [00:03<00:03, 23.14it/s] propagate_in_video: 55%\|█████▍ \| 81/148 [00:03<00:02, 24.28it/s] propagate_in_video: 66%\|██████▌ \| 97/148 [00:04<00:02, 24.35it/s] propagate_in_video: 76%\|███████▋ \| 113/148 [00:04<00:01, 25.33it/s] propagate_in_video: 87%\|████████▋ \| 129/148 [00:05<00:00, 25.18it/s] propagate_in_video: 98%\|█████████▊\| 145/148 [00:05<00:00, 32.21it/s] propagate_in_video: 100%\|██████████\| 148/148 [00:05<00:00, 25.35it/s]
	[0m[32mINFO 2026-08-01 19:42:10,661 4074910 sam3_multiplex_tracking.py:2328:[0m Fetching existing VG predictions without running any propagation (reverse=True).
	0it [00:00, ?it/s] 0it [00:00, ?it/s]
	[0m[32mINFO 2026-08-01 19:42:10,662 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session fe609818-3238-4ab9-8eda-fd8b481ea308
	19:42:10 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' phrase 'green drawer' matched nothing
	[0m[32mINFO 2026-08-01 19:42:11,359 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 19562233856 bytes (free_pct 8.9% -> 22.0%, reserved 27919384576 -> 8357150720 bytes)
	[0m[32mINFO 2026-08-01 19:42:11,359 4074910 sam3_base_predictor.py: 410:[0m removed session fe609818-3238-4ab9-8eda-fd8b481ea308
	19:42:11 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' -> winning phrase 'drawer' (score 0.7300) among 8 candidates
	19:42:18 INFO fpgm.datagen.object_masks: prompt_masks: role 'drawer' -- path=prompt is_moving=True mean_displacement_px=102.41 (gate=6.0)
	19:42:25 INFO fpgm.datagen.object_masks: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/prompt_masks_debug.mp4
	19:42:25 INFO fpgm.datagen.pipeline: picked GPU 1 (23328 MB free >= 20000 MB required)
	19:42:26 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 36; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:42:27 INFO fpgm.datagen.object_masks: prompt_masks: reusing cached results for 2 roles
	19:42:28 INFO fpgm.datagen.static_span: static span brick: frames 0-30 (30 frames) via tracks, max step 0.76 px, drift 0.61 px
	19:42:28 INFO fpgm.datagen.static_span: static span drawer: frames 0-30 (30 frames) via tracks, max step 0.56 px, drift 0.47 px
	19:42:28 INFO fpgm.robot.kinematics: flange table built: g=0.0 offset=[0.0, 0.00024, 0.11498] sep=0.0941; g=1.0 offset=[0.0, 0.0003, 0.12627] sep=0.0162
	19:42:29 INFO fpgm.geometry.rigid_lifting: rigid seed: frame 0 with 300 anchor points (mean confidence 1.00)
	19:42:29 INFO fpgm.geometry.rigid_lifting: rigid PnP solved 146/148 frames (median inlier fraction 100%, seed planarity 0.3095)
	19:42:29 WARNING fpgm.datagen.pipeline: S6: fixture 'drawer' failed the prismatic precondition: build_observed_surface_cloud('drawer'): mesh-free precondition failed -- residual_m=0.00021, rotation_range_deg=0.644, n_frames=112 did not satisfy PrismaticFitter's gates; refusing to carry an observed point cloud rigidly for an object not confirmed prismatic -- see FixtureNotPrismaticError's own docstring
	19:42:29 INFO fpgm.datagen.pipeline: S6: brick -- pose_noise_mm=1.017 silhouette_iou(mean over valid)=0.706 rigid_lift={'planarity': 0.1736893205794351, 'n_rejected_implausible': 0, 'n_solved': 103, 'n_frames': 148} pose_source_counts={0: 98, 3: 50}
	19:42:29 WARNING fpgm.datagen.pipeline: S6: role 'drawer' has no mesh/geometry built; skipping
	19:42:29 INFO fpgm.datagen.events: EventLabeler.label: 50/148 frames missing pose data, gap-filling for windowed computations (see EventTimeline.valid for which)
	19:42:31 INFO fpgm.datagen.object_poses: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/object_poses_debug.mp4
	19:42:31 INFO fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/poses.npz
	19:42:31 INFO fpgm.datagen.geometry_ops: wrote /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/master/events.json
	19:42:32 INFO fpgm.datagen.export_vace: S8: AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760 -- 148 video frames -> 3 window(s) (window=81 stride=40): [(0, 81), (40, 121), (67, 148)] -- mesh objects=('brick',), point-cloud objects=()
	19:42:46 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 37; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:43:07 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 38; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:43:27 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 39; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:43:28 INFO fpgm.datagen.export_vace: S8: wrote window [0, 81) -- 81 frames, overlaps_pose_gap=False -> /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/vace/window_00000_00081
	19:43:47 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 40; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:44:07 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 41; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:44:28 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 42; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:44:48 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 43; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:44:50 INFO fpgm.datagen.export_vace: S8: wrote window [40, 121) -- 81 frames, overlaps_pose_gap=True -> /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/vace/window_00040_00121
	19:45:08 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 44; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:45:28 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 45; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:45:49 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 46; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:46:02 INFO fpgm.datagen.export_vace: S8: wrote window [67, 148) -- 81 frames, overlaps_pose_gap=True -> /srv/data/foundation-physics-graph-model/outputs/datagen/AUTOLab+0d4edc83+2023-10-21-19h-08m-08s/22008760/vace/window_00067_00148
	19:46:09 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 47; measured free MB: {0: 19304, 1: 22724, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:46:13 INFO fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-08m-08s -> ok (702.1s)
	19:46:14 INFO fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
	19:46:14 INFO fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 166 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
	19:46:15 INFO fpgm.datagen.pipeline: picked GPU 1 (23328 MB free >= 20000 MB required)
	19:46:15 INFO fpgm.pipeline.frames: extracted 165 frames [0, 164] stride=1 from 22008760.mp4 -> /tmp/fpgm_frames_9azp77th
	19:46:15 INFO fpgm.segmentation.sam3: building SAM 3.1 predictor (use_fa3=False)
	19:46:29 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 48; measured free MB: {0: 19304, 1: 21838, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	[0m[32mINFO 2026-08-01 19:46:30,485 4074910 sam3_multiplex_base.py: 336:[0m `setting max_num_objects` to 16 -- creating num_obj_for_compile=1 objects for torch.compile cache
	19:46:32 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: init_state does not accept ['offload_state_to_cpu']
	19:46:32 INFO fpgm.segmentation.sam3: applied sam3 compatibility shim: _build_sam2_output was discarding refined masks on frames with no prior cache entry, which breaks point-prompt video propagation
	dynamic_multimask_via_stability is reset to False in the multiplex model
	frame loading (image folder) [rank=0]: 0%\| \| 0/165 [00:00<?, ?it/s][0m[32mINFO 2026-08-01 19:46:33,136 4074910 sam3_base_predictor.py: 146:[0m started new session f67b1fcd-4ae8-4d43-92a9-7953a0848bec
	[0m[32mINFO 2026-08-01 19:46:33,137 4074910 sam3_multiplex_tracking.py:1677:[0m Running add_prompt on frame 82
	frame loading (image folder) [rank=0]: 2%\|▏ \| 3/165 [00:00<00:05, 28.46it/s] frame loading (image folder) [rank=0]: 4%\|▎ \| 6/165 [00:00<00:06, 24.06it/s] frame loading (image folder) [rank=0]: 6%\|▌ \| 10/165 [00:00<00:05, 28.19it/s][0m[32mINFO 2026-08-01 19:46:33,515 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=False).

	propagate_in_video: 0%\| \| 0/83 [00:00<?, ?it/s][A frame loading (image folder) [rank=0]: 8%\|▊ \| 13/165 [00:00<00:05, 26.10it/s] frame loading (image folder) [rank=0]: 10%\|▉ \| 16/165 [00:00<00:05, 24.98it/s] frame loading (image folder) [rank=0]: 12%\|█▏ \| 19/165 [00:00<00:05, 24.81it/s] frame loading (image folder) [rank=0]: 13%\|█▎ \| 22/165 [00:00<00:05, 24.96it/s] frame loading (image folder) [rank=0]: 15%\|█▌ \| 25/165 [00:01<00:06, 21.87it/s] frame loading (image folder) [rank=0]: 17%\|█▋ \| 28/165 [00:01<00:06, 21.50it/s] frame loading (image folder) [rank=0]: 19%\|█▉ \| 31/165 [00:01<00:05, 22.57it/s] frame loading (image folder) [rank=0]: 21%\|██ \| 35/165 [00:01<00:05, 25.32it/s] frame loading (image folder) [rank=0]: 24%\|██▎ \| 39/165 [00:01<00:04, 26.05it/s]
	propagate_in_video: 1%\| \| 1/83 [00:01<01:39, 1.21s/it][A frame loading (image folder) [rank=0]: 26%\|██▌ \| 43/165 [00:01<00:04, 29.00it/s] frame loading (image folder) [rank=0]: 28%\|██▊ \| 46/165 [00:01<00:04, 28.93it/s] frame loading (image folder) [rank=0]: 30%\|██▉ \| 49/165 [00:01<00:04, 26.50it/s] frame loading (image folder) [rank=0]: 32%\|███▏ \| 53/165 [00:02<00:03, 28.92it/s] frame loading (image folder) [rank=0]: 35%\|███▍ \| 57/165 [00:02<00:03, 31.14it/s] frame loading (image folder) [rank=0]: 38%\|███▊ \| 62/165 [00:02<00:03, 32.92it/s] frame loading (image folder) [rank=0]: 40%\|████ \| 66/165 [00:02<00:03, 27.96it/s] frame loading (image folder) [rank=0]: 42%\|████▏ \| 69/165 [00:02<00:03, 27.14it/s] frame loading (image folder) [rank=0]: 45%\|████▍ \| 74/165 [00:02<00:02, 30.82it/s]
	propagate_in_video: 18%\|█▊ \| 15/83 [00:02<00:08, 7.70it/s][A frame loading (image folder) [rank=0]: 47%\|████▋ \| 78/165 [00:02<00:02, 29.04it/s] frame loading (image folder) [rank=0]: 49%\|████▉ \| 81/165 [00:02<00:02, 28.79it/s] frame loading (image folder) [rank=0]: 71%\|███████ \| 117/165 [00:03<00:00, 108.16it/s] frame loading (image folder) [rank=0]: 78%\|███████▊ \| 129/165 [00:03<00:00, 53.86it/s] frame loading (image folder) [rank=0]: 84%\|████████▍ \| 139/165 [00:03<00:00, 47.91it/s]
	propagate_in_video: 37%\|███▋ \| 31/83 [00:03<00:05, 9.48it/s][A frame loading (image folder) [rank=0]: 89%\|████████▉ \| 147/165 [00:04<00:00, 40.96it/s] frame loading (image folder) [rank=0]: 93%\|█████████▎\| 153/165 [00:04<00:00, 32.81it/s] frame loading (image folder) [rank=0]: 96%\|█████████▌\| 158/165 [00:04<00:00, 34.00it/s]
	propagate_in_video: 57%\|█████▋ \| 47/83 [00:04<00:02, 13.48it/s][A frame loading (image folder) [rank=0]: 99%\|█████████▉\| 163/165 [00:04<00:00, 29.41it/s] frame loading (image folder) [rank=0]: 100%\|██████████\| 165/165 [00:04<00:00, 33.29it/s]

	propagate_in_video: 76%\|███████▌ \| 63/83 [00:05<00:01, 16.13it/s][A
	propagate_in_video: 81%\|████████ \| 67/83 [00:05<00:00, 17.41it/s][A
	propagate_in_video: 88%\|████████▊ \| 73/83 [00:05<00:00, 20.38it/s][A
	propagate_in_video: 94%\|█████████▍\| 78/83 [00:05<00:00, 19.68it/s][A
	propagate_in_video: 98%\|█████████▊\| 81/83 [00:05<00:00, 17.15it/s][A propagate_in_video: 100%\|██████████\| 83/83 [00:05<00:00, 14.16it/s]
	[0m[32mINFO 2026-08-01 19:46:39,377 4074910 sam3_multiplex_tracking.py: 580:[0m Bucket utilization rate: 6.25%, subscription rate: 100.00%
	[0m[32mINFO 2026-08-01 19:46:39,377 4074910 sam3_multiplex_tracking.py:2316:[0m Running full VG propagation (reverse=True).
	propagate_in_video: 0%\| \| 0/82 [00:00<?, ?it/s] propagate_in_video: 1%\| \| 1/82 [00:00<00:42, 1.89it/s] propagate_in_video: 4%\|▎ \| 3/82 [00:01<00:28, 2.76it/s] propagate_in_video: 11%\|█ \| 9/82 [00:01<00:07, 9.89it/s] propagate_in_video: 17%\|█▋ \| 14/82 [00:01<00:04, 15.58it/s] propagate_in_video: 22%\|██▏ \| 18/82 [00:02<00:07, 8.68it/s]
	[0m[32mINFO 2026-08-01 19:46:41,451 4074910 sam3_base_predictor.py: 305:[0m propagation ended in session f67b1fcd-4ae8-4d43-92a9-7953a0848bec
	[0m[32mINFO 2026-08-01 19:46:42,065 4074910 sam3_base_predictor.py: 398:[0m empty_cache freed 2705326080 bytes (free_pct 0.8% -> 2.6%, reserved 32029802496 -> 29324476416 bytes)
	[0m[32mINFO 2026-08-01 19:46:42,065 4074910 sam3_base_predictor.py: 410:[0m removed session f67b1fcd-4ae8-4d43-92a9-7953a0848bec
	19:46:42 WARNING fpgm.datagen.pipeline: stage s2 failed for AUTOLab+0d4edc83+2023-10-21-19h-08m-41s/ext1: CUDA out of memory. Tried to allocate 1.27 GiB. GPU 0 has a total capacity of 139.81 GiB of which 1.12 GiB is free. Including non-PyTorch memory, this process has 30.51 GiB memory in use. Of the allocated memory 29.55 GiB is allocated by PyTorch, and 283.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
	19:46:42 INFO fpgm.datagen.batch.worker.0: shard 0: AUTOLab+0d4edc83+2023-10-21-19h-08m-41s -> failed (28.2s)
	19:46:42 INFO fpgm.datagen.pipeline: extrinsics candidate 'optimized_cameras_json': Bundle-adjusted world->camera extrinsic from cameras.json's optimized_extrinsics (Phase 0 measured this byte-identical to SceneFlowClip.extrinsic). Expected to win the gate since it is jointly optimized against the whole scene rather than a single onboard frame estimate -- but that expectation is exactly what the gate verifies, not assumes.
	19:46:42 INFO fpgm.datagen.pipeline: extrinsics candidate 'trajectory_h5_row0': DROID's per-frame 6-vector cam2world at trajectory row 0, converted via cam2world_vector_to_world2cam. Row 0 stands in for the whole episode under the fixed-camera assumption; max translation spread across all 125 rows was 0.00 mm (a large spread here would mean this single-row candidate is not representative even on its own terms).
	19:46:43 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 1); retrying in 20s
	19:46:49 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 49; measured free MB: {0: 19304, 1: 3732, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:47:03 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 2); retrying in 20s
	19:47:10 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 50; measured free MB: {0: 19304, 1: 18926, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:47:23 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 3); retrying in 20s
	19:47:30 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 51; measured free MB: {0: 19304, 1: 18926, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:47:43 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 4); retrying in 20s
	19:47:50 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 52; measured free MB: {0: 19304, 1: 18926, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:48:04 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 5); retrying in 20s
	19:48:10 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 53; measured free MB: {0: 19304, 1: 18926, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:48:24 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 6); retrying in 20s
	19:48:31 INFO fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 54; measured free MB: {0: 19304, 1: 18926, 2: 3451, 3: 11677}; claims: {0: 0, 1: 1, 2: 0, 3: 0}); sleeping 20s
	19:48:44 WARNING fpgm.datagen.pipeline: no GPU with >= 20000 MB free (attempt 7); retrying in 20s
	/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
	warnings.warn('resource_tracker: There appear to be %d '

Xet Storage Details

Size:: 101 kB
Xet hash:: cd7bf5f395d885723a1823b3a815bf9900c3c67b32eedb25d565d0ddea15464e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.