logs/batch_v6.log

20:27:39 INFO    fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677})
20:27:39 INFO    fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
20:27:39 INFO    fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
20:27:39 INFO    fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
20:27:39 INFO    fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
20:27:40 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:28:00 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:28:20 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:28:40 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:29:01 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:29:21 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:29:41 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:30:01 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 8; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:30:22 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 9; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:30:42 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 10; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:31:02 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 11; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
20:31:22 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 12; measured free MB: {0: 19304, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 5 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '