logs/batch_run2.log

19:01:57 INFO    fpgm.datagen.batch.gpu_pool: gpu pool: discovered 4 device(s): [0, 1, 2, 3] (measured free MB: {0: 20846, 1: 31535, 2: 3451, 3: 11677})
19:01:57 INFO    fpgm.datagen.batch.summary: batch summary: resumed 1 episode record(s) from /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:01:57 INFO    fpgm.datagen.batch.runner: batch: skipping 1 episode(s) already 'ok' in /srv/data/foundation-physics-graph-model/outputs/datagen/batch_summary.json
19:01:57 INFO    fpgm.datagen.batch.runner: batch: pilot -- running 7/61 episode(s) first
19:01:57 INFO    fpgm.datagen.batch.runner: batch: 7 episode(s) -> 2 shard(s) (concurrency=2, 4 GPU device(s) visible: (0, 1, 2, 3))
19:01:57 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 1; measured free MB: {0: 20846, 1: 31535, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:02:18 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 2; measured free MB: {0: 20846, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:02:38 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 3; measured free MB: {0: 20846, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:02:58 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 4; measured free MB: {0: 20846, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:03:18 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 5; measured free MB: {0: 20846, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:03:38 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 6; measured free MB: {0: 22388, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:03:58 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 7; measured free MB: {0: 22388, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:04:19 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 8; measured free MB: {0: 21154, 1: 24659, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:04:39 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 9; measured free MB: {0: 21154, 1: 25187, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:04:59 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 10; measured free MB: {0: 21154, 1: 25187, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:05:20 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 11; measured free MB: {0: 21154, 1: 25187, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:05:40 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 12; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:06:00 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 13; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:06:20 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 14; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:06:41 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 15; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:07:01 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 16; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:07:21 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 17; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:07:41 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 18; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:08:02 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 19; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:08:22 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 20; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:08:42 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 21; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:09:02 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 22; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:09:22 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 23; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:09:43 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 24; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:10:03 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 25; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:10:23 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 26; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:10:43 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 27; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:11:04 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 28; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:11:24 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 29; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:11:44 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 30; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:12:04 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 31; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:12:24 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 32; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:12:44 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 33; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:13:05 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 34; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:13:25 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 35; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:13:45 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 36; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:14:05 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 37; measured free MB: {0: 21154, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:14:25 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 38; measured free MB: {0: 21682, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:14:46 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 39; measured free MB: {0: 23224, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:15:06 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 40; measured free MB: {0: 23224, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:15:26 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 41; measured free MB: {0: 21568, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:15:46 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 42; measured free MB: {0: 23110, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:16:07 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 43; measured free MB: {0: 22468, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:16:27 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 44; measured free MB: {0: 12318, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:16:47 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 45; measured free MB: {0: 10774, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:17:07 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 46; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:17:27 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 47; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:17:47 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 48; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:18:08 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 49; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:18:28 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 50; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:18:48 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 51; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:19:08 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 52; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:19:29 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 53; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:19:49 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 54; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:20:09 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 55; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:20:29 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 56; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:20:49 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 57; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:21:10 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 58; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:21:30 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 59; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:21:50 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 60; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:22:10 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 61; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
19:22:31 INFO    fpgm.datagen.batch.gpu_pool: gpu claim: no device available yet (attempt 62; measured free MB: {0: 19410, 1: 24765, 2: 3451, 3: 11677}; claims: {0: 0, 1: 0, 2: 0, 3: 0}); sleeping 20s
/home/quang/miniconda3/envs/fpgm/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 5 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '