Clarify GPU memory footprint and add ROCm utilization averages
Browse filesDistinguish tensor allocation, allocator reservation and device-wide VRAM occupancy. GPU utilization averages devices 1–7 for hit runs, retains intermittent zeros, excludes logging-biased device 0, and marks PF utilization unavailable. Include full snapshot audit data.
- README.md +70 -17
- images/v3.2.0/comparison_data.json +58 -9
- images/v3.2.0/memory_rocm_data.json +0 -0
README.md
CHANGED
|
@@ -99,33 +99,86 @@ uses saved PU-only test inputs and must not be mixed with these numbers.
|
|
| 99 |
|
| 100 |
## CLIC/CLD training compute and I/O
|
| 101 |
|
| 102 |
-
| Detector | Model | Training s/step¹ | Logical devices | Loader wait [ms]² |
|
| 103 |
-
| --- | --- | ---: | ---: | ---: |
|
| 104 |
-
| CLIC | Track/cluster attention | 0.112 | 1 | 2.05 |
|
| 105 |
-
| CLIC | Track/cluster HEPTv2 | 0.400 | 1 | 1.99 |
|
| 106 |
-
| CLIC | Hits attention | 2.356 | 8 | 17.11 |
|
| 107 |
-
| CLIC | Hits HEPTv2 | 2.888 | 8 | 16.89 |
|
| 108 |
-
| CLD | Track/cluster attention | 0.112 | 1 | 2.07 |
|
| 109 |
-
| CLD | Track/cluster HEPTv2 | 0.412 | 1 | 2.01 |
|
| 110 |
-
| CLD | Hits attention | 2.378 | 8 | 16.91 |
|
| 111 |
-
| CLD | Hits HEPTv2 | 2.842 | 8 | 16.71 |
|
| 112 |
|
| 113 |
All these runs use global batch 128 on the LUMI MI250X platform: 128 events on
|
| 114 |
one logical device for track/cluster, or 16 events per device across eight devices
|
| 115 |
for hits. HEPTv2 uses activation checkpointing and has more parameters than
|
| 116 |
-
attention. Logical
|
| 117 |
-
dual-GCD packages.
|
| 118 |
|
| 119 |
¹ Median elapsed seconds per step in 100-step intervals, excluding intervals
|
| 120 |
-
across validation, warmup and process restarts. This
|
| 121 |
-
throughput
|
| 122 |
|
| 123 |
² Median exposed rank-zero wait for the next batch on instrumented steps.
|
| 124 |
Prefetch work, storage byte throughput and CPU utilization are not measured by
|
| 125 |
-
this number. Small exposed wait does not prove that
|
|
|
|
|
|
|
| 126 |
|
| 127 |
-
|
| 128 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
Hit HEPTv2 projects to about **55–58 times the training device cost** of
|
| 131 |
track/cluster HEPTv2 for the same step count. This excludes validation, queueing,
|
|
|
|
| 99 |
|
| 100 |
## CLIC/CLD training compute and I/O
|
| 101 |
|
| 102 |
+
| Detector | Model | Training s/step¹ | Logical devices | Loader wait [ms]² |
|
| 103 |
+
| --- | --- | ---: | ---: | ---: |
|
| 104 |
+
| CLIC | Track/cluster attention | 0.112 | 1 | 2.05 |
|
| 105 |
+
| CLIC | Track/cluster HEPTv2 | 0.400 | 1 | 1.99 |
|
| 106 |
+
| CLIC | Hits attention | 2.356 | 8 | 17.11 |
|
| 107 |
+
| CLIC | Hits HEPTv2 | 2.888 | 8 | 16.89 |
|
| 108 |
+
| CLD | Track/cluster attention | 0.112 | 1 | 2.07 |
|
| 109 |
+
| CLD | Track/cluster HEPTv2 | 0.412 | 1 | 2.01 |
|
| 110 |
+
| CLD | Hits attention | 2.378 | 8 | 16.91 |
|
| 111 |
+
| CLD | Hits HEPTv2 | 2.842 | 8 | 16.71 |
|
| 112 |
|
| 113 |
All these runs use global batch 128 on the LUMI MI250X platform: 128 events on
|
| 114 |
one logical device for track/cluster, or 16 events per device across eight devices
|
| 115 |
for hits. HEPTv2 uses activation checkpointing and has more parameters than
|
| 116 |
+
attention. Logical devices are MI250X GCDs, not entire dual-GCD packages.
|
|
|
|
| 117 |
|
| 118 |
¹ Median elapsed seconds per step in 100-step intervals, excluding intervals
|
| 119 |
+
across validation, warmup and process restarts. This estimates ordinary training
|
| 120 |
+
throughput, not total campaign wall time or inference latency.
|
| 121 |
|
| 122 |
² Median exposed rank-zero wait for the next batch on instrumented steps.
|
| 123 |
Prefetch work, storage byte throughput and CPU utilization are not measured by
|
| 124 |
+
this number. Small exposed wait does not prove that storage is idle.
|
| 125 |
+
|
| 126 |
+
### GPU memory: tensors, allocator reservation and VRAM occupancy
|
| 127 |
|
| 128 |
+
| Detector | Model | Peak PyTorch allocated/device [GiB] | Peak PyTorch reserved/device [GiB] | ROCm VRAM median | ROCm VRAM maximum observed |
|
| 129 |
+
| --- | --- | ---: | ---: | ---: | ---: |
|
| 130 |
+
| CLIC | Track/cluster attention | 2.25 | 2.77 | 5% | 5% |
|
| 131 |
+
| CLIC | Track/cluster HEPTv2 | 3.03 | 3.76 | 7% | 7% |
|
| 132 |
+
| CLIC | Hits attention | 11.86 | 28.25 | 37% | 47% |
|
| 133 |
+
| CLIC | Hits HEPTv2 | 14.50 | 35.66 | 40% | 59% |
|
| 134 |
+
| CLD | Track/cluster attention | 2.27 | 2.79 | 5% | 5% |
|
| 135 |
+
| CLD | Track/cluster HEPTv2 | 3.03 | 3.81 | 7% | 7% |
|
| 136 |
+
| CLD | Hits attention | 11.69 | 28.48 | 37% | 47% |
|
| 137 |
+
| CLD | Hits HEPTv2 | 14.57 | 32.77 | 40% | 54% |
|
| 138 |
+
|
| 139 |
+
**Allocated** measures memory occupied by tensors. **Reserved** also includes
|
| 140 |
+
unused blocks retained by PyTorch's caching allocator. Both columns use the
|
| 141 |
+
maximum recorded historical high-water mark across ranks and sessions; counters
|
| 142 |
+
reset on restart and may include earlier validation allocations. **ROCm VRAM
|
| 143 |
+
occupancy** measures device-wide usage, including allocations outside the PyTorch
|
| 144 |
+
allocator. Its statistics include all 500 snapshots and all participating
|
| 145 |
+
devices: device 0 for PF and devices 0–7 for hits. Its maximum is the largest
|
| 146 |
+
observed snapshot, not a continuously measured peak.
|
| 147 |
+
|
| 148 |
+
The earlier approximately 14.5 GiB figures correctly described tensor allocation,
|
| 149 |
+
but should not be interpreted as total GPU-memory requirements. HEPTv2 hit runs
|
| 150 |
+
reserve up to **35.66 GiB/device for CLIC** and **32.77 GiB/device for CLD**, with
|
| 151 |
+
median ROCm occupancy **40%** for both. Definitions follow the
|
| 152 |
+
[PyTorch memory documentation](https://docs.pytorch.org/docs/stable/notes/cuda.html);
|
| 153 |
+
[LUMI hardware documentation](https://docs.lumi-supercomputer.eu/hardware/lumig/)
|
| 154 |
+
explains the logical-device/GCD layout.
|
| 155 |
+
|
| 156 |
+
### Average GPU utilization from ROCm
|
| 157 |
+
|
| 158 |
+
| Detector | Model | Mean ROCm GPU busy | Devices averaged | Readings used | Zero readings retained |
|
| 159 |
+
| --- | --- | ---: | --- | ---: | ---: |
|
| 160 |
+
| CLIC | Track/cluster attention | Unavailable | None | — | — |
|
| 161 |
+
| CLIC | Track/cluster HEPTv2 | Unavailable | None | — | — |
|
| 162 |
+
| CLIC | Hits attention | 96.17% | 1–7 | 3500 | 47 |
|
| 163 |
+
| CLIC | Hits HEPTv2 | 96.59% | 1–7 | 3500 | 49 |
|
| 164 |
+
| CLD | Track/cluster attention | Unavailable | None | — | — |
|
| 165 |
+
| CLD | Track/cluster HEPTv2 | Unavailable | None | — | — |
|
| 166 |
+
| CLD | Hits attention | 95.17% | 1–7 | 3500 | 66 |
|
| 167 |
+
| CLD | Hits HEPTv2 | 96.38% | 1–7 | 3500 | 40 |
|
| 168 |
+
|
| 169 |
+
Device 0 is explicitly excluded from utilization because rank-zero logging biases
|
| 170 |
+
its snapshots toward an idle period. For hit models, the reported value is the
|
| 171 |
+
arithmetic mean over **500 snapshots × devices 1–7 = 3,500 readings**.
|
| 172 |
+
**Intermittent zero readings on those devices remain in the mean.** PF uses only
|
| 173 |
+
device 0, so its utilization is unavailable under this rule; unused devices 1–7
|
| 174 |
+
are not substituted. Excluding device 0 from GPU-busy statistics does not remove
|
| 175 |
+
its VRAM readings from the memory statistics.
|
| 176 |
+
|
| 177 |
+
These are snapshot averages at logged steps 100–50,000 after deduplicating resumed
|
| 178 |
+
steps, not time-weighted utilization over the full training run. GPU busy is not
|
| 179 |
+
FLOP efficiency and does not distinguish useful kernels from communication work.
|
| 180 |
+
The [memory and ROCm audit data](images/v3.2.0/memory_rocm_data.json) retain all
|
| 181 |
+
raw device snapshots, sample counts and source-log links.
|
| 182 |
|
| 183 |
Hit HEPTv2 projects to about **55–58 times the training device cost** of
|
| 184 |
track/cluster HEPTv2 for the same step count. This excludes validation, queueing,
|
images/v3.2.0/comparison_data.json
CHANGED
|
@@ -31,7 +31,13 @@
|
|
| 31 |
"transfer_ms": 1.2761279940605164,
|
| 32 |
"compute_s": 0.2169132666895166,
|
| 33 |
"sampled_s": 1.3458307981491089,
|
| 34 |
-
"peak_gib": 2.252988338470459
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
},
|
| 36 |
{
|
| 37 |
"run": "clic_pf_heptv2",
|
|
@@ -52,7 +58,13 @@
|
|
| 52 |
"transfer_ms": 1.2515810085460544,
|
| 53 |
"compute_s": 0.5076365884160623,
|
| 54 |
"sampled_s": 1.6224647760391235,
|
| 55 |
-
"peak_gib": 3.030461311340332
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
},
|
| 57 |
{
|
| 58 |
"run": "clic_hits_attention",
|
|
@@ -73,7 +85,13 @@
|
|
| 73 |
"transfer_ms": 5.6031260173767805,
|
| 74 |
"compute_s": 2.3847950026392937,
|
| 75 |
"sampled_s": 4.924937963485718,
|
| 76 |
-
"peak_gib": 11.855772018432617
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
},
|
| 78 |
{
|
| 79 |
"run": "clic_hits_heptv2",
|
|
@@ -94,7 +112,13 @@
|
|
| 94 |
"transfer_ms": 5.54687948897481,
|
| 95 |
"compute_s": 2.8723186030983925,
|
| 96 |
"sampled_s": 5.453466176986694,
|
| 97 |
-
"peak_gib": 14.498655319213867
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
},
|
| 99 |
{
|
| 100 |
"run": "cld_pf_attention",
|
|
@@ -115,7 +139,13 @@
|
|
| 115 |
"transfer_ms": 1.2848689802922308,
|
| 116 |
"compute_s": 0.2151729288743809,
|
| 117 |
"sampled_s": 1.3415029048919678,
|
| 118 |
-
"peak_gib": 2.2739763259887695
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
},
|
| 120 |
{
|
| 121 |
"run": "cld_pf_heptv2",
|
|
@@ -136,7 +166,13 @@
|
|
| 136 |
"transfer_ms": 1.2725715059787035,
|
| 137 |
"compute_s": 0.5150086318608373,
|
| 138 |
"sampled_s": 1.6560965776443481,
|
| 139 |
-
"peak_gib": 3.030461311340332
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
},
|
| 141 |
{
|
| 142 |
"run": "cld_hits_attention",
|
|
@@ -157,7 +193,13 @@
|
|
| 157 |
"transfer_ms": 5.489614559337497,
|
| 158 |
"compute_s": 2.324357531964779,
|
| 159 |
"sampled_s": 4.865405797958374,
|
| 160 |
-
"peak_gib": 11.692705154418945
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 161 |
},
|
| 162 |
{
|
| 163 |
"run": "cld_hits_heptv2",
|
|
@@ -178,7 +220,13 @@
|
|
| 178 |
"transfer_ms": 5.5203549563884735,
|
| 179 |
"compute_s": 2.839740104973316,
|
| 180 |
"sampled_s": 5.353001117706299,
|
| 181 |
-
"peak_gib": 14.568944931030273
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 182 |
}
|
| 183 |
],
|
| 184 |
"cms": {
|
|
@@ -254,5 +302,6 @@
|
|
| 254 |
"code_revision": "6b80b983477321cf708e5043fb1d2ec15dd398bc"
|
| 255 |
}
|
| 256 |
},
|
| 257 |
-
"note": "Charts use release validation metrics and CLIC/CLD equal-process jet IQR. No cross-detector ranking or statistical-significance claim."
|
|
|
|
| 258 |
}
|
|
|
|
| 31 |
"transfer_ms": 1.2761279940605164,
|
| 32 |
"compute_s": 0.2169132666895166,
|
| 33 |
"sampled_s": 1.3458307981491089,
|
| 34 |
+
"peak_gib": 2.252988338470459,
|
| 35 |
+
"peak_reserved_gib": 2.771484375,
|
| 36 |
+
"rocm_vram_median_pct": 5.0,
|
| 37 |
+
"rocm_vram_max_pct": 5.0,
|
| 38 |
+
"rocm_gpu_busy_mean_pct": null,
|
| 39 |
+
"rocm_gpu_readings": 0,
|
| 40 |
+
"rocm_gpu_zero_readings": 0
|
| 41 |
},
|
| 42 |
{
|
| 43 |
"run": "clic_pf_heptv2",
|
|
|
|
| 58 |
"transfer_ms": 1.2515810085460544,
|
| 59 |
"compute_s": 0.5076365884160623,
|
| 60 |
"sampled_s": 1.6224647760391235,
|
| 61 |
+
"peak_gib": 3.030461311340332,
|
| 62 |
+
"peak_reserved_gib": 3.7578125,
|
| 63 |
+
"rocm_vram_median_pct": 7.0,
|
| 64 |
+
"rocm_vram_max_pct": 7.0,
|
| 65 |
+
"rocm_gpu_busy_mean_pct": null,
|
| 66 |
+
"rocm_gpu_readings": 0,
|
| 67 |
+
"rocm_gpu_zero_readings": 0
|
| 68 |
},
|
| 69 |
{
|
| 70 |
"run": "clic_hits_attention",
|
|
|
|
| 85 |
"transfer_ms": 5.6031260173767805,
|
| 86 |
"compute_s": 2.3847950026392937,
|
| 87 |
"sampled_s": 4.924937963485718,
|
| 88 |
+
"peak_gib": 11.855772018432617,
|
| 89 |
+
"peak_reserved_gib": 28.251953125,
|
| 90 |
+
"rocm_vram_median_pct": 37.0,
|
| 91 |
+
"rocm_vram_max_pct": 47.0,
|
| 92 |
+
"rocm_gpu_busy_mean_pct": 96.16542857142858,
|
| 93 |
+
"rocm_gpu_readings": 3500,
|
| 94 |
+
"rocm_gpu_zero_readings": 47
|
| 95 |
},
|
| 96 |
{
|
| 97 |
"run": "clic_hits_heptv2",
|
|
|
|
| 112 |
"transfer_ms": 5.54687948897481,
|
| 113 |
"compute_s": 2.8723186030983925,
|
| 114 |
"sampled_s": 5.453466176986694,
|
| 115 |
+
"peak_gib": 14.498655319213867,
|
| 116 |
+
"peak_reserved_gib": 35.662109375,
|
| 117 |
+
"rocm_vram_median_pct": 40.0,
|
| 118 |
+
"rocm_vram_max_pct": 59.0,
|
| 119 |
+
"rocm_gpu_busy_mean_pct": 96.59314285714285,
|
| 120 |
+
"rocm_gpu_readings": 3500,
|
| 121 |
+
"rocm_gpu_zero_readings": 49
|
| 122 |
},
|
| 123 |
{
|
| 124 |
"run": "cld_pf_attention",
|
|
|
|
| 139 |
"transfer_ms": 1.2848689802922308,
|
| 140 |
"compute_s": 0.2151729288743809,
|
| 141 |
"sampled_s": 1.3415029048919678,
|
| 142 |
+
"peak_gib": 2.2739763259887695,
|
| 143 |
+
"peak_reserved_gib": 2.7890625,
|
| 144 |
+
"rocm_vram_median_pct": 5.0,
|
| 145 |
+
"rocm_vram_max_pct": 5.0,
|
| 146 |
+
"rocm_gpu_busy_mean_pct": null,
|
| 147 |
+
"rocm_gpu_readings": 0,
|
| 148 |
+
"rocm_gpu_zero_readings": 0
|
| 149 |
},
|
| 150 |
{
|
| 151 |
"run": "cld_pf_heptv2",
|
|
|
|
| 166 |
"transfer_ms": 1.2725715059787035,
|
| 167 |
"compute_s": 0.5150086318608373,
|
| 168 |
"sampled_s": 1.6560965776443481,
|
| 169 |
+
"peak_gib": 3.030461311340332,
|
| 170 |
+
"peak_reserved_gib": 3.814453125,
|
| 171 |
+
"rocm_vram_median_pct": 7.0,
|
| 172 |
+
"rocm_vram_max_pct": 7.0,
|
| 173 |
+
"rocm_gpu_busy_mean_pct": null,
|
| 174 |
+
"rocm_gpu_readings": 0,
|
| 175 |
+
"rocm_gpu_zero_readings": 0
|
| 176 |
},
|
| 177 |
{
|
| 178 |
"run": "cld_hits_attention",
|
|
|
|
| 193 |
"transfer_ms": 5.489614559337497,
|
| 194 |
"compute_s": 2.324357531964779,
|
| 195 |
"sampled_s": 4.865405797958374,
|
| 196 |
+
"peak_gib": 11.692705154418945,
|
| 197 |
+
"peak_reserved_gib": 28.48046875,
|
| 198 |
+
"rocm_vram_median_pct": 37.0,
|
| 199 |
+
"rocm_vram_max_pct": 47.0,
|
| 200 |
+
"rocm_gpu_busy_mean_pct": 95.17,
|
| 201 |
+
"rocm_gpu_readings": 3500,
|
| 202 |
+
"rocm_gpu_zero_readings": 66
|
| 203 |
},
|
| 204 |
{
|
| 205 |
"run": "cld_hits_heptv2",
|
|
|
|
| 220 |
"transfer_ms": 5.5203549563884735,
|
| 221 |
"compute_s": 2.839740104973316,
|
| 222 |
"sampled_s": 5.353001117706299,
|
| 223 |
+
"peak_gib": 14.568944931030273,
|
| 224 |
+
"peak_reserved_gib": 32.767578125,
|
| 225 |
+
"rocm_vram_median_pct": 40.0,
|
| 226 |
+
"rocm_vram_max_pct": 54.0,
|
| 227 |
+
"rocm_gpu_busy_mean_pct": 96.37771428571429,
|
| 228 |
+
"rocm_gpu_readings": 3500,
|
| 229 |
+
"rocm_gpu_zero_readings": 40
|
| 230 |
}
|
| 231 |
],
|
| 232 |
"cms": {
|
|
|
|
| 302 |
"code_revision": "6b80b983477321cf708e5043fb1d2ec15dd398bc"
|
| 303 |
}
|
| 304 |
},
|
| 305 |
+
"note": "Charts use release validation metrics and CLIC/CLD equal-process jet IQR. No cross-detector ranking or statistical-significance claim.",
|
| 306 |
+
"memory_rocm_audit": "memory_rocm_data.json"
|
| 307 |
}
|
images/v3.2.0/memory_rocm_data.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|