jpata commited on
Commit
1a46335
·
verified ·
1 Parent(s): b925e87

Clarify GPU memory footprint and add ROCm utilization averages

Browse files

Distinguish tensor allocation, allocator reservation and device-wide VRAM occupancy. GPU utilization averages devices 1–7 for hit runs, retains intermittent zeros, excludes logging-biased device 0, and marks PF utilization unavailable. Include full snapshot audit data.

README.md CHANGED
@@ -99,33 +99,86 @@ uses saved PU-only test inputs and must not be mixed with these numbers.
99
 
100
  ## CLIC/CLD training compute and I/O
101
 
102
- | Detector | Model | Training s/step¹ | Logical devices | Loader wait [ms]² | Peak allocation/device [GiB]³ |
103
- | --- | --- | ---: | ---: | ---: | ---: |
104
- | CLIC | Track/cluster attention | 0.112 | 1 | 2.05 | 2.25 |
105
- | CLIC | Track/cluster HEPTv2 | 0.400 | 1 | 1.99 | 3.03 |
106
- | CLIC | Hits attention | 2.356 | 8 | 17.11 | 11.86 |
107
- | CLIC | Hits HEPTv2 | 2.888 | 8 | 16.89 | 14.50 |
108
- | CLD | Track/cluster attention | 0.112 | 1 | 2.07 | 2.27 |
109
- | CLD | Track/cluster HEPTv2 | 0.412 | 1 | 2.01 | 3.03 |
110
- | CLD | Hits attention | 2.378 | 8 | 16.91 | 11.69 |
111
- | CLD | Hits HEPTv2 | 2.842 | 8 | 16.71 | 14.57 |
112
 
113
  All these runs use global batch 128 on the LUMI MI250X platform: 128 events on
114
  one logical device for track/cluster, or 16 events per device across eight devices
115
  for hits. HEPTv2 uses activation checkpointing and has more parameters than
116
- attention. Logical-device counts refer to accelerator devices/GCDs, not entire
117
- dual-GCD packages.
118
 
119
  ¹ Median elapsed seconds per step in 100-step intervals, excluding intervals
120
- across validation, warmup and process restarts. This is an ordinary training
121
- throughput estimate, not total campaign wall time or inference latency.
122
 
123
  ² Median exposed rank-zero wait for the next batch on instrumented steps.
124
  Prefetch work, storage byte throughput and CPU utilization are not measured by
125
- this number. Small exposed wait does not prove that the storage system is idle.
 
 
126
 
127
- ³ Maximum recorded historical allocator peak across ranks and sessions. It can
128
- include earlier validation allocations and is not total VRAM occupancy.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
 
130
  Hit HEPTv2 projects to about **55–58 times the training device cost** of
131
  track/cluster HEPTv2 for the same step count. This excludes validation, queueing,
 
99
 
100
  ## CLIC/CLD training compute and I/O
101
 
102
+ | Detector | Model | Training s/step¹ | Logical devices | Loader wait [ms]² |
103
+ | --- | --- | ---: | ---: | ---: |
104
+ | CLIC | Track/cluster attention | 0.112 | 1 | 2.05 |
105
+ | CLIC | Track/cluster HEPTv2 | 0.400 | 1 | 1.99 |
106
+ | CLIC | Hits attention | 2.356 | 8 | 17.11 |
107
+ | CLIC | Hits HEPTv2 | 2.888 | 8 | 16.89 |
108
+ | CLD | Track/cluster attention | 0.112 | 1 | 2.07 |
109
+ | CLD | Track/cluster HEPTv2 | 0.412 | 1 | 2.01 |
110
+ | CLD | Hits attention | 2.378 | 8 | 16.91 |
111
+ | CLD | Hits HEPTv2 | 2.842 | 8 | 16.71 |
112
 
113
  All these runs use global batch 128 on the LUMI MI250X platform: 128 events on
114
  one logical device for track/cluster, or 16 events per device across eight devices
115
  for hits. HEPTv2 uses activation checkpointing and has more parameters than
116
+ attention. Logical devices are MI250X GCDs, not entire dual-GCD packages.
 
117
 
118
  ¹ Median elapsed seconds per step in 100-step intervals, excluding intervals
119
+ across validation, warmup and process restarts. This estimates ordinary training
120
+ throughput, not total campaign wall time or inference latency.
121
 
122
  ² Median exposed rank-zero wait for the next batch on instrumented steps.
123
  Prefetch work, storage byte throughput and CPU utilization are not measured by
124
+ this number. Small exposed wait does not prove that storage is idle.
125
+
126
+ ### GPU memory: tensors, allocator reservation and VRAM occupancy
127
 
128
+ | Detector | Model | Peak PyTorch allocated/device [GiB] | Peak PyTorch reserved/device [GiB] | ROCm VRAM median | ROCm VRAM maximum observed |
129
+ | --- | --- | ---: | ---: | ---: | ---: |
130
+ | CLIC | Track/cluster attention | 2.25 | 2.77 | 5% | 5% |
131
+ | CLIC | Track/cluster HEPTv2 | 3.03 | 3.76 | 7% | 7% |
132
+ | CLIC | Hits attention | 11.86 | 28.25 | 37% | 47% |
133
+ | CLIC | Hits HEPTv2 | 14.50 | 35.66 | 40% | 59% |
134
+ | CLD | Track/cluster attention | 2.27 | 2.79 | 5% | 5% |
135
+ | CLD | Track/cluster HEPTv2 | 3.03 | 3.81 | 7% | 7% |
136
+ | CLD | Hits attention | 11.69 | 28.48 | 37% | 47% |
137
+ | CLD | Hits HEPTv2 | 14.57 | 32.77 | 40% | 54% |
138
+
139
+ **Allocated** measures memory occupied by tensors. **Reserved** also includes
140
+ unused blocks retained by PyTorch's caching allocator. Both columns use the
141
+ maximum recorded historical high-water mark across ranks and sessions; counters
142
+ reset on restart and may include earlier validation allocations. **ROCm VRAM
143
+ occupancy** measures device-wide usage, including allocations outside the PyTorch
144
+ allocator. Its statistics include all 500 snapshots and all participating
145
+ devices: device 0 for PF and devices 0–7 for hits. Its maximum is the largest
146
+ observed snapshot, not a continuously measured peak.
147
+
148
+ The earlier approximately 14.5 GiB figures correctly described tensor allocation,
149
+ but should not be interpreted as total GPU-memory requirements. HEPTv2 hit runs
150
+ reserve up to **35.66 GiB/device for CLIC** and **32.77 GiB/device for CLD**, with
151
+ median ROCm occupancy **40%** for both. Definitions follow the
152
+ [PyTorch memory documentation](https://docs.pytorch.org/docs/stable/notes/cuda.html);
153
+ [LUMI hardware documentation](https://docs.lumi-supercomputer.eu/hardware/lumig/)
154
+ explains the logical-device/GCD layout.
155
+
156
+ ### Average GPU utilization from ROCm
157
+
158
+ | Detector | Model | Mean ROCm GPU busy | Devices averaged | Readings used | Zero readings retained |
159
+ | --- | --- | ---: | --- | ---: | ---: |
160
+ | CLIC | Track/cluster attention | Unavailable | None | — | — |
161
+ | CLIC | Track/cluster HEPTv2 | Unavailable | None | — | — |
162
+ | CLIC | Hits attention | 96.17% | 1–7 | 3500 | 47 |
163
+ | CLIC | Hits HEPTv2 | 96.59% | 1–7 | 3500 | 49 |
164
+ | CLD | Track/cluster attention | Unavailable | None | — | — |
165
+ | CLD | Track/cluster HEPTv2 | Unavailable | None | — | — |
166
+ | CLD | Hits attention | 95.17% | 1–7 | 3500 | 66 |
167
+ | CLD | Hits HEPTv2 | 96.38% | 1–7 | 3500 | 40 |
168
+
169
+ Device 0 is explicitly excluded from utilization because rank-zero logging biases
170
+ its snapshots toward an idle period. For hit models, the reported value is the
171
+ arithmetic mean over **500 snapshots × devices 1–7 = 3,500 readings**.
172
+ **Intermittent zero readings on those devices remain in the mean.** PF uses only
173
+ device 0, so its utilization is unavailable under this rule; unused devices 1–7
174
+ are not substituted. Excluding device 0 from GPU-busy statistics does not remove
175
+ its VRAM readings from the memory statistics.
176
+
177
+ These are snapshot averages at logged steps 100–50,000 after deduplicating resumed
178
+ steps, not time-weighted utilization over the full training run. GPU busy is not
179
+ FLOP efficiency and does not distinguish useful kernels from communication work.
180
+ The [memory and ROCm audit data](images/v3.2.0/memory_rocm_data.json) retain all
181
+ raw device snapshots, sample counts and source-log links.
182
 
183
  Hit HEPTv2 projects to about **55–58 times the training device cost** of
184
  track/cluster HEPTv2 for the same step count. This excludes validation, queueing,
images/v3.2.0/comparison_data.json CHANGED
@@ -31,7 +31,13 @@
31
  "transfer_ms": 1.2761279940605164,
32
  "compute_s": 0.2169132666895166,
33
  "sampled_s": 1.3458307981491089,
34
- "peak_gib": 2.252988338470459
 
 
 
 
 
 
35
  },
36
  {
37
  "run": "clic_pf_heptv2",
@@ -52,7 +58,13 @@
52
  "transfer_ms": 1.2515810085460544,
53
  "compute_s": 0.5076365884160623,
54
  "sampled_s": 1.6224647760391235,
55
- "peak_gib": 3.030461311340332
 
 
 
 
 
 
56
  },
57
  {
58
  "run": "clic_hits_attention",
@@ -73,7 +85,13 @@
73
  "transfer_ms": 5.6031260173767805,
74
  "compute_s": 2.3847950026392937,
75
  "sampled_s": 4.924937963485718,
76
- "peak_gib": 11.855772018432617
 
 
 
 
 
 
77
  },
78
  {
79
  "run": "clic_hits_heptv2",
@@ -94,7 +112,13 @@
94
  "transfer_ms": 5.54687948897481,
95
  "compute_s": 2.8723186030983925,
96
  "sampled_s": 5.453466176986694,
97
- "peak_gib": 14.498655319213867
 
 
 
 
 
 
98
  },
99
  {
100
  "run": "cld_pf_attention",
@@ -115,7 +139,13 @@
115
  "transfer_ms": 1.2848689802922308,
116
  "compute_s": 0.2151729288743809,
117
  "sampled_s": 1.3415029048919678,
118
- "peak_gib": 2.2739763259887695
 
 
 
 
 
 
119
  },
120
  {
121
  "run": "cld_pf_heptv2",
@@ -136,7 +166,13 @@
136
  "transfer_ms": 1.2725715059787035,
137
  "compute_s": 0.5150086318608373,
138
  "sampled_s": 1.6560965776443481,
139
- "peak_gib": 3.030461311340332
 
 
 
 
 
 
140
  },
141
  {
142
  "run": "cld_hits_attention",
@@ -157,7 +193,13 @@
157
  "transfer_ms": 5.489614559337497,
158
  "compute_s": 2.324357531964779,
159
  "sampled_s": 4.865405797958374,
160
- "peak_gib": 11.692705154418945
 
 
 
 
 
 
161
  },
162
  {
163
  "run": "cld_hits_heptv2",
@@ -178,7 +220,13 @@
178
  "transfer_ms": 5.5203549563884735,
179
  "compute_s": 2.839740104973316,
180
  "sampled_s": 5.353001117706299,
181
- "peak_gib": 14.568944931030273
 
 
 
 
 
 
182
  }
183
  ],
184
  "cms": {
@@ -254,5 +302,6 @@
254
  "code_revision": "6b80b983477321cf708e5043fb1d2ec15dd398bc"
255
  }
256
  },
257
- "note": "Charts use release validation metrics and CLIC/CLD equal-process jet IQR. No cross-detector ranking or statistical-significance claim."
 
258
  }
 
31
  "transfer_ms": 1.2761279940605164,
32
  "compute_s": 0.2169132666895166,
33
  "sampled_s": 1.3458307981491089,
34
+ "peak_gib": 2.252988338470459,
35
+ "peak_reserved_gib": 2.771484375,
36
+ "rocm_vram_median_pct": 5.0,
37
+ "rocm_vram_max_pct": 5.0,
38
+ "rocm_gpu_busy_mean_pct": null,
39
+ "rocm_gpu_readings": 0,
40
+ "rocm_gpu_zero_readings": 0
41
  },
42
  {
43
  "run": "clic_pf_heptv2",
 
58
  "transfer_ms": 1.2515810085460544,
59
  "compute_s": 0.5076365884160623,
60
  "sampled_s": 1.6224647760391235,
61
+ "peak_gib": 3.030461311340332,
62
+ "peak_reserved_gib": 3.7578125,
63
+ "rocm_vram_median_pct": 7.0,
64
+ "rocm_vram_max_pct": 7.0,
65
+ "rocm_gpu_busy_mean_pct": null,
66
+ "rocm_gpu_readings": 0,
67
+ "rocm_gpu_zero_readings": 0
68
  },
69
  {
70
  "run": "clic_hits_attention",
 
85
  "transfer_ms": 5.6031260173767805,
86
  "compute_s": 2.3847950026392937,
87
  "sampled_s": 4.924937963485718,
88
+ "peak_gib": 11.855772018432617,
89
+ "peak_reserved_gib": 28.251953125,
90
+ "rocm_vram_median_pct": 37.0,
91
+ "rocm_vram_max_pct": 47.0,
92
+ "rocm_gpu_busy_mean_pct": 96.16542857142858,
93
+ "rocm_gpu_readings": 3500,
94
+ "rocm_gpu_zero_readings": 47
95
  },
96
  {
97
  "run": "clic_hits_heptv2",
 
112
  "transfer_ms": 5.54687948897481,
113
  "compute_s": 2.8723186030983925,
114
  "sampled_s": 5.453466176986694,
115
+ "peak_gib": 14.498655319213867,
116
+ "peak_reserved_gib": 35.662109375,
117
+ "rocm_vram_median_pct": 40.0,
118
+ "rocm_vram_max_pct": 59.0,
119
+ "rocm_gpu_busy_mean_pct": 96.59314285714285,
120
+ "rocm_gpu_readings": 3500,
121
+ "rocm_gpu_zero_readings": 49
122
  },
123
  {
124
  "run": "cld_pf_attention",
 
139
  "transfer_ms": 1.2848689802922308,
140
  "compute_s": 0.2151729288743809,
141
  "sampled_s": 1.3415029048919678,
142
+ "peak_gib": 2.2739763259887695,
143
+ "peak_reserved_gib": 2.7890625,
144
+ "rocm_vram_median_pct": 5.0,
145
+ "rocm_vram_max_pct": 5.0,
146
+ "rocm_gpu_busy_mean_pct": null,
147
+ "rocm_gpu_readings": 0,
148
+ "rocm_gpu_zero_readings": 0
149
  },
150
  {
151
  "run": "cld_pf_heptv2",
 
166
  "transfer_ms": 1.2725715059787035,
167
  "compute_s": 0.5150086318608373,
168
  "sampled_s": 1.6560965776443481,
169
+ "peak_gib": 3.030461311340332,
170
+ "peak_reserved_gib": 3.814453125,
171
+ "rocm_vram_median_pct": 7.0,
172
+ "rocm_vram_max_pct": 7.0,
173
+ "rocm_gpu_busy_mean_pct": null,
174
+ "rocm_gpu_readings": 0,
175
+ "rocm_gpu_zero_readings": 0
176
  },
177
  {
178
  "run": "cld_hits_attention",
 
193
  "transfer_ms": 5.489614559337497,
194
  "compute_s": 2.324357531964779,
195
  "sampled_s": 4.865405797958374,
196
+ "peak_gib": 11.692705154418945,
197
+ "peak_reserved_gib": 28.48046875,
198
+ "rocm_vram_median_pct": 37.0,
199
+ "rocm_vram_max_pct": 47.0,
200
+ "rocm_gpu_busy_mean_pct": 95.17,
201
+ "rocm_gpu_readings": 3500,
202
+ "rocm_gpu_zero_readings": 66
203
  },
204
  {
205
  "run": "cld_hits_heptv2",
 
220
  "transfer_ms": 5.5203549563884735,
221
  "compute_s": 2.839740104973316,
222
  "sampled_s": 5.353001117706299,
223
+ "peak_gib": 14.568944931030273,
224
+ "peak_reserved_gib": 32.767578125,
225
+ "rocm_vram_median_pct": 40.0,
226
+ "rocm_vram_max_pct": 54.0,
227
+ "rocm_gpu_busy_mean_pct": 96.37771428571429,
228
+ "rocm_gpu_readings": 3500,
229
+ "rocm_gpu_zero_readings": 40
230
  }
231
  ],
232
  "cms": {
 
302
  "code_revision": "6b80b983477321cf708e5043fb1d2ec15dd398bc"
303
  }
304
  },
305
+ "note": "Charts use release validation metrics and CLIC/CLD equal-process jet IQR. No cross-detector ranking or statistical-significance claim.",
306
+ "memory_rocm_audit": "memory_rocm_data.json"
307
  }
images/v3.2.0/memory_rocm_data.json ADDED
The diff for this file is too large to render. See raw diff