Add portable CLD benchmark evidence for v3.2.0
Browse files- .gitattributes +4 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/README.md +123 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/cases.csv +31 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.png +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.svg +1553 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.png +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.svg +1549 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.png +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.svg +1566 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png +3 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.svg +1509 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png +3 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.svg +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png +3 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.svg +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.png +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.svg +1563 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/manifest.json +114 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/render_plots.py +98 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/results.json +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.html +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf +3 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/steps.csv +0 -0
- benchmarks/v3.2.0/cld_gpu_batch_sweep/validation.csv +25 -0
.gitattributes
CHANGED
|
@@ -100,3 +100,7 @@ images/event_displays/pf_event.png filter=lfs diff=lfs merge=lfs -text
|
|
| 100 |
images/event_displays/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
|
| 101 |
images/pf_event.png filter=lfs diff=lfs merge=lfs -text
|
| 102 |
images/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
images/event_displays/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
|
| 101 |
images/pf_event.png filter=lfs diff=lfs merge=lfs -text
|
| 102 |
images/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
|
| 103 |
+
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png filter=lfs diff=lfs merge=lfs -text
|
| 104 |
+
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png filter=lfs diff=lfs merge=lfs -text
|
| 105 |
+
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png filter=lfs diff=lfs merge=lfs -text
|
| 106 |
+
benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf filter=lfs diff=lfs merge=lfs -text
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/README.md
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CLD GPU batch-sweep benchmark evidence, v3.2.0
|
| 2 |
+
|
| 3 |
+
Recorded on 29–30 October 2026. **v3.2.0 is the benchmark evidence release tag,
|
| 4 |
+
not the TFDS dataset version.** These are full platform/recipe measurements,
|
| 5 |
+
not isolated hardware rankings or a convergence study.
|
| 6 |
+
|
| 7 |
+
## Results
|
| 8 |
+
|
| 9 |
+
| Platform | Model | Best tested batch | Steady events/s | Steady kHits/s |
|
| 10 |
+
| --- | --- | ---: | ---: | ---: |
|
| 11 |
+
| GH200 | Attention | 32 | 14.55 | 140.6 |
|
| 12 |
+
| GH200 | HEPTv2 | 32 | 13.83 | 133.7 |
|
| 13 |
+
| H100 PCIe | Attention | 16 | 12.81 | 123.6 |
|
| 14 |
+
| H100 PCIe | HEPTv2 | 16 | 11.21 | 108.1 |
|
| 15 |
+
| MI250X (one GCD) | Attention | 16 | 7.35 | 70.9 |
|
| 16 |
+
| MI250X (one GCD) | HEPTv2 | 16 | 6.18 | 59.6 |
|
| 17 |
+
|
| 18 |
+
All 24 cases at batches 8/16/32/64 complete. Both models run out of device memory
|
| 19 |
+
at batch 128 on all three platforms. H100 HEPTv2 wraps its OOM in a fused-attention
|
| 20 |
+
RuntimeError; the original chained exception and sweep state identify OOM.
|
| 21 |
+
Batch 64 is the largest **tested** successful batch, not the exact capacity limit.
|
| 22 |
+
All six recipes slow down at batch 64 relative to their observed throughput peak.
|
| 23 |
+
|
| 24 |
+

|
| 25 |
+

|
| 26 |
+

|
| 27 |
+
|
| 28 |
+
## Total unweighted losses
|
| 29 |
+
|
| 30 |
+
The plots retain all 120 raw training losses per completed case, without smoothing
|
| 31 |
+
or warmup exclusion. Compare platforms within a fixed model and batch size.
|
| 32 |
+
`Total_unweighted` is the fixed sum of binary classification, class classification,
|
| 33 |
+
and the five regression tasks (pT, eta, sin(phi), cos(phi), energy). It excludes
|
| 34 |
+
cardinality, hit grouping and auxiliary losses, and differs from calibrated `Total`.
|
| 35 |
+
|
| 36 |
+

|
| 37 |
+

|
| 38 |
+

|
| 39 |
+
|
| 40 |
+
Training trajectories are broadly similar across platforms, with some transient
|
| 41 |
+
differences. The largest step-120 validation spread at a matched batch is 0.169
|
| 42 |
+
for Attention at batch 32. There is only one validation point per run, based on
|
| 43 |
+
32 validation events. One seed and one run do not support statistical significance
|
| 44 |
+
or a convergence claim. Larger training batches process more events in 120 steps.
|
| 45 |
+
|
| 46 |
+
## Protocol and scope
|
| 47 |
+
|
| 48 |
+
- One visible device and one rank. LUMI uses **one 64 GiB MI250X GCD**, half of a
|
| 49 |
+
physical two-GCD module. H100 is PCIe. Observed PyTorch capacities are about
|
| 50 |
+
95.0 GiB (GH200), 79.2 GiB (H100), and 64.0 GiB (MI250X GCD).
|
| 51 |
+
- Seed 12345, BF16, LAMB, LR 0.001, cosine schedule. Six-layer, width-256 backbones
|
| 52 |
+
with 16 heads, and a shared four-layer, 256-slot set decoder. HEPTv2 enables
|
| 53 |
+
activation checkpointing. Top-level model compilation is disabled; internal
|
| 54 |
+
compiled kernels may still run.
|
| 55 |
+
- 120 steps per case. Throughput and p50/p99 exclude the first 20 warmup steps.
|
| 56 |
+
Throughput = summed events or hits / summed host step durations. Those durations
|
| 57 |
+
include data wait and existing synchronizations, but exclude validation,
|
| 58 |
+
checkpoint writes, launch and teardown. The raw CSV includes the full 120 steps.
|
| 59 |
+
- End-of-case validation uses 32 events and testing uses 8 events. Process memory
|
| 60 |
+
peaks include evaluation; allocated, reserved and device-wide memory differ.
|
| 61 |
+
- Failed cases retain observed work (GH200 Attention batch 128 completes two
|
| 62 |
+
steps/256 events before failure). Failed throughput is unavailable, never zero.
|
| 63 |
+
- GPU busy measures device activity, not SM occupancy or FLOP efficiency. LUMI
|
| 64 |
+
power coverage is zero/unavailable. GH200 Attention batch 32 has no usable
|
| 65 |
+
phase-attributed busy/power interval. No energy or scheduler-cost ranking is made.
|
| 66 |
+
|
| 67 |
+
## Provenance and limitations
|
| 68 |
+
|
| 69 |
+
| Platform | Original sweep | Job | Recorded Git base | TFDS |
|
| 70 |
+
| --- | --- | --- | --- | --- |
|
| 71 |
+
| GH200 | gh200_batch_sweep_153260 | 153260 | d2d991cea54ab47382e9de63bc900d2850e75eb4 | 3.2.2 |
|
| 72 |
+
| H100 PCIe | h100_approx_321_full_20261029_1038 | 7363958 | 9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0 | 3.2.1 |
|
| 73 |
+
| MI250X (1 GCD) | lumi_retry_20261030_144129 | 23308832 | 20ac400a43a68da3ce1c029e96b3eaf90552f36d | 3.2.2 |
|
| 74 |
+
|
| 75 |
+
All runs use CLD `cld_edm_ttbar_hits`, config `1`, and all recorded Git trees were
|
| 76 |
+
dirty. The analysis originally reviewed the branch at `9ee676ac`. This release
|
| 77 |
+
preserves historical provenance and does not relabel the runs as measurements of
|
| 78 |
+
the later portable-launcher changes.
|
| 79 |
+
|
| 80 |
+
NVIDIA runs use PyTorch 2.11/CUDA 12.8 and native attention. LUMI uses its PyTorch
|
| 81 |
+
2.11/ROCm 7.2 build and external FlashAttention. GH200 is aarch64; the others are
|
| 82 |
+
x86_64. NVIDIA uses four loader workers; LUMI uses two and file-descriptor sharing.
|
| 83 |
+
The package preserves per-case model/data/runtime configuration and exact package
|
| 84 |
+
versions. `data_verification.status=not_recorded` means the original sweep did
|
| 85 |
+
not record the new preflight result, not that its data was verified by this release.
|
| 86 |
+
|
| 87 |
+
All 120 per-step sequence-length vectors match across platforms for each completed
|
| 88 |
+
model/batch pair. The `sequence_lengths_sha256` values retain that check without
|
| 89 |
+
publishing the full length vectors. Matching sizes **do not prove matching dataset
|
| 90 |
+
payloads, features or targets**. TFDS versions, dirty code bases and backends differ.
|
| 91 |
+
Loss and throughput comparisons therefore describe these recorded platforms.
|
| 92 |
+
|
| 93 |
+
## Files and regeneration
|
| 94 |
+
|
| 95 |
+
- [19-slide PDF](slides/gpu_batch_sweep.slides.pdf) and
|
| 96 |
+
[standalone HTML slides](slides/gpu_batch_sweep.slides.html). Download the HTML
|
| 97 |
+
to present offline (P enters presentation mode, arrows navigate, Esc exits).
|
| 98 |
+
- `results.json`: all 30 cases, scientific/runtime configuration, memory and
|
| 99 |
+
telemetry scope, source provenance, raw-source hashes and sequence signatures.
|
| 100 |
+
- `cases.csv`: flat per-case metrics. `steps.csv`: host-step work/timing and
|
| 101 |
+
unweighted losses, including partial work from failed cases.
|
| 102 |
+
- `validation.csv`: train/validation loss snapshots for 24 completed cases.
|
| 103 |
+
- `figures/`: PNG and SVG plots. `render_plots.py`: standalone plotting source.
|
| 104 |
+
- `manifest.json`: file sizes and SHA-256 digests for this package.
|
| 105 |
+
|
| 106 |
+
With Python, pandas, numpy and matplotlib installed:
|
| 107 |
+
|
| 108 |
+
```bash
|
| 109 |
+
python render_plots.py
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
The script regenerates the seven data figures from the compact CSVs. The slides
|
| 113 |
+
are archived presentation outputs, not regenerated by that command. Full original
|
| 114 |
+
inputs, histories, TensorBoard writers and frozen study sources remain in the dated
|
| 115 |
+
local/private study archive. Checkpoints, predictions and dataset caches are
|
| 116 |
+
intentionally omitted from this compact public release. Machine-specific absolute
|
| 117 |
+
paths and device UUIDs are omitted from exported metadata.
|
| 118 |
+
|
| 119 |
+
The generic exporter reads raw sweep directories without importing training/GPU
|
| 120 |
+
libraries. The plotting script requires one single-device session per platform,
|
| 121 |
+
model and batch and rejects ambiguous repeats. Raw throughput is checked against
|
| 122 |
+
summaries during export. An additive Hub commit carries this package under
|
| 123 |
+
`benchmarks/v3.2.0/cld_gpu_batch_sweep/`, with tag `v3.2.0` pointing to that commit.
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/cases.csv
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
platform,variant,batch,seed,run_id,session_id,rank,status,worker_status,source_sweep,world_size,architecture,python,torch,cuda_runtime,hip_runtime,dirty,steps,warmup_steps,events,hits,started_at_unix_time,finished_at_unix_time,worker_seconds,case_seconds,events_per_second,hits_per_second,step_p50_seconds,step_p99_seconds,device_memory_peak_bytes,scenario_sha256,spec_sha256,source_summary_sha256,source_metadata_sha256,sequence_lengths_sha256,device_name,device_memory_bytes,allocated_bytes,reserved_bytes,git_revision
|
| 2 |
+
GH200,attention,128,12345,attention_seed12345_20261029_065651_646012,alphagh01_rank0_3001875_7b652ed64458,0,oom,failed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,2,2,256,2452792,1793271428.2599475,1793271469.4534717,41.1932246722281,61.89646673202515,,,,,101581848576.0,,,c9f1576626f3b6bd348d8bdb4e2187370dbfd728a3a6538e3e4056ee68d1520d,493a3e099a1211a67dbf909f96b6ef1625afb1155dfaf7d63d6c430cc0ae61fc,d650c7bfed7b1bfa372dc0e96d993e9bbd1174ff0948d0c6385f20d81118b5a1,NVIDIA GH200 480GB,102005473280,86645364224,100908662784,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 3 |
+
GH200,heptv2,128,12345,heptv2_seed12345_20261029_065753_665332,alphagh01_rank0_3003853_a3fb6b54d52b,0,oom,failed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,0,0,0,0,1793271490.253663,1793271517.012132,26.75677142944187,48.1813178062439,,,,,98864988160.0,,,02ebca65d9f4cc0ce2651302ec410a5023ebcf4dc3355186e14d35beb1b53b6d,292cf2cdb3167b619bbb9600b902f18f25c082c2b351062452ee1a09ac34f9c2,,NVIDIA GH200 480GB,102005473280,93710633472,98039758848,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 4 |
+
GH200,attention,16,12345,attention_seed12345_20261029_062032_191329,alphagh01_rank0_2963140_f80016108aa1,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793269248.502339,1793269394.1430855,145.64032928738743,166.31411004066467,14.16103246161317,136614.56018174265,1.1265132501721382,1.3897758059576155,58485374976.0,,,426068a7a9794ae46693f6ec6a0109aa0f9c34af05d40c8bfcd431389953cdac,4f17b0f20fad5bdb358e7c12ded1a6376736a1bbfdd46bc8bf35db9f95cb00ba,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA GH200 480GB,102005473280,11841815040,21887975424,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 5 |
+
GH200,heptv2,16,12345,heptv2_seed12345_20261029_062318_186034,alphagh01_rank0_2966282_3c1d7090284d,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793269414.3337042,1793269573.2979531,158.9638429991901,179.52954769134521,13.4722942636621,129970.15298555062,1.1703924494795501,1.4493609412107618,59380858880.0,,,4dc39826e1c73d032711a5adbdf41082f89c8def27ea25c019a6bef750c2fec2,1bf689cc891339f82d0e08ea5831d2527cd1bdcd06933202ab827f968c75a0c9,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA GH200 480GB,102005473280,14239806464,22789750784,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 6 |
+
GH200,attention,32,12345,attention_seed12345_20261029_062618_170573,alphagh01_rank0_2969691_16a386b5ec65,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793269594.5457299,1793269869.5912952,275.0450903531164,295.8008642196655,14.547862661382029,140599.04616683867,2.1722677536308765,2.530936222132296,,,,4f309e232952ddc66592a6007933599b86ecc9c604701da945fefa183250a443,64b205c60d4771926f1db38db065022b2ede9ceb37a760c13c11348dd32d2602,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA GH200 480GB,102005473280,23358533632,48486154240,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 7 |
+
GH200,heptv2,32,12345,heptv2_seed12345_20261029_063114_050085,alphagh01_rank0_2974716_6b9de441032e,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793269890.665387,1793270185.674123,295.0081489570439,316.4379062652588,13.830442036952725,133665.4740096765,2.285853264387697,2.5668283491674813,86391128064.0,,,cec824d51019a3bf432a632fa23e13d1da9dbec1397a04ad3dc7b72bb4879d9b,632e01cb75d5c931bdd4f3d0d700cd41c1a6d36175d6113d74639608b619feca,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA GH200 480GB,102005473280,29242608128,49798971392,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 8 |
+
GH200,attention,64,12345,attention_seed12345_20261029_063630_231892,alphagh01_rank0_2980492_cefdd4e73f6b,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793270206.9355567,1793270778.2902935,571.3542380556464,592.1047451496124,13.837633818324782,133021.27767780976,4.618271645624191,5.18914441172034,101188632576.0,,,aa17684bc3e406284a5d1eb501187a35f8c071480d572bd385bf84fa14296650,dd6ff44afda20afa4f91e291dd68cd1055b61b867339e61ac66d6927f05b36b8,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA GH200 480GB,102005473280,46277582336,96154419200,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 9 |
+
GH200,heptv2,64,12345,heptv2_seed12345_20261029_064622_331901,alphagh01_rank0_2991102_2bb5ab0d5fc0,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793270798.354628,1793271407.1352649,608.7801285814494,629.1579208374023,13.081137392713957,125749.07186468971,4.868631860706955,5.330362756364049,101357453312.0,,,4a36b6a4ded75a0cfacfd41a5f045cdc442c1b2acd6acbfb5bbf3d52fa5fede9,57605d3217699f464b32d3d50d062d1870ade1b9889f03ce169f1c881c651b60,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA GH200 480GB,102005473280,58237065216,99444850688,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 10 |
+
GH200,attention,8,12345,attention_seed12345_20261029_061642_376997,alphagh01_rank0_2958727_aca0c7649228,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793269020.9687698,1793269107.0918162,86.12262949720025,108.35192370414734,12.56589640592102,122168.80203983614,0.6320971548557281,0.8602891777828339,49381638144.0,,,54cb095fcc22c17ca61335dfb048b9a8d8d48b682a6d6111276e442d492295ea,38048fb5876cbd44603b16067a3d2ad36474cba29abb272e64111924b8220c1f,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA GH200 480GB,102005473280,6212977664,12784238592,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 11 |
+
GH200,heptv2,8,12345,heptv2_seed12345_20261029_061831_709291,alphagh01_rank0_2960929_85567b5b0c80,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793269129.363011,1793269227.2062633,97.84284777846187,120.66704940795898,11.59590143653919,112738.26723626994,0.6744112796150148,0.939555611154065,48215621632.0,,,9275dae65558becf6e800ae68da0681aa9597c8c49fd49e8488a48b0ecd0262d,72e022d2dfd405d70687ce346725656b4334d1393264bcffc867f5fed4258ea3,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA GH200 480GB,102005473280,7242431488,11624513536,d2d991cea54ab47382e9de63bc900d2850e75eb4
|
| 12 |
+
H100 PCIe,attention,128,12345,attention_seed12345_20261029_112837_250666,workergpu158_rank0_1636267_32b4965342eb,0,oom,failed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,0,0,0,0,1793287749.437335,1793287760.1536057,10.715927304991055,53.80491065979004,,,,,79621521408.0,,,c19afde5deb9218f17fec97b131bcab8d59bd63352ab8dc19be93b1d4898e907,2faa3dd186ac2cf4638c999d1a87181d9bb43af564d8720daf7865d11e10c8ea,,NVIDIA H100 PCIe,85064286208,82823076864,84414562304,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 13 |
+
H100 PCIe,heptv2,128,12345,heptv2_seed12345_20261029_112929_059759,workergpu158_rank0_1645833_65e5cf1b3e6f,0,oom,failed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,0,0,0,0,1793287788.980532,1793287815.3894665,26.407711561012547,55.7578341960907,,,,,71440531456.0,,,b84797600bfc6b47c003bb90214f64d6de40122afc5308c7e802d5abb3603322,bd04558f9874d86e8a1c4dac8e9f08530632fdda541f3769439150bba7e25fba,,NVIDIA H100 PCIe,85064286208,78372063744,84064337920,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 14 |
+
H100 PCIe,attention,16,12345,attention_seed12345_20261029_104229_564372,workergpu158_rank0_1259903_683085c8df3b,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793284969.513365,1793285130.2373576,160.72338473401032,188.47921133041382,12.809366597085937,123574.74559932291,1.2254043756984174,1.813039491510718,22564306944.0,,,cc4746edf0c3de7a483bdde44219c516f119a358f247938acab75f344c68d402,1286ef919b1653a4ca9c02bff283bc719eb65f63b680eafd9f091bf2c7d27005,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA H100 PCIe,85064286208,11846225408,21850226688,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 15 |
+
H100 PCIe,heptv2,16,12345,heptv2_seed12345_20261029_104538_814383,workergpu158_rank0_1276037_80bfd5ff3556,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793285161.1348226,1793285351.6403244,190.50486456701765,222.06918454170227,11.2054484043102,108101.39794140843,1.3936438743548933,1.995739113763557,23419944960.0,,,1be9b64d044ff72abeb43e95730932459f28c7527c58ce6bbfcfa72078d3e52c,a3e5e6e6bd538336aeddc1280de234164baab11ec49a8f612aa61ddf768b1d5a,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA H100 PCIe,85064286208,14239962112,22705864704,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 16 |
+
H100 PCIe,attention,32,12345,attention_seed12345_20261029_104920_111739,workergpu158_rank0_1302660_15e949a8d27e,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793285380.326085,1793285700.1593618,319.83263114100555,347.7678084373474,12.438171399239144,120209.75695865852,2.5421976941579487,3.227651218120009,49202331648.0,,,d86be67cb9d134b59ba35a3c4a00568a64590a6524f60a89905bf9e3f50fb59f,7762de7231f26e1b8b7cd2f0afa3186285a5d37dd8dbfa4f6ea98fe4b581d171,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA H100 PCIe,85064286208,23358514176,48488251392,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 17 |
+
H100 PCIe,heptv2,32,12345,heptv2_seed12345_20261029_105508_955409,workergpu158_rank0_1345892_3a3528b8bf0e,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793285732.0663805,1793286107.4197798,375.352714884968,407.9657735824585,10.815708218172245,104529.32464990659,2.9217955035564955,3.493805446018232,51238666240.0,,,c8a17f9ba4c57520b1ee7bab486c1b748f8300a6a3d3d7950db1b270965e0d92,00bb79480860940c3d7e1266071542a50521d6711688b644d4f244f0f5f9a2db,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA H100 PCIe,85064286208,29242608128,50524585984,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 18 |
+
H100 PCIe,attention,64,12345,attention_seed12345_20261029_110158_000992,workergpu158_rank0_1397341_2dd8ef558a94,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793286149.1802657,1793286856.8331676,707.6522006490268,748.5393121242523,11.064622905911989,106364.30297920374,5.763027106382651,6.416846876736965,82727403520.0,,,44f1ae23f7db2f0f3ea02abae02263b29e3e3caf12cafb79f05549508ddc15ef,77db909f565c4a61a41cb1832f4b9d3a4c913eee3624744df13146ee519938d7,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA H100 PCIe,85064286208,46279458304,84074823680,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 19 |
+
H100 PCIe,heptv2,64,12345,heptv2_seed12345_20261029_111426_347405,workergpu158_rank0_1498617_e462b89737dc,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793286890.1019304,1793287705.664468,815.5618177449796,849.5290899276733,9.674555130611186,93001.5710297288,6.575758987048175,7.331884939683952,83606110208.0,,,120ddb7e424abb7aefd5142fb554ef521657db2d9456ec2265b96013654444a2,c6ed9d1d2cec4997e396f86f42502e17f2bb9ee446fb716b8298d5e0594ba1df,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA H100 PCIe,85064286208,58236852224,82892029952,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 20 |
+
H100 PCIe,attention,8,12345,attention_seed12345_20261029_103730_624725,workergpu158_rank0_1235013_e1123604fe01,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793284683.2652156,1793284778.9086616,95.64288120501442,133.60123825073242,11.900117044586105,115695.92781187355,0.6475720765884034,1.3319195346685593,13643022336.0,,,0db34b576230877eea086983e40861ad5dc50bf8d1ca4f52f1171e862ecfe8d1,e031741435d457a019cef17d5bd8a719aece39e6e5e5c4402f14eeeaef556df9,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA H100 PCIe,85064286208,6213974016,12928942080,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 21 |
+
H100 PCIe,heptv2,8,12345,heptv2_seed12345_20261029_103949_700914,workergpu158_rank0_1247010_f21641cb321b,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793284821.6578064,1793284940.9059522,119.24748125701444,162.636616230011,10.490352668650944,101989.84434573248,0.734243988757953,1.3537986898882084,12523143168.0,,,4f8709a969efac1aef823ff96817fc20169c2f8304f35c73b5fb409bbf51a4df,4e663095850d9f7ca01a513d78eab2410785b3b11fb2d2e029616f5acac8e9c6,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA H100 PCIe,85064286208,7241464320,11809062912,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
|
| 22 |
+
MI250X (1 GCD),attention,128,12345,attention_seed12345_20261030_175323_702459,nid007882_rank0_1785_c17cbf4f192f,0,oom,failed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,0,0,0,0,1793375612.3400874,1793375631.988239,19.64776315400377,34.78092813491821,,,,,67848052736.0,,,80a1cecbdea69f4107c4bfb888b507a01b27c6669ed0bd9dbfe6e7be315d975c,71ba888bfee367fba3384fc34e32eff9a9cb93310c905b219d23d735e7841829,,AMD Instinct MI250X,68702699520,66094283776,68023222272,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 23 |
+
MI250X (1 GCD),heptv2,128,12345,heptv2_seed12345_20261030_175357_109122,nid007882_rank0_3202_4345cf687189,0,oom,failed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,0,0,0,0,1793375644.9217677,1793375681.3851423,36.46110969898291,50.256638050079346,,,,,68259753984.0,,,66e42957b2fed2e0f60669b51017829302584d0d382d7e9777a36a9bebbd6087,abb9385c1ef8585c13c88b6b433ac2e2600c0bb7ae36bb86921992767114ae72,,AMD Instinct MI250X,68702699520,61789143040,67459088384,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 24 |
+
MI250X (1 GCD),attention,16,12345,attention_seed12345_20261030_163541_037698,nid007882_rank0_77698_8bca43ff61b1,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,1920,18508648,1793370948.8368967,1793371226.9412224,278.10355572309345,290.8300964832306,7.347870070138632,70886.50072836764,2.1180435339920223,3.23811617057887,22263279616.0,,,ea52c515dbbe1c9563883b36b46944ea1301790aaf30fc246a77f609f45b0c4f,d929cbf2063c7ea854d163e1d1bd0cb375d2791b6b763440be12e8fbe903cdce,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,AMD Instinct MI250X,68702699520,11927367168,21353201664,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 25 |
+
MI250X (1 GCD),heptv2,16,12345,heptv2_seed12345_20261030_164032_134980,nid007882_rank0_80833_a2c3ec49ebc1,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,1920,18508648,1793371239.9030964,1793371572.4276152,332.5237337300787,346.27485871315,6.17795217003457,59600.04828892048,2.5454883782658726,3.6300434995302937,23697174528.0,,,a708686d1cdceed0d8d9ee182b51b3218007b6b4b8f000e71b5320b5a2a7ae9e,03bf3af4aa95bc4f7c023240eec8a35c0af0db4570d1dc41c339d76be953f821,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,AMD Instinct MI250X,68702699520,14329852928,22758293504,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 26 |
+
MI250X (1 GCD),attention,32,12345,attention_seed12345_20261030_164618_457690,nid007882_rank0_84145_ef09bea576c4,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,3840,37085423,1793371586.4922276,1793372145.7281442,559.2351652139332,572.4081828594208,7.095937882042429,68579.29038075786,4.479588077578228,5.675255642548437,47670607872.0,,,ee1c65d604d38cfd7ba8f7ff353c172310550c502bd2e9a1de7c66e79c3739cf,70c9e592e9ad861b0155b680eb576f8541cc525c448e8437b4bf39eb2b1841ee,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,AMD Instinct MI250X,68702699520,23447246848,46760198144,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 27 |
+
MI250X (1 GCD),heptv2,32,12345,heptv2_seed12345_20261030_165550_661732,nid007882_rank0_89756_b734840a058e,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,3840,37085423,1793372158.7184627,1793372834.6671145,675.9478112340439,689.7471876144409,5.866772572121572,56699.91853220471,5.390423756791279,6.600580013847212,51459600384.0,,,7cefc73bda64b0d074b0349fd33d1c821a9016f7c9cca64db56a02c9fbacb4b9,25d8ac642d028a4e189705e36d94ef847f6cdc0ea9da2e228f74c5e1fcde084c,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,AMD Instinct MI250X,68702699520,29332572672,50520391680,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 28 |
+
MI250X (1 GCD),attention,64,12345,attention_seed12345_20261030_170721_046856,nid007882_rank0_95827_bf35c0cbc8e9,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,7680,73925923,1793372849.5186274,1793374089.7862635,1240.2667897279607,1254.2669656276703,6.298893842190149,60551.31374667772,10.104078602686059,11.602974367812276,68359630848.0,,,99468c584d577b3fd8ee0040f86788736765ddc8bcf6819a8cfc1b8d8fedba93,9c4fc33641013a47b46f70e542f6b7621eeebc637a10d4652860dbe0a7a738cb,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,AMD Instinct MI250X,68702699520,46362395136,67926753280,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 29 |
+
MI250X (1 GCD),heptv2,64,12345,heptv2_seed12345_20261030_172815_671846,nid007882_rank0_121143_7546f178b8dd,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,7680,73925923,1793374104.72147,1793375596.1859314,1491.4635927228956,1507.1206331253052,5.249767490781045,50466.05426213437,12.11482840788085,13.497417909321376,68279242752.0,,,d56f05a803a61b5157d8bd6ece44213ace216f496229fd381417c2fdc947f86d,a8b202304fa767521a170e6aa070a9ad704d62c0a7e3e0203ae0011ea08831ea,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,AMD Instinct MI250X,68702699520,58328029184,67626860544,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 30 |
+
MI250X (1 GCD),attention,8,12345,attention_seed12345_20261030_162850_951873,nid007882_rank0_73181_e2a0f0197eb7,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,960,9316574,1793370555.6356494,1793370733.4232671,177.78685278398916,207.6635971069336,7.030669051986354,68353.93097901065,1.0991682871826924,2.2399486443423666,13298425856.0,,,3e30d19c78dc7ece2b5602febf23155e0c4f0b7bade287e4d1971ddb8a1f5683,931b0a54d71ac40f987851e7a09bb45fe4b524a526c004d3f1f7ef2c0722080d,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,AMD Instinct MI250X,68702699520,6305170432,12379488256,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
| 31 |
+
MI250X (1 GCD),heptv2,8,12345,heptv2_seed12345_20261030_163218_908588,nid007882_rank0_75513_3099f92c9e96,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,960,9316574,1793370746.7258735,1793370935.0312088,188.30460606899578,202.32765364646912,6.020502489276655,58532.837851998076,1.280680806026794,2.432090930375271,12788461568.0,,,77d2d22e7d9ad4b8ad41b3ff8e9c020a601622c402a4a7562850fd33eabbe181,52c73b40d6692cbabf242e6f5a53f981e30aa5304533917007db5faf9b69bd98,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,AMD Instinct MI250X,68702699520,7332608512,11840520192,20ac400a43a68da3ce1c029e96b3eaf90552f36d
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.png
ADDED
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.png
ADDED
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.png
ADDED
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png
ADDED
|
Git LFS Details
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png
ADDED
|
Git LFS Details
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png
ADDED
|
Git LFS Details
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.png
ADDED
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.svg
ADDED
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/manifest.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"files": [
|
| 3 |
+
{
|
| 4 |
+
"path": "README.md",
|
| 5 |
+
"bytes": 7002,
|
| 6 |
+
"sha256": "5fb19165312d3cd74a04d235193dd8ddc8aad250e6059944a0b5fe7c31d7d308"
|
| 7 |
+
},
|
| 8 |
+
{
|
| 9 |
+
"path": "cases.csv",
|
| 10 |
+
"bytes": 20206,
|
| 11 |
+
"sha256": "d9e090aab5a7fb90e38adcc5826de324295fd1811c1dfa65a30838d0da2a28e8"
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"path": "figures/allocated_gib.png",
|
| 15 |
+
"bytes": 80709,
|
| 16 |
+
"sha256": "8540ea1ff3b4b4b9a08b2f039698f6e5cf9cea8cee7641b41b86b126a0ecf5a7"
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"path": "figures/allocated_gib.svg",
|
| 20 |
+
"bytes": 47364,
|
| 21 |
+
"sha256": "050b6fc9d3ec84587e9329f0c97398c84a10ee898901efaa217c4f5ece9719b0"
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"path": "figures/events_per_second.png",
|
| 25 |
+
"bytes": 82104,
|
| 26 |
+
"sha256": "349b27a480cb8909b8b7cc75b827223ae1a532d1f9a8fe2eeea67fe52e16bbd4"
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"path": "figures/events_per_second.svg",
|
| 30 |
+
"bytes": 47035,
|
| 31 |
+
"sha256": "ce2a4e19fa5f8bd6da39cac3482e34bd8e812b47c277f5de84d6af6b3f58e19e"
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"path": "figures/khits_per_second.png",
|
| 35 |
+
"bytes": 85807,
|
| 36 |
+
"sha256": "258b9cf7e95d477a3e697571fe97d7d2d9d321732803e10a409bb871003525b0"
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"path": "figures/khits_per_second.svg",
|
| 40 |
+
"bytes": 47490,
|
| 41 |
+
"sha256": "53882f3d7b61d048b3646f3183034c053814b999662028b3e3684bc11668c56d"
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"path": "figures/p99_over_p50.png",
|
| 45 |
+
"bytes": 105379,
|
| 46 |
+
"sha256": "e241eb32bc6d4f38f08dac6ec1126db719ae5217675572063fcace3ff87fe125"
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"path": "figures/p99_over_p50.svg",
|
| 50 |
+
"bytes": 46133,
|
| 51 |
+
"sha256": "625700ea3deff968da9a2610dd47c69afe824b6d649a9da9badf60a8af34faf7"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"path": "figures/training_loss_1.png",
|
| 55 |
+
"bytes": 238301,
|
| 56 |
+
"sha256": "9040cc4534a92f0cb960a372dedf7f787f03addcbb77fbf0ddc5e8ed0cbfa0a1"
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"path": "figures/training_loss_1.svg",
|
| 60 |
+
"bytes": 115074,
|
| 61 |
+
"sha256": "e3ee8db077208248d55a6e7140ac690473c44f5c16a96b9bf3973c0057741ad4"
|
| 62 |
+
},
|
| 63 |
+
{
|
| 64 |
+
"path": "figures/training_loss_2.png",
|
| 65 |
+
"bytes": 246512,
|
| 66 |
+
"sha256": "8307c896b3946b81872fa4fc2f3e17eb1a6bd7c4d49f12e47da4dc210d7f0f1a"
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"path": "figures/training_loss_2.svg",
|
| 70 |
+
"bytes": 112017,
|
| 71 |
+
"sha256": "7ebdc363d6bf6f96ef4a4226902e8a6d24d9dc9a1288d743584e8c0882138afc"
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"path": "figures/validation_loss.png",
|
| 75 |
+
"bytes": 98278,
|
| 76 |
+
"sha256": "fd8e87b1d9a94d26f201863258b1a82d959241fa17547a8112ba6bfc93408682"
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"path": "figures/validation_loss.svg",
|
| 80 |
+
"bytes": 48587,
|
| 81 |
+
"sha256": "8ba148bf29473e842450ecbbbaff46470f2cbad8118cfd27760550dc34c78a6e"
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"path": "render_plots.py",
|
| 85 |
+
"bytes": 4891,
|
| 86 |
+
"sha256": "23f1a531cb752e3fb8cb71f6c879d8a277ef349a5fb3aab660ce3ec5811a7fb7"
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"path": "results.json",
|
| 90 |
+
"bytes": 237951,
|
| 91 |
+
"sha256": "78df2ade8ca3ef41cb0704f32fd322c21c4a6fe0ff761160fc80102b4c8595e4"
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"path": "slides/gpu_batch_sweep.slides.html",
|
| 95 |
+
"bytes": 1422163,
|
| 96 |
+
"sha256": "2170b1bde53265981104ade7d6799fa4b9567b2b54a44ebf85530714d654eaeb"
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
"path": "slides/gpu_batch_sweep.slides.pdf",
|
| 100 |
+
"bytes": 1077663,
|
| 101 |
+
"sha256": "6aa345063e56d21b11fec33aec4984e5dd310d67d473d8dea5557ca2bf69302a"
|
| 102 |
+
},
|
| 103 |
+
{
|
| 104 |
+
"path": "steps.csv",
|
| 105 |
+
"bytes": 539440,
|
| 106 |
+
"sha256": "bfcb71811c4cf240bcac42656963e37d46891e9837c3fe1df3e654a5f5c6e664"
|
| 107 |
+
},
|
| 108 |
+
{
|
| 109 |
+
"path": "validation.csv",
|
| 110 |
+
"bytes": 3721,
|
| 111 |
+
"sha256": "14c0a2f8ba86bf5b254da2d558d01d2a27cda35585c834ce91ba5c23eef06769"
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/render_plots.py
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Render portable sweep CSVs. Requires matplotlib, numpy and pandas; no training stack."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
def render(directory):
|
| 8 |
+
import matplotlib
|
| 9 |
+
|
| 10 |
+
matplotlib.use("Agg")
|
| 11 |
+
import matplotlib.pyplot as plt
|
| 12 |
+
import pandas as pd
|
| 13 |
+
|
| 14 |
+
directory = Path(directory)
|
| 15 |
+
cases = pd.read_csv(directory / "cases.csv")
|
| 16 |
+
if (cases.world_size != 1).any() or cases.duplicated(["platform", "variant", "batch"]).any():
|
| 17 |
+
raise ValueError("Plot one single-device session per platform/model/batch; select repeats explicitly")
|
| 18 |
+
completed = cases[cases.status == "completed"].copy()
|
| 19 |
+
if completed.empty:
|
| 20 |
+
raise ValueError("No completed cases to plot")
|
| 21 |
+
completed["allocated_gib"] = completed.allocated_bytes / 2**30
|
| 22 |
+
completed["khits_per_second"] = completed.hits_per_second / 1000
|
| 23 |
+
completed["p99_over_p50"] = completed.step_p99_seconds / completed.step_p50_seconds
|
| 24 |
+
platforms = list(cases.platform.drop_duplicates())
|
| 25 |
+
variants = list(cases.variant.drop_duplicates())
|
| 26 |
+
colors = dict(zip(platforms, plt.get_cmap("tab10").colors * (1 + len(platforms) // 10)))
|
| 27 |
+
figdir = directory / "figures"
|
| 28 |
+
figdir.mkdir(exist_ok=True)
|
| 29 |
+
plt.rcParams.update({"font.size": 12, "axes.spines.top": False, "axes.spines.right": False})
|
| 30 |
+
|
| 31 |
+
def save(fig, name):
|
| 32 |
+
for ext in ("png", "svg"):
|
| 33 |
+
fig.savefig(figdir / f"{name}.{ext}", dpi=160, bbox_inches="tight")
|
| 34 |
+
plt.close(fig)
|
| 35 |
+
|
| 36 |
+
def compare(data, metric, ylabel, name):
|
| 37 |
+
fig, axes = plt.subplots(1, len(variants), figsize=(6 * len(variants), 4), squeeze=False, layout="constrained", sharey=True)
|
| 38 |
+
for ax, variant in zip(axes.flat, variants):
|
| 39 |
+
for platform in platforms:
|
| 40 |
+
rows = data[(data.platform == platform) & (data.variant == variant)].sort_values("batch")
|
| 41 |
+
ax.plot(rows.batch, rows[metric], "-o", label=platform, color=colors[platform])
|
| 42 |
+
ax.set(title=variant, xlabel="Training batch size", ylabel=ylabel, xscale="log")
|
| 43 |
+
batches = sorted(data.batch.unique())
|
| 44 |
+
ax.set_xticks(batches, labels=batches)
|
| 45 |
+
ax.xaxis.set_minor_locator(matplotlib.ticker.NullLocator())
|
| 46 |
+
ax.grid(alpha=0.2)
|
| 47 |
+
axes.flat[-1].legend(fontsize=10)
|
| 48 |
+
save(fig, name)
|
| 49 |
+
|
| 50 |
+
for metric, ylabel in [
|
| 51 |
+
("events_per_second", "Steady training events/s"),
|
| 52 |
+
("khits_per_second", "Steady training kHits/s"),
|
| 53 |
+
("allocated_gib", "Process peak allocated GiB"),
|
| 54 |
+
("p99_over_p50", "Host step p99 / p50"),
|
| 55 |
+
]:
|
| 56 |
+
compare(completed, metric, ylabel, metric)
|
| 57 |
+
steps = pd.read_csv(directory / "steps.csv")
|
| 58 |
+
keys = completed[["platform", "variant", "batch", "seed", "run_id", "session_id", "rank", "warmup_steps"]]
|
| 59 |
+
steps = steps.merge(keys, on=list(keys.columns[:-1]), validate="many_to_one")
|
| 60 |
+
for index, variant in enumerate(variants):
|
| 61 |
+
selected = steps[steps.variant == variant]
|
| 62 |
+
batches = sorted(selected.batch.unique())
|
| 63 |
+
ncols = min(2, len(batches))
|
| 64 |
+
nrows = (len(batches) + ncols - 1) // ncols
|
| 65 |
+
fig, axes = plt.subplots(nrows, ncols, figsize=(6 * ncols, 3.5 * nrows), squeeze=False, layout="constrained", sharey=True)
|
| 66 |
+
for ax, batch in zip(axes.flat, batches):
|
| 67 |
+
for i, platform in enumerate(platforms):
|
| 68 |
+
rows = selected[(selected.platform == platform) & (selected.batch == batch)].sort_values("step")
|
| 69 |
+
ax.plot(rows.step, rows.total_unweighted, ["-", "--", ":", "-."][i % 4], label=platform, color=colors[platform])
|
| 70 |
+
ax.set(title=f"{variant}, batch {batch}", xlabel="Training step", ylabel="Total unweighted loss")
|
| 71 |
+
ax.grid(alpha=0.2)
|
| 72 |
+
for ax in list(axes.flat)[len(batches) :]:
|
| 73 |
+
ax.set_visible(False)
|
| 74 |
+
axes.flat[0].legend(fontsize=10)
|
| 75 |
+
save(fig, f"training_loss_{index + 1}")
|
| 76 |
+
validation = pd.read_csv(directory / "validation.csv")
|
| 77 |
+
if not validation.empty:
|
| 78 |
+
endpoints = validation.merge(
|
| 79 |
+
completed[["platform", "variant", "batch", "seed", "run_id", "session_id", "rank", "steps"]],
|
| 80 |
+
on=["platform", "variant", "batch", "seed", "run_id", "session_id", "rank"],
|
| 81 |
+
validate="many_to_one",
|
| 82 |
+
)
|
| 83 |
+
endpoints = endpoints[endpoints.step == endpoints.steps]
|
| 84 |
+
if endpoints.step.nunique() != 1:
|
| 85 |
+
raise ValueError("Validation endpoints have different training lengths; compare explicitly")
|
| 86 |
+
compare(endpoints, "valid_total_unweighted", f"Validation loss at step {endpoints.step.iloc[0]}", "validation_loss")
|
| 87 |
+
print(f"Wrote figures to {figdir}")
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def main(argv=None):
|
| 91 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 92 |
+
parser.add_argument("directory", nargs="?", type=Path, default=Path(__file__).resolve().parent)
|
| 93 |
+
args = parser.parse_args(argv)
|
| 94 |
+
render(args.directory)
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
if __name__ == "__main__":
|
| 98 |
+
main()
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/results.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.html
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6aa345063e56d21b11fec33aec4984e5dd310d67d473d8dea5557ca2bf69302a
|
| 3 |
+
size 1077663
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/steps.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmarks/v3.2.0/cld_gpu_batch_sweep/validation.csv
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
platform,variant,batch,seed,run_id,session_id,rank,step,train_total_unweighted,valid_total_unweighted
|
| 2 |
+
GH200,attention,16,12345,attention_seed12345_20261029_062032_191329,alphagh01_rank0_2963140_f80016108aa1,0,120,3.4413122558034956,3.3920575259253383
|
| 3 |
+
GH200,heptv2,16,12345,heptv2_seed12345_20261029_062318_186034,alphagh01_rank0_2966282_3c1d7090284d,0,120,3.2092463737353683,3.2746193357743323
|
| 4 |
+
GH200,attention,32,12345,attention_seed12345_20261029_062618_170573,alphagh01_rank0_2969691_16a386b5ec65,0,120,3.3299973998218775,3.5785752404481173
|
| 5 |
+
GH200,heptv2,32,12345,heptv2_seed12345_20261029_063114_050085,alphagh01_rank0_2974716_6b9de441032e,0,120,3.210159281268716,3.3225052603520453
|
| 6 |
+
GH200,attention,64,12345,attention_seed12345_20261029_063630_231892,alphagh01_rank0_2980492_cefdd4e73f6b,0,120,3.325105535797775,3.5525365620851517
|
| 7 |
+
GH200,heptv2,64,12345,heptv2_seed12345_20261029_064622_331901,alphagh01_rank0_2991102_2bb5ab0d5fc0,0,120,2.8526693456806242,3.100961511954665
|
| 8 |
+
GH200,attention,8,12345,attention_seed12345_20261029_061642_376997,alphagh01_rank0_2958727_aca0c7649228,0,120,3.4185724649578333,3.4165319274179637
|
| 9 |
+
GH200,heptv2,8,12345,heptv2_seed12345_20261029_061831_709291,alphagh01_rank0_2960929_85567b5b0c80,0,120,3.235496461391449,3.3162532565183938
|
| 10 |
+
H100 PCIe,attention,16,12345,attention_seed12345_20261029_104229_564372,workergpu158_rank0_1259903_683085c8df3b,0,120,3.4205480571836233,3.3728219415061176
|
| 11 |
+
H100 PCIe,heptv2,16,12345,heptv2_seed12345_20261029_104538_814383,workergpu158_rank0_1276037_80bfd5ff3556,0,120,3.1876705032773316,3.258329316973686
|
| 12 |
+
H100 PCIe,attention,32,12345,attention_seed12345_20261029_104920_111739,workergpu158_rank0_1302660_15e949a8d27e,0,120,3.316897104959935,3.4092630185186863
|
| 13 |
+
H100 PCIe,heptv2,32,12345,heptv2_seed12345_20261029_105508_955409,workergpu158_rank0_1345892_3a3528b8bf0e,0,120,3.1633353787474334,3.3145826999098063
|
| 14 |
+
H100 PCIe,attention,64,12345,attention_seed12345_20261029_110158_000992,workergpu158_rank0_1397341_2dd8ef558a94,0,120,3.3548554317094386,3.5806318521499634
|
| 15 |
+
H100 PCIe,heptv2,64,12345,heptv2_seed12345_20261029_111426_347405,workergpu158_rank0_1498617_e462b89737dc,0,120,2.865398366935551,3.115826594643295
|
| 16 |
+
H100 PCIe,attention,8,12345,attention_seed12345_20261029_103730_624725,workergpu158_rank0_1235013_e1123604fe01,0,120,3.4219354861415923,3.3811078616417944
|
| 17 |
+
H100 PCIe,heptv2,8,12345,heptv2_seed12345_20261029_103949_700914,workergpu158_rank0_1247010_f21641cb321b,0,120,3.2621626425534487,3.2805806999094784
|
| 18 |
+
MI250X (1 GCD),attention,16,12345,attention_seed12345_20261030_163541_037698,nid007882_rank0_77698_8bca43ff61b1,0,120,3.39885445125401,3.375275770202279
|
| 19 |
+
MI250X (1 GCD),heptv2,16,12345,heptv2_seed12345_20261030_164032_134980,nid007882_rank0_80833_a2c3ec49ebc1,0,120,3.2108154979068786,3.2590630035847425
|
| 20 |
+
MI250X (1 GCD),attention,32,12345,attention_seed12345_20261030_164618_457690,nid007882_rank0_84145_ef09bea576c4,0,120,3.3251947122626007,3.5192190841771662
|
| 21 |
+
MI250X (1 GCD),heptv2,32,12345,heptv2_seed12345_20261030_165550_661732,nid007882_rank0_89756_b734840a058e,0,120,3.172804694622755,3.3399899369105697
|
| 22 |
+
MI250X (1 GCD),attention,64,12345,attention_seed12345_20261030_170721_046856,nid007882_rank0_95827_bf35c0cbc8e9,0,120,3.3100380515679717,3.5173501721583307
|
| 23 |
+
MI250X (1 GCD),heptv2,64,12345,heptv2_seed12345_20261030_172815_671846,nid007882_rank0_121143_7546f178b8dd,0,120,2.812488204333931,3.0608929730951786
|
| 24 |
+
MI250X (1 GCD),attention,8,12345,attention_seed12345_20261030_162850_951873,nid007882_rank0_73181_e2a0f0197eb7,0,120,3.380337958689779,3.359904798679054
|
| 25 |
+
MI250X (1 GCD),heptv2,8,12345,heptv2_seed12345_20261030_163218_908588,nid007882_rank0_75513_3099f92c9e96,0,120,3.289946631528437,3.290073842741549
|