jpata commited on
Commit
5a1fbd8
·
verified ·
1 Parent(s): 0acee68

Add portable CLD benchmark evidence for v3.2.0

Browse files
Files changed (24) hide show
  1. .gitattributes +4 -0
  2. benchmarks/v3.2.0/cld_gpu_batch_sweep/README.md +123 -0
  3. benchmarks/v3.2.0/cld_gpu_batch_sweep/cases.csv +31 -0
  4. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.png +0 -0
  5. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.svg +1553 -0
  6. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.png +0 -0
  7. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.svg +1549 -0
  8. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.png +0 -0
  9. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.svg +1566 -0
  10. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png +3 -0
  11. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.svg +1509 -0
  12. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png +3 -0
  13. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.svg +0 -0
  14. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png +3 -0
  15. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.svg +0 -0
  16. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.png +0 -0
  17. benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.svg +1563 -0
  18. benchmarks/v3.2.0/cld_gpu_batch_sweep/manifest.json +114 -0
  19. benchmarks/v3.2.0/cld_gpu_batch_sweep/render_plots.py +98 -0
  20. benchmarks/v3.2.0/cld_gpu_batch_sweep/results.json +0 -0
  21. benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.html +0 -0
  22. benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf +3 -0
  23. benchmarks/v3.2.0/cld_gpu_batch_sweep/steps.csv +0 -0
  24. benchmarks/v3.2.0/cld_gpu_batch_sweep/validation.csv +25 -0
.gitattributes CHANGED
@@ -100,3 +100,7 @@ images/event_displays/pf_event.png filter=lfs diff=lfs merge=lfs -text
100
  images/event_displays/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
101
  images/pf_event.png filter=lfs diff=lfs merge=lfs -text
102
  images/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
100
  images/event_displays/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
101
  images/pf_event.png filter=lfs diff=lfs merge=lfs -text
102
  images/pf_event_transparent.png filter=lfs diff=lfs merge=lfs -text
103
+ benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png filter=lfs diff=lfs merge=lfs -text
104
+ benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png filter=lfs diff=lfs merge=lfs -text
105
+ benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png filter=lfs diff=lfs merge=lfs -text
106
+ benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf filter=lfs diff=lfs merge=lfs -text
benchmarks/v3.2.0/cld_gpu_batch_sweep/README.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CLD GPU batch-sweep benchmark evidence, v3.2.0
2
+
3
+ Recorded on 29–30 October 2026. **v3.2.0 is the benchmark evidence release tag,
4
+ not the TFDS dataset version.** These are full platform/recipe measurements,
5
+ not isolated hardware rankings or a convergence study.
6
+
7
+ ## Results
8
+
9
+ | Platform | Model | Best tested batch | Steady events/s | Steady kHits/s |
10
+ | --- | --- | ---: | ---: | ---: |
11
+ | GH200 | Attention | 32 | 14.55 | 140.6 |
12
+ | GH200 | HEPTv2 | 32 | 13.83 | 133.7 |
13
+ | H100 PCIe | Attention | 16 | 12.81 | 123.6 |
14
+ | H100 PCIe | HEPTv2 | 16 | 11.21 | 108.1 |
15
+ | MI250X (one GCD) | Attention | 16 | 7.35 | 70.9 |
16
+ | MI250X (one GCD) | HEPTv2 | 16 | 6.18 | 59.6 |
17
+
18
+ All 24 cases at batches 8/16/32/64 complete. Both models run out of device memory
19
+ at batch 128 on all three platforms. H100 HEPTv2 wraps its OOM in a fused-attention
20
+ RuntimeError; the original chained exception and sweep state identify OOM.
21
+ Batch 64 is the largest **tested** successful batch, not the exact capacity limit.
22
+ All six recipes slow down at batch 64 relative to their observed throughput peak.
23
+
24
+ ![Training event throughput](figures/events_per_second.png)
25
+ ![Training hit throughput](figures/khits_per_second.png)
26
+ ![Framework allocated memory](figures/allocated_gib.png)
27
+
28
+ ## Total unweighted losses
29
+
30
+ The plots retain all 120 raw training losses per completed case, without smoothing
31
+ or warmup exclusion. Compare platforms within a fixed model and batch size.
32
+ `Total_unweighted` is the fixed sum of binary classification, class classification,
33
+ and the five regression tasks (pT, eta, sin(phi), cos(phi), energy). It excludes
34
+ cardinality, hit grouping and auxiliary losses, and differs from calibrated `Total`.
35
+
36
+ ![Attention loss](figures/training_loss_1.png)
37
+ ![HEPTv2 loss](figures/training_loss_2.png)
38
+ ![Step-120 validation loss](figures/validation_loss.png)
39
+
40
+ Training trajectories are broadly similar across platforms, with some transient
41
+ differences. The largest step-120 validation spread at a matched batch is 0.169
42
+ for Attention at batch 32. There is only one validation point per run, based on
43
+ 32 validation events. One seed and one run do not support statistical significance
44
+ or a convergence claim. Larger training batches process more events in 120 steps.
45
+
46
+ ## Protocol and scope
47
+
48
+ - One visible device and one rank. LUMI uses **one 64 GiB MI250X GCD**, half of a
49
+ physical two-GCD module. H100 is PCIe. Observed PyTorch capacities are about
50
+ 95.0 GiB (GH200), 79.2 GiB (H100), and 64.0 GiB (MI250X GCD).
51
+ - Seed 12345, BF16, LAMB, LR 0.001, cosine schedule. Six-layer, width-256 backbones
52
+ with 16 heads, and a shared four-layer, 256-slot set decoder. HEPTv2 enables
53
+ activation checkpointing. Top-level model compilation is disabled; internal
54
+ compiled kernels may still run.
55
+ - 120 steps per case. Throughput and p50/p99 exclude the first 20 warmup steps.
56
+ Throughput = summed events or hits / summed host step durations. Those durations
57
+ include data wait and existing synchronizations, but exclude validation,
58
+ checkpoint writes, launch and teardown. The raw CSV includes the full 120 steps.
59
+ - End-of-case validation uses 32 events and testing uses 8 events. Process memory
60
+ peaks include evaluation; allocated, reserved and device-wide memory differ.
61
+ - Failed cases retain observed work (GH200 Attention batch 128 completes two
62
+ steps/256 events before failure). Failed throughput is unavailable, never zero.
63
+ - GPU busy measures device activity, not SM occupancy or FLOP efficiency. LUMI
64
+ power coverage is zero/unavailable. GH200 Attention batch 32 has no usable
65
+ phase-attributed busy/power interval. No energy or scheduler-cost ranking is made.
66
+
67
+ ## Provenance and limitations
68
+
69
+ | Platform | Original sweep | Job | Recorded Git base | TFDS |
70
+ | --- | --- | --- | --- | --- |
71
+ | GH200 | gh200_batch_sweep_153260 | 153260 | d2d991cea54ab47382e9de63bc900d2850e75eb4 | 3.2.2 |
72
+ | H100 PCIe | h100_approx_321_full_20261029_1038 | 7363958 | 9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0 | 3.2.1 |
73
+ | MI250X (1 GCD) | lumi_retry_20261030_144129 | 23308832 | 20ac400a43a68da3ce1c029e96b3eaf90552f36d | 3.2.2 |
74
+
75
+ All runs use CLD `cld_edm_ttbar_hits`, config `1`, and all recorded Git trees were
76
+ dirty. The analysis originally reviewed the branch at `9ee676ac`. This release
77
+ preserves historical provenance and does not relabel the runs as measurements of
78
+ the later portable-launcher changes.
79
+
80
+ NVIDIA runs use PyTorch 2.11/CUDA 12.8 and native attention. LUMI uses its PyTorch
81
+ 2.11/ROCm 7.2 build and external FlashAttention. GH200 is aarch64; the others are
82
+ x86_64. NVIDIA uses four loader workers; LUMI uses two and file-descriptor sharing.
83
+ The package preserves per-case model/data/runtime configuration and exact package
84
+ versions. `data_verification.status=not_recorded` means the original sweep did
85
+ not record the new preflight result, not that its data was verified by this release.
86
+
87
+ All 120 per-step sequence-length vectors match across platforms for each completed
88
+ model/batch pair. The `sequence_lengths_sha256` values retain that check without
89
+ publishing the full length vectors. Matching sizes **do not prove matching dataset
90
+ payloads, features or targets**. TFDS versions, dirty code bases and backends differ.
91
+ Loss and throughput comparisons therefore describe these recorded platforms.
92
+
93
+ ## Files and regeneration
94
+
95
+ - [19-slide PDF](slides/gpu_batch_sweep.slides.pdf) and
96
+ [standalone HTML slides](slides/gpu_batch_sweep.slides.html). Download the HTML
97
+ to present offline (P enters presentation mode, arrows navigate, Esc exits).
98
+ - `results.json`: all 30 cases, scientific/runtime configuration, memory and
99
+ telemetry scope, source provenance, raw-source hashes and sequence signatures.
100
+ - `cases.csv`: flat per-case metrics. `steps.csv`: host-step work/timing and
101
+ unweighted losses, including partial work from failed cases.
102
+ - `validation.csv`: train/validation loss snapshots for 24 completed cases.
103
+ - `figures/`: PNG and SVG plots. `render_plots.py`: standalone plotting source.
104
+ - `manifest.json`: file sizes and SHA-256 digests for this package.
105
+
106
+ With Python, pandas, numpy and matplotlib installed:
107
+
108
+ ```bash
109
+ python render_plots.py
110
+ ```
111
+
112
+ The script regenerates the seven data figures from the compact CSVs. The slides
113
+ are archived presentation outputs, not regenerated by that command. Full original
114
+ inputs, histories, TensorBoard writers and frozen study sources remain in the dated
115
+ local/private study archive. Checkpoints, predictions and dataset caches are
116
+ intentionally omitted from this compact public release. Machine-specific absolute
117
+ paths and device UUIDs are omitted from exported metadata.
118
+
119
+ The generic exporter reads raw sweep directories without importing training/GPU
120
+ libraries. The plotting script requires one single-device session per platform,
121
+ model and batch and rejects ambiguous repeats. Raw throughput is checked against
122
+ summaries during export. An additive Hub commit carries this package under
123
+ `benchmarks/v3.2.0/cld_gpu_batch_sweep/`, with tag `v3.2.0` pointing to that commit.
benchmarks/v3.2.0/cld_gpu_batch_sweep/cases.csv ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ platform,variant,batch,seed,run_id,session_id,rank,status,worker_status,source_sweep,world_size,architecture,python,torch,cuda_runtime,hip_runtime,dirty,steps,warmup_steps,events,hits,started_at_unix_time,finished_at_unix_time,worker_seconds,case_seconds,events_per_second,hits_per_second,step_p50_seconds,step_p99_seconds,device_memory_peak_bytes,scenario_sha256,spec_sha256,source_summary_sha256,source_metadata_sha256,sequence_lengths_sha256,device_name,device_memory_bytes,allocated_bytes,reserved_bytes,git_revision
2
+ GH200,attention,128,12345,attention_seed12345_20261029_065651_646012,alphagh01_rank0_3001875_7b652ed64458,0,oom,failed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,2,2,256,2452792,1793271428.2599475,1793271469.4534717,41.1932246722281,61.89646673202515,,,,,101581848576.0,,,c9f1576626f3b6bd348d8bdb4e2187370dbfd728a3a6538e3e4056ee68d1520d,493a3e099a1211a67dbf909f96b6ef1625afb1155dfaf7d63d6c430cc0ae61fc,d650c7bfed7b1bfa372dc0e96d993e9bbd1174ff0948d0c6385f20d81118b5a1,NVIDIA GH200 480GB,102005473280,86645364224,100908662784,d2d991cea54ab47382e9de63bc900d2850e75eb4
3
+ GH200,heptv2,128,12345,heptv2_seed12345_20261029_065753_665332,alphagh01_rank0_3003853_a3fb6b54d52b,0,oom,failed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,0,0,0,0,1793271490.253663,1793271517.012132,26.75677142944187,48.1813178062439,,,,,98864988160.0,,,02ebca65d9f4cc0ce2651302ec410a5023ebcf4dc3355186e14d35beb1b53b6d,292cf2cdb3167b619bbb9600b902f18f25c082c2b351062452ee1a09ac34f9c2,,NVIDIA GH200 480GB,102005473280,93710633472,98039758848,d2d991cea54ab47382e9de63bc900d2850e75eb4
4
+ GH200,attention,16,12345,attention_seed12345_20261029_062032_191329,alphagh01_rank0_2963140_f80016108aa1,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793269248.502339,1793269394.1430855,145.64032928738743,166.31411004066467,14.16103246161317,136614.56018174265,1.1265132501721382,1.3897758059576155,58485374976.0,,,426068a7a9794ae46693f6ec6a0109aa0f9c34af05d40c8bfcd431389953cdac,4f17b0f20fad5bdb358e7c12ded1a6376736a1bbfdd46bc8bf35db9f95cb00ba,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA GH200 480GB,102005473280,11841815040,21887975424,d2d991cea54ab47382e9de63bc900d2850e75eb4
5
+ GH200,heptv2,16,12345,heptv2_seed12345_20261029_062318_186034,alphagh01_rank0_2966282_3c1d7090284d,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793269414.3337042,1793269573.2979531,158.9638429991901,179.52954769134521,13.4722942636621,129970.15298555062,1.1703924494795501,1.4493609412107618,59380858880.0,,,4dc39826e1c73d032711a5adbdf41082f89c8def27ea25c019a6bef750c2fec2,1bf689cc891339f82d0e08ea5831d2527cd1bdcd06933202ab827f968c75a0c9,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA GH200 480GB,102005473280,14239806464,22789750784,d2d991cea54ab47382e9de63bc900d2850e75eb4
6
+ GH200,attention,32,12345,attention_seed12345_20261029_062618_170573,alphagh01_rank0_2969691_16a386b5ec65,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793269594.5457299,1793269869.5912952,275.0450903531164,295.8008642196655,14.547862661382029,140599.04616683867,2.1722677536308765,2.530936222132296,,,,4f309e232952ddc66592a6007933599b86ecc9c604701da945fefa183250a443,64b205c60d4771926f1db38db065022b2ede9ceb37a760c13c11348dd32d2602,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA GH200 480GB,102005473280,23358533632,48486154240,d2d991cea54ab47382e9de63bc900d2850e75eb4
7
+ GH200,heptv2,32,12345,heptv2_seed12345_20261029_063114_050085,alphagh01_rank0_2974716_6b9de441032e,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793269890.665387,1793270185.674123,295.0081489570439,316.4379062652588,13.830442036952725,133665.4740096765,2.285853264387697,2.5668283491674813,86391128064.0,,,cec824d51019a3bf432a632fa23e13d1da9dbec1397a04ad3dc7b72bb4879d9b,632e01cb75d5c931bdd4f3d0d700cd41c1a6d36175d6113d74639608b619feca,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA GH200 480GB,102005473280,29242608128,49798971392,d2d991cea54ab47382e9de63bc900d2850e75eb4
8
+ GH200,attention,64,12345,attention_seed12345_20261029_063630_231892,alphagh01_rank0_2980492_cefdd4e73f6b,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793270206.9355567,1793270778.2902935,571.3542380556464,592.1047451496124,13.837633818324782,133021.27767780976,4.618271645624191,5.18914441172034,101188632576.0,,,aa17684bc3e406284a5d1eb501187a35f8c071480d572bd385bf84fa14296650,dd6ff44afda20afa4f91e291dd68cd1055b61b867339e61ac66d6927f05b36b8,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA GH200 480GB,102005473280,46277582336,96154419200,d2d991cea54ab47382e9de63bc900d2850e75eb4
9
+ GH200,heptv2,64,12345,heptv2_seed12345_20261029_064622_331901,alphagh01_rank0_2991102_2bb5ab0d5fc0,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793270798.354628,1793271407.1352649,608.7801285814494,629.1579208374023,13.081137392713957,125749.07186468971,4.868631860706955,5.330362756364049,101357453312.0,,,4a36b6a4ded75a0cfacfd41a5f045cdc442c1b2acd6acbfb5bbf3d52fa5fede9,57605d3217699f464b32d3d50d062d1870ade1b9889f03ce169f1c881c651b60,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA GH200 480GB,102005473280,58237065216,99444850688,d2d991cea54ab47382e9de63bc900d2850e75eb4
10
+ GH200,attention,8,12345,attention_seed12345_20261029_061642_376997,alphagh01_rank0_2958727_aca0c7649228,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793269020.9687698,1793269107.0918162,86.12262949720025,108.35192370414734,12.56589640592102,122168.80203983614,0.6320971548557281,0.8602891777828339,49381638144.0,,,54cb095fcc22c17ca61335dfb048b9a8d8d48b682a6d6111276e442d492295ea,38048fb5876cbd44603b16067a3d2ad36474cba29abb272e64111924b8220c1f,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA GH200 480GB,102005473280,6212977664,12784238592,d2d991cea54ab47382e9de63bc900d2850e75eb4
11
+ GH200,heptv2,8,12345,heptv2_seed12345_20261029_061831_709291,alphagh01_rank0_2960929_85567b5b0c80,0,completed,completed,gh200_batch_sweep_153260,1,aarch64,3.11.18,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793269129.363011,1793269227.2062633,97.84284777846187,120.66704940795898,11.59590143653919,112738.26723626994,0.6744112796150148,0.939555611154065,48215621632.0,,,9275dae65558becf6e800ae68da0681aa9597c8c49fd49e8488a48b0ecd0262d,72e022d2dfd405d70687ce346725656b4334d1393264bcffc867f5fed4258ea3,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA GH200 480GB,102005473280,7242431488,11624513536,d2d991cea54ab47382e9de63bc900d2850e75eb4
12
+ H100 PCIe,attention,128,12345,attention_seed12345_20261029_112837_250666,workergpu158_rank0_1636267_32b4965342eb,0,oom,failed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,0,0,0,0,1793287749.437335,1793287760.1536057,10.715927304991055,53.80491065979004,,,,,79621521408.0,,,c19afde5deb9218f17fec97b131bcab8d59bd63352ab8dc19be93b1d4898e907,2faa3dd186ac2cf4638c999d1a87181d9bb43af564d8720daf7865d11e10c8ea,,NVIDIA H100 PCIe,85064286208,82823076864,84414562304,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
13
+ H100 PCIe,heptv2,128,12345,heptv2_seed12345_20261029_112929_059759,workergpu158_rank0_1645833_65e5cf1b3e6f,0,oom,failed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,0,0,0,0,1793287788.980532,1793287815.3894665,26.407711561012547,55.7578341960907,,,,,71440531456.0,,,b84797600bfc6b47c003bb90214f64d6de40122afc5308c7e802d5abb3603322,bd04558f9874d86e8a1c4dac8e9f08530632fdda541f3769439150bba7e25fba,,NVIDIA H100 PCIe,85064286208,78372063744,84064337920,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
14
+ H100 PCIe,attention,16,12345,attention_seed12345_20261029_104229_564372,workergpu158_rank0_1259903_683085c8df3b,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793284969.513365,1793285130.2373576,160.72338473401032,188.47921133041382,12.809366597085937,123574.74559932291,1.2254043756984174,1.813039491510718,22564306944.0,,,cc4746edf0c3de7a483bdde44219c516f119a358f247938acab75f344c68d402,1286ef919b1653a4ca9c02bff283bc719eb65f63b680eafd9f091bf2c7d27005,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA H100 PCIe,85064286208,11846225408,21850226688,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
15
+ H100 PCIe,heptv2,16,12345,heptv2_seed12345_20261029_104538_814383,workergpu158_rank0_1276037_80bfd5ff3556,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,1920,18508648,1793285161.1348226,1793285351.6403244,190.50486456701765,222.06918454170227,11.2054484043102,108101.39794140843,1.3936438743548933,1.995739113763557,23419944960.0,,,1be9b64d044ff72abeb43e95730932459f28c7527c58ce6bbfcfa72078d3e52c,a3e5e6e6bd538336aeddc1280de234164baab11ec49a8f612aa61ddf768b1d5a,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,NVIDIA H100 PCIe,85064286208,14239962112,22705864704,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
16
+ H100 PCIe,attention,32,12345,attention_seed12345_20261029_104920_111739,workergpu158_rank0_1302660_15e949a8d27e,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793285380.326085,1793285700.1593618,319.83263114100555,347.7678084373474,12.438171399239144,120209.75695865852,2.5421976941579487,3.227651218120009,49202331648.0,,,d86be67cb9d134b59ba35a3c4a00568a64590a6524f60a89905bf9e3f50fb59f,7762de7231f26e1b8b7cd2f0afa3186285a5d37dd8dbfa4f6ea98fe4b581d171,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA H100 PCIe,85064286208,23358514176,48488251392,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
17
+ H100 PCIe,heptv2,32,12345,heptv2_seed12345_20261029_105508_955409,workergpu158_rank0_1345892_3a3528b8bf0e,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,3840,37085423,1793285732.0663805,1793286107.4197798,375.352714884968,407.9657735824585,10.815708218172245,104529.32464990659,2.9217955035564955,3.493805446018232,51238666240.0,,,c8a17f9ba4c57520b1ee7bab486c1b748f8300a6a3d3d7950db1b270965e0d92,00bb79480860940c3d7e1266071542a50521d6711688b644d4f244f0f5f9a2db,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,NVIDIA H100 PCIe,85064286208,29242608128,50524585984,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
18
+ H100 PCIe,attention,64,12345,attention_seed12345_20261029_110158_000992,workergpu158_rank0_1397341_2dd8ef558a94,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793286149.1802657,1793286856.8331676,707.6522006490268,748.5393121242523,11.064622905911989,106364.30297920374,5.763027106382651,6.416846876736965,82727403520.0,,,44f1ae23f7db2f0f3ea02abae02263b29e3e3caf12cafb79f05549508ddc15ef,77db909f565c4a61a41cb1832f4b9d3a4c913eee3624744df13146ee519938d7,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA H100 PCIe,85064286208,46279458304,84074823680,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
19
+ H100 PCIe,heptv2,64,12345,heptv2_seed12345_20261029_111426_347405,workergpu158_rank0_1498617_e462b89737dc,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,7680,73925923,1793286890.1019304,1793287705.664468,815.5618177449796,849.5290899276733,9.674555130611186,93001.5710297288,6.575758987048175,7.331884939683952,83606110208.0,,,120ddb7e424abb7aefd5142fb554ef521657db2d9456ec2265b96013654444a2,c6ed9d1d2cec4997e396f86f42502e17f2bb9ee446fb716b8298d5e0594ba1df,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,NVIDIA H100 PCIe,85064286208,58236852224,82892029952,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
20
+ H100 PCIe,attention,8,12345,attention_seed12345_20261029_103730_624725,workergpu158_rank0_1235013_e1123604fe01,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793284683.2652156,1793284778.9086616,95.64288120501442,133.60123825073242,11.900117044586105,115695.92781187355,0.6475720765884034,1.3319195346685593,13643022336.0,,,0db34b576230877eea086983e40861ad5dc50bf8d1ca4f52f1171e862ecfe8d1,e031741435d457a019cef17d5bd8a719aece39e6e5e5c4402f14eeeaef556df9,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA H100 PCIe,85064286208,6213974016,12928942080,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
21
+ H100 PCIe,heptv2,8,12345,heptv2_seed12345_20261029_103949_700914,workergpu158_rank0_1247010_f21641cb321b,0,completed,completed,h100_approx_321_full_20261029_1038,1,x86_64,3.11.15,2.11.0+cu128,12.8,,True,120,20,960,9316574,1793284821.6578064,1793284940.9059522,119.24748125701444,162.636616230011,10.490352668650944,101989.84434573248,0.734243988757953,1.3537986898882084,12523143168.0,,,4f8709a969efac1aef823ff96817fc20169c2f8304f35c73b5fb409bbf51a4df,4e663095850d9f7ca01a513d78eab2410785b3b11fb2d2e029616f5acac8e9c6,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,NVIDIA H100 PCIe,85064286208,7241464320,11809062912,9e3d96bff0f3f0cc7348bb566c1a9974ea6420d0
22
+ MI250X (1 GCD),attention,128,12345,attention_seed12345_20261030_175323_702459,nid007882_rank0_1785_c17cbf4f192f,0,oom,failed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,0,0,0,0,1793375612.3400874,1793375631.988239,19.64776315400377,34.78092813491821,,,,,67848052736.0,,,80a1cecbdea69f4107c4bfb888b507a01b27c6669ed0bd9dbfe6e7be315d975c,71ba888bfee367fba3384fc34e32eff9a9cb93310c905b219d23d735e7841829,,AMD Instinct MI250X,68702699520,66094283776,68023222272,20ac400a43a68da3ce1c029e96b3eaf90552f36d
23
+ MI250X (1 GCD),heptv2,128,12345,heptv2_seed12345_20261030_175357_109122,nid007882_rank0_3202_4345cf687189,0,oom,failed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,0,0,0,0,1793375644.9217677,1793375681.3851423,36.46110969898291,50.256638050079346,,,,,68259753984.0,,,66e42957b2fed2e0f60669b51017829302584d0d382d7e9777a36a9bebbd6087,abb9385c1ef8585c13c88b6b433ac2e2600c0bb7ae36bb86921992767114ae72,,AMD Instinct MI250X,68702699520,61789143040,67459088384,20ac400a43a68da3ce1c029e96b3eaf90552f36d
24
+ MI250X (1 GCD),attention,16,12345,attention_seed12345_20261030_163541_037698,nid007882_rank0_77698_8bca43ff61b1,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,1920,18508648,1793370948.8368967,1793371226.9412224,278.10355572309345,290.8300964832306,7.347870070138632,70886.50072836764,2.1180435339920223,3.23811617057887,22263279616.0,,,ea52c515dbbe1c9563883b36b46944ea1301790aaf30fc246a77f609f45b0c4f,d929cbf2063c7ea854d163e1d1bd0cb375d2791b6b763440be12e8fbe903cdce,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,AMD Instinct MI250X,68702699520,11927367168,21353201664,20ac400a43a68da3ce1c029e96b3eaf90552f36d
25
+ MI250X (1 GCD),heptv2,16,12345,heptv2_seed12345_20261030_164032_134980,nid007882_rank0_80833_a2c3ec49ebc1,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,1920,18508648,1793371239.9030964,1793371572.4276152,332.5237337300787,346.27485871315,6.17795217003457,59600.04828892048,2.5454883782658726,3.6300434995302937,23697174528.0,,,a708686d1cdceed0d8d9ee182b51b3218007b6b4b8f000e71b5320b5a2a7ae9e,03bf3af4aa95bc4f7c023240eec8a35c0af0db4570d1dc41c339d76be953f821,fcab86177fffa2f028dc6d8641960a6b79ef70a2515ab3d35fd226c30c5c31e2,AMD Instinct MI250X,68702699520,14329852928,22758293504,20ac400a43a68da3ce1c029e96b3eaf90552f36d
26
+ MI250X (1 GCD),attention,32,12345,attention_seed12345_20261030_164618_457690,nid007882_rank0_84145_ef09bea576c4,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,3840,37085423,1793371586.4922276,1793372145.7281442,559.2351652139332,572.4081828594208,7.095937882042429,68579.29038075786,4.479588077578228,5.675255642548437,47670607872.0,,,ee1c65d604d38cfd7ba8f7ff353c172310550c502bd2e9a1de7c66e79c3739cf,70c9e592e9ad861b0155b680eb576f8541cc525c448e8437b4bf39eb2b1841ee,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,AMD Instinct MI250X,68702699520,23447246848,46760198144,20ac400a43a68da3ce1c029e96b3eaf90552f36d
27
+ MI250X (1 GCD),heptv2,32,12345,heptv2_seed12345_20261030_165550_661732,nid007882_rank0_89756_b734840a058e,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,3840,37085423,1793372158.7184627,1793372834.6671145,675.9478112340439,689.7471876144409,5.866772572121572,56699.91853220471,5.390423756791279,6.600580013847212,51459600384.0,,,7cefc73bda64b0d074b0349fd33d1c821a9016f7c9cca64db56a02c9fbacb4b9,25d8ac642d028a4e189705e36d94ef847f6cdc0ea9da2e228f74c5e1fcde084c,750ebd36d1f143d3ac57e9549db9762dee2b7fb22cc5a28e6f2c95eefae7e929,AMD Instinct MI250X,68702699520,29332572672,50520391680,20ac400a43a68da3ce1c029e96b3eaf90552f36d
28
+ MI250X (1 GCD),attention,64,12345,attention_seed12345_20261030_170721_046856,nid007882_rank0_95827_bf35c0cbc8e9,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,7680,73925923,1793372849.5186274,1793374089.7862635,1240.2667897279607,1254.2669656276703,6.298893842190149,60551.31374667772,10.104078602686059,11.602974367812276,68359630848.0,,,99468c584d577b3fd8ee0040f86788736765ddc8bcf6819a8cfc1b8d8fedba93,9c4fc33641013a47b46f70e542f6b7621eeebc637a10d4652860dbe0a7a738cb,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,AMD Instinct MI250X,68702699520,46362395136,67926753280,20ac400a43a68da3ce1c029e96b3eaf90552f36d
29
+ MI250X (1 GCD),heptv2,64,12345,heptv2_seed12345_20261030_172815_671846,nid007882_rank0_121143_7546f178b8dd,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,7680,73925923,1793374104.72147,1793375596.1859314,1491.4635927228956,1507.1206331253052,5.249767490781045,50466.05426213437,12.11482840788085,13.497417909321376,68279242752.0,,,d56f05a803a61b5157d8bd6ece44213ace216f496229fd381417c2fdc947f86d,a8b202304fa767521a170e6aa070a9ad704d62c0a7e3e0203ae0011ea08831ea,523871de2fcc2d80296f788542abd4fa259bed36e3b51abde4f0813022a26461,AMD Instinct MI250X,68702699520,58328029184,67626860544,20ac400a43a68da3ce1c029e96b3eaf90552f36d
30
+ MI250X (1 GCD),attention,8,12345,attention_seed12345_20261030_162850_951873,nid007882_rank0_73181_e2a0f0197eb7,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,960,9316574,1793370555.6356494,1793370733.4232671,177.78685278398916,207.6635971069336,7.030669051986354,68353.93097901065,1.0991682871826924,2.2399486443423666,13298425856.0,,,3e30d19c78dc7ece2b5602febf23155e0c4f0b7bade287e4d1971ddb8a1f5683,931b0a54d71ac40f987851e7a09bb45fe4b524a526c004d3f1f7ef2c0722080d,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,AMD Instinct MI250X,68702699520,6305170432,12379488256,20ac400a43a68da3ce1c029e96b3eaf90552f36d
31
+ MI250X (1 GCD),heptv2,8,12345,heptv2_seed12345_20261030_163218_908588,nid007882_rank0_75513_3099f92c9e96,0,completed,completed,lumi_retry_20261030_144129,1,x86_64,3.12.3,2.11.0+rocm7.2_lumi_aif_20261028104748,,7.2.53211,True,120,20,960,9316574,1793370746.7258735,1793370935.0312088,188.30460606899578,202.32765364646912,6.020502489276655,58532.837851998076,1.280680806026794,2.432090930375271,12788461568.0,,,77d2d22e7d9ad4b8ad41b3ff8e9c020a601622c402a4a7562850fd33eabbe181,52c73b40d6692cbabf242e6f5a53f981e30aa5304533917007db5faf9b69bd98,7fcc9de94e5c3b2b64c8d844ea5408d8a8bdb3aebcaf3c77bb112cc89b5260fa,AMD Instinct MI250X,68702699520,7332608512,11840520192,20ac400a43a68da3ce1c029e96b3eaf90552f36d
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.png ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/allocated_gib.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.png ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/events_per_second.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.png ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/khits_per_second.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.png ADDED

Git LFS Details

  • SHA256: e241eb32bc6d4f38f08dac6ec1126db719ae5217675572063fcace3ff87fe125
  • Pointer size: 131 Bytes
  • Size of remote file: 105 kB
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/p99_over_p50.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.png ADDED

Git LFS Details

  • SHA256: 9040cc4534a92f0cb960a372dedf7f787f03addcbb77fbf0ddc5e8ed0cbfa0a1
  • Pointer size: 131 Bytes
  • Size of remote file: 238 kB
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_1.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.png ADDED

Git LFS Details

  • SHA256: 8307c896b3946b81872fa4fc2f3e17eb1a6bd7c4d49f12e47da4dc210d7f0f1a
  • Pointer size: 131 Bytes
  • Size of remote file: 247 kB
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/training_loss_2.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.png ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/figures/validation_loss.svg ADDED
benchmarks/v3.2.0/cld_gpu_batch_sweep/manifest.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "files": [
3
+ {
4
+ "path": "README.md",
5
+ "bytes": 7002,
6
+ "sha256": "5fb19165312d3cd74a04d235193dd8ddc8aad250e6059944a0b5fe7c31d7d308"
7
+ },
8
+ {
9
+ "path": "cases.csv",
10
+ "bytes": 20206,
11
+ "sha256": "d9e090aab5a7fb90e38adcc5826de324295fd1811c1dfa65a30838d0da2a28e8"
12
+ },
13
+ {
14
+ "path": "figures/allocated_gib.png",
15
+ "bytes": 80709,
16
+ "sha256": "8540ea1ff3b4b4b9a08b2f039698f6e5cf9cea8cee7641b41b86b126a0ecf5a7"
17
+ },
18
+ {
19
+ "path": "figures/allocated_gib.svg",
20
+ "bytes": 47364,
21
+ "sha256": "050b6fc9d3ec84587e9329f0c97398c84a10ee898901efaa217c4f5ece9719b0"
22
+ },
23
+ {
24
+ "path": "figures/events_per_second.png",
25
+ "bytes": 82104,
26
+ "sha256": "349b27a480cb8909b8b7cc75b827223ae1a532d1f9a8fe2eeea67fe52e16bbd4"
27
+ },
28
+ {
29
+ "path": "figures/events_per_second.svg",
30
+ "bytes": 47035,
31
+ "sha256": "ce2a4e19fa5f8bd6da39cac3482e34bd8e812b47c277f5de84d6af6b3f58e19e"
32
+ },
33
+ {
34
+ "path": "figures/khits_per_second.png",
35
+ "bytes": 85807,
36
+ "sha256": "258b9cf7e95d477a3e697571fe97d7d2d9d321732803e10a409bb871003525b0"
37
+ },
38
+ {
39
+ "path": "figures/khits_per_second.svg",
40
+ "bytes": 47490,
41
+ "sha256": "53882f3d7b61d048b3646f3183034c053814b999662028b3e3684bc11668c56d"
42
+ },
43
+ {
44
+ "path": "figures/p99_over_p50.png",
45
+ "bytes": 105379,
46
+ "sha256": "e241eb32bc6d4f38f08dac6ec1126db719ae5217675572063fcace3ff87fe125"
47
+ },
48
+ {
49
+ "path": "figures/p99_over_p50.svg",
50
+ "bytes": 46133,
51
+ "sha256": "625700ea3deff968da9a2610dd47c69afe824b6d649a9da9badf60a8af34faf7"
52
+ },
53
+ {
54
+ "path": "figures/training_loss_1.png",
55
+ "bytes": 238301,
56
+ "sha256": "9040cc4534a92f0cb960a372dedf7f787f03addcbb77fbf0ddc5e8ed0cbfa0a1"
57
+ },
58
+ {
59
+ "path": "figures/training_loss_1.svg",
60
+ "bytes": 115074,
61
+ "sha256": "e3ee8db077208248d55a6e7140ac690473c44f5c16a96b9bf3973c0057741ad4"
62
+ },
63
+ {
64
+ "path": "figures/training_loss_2.png",
65
+ "bytes": 246512,
66
+ "sha256": "8307c896b3946b81872fa4fc2f3e17eb1a6bd7c4d49f12e47da4dc210d7f0f1a"
67
+ },
68
+ {
69
+ "path": "figures/training_loss_2.svg",
70
+ "bytes": 112017,
71
+ "sha256": "7ebdc363d6bf6f96ef4a4226902e8a6d24d9dc9a1288d743584e8c0882138afc"
72
+ },
73
+ {
74
+ "path": "figures/validation_loss.png",
75
+ "bytes": 98278,
76
+ "sha256": "fd8e87b1d9a94d26f201863258b1a82d959241fa17547a8112ba6bfc93408682"
77
+ },
78
+ {
79
+ "path": "figures/validation_loss.svg",
80
+ "bytes": 48587,
81
+ "sha256": "8ba148bf29473e842450ecbbbaff46470f2cbad8118cfd27760550dc34c78a6e"
82
+ },
83
+ {
84
+ "path": "render_plots.py",
85
+ "bytes": 4891,
86
+ "sha256": "23f1a531cb752e3fb8cb71f6c879d8a277ef349a5fb3aab660ce3ec5811a7fb7"
87
+ },
88
+ {
89
+ "path": "results.json",
90
+ "bytes": 237951,
91
+ "sha256": "78df2ade8ca3ef41cb0704f32fd322c21c4a6fe0ff761160fc80102b4c8595e4"
92
+ },
93
+ {
94
+ "path": "slides/gpu_batch_sweep.slides.html",
95
+ "bytes": 1422163,
96
+ "sha256": "2170b1bde53265981104ade7d6799fa4b9567b2b54a44ebf85530714d654eaeb"
97
+ },
98
+ {
99
+ "path": "slides/gpu_batch_sweep.slides.pdf",
100
+ "bytes": 1077663,
101
+ "sha256": "6aa345063e56d21b11fec33aec4984e5dd310d67d473d8dea5557ca2bf69302a"
102
+ },
103
+ {
104
+ "path": "steps.csv",
105
+ "bytes": 539440,
106
+ "sha256": "bfcb71811c4cf240bcac42656963e37d46891e9837c3fe1df3e654a5f5c6e664"
107
+ },
108
+ {
109
+ "path": "validation.csv",
110
+ "bytes": 3721,
111
+ "sha256": "14c0a2f8ba86bf5b254da2d558d01d2a27cda35585c834ce91ba5c23eef06769"
112
+ }
113
+ ]
114
+ }
benchmarks/v3.2.0/cld_gpu_batch_sweep/render_plots.py ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Render portable sweep CSVs. Requires matplotlib, numpy and pandas; no training stack."""
2
+
3
+ import argparse
4
+ from pathlib import Path
5
+
6
+
7
+ def render(directory):
8
+ import matplotlib
9
+
10
+ matplotlib.use("Agg")
11
+ import matplotlib.pyplot as plt
12
+ import pandas as pd
13
+
14
+ directory = Path(directory)
15
+ cases = pd.read_csv(directory / "cases.csv")
16
+ if (cases.world_size != 1).any() or cases.duplicated(["platform", "variant", "batch"]).any():
17
+ raise ValueError("Plot one single-device session per platform/model/batch; select repeats explicitly")
18
+ completed = cases[cases.status == "completed"].copy()
19
+ if completed.empty:
20
+ raise ValueError("No completed cases to plot")
21
+ completed["allocated_gib"] = completed.allocated_bytes / 2**30
22
+ completed["khits_per_second"] = completed.hits_per_second / 1000
23
+ completed["p99_over_p50"] = completed.step_p99_seconds / completed.step_p50_seconds
24
+ platforms = list(cases.platform.drop_duplicates())
25
+ variants = list(cases.variant.drop_duplicates())
26
+ colors = dict(zip(platforms, plt.get_cmap("tab10").colors * (1 + len(platforms) // 10)))
27
+ figdir = directory / "figures"
28
+ figdir.mkdir(exist_ok=True)
29
+ plt.rcParams.update({"font.size": 12, "axes.spines.top": False, "axes.spines.right": False})
30
+
31
+ def save(fig, name):
32
+ for ext in ("png", "svg"):
33
+ fig.savefig(figdir / f"{name}.{ext}", dpi=160, bbox_inches="tight")
34
+ plt.close(fig)
35
+
36
+ def compare(data, metric, ylabel, name):
37
+ fig, axes = plt.subplots(1, len(variants), figsize=(6 * len(variants), 4), squeeze=False, layout="constrained", sharey=True)
38
+ for ax, variant in zip(axes.flat, variants):
39
+ for platform in platforms:
40
+ rows = data[(data.platform == platform) & (data.variant == variant)].sort_values("batch")
41
+ ax.plot(rows.batch, rows[metric], "-o", label=platform, color=colors[platform])
42
+ ax.set(title=variant, xlabel="Training batch size", ylabel=ylabel, xscale="log")
43
+ batches = sorted(data.batch.unique())
44
+ ax.set_xticks(batches, labels=batches)
45
+ ax.xaxis.set_minor_locator(matplotlib.ticker.NullLocator())
46
+ ax.grid(alpha=0.2)
47
+ axes.flat[-1].legend(fontsize=10)
48
+ save(fig, name)
49
+
50
+ for metric, ylabel in [
51
+ ("events_per_second", "Steady training events/s"),
52
+ ("khits_per_second", "Steady training kHits/s"),
53
+ ("allocated_gib", "Process peak allocated GiB"),
54
+ ("p99_over_p50", "Host step p99 / p50"),
55
+ ]:
56
+ compare(completed, metric, ylabel, metric)
57
+ steps = pd.read_csv(directory / "steps.csv")
58
+ keys = completed[["platform", "variant", "batch", "seed", "run_id", "session_id", "rank", "warmup_steps"]]
59
+ steps = steps.merge(keys, on=list(keys.columns[:-1]), validate="many_to_one")
60
+ for index, variant in enumerate(variants):
61
+ selected = steps[steps.variant == variant]
62
+ batches = sorted(selected.batch.unique())
63
+ ncols = min(2, len(batches))
64
+ nrows = (len(batches) + ncols - 1) // ncols
65
+ fig, axes = plt.subplots(nrows, ncols, figsize=(6 * ncols, 3.5 * nrows), squeeze=False, layout="constrained", sharey=True)
66
+ for ax, batch in zip(axes.flat, batches):
67
+ for i, platform in enumerate(platforms):
68
+ rows = selected[(selected.platform == platform) & (selected.batch == batch)].sort_values("step")
69
+ ax.plot(rows.step, rows.total_unweighted, ["-", "--", ":", "-."][i % 4], label=platform, color=colors[platform])
70
+ ax.set(title=f"{variant}, batch {batch}", xlabel="Training step", ylabel="Total unweighted loss")
71
+ ax.grid(alpha=0.2)
72
+ for ax in list(axes.flat)[len(batches) :]:
73
+ ax.set_visible(False)
74
+ axes.flat[0].legend(fontsize=10)
75
+ save(fig, f"training_loss_{index + 1}")
76
+ validation = pd.read_csv(directory / "validation.csv")
77
+ if not validation.empty:
78
+ endpoints = validation.merge(
79
+ completed[["platform", "variant", "batch", "seed", "run_id", "session_id", "rank", "steps"]],
80
+ on=["platform", "variant", "batch", "seed", "run_id", "session_id", "rank"],
81
+ validate="many_to_one",
82
+ )
83
+ endpoints = endpoints[endpoints.step == endpoints.steps]
84
+ if endpoints.step.nunique() != 1:
85
+ raise ValueError("Validation endpoints have different training lengths; compare explicitly")
86
+ compare(endpoints, "valid_total_unweighted", f"Validation loss at step {endpoints.step.iloc[0]}", "validation_loss")
87
+ print(f"Wrote figures to {figdir}")
88
+
89
+
90
+ def main(argv=None):
91
+ parser = argparse.ArgumentParser(description=__doc__)
92
+ parser.add_argument("directory", nargs="?", type=Path, default=Path(__file__).resolve().parent)
93
+ args = parser.parse_args(argv)
94
+ render(args.directory)
95
+
96
+
97
+ if __name__ == "__main__":
98
+ main()
benchmarks/v3.2.0/cld_gpu_batch_sweep/results.json ADDED
The diff for this file is too large to render. See raw diff
 
benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.html ADDED
The diff for this file is too large to render. See raw diff
 
benchmarks/v3.2.0/cld_gpu_batch_sweep/slides/gpu_batch_sweep.slides.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6aa345063e56d21b11fec33aec4984e5dd310d67d473d8dea5557ca2bf69302a
3
+ size 1077663
benchmarks/v3.2.0/cld_gpu_batch_sweep/steps.csv ADDED
The diff for this file is too large to render. See raw diff
 
benchmarks/v3.2.0/cld_gpu_batch_sweep/validation.csv ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ platform,variant,batch,seed,run_id,session_id,rank,step,train_total_unweighted,valid_total_unweighted
2
+ GH200,attention,16,12345,attention_seed12345_20261029_062032_191329,alphagh01_rank0_2963140_f80016108aa1,0,120,3.4413122558034956,3.3920575259253383
3
+ GH200,heptv2,16,12345,heptv2_seed12345_20261029_062318_186034,alphagh01_rank0_2966282_3c1d7090284d,0,120,3.2092463737353683,3.2746193357743323
4
+ GH200,attention,32,12345,attention_seed12345_20261029_062618_170573,alphagh01_rank0_2969691_16a386b5ec65,0,120,3.3299973998218775,3.5785752404481173
5
+ GH200,heptv2,32,12345,heptv2_seed12345_20261029_063114_050085,alphagh01_rank0_2974716_6b9de441032e,0,120,3.210159281268716,3.3225052603520453
6
+ GH200,attention,64,12345,attention_seed12345_20261029_063630_231892,alphagh01_rank0_2980492_cefdd4e73f6b,0,120,3.325105535797775,3.5525365620851517
7
+ GH200,heptv2,64,12345,heptv2_seed12345_20261029_064622_331901,alphagh01_rank0_2991102_2bb5ab0d5fc0,0,120,2.8526693456806242,3.100961511954665
8
+ GH200,attention,8,12345,attention_seed12345_20261029_061642_376997,alphagh01_rank0_2958727_aca0c7649228,0,120,3.4185724649578333,3.4165319274179637
9
+ GH200,heptv2,8,12345,heptv2_seed12345_20261029_061831_709291,alphagh01_rank0_2960929_85567b5b0c80,0,120,3.235496461391449,3.3162532565183938
10
+ H100 PCIe,attention,16,12345,attention_seed12345_20261029_104229_564372,workergpu158_rank0_1259903_683085c8df3b,0,120,3.4205480571836233,3.3728219415061176
11
+ H100 PCIe,heptv2,16,12345,heptv2_seed12345_20261029_104538_814383,workergpu158_rank0_1276037_80bfd5ff3556,0,120,3.1876705032773316,3.258329316973686
12
+ H100 PCIe,attention,32,12345,attention_seed12345_20261029_104920_111739,workergpu158_rank0_1302660_15e949a8d27e,0,120,3.316897104959935,3.4092630185186863
13
+ H100 PCIe,heptv2,32,12345,heptv2_seed12345_20261029_105508_955409,workergpu158_rank0_1345892_3a3528b8bf0e,0,120,3.1633353787474334,3.3145826999098063
14
+ H100 PCIe,attention,64,12345,attention_seed12345_20261029_110158_000992,workergpu158_rank0_1397341_2dd8ef558a94,0,120,3.3548554317094386,3.5806318521499634
15
+ H100 PCIe,heptv2,64,12345,heptv2_seed12345_20261029_111426_347405,workergpu158_rank0_1498617_e462b89737dc,0,120,2.865398366935551,3.115826594643295
16
+ H100 PCIe,attention,8,12345,attention_seed12345_20261029_103730_624725,workergpu158_rank0_1235013_e1123604fe01,0,120,3.4219354861415923,3.3811078616417944
17
+ H100 PCIe,heptv2,8,12345,heptv2_seed12345_20261029_103949_700914,workergpu158_rank0_1247010_f21641cb321b,0,120,3.2621626425534487,3.2805806999094784
18
+ MI250X (1 GCD),attention,16,12345,attention_seed12345_20261030_163541_037698,nid007882_rank0_77698_8bca43ff61b1,0,120,3.39885445125401,3.375275770202279
19
+ MI250X (1 GCD),heptv2,16,12345,heptv2_seed12345_20261030_164032_134980,nid007882_rank0_80833_a2c3ec49ebc1,0,120,3.2108154979068786,3.2590630035847425
20
+ MI250X (1 GCD),attention,32,12345,attention_seed12345_20261030_164618_457690,nid007882_rank0_84145_ef09bea576c4,0,120,3.3251947122626007,3.5192190841771662
21
+ MI250X (1 GCD),heptv2,32,12345,heptv2_seed12345_20261030_165550_661732,nid007882_rank0_89756_b734840a058e,0,120,3.172804694622755,3.3399899369105697
22
+ MI250X (1 GCD),attention,64,12345,attention_seed12345_20261030_170721_046856,nid007882_rank0_95827_bf35c0cbc8e9,0,120,3.3100380515679717,3.5173501721583307
23
+ MI250X (1 GCD),heptv2,64,12345,heptv2_seed12345_20261030_172815_671846,nid007882_rank0_121143_7546f178b8dd,0,120,2.812488204333931,3.0608929730951786
24
+ MI250X (1 GCD),attention,8,12345,attention_seed12345_20261030_162850_951873,nid007882_rank0_73181_e2a0f0197eb7,0,120,3.380337958689779,3.359904798679054
25
+ MI250X (1 GCD),heptv2,8,12345,heptv2_seed12345_20261030_163218_908588,nid007882_rank0_75513_3099f92c9e96,0,120,3.289946631528437,3.290073842741549