File size: 3,688 Bytes
f30427a 4bb4e7c f30427a 4bb4e7c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | # The L2-band cliff: independently reported, reproduced, root-caused, fixed
**Status: fixed in v3** (`relu_cuda/relu.cu`, pre-Hopper band branch).
Credit: found by an independent recreation of the model-card results on
RTX 4090 against the v2 release (report `RELU_KERNEL_RESULTS.md` in the
development tree's `tests/` directory, not shipped in this repo).
## The bug
v2's `launch_relu()` used evict-first streaming loads (`__ldcs`) for every
working set ≥ 32 MiB. On pre-Hopper GPUs, for working sets in
**(32 MiB, L2]** (4090: 72 MiB), that discards exactly the input-reuse
residency `torch.relu` exploits: torch serves repeated calls from L2 at up
to ~3.5 TB/s while the streaming kernel re-fetches from DRAM every call.
Measured impact (4090, correctness-gated, multiple methodologies —
independent report's single-workload sweep + our robust_bench pairwise):
**0.34–0.65x vs torch across the band, for every dtype at its own byte
sizes.** The band is byte-defined, so the same two nominal test sizes
(1024², 4096²) dodge it for fp32 (8 / 128 MiB) and land fp16/bf16's
"large" right inside it (64 MiB) — which is why the published v2 card
showed good fp32 numbers and the fp16/bf16 claims failed to reproduce.
## The fix (v3)
Working sets in (32 MiB, `l2CacheSize`] on `major < 9` now use **default
loads + streaming stores**: the re-read input stays L2-resident; the
write-once output still streams (evicting it early keeps more L2 for the
input). Above L2, streaming loads return — they genuinely win there
(1.13–1.30x vs torch at 76–84 MiB). Hopper is untouched: both of its band
points measured cliff-free with streaming (1.43–1.55x over torch; its
evict-first retention behaves differently).
Config selection data (4090, candidate-vs-torch anchored pairs, nsys):
| config @ band size | 33.6 MB | 52.4 MB | 75.5 MB |
|---|:---:|:---:|:---:|
| default ld + `__stcs` st (shipped) | 1.00x | **1.05x** | 1.05x |
| fully default | 1.01x | 0.99x | 1.07x |
| streaming (v2, control) | 0.40x | 0.54x | **1.13x** |
Post-fix validation: pairwise vs torch 0.99–1.06x across the band (was
0.40–0.54x); official harness fp16/bf16 `large` 1.02–1.03x (was 0.35x),
int8 `large` 1.14x (was 0.57x); no change outside the band on the 4090
(1024² 1.02x, ≥128 MiB 1.05x) and none on H100.
## Three measurement lessons this also exposed
1. **The v2 card's 4090 fp16/bf16 "1.85x/1.81x vs torch" were artifacts**
of a fixed-order shared-buffer interleave: ours always ran on input the
previous case had just re-cached, torch always ran on input our
evict-first pass had just evicted. Separate-buffer symmetric pairing
measures parity-to-1.3x depending on regime. Corrected everywhere.
2. **The allocating call pattern doubles the output footprint.**
`out = kernel.relu(x)` keeps the previous output alive at allocation
time, so the caching allocator alternates two output blocks; at
L2-boundary sizes that alone pushes a timed loop off the cliff.
`torch.relu` itself measures ~0.33x *against its own single-shot* under
that loop. The dtype benchmark class now uses a preallocated `out=`
and times the reference in the same form.
3. **Sweep coverage must be gap-free in bytes, not element counts.** The
original 4090 sweep jumped from 33.6 MB to 134 MB of working set —
the entire band containing the L2 boundary was never sampled, and the
32 MB threshold got extrapolated across it.
Repro: `bench/spec_pair_h100.py` (symmetric pairwise A/B) plus the
band-size sweep pattern described above; the independent report's own
sweep script and raw JSON live in the development tree's `tests/`
directory (not shipped in this repo).
|