L2_BAND_CLIFF.md

The L2-band cliff: independently reported, reproduced, root-caused, fixed

Status: fixed in v3 (relu_cuda/relu.cu, pre-Hopper band branch). Credit: found by an independent recreation of the model-card results on RTX 4090 against the v2 release (report RELU_KERNEL_RESULTS.md in the development tree's tests/ directory, not shipped in this repo).

The bug

v2's launch_relu() used evict-first streaming loads (__ldcs) for every working set ≥ 32 MiB. On pre-Hopper GPUs, for working sets in (32 MiB, L2] (4090: 72 MiB), that discards exactly the input-reuse residency torch.relu exploits: torch serves repeated calls from L2 at up to ~3.5 TB/s while the streaming kernel re-fetches from DRAM every call.

Measured impact (4090, correctness-gated, multiple methodologies — independent report's single-workload sweep + our robust_bench pairwise): 0.34–0.65x vs torch across the band, for every dtype at its own byte sizes. The band is byte-defined, so the same two nominal test sizes (1024², 4096²) dodge it for fp32 (8 / 128 MiB) and land fp16/bf16's "large" right inside it (64 MiB) — which is why the published v2 card showed good fp32 numbers and the fp16/bf16 claims failed to reproduce.

The fix (v3)

Working sets in (32 MiB, l2CacheSize] on major < 9 now use default loads + streaming stores: the re-read input stays L2-resident; the write-once output still streams (evicting it early keeps more L2 for the input). Above L2, streaming loads return — they genuinely win there (1.13–1.30x vs torch at 76–84 MiB). Hopper is untouched: both of its band points measured cliff-free with streaming (1.43–1.55x over torch; its evict-first retention behaves differently).

Config selection data (4090, candidate-vs-torch anchored pairs, nsys):

config @ band size	33.6 MB	52.4 MB	75.5 MB
default ld + __stcs st (shipped)	1.00x	1.05x	1.05x
fully default	1.01x	0.99x	1.07x
streaming (v2, control)	0.40x	0.54x	1.13x

Post-fix validation: pairwise vs torch 0.99–1.06x across the band (was 0.40–0.54x); official harness fp16/bf16 large 1.02–1.03x (was 0.35x), int8 large 1.14x (was 0.57x); no change outside the band on the 4090 (1024² 1.02x, ≥128 MiB 1.05x) and none on H100.

Three measurement lessons this also exposed

1. The v2 card's 4090 fp16/bf16 "1.85x/1.81x vs torch" were artifacts of a fixed-order shared-buffer interleave: ours always ran on input the previous case had just re-cached, torch always ran on input our evict-first pass had just evicted. Separate-buffer symmetric pairing measures parity-to-1.3x depending on regime. Corrected everywhere.
2. The allocating call pattern doubles the output footprint. out = kernel.relu(x) keeps the previous output alive at allocation time, so the caching allocator alternates two output blocks; at L2-boundary sizes that alone pushes a timed loop off the cliff. torch.relu itself measures ~0.33x against its own single-shot under that loop. The dtype benchmark class now uses a preallocated out= and times the reference in the same form.
3. Sweep coverage must be gap-free in bytes, not element counts. The original 4090 sweep jumped from 33.6 MB to 134 MB of working set — the entire band containing the L2 boundary was never sampled, and the 32 MB threshold got extrapolated across it.

Repro: bench/spec_pair_h100.py (symmetric pairwise A/B) plus the band-size sweep pattern described above; the independent report's own sweep script and raw JSON live in the development tree's tests/ directory (not shipped in this repo).