L2_BAND_CLIFF.md

# The L2-band cliff: independently reported, reproduced, root-caused, fixed

**Status: fixed in v3** (`relu_cuda/relu.cu`, pre-Hopper band branch).
Credit: found by an independent recreation of the model-card results on
RTX 4090 against the v2 release (report `RELU_KERNEL_RESULTS.md` in the
development tree's `tests/` directory, not shipped in this repo).

## The bug

v2's `launch_relu()` used evict-first streaming loads (`__ldcs`) for every
working set ≥ 32 MiB. On pre-Hopper GPUs, for working sets in
**(32 MiB, L2]** (4090: 72 MiB), that discards exactly the input-reuse
residency `torch.relu` exploits: torch serves repeated calls from L2 at up
to ~3.5 TB/s while the streaming kernel re-fetches from DRAM every call.

Measured impact (4090, correctness-gated, multiple methodologies —
independent report's single-workload sweep + our robust_bench pairwise):
**0.34–0.65x vs torch across the band, for every dtype at its own byte
sizes.** The band is byte-defined, so the same two nominal test sizes
(1024², 4096²) dodge it for fp32 (8 / 128 MiB) and land fp16/bf16's
"large" right inside it (64 MiB) — which is why the published v2 card
showed good fp32 numbers and the fp16/bf16 claims failed to reproduce.

## The fix (v3)

Working sets in (32 MiB, `l2CacheSize`] on `major < 9` now use **default
loads + streaming stores**: the re-read input stays L2-resident; the
write-once output still streams (evicting it early keeps more L2 for the
input). Above L2, streaming loads return — they genuinely win there
(1.13–1.30x vs torch at 76–84 MiB). Hopper is untouched: both of its band
points measured cliff-free with streaming (1.43–1.55x over torch; its
evict-first retention behaves differently).

Config selection data (4090, candidate-vs-torch anchored pairs, nsys):

| config @ band size | 33.6 MB | 52.4 MB | 75.5 MB |
|---|:---:|:---:|:---:|
| default ld + `__stcs` st (shipped) | 1.00x | **1.05x** | 1.05x |
| fully default | 1.01x | 0.99x | 1.07x |
| streaming (v2, control) | 0.40x | 0.54x | **1.13x** |

Post-fix validation: pairwise vs torch 0.99–1.06x across the band (was
0.40–0.54x); official harness fp16/bf16 `large` 1.02–1.03x (was 0.35x),
int8 `large` 1.14x (was 0.57x); no change outside the band on the 4090
(1024² 1.02x, ≥128 MiB 1.05x) and none on H100.

## Three measurement lessons this also exposed

1. **The v2 card's 4090 fp16/bf16 "1.85x/1.81x vs torch" were artifacts**
   of a fixed-order shared-buffer interleave: ours always ran on input the
   previous case had just re-cached, torch always ran on input our
   evict-first pass had just evicted. Separate-buffer symmetric pairing
   measures parity-to-1.3x depending on regime. Corrected everywhere.
2. **The allocating call pattern doubles the output footprint.**
   `out = kernel.relu(x)` keeps the previous output alive at allocation
   time, so the caching allocator alternates two output blocks; at
   L2-boundary sizes that alone pushes a timed loop off the cliff.
   `torch.relu` itself measures ~0.33x *against its own single-shot* under
   that loop. The dtype benchmark class now uses a preallocated `out=`
   and times the reference in the same form.
3. **Sweep coverage must be gap-free in bytes, not element counts.** The
   original 4090 sweep jumped from 33.6 MB to 134 MB of working set —
   the entire band containing the L2 boundary was never sampled, and the
   32 MB threshold got extrapolated across it.

Repro: `bench/spec_pair_h100.py` (symmetric pairwise A/B) plus the
band-size sweep pattern described above; the independent report's own
sweep script and raw JSON live in the development tree's `tests/`
directory (not shipped in this repo).