OPTIMIZATION.md

ReLU Optimization for RTX 4090 (sm_89)

Erratum (2026-07-31, see L2_BAND_CLIFF.md): this doc records the original v1 tuning. Two of its conclusions were later corrected: the vs-torch/vs-Hub margins measured with shared-buffer interleaves were position-biased (v3 numbers: ~parity vs torch for fp32 on the 4090), and the "streaming everywhere ≥ 32 MB" config had a real 0.34–0.65x cliff vs torch for working sets in (32 MiB, L2] — the sweep below never sampled that band (its sizes jump 33.6 MB → 134 MB). Fixed in v3 with default loads + streaming stores in-band. The sweep's DRAM-wall findings and config rankings within the same measurement layout remain valid.

Float32 ReLU (out[i] = max(in[i], 0)) tuned for the NVIDIA RTX 4090 (Ada, sm_89), built via kernel-builder (torch212-cu126) and benchmarked with the official kernels ReluBenchmark plus controlled timing. Method: empirical design-space sweep + independent design fan-out + roofline analysis.

TL;DR

Deployed kernel (relu_cuda/relu.cu): float4 vectorized + streaming load/store (__ldcs/__stcs) + adaptive block size (128 L2-resident / 512 DRAM), unroll 1, flat grid-stride, scalar tail.

Workload	Hub (orig)	Optimized	torch.relu	Optimized result
base 1024² (L2)	1836 GB/s	2335 GB/s	2285	1.27× vs Hub, beats torch
large 4096² (DRAM)	940 GB/s	945 GB/s	936	at the roofline (1.01×)

Bit-identical to the upstream kernel on all inputs (incl. NaN→0). Correctness verified incl. tail sizes 4097/333/1.

Method (ultracode)

1. Empirical sweep (bench/sweep.cu, nvcc -arch=sm_89 -O3): 31 configs × 8 sizes (64K → 134M elems), each correctness-checked, CUDA-event timed. Configs = {vec2,vec4,scalar} × unroll{1,2,4} × store{default,__stcs} × load{default,__ldcs} × block{128,256,512}.
2. Design fan-out (5 independent senior-engineer agents + roofline analyst). The streaming agent verified in SASS that __ldcs/__stcs lower to LDG.E.EF.128 / STG.E.EF.128 (Evict-First, not a hard L2 bypass on sm_89) — explaining why the large-case gain is small.
3. Roofline: theoretical 1008 GB/s; practical achievable copy ~960; ReLU target ~955. So 942→ceiling headroom is only ~+1.5–2% (in the noise).

What the sweep proved

• DRAM-bound large (working set > ~50 MB): hard wall ~944 GB/s (93.7% of peak, ~98% of the ~960 practical achievable). Every config — scalar to float4, streaming or not, any block/unroll — lands 940–945. Nothing beats the wall.
• unroll ≥ 4 is consistently SLOWER on large (929–936) — register pressure. This is why the earlier "v2" (ILP-4 + occupancy-capped grid) regressed.
• Under-subscribing the grid (v2's other bug) costs ~2% — keep a full grid.
• L2-resident (≤ ~33 MB): streaming load+store wins. On base 1024², float4 + __ldcs + __stcs + block128 = 2357 GB/s in the microbench (2335 in the kernels harness) vs 2218 for plain float4 — +6%, and ahead of torch. Streaming avoids L2 write-allocate pollution; smaller blocks balance SMs.
• Block size only matters in the L2 regime: 128 best for ≤~2–4 M elems, 512 for the 16–33 MB band; irrelevant on DRAM. Hence the adaptive threshold.

Why not "more"?

The large case is memory-bound at the GDDR6X wall — a read+write elementwise op cannot exceed copy bandwidth. ILP, wider vectors, bigger grids, streaming: all measured flat-or-worse on large. The only real headroom is L2-resident sizes, which the streaming + block-128 path captures. Knowing when to stop is the optimization.

Semantics

out = x > 0 ? x : 0 (matches upstream kernels-community/relu exactly, including NaN→0). PyTorch's F.relu propagates NaN→NaN; the upstream kernel and this one return 0 for NaN. This is a pre-existing upstream semantic, intentionally preserved — the optimization is performance-only, not a numerics change.

Reproduce

# microbench sweep
cd bench && nvcc -arch=sm_89 -O3 -o sweep sweep.cu && ./sweep
# build + official benchmark
cd relu-build && kernel-builder build --variant torch212-cxx11-cu126-x86_64-linux
kernels benchmark kernels-community/relu --version 1     # official Hub baseline