DTYPE_THROUGHPUT.md

# ReLU throughput vs dtype on RTX 4090 (sm_89)

ReLU is **memory-bound** (read N + write N, trivial compute). So at the GPU's
byte-bandwidth wall, **element throughput scales as 1/sizeof(dtype)** — precision
choice trades only bytes moved, not compute. Measured two ways (custom CUDA
kernel + `torch.relu`), all correctness-verified.

Harness: `bench/relu_dtype.cu` (custom, covers fp8/int8 that PyTorch has no
ReLU for) and a `torch.relu` cross-check. 128-bit (int4) vectorized loads so every
dtype gets an identical memory access pattern.

## Result 1 — intrinsic precision effect (DRAM-bound, N=64M, all > L2)

Byte-bandwidth is **dtype-invariant** (~943 GB/s = the wall); element rate = 1/sizeof.

| dtype | bytes | GB/s | Gelem/s | vs fp32 |
|-------|:---:|:---:|:---:|:---:|
| fp64 / int64 | 8 | ~943 | 59 | **0.5×** |
| fp32 / int32 | 4 | ~944 | 118 | 1.0× |
| **fp16 / bf16** / int16 | 2 | ~944 | 236 | **2.0×** |
| **fp8 e4m3 / e5m2** / int8 / uint8 | 1 | ~941 | 471 | **4.0×** |

Cross-checked in `torch.relu` @ 64M: fp64 0.50×, bf16 2.01×, fp16 2.01×, int8 4.02×
— identical to the custom kernel. fp16 and bf16 are exactly equal (both 2 bytes);
the format (5e10m vs 8e7m) is irrelevant to a memory-bound op. fp8 only measurable
via the custom kernel (`torch.relu` raises NotImplementedError for float8).

## Result 2 — L2-residency bonus (fixed element count near the 72 MB L2)

At a *fixed element count* where the low-precision working set fits in the 72 MB L2
but fp32 spills to DRAM, low precision gets an EXTRA super-linear win. `torch.relu`
@ N=16.7M (4096²):

| dtype | working set | GB/s | Gelem/s | vs fp32 | regime |
|-------|:---:|:---:|:---:|:---:|:---|
| fp32 | 134 MB | 935 | 117 | 1.0× | DRAM (spills) |
| bf16 | 67 MB | 3753 | 938 | **8.0×** | **L2-resident** |
| fp16 | 67 MB | 3598 | 900 | 7.7× | L2-resident |
| int8 | 34 MB | 3631 | 1816 | **15.5×** | L2-resident |

This is two effects stacked: 2× (bf16 bytes) × ~4× (L2 vs DRAM BW). Real activation
tensors that hover around L2 size see exactly this — a strong reason to keep
activations in bf16/fp16.

## L2-resident pure (custom kernel, N=2M, all fit in L2)

Absolute L2 bandwidth is higher and rises with larger dtypes (more in-flight bytes
per int4 chunk): fp64 ~3.4 TB/s → fp8 ~1.6 TB/s. Element rate still favors small
dtypes (fp8 ~808 vs fp32 ~395 Gelem/s) but sub-linearly, because tiny byte working
sets underutilize the memory system.

## Takeaways

1. **bf16/fp16 ReLU = 2× fp32 throughput** intrinsically (memory-bound, fewer
   bytes), and up to ~8× at L2-boundary sizes. fp16 == bf16 for this op.
2. **fp8/int8 = 4× fp32** intrinsically, up to ~15× at L2-boundary sizes.
3. **fp64/int64 = 0.5× fp32** — avoid 64-bit for elementwise.
4. The byte-bandwidth wall (~943 GB/s) is the same for every dtype; you cannot beat
   it within one dtype — only move fewer bytes (lower precision) or skip the
   round-trip (fusion).
5. The shipped `kernels-community/relu` kernel is **fp32-only** (`TORCH_CHECK`).
   Adding bf16/fp16 (and int8/fp8) would unlock the 2–4× shown here for the dtypes
   real models actually use; the vectorized + streaming kernel design generalizes
   directly (template on T, 16-byte loads, per-element relu).

## Reproduce
```
cd bench && nvcc -arch=sm_89 -O3 -o relu_dtype relu_dtype.cu && ./relu_dtype
```