DTYPE_THROUGHPUT.md

ReLU throughput vs dtype on RTX 4090 (sm_89)

ReLU is memory-bound (read N + write N, trivial compute). So at the GPU's byte-bandwidth wall, element throughput scales as 1/sizeof(dtype) — precision choice trades only bytes moved, not compute. Measured two ways (custom CUDA kernel + torch.relu), all correctness-verified.

Harness: bench/relu_dtype.cu (custom, covers fp8/int8 that PyTorch has no ReLU for) and a torch.relu cross-check. 128-bit (int4) vectorized loads so every dtype gets an identical memory access pattern.

Result 1 — intrinsic precision effect (DRAM-bound, N=64M, all > L2)

Byte-bandwidth is dtype-invariant (~943 GB/s = the wall); element rate = 1/sizeof.

dtype	bytes	GB/s	Gelem/s	vs fp32
fp64 / int64	8	~943	59	0.5×
fp32 / int32	4	~944	118	1.0×
fp16 / bf16 / int16	2	~944	236	2.0×
fp8 e4m3 / e5m2 / int8 / uint8	1	~941	471	4.0×

Cross-checked in torch.relu @ 64M: fp64 0.50×, bf16 2.01×, fp16 2.01×, int8 4.02× — identical to the custom kernel. fp16 and bf16 are exactly equal (both 2 bytes); the format (5e10m vs 8e7m) is irrelevant to a memory-bound op. fp8 only measurable via the custom kernel (torch.relu raises NotImplementedError for float8).

Result 2 — L2-residency bonus (fixed element count near the 72 MB L2)

At a fixed element count where the low-precision working set fits in the 72 MB L2 but fp32 spills to DRAM, low precision gets an EXTRA super-linear win. torch.relu @ N=16.7M (4096²):

dtype	working set	GB/s	Gelem/s	vs fp32	regime
fp32	134 MB	935	117	1.0×	DRAM (spills)
bf16	67 MB	3753	938	8.0×	L2-resident
fp16	67 MB	3598	900	7.7×	L2-resident
int8	34 MB	3631	1816	15.5×	L2-resident

This is two effects stacked: 2× (bf16 bytes) × ~4× (L2 vs DRAM BW). Real activation tensors that hover around L2 size see exactly this — a strong reason to keep activations in bf16/fp16.

L2-resident pure (custom kernel, N=2M, all fit in L2)

Absolute L2 bandwidth is higher and rises with larger dtypes (more in-flight bytes per int4 chunk): fp64 ~3.4 TB/s → fp8 ~1.6 TB/s. Element rate still favors small dtypes (fp8 ~808 vs fp32 ~395 Gelem/s) but sub-linearly, because tiny byte working sets underutilize the memory system.

Takeaways

1. bf16/fp16 ReLU = 2× fp32 throughput intrinsically (memory-bound, fewer bytes), and up to ~8× at L2-boundary sizes. fp16 == bf16 for this op.
2. fp8/int8 = 4× fp32 intrinsically, up to ~15× at L2-boundary sizes.
3. fp64/int64 = 0.5× fp32 — avoid 64-bit for elementwise.
4. The byte-bandwidth wall (~943 GB/s) is the same for every dtype; you cannot beat it within one dtype — only move fewer bytes (lower precision) or skip the round-trip (fusion).
5. The shipped kernels-community/relu kernel is fp32-only (TORCH_CHECK). Adding bf16/fp16 (and int8/fp8) would unlock the 2–4× shown here for the dtypes real models actually use; the vectorized + streaming kernel design generalizes directly (template on T, 16-byte loads, per-element relu).

Reproduce

cd bench && nvcc -arch=sm_89 -O3 -o relu_dtype relu_dtype.cu && ./relu_dtype