Validation

Date: June 20, 2026

Local environment:

• GPU: NVIDIA GeForce RTX 5090
• PyTorch: 2.9.1+cu128
• CUDA runtime reported by PyTorch: 12.8
• Source build target: sm_120a

Source Correctness

Command:

python fp8-gemm/tests/test_fp8_gemm.py --backend source --mode full

Result: 14/14 checks passed, plus the blockwise custom op passed torch.compile(fullgraph=True) with bitwise-equal output to the eager wrapper.

Covered public v1 rows:

• M=1 decode GEMV: K in {512,4096}, N in {512,2048,8192}
• small-M GEMM: M in {8,16,32,64} with representative transformer/diffuser-adjacent K,N rows
• M=1 residual-add GEMV
• block-128 scaled FP8 GEMM at:
  • (M,K,N)=(1,1024,1024)
  • (51,1536,1536)
  • (277,2048,2048)
  • (1024,1152,1152)
  • (2520,3072,3072)
  • (128,4096,12288)

Metrics:

• max_abs
• mean_abs
• p99_abs
• cosine similarity
• output dtype
• tolerance

The blockwise rows use the stricter gate:

• max_abs <= 0.0625
• mean_abs <= 0.003
• p99_abs <= 0.015625
• cosine similarity >= 0.9999

The release benchmark also compares the Tensor wrapper against an independent binding of the original FlashRT pointer API. Matching source code alone is not treated as proof of zero wrapper overhead.

Source Benchmark

Command:

python fp8-gemm/benchmarks/benchmark.py \
  --backend source --mode headline --warmup 20 --iterations 100 --compile-ref

Result: all public rows passed. Headline rows are recorded in benchmarks/RESULTS.md.

Architecture Scope Boundary

On SM120, the public per-tensor path supports M=1 and 2 <= M <= 64. The blockwise path retains its independent unrestricted-M contract.

On SM110, the public per-tensor path uses the production CUTLASS Sq/T1/Wide family and supports the validated model-shape matrix. The current full sweep covers the large-M boundary 65, exact PI0.5 prefill rows 712/768/970, and representative K,N rows from PI0.5, GROOT N1.6/N1.7, Cosmos Edge, and LingBot VLA. It also gates BF16 bias, in-place bias+residual, and tanh-GELU bias epilogues on SigLIP dimensions 1152/3456/4304. SM110 blockwise scaling is not claimed.

The August 8 Thor source gate passed 39/39 with zero failures. Plain large-M GEMM was bitwise equal to the reference, and every PI0.5 prefill auto tile was within 2% of the fastest validated package/native tile.

Thor SM110 Increment

Validated August 2, 2026:

• GPU: NVIDIA Thor, compute capability 11.0;
• PyTorch: 2.11.0+cu130;
• CUDA: 13.0;
• CUTLASS: 4.5.2, matching the current kernel-builder cutlass_4_5 dependency;
• pinned builder: e9152aa24e0d99eca255ca9f1beb996de32f9ca4;
• source correctness: 23/23;
• locally installed aarch64 artifact correctness: 23/23;
• torch.compile(fullgraph=True): exact output parity;
• CUDA Graph capture/replay: exact output parity;
• original SM120 source regression on RTX 5090: 14/14.

The 23 Thor rows include 20 production auto-dispatch checks and three forced Sq/T1/Wide diagnostics. Ordinary GEMMs were bitwise equal to the FP32 accumulation reference after BF16 output conversion. The residual row passed with max_abs=0.0625, p99_abs=0.0625, and cosine 0.9999958 under the documented BF16 residual contract.

Source-to-installed-artifact performance parity passed over 17 public auto-dispatch shapes: median artifact/source 0.9986, p95 1.0195, and max 1.0244. Comparisons against the original FlashRT pointer entry are reported separately in benchmarks/RESULTS.md.

The final clean local artifact was built from d31c69b1cb97ecd703aba01e29f423097f11c86a. All 17 production rows passed the dispatcher gate; the worst auto/fastest-valid-tile paired ratio was 1.0028. Sixteen rows matched the original CUTLASS 4.4.2 native entry within about 1.3% in the paired graph comparison. The PI0.5 gate/up row is a documented CUTLASS 4.5.2 version outlier at 1.128x; it is not described as native-performance parity.

Before the SM110 update was published, the existing Thor pipeline dependency set was cold-loaded from Hub using both kernels==0.16.0 and kernels==0.12.3: 20/20 package imports passed for each client. The Thor host required HF_ENDPOINT=https://hf-mirror.com; direct access to huggingface.co timed out, so official-endpoint cold loading remains a post-publication check on a host with direct Hub access.

HF Jobs Publish Status

flashrt/fp8-gemm v1 was built and uploaded through the repository HF Jobs workflow.

• Hub revision checked on June 20, 2026: 166f09be
• Uploaded variants:
  • torch211-cxx11-cu128-x86_64-linux
  • torch211-cxx11-cu130-x86_64-linux
  • torch212-cxx11-cu130-x86_64-linux
  • torch212-cxx11-cu132-x86_64-linux

The existing SM120 Hub variants remain published. The new torch211-cxx11-cu130-aarch64-linux SM110 artifact is not included in the older Hub revision above; it must be published only after the clean-commit artifact rebuild and cold-cache checks.

SM89 Increment

The source now also exposes block-128 scaled FP8 GEMM/GEMV and fp8_blockwise_swiglu_quantize_fp8 on SM89. The fused producer performs the gate/up GEMMs, SiLU product, and block-128 FP8 requantization in one launch. It requires 1<=M<=256, N%128==0, and K%128==0 and uses the upstream measured 32x128-w4-s1 tile. SM120 source regression remains 14/14. SM89 installed correctness, tile parity, and performance claims remain gated on an SM89 release artifact run; source presence alone is not recorded as runtime validation.