| # Validation |
|
|
| Date: June 20, 2026 |
|
|
| Local environment: |
|
|
| - GPU: NVIDIA GeForce RTX 5090 |
| - PyTorch: 2.9.1+cu128 |
| - CUDA runtime reported by PyTorch: 12.8 |
| - Source build target: `sm_120a` |
|
|
| ## Source Correctness |
|
|
| Command: |
|
|
| ```bash |
| python fp8-gemm/tests/test_fp8_gemm.py --backend source --mode full |
| ``` |
|
|
| Result: 14/14 checks passed, plus the blockwise custom op passed |
| `torch.compile(fullgraph=True)` with bitwise-equal output to the eager wrapper. |
|
|
| Covered public v1 rows: |
|
|
| - M=1 decode GEMV: `K in {512,4096}`, `N in {512,2048,8192}` |
| - small-M GEMM: `M in {8,16,32,64}` with representative |
| transformer/diffuser-adjacent `K,N` rows |
| - M=1 residual-add GEMV |
| - block-128 scaled FP8 GEMM at: |
| - `(M,K,N)=(1,1024,1024)` |
| - `(51,1536,1536)` |
| - `(277,2048,2048)` |
| - `(1024,1152,1152)` |
| - `(2520,3072,3072)` |
| - `(128,4096,12288)` |
|
|
| Metrics: |
|
|
| - `max_abs` |
| - `mean_abs` |
| - `p99_abs` |
| - cosine similarity |
| - output dtype |
| - tolerance |
|
|
| The blockwise rows use the stricter gate: |
|
|
| - `max_abs <= 0.0625` |
| - `mean_abs <= 0.003` |
| - `p99_abs <= 0.015625` |
| - cosine similarity `>= 0.9999` |
|
|
| The release benchmark also compares the Tensor wrapper against an independent |
| binding of the original FlashRT pointer API. Matching source code alone is not |
| treated as proof of zero wrapper overhead. |
|
|
| ## Source Benchmark |
|
|
| Command: |
|
|
| ```bash |
| python fp8-gemm/benchmarks/benchmark.py \ |
| --backend source --mode headline --warmup 20 --iterations 100 --compile-ref |
| ``` |
|
|
| Result: all public rows passed. Headline rows are recorded in |
| `benchmarks/RESULTS.md`. |
|
|
| ## Architecture Scope Boundary |
|
|
| On SM120, the public per-tensor path supports `M=1` and `2 <= M <= 64`. |
| The blockwise path retains its independent unrestricted-M contract. |
|
|
| On SM110, the public per-tensor path uses the production CUTLASS Sq/T1/Wide |
| family and supports the validated model-shape matrix. The current full sweep |
| covers the large-M boundary `65`, exact PI0.5 prefill rows `712/768/970`, and |
| representative `K,N` rows from PI0.5, GROOT N1.6/N1.7, Cosmos Edge, and |
| LingBot VLA. It also gates BF16 bias, in-place bias+residual, and tanh-GELU |
| bias epilogues on SigLIP dimensions `1152/3456/4304`. SM110 blockwise scaling |
| is not claimed. |
|
|
| The August 8 Thor source gate passed `39/39` with zero failures. Plain large-M |
| GEMM was bitwise equal to the reference, and every PI0.5 prefill auto tile was |
| within 2% of the fastest validated package/native tile. |
|
|
| ## Thor SM110 Increment |
|
|
| Validated August 2, 2026: |
|
|
| - GPU: NVIDIA Thor, compute capability 11.0; |
| - PyTorch: 2.11.0+cu130; |
| - CUDA: 13.0; |
| - CUTLASS: 4.5.2, matching the current `kernel-builder` `cutlass_4_5` |
| dependency; |
| - pinned builder: `e9152aa24e0d99eca255ca9f1beb996de32f9ca4`; |
| - source correctness: 23/23; |
| - locally installed aarch64 artifact correctness: 23/23; |
| - `torch.compile(fullgraph=True)`: exact output parity; |
| - CUDA Graph capture/replay: exact output parity; |
| - original SM120 source regression on RTX 5090: 14/14. |
|
|
| The 23 Thor rows include 20 production auto-dispatch checks and three forced |
| Sq/T1/Wide diagnostics. Ordinary GEMMs were bitwise equal to the FP32 |
| accumulation reference after BF16 output conversion. The residual row passed |
| with `max_abs=0.0625`, `p99_abs=0.0625`, and cosine `0.9999958` under the |
| documented BF16 residual contract. |
|
|
| Source-to-installed-artifact performance parity passed over 17 public |
| auto-dispatch shapes: median artifact/source `0.9986`, p95 `1.0195`, and max |
| `1.0244`. |
| Comparisons against the original FlashRT pointer entry are reported separately |
| in `benchmarks/RESULTS.md`. |
|
|
| The final clean local artifact was built from |
| `d31c69b1cb97ecd703aba01e29f423097f11c86a`. All 17 production rows passed the |
| dispatcher gate; the worst auto/fastest-valid-tile paired ratio was `1.0028`. |
| Sixteen rows matched the original CUTLASS 4.4.2 native entry within about 1.3% |
| in the paired graph comparison. The PI0.5 gate/up row is a documented CUTLASS |
| 4.5.2 version outlier at `1.128x`; it is not described as native-performance |
| parity. |
|
|
| Before the SM110 update was published, the existing Thor pipeline dependency |
| set was cold-loaded from Hub using both `kernels==0.16.0` and |
| `kernels==0.12.3`: 20/20 package imports passed for each client. The Thor host |
| required `HF_ENDPOINT=https://hf-mirror.com`; direct access to |
| `huggingface.co` timed out, so official-endpoint cold loading remains a |
| post-publication check on a host with direct Hub access. |
|
|
| ## HF Jobs Publish Status |
|
|
| `flashrt/fp8-gemm` v1 was built and uploaded through the repository HF Jobs |
| workflow. |
|
|
| - Hub revision checked on June 20, 2026: `166f09be` |
| - Uploaded variants: |
| - `torch211-cxx11-cu128-x86_64-linux` |
| - `torch211-cxx11-cu130-x86_64-linux` |
| - `torch212-cxx11-cu130-x86_64-linux` |
| - `torch212-cxx11-cu132-x86_64-linux` |
|
|
| The existing SM120 Hub variants remain published. The new |
| `torch211-cxx11-cu130-aarch64-linux` SM110 artifact is not included in the |
| older Hub revision above; it must be published only after the clean-commit |
| artifact rebuild and cold-cache checks. |
|
|
| ## SM89 Increment |
|
|
| The source now also exposes block-128 scaled FP8 GEMM/GEMV and |
| `fp8_blockwise_swiglu_quantize_fp8` on SM89. The fused producer performs the |
| gate/up GEMMs, SiLU product, and block-128 FP8 requantization in one launch. |
| It requires `1<=M<=256`, `N%128==0`, and `K%128==0` and uses the upstream |
| measured `32x128-w4-s1` tile. SM120 source regression remains 14/14. SM89 |
| installed correctness, tile parity, and performance claims remain gated on an |
| SM89 release artifact run; source presence alone is not recorded as runtime |
| validation. |
|
|
