nvfp4/README.md

NVFP4 Kernel for Candle

Overview

This repository contains an NVFP4 (NVIDIA 4-bit Floating Point) kernel implementation for the Candle ML framework.

NVFP4 Format

• Element format: E2M1 (4-bit float: 1 sign, 2 exponent, 1 mantissa)
• Representable values: ±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6
• Block size: 16 elements
• Block scale: E4M3 FP8 (8 bits per block)
• Memory layout: 9 bytes per 16 elements (8 data + 1 scale)
• Compression: ~7.1x vs FP32

Files

• candle-kernels/src/nvfp4.cu - CUDA kernels for quantize, dequantize, matmul
• candle-core/src/quantized/mod.rs - GgmlDType enum with NVFP4 variant
• candle-core/src/quantized/cuda.rs - CUDA backend integration
• candle-core/src/quantized/k_quants.rs - BlockNvfp4 Rust type and GgmlType impl
• examples/nvfp4-inference.rs - Rust inference example
• test_nvfp4.py - Python test suite (PyTorch-based validation)

GPU

Tested on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM 12.0, MIG 2g.48gb).

Test Results

• E2M1 round-trip: max error 0.25 (expected for 4-bit precision)
• E4M3 scale round-trip: max relative error 7.1% (3 mantissa bits)
• Block quantization: relative error < 16% across distributions
• MatMul (1x256x512): relative error 10.1%
• MatMul (4x1024x2048): relative error 9.4%
• Memory savings: 7.11x compression vs FP32