# NVFP4 Kernel for Candle

## Overview
This repository contains an NVFP4 (NVIDIA 4-bit Floating Point) kernel implementation
for the [Candle](https://github.com/huggingface/candle) ML framework.

## NVFP4 Format
- **Element format**: E2M1 (4-bit float: 1 sign, 2 exponent, 1 mantissa)
- **Representable values**: ±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6
- **Block size**: 16 elements
- **Block scale**: E4M3 FP8 (8 bits per block)
- **Memory layout**: 9 bytes per 16 elements (8 data + 1 scale)
- **Compression**: ~7.1x vs FP32

## Files
- `candle-kernels/src/nvfp4.cu` - CUDA kernels for quantize, dequantize, matmul
- `candle-core/src/quantized/mod.rs` - GgmlDType enum with NVFP4 variant
- `candle-core/src/quantized/cuda.rs` - CUDA backend integration
- `candle-core/src/quantized/k_quants.rs` - BlockNvfp4 Rust type and GgmlType impl
- `examples/nvfp4-inference.rs` - Rust inference example
- `test_nvfp4.py` - Python test suite (PyTorch-based validation)

## GPU
Tested on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM 12.0, MIG 2g.48gb).

## Test Results
- E2M1 round-trip: max error 0.25 (expected for 4-bit precision)
- E4M3 scale round-trip: max relative error 7.1% (3 mantissa bits)
- Block quantization: relative error < 16% across distributions
- MatMul (1x256x512): relative error 10.1%
- MatMul (4x1024x2048): relative error 9.4%
- Memory savings: 7.11x compression vs FP32