| | <!DOCTYPE html> |
| | <html lang="en"> |
| | <head> |
| | <meta charset="UTF-8"> |
| | <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| | <title>LLM Quantization Formats & CUDA Support Reference</title> |
| | <style> |
| | * { |
| | margin: 0; |
| | padding: 0; |
| | box-sizing: border-box; |
| | } |
| | |
| | body { |
| | font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', sans-serif; |
| | background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); |
| | min-height: 100vh; |
| | padding: 2rem; |
| | color: #333; |
| | } |
| | |
| | .container { |
| | max-width: 1400px; |
| | margin: 0 auto; |
| | background: white; |
| | border-radius: 16px; |
| | box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3); |
| | overflow: hidden; |
| | } |
| | |
| | header { |
| | background: linear-gradient(135deg, #1e3c72 0%, #2a5298 100%); |
| | color: white; |
| | padding: 3rem 2rem; |
| | text-align: center; |
| | } |
| | |
| | h1 { |
| | font-size: 2.5rem; |
| | margin-bottom: 0.5rem; |
| | font-weight: 700; |
| | } |
| | |
| | .subtitle { |
| | font-size: 1.1rem; |
| | opacity: 0.9; |
| | font-weight: 300; |
| | } |
| | |
| | .content { |
| | padding: 2rem; |
| | } |
| | |
| | .section { |
| | margin-bottom: 3rem; |
| | } |
| | |
| | h2 { |
| | color: #1e3c72; |
| | font-size: 1.8rem; |
| | margin-bottom: 1.5rem; |
| | border-bottom: 3px solid #667eea; |
| | padding-bottom: 0.5rem; |
| | } |
| | |
| | .table-wrapper { |
| | overflow-x: auto; |
| | margin-bottom: 2rem; |
| | border-radius: 8px; |
| | box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); |
| | } |
| | |
| | table { |
| | width: 100%; |
| | border-collapse: collapse; |
| | font-size: 0.95rem; |
| | } |
| | |
| | thead { |
| | background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); |
| | color: white; |
| | } |
| | |
| | th { |
| | padding: 1rem; |
| | text-align: left; |
| | font-weight: 600; |
| | text-transform: uppercase; |
| | font-size: 0.85rem; |
| | letter-spacing: 0.5px; |
| | } |
| | |
| | td { |
| | padding: 0.9rem 1rem; |
| | border-bottom: 1px solid #e5e7eb; |
| | } |
| | |
| | tbody tr { |
| | transition: background-color 0.2s; |
| | } |
| | |
| | tbody tr:hover { |
| | background-color: #f3f4f6; |
| | } |
| | |
| | tbody tr:nth-child(even) { |
| | background-color: #f9fafb; |
| | } |
| | |
| | .highlight { |
| | background: linear-gradient(135deg, #fef3c7 0%, #fde68a 100%); |
| | font-weight: 600; |
| | } |
| | |
| | .cuda-grid { |
| | display: grid; |
| | grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); |
| | gap: 1.5rem; |
| | margin-top: 1rem; |
| | } |
| | |
| | .cuda-card { |
| | background: linear-gradient(135deg, #f3f4f6 0%, #e5e7eb 100%); |
| | padding: 1.5rem; |
| | border-radius: 8px; |
| | border-left: 4px solid #667eea; |
| | } |
| | |
| | .cuda-card h3 { |
| | color: #1e3c72; |
| | font-size: 1.2rem; |
| | margin-bottom: 0.5rem; |
| | } |
| | |
| | .cuda-card p { |
| | color: #6b7280; |
| | line-height: 1.6; |
| | } |
| | |
| | .notes-grid { |
| | display: grid; |
| | gap: 1rem; |
| | margin-top: 1rem; |
| | } |
| | |
| | .note-item { |
| | background: #f0f9ff; |
| | padding: 1rem; |
| | border-radius: 6px; |
| | border-left: 3px solid #3b82f6; |
| | } |
| | |
| | .note-item strong { |
| | color: #1e40af; |
| | } |
| | |
| | footer { |
| | background: #f9fafb; |
| | padding: 2rem; |
| | text-align: center; |
| | color: #6b7280; |
| | border-top: 1px solid #e5e7eb; |
| | } |
| | |
| | @media (max-width: 768px) { |
| | body { |
| | padding: 1rem; |
| | } |
| | |
| | h1 { |
| | font-size: 1.8rem; |
| | } |
| | |
| | .content { |
| | padding: 1rem; |
| | } |
| | |
| | table { |
| | font-size: 0.85rem; |
| | } |
| | |
| | th, td { |
| | padding: 0.7rem 0.5rem; |
| | } |
| | } |
| | </style> |
| | </head> |
| | <body> |
| | <div class="container"> |
| | <header> |
| | <h1>🚀 Quantization Formats & CUDA Support</h1> |
| | <p class="subtitle">Complete reference guide for LLM quantization methods and hardware requirements</p> |
| | </header> |
| |
|
| | <div class="content"> |
| | <div class="section"> |
| | <h2>📊 Quantization Formats</h2> |
| | <div class="table-wrapper"> |
| | <table> |
| | <thead> |
| | <tr> |
| | <th>Format</th> |
| | <th>Bits</th> |
| | <th>Min CUDA</th> |
| | <th>GPU Examples</th> |
| | <th>Notes</th> |
| | </tr> |
| | </thead> |
| | <tbody> |
| | <tr> |
| | <td><strong>FP16</strong></td> |
| | <td>16</td> |
| | <td>5.3+</td> |
| | <td>GTX 1000, RTX 2000+</td> |
| | <td>Native half precision</td> |
| | </tr> |
| | <tr> |
| | <td><strong>BF16</strong></td> |
| | <td>16</td> |
| | <td>8.0+</td> |
| | <td>A100, RTX 3090, 4090</td> |
| | <td>Better range than FP16</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>FP8 (E4M3/E5M2)</strong></td> |
| | <td>8</td> |
| | <td>8.9+</td> |
| | <td>H100, H200, L40S</td> |
| | <td>Transformer Engine support</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>MXFP8</strong></td> |
| | <td>8</td> |
| | <td>8.9+</td> |
| | <td>H100, H200, Blackwell</td> |
| | <td>Block-size 32, E8M0 scale</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>FP6</strong></td> |
| | <td>6</td> |
| | <td>10.0+</td> |
| | <td>GB200, B100, B200</td> |
| | <td>Blackwell native support</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>MXFP6</strong></td> |
| | <td>6</td> |
| | <td>8.9+</td> |
| | <td>H100+, Blackwell</td> |
| | <td>E2M3/E3M2, block-size 32</td> |
| | </tr> |
| | <tr> |
| | <td><strong>INT8</strong></td> |
| | <td>8</td> |
| | <td>6.1+</td> |
| | <td>GTX 1080+, P100+</td> |
| | <td>Wide compatibility</td> |
| | </tr> |
| | <tr> |
| | <td><strong>INT4</strong></td> |
| | <td>4</td> |
| | <td>7.5+</td> |
| | <td>RTX 2080+, T4, V100</td> |
| | <td>CUTLASS kernels</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>MXFP4</strong></td> |
| | <td>4</td> |
| | <td>9.0+</td> |
| | <td>H100, H200, GB200</td> |
| | <td>E2M1, block-size 32, OpenAI</td> |
| | </tr> |
| | <tr class="highlight"> |
| | <td><strong>NVFP4</strong></td> |
| | <td>4</td> |
| | <td>10.0+</td> |
| | <td>GB200, B100, B200</td> |
| | <td>E2M1, block-size 16, dual-scale</td> |
| | </tr> |
| | <tr> |
| | <td><strong>GPTQ</strong></td> |
| | <td>2-8</td> |
| | <td>7.0+</td> |
| | <td>RTX 2000+, V100+</td> |
| | <td>Group-wise quantization</td> |
| | </tr> |
| | <tr> |
| | <td><strong>AWQ</strong></td> |
| | <td>4</td> |
| | <td>7.5+</td> |
| | <td>RTX 3000+, A100+</td> |
| | <td>Activation-aware</td> |
| | </tr> |
| | <tr> |
| | <td><strong>QuIP</strong></td> |
| | <td>2-4</td> |
| | <td>7.0+</td> |
| | <td>RTX 2000+, V100+</td> |
| | <td>Incoherence processing</td> |
| | </tr> |
| | <tr> |
| | <td><strong>QuIP#</strong></td> |
| | <td>2-4</td> |
| | <td>8.0+</td> |
| | <td>RTX 3090+, A100+</td> |
| | <td>E8P lattice codebook</td> |
| | </tr> |
| | <tr> |
| | <td><strong>GGUF/GGML</strong></td> |
| | <td>2-8</td> |
| | <td>6.1+</td> |
| | <td>GTX 1060+, most GPUs</td> |
| | <td>CPU fallback available</td> |
| | </tr> |
| | <tr> |
| | <td><strong>EXL2</strong></td> |
| | <td>2-8</td> |
| | <td>7.5+</td> |
| | <td>RTX 2000+, V100+</td> |
| | <td>Variable bit-width</td> |
| | </tr> |
| | <tr> |
| | <td><strong>NF4</strong></td> |
| | <td>4</td> |
| | <td>7.0+</td> |
| | <td>RTX 2000+, V100+</td> |
| | <td>QLoRA, normal float</td> |
| | </tr> |
| | <tr> |
| | <td><strong>GGUF-IQ</strong></td> |
| | <td>1-8</td> |
| | <td>6.1+</td> |
| | <td>GTX 1060+</td> |
| | <td>Importance matrix</td> |
| | </tr> |
| | </tbody> |
| | </table> |
| | </div> |
| | </div> |
| |
|
| | <div class="section"> |
| | <h2>🎯 CUDA Compute Capabilities</h2> |
| | <div class="cuda-grid"> |
| | <div class="cuda-card"> |
| | <h3>6.1 - Pascal</h3> |
| | <p>GTX 1000 series, Tesla P100</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>7.0 - Volta</h3> |
| | <p>Tesla V100, Titan V</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>7.5 - Turing</h3> |
| | <p>RTX 2000 series, T4, RTX 6000</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>8.0 - Ampere</h3> |
| | <p>A100, RTX 3090</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>8.6 - Ampere</h3> |
| | <p>RTX 3000 series consumer</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>8.9 - Ada Lovelace</h3> |
| | <p>RTX 4000 series, L40S</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>9.0 - Hopper</h3> |
| | <p>H100, H200</p> |
| | </div> |
| | <div class="cuda-card"> |
| | <h3>10.0 - Blackwell</h3> |
| | <p>GB200, B100, B200</p> |
| | </div> |
| | </div> |
| | </div> |
| |
|
| | <div class="section"> |
| | <h2>⚡ Performance Notes</h2> |
| | <div class="notes-grid"> |
| | <div class="note-item"> |
| | <strong>FP8/MXFP8:</strong> Transformer Engine support, 2x faster than BF16 on H100+ |
| | </div> |
| | <div class="note-item"> |
| | <strong>NVFP4:</strong> Native on Blackwell, 2x faster than FP8, ~3.5x memory reduction vs FP16 |
| | </div> |
| | <div class="note-item"> |
| | <strong>MXFP4:</strong> Requires H100+ (CC 9.0), uses Triton kernels, OpenAI GPT-OSS format |
| | </div> |
| | <div class="note-item"> |
| | <strong>MXFP6:</strong> Training & inference on H100+, better accuracy than MXFP4 |
| | </div> |
| | <div class="note-item"> |
| | <strong>QuIP#:</strong> 2-4 bit with E8P lattice codebook, 50% peak bandwidth on RTX 4090 |
| | </div> |
| | <div class="note-item"> |
| | <strong>INT4/GPTQ/AWQ:</strong> ~3-4x memory reduction, 1.5-2x faster inference |
| | </div> |
| | <div class="note-item"> |
| | <strong>GGUF:</strong> Best CPU/GPU hybrid performance |
| | </div> |
| | <div class="note-item"> |
| | <strong>EXL2:</strong> Highest quality at low bits, slower than GPTQ |
| | </div> |
| | </div> |
| | </div> |
| | </div> |
| |
|
| | <footer> |
| | <p>Last updated: November 2025 | Reference for LLM quantization formats and CUDA compute capability requirements</p> |
| | </footer> |
| | </div> |
| | </body> |
| | </html> |
