| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>LLM Quantization Formats & CUDA Support Reference</title> |
| <style> |
| * { |
| margin: 0; |
| padding: 0; |
| box-sizing: border-box; |
| } |
| |
| body { |
| font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', sans-serif; |
| background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); |
| min-height: 100vh; |
| padding: 2rem; |
| color: #333; |
| } |
| |
| .container { |
| max-width: 1400px; |
| margin: 0 auto; |
| background: white; |
| border-radius: 16px; |
| box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3); |
| overflow: hidden; |
| } |
| |
| header { |
| background: linear-gradient(135deg, #1e3c72 0%, #2a5298 100%); |
| color: white; |
| padding: 3rem 2rem; |
| text-align: center; |
| } |
| |
| h1 { |
| font-size: 2.5rem; |
| margin-bottom: 0.5rem; |
| font-weight: 700; |
| } |
| |
| .subtitle { |
| font-size: 1.1rem; |
| opacity: 0.9; |
| font-weight: 300; |
| } |
| |
| .content { |
| padding: 2rem; |
| } |
| |
| .section { |
| margin-bottom: 3rem; |
| } |
| |
| h2 { |
| color: #1e3c72; |
| font-size: 1.8rem; |
| margin-bottom: 1.5rem; |
| border-bottom: 3px solid #667eea; |
| padding-bottom: 0.5rem; |
| } |
| |
| .table-wrapper { |
| overflow-x: auto; |
| margin-bottom: 2rem; |
| border-radius: 8px; |
| box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); |
| } |
| |
| table { |
| width: 100%; |
| border-collapse: collapse; |
| font-size: 0.95rem; |
| } |
| |
| thead { |
| background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); |
| color: white; |
| } |
| |
| th { |
| padding: 1rem; |
| text-align: left; |
| font-weight: 600; |
| text-transform: uppercase; |
| font-size: 0.85rem; |
| letter-spacing: 0.5px; |
| } |
| |
| td { |
| padding: 0.9rem 1rem; |
| border-bottom: 1px solid #e5e7eb; |
| } |
| |
| tbody tr { |
| transition: background-color 0.2s; |
| } |
| |
| tbody tr:hover { |
| background-color: #f3f4f6; |
| } |
| |
| tbody tr:nth-child(even) { |
| background-color: #f9fafb; |
| } |
| |
| .highlight { |
| background: linear-gradient(135deg, #fef3c7 0%, #fde68a 100%); |
| font-weight: 600; |
| } |
| |
| .cuda-grid { |
| display: grid; |
| grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); |
| gap: 1.5rem; |
| margin-top: 1rem; |
| } |
| |
| .cuda-card { |
| background: linear-gradient(135deg, #f3f4f6 0%, #e5e7eb 100%); |
| padding: 1.5rem; |
| border-radius: 8px; |
| border-left: 4px solid #667eea; |
| } |
| |
| .cuda-card h3 { |
| color: #1e3c72; |
| font-size: 1.2rem; |
| margin-bottom: 0.5rem; |
| } |
| |
| .cuda-card p { |
| color: #6b7280; |
| line-height: 1.6; |
| } |
| |
| .notes-grid { |
| display: grid; |
| gap: 1rem; |
| margin-top: 1rem; |
| } |
| |
| .note-item { |
| background: #f0f9ff; |
| padding: 1rem; |
| border-radius: 6px; |
| border-left: 3px solid #3b82f6; |
| } |
| |
| .note-item strong { |
| color: #1e40af; |
| } |
| |
| footer { |
| background: #f9fafb; |
| padding: 2rem; |
| text-align: center; |
| color: #6b7280; |
| border-top: 1px solid #e5e7eb; |
| } |
| |
| @media (max-width: 768px) { |
| body { |
| padding: 1rem; |
| } |
| |
| h1 { |
| font-size: 1.8rem; |
| } |
| |
| .content { |
| padding: 1rem; |
| } |
| |
| table { |
| font-size: 0.85rem; |
| } |
| |
| th, td { |
| padding: 0.7rem 0.5rem; |
| } |
| } |
| </style> |
| </head> |
| <body> |
| <div class="container"> |
| <header> |
| <h1>🚀 Quantization Formats & CUDA Support</h1> |
| <p class="subtitle">Complete reference guide for LLM quantization methods and hardware requirements</p> |
| </header> |
|
|
| <div class="content"> |
| <div class="section"> |
| <h2>📊 Quantization Formats</h2> |
| <div class="table-wrapper"> |
| <table> |
| <thead> |
| <tr> |
| <th>Format</th> |
| <th>Bits</th> |
| <th>Min CUDA</th> |
| <th>GPU Examples</th> |
| <th>Notes</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><strong>FP16</strong></td> |
| <td>16</td> |
| <td>5.3+</td> |
| <td>GTX 1000, RTX 2000+</td> |
| <td>Native half precision</td> |
| </tr> |
| <tr> |
| <td><strong>BF16</strong></td> |
| <td>16</td> |
| <td>8.0+</td> |
| <td>A100, RTX 3090, 4090</td> |
| <td>Better range than FP16</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>FP8 (E4M3/E5M2)</strong></td> |
| <td>8</td> |
| <td>8.9+</td> |
| <td>H100, H200, L40S</td> |
| <td>Transformer Engine support</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>MXFP8</strong></td> |
| <td>8</td> |
| <td>8.9+</td> |
| <td>H100, H200, Blackwell</td> |
| <td>Block-size 32, E8M0 scale</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>FP6</strong></td> |
| <td>6</td> |
| <td>10.0+</td> |
| <td>GB200, B100, B200</td> |
| <td>Blackwell native support</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>MXFP6</strong></td> |
| <td>6</td> |
| <td>8.9+</td> |
| <td>H100+, Blackwell</td> |
| <td>E2M3/E3M2, block-size 32</td> |
| </tr> |
| <tr> |
| <td><strong>INT8</strong></td> |
| <td>8</td> |
| <td>6.1+</td> |
| <td>GTX 1080+, P100+</td> |
| <td>Wide compatibility</td> |
| </tr> |
| <tr> |
| <td><strong>INT4</strong></td> |
| <td>4</td> |
| <td>7.5+</td> |
| <td>RTX 2080+, T4, V100</td> |
| <td>CUTLASS kernels</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>MXFP4</strong></td> |
| <td>4</td> |
| <td>9.0+</td> |
| <td>H100, H200, GB200</td> |
| <td>E2M1, block-size 32, OpenAI</td> |
| </tr> |
| <tr class="highlight"> |
| <td><strong>NVFP4</strong></td> |
| <td>4</td> |
| <td>10.0+</td> |
| <td>GB200, B100, B200</td> |
| <td>E2M1, block-size 16, dual-scale</td> |
| </tr> |
| <tr> |
| <td><strong>GPTQ</strong></td> |
| <td>2-8</td> |
| <td>7.0+</td> |
| <td>RTX 2000+, V100+</td> |
| <td>Group-wise quantization</td> |
| </tr> |
| <tr> |
| <td><strong>AWQ</strong></td> |
| <td>4</td> |
| <td>7.5+</td> |
| <td>RTX 3000+, A100+</td> |
| <td>Activation-aware</td> |
| </tr> |
| <tr> |
| <td><strong>QuIP</strong></td> |
| <td>2-4</td> |
| <td>7.0+</td> |
| <td>RTX 2000+, V100+</td> |
| <td>Incoherence processing</td> |
| </tr> |
| <tr> |
| <td><strong>QuIP#</strong></td> |
| <td>2-4</td> |
| <td>8.0+</td> |
| <td>RTX 3090+, A100+</td> |
| <td>E8P lattice codebook</td> |
| </tr> |
| <tr> |
| <td><strong>GGUF/GGML</strong></td> |
| <td>2-8</td> |
| <td>6.1+</td> |
| <td>GTX 1060+, most GPUs</td> |
| <td>CPU fallback available</td> |
| </tr> |
| <tr> |
| <td><strong>EXL2</strong></td> |
| <td>2-8</td> |
| <td>7.5+</td> |
| <td>RTX 2000+, V100+</td> |
| <td>Variable bit-width</td> |
| </tr> |
| <tr> |
| <td><strong>NF4</strong></td> |
| <td>4</td> |
| <td>7.0+</td> |
| <td>RTX 2000+, V100+</td> |
| <td>QLoRA, normal float</td> |
| </tr> |
| <tr> |
| <td><strong>GGUF-IQ</strong></td> |
| <td>1-8</td> |
| <td>6.1+</td> |
| <td>GTX 1060+</td> |
| <td>Importance matrix</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| </div> |
|
|
| <div class="section"> |
| <h2>🎯 CUDA Compute Capabilities</h2> |
| <div class="cuda-grid"> |
| <div class="cuda-card"> |
| <h3>6.1 - Pascal</h3> |
| <p>GTX 1000 series, Tesla P100</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>7.0 - Volta</h3> |
| <p>Tesla V100, Titan V</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>7.5 - Turing</h3> |
| <p>RTX 2000 series, T4, RTX 6000</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>8.0 - Ampere</h3> |
| <p>A100, RTX 3090</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>8.6 - Ampere</h3> |
| <p>RTX 3000 series consumer</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>8.9 - Ada Lovelace</h3> |
| <p>RTX 4000 series, L40S</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>9.0 - Hopper</h3> |
| <p>H100, H200</p> |
| </div> |
| <div class="cuda-card"> |
| <h3>10.0 - Blackwell</h3> |
| <p>GB200, B100, B200</p> |
| </div> |
| </div> |
| </div> |
|
|
| <div class="section"> |
| <h2>⚡ Performance Notes</h2> |
| <div class="notes-grid"> |
| <div class="note-item"> |
| <strong>FP8/MXFP8:</strong> Transformer Engine support, 2x faster than BF16 on H100+ |
| </div> |
| <div class="note-item"> |
| <strong>NVFP4:</strong> Native on Blackwell, 2x faster than FP8, ~3.5x memory reduction vs FP16 |
| </div> |
| <div class="note-item"> |
| <strong>MXFP4:</strong> Requires H100+ (CC 9.0), uses Triton kernels, OpenAI GPT-OSS format |
| </div> |
| <div class="note-item"> |
| <strong>MXFP6:</strong> Training & inference on H100+, better accuracy than MXFP4 |
| </div> |
| <div class="note-item"> |
| <strong>QuIP#:</strong> 2-4 bit with E8P lattice codebook, 50% peak bandwidth on RTX 4090 |
| </div> |
| <div class="note-item"> |
| <strong>INT4/GPTQ/AWQ:</strong> ~3-4x memory reduction, 1.5-2x faster inference |
| </div> |
| <div class="note-item"> |
| <strong>GGUF:</strong> Best CPU/GPU hybrid performance |
| </div> |
| <div class="note-item"> |
| <strong>EXL2:</strong> Highest quality at low bits, slower than GPTQ |
| </div> |
| </div> |
| </div> |
| </div> |
|
|
| <footer> |
| <p>Last updated: November 2025 | Reference for LLM quantization formats and CUDA compute capability requirements</p> |
| </footer> |
| </div> |
| </body> |
| </html> |
