README.md

---
license: mit
title: IndiaAI GPU Infrastructure Recommender for AI Models
sdk: gradio
emoji: 🚀
colorFrom: purple
colorTo: gray
short_description: Calculate VRAM requirements +get optimal GPU recommendations
pinned: false
thumbnail: >-
  https://cdn-uploads.huggingface.co/production/uploads/69312fe079dac228ed39fb04/kZDf4ccek1FcfpojnV_yJ.webp
---
# IndiaAI GPU Infrastructure Recommender for AI Models

A comprehensive tool for estimating VRAM requirements and recommending optimal GPU configurations for Large Language Model (LLM) deployment and training using IndiaAI's price list.

## Overview

This recommender provides accurate VRAM estimates and GPU recommendations for:
- **Inference**: Single-stream and batched inference workloads with quantization speedups
- **Training**: Full fine-tuning, LoRA, and QLoRA with accurate precision handling
- **Multiple Quantization**: fp16, bf16, int8, int4, nf4, AWQ, GPTQ with realistic throughput
- **Framework Support**: vLLM, HuggingFace Transformers with framework-specific optimizations

### Key Features

- **Accurate VRAM Estimation**: Based on empirically-validated formulas from academic papers and production deployments
- **Quantization-Aware Throughput**: INT4 is 3x faster, INT8 is 1.8x faster than FP16 baseline
- **Framework-Specific Performance**: HuggingFace 30% slower than vLLM baseline
- **Precision Override Logic**: Automatically upgrades to bf16 for Full FT and LoRA (cannot train quantized weights)
- **LoRA Rank Impact**: Accounts for 2-4.5x throughput improvement based on rank size and model size
- **Sequence Length Scaling**: Realistic throughput reduction for longer contexts
- **Optimized Batch Scaling**: Model-size-dependent batch optimization for accurate time estimates
- **Cost Analysis**: Compare costs across different GPU configurations with accurate time estimates
- **Multiple Pricing Tiers**: On-demand, 1-month, 6-month, and 12-month reserved pricing
- **Comprehensive GPU Database**: 55+ configurations from 1x to 8x GPUs across all major vendors
- **GPU Manufacturer Filtering**: Filter recommendations by Nvidia, AMD, or Intel
- **Interactive Comparison Charts**: Visual comparison of GPUs with 3 different views
- **CSV Export**: Download detailed analysis with all formulas and calculations
- **Production-Ready**: ±15-20% accuracy validated against real deployments

### Interactive Visualization

The tool provides **interactive comparison charts** (Top 10 GPUs by cost):
1. **Throughput**: Tokens/sec comparison - find fastest processing
2. **Cost**: ₹/hour pricing - compare rental costs
3. **VRAM Utilization**: % of GPU memory used - identify over-provisioned GPUs

## Quick Start

### Installation

```bash
# Clone the repository
git clone https://github.com/CoE-HCC/indiaai-gpu-infrastructure-recommender.git
cd indiaai-gpu-infrastructure-recommender

# Install dependencies
pip install -r requirements.txt
```

### HuggingFace Token (for automatic model config resolution)

For gated models that require authentication or automatic config resolution:

```bash
# Set your HuggingFace token
export HF_TOKEN="your_token_here"
```

**Note**: The tool works without this token using fallback estimation.

### Basic Usage

```bash
# Run the application
python app.py
```

### Access the Web Interface

```
http://localhost:7860
```

The interface will open in your default browser.

## User Interface

### Input Configuration

**Model Selection**:
- Pre-configured popular models from HuggingFace Hub
- Or enter custom size (e.g., "7B", "70B", "405B")
- Or enter HuggingFace model ID (e.g., "meta-llama/Llama-3.1-70B-Instruct")

**Task Selection**:
- **Inference**: For model serving and generation
- **Training**: For fine-tuning and training workloads

**Precision/Quantization**:
- fp16/bf16: 2 bytes/param (baseline)
- int8: 1 byte/param (1.8x faster inference)
- int4/nf4: 0.5 bytes/param (3.0x faster inference)
- AWQ/GPTQ: 0.52 bytes/param (3.2x faster inference)

**Training Options** (when Task=Training):
- **Full Fine-Tuning**: Train all parameters (requires bf16)
- **LoRA**: Parameter-efficient training (requires bf16 base)
- **QLoRA**: Memory-efficient training (quantized base + fp16 adapters)
- **LoRA Rank**: 8, 16, 32, 64, 128, 256 (affects memory and speed)

**Workload Parameters**:
- Max Context Length: Sequence length for KV cache sizing
- Batch Size: Samples processed in parallel
- Samples: Total number of samples to process
- Input/Output Tokens: Average token counts per sample

**Pricing & Filtering**:
- Pricing Tier: On Demand, 1/6/12 Month Reserved
- GPU Manufacturers: Nvidia, AMD, Intel (multi-select)

### Recommendations Panel

Two recommendation cards are displayed:
- **🥇 Best Budget**: Lowest cost GPU meeting requirements
- **🥈 Budget Runner-up**: Second most affordable option

Each card shows:
- GPU configuration name and count
- Total VRAM and per-GPU VRAM
- VRAM utilization percentage
- Performance (TFLOPS) and bandwidth
- Estimated throughput (tokens/sec)
- Time estimate for your workload
- Hourly, daily, and monthly pricing

### GPU Comparison Charts

Expandable section with interactive Plotly charts comparing **Top 10 GPUs by cost**:

| Chart | Y-Axis | Description | Use Case |
|-------|--------|-------------|----------|
| **Throughput** | Tokens/sec | Processing speed | Find fastest GPUs |
| **Cost** | ₹/hour | Hourly pricing | Compare costs |
| **VRAM Utilization** | % Used | Memory efficiency | Avoid over-provisioning |

### CSV Export

Download button provides comprehensive analysis including:
- All input parameters
- Derived model architecture
- Precision parameters
- VRAM calculation breakdown with formulas
- GPU recommendations (all viable options)
- Throughput calculations
- Cost projections

## Supported Models

### Pre-configured Models

**Llama 3 Family**:
- Llama 3.3: 70B
- Llama 3.1: 8B, 70B, 405B
- Llama 3.2: 1B, 3B

**Qwen 2.5 Family**:
- Qwen 2.5: 1.5B, 3B, 7B, 14B, 32B, 72B
- Qwen 2.5 Coder: 32B

**Mistral Family**:
- Mistral: 7B, Small, Large, Nemo
- Mixtral: 8x22B
- Ministral: 8B

Or enter any:
- Model size: "7B", "13B", "70B", etc.
- HuggingFace ID: "organization/model-name"

## GPU Database

### Comprehensive Hardware Catalog

**55+ GPU configurations** across all major vendors:

**NVIDIA H-Series (Hopper)**:
- H100 SXM: 80GB (1x, 2x, 4x, 8x)
- H100 NVL: 94GB (1x, 2x, 4x, 8x)
- H100 PCIe: 80GB (1x, 8x)
- H200 SXM: 141GB (1x, 2x, 4x, 8x)
- H200 NVL: 141GB (1x, 2x, 4x, 8x)
- H200 PCIe: 141GB (8x)

**NVIDIA B-Series (Blackwell)**:
- B200 SXM: 180GB (1x, 2x, 4x, 8x)

**NVIDIA A-Series (Ampere)**:
- A100 40GB: (1x, 2x, 4x, 8x)
- A100 80GB: (1x, 2x, 4x, 8x)

**NVIDIA L-Series (Ada)**:
- L40S: 48GB (1x, 2x, 4x, 8x)
- L4: 24GB (1x, 2x, 4x, 8x)

**AMD Instinct**:
- MI300X: 192GB (1x, 2x, 4x, 8x)
- MI325X: 256GB (1x, 2x, 4x, 8x)

**Intel Gaudi**:
- Gaudi 2: 96GB (1x, 2x, 4x, 8x)
- Gaudi 3: 128GB (1x, 2x, 4x, 8x)

### Manufacturer Filtering

Filter recommendations by vendor preference:
- **Nvidia**: Industry standard, best software ecosystem
- **AMD**: Competitive performance, often better value
- **Intel**: Gaudi accelerators for specific workloads

Default: All manufacturers selected

## Technical Details

### VRAM Calculation Components

Total VRAM is calculated as:

```
Total VRAM = Model Weights + KV Cache + [Dynamic Components] + Safety Buffer
```

Where dynamic components depend on the task:
- **Inference**: Framework Overhead
- **Training**: Activations + Optimizer States + Gradients + [LoRA Adapters]

#### 1. Model Weights

**Formula:**
```python
memory_gb = (num_parameters × bytes_per_parameter) / (1024³)
```

**Quantization formats:**

| Format | Bytes/Param | Use Case |
|--------|-------------|----------|
| fp16/bf16 | 2.0      | Standard training/inference |
| int8   | 1.0         | 4x compression, 1.8x faster |
| nf4    | 0.5625      | QLoRA's format, 3x faster |
| int4   | 0.50        | 8x compression, 3x faster |
| awq/gptq | 0.52      | Optimized int4, 3.2x faster |

**Important**: For training with LoRA or Full FT, the code automatically overrides quantized formats to bf16 because you cannot train quantized weights. Only QLoRA keeps the base model quantized.

#### 2. KV Cache (Inference & Training)

**Formula:**
```python
kv_memory = 2 × layers × batch × seq_len × kv_heads × head_dim × bytes_per_elem / (1024³)
```

**Components:**
- **2**: Separate tensors for Keys and Values
- **layers**: Number of transformer layers
- **batch**: Batch size
- **seq_len**: Sequence length
- **kv_heads**: Number of KV heads (for GQA/MQA)
- **head_dim**: Dimension per attention head
- **bytes_per_elem**: Precision in bytes

**GQA Support**: 
- Standard MHA: kv_heads = heads
- Grouped Query Attention (GQA): kv_heads < heads
- Example: Llama 3 uses 32 heads but only 8 kv_heads (4x memory reduction)

**Precision Handling**: 
- KV cache kept at fp16/bf16 even for quantized models (maintains quality)
- For training, always uses bf16 compute precision

#### 3. Activations (Training Only)

**Formula:**
```python
activation_memory = batch × seq × hidden × layers × multiplier × bytes / (1024³)
```

**Multiplier:**
- With gradient checkpointing: **12x** (default in most frameworks)
- Without checkpointing: **34x** (stores all intermediate activations)

**Precision**: Always at compute precision (bf16), even for QLoRA.

#### 4. Optimizer States (Training Only)

**Full Fine-Tuning (Adam):**
```python
optimizer_memory = model_weights_gb × 4
```
- Model weights: fp16 (2 bytes/param)
- Optimizer states: 2 states × fp32 (8 bytes/param total)
- Ratio: 8/2 = **4x model weights**

**LoRA/QLoRA (Adam for adapters only):**
```python
adapter_params = 2 × rank × hidden_dim × layers
adapter_params_gb = (adapter_params × 2) / (1024³)  # fp16
optimizer_memory = adapter_params_gb × 4  # 2 states at fp32
```

**Example** (7B model, rank=64):
- Full FT: 52.15 GB
- LoRA: 0.125 GB
- **417x smaller!**

#### 5. Gradients (Training Only)

**Full Fine-Tuning:**
```python
gradients_memory = model_weights_gb × 2
```
- fp32 gradients for numerical stability
- Ratio: 4/2 = **2x model weights**

**LoRA/QLoRA:**
```python
gradients_memory = (adapter_params × 2) / (1024³)  # fp16
```
- Only adapters need gradients

#### 6. LoRA Adapters (LoRA/QLoRA Only)

```python
adapter_params = 2 × rank × hidden_dim × layers
adapter_memory_gb = (adapter_params × 2) / (1024³)  # fp16
```

**Example** (7B model, rank=64):
- Adapter params: 16,777,216
- Adapter memory: 0.0312 GB

#### 7. Framework Overhead (Inference Only)

| Framework | Overhead | Description |
|-----------|----------|-------------|
| vLLM      | 1.5 GB   | PagedAttention + continuous batching |

#### 8. Safety Buffer

The tool adds a **10% safety buffer** to all VRAM calculations to ensure reliability:
```python
total_vram_with_buffer = calculated_vram × 1.10
```

### Throughput Estimation

#### Base Throughput

Uses empirical benchmarks from:
- MLPerf Training v3.1 (November 2023)
- NVIDIA hardware specifications
- vLLM project benchmarks (Q4 2024)
- Real production deployments

**Interpolation**: Log-linear interpolation between benchmark points for model sizes not directly measured.

#### Quantization Speedup (Inference Only)

```python
QUANTIZATION_SPEEDUP = {
    "fp16": 1.0,     # Baseline
    "bf16": 1.0,     # Same as fp16
    "int8": 1.8,     # ~2x faster (INT8 Tensor Cores)
    "int4": 3.0,     # ~3-4x faster (INT4 Tensor Cores)
    "nf4": 3.0,      # Similar to int4
    "awq": 3.2,      # Optimized int4
    "gptq": 3.2,     # Optimized int4
}
```

**Note**: Speedup only applies to inference. Training does not benefit from quantization speedups.

#### Framework Efficiency

```python
FRAMEWORK_SPEEDUP = {
    "vllm": 1.0,           # Baseline (highly optimized)
    "huggingface": 0.7,    # ~30% slower
}
```

#### LoRA Training Speedup

LoRA/QLoRA train only adapter parameters (~0.1-1% of model), resulting in significant speedup:

**Speedup factors by model size and rank:**

| Model Size | Rank 8 | Rank 16 | Rank 32 | Rank 64 | Rank 128 | Rank 256 |
|------------|--------|---------|---------|---------|----------|----------|
| 7B         | 3.5x   | 3.2x    | 2.8x    | 2.4x    | 2.0x     | 1.6x     |
| 13B        | 3.2x   | 2.9x    | 2.5x    | 2.2x    | 1.8x     | 1.5x     |
| 70B        | 4.0x   | 3.6x    | 3.0x    | 2.5x    | 2.0x     | 1.6x     |
| 405B       | 4.5x   | 4.0x    | 3.4x    | 2.8x    | 2.2x     | 1.8x     |

**Why less than param ratio?**
1. Forward pass still processes full model
2. Adapter computations add latency
3. Quantization/dequantization overhead (QLoRA)

#### Batch Scaling

**Inference with batch ≥ 8:**
```python
batch_efficiency = (batch_size / 32) ** 0.7
tps_per_gpu *= batch_efficiency
```

**Inference with batch < 8:**
```python
batch_efficiency = min(1.0, (batch_size / 8) ** 0.6)
tps_per_gpu *= batch_efficiency * batch_size
```

**Training:**
```python
batch_efficiency = (batch_size / 8) ** 0.7
tps_per_gpu *= batch_efficiency
```

Diminishing returns model realistic throughput scaling.

#### Sequence Length Scaling

Longer sequences reduce throughput due to:
1. Increased KV cache memory bandwidth
2. O(n²) attention complexity
3. More memory pressure

```python
seq_factor = (2048 / seq_len) ** 0.15
tps_per_gpu *= seq_factor
```

Baseline: 2048 tokens, ~15% reduction per doubling.

#### Multi-GPU Communication Overhead

```python
if gpu_count <= 4:
    comm_efficiency = 0.90  # 10% overhead
elif gpu_count <= 8:
    comm_efficiency = 0.85  # 15% overhead
else:
    comm_efficiency = 0.75  # 25% overhead
```

Based on NCCL performance benchmarks.

#### Combined Throughput Formula

```python
final_throughput = (
    base_throughput
    × quant_speedup          # Inference only
    × framework_speedup
    × batch_efficiency
    × seq_len_factor
    × lora_speedup           # Training only, if LoRA/QLoRA
    × comm_efficiency        # Multi-GPU only
)
```

## Pricing Tiers (IndiaAI)

| Tier | Discount | Description |
|------|----------|-------------|
| On Demand | 0% | Pay-as-you-go hourly |
| 1 Month | ~10-12% | 1-month commitment |
| 6 Month | ~18-20% | 6-month commitment |
| 12 Month | ~23-30% | 12-month commitment |

Prices in Indian Rupees (INR) per hour.

## Example Workflows

### Example 1: Inference with Quantization

**Configuration:**
- Model: Llama 3.1 70B
- Task: Inference
- Quantization: int4
- Framework: vLLM
- Batch: 32
- Seq len: 2048

**Result:**
- VRAM: ~18 GB (vs ~140 GB for fp16)
- Throughput: ~270 tokens/sec (3x faster than fp16)
- Recommended: Nvidia L40S (1x) - Budget-friendly

### Example 2: QLoRA Training

**Configuration:**
- Model: Llama 3.1 8B
- Task: Training
- Quantization: nf4
- Method: QLoRA
- Rank: 64
- Batch: 16
- Seq len: 2048

**Result:**
- VRAM: ~104 GB
- Base model: 3.67 GB (stays nf4)
- Activations: 96 GB (bf16 compute)
- Adapters + Optimizer + Gradients: ~0.16 GB
- Throughput: ~4,176 tokens/sec (2.4x faster than Full FT)
- Recommended: Nvidia H100 SXM (2x)

### Example 3: Full Fine-Tuning

**Configuration:**
- Model: Llama 3.1 8B
- Task: Training
- Quantization: bf16 (auto-upgraded from nf4)
- Method: Full Fine-Tuning
- Batch: 16
- Seq len: 2048

**Result:**
- VRAM: ~187 GB
- Model: 13.04 GB (bf16 required)
- Optimizer: 52.15 GB (4x model)
- Gradients: 26.08 GB (2x model)
- Activations: 96 GB
- Throughput: ~1,740 tokens/sec (baseline)
- Recommended: Nvidia H100 SXM (4x)

## Development

### Project Structure

```
indiaai-gpu-infrastructure-recommender/
├── app.py              # Main application with Gradio UI
├── requirements.txt        # Python dependencies
└── README.md               # This file
```

### Key Components

**Core Calculation Functions:**
- `calculate_model_weights()`: Handles all quantization formats
- `calculate_kv_cache()`: GQA-aware KV cache with precision handling
- `calculate_activations()`: Megatron-LM formulas with checkpointing
- `calculate_optimizer_states()`: Full FT vs LoRA/QLoRA
- `calculate_gradients()`: Training-specific gradient memory
- `calculate_vram()`: Main integration function with precision override logic

**Throughput Functions:**
- `calculate_throughput()`: Applies all speedup/overhead factors
- `interpolate_throughput()`: Log-linear model size interpolation
- `get_lora_overhead_factor()`: Rank-dependent speedup calculation
- `get_gpu_family()`: Maps GPU names to benchmark families

**Hardware & Visualization:**
- `recommend_hardware()`: Finds optimal GPUs, generates chart data
- `get_manufacturer()`: Extracts vendor from GPU name
- `create_comparison_plot()`: Plotly bar charts with 4 views
- `format_vram_report()`: Detailed breakdown with formulas

**UI Functions:**
- `create_interface()`: Gradio interface definition
- `process_request()`: Main request handler
- `update_ui_on_task()`: Dynamic UI updates
- `update_rank_visibility()`: LoRA rank visibility control

### Adding New GPUs

1. Add entry to `GPU_DATABASE`:
```python
GPUConfig(
    name='Vendor ModelName (Nx)',
    vram=total_vram_gb,
    count=num_gpus,
    tflops=total_tflops,
    bandwidth=total_bandwidth_gbps,
    price_od=on_demand_price_inr,
    price_1m=one_month_price_inr,
    price_6m=six_month_price_inr,
    price_12m=twelve_month_price_inr
)
```

2. Add throughput benchmarks to `GPU_THROUGHPUT_BENCHMARKS`:
```python
'GPUFamily': {
    7: (single_inf_tps, batched_inf_tps, training_tps),
    13: (...),
    70: (...),
    405: (...),
}
```

3. Update `get_gpu_family()` if new GPU family.

## Contributing

**Contributions welcome!** Please:
1. Fork the repository
2. Create a feature branch
3. Add tests for new features
4. Update documentation
5. Submit a pull request


## Acknowledgements

Formulas validated against **20+ authoritative sources**:

### Academic Papers
- Vaswani et al. (2017) - "Attention Is All You Need"
- Hu et al. (2021) - "LoRA: Low-Rank Adaptation of Large Language Models"
- Dettmers et al. (2023) - "QLoRA: Efficient Finetuning of Quantized LLMs"
- Dettmers et al. (2022) - "LLM.int8(): 8-bit Matrix Multiplication"
- Rajbhandari et al. (2019) - "ZeRO: Memory Optimizations"
- Shoeybi et al. (2019) - "Megatron-LM: Training Multi-Billion Parameter Models"

### Official Documentation
- HuggingFace Transformers: Model Memory Anatomy
- PyTorch Automatic Mixed Precision (AMP)
- NVIDIA Apex: Mixed Precision Training
- vLLM: PagedAttention Documentation
- NVIDIA GPU Architecture Documentation (H100, H200, B200)
- NVIDIA TensorRT-LLM Documentation

### Benchmarks & Production Data
- MLPerf Training v3.1 (November 2023)
- MLPerf Inference v4.0 (2024)
- NVIDIA H100/H200/B200 Official Benchmarks
- Databricks: Serving Quantized LLMs
- Anyscale: Fine-Tuning LLMs with LoRA at Scale
- Axolotl: LoRA Training Benchmarks
- vLLM Community Benchmarks (2024)
- HuggingFace PEFT Library Documentation

### Validation Sources
- NVIDIA H100 Datasheet: INT8 2x, INT4 3-4x speedup
- NVIDIA TensorRT-LLM: AWQ/GPTQ speedup validation
- Databricks Production: FP8 2.2x improvement
- Community GPTQ/AWQ: 3-4x speedup confirmation
- QLoRA Paper: Empirical training times
- HuggingFace PEFT: LoRA rank performance
- Axolotl Logs: Real-world rank overhead

## Limitations & Disclaimers

### Accuracy Expectations

This tool provides **estimates for planning purposes**. Actual performance varies ±15-20% due to:

**Architecture Factors:**
- Specific model implementation (attention mechanisms, FFN design)
- Number of layers, heads, and head dimensions
- Presence of special tokens, embeddings

**Software Factors:**
- Framework version and optimizations
- CUDA/ROCm version
- Kernel efficiency and fusion
- Memory allocator behavior
- Python/C++ interface overhead

**Hardware Factors:**
- GPU batch/frequency
- PCIe vs NVLink bandwidth
- Thermal throttling
- Shared system resources

**Runtime Factors:**
- Memory fragmentation
- Concurrent workloads
- Input sequence distribution
- Actual vs average token counts

### Recommendations

1. **Add Safety Buffer**: Tool includes 10% buffer, but consider 15-20% for production
2. **Test Your Config**: Validate with your specific setup before deployment
3. **Monitor Actual Usage**: Track real VRAM and throughput in production
4. **Framework Optimizations**: Results may improve with framework updates
5. **Model-Specific Tuning**: Some models may have architecture-specific optimizations

### Conservative Estimates

The tool is intentionally conservative:
- Estimates are **80-90% of theoretical maximum**
- Throughput uses real-world benchmarks, not peak specs
- Safety buffers ensure reliability over performance

### Known Limitations

1. **Model Support**: Best accuracy for decoder-only transformers (GPT-style)
2. **Benchmark Coverage**: Limited data for some GPU/model combinations
3. **Framework Versions**: Benchmarks may not reflect latest optimizations
4. **Custom Architectures**: May require manual adjustment for exotic designs
5. **Multi-Node**: Currently models single-node multi-GPU only

---

**Made with ❤️ for the AI/ML community by CHCC@IIITD**

*Last updated: December 18, 2025*