Hugging Face's logo

Spaces:
CHCC-PSU
/
IndiaAI-GPU-Reco
Sleeping

App Files Community
IndiaAI-GPU-Reco / README.md
rhui96's picture
rhui96
Update README.md
4b0704d verified
preview code
|
Raw
History Blame Contribute Delete
21.1 kB

A newer version of the Gradio SDK is available: 6.26.0

Upgrade
metadata 
license: mit
title: IndiaAI GPU Infrastructure Recommender for AI Models
sdk: gradio
emoji: 🚀
colorFrom: purple
colorTo: gray
short_description: Calculate VRAM requirements +get optimal GPU recommendations
pinned: false
thumbnail: >-
  https://cdn-uploads.huggingface.co/production/uploads/69312fe079dac228ed39fb04/kZDf4ccek1FcfpojnV_yJ.webp

IndiaAI GPU Infrastructure Recommender for AI Models

A comprehensive tool for estimating VRAM requirements and recommending optimal GPU configurations for Large Language Model (LLM) deployment and training using IndiaAI's price list.

Overview

This recommender provides accurate VRAM estimates and GPU recommendations for:

  • Inference: Single-stream and batched inference workloads with quantization speedups
  • Training: Full fine-tuning, LoRA, and QLoRA with accurate precision handling
  • Multiple Quantization: fp16, bf16, int8, int4, nf4, AWQ, GPTQ with realistic throughput
  • Framework Support: vLLM, HuggingFace Transformers with framework-specific optimizations

Key Features

  • Accurate VRAM Estimation: Based on empirically-validated formulas from academic papers and production deployments
  • Quantization-Aware Throughput: INT4 is 3x faster, INT8 is 1.8x faster than FP16 baseline
  • Framework-Specific Performance: HuggingFace 30% slower than vLLM baseline
  • Precision Override Logic: Automatically upgrades to bf16 for Full FT and LoRA (cannot train quantized weights)
  • LoRA Rank Impact: Accounts for 2-4.5x throughput improvement based on rank size and model size
  • Sequence Length Scaling: Realistic throughput reduction for longer contexts
  • Optimized Batch Scaling: Model-size-dependent batch optimization for accurate time estimates
  • Cost Analysis: Compare costs across different GPU configurations with accurate time estimates
  • Multiple Pricing Tiers: On-demand, 1-month, 6-month, and 12-month reserved pricing
  • Comprehensive GPU Database: 55+ configurations from 1x to 8x GPUs across all major vendors
  • GPU Manufacturer Filtering: Filter recommendations by Nvidia, AMD, or Intel
  • Interactive Comparison Charts: Visual comparison of GPUs with 3 different views
  • CSV Export: Download detailed analysis with all formulas and calculations
  • Production-Ready: ±15-20% accuracy validated against real deployments

Interactive Visualization

The tool provides interactive comparison charts (Top 10 GPUs by cost):

  1. Throughput: Tokens/sec comparison - find fastest processing
  2. Cost: ₹/hour pricing - compare rental costs
  3. VRAM Utilization: % of GPU memory used - identify over-provisioned GPUs

Quick Start

Installation 

# Clone the repository
git clone https://github.com/CoE-HCC/indiaai-gpu-infrastructure-recommender.git
cd indiaai-gpu-infrastructure-recommender

# Install dependencies
pip install -r requirements.txt

HuggingFace Token (for automatic model config resolution)

For gated models that require authentication or automatic config resolution:

# Set your HuggingFace token
export HF_TOKEN="your_token_here"

Note: The tool works without this token using fallback estimation.

Basic Usage 

# Run the application
python app.py

Access the Web Interface 

http://localhost:7860

The interface will open in your default browser.

User Interface

Input Configuration

Model Selection:

  • Pre-configured popular models from HuggingFace Hub
  • Or enter custom size (e.g., "7B", "70B", "405B")
  • Or enter HuggingFace model ID (e.g., "meta-llama/Llama-3.1-70B-Instruct")

Task Selection:

  • Inference: For model serving and generation
  • Training: For fine-tuning and training workloads

Precision/Quantization:

  • fp16/bf16: 2 bytes/param (baseline)
  • int8: 1 byte/param (1.8x faster inference)
  • int4/nf4: 0.5 bytes/param (3.0x faster inference)
  • AWQ/GPTQ: 0.52 bytes/param (3.2x faster inference)

Training Options (when Task=Training):

  • Full Fine-Tuning: Train all parameters (requires bf16)
  • LoRA: Parameter-efficient training (requires bf16 base)
  • QLoRA: Memory-efficient training (quantized base + fp16 adapters)
  • LoRA Rank: 8, 16, 32, 64, 128, 256 (affects memory and speed)

Workload Parameters:

  • Max Context Length: Sequence length for KV cache sizing
  • Batch Size: Samples processed in parallel
  • Samples: Total number of samples to process
  • Input/Output Tokens: Average token counts per sample

Pricing & Filtering:

  • Pricing Tier: On Demand, 1/6/12 Month Reserved
  • GPU Manufacturers: Nvidia, AMD, Intel (multi-select)

Recommendations Panel

Two recommendation cards are displayed:

  • 🥇 Best Budget: Lowest cost GPU meeting requirements
  • 🥈 Budget Runner-up: Second most affordable option

Each card shows:

  • GPU configuration name and count
  • Total VRAM and per-GPU VRAM
  • VRAM utilization percentage
  • Performance (TFLOPS) and bandwidth
  • Estimated throughput (tokens/sec)
  • Time estimate for your workload
  • Hourly, daily, and monthly pricing

GPU Comparison Charts

Expandable section with interactive Plotly charts comparing Top 10 GPUs by cost:

Chart Y-Axis Description Use Case
Throughput Tokens/sec Processing speed Find fastest GPUs
Cost ₹/hour Hourly pricing Compare costs
VRAM Utilization % Used Memory efficiency Avoid over-provisioning

CSV Export

Download button provides comprehensive analysis including:

  • All input parameters
  • Derived model architecture
  • Precision parameters
  • VRAM calculation breakdown with formulas
  • GPU recommendations (all viable options)
  • Throughput calculations
  • Cost projections

Supported Models

Pre-configured Models

Llama 3 Family:

  • Llama 3.3: 70B
  • Llama 3.1: 8B, 70B, 405B
  • Llama 3.2: 1B, 3B

Qwen 2.5 Family:

  • Qwen 2.5: 1.5B, 3B, 7B, 14B, 32B, 72B
  • Qwen 2.5 Coder: 32B

Mistral Family:

  • Mistral: 7B, Small, Large, Nemo
  • Mixtral: 8x22B
  • Ministral: 8B

Or enter any:

  • Model size: "7B", "13B", "70B", etc.
  • HuggingFace ID: "organization/model-name"

GPU Database

Comprehensive Hardware Catalog

55+ GPU configurations across all major vendors:

NVIDIA H-Series (Hopper):

  • H100 SXM: 80GB (1x, 2x, 4x, 8x)
  • H100 NVL: 94GB (1x, 2x, 4x, 8x)
  • H100 PCIe: 80GB (1x, 8x)
  • H200 SXM: 141GB (1x, 2x, 4x, 8x)
  • H200 NVL: 141GB (1x, 2x, 4x, 8x)
  • H200 PCIe: 141GB (8x)

NVIDIA B-Series (Blackwell):

  • B200 SXM: 180GB (1x, 2x, 4x, 8x)

NVIDIA A-Series (Ampere):

  • A100 40GB: (1x, 2x, 4x, 8x)
  • A100 80GB: (1x, 2x, 4x, 8x)

NVIDIA L-Series (Ada):

  • L40S: 48GB (1x, 2x, 4x, 8x)
  • L4: 24GB (1x, 2x, 4x, 8x)

AMD Instinct:

  • MI300X: 192GB (1x, 2x, 4x, 8x)
  • MI325X: 256GB (1x, 2x, 4x, 8x)

Intel Gaudi:

  • Gaudi 2: 96GB (1x, 2x, 4x, 8x)
  • Gaudi 3: 128GB (1x, 2x, 4x, 8x)

Manufacturer Filtering

Filter recommendations by vendor preference:

  • Nvidia: Industry standard, best software ecosystem
  • AMD: Competitive performance, often better value
  • Intel: Gaudi accelerators for specific workloads

Default: All manufacturers selected

Technical Details

VRAM Calculation Components

Total VRAM is calculated as:

Total VRAM = Model Weights + KV Cache + [Dynamic Components] + Safety Buffer

Where dynamic components depend on the task:

  • Inference: Framework Overhead
  • Training: Activations + Optimizer States + Gradients + [LoRA Adapters]

1. Model Weights

Formula:

memory_gb = (num_parameters × bytes_per_parameter) / (1024³)

Quantization formats:

Format Bytes/Param Use Case
fp16/bf16 2.0 Standard training/inference
int8 1.0 4x compression, 1.8x faster
nf4 0.5625 QLoRA's format, 3x faster
int4 0.50 8x compression, 3x faster
awq/gptq 0.52 Optimized int4, 3.2x faster

Important: For training with LoRA or Full FT, the code automatically overrides quantized formats to bf16 because you cannot train quantized weights. Only QLoRA keeps the base model quantized.

2. KV Cache (Inference & Training)

Formula:

kv_memory = 2 × layers × batch × seq_len × kv_heads × head_dim × bytes_per_elem / (1024³)

Components:

  • 2: Separate tensors for Keys and Values
  • layers: Number of transformer layers
  • batch: Batch size
  • seq_len: Sequence length
  • kv_heads: Number of KV heads (for GQA/MQA)
  • head_dim: Dimension per attention head
  • bytes_per_elem: Precision in bytes

GQA Support:

  • Standard MHA: kv_heads = heads
  • Grouped Query Attention (GQA): kv_heads < heads
  • Example: Llama 3 uses 32 heads but only 8 kv_heads (4x memory reduction)

Precision Handling:

  • KV cache kept at fp16/bf16 even for quantized models (maintains quality)
  • For training, always uses bf16 compute precision

3. Activations (Training Only)

Formula:

activation_memory = batch × seq × hidden × layers × multiplier × bytes / (1024³)

Multiplier:

  • With gradient checkpointing: 12x (default in most frameworks)
  • Without checkpointing: 34x (stores all intermediate activations)

Precision: Always at compute precision (bf16), even for QLoRA.

4. Optimizer States (Training Only)

Full Fine-Tuning (Adam):

optimizer_memory = model_weights_gb × 4
  • Model weights: fp16 (2 bytes/param)
  • Optimizer states: 2 states × fp32 (8 bytes/param total)
  • Ratio: 8/2 = 4x model weights

LoRA/QLoRA (Adam for adapters only):

adapter_params = 2 × rank × hidden_dim × layers
adapter_params_gb = (adapter_params × 2) / (1024³)  # fp16
optimizer_memory = adapter_params_gb × 4  # 2 states at fp32

Example (7B model, rank=64):

  • Full FT: 52.15 GB
  • LoRA: 0.125 GB
  • 417x smaller!

5. Gradients (Training Only)

Full Fine-Tuning:

gradients_memory = model_weights_gb × 2
  • fp32 gradients for numerical stability
  • Ratio: 4/2 = 2x model weights

LoRA/QLoRA:

gradients_memory = (adapter_params × 2) / (1024³)  # fp16
  • Only adapters need gradients

6. LoRA Adapters (LoRA/QLoRA Only) 

adapter_params = 2 × rank × hidden_dim × layers
adapter_memory_gb = (adapter_params × 2) / (1024³)  # fp16

Example (7B model, rank=64):

  • Adapter params: 16,777,216
  • Adapter memory: 0.0312 GB

7. Framework Overhead (Inference Only)

Framework Overhead Description
vLLM 1.5 GB PagedAttention + continuous batching

8. Safety Buffer

The tool adds a 10% safety buffer to all VRAM calculations to ensure reliability:

total_vram_with_buffer = calculated_vram × 1.10

Throughput Estimation

Base Throughput

Uses empirical benchmarks from:

  • MLPerf Training v3.1 (November 2023)
  • NVIDIA hardware specifications
  • vLLM project benchmarks (Q4 2024)
  • Real production deployments

Interpolation: Log-linear interpolation between benchmark points for model sizes not directly measured.

Quantization Speedup (Inference Only) 

QUANTIZATION_SPEEDUP = {
    "fp16": 1.0,     # Baseline
    "bf16": 1.0,     # Same as fp16
    "int8": 1.8,     # ~2x faster (INT8 Tensor Cores)
    "int4": 3.0,     # ~3-4x faster (INT4 Tensor Cores)
    "nf4": 3.0,      # Similar to int4
    "awq": 3.2,      # Optimized int4
    "gptq": 3.2,     # Optimized int4
}

Note: Speedup only applies to inference. Training does not benefit from quantization speedups.

Framework Efficiency 

FRAMEWORK_SPEEDUP = {
    "vllm": 1.0,           # Baseline (highly optimized)
    "huggingface": 0.7,    # ~30% slower
}

LoRA Training Speedup

LoRA/QLoRA train only adapter parameters (~0.1-1% of model), resulting in significant speedup:

Speedup factors by model size and rank:

Model Size Rank 8 Rank 16 Rank 32 Rank 64 Rank 128 Rank 256
7B 3.5x 3.2x 2.8x 2.4x 2.0x 1.6x
13B 3.2x 2.9x 2.5x 2.2x 1.8x 1.5x
70B 4.0x 3.6x 3.0x 2.5x 2.0x 1.6x
405B 4.5x 4.0x 3.4x 2.8x 2.2x 1.8x

Why less than param ratio?

  1. Forward pass still processes full model
  2. Adapter computations add latency
  3. Quantization/dequantization overhead (QLoRA)

Batch Scaling

Inference with batch ≥ 8:

batch_efficiency = (batch_size / 32) ** 0.7
tps_per_gpu *= batch_efficiency

Inference with batch < 8:

batch_efficiency = min(1.0, (batch_size / 8) ** 0.6)
tps_per_gpu *= batch_efficiency * batch_size

Training:

batch_efficiency = (batch_size / 8) ** 0.7
tps_per_gpu *= batch_efficiency

Diminishing returns model realistic throughput scaling.

Sequence Length Scaling

Longer sequences reduce throughput due to:

  1. Increased KV cache memory bandwidth
  2. O(n²) attention complexity
  3. More memory pressure
seq_factor = (2048 / seq_len) ** 0.15
tps_per_gpu *= seq_factor

Baseline: 2048 tokens, ~15% reduction per doubling.

Multi-GPU Communication Overhead 

if gpu_count <= 4:
    comm_efficiency = 0.90  # 10% overhead
elif gpu_count <= 8:
    comm_efficiency = 0.85  # 15% overhead
else:
    comm_efficiency = 0.75  # 25% overhead

Based on NCCL performance benchmarks.

Combined Throughput Formula 

final_throughput = (
    base_throughput
    × quant_speedup          # Inference only
    × framework_speedup
    × batch_efficiency
    × seq_len_factor
    × lora_speedup           # Training only, if LoRA/QLoRA
    × comm_efficiency        # Multi-GPU only
)

Pricing Tiers (IndiaAI)

Tier Discount Description
On Demand 0% Pay-as-you-go hourly
1 Month ~10-12% 1-month commitment
6 Month ~18-20% 6-month commitment
12 Month ~23-30% 12-month commitment

Prices in Indian Rupees (INR) per hour.

Example Workflows

Example 1: Inference with Quantization

Configuration:

  • Model: Llama 3.1 70B
  • Task: Inference
  • Quantization: int4
  • Framework: vLLM
  • Batch: 32
  • Seq len: 2048

Result:

  • VRAM: ~18 GB (vs ~140 GB for fp16)
  • Throughput: ~270 tokens/sec (3x faster than fp16)
  • Recommended: Nvidia L40S (1x) - Budget-friendly

Example 2: QLoRA Training

Configuration:

  • Model: Llama 3.1 8B
  • Task: Training
  • Quantization: nf4
  • Method: QLoRA
  • Rank: 64
  • Batch: 16
  • Seq len: 2048

Result:

  • VRAM: ~104 GB
  • Base model: 3.67 GB (stays nf4)
  • Activations: 96 GB (bf16 compute)
  • Adapters + Optimizer + Gradients: ~0.16 GB
  • Throughput: ~4,176 tokens/sec (2.4x faster than Full FT)
  • Recommended: Nvidia H100 SXM (2x)

Example 3: Full Fine-Tuning

Configuration:

  • Model: Llama 3.1 8B
  • Task: Training
  • Quantization: bf16 (auto-upgraded from nf4)
  • Method: Full Fine-Tuning
  • Batch: 16
  • Seq len: 2048

Result:

  • VRAM: ~187 GB
  • Model: 13.04 GB (bf16 required)
  • Optimizer: 52.15 GB (4x model)
  • Gradients: 26.08 GB (2x model)
  • Activations: 96 GB
  • Throughput: ~1,740 tokens/sec (baseline)
  • Recommended: Nvidia H100 SXM (4x)

Development

Project Structure 

indiaai-gpu-infrastructure-recommender/
├── app.py              # Main application with Gradio UI
├── requirements.txt        # Python dependencies
└── README.md               # This file

Key Components

Core Calculation Functions:

  • calculate_model_weights(): Handles all quantization formats
  • calculate_kv_cache(): GQA-aware KV cache with precision handling
  • calculate_activations(): Megatron-LM formulas with checkpointing
  • calculate_optimizer_states(): Full FT vs LoRA/QLoRA
  • calculate_gradients(): Training-specific gradient memory
  • calculate_vram(): Main integration function with precision override logic

Throughput Functions:

  • calculate_throughput(): Applies all speedup/overhead factors
  • interpolate_throughput(): Log-linear model size interpolation
  • get_lora_overhead_factor(): Rank-dependent speedup calculation
  • get_gpu_family(): Maps GPU names to benchmark families

Hardware & Visualization:

  • recommend_hardware(): Finds optimal GPUs, generates chart data
  • get_manufacturer(): Extracts vendor from GPU name
  • create_comparison_plot(): Plotly bar charts with 4 views
  • format_vram_report(): Detailed breakdown with formulas

UI Functions:

  • create_interface(): Gradio interface definition
  • process_request(): Main request handler
  • update_ui_on_task(): Dynamic UI updates
  • update_rank_visibility(): LoRA rank visibility control

Adding New GPUs

  1. Add entry to GPU_DATABASE:
GPUConfig(
    name='Vendor ModelName (Nx)',
    vram=total_vram_gb,
    count=num_gpus,
    tflops=total_tflops,
    bandwidth=total_bandwidth_gbps,
    price_od=on_demand_price_inr,
    price_1m=one_month_price_inr,
    price_6m=six_month_price_inr,
    price_12m=twelve_month_price_inr
)
  1. Add throughput benchmarks to GPU_THROUGHPUT_BENCHMARKS:
'GPUFamily': {
    7: (single_inf_tps, batched_inf_tps, training_tps),
    13: (...),
    70: (...),
    405: (...),
}
  1. Update get_gpu_family() if new GPU family.

Contributing

Contributions welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Add tests for new features
  4. Update documentation
  5. Submit a pull request

Acknowledgements

Formulas validated against 20+ authoritative sources:

Academic Papers

  • Vaswani et al. (2017) - "Attention Is All You Need"
  • Hu et al. (2021) - "LoRA: Low-Rank Adaptation of Large Language Models"
  • Dettmers et al. (2023) - "QLoRA: Efficient Finetuning of Quantized LLMs"
  • Dettmers et al. (2022) - "LLM.int8(): 8-bit Matrix Multiplication"
  • Rajbhandari et al. (2019) - "ZeRO: Memory Optimizations"
  • Shoeybi et al. (2019) - "Megatron-LM: Training Multi-Billion Parameter Models"

Official Documentation

  • HuggingFace Transformers: Model Memory Anatomy
  • PyTorch Automatic Mixed Precision (AMP)
  • NVIDIA Apex: Mixed Precision Training
  • vLLM: PagedAttention Documentation
  • NVIDIA GPU Architecture Documentation (H100, H200, B200)
  • NVIDIA TensorRT-LLM Documentation

Benchmarks & Production Data

  • MLPerf Training v3.1 (November 2023)
  • MLPerf Inference v4.0 (2024)
  • NVIDIA H100/H200/B200 Official Benchmarks
  • Databricks: Serving Quantized LLMs
  • Anyscale: Fine-Tuning LLMs with LoRA at Scale
  • Axolotl: LoRA Training Benchmarks
  • vLLM Community Benchmarks (2024)
  • HuggingFace PEFT Library Documentation

Validation Sources

  • NVIDIA H100 Datasheet: INT8 2x, INT4 3-4x speedup
  • NVIDIA TensorRT-LLM: AWQ/GPTQ speedup validation
  • Databricks Production: FP8 2.2x improvement
  • Community GPTQ/AWQ: 3-4x speedup confirmation
  • QLoRA Paper: Empirical training times
  • HuggingFace PEFT: LoRA rank performance
  • Axolotl Logs: Real-world rank overhead

Limitations & Disclaimers

Accuracy Expectations

This tool provides estimates for planning purposes. Actual performance varies ±15-20% due to:

Architecture Factors:

  • Specific model implementation (attention mechanisms, FFN design)
  • Number of layers, heads, and head dimensions
  • Presence of special tokens, embeddings

Software Factors:

  • Framework version and optimizations
  • CUDA/ROCm version
  • Kernel efficiency and fusion
  • Memory allocator behavior
  • Python/C++ interface overhead

Hardware Factors:

  • GPU batch/frequency
  • PCIe vs NVLink bandwidth
  • Thermal throttling
  • Shared system resources

Runtime Factors:

  • Memory fragmentation
  • Concurrent workloads
  • Input sequence distribution
  • Actual vs average token counts

Recommendations

  1. Add Safety Buffer: Tool includes 10% buffer, but consider 15-20% for production
  2. Test Your Config: Validate with your specific setup before deployment
  3. Monitor Actual Usage: Track real VRAM and throughput in production
  4. Framework Optimizations: Results may improve with framework updates
  5. Model-Specific Tuning: Some models may have architecture-specific optimizations

Conservative Estimates

The tool is intentionally conservative:

  • Estimates are 80-90% of theoretical maximum
  • Throughput uses real-world benchmarks, not peak specs
  • Safety buffers ensure reliability over performance

Known Limitations

  1. Model Support: Best accuracy for decoder-only transformers (GPT-style)
  2. Benchmark Coverage: Limited data for some GPU/model combinations
  3. Framework Versions: Benchmarks may not reflect latest optimizations
  4. Custom Architectures: May require manual adjustment for exotic designs
  5. Multi-Node: Currently models single-node multi-GPU only

Made with ❤️ for the AI/ML community by CHCC@IIITD

Last updated: December 18, 2025