One GPU Capacity: calibrated step-time and capacity model

This repository holds the calibrated parameters of the analytical model in How Many Users Can One GPU Serve? (Thanatchon Chongmankhong, November 2026). It holds no neural-network weights: the "model" is a token roofline for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks.

Fitted step-time parameters

Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)).

engine η_β (bandwidth eff.) η_c (compute eff.) t₀ (ms) a (µs/seq) rows leave-one-GPU-out MAPE
vLLM 0.77 0.61 2.2 81 188 10.4%
TensorRT-LLM 0.78 0.56 1.4 19 184 8.7%

Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM, GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap.

Validation

layer test result
V1 simulator vs exact queueing results all identities within the 95% CI
V2 leave-one-GPU-out batch latency MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM)
V3 consumer-card decode speed, fitted on datacenter cards only Spearman ρ = 0.99
V4 48 MLPerf Inference systems, after the calibration was frozen 90% within 0.67–1.5×

Files

  • calibration.json: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer)
  • gpus.csv, models.csv: the hardware and architecture catalogues the model is evaluated on

Use

git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity
pip install -r requirements.txt
PYTHONPATH=src python -c "from ogc import capacity; help(capacity)"

Limits

No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper.

Citation

@misc{chongmankhong2026onegpu,
  title  = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators},
  author = {Chongmankhong, Thanatchon},
  year   = {2026},
  note   = {Preprint},
  doi    = {10.5281/zenodo.24243858},
  url    = {https://github.com/thanatchon36/one-gpu-capacity}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train thanatchon36/one-gpu-capacity