One GPU Capacity: calibrated step-time and capacity model
This repository holds the calibrated parameters of the analytical model in How Many Users Can One GPU Serve? (Thanatchon Chongmankhong, November 2026). It holds no neural-network weights: the "model" is a token roofline for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks.
- 📖 Paper online: https://thanatchon36.github.io/one-gpu-capacity/paper/
- 💻 Code (simulator in C, Python reference): https://github.com/thanatchon36/one-gpu-capacity
- 📊 Dataset with every result:
datasets/thanatchon36/one-gpu-capacity
Fitted step-time parameters
Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)).
| engine | η_β (bandwidth eff.) | η_c (compute eff.) | t₀ (ms) | a (µs/seq) | rows | leave-one-GPU-out MAPE |
|---|---|---|---|---|---|---|
| vLLM | 0.77 | 0.61 | 2.2 | 81 | 188 | 10.4% |
| TensorRT-LLM | 0.78 | 0.56 | 1.4 | 19 | 184 | 8.7% |
Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM, GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap.
Validation
| layer | test | result |
|---|---|---|
| V1 | simulator vs exact queueing results | all identities within the 95% CI |
| V2 | leave-one-GPU-out batch latency | MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM) |
| V3 | consumer-card decode speed, fitted on datacenter cards only | Spearman ρ = 0.99 |
| V4 | 48 MLPerf Inference systems, after the calibration was frozen | 90% within 0.67–1.5× |
Files
calibration.json: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer)gpus.csv,models.csv: the hardware and architecture catalogues the model is evaluated on
Use
git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity
pip install -r requirements.txt
PYTHONPATH=src python -c "from ogc import capacity; help(capacity)"
Limits
No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper.
Citation
@misc{chongmankhong2026onegpu,
title = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators},
author = {Chongmankhong, Thanatchon},
year = {2026},
note = {Preprint},
doi = {10.5281/zenodo.24243858},
url = {https://github.com/thanatchon36/one-gpu-capacity}
}