File size: 3,731 Bytes
1f89cd9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | ---
license: bsd-3-clause
library_name: onnx
tags:
- image-classification
- resnet
- onnx
- tensorrt
- quantization
- inference-optimization
datasets:
- imagenet-1k
metrics:
- accuracy
pipeline_tag: image-classification
---
# ResNet-50 (ONNX) — inference-optimization benchmark
This is a standard torchvision ResNet-50 (`IMAGENET1K_V2` weights) exported to
ONNX, published as the portable artifact from an inference-optimization study.
The point of the repo is not the weights — they're the stock torchvision model —
but the benchmark: what FP16, ONNX Runtime, and TensorRT (FP16 / INT8) do to
latency and accuracy on the same hardware, measured the same way.
The `.onnx` file lets you reproduce the ONNX Runtime and TensorRT results, or
build your own engine, without re-exporting.
## Benchmark summary
NVIDIA GPU, CUDA 12.1, TensorRT 10.13. 200 timed iterations, 20 warm-up
discarded, `torch.cuda.synchronize()` around each timed region. Accuracy on a
fixed 3,200-image ImageNet-val slice, identical across variants.
Batch-1 latency (single-request):
| Variant | p50 (ms) | Speedup vs FP32 | Top-1 |
|---------------|---------:|----------------:|-------:|
| PyTorch FP32 | 6.87 | 1.0x | 85.25% |
| PyTorch FP16 | 7.92 | 0.87x | 85.25% |
| ONNX Runtime | 2.67 | 2.6x | — |
| TensorRT FP16 | 0.74 | 9.3x | 85.25% |
| TensorRT INT8 | 0.59 | 11.6x | 84.97% |
Batch-32 throughput:
| Variant | Throughput (img/s) | vs FP32 |
|---------------|-------------------:|--------:|
| PyTorch FP32 | 1068 | 1.0x |
| PyTorch FP16 | 2091 | 2.0x |
| ONNX Runtime | 975 | 0.9x |
| TensorRT FP16 | 5118 | 4.8x |
| TensorRT INT8 | 10021 | 9.4x |
Headline: TensorRT INT8 is 11.6x faster than FP32 at batch 1 for a 0.28-point
top-1 drop.
The accuracy figures are higher than the canonical ~80.3% because they're on an
easier 3,200-image slice; the relative gap between variants is the point, and
it's valid because every variant saw the same images.
## Intended use
Reference artifact for inference-optimization work: a fixed ResNet-50 ONNX graph
you can run in ONNX Runtime or compile with TensorRT to reproduce or extend the
numbers above. Not a new or improved model — the weights are stock torchvision.
## How to use
ONNX Runtime (CUDA):
```python
import onnxruntime as ort
import numpy as np
sess = ort.InferenceSession("resnet50.onnx", providers=["CUDAExecutionProvider"])
x = np.random.randn(1, 3, 224, 224).astype(np.float32) # NCHW, ImageNet-normalized
out = sess.run(None, {sess.get_inputs()[0].name: x})[0]
pred = out.argmax(1)
```
Preprocessing is standard ImageNet eval: resize 256, center-crop 224, normalize
with mean `[0.485, 0.456, 0.406]` and std `[0.229, 0.224, 0.225]`, in NCHW order.
The input has a dynamic batch axis, so any batch size works.
## Model details
- Architecture: ResNet-50
- Weights: torchvision `ResNet50_Weights.IMAGENET1K_V2`
- Input: `(batch, 3, 224, 224)` float32, ImageNet-normalized, NCHW
- Output: `(batch, 1000)` logits over ImageNet-1k classes
- ONNX opset: 17
## Limitations
Batch-1 p95 latency is noisy at sub-millisecond scale and shouldn't be read as a
tail-latency guarantee. Absolute accuracy is on a subset, not the full 50k val
set. INT8 numbers come from TensorRT 10.x implicit calibration (deprecated in
11.x, which uses explicit QDQ via NVIDIA ModelOpt).
Full code, benchmark scripts, and methodology: github.com/paulsaurav/resnet50-inference-optimization. |