README.md

---
license: bsd-3-clause
library_name: onnx
tags:
  - image-classification
  - resnet
  - onnx
  - tensorrt
  - quantization
  - inference-optimization
datasets:
  - imagenet-1k
metrics:
  - accuracy
pipeline_tag: image-classification
---

# ResNet-50 (ONNX) — inference-optimization benchmark

This is a standard torchvision ResNet-50 (`IMAGENET1K_V2` weights) exported to
ONNX, published as the portable artifact from an inference-optimization study.
The point of the repo is not the weights — they're the stock torchvision model —
but the benchmark: what FP16, ONNX Runtime, and TensorRT (FP16 / INT8) do to
latency and accuracy on the same hardware, measured the same way.

The `.onnx` file lets you reproduce the ONNX Runtime and TensorRT results, or
build your own engine, without re-exporting.

## Benchmark summary

NVIDIA GPU, CUDA 12.1, TensorRT 10.13. 200 timed iterations, 20 warm-up
discarded, `torch.cuda.synchronize()` around each timed region. Accuracy on a
fixed 3,200-image ImageNet-val slice, identical across variants.

Batch-1 latency (single-request):

| Variant       | p50 (ms) | Speedup vs FP32 | Top-1  |
|---------------|---------:|----------------:|-------:|
| PyTorch FP32  |     6.87 |            1.0x  | 85.25% |
| PyTorch FP16  |     7.92 |            0.87x | 85.25% |
| ONNX Runtime  |     2.67 |            2.6x  | —      |
| TensorRT FP16 |     0.74 |            9.3x  | 85.25% |
| TensorRT INT8 |     0.59 |           11.6x  | 84.97% |

Batch-32 throughput:

| Variant       | Throughput (img/s) | vs FP32 |
|---------------|-------------------:|--------:|
| PyTorch FP32  |               1068 |   1.0x  |
| PyTorch FP16  |               2091 |   2.0x  |
| ONNX Runtime  |                975 |   0.9x  |
| TensorRT FP16 |               5118 |   4.8x  |
| TensorRT INT8 |              10021 |   9.4x  |

Headline: TensorRT INT8 is 11.6x faster than FP32 at batch 1 for a 0.28-point
top-1 drop.

The accuracy figures are higher than the canonical ~80.3% because they're on an
easier 3,200-image slice; the relative gap between variants is the point, and
it's valid because every variant saw the same images.

## Intended use

Reference artifact for inference-optimization work: a fixed ResNet-50 ONNX graph
you can run in ONNX Runtime or compile with TensorRT to reproduce or extend the
numbers above. Not a new or improved model — the weights are stock torchvision.

## How to use

ONNX Runtime (CUDA):

```python
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("resnet50.onnx", providers=["CUDAExecutionProvider"])
x = np.random.randn(1, 3, 224, 224).astype(np.float32)  # NCHW, ImageNet-normalized
out = sess.run(None, {sess.get_inputs()[0].name: x})[0]
pred = out.argmax(1)
```

Preprocessing is standard ImageNet eval: resize 256, center-crop 224, normalize
with mean `[0.485, 0.456, 0.406]` and std `[0.229, 0.224, 0.225]`, in NCHW order.

The input has a dynamic batch axis, so any batch size works.

## Model details

- Architecture: ResNet-50
- Weights: torchvision `ResNet50_Weights.IMAGENET1K_V2`
- Input: `(batch, 3, 224, 224)` float32, ImageNet-normalized, NCHW
- Output: `(batch, 1000)` logits over ImageNet-1k classes
- ONNX opset: 17

## Limitations

Batch-1 p95 latency is noisy at sub-millisecond scale and shouldn't be read as a
tail-latency guarantee. Absolute accuracy is on a subset, not the full 50k val
set. INT8 numbers come from TensorRT 10.x implicit calibration (deprecated in
11.x, which uses explicit QDQ via NVIDIA ModelOpt).

Full code, benchmark scripts, and methodology: github.com/paulsaurav/resnet50-inference-optimization.