File size: 5,185 Bytes
5badec5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | # MixVPR — ONNX & CoreML Export
Exported and quantized models for [MixVPR](https://github.com/Vincentqyw/MixVPR): Feature Mixing for Visual Place Recognition (WACV 2023).
## Models
| Format | File | Size | Latency (M-series) | CosSim | Status |
|---|---|---|---|---|---|
| ONNX FP32 | `onnx/mixvpr_fp32.onnx` | 41.7 MB | 32.4 ms | 1.0000 | ✓ |
| ONNX FP16 | `onnx/mixvpr_fp16.onnx` | 21.0 MB | 38.2 ms | 0.9999 | ✓ Recommended |
| CoreML FP16 | `coreml/mixvpr_fp16.mlpackage/` | 20.8 MB | **3.1 ms** | 0.9999 | ✓ Recommended |
| CoreML INT8 | `coreml/mixvpr_int8.mlpackage/` | 10.5 MB | **3.3 ms** | 0.9983 | ✓ |
## Benchmark
All models were benchmarked against the original PyTorch FP32 model on a MacBook with Apple Silicon (M-series). The primary accuracy metric is **cosine similarity** between the exported model's 4096-dim descriptor and the PyTorch reference.
### Latency (batch=1, averaged over 4 images)
| Model | Latency | Speedup |
|---|---|---|
| PyTorch FP32 (reference) | 44.5 ms | 1.0× |
| ONNX FP32 | 32.4 ms | 1.4× |
| ONNX FP16 | 38.2 ms | 1.2× |
| **CoreML FP16** | **3.1 ms** | **14.6×** |
| **CoreML INT8** | **3.3 ms** | **13.5×** |
### Accuracy (cosine similarity vs PyTorch)
| Model | CosSim | Max Abs Error | Verdict |
|---|---|---|---|
| ONNX FP32 | 1.0000 | 1.9×10⁻⁷ | No drop |
| ONNX FP16 | 0.9999 | 1.7×10⁻⁴ | No drop |
| CoreML FP16 | 0.9999 | 3.2×10⁻⁴ | No drop |
| CoreML INT8 | 0.9983 | 3.5×10⁻³ | Negligible |
- CosSim > 0.9999 means retrieval results are **identical** to PyTorch.
- CosSim 0.9983 means top-1 may shift for borderline cases; top-10 remains stable.
### File Size
| Model | Size | vs PyTorch (.pth) |
|---|---|---|
| PyTorch .pth | ~98 MB | — |
| ONNX FP32 | 41.7 MB | -57% |
| ONNX FP16 | 21.0 MB | -79% |
| CoreML FP16 | 20.8 MB | -79% |
| CoreML INT8 | 10.5 MB | -89% |
## Model Architecture
```
Input: (1, 3, 320, 320) normalized RGB image
│
▼
ResNet50 backbone (layer4 cropped) → (1, 1024, 20, 20)
│
▼
MixVPR aggregator:
- 4× FeatureMixerLayer (LayerNorm → Linear → ReLU → Linear, residual)
- Channel projection (Linear: 1024 → 1024)
- Row projection (Linear: 400 → 4)
- Flatten + L2 normalize → (1, 4096)
```
Parameters: **10.88 M**
## Usage
### Download
```bash
# All MixVPR models
huggingface-cli download Realcat/image_retrieval_checkpoints mixvpr/ --local-dir .
# Single file
huggingface-cli download Realcat/image_retrieval_checkpoints mixvpr/onnx/mixvpr_fp16.onnx
```
### Install Dependencies
```bash
pip install onnxruntime # for ONNX
pip install coremltools # for CoreML (macOS only)
pip install torch torchvision Pillow numpy # for preprocessing
```
### ONNX Inference
```python
import onnxruntime as ort
import numpy as np
from PIL import Image
import torchvision.transforms as tvf
sess = ort.InferenceSession("mixvpr/onnx/mixvpr_fp16.onnx",
providers=['CPUExecutionProvider'])
# Images must be resized to 320×320
preprocess = tvf.Compose([
tvf.Resize((320, 320), interpolation=tvf.InterpolationMode.BICUBIC),
tvf.ToTensor(),
tvf.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
def extract_descriptor(image_path):
img = Image.open(image_path).convert("RGB")
tensor = preprocess(img).unsqueeze(0).numpy().astype(np.float32)
desc = sess.run(None, {'images': tensor})[0]
return desc # (1, 4096), L2-normalized
# Compare two images via cosine similarity
d1 = extract_descriptor("query.jpg")
d2 = extract_descriptor("reference.jpg")
similarity = np.dot(d1.flatten(), d2.flatten())
```
### CoreML Inference (Apple Silicon, ~15× faster)
```python
import coremltools as ct
import numpy as np
from PIL import Image
import torchvision.transforms as tvf
mlmodel = ct.models.MLModel("mixvpr/coreml/mixvpr_fp16.mlpackage")
preprocess = tvf.Compose([
tvf.Resize((320, 320), interpolation=tvf.InterpolationMode.BICUBIC),
tvf.ToTensor(),
tvf.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
def extract_descriptor(image_path):
img = Image.open(image_path).convert("RGB")
tensor = preprocess(img).unsqueeze(0).numpy().astype(np.float32)
desc = mlmodel.predict({'images': tensor})['descriptor']
return desc # (1, 4096), L2-normalized, runs on ANE + GPU
```
## Notes
- **Input**: 320×320 RGB images, normalized with ImageNet stats.
- **Output**: 4096-dim L2-normalized global descriptor. Use cosine similarity for retrieval.
- **ONNX INT8/INT4**: Quantized models exist but the ONNX Runtime CPU EP lacks `ConvInteger` kernels for this architecture. Use a GPU EP (CUDA/TensorRT) or switch to CoreML for quantized inference.
- **Re-export**: Scripts available in the [source repo](https://github.com/Vincentqyw/MixVPR) (`export_quant_onnx.py`, `export_coreml.py`).
## Reference
```bibtex
@inproceedings{ali2023mixvpr,
title={{MixVPR}: Feature Mixing for Visual Place Recognition},
author={Ali-bey, Amar and Chaib-draa, Brahim and Gigu{\`e}re, Philippe},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={2998--3007},
year={2023}
}
```
|