mixvpr/README.md

# MixVPR — ONNX & CoreML Export

Exported and quantized models for [MixVPR](https://github.com/Vincentqyw/MixVPR): Feature Mixing for Visual Place Recognition (WACV 2023).

## Models

| Format | File | Size | Latency (M-series) | CosSim | Status |
|---|---|---|---|---|---|
| ONNX FP32 | `onnx/mixvpr_fp32.onnx` | 41.7 MB | 32.4 ms | 1.0000 | ✓ |
| ONNX FP16 | `onnx/mixvpr_fp16.onnx` | 21.0 MB | 38.2 ms | 0.9999 | ✓ Recommended |
| CoreML FP16 | `coreml/mixvpr_fp16.mlpackage/` | 20.8 MB | **3.1 ms** | 0.9999 | ✓ Recommended |
| CoreML INT8 | `coreml/mixvpr_int8.mlpackage/` | 10.5 MB | **3.3 ms** | 0.9983 | ✓ |

## Benchmark

All models were benchmarked against the original PyTorch FP32 model on a MacBook with Apple Silicon (M-series). The primary accuracy metric is **cosine similarity** between the exported model's 4096-dim descriptor and the PyTorch reference.

### Latency (batch=1, averaged over 4 images)

| Model | Latency | Speedup |
|---|---|---|
| PyTorch FP32 (reference) | 44.5 ms | 1.0× |
| ONNX FP32 | 32.4 ms | 1.4× |
| ONNX FP16 | 38.2 ms | 1.2× |
| **CoreML FP16** | **3.1 ms** | **14.6×** |
| **CoreML INT8** | **3.3 ms** | **13.5×** |

### Accuracy (cosine similarity vs PyTorch)

| Model | CosSim | Max Abs Error | Verdict |
|---|---|---|---|
| ONNX FP32 | 1.0000 | 1.9×10⁻⁷ | No drop |
| ONNX FP16 | 0.9999 | 1.7×10⁻⁴ | No drop |
| CoreML FP16 | 0.9999 | 3.2×10⁻⁴ | No drop |
| CoreML INT8 | 0.9983 | 3.5×10⁻³ | Negligible |

- CosSim > 0.9999 means retrieval results are **identical** to PyTorch.
- CosSim 0.9983 means top-1 may shift for borderline cases; top-10 remains stable.

### File Size

| Model | Size | vs PyTorch (.pth) |
|---|---|---|
| PyTorch .pth | ~98 MB | — |
| ONNX FP32 | 41.7 MB | -57% |
| ONNX FP16 | 21.0 MB | -79% |
| CoreML FP16 | 20.8 MB | -79% |
| CoreML INT8 | 10.5 MB | -89% |

## Model Architecture

```
Input: (1, 3, 320, 320) normalized RGB image
  │
  ▼
ResNet50 backbone (layer4 cropped) → (1, 1024, 20, 20)
  │
  ▼
MixVPR aggregator:
  - 4× FeatureMixerLayer (LayerNorm → Linear → ReLU → Linear, residual)
  - Channel projection (Linear: 1024 → 1024)
  - Row projection (Linear: 400 → 4)
  - Flatten + L2 normalize → (1, 4096)
```

Parameters: **10.88 M**

## Usage

### Download

```bash
# All MixVPR models
huggingface-cli download Realcat/image_retrieval_checkpoints mixvpr/ --local-dir .

# Single file
huggingface-cli download Realcat/image_retrieval_checkpoints mixvpr/onnx/mixvpr_fp16.onnx
```

### Install Dependencies

```bash
pip install onnxruntime  # for ONNX
pip install coremltools  # for CoreML (macOS only)
pip install torch torchvision Pillow numpy  # for preprocessing
```

### ONNX Inference

```python
import onnxruntime as ort
import numpy as np
from PIL import Image
import torchvision.transforms as tvf

sess = ort.InferenceSession("mixvpr/onnx/mixvpr_fp16.onnx",
                            providers=['CPUExecutionProvider'])

# Images must be resized to 320×320
preprocess = tvf.Compose([
    tvf.Resize((320, 320), interpolation=tvf.InterpolationMode.BICUBIC),
    tvf.ToTensor(),
    tvf.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def extract_descriptor(image_path):
    img = Image.open(image_path).convert("RGB")
    tensor = preprocess(img).unsqueeze(0).numpy().astype(np.float32)
    desc = sess.run(None, {'images': tensor})[0]
    return desc  # (1, 4096), L2-normalized

# Compare two images via cosine similarity
d1 = extract_descriptor("query.jpg")
d2 = extract_descriptor("reference.jpg")
similarity = np.dot(d1.flatten(), d2.flatten())
```

### CoreML Inference (Apple Silicon, ~15× faster)

```python
import coremltools as ct
import numpy as np
from PIL import Image
import torchvision.transforms as tvf

mlmodel = ct.models.MLModel("mixvpr/coreml/mixvpr_fp16.mlpackage")

preprocess = tvf.Compose([
    tvf.Resize((320, 320), interpolation=tvf.InterpolationMode.BICUBIC),
    tvf.ToTensor(),
    tvf.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def extract_descriptor(image_path):
    img = Image.open(image_path).convert("RGB")
    tensor = preprocess(img).unsqueeze(0).numpy().astype(np.float32)
    desc = mlmodel.predict({'images': tensor})['descriptor']
    return desc  # (1, 4096), L2-normalized, runs on ANE + GPU
```

## Notes

- **Input**: 320×320 RGB images, normalized with ImageNet stats.
- **Output**: 4096-dim L2-normalized global descriptor. Use cosine similarity for retrieval.
- **ONNX INT8/INT4**: Quantized models exist but the ONNX Runtime CPU EP lacks `ConvInteger` kernels for this architecture. Use a GPU EP (CUDA/TensorRT) or switch to CoreML for quantized inference.
- **Re-export**: Scripts available in the [source repo](https://github.com/Vincentqyw/MixVPR) (`export_quant_onnx.py`, `export_coreml.py`).

## Reference

```bibtex
@inproceedings{ali2023mixvpr,
  title={{MixVPR}: Feature Mixing for Visual Place Recognition},
  author={Ali-bey, Amar and Chaib-draa, Brahim and Gigu{\`e}re, Philippe},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={2998--3007},
  year={2023}
}
```