---
library_name: onnx
license: other
license_name: mixed-upstream-model-licenses
license_link: https://huggingface.co/ketiswp/ONNX_Models/blob/main/LICENSES.md
tags:
- onnx
- onnxruntime
- onnx-mlir
- quantization
- fp32
---

# FP32 and Quantized Model ONNX/ONNX-MLIR Validation

This repository contains paired public FP32 and public quantized models and reproduces the following tasks.

1. Identify and collect paired public FP32 and quantized models.
2. Convert or prepare each model pair in ONNX format.
3. Validate the ONNX models with ONNX Runtime and compare their outputs with the source-model outputs.
4. Compare the task-level accuracy of the FP32 and quantized variants using the same evaluation dataset and protocol for each pair.
5. Generate Netron PNG images of the FP32 and quantized ONNX graphs.
6. Import the ONNX models into the ONNX-MLIR ONNX Dialect and lower them further where supported.
7. Generate MLIR graphs based on static operation order and SSA dependencies.

## Models

A total of 21 FP32/quantized model pairs are included.

| Task | Number of models |
|---|---:|
| Vision classification | 10 |
| Keyword spotting | 4 |
| Semantic segmentation | 3 |
| Object detection | 2 |
| Anomaly detection | 1 |
| Language model | 1 |

Model names, public URLs, licenses, and original-file SHA-256 checksums are listed in [`model_registry.csv`](model_registry.csv).

## Accuracy Summary

The delta is `quantized - FP32`. Higher values are better except for SP02 (FP/FN) and VC13 (error), where lower values are better.

| Model | Metric | FP32 | Quantized | Delta |
|---|---|---:|---:|---:|
| AD01 | AUC / pAUC (max_fpr=0.1) | 0.876001 / 0.764121 | 0.840250 / 0.720049 | -0.035750 / -0.044071 |
| LM04 | AUROC / TPR@FPR 5% / 1% | 0.667078 / 0.318983 / 0.233873 | 0.668862 / 0.322615 / 0.235853 | +0.001785 / +0.003632 / +0.001981 |
| OD06 | COCO bbox mAP | 24.8751% | 24.2822% | -0.5929 pp |
| OD07 | COCO bbox mAP | 31.8594% | 31.4191% | -0.4403 pp |
| SG06 | mIoU | 75.6398% | 74.1290% | -1.5108 pp |
| SG07 | mIoU | 70.6473% | 69.6191% | -1.0282 pp |
| SG08 | mIoU (CamVid cross-dataset) | 50.6498% | 51.1600% | +0.5102 pp |
| SP01 | Top-1 accuracy | 91.86% | 91.66% | -0.2045 pp |
| SP02 | FP / FN (1 s) | 5 / 6 | 4 / 6 | -1 / +0 |
| SP08 | Top-1 accuracy (yes/no subset) | 94.05% | 94.05% | +0.0000 pp |
| SP09 | Top-1 accuracy | 95.06% | 94.70% | -0.3590 pp |
| VC01 | Top-1 accuracy | 85.10% | 85.60% | +0.5000 pp |
| VC02 | Top-1 accuracy | 87.00% | 87.00% | +0.0000 pp |
| VC03 | Top-1 / Top-5 accuracy | 49.80% / 74.20% | 48.00% / 72.80% | -1.8000 / -1.4000 pp |
| VC04 | Top-1 / Top-5 accuracy | 63.30% / 84.90% | 60.70% / 83.20% | -2.6000 / -1.7000 pp |
| VC05 | Top-1 accuracy | 58.13% | 56.77% | -1.3600 pp |
| VC06 | Top-1 accuracy | 66.20% | 65.31% | -0.8900 pp |
| VC09 | Top-1 / Top-5 accuracy | 56.85% / 79.87% | 56.48% / 79.76% | -0.3700 / -0.1100 pp |
| VC11 | Top-1 accuracy | 75.10% | 74.40% | -0.7000 pp |
| VC12 | Top-1 / Top-5 accuracy | 69.48% / 89.26% | 68.30% / 88.44% | -1.1800 / -0.8200 pp |
| VC13 | Top-1 / Top-5 error | 33.65% / 13.43% | 33.85% / 13.66% | +0.2000 / +0.2300 pp |

Full model names and published-result comparisons are available in [`reports/accuracy/model_accuracy.csv`](reports/accuracy/model_accuracy.csv).

## Results

| Result | File |
|---|---|
| FP32/quantized accuracy | [`reports/accuracy/model_accuracy.csv`](reports/accuracy/model_accuracy.csv) |
| conversion status | [`reports/conversion/pipeline_status.csv`](reports/conversion/pipeline_status.csv) |
| MLIR stage coverage | [`reports/conversion/ir_stage_coverage.csv`](reports/conversion/ir_stage_coverage.csv) |
| Netron ONNX graphs | [`reports/graphs/netron/README.md`](reports/graphs/netron/README.md) |
| ONNX Dialect static-order graphs | [`reports/graphs/mlir/README.md`](reports/graphs/mlir/README.md) |

## Directories

```text
configs/        Model conversion, MLIR conversion, and accuracy evaluation configurations
environment/    Python, Netron, and ONNX-MLIR versions and installation scripts
models/         Per-model conversion outputs
reports/        Accuracy, conversion status, Netron, and IR graph results
research/       Public original models and source materials
scripts/        Conversion, evaluation, graph generation, and validation code
schemas/        Configuration and execution-result formats
tests/          Tests for reproducibility code
```

## Reproduction

```bash
git lfs install
git lfs pull
make setup
make validate
make convert
make accuracy
make netron
make mlir
make mlir-graphs
make test
```

`make mlir-graphs` updates `reports/conversion/ir_stage_coverage.csv` using the checksums of the current MLIR results in `models/`, and then generates the graphs.

Individual scripts are documented in [`scripts/README.md`](scripts/README.md), configuration files in [`configs/README.md`](configs/README.md), and result files in [`reports/README.md`](reports/README.md).