ketiswp/ONNX_Models

FP32 and Quantized Model ONNX/ONNX-MLIR Validation

This repository contains paired public FP32 and public quantized models and reproduces the following tasks.

1. Identify and collect paired public FP32 and quantized models.
2. Convert or prepare each model pair in ONNX format.
3. Validate the ONNX models with ONNX Runtime and compare their outputs with the source-model outputs.
4. Compare the task-level accuracy of the FP32 and quantized variants using the same evaluation dataset and protocol for each pair.
5. Generate Netron PNG images of the FP32 and quantized ONNX graphs.
6. Import the ONNX models into the ONNX-MLIR ONNX Dialect and lower them further where supported.
7. Generate MLIR graphs based on static operation order and SSA dependencies.

Models

A total of 21 FP32/quantized model pairs are included.

Task	Number of models
Vision classification	10
Keyword spotting	4
Semantic segmentation	3
Object detection	2
Anomaly detection	1
Language model	1

Model names, public URLs, licenses, and original-file SHA-256 checksums are listed in model_registry.csv.

Accuracy Summary

The delta is quantized - FP32. Higher values are better except for SP02 (FP/FN) and VC13 (error), where lower values are better.

Model	Metric	FP32	Quantized	Delta
AD01	AUC / pAUC (max_fpr=0.1)	0.876001 / 0.764121	0.840250 / 0.720049	-0.035750 / -0.044071
LM04	AUROC / TPR@FPR 5% / 1%	0.667078 / 0.318983 / 0.233873	0.668862 / 0.322615 / 0.235853	+0.001785 / +0.003632 / +0.001981
OD06	COCO bbox mAP	24.8751%	24.2822%	-0.5929 pp
OD07	COCO bbox mAP	31.8594%	31.4191%	-0.4403 pp
SG06	mIoU	75.6398%	74.1290%	-1.5108 pp
SG07	mIoU	70.6473%	69.6191%	-1.0282 pp
SG08	mIoU (CamVid cross-dataset)	50.6498%	51.1600%	+0.5102 pp
SP01	Top-1 accuracy	91.86%	91.66%	-0.2045 pp
SP02	FP / FN (1 s)	5 / 6	4 / 6	-1 / +0
SP08	Top-1 accuracy (yes/no subset)	94.05%	94.05%	+0.0000 pp
SP09	Top-1 accuracy	95.06%	94.70%	-0.3590 pp
VC01	Top-1 accuracy	85.10%	85.60%	+0.5000 pp
VC02	Top-1 accuracy	87.00%	87.00%	+0.0000 pp
VC03	Top-1 / Top-5 accuracy	49.80% / 74.20%	48.00% / 72.80%	-1.8000 / -1.4000 pp
VC04	Top-1 / Top-5 accuracy	63.30% / 84.90%	60.70% / 83.20%	-2.6000 / -1.7000 pp
VC05	Top-1 accuracy	58.13%	56.77%	-1.3600 pp
VC06	Top-1 accuracy	66.20%	65.31%	-0.8900 pp
VC09	Top-1 / Top-5 accuracy	56.85% / 79.87%	56.48% / 79.76%	-0.3700 / -0.1100 pp
VC11	Top-1 accuracy	75.10%	74.40%	-0.7000 pp
VC12	Top-1 / Top-5 accuracy	69.48% / 89.26%	68.30% / 88.44%	-1.1800 / -0.8200 pp
VC13	Top-1 / Top-5 error	33.65% / 13.43%	33.85% / 13.66%	+0.2000 / +0.2300 pp

Full model names and published-result comparisons are available in reports/accuracy/model_accuracy.csv.

Results

Result	File
FP32/quantized accuracy	reports/accuracy/model_accuracy.csv
conversion status	reports/conversion/pipeline_status.csv
MLIR stage coverage	reports/conversion/ir_stage_coverage.csv
Netron ONNX graphs	reports/graphs/netron/README.md
ONNX Dialect static-order graphs	reports/graphs/mlir/README.md

Directories

configs/        Model conversion, MLIR conversion, and accuracy evaluation configurations
environment/    Python, Netron, and ONNX-MLIR versions and installation scripts
models/         Per-model conversion outputs
reports/        Accuracy, conversion status, Netron, and IR graph results
research/       Public original models and source materials
scripts/        Conversion, evaluation, graph generation, and validation code
schemas/        Configuration and execution-result formats
tests/          Tests for reproducibility code

Reproduction

git lfs install
git lfs pull
make setup
make validate
make convert
make accuracy
make netron
make mlir
make mlir-graphs
make test

make mlir-graphs updates reports/conversion/ir_stage_coverage.csv using the checksums of the current MLIR results in models/, and then generates the graphs.

Individual scripts are documented in scripts/README.md, configuration files in configs/README.md, and result files in reports/README.md.