| --- |
| library_name: onnx |
| license: other |
| license_name: mixed-upstream-model-licenses |
| license_link: https://huggingface.co/ketiswp/ONNX_Models/blob/main/LICENSES.md |
| tags: |
| - onnx |
| - onnxruntime |
| - onnx-mlir |
| - quantization |
| - fp32 |
| --- |
| |
| # FP32 and Quantized Model ONNX/ONNX-MLIR Validation |
|
|
| This repository contains paired public FP32 and public quantized models and reproduces the following tasks. |
|
|
| 1. Identify and collect paired public FP32 and quantized models. |
| 2. Convert or prepare each model pair in ONNX format. |
| 3. Validate the ONNX models with ONNX Runtime and compare their outputs with the source-model outputs. |
| 4. Compare the task-level accuracy of the FP32 and quantized variants using the same evaluation dataset and protocol for each pair. |
| 5. Generate Netron PNG images of the FP32 and quantized ONNX graphs. |
| 6. Import the ONNX models into the ONNX-MLIR ONNX Dialect and lower them further where supported. |
| 7. Generate MLIR graphs based on static operation order and SSA dependencies. |
|
|
| ## Models |
|
|
| A total of 21 FP32/quantized model pairs are included. |
|
|
| | Task | Number of models | |
| |---|---:| |
| | Vision classification | 10 | |
| | Keyword spotting | 4 | |
| | Semantic segmentation | 3 | |
| | Object detection | 2 | |
| | Anomaly detection | 1 | |
| | Language model | 1 | |
|
|
| Model names, public URLs, licenses, and original-file SHA-256 checksums are listed in [`model_registry.csv`](model_registry.csv). |
|
|
| ## Accuracy Summary |
|
|
| The delta is `quantized - FP32`. Higher values are better except for SP02 (FP/FN) and VC13 (error), where lower values are better. |
|
|
| | Model | Metric | FP32 | Quantized | Delta | |
| |---|---|---:|---:|---:| |
| | AD01 | AUC / pAUC (max_fpr=0.1) | 0.876001 / 0.764121 | 0.840250 / 0.720049 | -0.035750 / -0.044071 | |
| | LM04 | AUROC / TPR@FPR 5% / 1% | 0.667078 / 0.318983 / 0.233873 | 0.668862 / 0.322615 / 0.235853 | +0.001785 / +0.003632 / +0.001981 | |
| | OD06 | COCO bbox mAP | 24.8751% | 24.2822% | -0.5929 pp | |
| | OD07 | COCO bbox mAP | 31.8594% | 31.4191% | -0.4403 pp | |
| | SG06 | mIoU | 75.6398% | 74.1290% | -1.5108 pp | |
| | SG07 | mIoU | 70.6473% | 69.6191% | -1.0282 pp | |
| | SG08 | mIoU (CamVid cross-dataset) | 50.6498% | 51.1600% | +0.5102 pp | |
| | SP01 | Top-1 accuracy | 91.86% | 91.66% | -0.2045 pp | |
| | SP02 | FP / FN (1 s) | 5 / 6 | 4 / 6 | -1 / +0 | |
| | SP08 | Top-1 accuracy (yes/no subset) | 94.05% | 94.05% | +0.0000 pp | |
| | SP09 | Top-1 accuracy | 95.06% | 94.70% | -0.3590 pp | |
| | VC01 | Top-1 accuracy | 85.10% | 85.60% | +0.5000 pp | |
| | VC02 | Top-1 accuracy | 87.00% | 87.00% | +0.0000 pp | |
| | VC03 | Top-1 / Top-5 accuracy | 49.80% / 74.20% | 48.00% / 72.80% | -1.8000 / -1.4000 pp | |
| | VC04 | Top-1 / Top-5 accuracy | 63.30% / 84.90% | 60.70% / 83.20% | -2.6000 / -1.7000 pp | |
| | VC05 | Top-1 accuracy | 58.13% | 56.77% | -1.3600 pp | |
| | VC06 | Top-1 accuracy | 66.20% | 65.31% | -0.8900 pp | |
| | VC09 | Top-1 / Top-5 accuracy | 56.85% / 79.87% | 56.48% / 79.76% | -0.3700 / -0.1100 pp | |
| | VC11 | Top-1 accuracy | 75.10% | 74.40% | -0.7000 pp | |
| | VC12 | Top-1 / Top-5 accuracy | 69.48% / 89.26% | 68.30% / 88.44% | -1.1800 / -0.8200 pp | |
| | VC13 | Top-1 / Top-5 error | 33.65% / 13.43% | 33.85% / 13.66% | +0.2000 / +0.2300 pp | |
| |
| Full model names and published-result comparisons are available in [`reports/accuracy/model_accuracy.csv`](reports/accuracy/model_accuracy.csv). |
| |
| ## Results |
| |
| | Result | File | |
| |---|---| |
| | FP32/quantized accuracy | [`reports/accuracy/model_accuracy.csv`](reports/accuracy/model_accuracy.csv) | |
| | conversion status | [`reports/conversion/pipeline_status.csv`](reports/conversion/pipeline_status.csv) | |
| | MLIR stage coverage | [`reports/conversion/ir_stage_coverage.csv`](reports/conversion/ir_stage_coverage.csv) | |
| | Netron ONNX graphs | [`reports/graphs/netron/README.md`](reports/graphs/netron/README.md) | |
| | ONNX Dialect static-order graphs | [`reports/graphs/mlir/README.md`](reports/graphs/mlir/README.md) | |
| |
| ## Directories |
| |
| ```text |
| configs/ Model conversion, MLIR conversion, and accuracy evaluation configurations |
| environment/ Python, Netron, and ONNX-MLIR versions and installation scripts |
| models/ Per-model conversion outputs |
| reports/ Accuracy, conversion status, Netron, and IR graph results |
| research/ Public original models and source materials |
| scripts/ Conversion, evaluation, graph generation, and validation code |
| schemas/ Configuration and execution-result formats |
| tests/ Tests for reproducibility code |
| ``` |
| |
| ## Reproduction |
| |
| ```bash |
| git lfs install |
| git lfs pull |
| make setup |
| make validate |
| make convert |
| make accuracy |
| make netron |
| make mlir |
| make mlir-graphs |
| make test |
| ``` |
| |
| `make mlir-graphs` updates `reports/conversion/ir_stage_coverage.csv` using the checksums of the current MLIR results in `models/`, and then generates the graphs. |
| |
| Individual scripts are documented in [`scripts/README.md`](scripts/README.md), configuration files in [`configs/README.md`](configs/README.md), and result files in [`reports/README.md`](reports/README.md). |
| |
