ONNX
onnxruntime
onnx-mlir
quantization
fp32
ONNX_Models / reports /README.md
purejomo's picture
Finalize public ONNX/ONNX-MLIR validation release
ed3aeeb
|
Raw
History Blame Contribute Delete
3 kB

๊ฒฐ๊ณผ

๊ฒฝ๋กœ ๋‚ด์šฉ
accuracy/model_accuracy.csv 21๊ฐœ ๋ชจ๋ธ์˜ FP32ยท์–‘์žํ™” ์ •ํ™•๋„์™€ ๋ณ€ํ™”๋Ÿ‰
accuracy/onnx_mlir_output_check.csv ์ƒ์„ฑ๋œ native ONNX-MLIR artifact์˜ ์ถœ๋ ฅ ๋น„๊ต
conversion/pipeline_status.csv ์›๋ณธ, ONNX, ONNX-MLIR ๋ณ€ํ™˜ ์ƒํƒœ
conversion/ir_stage_coverage.csv ONNX Dialect, Krnl, Affine/SCF/MemRef, LLVM ๋‹จ๊ณ„๋ณ„ ์ƒํƒœ
conversion/quantization_preservation.csv ONNX์™€ MLIR์˜ ์–‘์žํ™” ์ •๋ณด ๋ณด์กด ์ƒํƒœ
graphs/netron/ Netron FP32ยท์–‘์žํ™” ONNX ๊ทธ๋ž˜ํ”„ ์ƒ์„ฑยท์ €์žฅ ์œ„์น˜
graphs/mlir/ ONNX Dialect 42๊ฐœ์˜ ์ •์  operation ์ˆœ์„œ์™€ SSA ์—ฐ๊ฒฐ ๊ทธ๋ž˜ํ”„

๋ชจ๋ธ ์ •ํ™•๋„ ๊ฒฐ๊ณผ

์ „์ฒด CSV: accuracy/model_accuracy.csv

ID ์ •ํ™•๋„ ์ง€ํ‘œ FP32 ์–‘์žํ™” ๋ณ€ํ™”๋Ÿ‰ ๊ณต๊ฐœ ์ˆ˜์น˜ ๋น„๊ต
AD01 AUC / pAUC 0.876001 / 0.764121 0.840250 / 0.720049 -0.035750 / -0.044071 ๊ณต๊ฐœ FP32 ์ˆ˜์น˜์™€ ๋™์ผ
LM04 AUROC / TPR@FPR 5% / 1% 0.667078 / 0.318983 / 0.233873 0.668862 / 0.322615 / 0.235853 +0.001785 / +0.003632 / +0.001981 ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
OD06 COCO bbox mAP 24.8751% 24.2822% -0.5929 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
OD07 COCO bbox mAP 31.8594% 31.4191% -0.4403 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
SG06 mIoU 75.6398% 74.1290% -1.5108 pp ๊ณต๊ฐœ FP32 75.32%, ์ฐจ์ด +0.3198 pp
SG07 mIoU 70.6473% 69.6191% -1.0282 pp ๊ณต๊ฐœ FP32 70.19%, ์ฐจ์ด +0.4573 pp
SG08 CamVid mIoU 50.6498% 51.1600% +0.5102 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
SP01 Top-1 accuracy 91.86% 91.66% -0.2045 pp ๊ณต๊ฐœ ์‹œํ—˜ ์ •ํ™•๋„ ์•ฝ 92%์™€ ๊ทผ์ ‘
SP02 FP / FN 5 / 6 4 / 6 -1 / 0 ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
SP08 Yes/No Top-1 accuracy 94.05% 94.05% 0.0000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
SP09 Top-1 accuracy 95.06% 94.70% -0.3590 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC01 Top-1 accuracy 85.10% 85.60% +0.5000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC02 Top-1 accuracy 87.00% 87.00% 0.0000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC03 Top-1 / Top-5 accuracy 49.80% / 74.20% 48.00% / 72.80% -1.8000 / -1.4000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC04 Top-1 / Top-5 accuracy 63.30% / 84.90% 60.70% / 83.20% -2.6000 / -1.7000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC05 Top-1 accuracy 58.13% 56.77% -1.3600 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC06 Top-1 accuracy 66.20% 65.31% -0.8900 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC09 Top-1 / Top-5 accuracy 56.85% / 79.87% 56.48% / 79.76% -0.3700 / -0.1100 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC11 Top-1 accuracy 75.10% 74.40% -0.7000 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC12 Top-1 / Top-5 accuracy 69.48% / 89.26% 68.30% / 88.44% -1.1800 / -0.8200 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.
VC13 Top-1 / Top-5 error 33.65% / 13.43% 33.85% / 13.66% +0.2000 / +0.2300 pp ๊ณต๊ฐœ ์ˆ˜์น˜ ์—†์Œ.

VC13์€ accuracy๊ฐ€ ์•„๋‹ˆ๋ผ error์ด๋ฏ€๋กœ ๊ฐ’์ด ๋‚ฎ์„์ˆ˜๋ก ์ข‹์Œ