| # ONNX parity |
|
|
| `onnx/model.onnx` and `onnx/model_int8.onnx` were exported from the PyTorch weights in this repository (opset 17, |
| dynamic batch and length) and checked with onnxruntime 1.30.0 against PyTorch 2.8.0 on the 160 development documents |
| of the SporeLabs Scrub benchmark (716 marked items), running the full scrubber (rules + model). |
|
|
| | graph | size | largest logit difference | findings identical to PyTorch | secrets recall | direct-identifier recall | precision | |
| |---|---|---|---|---|---|---| |
| | PyTorch | | | | 0.986 | 0.954 | 0.878 | |
| | fp32 ONNX | 67 MB | 0.00004 | 796 of 796 | 0.986 | 0.954 | 0.878 | |
| | int8 ONNX | 19 MB | 3.1 | 782 of 796 | 0.986 | 0.948 | 0.883 | |
|
|
| fp32 gives the same findings as PyTorch and is the default. int8 is a quarter of the size and catches slightly fewer |
| direct identifiers; use it where size matters more (`Scrub(precision="int8")`). |
|
|
