Update benchmark results in model card
Browse files- MODEL_CARD.md +94 -15
MODEL_CARD.md
CHANGED
|
@@ -11,16 +11,13 @@ tags:
|
|
| 11 |
|
| 12 |
# OpenThai-SystemOne ONNX
|
| 13 |
|
| 14 |
-
This is an FP32 ONNX Runtime conversion of `iapp/OpenThai-SystemOne`
|
| 15 |
-
`f3709948b5e3cc9606a57e74ba62b7a639d17dd3`.
|
| 16 |
|
| 17 |
-
The
|
| 18 |
tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
|
| 19 |
-
is validated through 65,536 tokens only.
|
| 20 |
-
available hardware and have structural—not numerical—validation.
|
| 21 |
|
| 22 |
-
The model requires ONNX Runtime 1.30.0.
|
| 23 |
-
will not run in arbitrary ONNX engines.
|
| 24 |
|
| 25 |
## Usage
|
| 26 |
|
|
@@ -37,25 +34,107 @@ result = client.system_one(
|
|
| 37 |
)
|
| 38 |
```
|
| 39 |
|
| 40 |
-
The client subclasses the upstream `SystemOneClient`
|
| 41 |
-
`5d04bcca0c58bd10e7dac2d3d369d8f760bea6cf`. Formatting, public methods,
|
| 42 |
response classes, decoding, confidence, permutations, and usage accounting are
|
| 43 |
inherited unchanged.
|
| 44 |
|
| 45 |
## Validation
|
| 46 |
|
| 47 |
-
The uploaded `validation-report.json` is the machine-readable source of truth.
|
| 48 |
-
Upload is blocked unless the client suite, standard parity suite, 65,536-token
|
| 49 |
-
parity test, and structural 262,144-token checks all pass.
|
| 50 |
-
|
| 51 |
See `artifact-manifest.json` for the tensor schema, operator domains, runtime
|
| 52 |
versions, source revisions, limits, and SHA-256 checksums.
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
## Limitations
|
| 55 |
|
| 56 |
- FP32 only.
|
| 57 |
-
- CPU and CUDA execution providers only through the packaged client.
|
| 58 |
-
- ONNX Runtime contrib operators make this artifact runtime-specific.
|
| 59 |
- The model performs complete-request inference; generation caches are
|
| 60 |
deliberately internal and reset for every call.
|
| 61 |
- 262,144 tokens is an accepted graph boundary, not a performance guarantee.
|
|
|
|
| 11 |
|
| 12 |
# OpenThai-SystemOne ONNX
|
| 13 |
|
| 14 |
+
This is an FP32 ONNX Runtime conversion of `iapp/OpenThai-SystemOne`.
|
|
|
|
| 15 |
|
| 16 |
+
The accepted structural limit is 262,144
|
| 17 |
tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
|
| 18 |
+
is validated through 65,536 tokens only.
|
|
|
|
| 19 |
|
| 20 |
+
The model requires ONNX Runtime 1.30.0.
|
|
|
|
| 21 |
|
| 22 |
## Usage
|
| 23 |
|
|
|
|
| 34 |
)
|
| 35 |
```
|
| 36 |
|
| 37 |
+
The client subclasses the upstream `SystemOneClient`. Formatting, public methods,
|
|
|
|
| 38 |
response classes, decoding, confidence, permutations, and usage accounting are
|
| 39 |
inherited unchanged.
|
| 40 |
|
| 41 |
## Validation
|
| 42 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
See `artifact-manifest.json` for the tensor schema, operator domains, runtime
|
| 44 |
versions, source revisions, limits, and SHA-256 checksums.
|
| 45 |
|
| 46 |
+
### Benchmark on Thai Datasets
|
| 47 |
+
|
| 48 |
+
#### Result Summary
|
| 49 |
+
|
| 50 |
+
- **OpenThaiONNX and OpenThai produced identical predictions across all
|
| 51 |
+
2,704 evaluated examples** (Cohen's kappa and raw agreement were both
|
| 52 |
+
1.00). This benchmark only dips its toes in with up to 500 samples per task (204 for SIB-200). _Want to help it dive deeper? Larger and more thorough evaluation contributions are very welcome!_
|
| 53 |
+
|
| 54 |
+
- **That parity preserves OpenThai's accuracy advantage:** the OpenThai models
|
| 55 |
+
averaged 76.5% accuracy across the six selected Thai CPU task configurations,
|
| 56 |
+
compared with 60.6% for Laya. That is a gain of 15.9 percentage points, or a
|
| 57 |
+
26.2% relative improvement.
|
| 58 |
+
|
| 59 |
+
- **OpenThaiONNX had 34.8% ± 4.4 percentage points lower CPU runtime than
|
| 60 |
+
OpenThai** This makes the ONNX version a better fit for
|
| 61 |
+
latency-sensitive, offline, and edge applications without a dedicated
|
| 62 |
+
accelerator, provided the device has sufficient memory for the FP32 model. _Sadly, Laya still crosses the finish line about 10× sooner than both OpenThaiONNX and OpenThai. The good news: that leaves us a delightfully large optimization playground—contributions are very welcome!_lly
|
| 63 |
+
large optimization playground, and contributions are very welcome!
|
| 64 |
+
|
| 65 |
+
#### CPU
|
| 66 |
+
|
| 67 |
+
| Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
|
| 68 |
+
|---|---|---:|---|---:|---:|---:|
|
| 69 |
+
| xnli_th | ONNX | 500 | choice | 0.8100 | 0.0246 | 407.935 |
|
| 70 |
+
| xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 686.813 |
|
| 71 |
+
| xnli_th | Laya | 500 | choice | 0.6740 | 0.2347 | 39.480 |
|
| 72 |
+
| | | | | | | |
|
| 73 |
+
| xnli_th | ONNX | 500 | noul | 0.8680 | 0.0247 | 407.935 |
|
| 74 |
+
| xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 686.813 |
|
| 75 |
+
| xnli_th | Laya | 500 | noul | 0.8180 | 0.1511 | 39.480 |
|
| 76 |
+
| | | | | | | |
|
| 77 |
+
| sib200_th | ONNX | 204 | choice | 0.7843 | 0.0816 | 126.797 |
|
| 78 |
+
| sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 199.093 |
|
| 79 |
+
| sib200_th | Laya | 204 | choice | 0.7500 | 0.1572 | 7.244 |
|
| 80 |
+
| | | | | | | |
|
| 81 |
+
| wongnai | ONNX | 500 | score | 0.6400 | 0.0428 | 2866.504 |
|
| 82 |
+
| wongnai | OpenThai | 500 | score | 0.6400 | 0.0428 | 4008.158 |
|
| 83 |
+
| wongnai | Laya | 500 | score | 0.2640 | 0.2493 | 167.471 |
|
| 84 |
+
| | | | | | | |
|
| 85 |
+
| wisesight | ONNX | 500 | choice | 0.4960 | 0.3603 | 1397.613 |
|
| 86 |
+
| wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2125.524 |
|
| 87 |
+
| wisesight | Laya | 500 | choice | 0.2800 | 0.2753 | 91.182 |
|
| 88 |
+
| | | | | | | |
|
| 89 |
+
| xlam_tools | ONNX | 500 | choice | 0.9920 | 0.0072 | 448.700 |
|
| 90 |
+
| xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 682.976 |
|
| 91 |
+
| xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 80.015 |
|
| 92 |
+
|
| 93 |
+
#### GPU/MPS
|
| 94 |
+
|
| 95 |
+
| Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
|
| 96 |
+
|---|---|---:|---|---:|---:|---:|
|
| 97 |
+
| xnli_th | ONNX | 500 | choice | ??? | ??? | ??? |
|
| 98 |
+
| xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 165.242 |
|
| 99 |
+
| xnli_th | Laya | 500 | choice | 0.6700 | 0.2262 | 27.470 |
|
| 100 |
+
| | | | | | | |
|
| 101 |
+
| xnli_th | ONNX | 500 | noul | ??? | ??? | ??? |
|
| 102 |
+
| xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 165.242 |
|
| 103 |
+
| xnli_th | Laya | 500 | noul | 0.7640 | 0.1072 | 27.470 |
|
| 104 |
+
| | | | | | | |
|
| 105 |
+
| sib200_th | OpenThai | 204 | choice | ??? | ??? | ??? |
|
| 106 |
+
| sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 51.089 |
|
| 107 |
+
| sib200_th | Laya | 204 | choice | 0.6176 | 0.1412 | 5.103 |
|
| 108 |
+
| | | | | | | |
|
| 109 |
+
| wongnai | OpenThai | 500 | score | ??? | ??? | ??? |
|
| 110 |
+
| wongnai | OpenThai | 500 | score | 0.6400 | 0.0424 | 3178.868 |
|
| 111 |
+
| wongnai | Laya | 500 | score | 0.0340 | 0.0173 | 124.185 |
|
| 112 |
+
| | | | | | | |
|
| 113 |
+
| wisesight | OpenThai | 500 | choice | ??? | ??? | ??? |
|
| 114 |
+
| wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2000.341 |
|
| 115 |
+
| wisesight | Laya | 500 | choice | 0.2500 | 0.2002 | 74.300 |
|
| 116 |
+
| | | | | | | |
|
| 117 |
+
| xlam_tools | OpenThai | 500 | choice | ??? | ??? | ??? |
|
| 118 |
+
| xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 609.601 |
|
| 119 |
+
| xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 45.588 |
|
| 120 |
+
|
| 121 |
+
### OpenThaiONNX vs OpenThai Agreement (Cohen's kappa)
|
| 122 |
+
|
| 123 |
+
#### CPU
|
| 124 |
+
|
| 125 |
+
| Task | n | Kappa |
|
| 126 |
+
|---|---:|---:|
|
| 127 |
+
| **Overall** | **2704** | **1.000000** |
|
| 128 |
+
| sib200_th / choice | 204 | 1.000000 |
|
| 129 |
+
| wisesight / choice | 500 | 1.000000 |
|
| 130 |
+
| wongnai / score | 500 | 1.000000 |
|
| 131 |
+
| xlam_tools / choice | 500 | 1.000000 |
|
| 132 |
+
| xnli_th / noul | 500 | 1.000000 |
|
| 133 |
+
| xnli_th / choice | 500 | 1.000000 |
|
| 134 |
+
|
| 135 |
## Limitations
|
| 136 |
|
| 137 |
- FP32 only.
|
|
|
|
|
|
|
| 138 |
- The model performs complete-request inference; generation caches are
|
| 139 |
deliberately internal and reset for every call.
|
| 140 |
- 262,144 tokens is an accepted graph boundary, not a performance guarantee.
|