imtk commited on
Commit
fdcea56
·
verified ·
1 Parent(s): 98f6291

Update benchmark results in model card

Browse files
Files changed (1) hide show
  1. MODEL_CARD.md +94 -15
MODEL_CARD.md CHANGED
@@ -11,16 +11,13 @@ tags:
11
 
12
  # OpenThai-SystemOne ONNX
13
 
14
- This is an FP32 ONNX Runtime conversion of `iapp/OpenThai-SystemOne` revision
15
- `f3709948b5e3cc9606a57e74ba62b7a639d17dd3`.
16
 
17
- The sequence dimension is symbolic. The accepted structural limit is 262,144
18
  tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
19
- is validated through 65,536 tokens only. Larger requests may be infeasible on
20
- available hardware and have structural—not numerical—validation.
21
 
22
- The model requires ONNX Runtime 1.30.0. It uses Microsoft contrib operators and
23
- will not run in arbitrary ONNX engines.
24
 
25
  ## Usage
26
 
@@ -37,25 +34,107 @@ result = client.system_one(
37
  )
38
  ```
39
 
40
- The client subclasses the upstream `SystemOneClient` at commit
41
- `5d04bcca0c58bd10e7dac2d3d369d8f760bea6cf`. Formatting, public methods,
42
  response classes, decoding, confidence, permutations, and usage accounting are
43
  inherited unchanged.
44
 
45
  ## Validation
46
 
47
- The uploaded `validation-report.json` is the machine-readable source of truth.
48
- Upload is blocked unless the client suite, standard parity suite, 65,536-token
49
- parity test, and structural 262,144-token checks all pass.
50
-
51
  See `artifact-manifest.json` for the tensor schema, operator domains, runtime
52
  versions, source revisions, limits, and SHA-256 checksums.
53
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
  ## Limitations
55
 
56
  - FP32 only.
57
- - CPU and CUDA execution providers only through the packaged client.
58
- - ONNX Runtime contrib operators make this artifact runtime-specific.
59
  - The model performs complete-request inference; generation caches are
60
  deliberately internal and reset for every call.
61
  - 262,144 tokens is an accepted graph boundary, not a performance guarantee.
 
11
 
12
  # OpenThai-SystemOne ONNX
13
 
14
+ This is an FP32 ONNX Runtime conversion of `iapp/OpenThai-SystemOne`.
 
15
 
16
+ The accepted structural limit is 262,144
17
  tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
18
+ is validated through 65,536 tokens only.
 
19
 
20
+ The model requires ONNX Runtime 1.30.0.
 
21
 
22
  ## Usage
23
 
 
34
  )
35
  ```
36
 
37
+ The client subclasses the upstream `SystemOneClient`. Formatting, public methods,
 
38
  response classes, decoding, confidence, permutations, and usage accounting are
39
  inherited unchanged.
40
 
41
  ## Validation
42
 
 
 
 
 
43
  See `artifact-manifest.json` for the tensor schema, operator domains, runtime
44
  versions, source revisions, limits, and SHA-256 checksums.
45
 
46
+ ### Benchmark on Thai Datasets
47
+
48
+ #### Result Summary
49
+
50
+ - **OpenThaiONNX and OpenThai produced identical predictions across all
51
+ 2,704 evaluated examples** (Cohen's kappa and raw agreement were both
52
+ 1.00). This benchmark only dips its toes in with up to 500 samples per task (204 for SIB-200). _Want to help it dive deeper? Larger and more thorough evaluation contributions are very welcome!_
53
+
54
+ - **That parity preserves OpenThai's accuracy advantage:** the OpenThai models
55
+ averaged 76.5% accuracy across the six selected Thai CPU task configurations,
56
+ compared with 60.6% for Laya. That is a gain of 15.9 percentage points, or a
57
+ 26.2% relative improvement.
58
+
59
+ - **OpenThaiONNX had 34.8% ± 4.4 percentage points lower CPU runtime than
60
+ OpenThai** This makes the ONNX version a better fit for
61
+ latency-sensitive, offline, and edge applications without a dedicated
62
+ accelerator, provided the device has sufficient memory for the FP32 model. _Sadly, Laya still crosses the finish line about 10× sooner than both OpenThaiONNX and OpenThai. The good news: that leaves us a delightfully large optimization playground—contributions are very welcome!_lly
63
+ large optimization playground, and contributions are very welcome!
64
+
65
+ #### CPU
66
+
67
+ | Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
68
+ |---|---|---:|---|---:|---:|---:|
69
+ | xnli_th | ONNX | 500 | choice | 0.8100 | 0.0246 | 407.935 |
70
+ | xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 686.813 |
71
+ | xnli_th | Laya | 500 | choice | 0.6740 | 0.2347 | 39.480 |
72
+ | | | | | | | |
73
+ | xnli_th | ONNX | 500 | noul | 0.8680 | 0.0247 | 407.935 |
74
+ | xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 686.813 |
75
+ | xnli_th | Laya | 500 | noul | 0.8180 | 0.1511 | 39.480 |
76
+ | | | | | | | |
77
+ | sib200_th | ONNX | 204 | choice | 0.7843 | 0.0816 | 126.797 |
78
+ | sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 199.093 |
79
+ | sib200_th | Laya | 204 | choice | 0.7500 | 0.1572 | 7.244 |
80
+ | | | | | | | |
81
+ | wongnai | ONNX | 500 | score | 0.6400 | 0.0428 | 2866.504 |
82
+ | wongnai | OpenThai | 500 | score | 0.6400 | 0.0428 | 4008.158 |
83
+ | wongnai | Laya | 500 | score | 0.2640 | 0.2493 | 167.471 |
84
+ | | | | | | | |
85
+ | wisesight | ONNX | 500 | choice | 0.4960 | 0.3603 | 1397.613 |
86
+ | wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2125.524 |
87
+ | wisesight | Laya | 500 | choice | 0.2800 | 0.2753 | 91.182 |
88
+ | | | | | | | |
89
+ | xlam_tools | ONNX | 500 | choice | 0.9920 | 0.0072 | 448.700 |
90
+ | xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 682.976 |
91
+ | xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 80.015 |
92
+
93
+ #### GPU/MPS
94
+
95
+ | Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
96
+ |---|---|---:|---|---:|---:|---:|
97
+ | xnli_th | ONNX | 500 | choice | ??? | ??? | ??? |
98
+ | xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 165.242 |
99
+ | xnli_th | Laya | 500 | choice | 0.6700 | 0.2262 | 27.470 |
100
+ | | | | | | | |
101
+ | xnli_th | ONNX | 500 | noul | ??? | ??? | ??? |
102
+ | xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 165.242 |
103
+ | xnli_th | Laya | 500 | noul | 0.7640 | 0.1072 | 27.470 |
104
+ | | | | | | | |
105
+ | sib200_th | OpenThai | 204 | choice | ??? | ??? | ??? |
106
+ | sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 51.089 |
107
+ | sib200_th | Laya | 204 | choice | 0.6176 | 0.1412 | 5.103 |
108
+ | | | | | | | |
109
+ | wongnai | OpenThai | 500 | score | ??? | ??? | ??? |
110
+ | wongnai | OpenThai | 500 | score | 0.6400 | 0.0424 | 3178.868 |
111
+ | wongnai | Laya | 500 | score | 0.0340 | 0.0173 | 124.185 |
112
+ | | | | | | | |
113
+ | wisesight | OpenThai | 500 | choice | ??? | ??? | ??? |
114
+ | wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2000.341 |
115
+ | wisesight | Laya | 500 | choice | 0.2500 | 0.2002 | 74.300 |
116
+ | | | | | | | |
117
+ | xlam_tools | OpenThai | 500 | choice | ??? | ??? | ??? |
118
+ | xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 609.601 |
119
+ | xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 45.588 |
120
+
121
+ ### OpenThaiONNX vs OpenThai Agreement (Cohen's kappa)
122
+
123
+ #### CPU
124
+
125
+ | Task | n | Kappa |
126
+ |---|---:|---:|
127
+ | **Overall** | **2704** | **1.000000** |
128
+ | sib200_th / choice | 204 | 1.000000 |
129
+ | wisesight / choice | 500 | 1.000000 |
130
+ | wongnai / score | 500 | 1.000000 |
131
+ | xlam_tools / choice | 500 | 1.000000 |
132
+ | xnli_th / noul | 500 | 1.000000 |
133
+ | xnli_th / choice | 500 | 1.000000 |
134
+
135
  ## Limitations
136
 
137
  - FP32 only.
 
 
138
  - The model performs complete-request inference; generation caches are
139
  deliberately internal and reset for every call.
140
  - 262,144 tokens is an accepted graph boundary, not a performance guarantee.