Publish validated FP32, INT8, and INT4 ONNX Runtime bundle

1910337 verified about 1 month ago

6.07 kB

---
license: apache-2.0
library_name: onnxruntime
base_model: iapp/OpenThai-SystemOne
tags:
  - onnx
  - onnxruntime
  - thai
  - qwen3.5
---
# OpenThai-SystemOne ONNX

This is a bundled ONNX Runtime conversion of `iapp/OpenThai-SystemOne` with
FP32, INT8, and INT4 versions.

The accepted structural limit is 262,144
tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
is validated through 65,536 tokens only.

The model requires ONNX Runtime 1.30.0.

## Usage

Clone this model repository and run `uv sync --frozen` once to install its
locked helper package and runtime dependencies, then:

```python
from openthai_systemone_onnx import OnnxSystemOneClient, Choice
client = OnnxSystemOneClient("imtk/OpenThai-SystemOne-ONNX")
result = client.system_one(
    "บริการดีมาก",
    {"sentiment": Choice(instructions="Choose sentiment", criteria={"positive": None, "negative": None})},
)
```

Pass `precision="int8"` or `precision="int4"` to select a quantized graph.
* INT8 dynamically quantizes activations and uses per-channel signed weights.
* INT4 uses asymmetric 32-value weight blocks with FP32 activations.

The client subclasses the upstream `SystemOneClient`. Formatting, public methods,
response classes, decoding, confidence, permutations, and usage accounting are
inherited unchanged.

## Validation

See `artifact-manifest.json` for the tensor schema, operator domains, runtime
versions, source revisions, limits, and SHA-256 checksums.

### Benchmark on Thai Datasets

#### Result Summary

- **OpenThaiONNX and OpenThai produced identical predictions across all
  2,704 evaluated examples** (Cohen's kappa and raw agreement were both
  1.00). This benchmark only dips its toes in with up to 500 samples per task (204 for SIB-200). _Want to help it dive deeper? Larger and more thorough evaluation contributions are very welcome!_
  
- **That parity preserves OpenThai's accuracy advantage:** the OpenThai models
  averaged 76.5% accuracy across the six selected Thai CPU task configurations,
  compared with 60.6% for Laya. That is a gain of 15.9 percentage points, or a
  26.2% relative improvement.

- **OpenThaiONNX had 34.8% ± 4.4 percentage points lower CPU runtime than
  OpenThai** This makes the ONNX version a better fit for
  latency-sensitive, offline, and edge applications without a dedicated
  accelerator, provided the device has sufficient memory for the FP32 model. _Sadly, Laya still crosses the finish line about 10× sooner than both OpenThaiONNX and OpenThai. The good news: that leaves us a delightfully large optimization playground—contributions are very welcome!_lly
  large optimization playground, and contributions are very welcome!

#### CPU

| Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
|---|---|---:|---|---:|---:|---:|
| xnli_th | ONNX | 500 | choice | 0.8100 | 0.0246 | 407.935 |
| xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 686.813 |
| xnli_th | Laya | 500 | choice | 0.6740 | 0.2347 | 39.480 |
| | | | | | | |
| xnli_th | ONNX | 500 | noul | 0.8680 | 0.0247 | 407.935 |
| xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 686.813 |
| xnli_th | Laya | 500 | noul | 0.8180 | 0.1511 | 39.480 |
| | | | | | | |
| sib200_th | ONNX | 204 | choice | 0.7843 | 0.0816 | 126.797 |
| sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 199.093 |
| sib200_th | Laya | 204 | choice | 0.7500 | 0.1572 | 7.244 |
| | | | | | | |
| wongnai | ONNX | 500 | score | 0.6400 | 0.0428 | 2866.504 |
| wongnai | OpenThai | 500 | score | 0.6400 | 0.0428 | 4008.158 |
| wongnai | Laya | 500 | score | 0.2640 | 0.2493 | 167.471 |
| | | | | | | |
| wisesight | ONNX | 500 | choice | 0.4960 | 0.3603 | 1397.613 |
| wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2125.524 |
| wisesight | Laya | 500 | choice | 0.2800 | 0.2753 | 91.182 |
| | | | | | | |
| xlam_tools | ONNX | 500 | choice | 0.9920 | 0.0072 | 448.700 |
| xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 682.976 |
| xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 80.015 |

#### GPU/MPS

| Dataset | Model | n | Type | Accuracy | ECE | Runtime (s) |
|---|---|---:|---|---:|---:|---:|
| xnli_th | ONNX | 500 | choice | ??? | ??? | ??? |
| xnli_th | OpenThai | 500 | choice | 0.8100 | 0.0246 | 165.242 |
| xnli_th | Laya | 500 | choice | 0.6700 | 0.2262 | 27.470 |
| | | | | | | |
| xnli_th | ONNX | 500 | noul | ??? | ??? | ??? |
| xnli_th | OpenThai | 500 | noul | 0.8680 | 0.0247 | 165.242 |
| xnli_th | Laya | 500 | noul | 0.7640 | 0.1072 | 27.470 |
| | | | | | | |
| sib200_th | OpenThai | 204 | choice | ??? | ??? | ??? |
| sib200_th | OpenThai | 204 | choice | 0.7843 | 0.0816 | 51.089 |
| sib200_th | Laya | 204 | choice | 0.6176 | 0.1412 | 5.103 |
| | | | | | | |
| wongnai | OpenThai | 500 | score | ??? | ??? | ??? |
| wongnai | OpenThai | 500 | score | 0.6400 | 0.0424 | 3178.868 |
| wongnai | Laya | 500 | score | 0.0340 | 0.0173 | 124.185 |
| | | | | | | |
| wisesight | OpenThai | 500 | choice | ??? | ??? | ??? |
| wisesight | OpenThai | 500 | choice | 0.4960 | 0.3603 | 2000.341 |
| wisesight | Laya | 500 | choice | 0.2500 | 0.2002 | 74.300 |
| | | | | | | |
| xlam_tools | OpenThai | 500 | choice | ??? | ??? | ??? |
| xlam_tools | OpenThai | 500 | choice | 0.9920 | 0.0072 | 609.601 |
| xlam_tools | Laya | 500 | choice | 0.8500 | 0.0598 | 45.588 |

### OpenThaiONNX vs OpenThai Agreement (Cohen's kappa)

| Task | n | Kappa |
|---|---:|---:|
| **Overall** | **2704** | **1.000000** |
| sib200_th / choice | 204 | 1.000000 |
| wisesight / choice | 500 | 1.000000 |
| wongnai / score | 500 | 1.000000 |
| xlam_tools / choice | 500 | 1.000000 |
| xnli_th / noul | 500 | 1.000000 |
| xnli_th / choice | 500 | 1.000000 |

## Limitations

- Quantized results are approximate and should be evaluated with the included
  held-out benchmark for the target workload.
- The model performs complete-request inference; generation caches are
  deliberately internal and reset for every call.
- 262,144 tokens is an accepted graph boundary, not a performance guarantee.

	---
	license: apache-2.0
	library_name: onnxruntime
	base_model: iapp/OpenThai-SystemOne
	tags:
	- onnx
	- onnxruntime
	- thai
	- qwen3.5
	---

	# OpenThai-SystemOne ONNX

	This is a bundled ONNX Runtime conversion of `iapp/OpenThai-SystemOne` with
	FP32, INT8, and INT4 versions.

	The accepted structural limit is 262,144
	tokens, inherited from the Qwen3.5 text backbone. PyTorch/ONNX numerical parity
	is validated through 65,536 tokens only.

	The model requires ONNX Runtime 1.30.0.

	## Usage

	Clone this model repository and run `uv sync --frozen` once to install its
	locked helper package and runtime dependencies, then:

	```python
	from openthai_systemone_onnx import OnnxSystemOneClient, Choice

	client = OnnxSystemOneClient("imtk/OpenThai-SystemOne-ONNX")
	result = client.system_one(
	"บริการดีมาก",
	{"sentiment": Choice(instructions="Choose sentiment", criteria={"positive": None, "negative": None})},
	)
	```

	Pass `precision="int8"` or `precision="int4"` to select a quantized graph.
	* INT8 dynamically quantizes activations and uses per-channel signed weights.
	* INT4 uses asymmetric 32-value weight blocks with FP32 activations.

	The client subclasses the upstream `SystemOneClient`. Formatting, public methods,
	response classes, decoding, confidence, permutations, and usage accounting are
	inherited unchanged.

	## Validation

	See `artifact-manifest.json` for the tensor schema, operator domains, runtime
	versions, source revisions, limits, and SHA-256 checksums.

	### Benchmark on Thai Datasets

	#### Result Summary

	- **OpenThaiONNX and OpenThai produced identical predictions across all
	2,704 evaluated examples** (Cohen's kappa and raw agreement were both
	1.00). This benchmark only dips its toes in with up to 500 samples per task (204 for SIB-200). _Want to help it dive deeper? Larger and more thorough evaluation contributions are very welcome!_

	- That parity preserves OpenThai's accuracy advantage: the OpenThai models
	averaged 76.5% accuracy across the six selected Thai CPU task configurations,
	compared with 60.6% for Laya. That is a gain of 15.9 percentage points, or a
	26.2% relative improvement.

	- **OpenThaiONNX had 34.8% ± 4.4 percentage points lower CPU runtime than
	OpenThai** This makes the ONNX version a better fit for
	latency-sensitive, offline, and edge applications without a dedicated
	accelerator, provided the device has sufficient memory for the FP32 model. _Sadly, Laya still crosses the finish line about 10× sooner than both OpenThaiONNX and OpenThai. The good news: that leaves us a delightfully large optimization playground—contributions are very welcome!_lly
	large optimization playground, and contributions are very welcome!

	#### CPU

	\| Dataset \| Model \| n \| Type \| Accuracy \| ECE \| Runtime (s) \|
	\|---\|---\|---:\|---\|---:\|---:\|---:\|
	\| xnli_th \| ONNX \| 500 \| choice \| 0.8100 \| 0.0246 \| 407.935 \|
	\| xnli_th \| OpenThai \| 500 \| choice \| 0.8100 \| 0.0246 \| 686.813 \|
	\| xnli_th \| Laya \| 500 \| choice \| 0.6740 \| 0.2347 \| 39.480 \|
	\| \| \| \| \| \| \| \|
	\| xnli_th \| ONNX \| 500 \| noul \| 0.8680 \| 0.0247 \| 407.935 \|
	\| xnli_th \| OpenThai \| 500 \| noul \| 0.8680 \| 0.0247 \| 686.813 \|
	\| xnli_th \| Laya \| 500 \| noul \| 0.8180 \| 0.1511 \| 39.480 \|
	\| \| \| \| \| \| \| \|
	\| sib200_th \| ONNX \| 204 \| choice \| 0.7843 \| 0.0816 \| 126.797 \|
	\| sib200_th \| OpenThai \| 204 \| choice \| 0.7843 \| 0.0816 \| 199.093 \|
	\| sib200_th \| Laya \| 204 \| choice \| 0.7500 \| 0.1572 \| 7.244 \|
	\| \| \| \| \| \| \| \|
	\| wongnai \| ONNX \| 500 \| score \| 0.6400 \| 0.0428 \| 2866.504 \|
	\| wongnai \| OpenThai \| 500 \| score \| 0.6400 \| 0.0428 \| 4008.158 \|
	\| wongnai \| Laya \| 500 \| score \| 0.2640 \| 0.2493 \| 167.471 \|
	\| \| \| \| \| \| \| \|
	\| wisesight \| ONNX \| 500 \| choice \| 0.4960 \| 0.3603 \| 1397.613 \|
	\| wisesight \| OpenThai \| 500 \| choice \| 0.4960 \| 0.3603 \| 2125.524 \|
	\| wisesight \| Laya \| 500 \| choice \| 0.2800 \| 0.2753 \| 91.182 \|
	\| \| \| \| \| \| \| \|
	\| xlam_tools \| ONNX \| 500 \| choice \| 0.9920 \| 0.0072 \| 448.700 \|
	\| xlam_tools \| OpenThai \| 500 \| choice \| 0.9920 \| 0.0072 \| 682.976 \|
	\| xlam_tools \| Laya \| 500 \| choice \| 0.8500 \| 0.0598 \| 80.015 \|

	#### GPU/MPS

	\| Dataset \| Model \| n \| Type \| Accuracy \| ECE \| Runtime (s) \|
	\|---\|---\|---:\|---\|---:\|---:\|---:\|
	\| xnli_th \| ONNX \| 500 \| choice \| ??? \| ??? \| ??? \|
	\| xnli_th \| OpenThai \| 500 \| choice \| 0.8100 \| 0.0246 \| 165.242 \|
	\| xnli_th \| Laya \| 500 \| choice \| 0.6700 \| 0.2262 \| 27.470 \|
	\| \| \| \| \| \| \| \|
	\| xnli_th \| ONNX \| 500 \| noul \| ??? \| ??? \| ??? \|
	\| xnli_th \| OpenThai \| 500 \| noul \| 0.8680 \| 0.0247 \| 165.242 \|
	\| xnli_th \| Laya \| 500 \| noul \| 0.7640 \| 0.1072 \| 27.470 \|
	\| \| \| \| \| \| \| \|
	\| sib200_th \| OpenThai \| 204 \| choice \| ??? \| ??? \| ??? \|
	\| sib200_th \| OpenThai \| 204 \| choice \| 0.7843 \| 0.0816 \| 51.089 \|
	\| sib200_th \| Laya \| 204 \| choice \| 0.6176 \| 0.1412 \| 5.103 \|
	\| \| \| \| \| \| \| \|
	\| wongnai \| OpenThai \| 500 \| score \| ??? \| ??? \| ??? \|
	\| wongnai \| OpenThai \| 500 \| score \| 0.6400 \| 0.0424 \| 3178.868 \|
	\| wongnai \| Laya \| 500 \| score \| 0.0340 \| 0.0173 \| 124.185 \|
	\| \| \| \| \| \| \| \|
	\| wisesight \| OpenThai \| 500 \| choice \| ??? \| ??? \| ??? \|
	\| wisesight \| OpenThai \| 500 \| choice \| 0.4960 \| 0.3603 \| 2000.341 \|
	\| wisesight \| Laya \| 500 \| choice \| 0.2500 \| 0.2002 \| 74.300 \|
	\| \| \| \| \| \| \| \|
	\| xlam_tools \| OpenThai \| 500 \| choice \| ??? \| ??? \| ??? \|
	\| xlam_tools \| OpenThai \| 500 \| choice \| 0.9920 \| 0.0072 \| 609.601 \|
	\| xlam_tools \| Laya \| 500 \| choice \| 0.8500 \| 0.0598 \| 45.588 \|

	### OpenThaiONNX vs OpenThai Agreement (Cohen's kappa)

	\| Task \| n \| Kappa \|
	\|---\|---:\|---:\|
	\| Overall \| 2704 \| 1.000000 \|
	\| sib200_th / choice \| 204 \| 1.000000 \|
	\| wisesight / choice \| 500 \| 1.000000 \|
	\| wongnai / score \| 500 \| 1.000000 \|
	\| xlam_tools / choice \| 500 \| 1.000000 \|
	\| xnli_th / noul \| 500 \| 1.000000 \|
	\| xnli_th / choice \| 500 \| 1.000000 \|

	## Limitations

	- Quantized results are approximate and should be evaluated with the included
	held-out benchmark for the target workload.
	- The model performs complete-request inference; generation caches are
	deliberately internal and reset for every call.
	- 262,144 tokens is an accepted graph boundary, not a performance guarantee.