| # Convert and quantize OpenThai-SystemOne for ONNX Runtime |
|
|
| These are the maintainer instructions for converting the pinned |
| `iapp/OpenThai-SystemOne` checkpoint, validating the result, and uploading it |
| to Hugging Face. For application-side `OnnxSystemOneClient` usage, see |
| [README.md](README.md). |
|
|
| Run every command from the repository root. The conversion and release code is |
| kept in `src/openthai_systemone_onnx_tools`; the independent runtime client is |
| in `src/openthai_systemone_onnx`. |
|
|
| ## 1. Create the conversion environment |
|
|
| ```bash |
| uv python pin 3.12 |
| uv sync --extra tools --extra dev --frozen |
| ``` |
|
|
| The first command selects Python 3.12. The second creates `.venv`, installs the |
| runtime package, adds the `tools` dependencies used to build and inspect ONNX, |
| and adds pytest from the `dev` extra. `--frozen` makes the operation fail if |
| `uv.lock` and `pyproject.toml` disagree instead of silently changing versions. |
|
|
| Important pinned versions include ONNX Runtime 1.30.0, ONNX Runtime GenAI |
| 0.16.0, source model revision |
| `f3709948b5e3cc9606a57e74ba62b7a639d17dd3`, and upstream client commit |
| `5d04bcca0c58bd10e7dac2d3d369d8f760bea6cf`. |
|
|
| ## 2. Authenticate with Hugging Face |
|
|
| ```bash |
| uv run hf auth login |
| ``` |
|
|
| This interactively stores a Hugging Face token. Downloading the public source |
| does not require write access; uploading the finished artifact does. |
|
|
| ## 3. Export the model |
|
|
| ```bash |
| uv run openthai-systemone-export \ |
| --output artifacts/OpenThai-SystemOne-ONNX \ |
| --cache-dir .cache/huggingface |
| ``` |
|
|
| This command downloads the pinned checkpoint, loads its safetensor weights, |
| builds the dynamic Qwen3.5 graph, appends the calibrated SystemOne head, and |
| writes: |
|
|
| - `onnx/model.onnx` for graph structure; |
| - `onnx/model.onnx.data` for external FP32 weights; |
| - tokenizer files and the separated helper packages; |
| - license, attribution, model card, and usage documentation; |
| - `artifact-manifest.json` with revisions, limits, tensor schema, operator |
| requirements, and checksums. |
|
|
| The graph keeps symbolic batch, sequence, and question dimensions. It derives |
| MRoPE positions from `attention_mask` and initializes KV, convolution, and |
| recurrent state internally for every complete request. |
|
|
| ## 4. Add INT8 and INT4 variants |
|
|
| ```bash |
| uv run openthai-systemone-quantize \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX |
| ``` |
|
|
| This adds `onnx/model.int8.onnx` and `onnx/model.int4.onnx` plus their external |
| data files. INT8 uses dynamic per-channel quantization. INT4 uses asymmetric |
| 32-value weight blocks and `MatMulNBits`. Both leave the calibrated decision |
| head in FP32. The command loads both variants with the CPU execution provider |
| before installing them and then rewrites the bundled manifest and checksums. |
| It removes an existing validation report because that report refers to the |
| previous manifest. |
|
|
| ## 5. Run fast compatibility and quantized runtime tests |
|
|
| ```bash |
| uv run openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --client |
| ``` |
|
|
| This checks upstream constructor/method signatures, response-class identity, |
| serialization, typed and raw questions, state formats, validation errors, |
| single/batch equivalence, automatic and explicit permutations, accounting, |
| exporter graph construction, and validator device diagnostics. It records the |
| result as `client_compatibility` in `validation-report.json`. |
|
|
| Then validate both quantized graphs on CPU: |
|
|
| ```bash |
| uv run openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --quantized |
| ``` |
|
|
| This checks all graph structures and runs raw-tensor and end-to-end client |
| inference for INT8 and INT4. Quantized validation checks runtime correctness, |
| not exact FP32 numerical parity. |
|
|
| ## 6. Run normal PyTorch/ONNX parity |
|
|
| On CPU, including macOS: |
|
|
| ```bash |
| PYTORCH_PARITY_DEVICE=cpu \ |
| ORT_PARITY_DEVICE=cpu \ |
| uv run openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --normal |
| ``` |
|
|
| `PYTORCH_PARITY_DEVICE` selects the reference-model device. |
| `ORT_PARITY_DEVICE` selects the ONNX Runtime provider. The command reruns the |
| fast tests and then compares FP32 logits and probabilities at lengths 63, 64, |
| 65, 511, 512, 513, 2,048, and 8,192. It requires `rtol=1e-3`, `atol=1e-4`, and |
| identical winning slots. |
|
|
| On Apple Silicon, ONNX must remain on CPU because the required Microsoft |
| contrib operators do not have an ONNX Runtime MPS implementation. Check whether |
| the installed PyTorch build can use MPS: |
|
|
| ```bash |
| uv run python -c 'import torch; print(torch.backends.mps.is_available())' |
| ``` |
|
|
| If this prints `True`, the PyTorch reference half may use the Apple GPU: |
|
|
| ```bash |
| PYTORCH_ENABLE_MPS_FALLBACK=1 \ |
| PYTORCH_PARITY_DEVICE=mps \ |
| ORT_PARITY_DEVICE=cpu \ |
| uv run openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --normal |
| ``` |
|
|
| `PYTORCH_ENABLE_MPS_FALLBACK=1` permits unsupported PyTorch operations to fall |
| back to CPU. If the availability check prints `False`, use the all-CPU command. |
|
|
| ## 7. Validate the dynamic 262,144-token structure |
|
|
| ```bash |
| uv run openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --structural |
| ``` |
|
|
| This runs the ONNX checker, creates an ONNX Runtime CPU session, verifies the |
| symbolic public dimensions, required contrib operator counts, dynamic position |
| construction, internal state, manifest boundary, and artifact checksums. It |
| does not attempt a 262,144-token numerical inference. Lengths above 65,536 are |
| structurally validated only. |
|
|
| ## 8. Prepare a CUDA host for the slow certification |
|
|
| The 65,536-token parity test is high-memory and should run on an NVIDIA Linux |
| or Windows machine. CUDA cannot run on macOS. |
|
|
| First create the environment as in step 1. Then replace the CPU ONNX Runtime |
| wheel with its GPU counterpart: |
|
|
| ```bash |
| uv pip uninstall onnxruntime |
| uv pip install onnxruntime-gpu==1.30.0 |
| ``` |
|
|
| The first command removes the CPU package. The second installs the build that |
| provides `CUDAExecutionProvider`. |
|
|
| Verify both CUDA stacks before starting the expensive test: |
|
|
| ```bash |
| uv run --no-sync python -c \ |
| 'import torch, onnxruntime as ort; print("torch CUDA:", torch.cuda.is_available()); print("ORT providers:", ort.get_available_providers())' |
| ``` |
|
|
| The output must include `torch CUDA: True` and `CUDAExecutionProvider`. |
| `--no-sync` is required after the manual wheel replacement; plain `uv run` may |
| synchronize from the lockfile and restore CPU-only `onnxruntime`. |
|
|
| ## 9. Run the slow 65,536-token parity test |
|
|
| ```bash |
| PYTORCH_PARITY_DEVICE=cuda \ |
| ORT_PARITY_DEVICE=cuda \ |
| uv run --no-sync openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --slow |
| ``` |
|
|
| This runs one numerical comparison at exactly 65,536 tokens. To reduce peak |
| GPU memory, the test executes PyTorch first, copies only its decision outputs |
| to CPU, deletes the reference model and clears its CUDA cache, and then creates |
| the ONNX Runtime session. Thus the two full model copies are not resident on |
| the GPU simultaneously. |
|
|
| If the process exits with signal 9, the host or driver likely exhausted |
| memory. Move the test to a larger-memory CUDA machine. If CUDA is requested on |
| macOS or a CPU-only ONNX Runtime installation, the validator stops before |
| pytest and explains how to correct the environment. |
|
|
| ## 10. Optionally run all validation stages together |
|
|
| After preparing the CUDA environment: |
|
|
| ```bash |
| PYTORCH_PARITY_DEVICE=cuda \ |
| ORT_PARITY_DEVICE=cuda \ |
| uv run --no-sync openthai-systemone-validate \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --all |
| ``` |
|
|
| This runs client/exporter tests, normal parity, slow parity, and structural |
| validation in sequence. Each stage records its command, timestamp, return code, |
| and test counts. The command stops at the first failed stage. |
|
|
| ## 11. Check the upload gate without uploading |
|
|
| ```bash |
| uv run --no-sync openthai-systemone-upload \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX |
| ``` |
|
|
| This is a local dry run. It requires every validation stage to have at least |
| one executed test, zero skips, and a passing result. It then recomputes all |
| manifest hashes. It does not contact or modify Hugging Face. |
|
|
| ## 12. Upload the validated artifact |
|
|
| ```bash |
| uv run --no-sync openthai-systemone-upload \ |
| --artifact artifacts/OpenThai-SystemOne-ONNX \ |
| --repo-id imtk/OpenThai-SystemOne-ONNX \ |
| --execute |
| ``` |
|
|
| This repeats the gate check, creates the Hugging Face model repository if it |
| does not exist, and uploads the artifact directory. `--execute` is the switch |
| that permits the external write; without it the command remains a dry run. |
|
|
| ## Exported tensor contract |
|
|
| All inputs are `int64`; outputs are `float32`. |
|
|
| | Tensor | Shape | |
| |---|---| |
| | `input_ids` | `[batch, sequence]` | |
| | `attention_mask` | `[batch, sequence]` | |
| | `answer_positions` | `[batch, questions]` | |
| | `option_counts` | `[batch, questions]` | |
| | `qtypes` | `[batch, questions]` | |
| | `logits` | `[batch, questions, 256]` | |
| | `probabilities` | `[batch, questions, 256]` | |
|
|
