INSTRUCTIONS.md

Convert and quantize OpenThai-SystemOne for ONNX Runtime

These are the maintainer instructions for converting the pinned iapp/OpenThai-SystemOne checkpoint, validating the result, and uploading it to Hugging Face. For application-side OnnxSystemOneClient usage, see README.md.

Run every command from the repository root. The conversion and release code is kept in src/openthai_systemone_onnx_tools; the independent runtime client is in src/openthai_systemone_onnx.

1. Create the conversion environment

uv python pin 3.12
uv sync --extra tools --extra dev --frozen

The first command selects Python 3.12. The second creates .venv, installs the runtime package, adds the tools dependencies used to build and inspect ONNX, and adds pytest from the dev extra. --frozen makes the operation fail if uv.lock and pyproject.toml disagree instead of silently changing versions.

Important pinned versions include ONNX Runtime 1.30.0, ONNX Runtime GenAI 0.16.0, source model revision f3709948b5e3cc9606a57e74ba62b7a639d17dd3, and upstream client commit 5d04bcca0c58bd10e7dac2d3d369d8f760bea6cf.

2. Authenticate with Hugging Face

uv run hf auth login

This interactively stores a Hugging Face token. Downloading the public source does not require write access; uploading the finished artifact does.

3. Export the model

uv run openthai-systemone-export \
  --output artifacts/OpenThai-SystemOne-ONNX \
  --cache-dir .cache/huggingface

This command downloads the pinned checkpoint, loads its safetensor weights, builds the dynamic Qwen3.5 graph, appends the calibrated SystemOne head, and writes:

• onnx/model.onnx for graph structure;
• onnx/model.onnx.data for external FP32 weights;
• tokenizer files and the separated helper packages;
• license, attribution, model card, and usage documentation;
• artifact-manifest.json with revisions, limits, tensor schema, operator requirements, and checksums.

The graph keeps symbolic batch, sequence, and question dimensions. It derives MRoPE positions from attention_mask and initializes KV, convolution, and recurrent state internally for every complete request.

4. Add INT8 and INT4 variants

uv run openthai-systemone-quantize \
  --artifact artifacts/OpenThai-SystemOne-ONNX

This adds onnx/model.int8.onnx and onnx/model.int4.onnx plus their external data files. INT8 uses dynamic per-channel quantization. INT4 uses asymmetric 32-value weight blocks and MatMulNBits. Both leave the calibrated decision head in FP32. The command loads both variants with the CPU execution provider before installing them and then rewrites the bundled manifest and checksums. It removes an existing validation report because that report refers to the previous manifest.

5. Run fast compatibility and quantized runtime tests

uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --client

This checks upstream constructor/method signatures, response-class identity, serialization, typed and raw questions, state formats, validation errors, single/batch equivalence, automatic and explicit permutations, accounting, exporter graph construction, and validator device diagnostics. It records the result as client_compatibility in validation-report.json.

Then validate both quantized graphs on CPU:

uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --quantized

This checks all graph structures and runs raw-tensor and end-to-end client inference for INT8 and INT4. Quantized validation checks runtime correctness, not exact FP32 numerical parity.

6. Run normal PyTorch/ONNX parity

On CPU, including macOS:

PYTORCH_PARITY_DEVICE=cpu \
ORT_PARITY_DEVICE=cpu \
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --normal

PYTORCH_PARITY_DEVICE selects the reference-model device. ORT_PARITY_DEVICE selects the ONNX Runtime provider. The command reruns the fast tests and then compares FP32 logits and probabilities at lengths 63, 64, 65, 511, 512, 513, 2,048, and 8,192. It requires rtol=1e-3, atol=1e-4, and identical winning slots.

On Apple Silicon, ONNX must remain on CPU because the required Microsoft contrib operators do not have an ONNX Runtime MPS implementation. Check whether the installed PyTorch build can use MPS:

uv run python -c 'import torch; print(torch.backends.mps.is_available())'

If this prints True, the PyTorch reference half may use the Apple GPU:

PYTORCH_ENABLE_MPS_FALLBACK=1 \
PYTORCH_PARITY_DEVICE=mps \
ORT_PARITY_DEVICE=cpu \
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --normal

PYTORCH_ENABLE_MPS_FALLBACK=1 permits unsupported PyTorch operations to fall back to CPU. If the availability check prints False, use the all-CPU command.

7. Validate the dynamic 262,144-token structure

uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --structural

This runs the ONNX checker, creates an ONNX Runtime CPU session, verifies the symbolic public dimensions, required contrib operator counts, dynamic position construction, internal state, manifest boundary, and artifact checksums. It does not attempt a 262,144-token numerical inference. Lengths above 65,536 are structurally validated only.

8. Prepare a CUDA host for the slow certification

The 65,536-token parity test is high-memory and should run on an NVIDIA Linux or Windows machine. CUDA cannot run on macOS.

First create the environment as in step 1. Then replace the CPU ONNX Runtime wheel with its GPU counterpart:

uv pip uninstall onnxruntime
uv pip install onnxruntime-gpu==1.30.0

The first command removes the CPU package. The second installs the build that provides CUDAExecutionProvider.

Verify both CUDA stacks before starting the expensive test:

uv run --no-sync python -c \
  'import torch, onnxruntime as ort; print("torch CUDA:", torch.cuda.is_available()); print("ORT providers:", ort.get_available_providers())'

The output must include torch CUDA: True and CUDAExecutionProvider. --no-sync is required after the manual wheel replacement; plain uv run may synchronize from the lockfile and restore CPU-only onnxruntime.

9. Run the slow 65,536-token parity test

PYTORCH_PARITY_DEVICE=cuda \
ORT_PARITY_DEVICE=cuda \
uv run --no-sync openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --slow

This runs one numerical comparison at exactly 65,536 tokens. To reduce peak GPU memory, the test executes PyTorch first, copies only its decision outputs to CPU, deletes the reference model and clears its CUDA cache, and then creates the ONNX Runtime session. Thus the two full model copies are not resident on the GPU simultaneously.

If the process exits with signal 9, the host or driver likely exhausted memory. Move the test to a larger-memory CUDA machine. If CUDA is requested on macOS or a CPU-only ONNX Runtime installation, the validator stops before pytest and explains how to correct the environment.

10. Optionally run all validation stages together

After preparing the CUDA environment:

PYTORCH_PARITY_DEVICE=cuda \
ORT_PARITY_DEVICE=cuda \
uv run --no-sync openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --all

This runs client/exporter tests, normal parity, slow parity, and structural validation in sequence. Each stage records its command, timestamp, return code, and test counts. The command stops at the first failed stage.

11. Check the upload gate without uploading

uv run --no-sync openthai-systemone-upload \
  --artifact artifacts/OpenThai-SystemOne-ONNX

This is a local dry run. It requires every validation stage to have at least one executed test, zero skips, and a passing result. It then recomputes all manifest hashes. It does not contact or modify Hugging Face.

12. Upload the validated artifact

uv run --no-sync openthai-systemone-upload \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --repo-id imtk/OpenThai-SystemOne-ONNX \
  --execute

This repeats the gate check, creates the Hugging Face model repository if it does not exist, and uploads the artifact directory. --execute is the switch that permits the external write; without it the command remains a dry run.

Exported tensor contract

All inputs are int64; outputs are float32.

Tensor	Shape
input_ids	[batch, sequence]
attention_mask	[batch, sequence]
answer_positions	[batch, questions]
option_counts	[batch, questions]
qtypes	[batch, questions]
logits	[batch, questions, 256]
probabilities	[batch, questions, 256]