# Convert and quantize OpenThai-SystemOne for ONNX Runtime

These are the maintainer instructions for converting the pinned
`iapp/OpenThai-SystemOne` checkpoint, validating the result, and uploading it
to Hugging Face. For application-side `OnnxSystemOneClient` usage, see
[README.md](README.md).

Run every command from the repository root. The conversion and release code is
kept in `src/openthai_systemone_onnx_tools`; the independent runtime client is
in `src/openthai_systemone_onnx`.

## 1. Create the conversion environment

```bash
uv python pin 3.12
uv sync --extra tools --extra dev --frozen
```

The first command selects Python 3.12. The second creates `.venv`, installs the
runtime package, adds the `tools` dependencies used to build and inspect ONNX,
and adds pytest from the `dev` extra. `--frozen` makes the operation fail if
`uv.lock` and `pyproject.toml` disagree instead of silently changing versions.

Important pinned versions include ONNX Runtime 1.30.0, ONNX Runtime GenAI
0.16.0, source model revision
`f3709948b5e3cc9606a57e74ba62b7a639d17dd3`, and upstream client commit
`5d04bcca0c58bd10e7dac2d3d369d8f760bea6cf`.

## 2. Authenticate with Hugging Face

```bash
uv run hf auth login
```

This interactively stores a Hugging Face token. Downloading the public source
does not require write access; uploading the finished artifact does.

## 3. Export the model

```bash
uv run openthai-systemone-export \
  --output artifacts/OpenThai-SystemOne-ONNX \
  --cache-dir .cache/huggingface
```

This command downloads the pinned checkpoint, loads its safetensor weights,
builds the dynamic Qwen3.5 graph, appends the calibrated SystemOne head, and
writes:

- `onnx/model.onnx` for graph structure;
- `onnx/model.onnx.data` for external FP32 weights;
- tokenizer files and the separated helper packages;
- license, attribution, model card, and usage documentation;
- `artifact-manifest.json` with revisions, limits, tensor schema, operator
  requirements, and checksums.

The graph keeps symbolic batch, sequence, and question dimensions. It derives
MRoPE positions from `attention_mask` and initializes KV, convolution, and
recurrent state internally for every complete request.

## 4. Add INT8 and INT4 variants

```bash
uv run openthai-systemone-quantize \
  --artifact artifacts/OpenThai-SystemOne-ONNX
```

This adds `onnx/model.int8.onnx` and `onnx/model.int4.onnx` plus their external
data files. INT8 uses dynamic per-channel quantization. INT4 uses asymmetric
32-value weight blocks and `MatMulNBits`. Both leave the calibrated decision
head in FP32. The command loads both variants with the CPU execution provider
before installing them and then rewrites the bundled manifest and checksums.
It removes an existing validation report because that report refers to the
previous manifest.

## 5. Run fast compatibility and quantized runtime tests

```bash
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --client
```

This checks upstream constructor/method signatures, response-class identity,
serialization, typed and raw questions, state formats, validation errors,
single/batch equivalence, automatic and explicit permutations, accounting,
exporter graph construction, and validator device diagnostics. It records the
result as `client_compatibility` in `validation-report.json`.

Then validate both quantized graphs on CPU:

```bash
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --quantized
```

This checks all graph structures and runs raw-tensor and end-to-end client
inference for INT8 and INT4. Quantized validation checks runtime correctness,
not exact FP32 numerical parity.

## 6. Run normal PyTorch/ONNX parity

On CPU, including macOS:

```bash
PYTORCH_PARITY_DEVICE=cpu \
ORT_PARITY_DEVICE=cpu \
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --normal
```

`PYTORCH_PARITY_DEVICE` selects the reference-model device.
`ORT_PARITY_DEVICE` selects the ONNX Runtime provider. The command reruns the
fast tests and then compares FP32 logits and probabilities at lengths 63, 64,
65, 511, 512, 513, 2,048, and 8,192. It requires `rtol=1e-3`, `atol=1e-4`, and
identical winning slots.

On Apple Silicon, ONNX must remain on CPU because the required Microsoft
contrib operators do not have an ONNX Runtime MPS implementation. Check whether
the installed PyTorch build can use MPS:

```bash
uv run python -c 'import torch; print(torch.backends.mps.is_available())'
```

If this prints `True`, the PyTorch reference half may use the Apple GPU:

```bash
PYTORCH_ENABLE_MPS_FALLBACK=1 \
PYTORCH_PARITY_DEVICE=mps \
ORT_PARITY_DEVICE=cpu \
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --normal
```

`PYTORCH_ENABLE_MPS_FALLBACK=1` permits unsupported PyTorch operations to fall
back to CPU. If the availability check prints `False`, use the all-CPU command.

## 7. Validate the dynamic 262,144-token structure

```bash
uv run openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --structural
```

This runs the ONNX checker, creates an ONNX Runtime CPU session, verifies the
symbolic public dimensions, required contrib operator counts, dynamic position
construction, internal state, manifest boundary, and artifact checksums. It
does not attempt a 262,144-token numerical inference. Lengths above 65,536 are
structurally validated only.

## 8. Prepare a CUDA host for the slow certification

The 65,536-token parity test is high-memory and should run on an NVIDIA Linux
or Windows machine. CUDA cannot run on macOS.

First create the environment as in step 1. Then replace the CPU ONNX Runtime
wheel with its GPU counterpart:

```bash
uv pip uninstall onnxruntime
uv pip install onnxruntime-gpu==1.30.0
```

The first command removes the CPU package. The second installs the build that
provides `CUDAExecutionProvider`.

Verify both CUDA stacks before starting the expensive test:

```bash
uv run --no-sync python -c \
  'import torch, onnxruntime as ort; print("torch CUDA:", torch.cuda.is_available()); print("ORT providers:", ort.get_available_providers())'
```

The output must include `torch CUDA: True` and `CUDAExecutionProvider`.
`--no-sync` is required after the manual wheel replacement; plain `uv run` may
synchronize from the lockfile and restore CPU-only `onnxruntime`.

## 9. Run the slow 65,536-token parity test

```bash
PYTORCH_PARITY_DEVICE=cuda \
ORT_PARITY_DEVICE=cuda \
uv run --no-sync openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --slow
```

This runs one numerical comparison at exactly 65,536 tokens. To reduce peak
GPU memory, the test executes PyTorch first, copies only its decision outputs
to CPU, deletes the reference model and clears its CUDA cache, and then creates
the ONNX Runtime session. Thus the two full model copies are not resident on
the GPU simultaneously.

If the process exits with signal 9, the host or driver likely exhausted
memory. Move the test to a larger-memory CUDA machine. If CUDA is requested on
macOS or a CPU-only ONNX Runtime installation, the validator stops before
pytest and explains how to correct the environment.

## 10. Optionally run all validation stages together

After preparing the CUDA environment:

```bash
PYTORCH_PARITY_DEVICE=cuda \
ORT_PARITY_DEVICE=cuda \
uv run --no-sync openthai-systemone-validate \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --all
```

This runs client/exporter tests, normal parity, slow parity, and structural
validation in sequence. Each stage records its command, timestamp, return code,
and test counts. The command stops at the first failed stage.

## 11. Check the upload gate without uploading

```bash
uv run --no-sync openthai-systemone-upload \
  --artifact artifacts/OpenThai-SystemOne-ONNX
```

This is a local dry run. It requires every validation stage to have at least
one executed test, zero skips, and a passing result. It then recomputes all
manifest hashes. It does not contact or modify Hugging Face.

## 12. Upload the validated artifact

```bash
uv run --no-sync openthai-systemone-upload \
  --artifact artifacts/OpenThai-SystemOne-ONNX \
  --repo-id imtk/OpenThai-SystemOne-ONNX \
  --execute
```

This repeats the gate check, creates the Hugging Face model repository if it
does not exist, and uploads the artifact directory. `--execute` is the switch
that permits the external write; without it the command remains a dry run.

## Exported tensor contract

All inputs are `int64`; outputs are `float32`.

| Tensor | Shape |
|---|---|
| `input_ids` | `[batch, sequence]` |
| `attention_mask` | `[batch, sequence]` |
| `answer_positions` | `[batch, questions]` |
| `option_counts` | `[batch, questions]` |
| `qtypes` | `[batch, questions]` |
| `logits` | `[batch, questions, 256]` |
| `probabilities` | `[batch, questions, 256]` |