d3-nano-onnx

An ONNX Runtime CPU build of the text path of vllm-sr/d3-nano at revision 462efc366229ea268cf4eed3738b3fc4415e4817. It answers Choice, Yes / No and Score questions with no torch and no transformers, and is the local backend of jev-lite (LocalTypeSafe): 0.0.5 and 0.0.6 use the one-sequence graph, 0.0.7 the state-once graph.

Weights	1.76 GiB: MatMul int8 and embeddings int4 (block size 32), fp32 readout
Inputs	Text only (the vision tower is removed)
Runtime	onnxruntime CPU provider, opset 21
License	Apache-2.0, as upstream; see LICENSE and NOTICE

Graph

input_ids int64 [1, L] → logits float32 [1, 255]. These are the code logits at the last prompt position, before the option mask and softmax. The graph contains the token embeddings, upstream's noncausal prompt encoder over the 24 Qwen3.5 text layers, and the 255-way readout. The length is dynamic, from 2 to 8,192 tokens.

Prompts are rendered as upstream does: chat_template.jinja, tokenizer.json and the answer codes in decision_config.json. To get the probabilities, keep the first n logits for an n-option question, divide by the temperature in decision_config.json (1.0), and apply a softmax.

How it was built

Built with export/d3_nano_onnx_export.py (in this repo, from jev-lite commit 3261b0af449d09b7951e39d7396b09a6a221672d): torch 2.14.0, transformers 5.17.0, onnxruntime 1.31.0. export/self_host_portability.py holds the 50 check requests the script imports.

• The export rewrites transformers' Gated DeltaNet reference code without Python loops. The triangular solve becomes a recursive block inverse, and the chunk recurrence becomes a Scan. The math is unchanged, and the torch logits are identical before and after.
• Weights are quantized weight-only, symmetric, at accuracy level 4.

Quality

On semantic-decisions@v0.0.2 (252 zh-Hant decisions × 2 option orders, jev-lite's evaluation bank), on the CPU:

Build	All /504	Hard /336	Same choices as bf16 d3-nano
bf16 d3-nano (upstream, torch)	449	296	—
this build	449	296	504/504

On 50 short zh-Hant requests on 20 Arm cores, p50 is 504 ms and p95 866 ms. Latency grows with the prompt length. Only zh-Hant was measured.

State-once graph

d3-nano-text-cached-int8-embedding-int4.onnx holds the same weights, quantized the same way, with each layer's past state as an extra input and output. One call reads the state and returns its past states; each later call scores a batch of questions on them:

• inputs: input_ids int64 [rows, width], lengths int64 [rows], and the past_* states (past_conv_*, past_recurrent_*, past_keys_*, past_values_*, length 0 for the first call);
• outputs: logits float32 [rows, 255] at each row's last real token, and the present_* states.

In the full-attention layers the state does not see the questions; each question sees the state and itself, at the positions it would have alone. For d3-nano this changes no score. Built with export/d3_nano_onnx_export.py export-cached from jev-lite commit 4223d381195f6b58a0308e69c15d9f80a008d023, then quantize.

Build	All /504	Hard /336	Same choices as bf16 d3-nano
state-once graph, one question per request	449	296	492/504

The state is read once per request instead of once per question. On 20 Arm cores with a 1,991-token state, 255 questions take 10.7× the time of one (about 22× faster than one call per question).

Files

MANIFEST.json lists the sha256 of every file.