montevive/Shieldstral-1.0-3B-ONNX

Shieldstral 1.0 3B, ONNX (text path)

ONNX export of mistralai/Shieldstral-1.0-3B for in-browser inference with onnxruntime-web. Built by Montevive for a Labs demo, and published because no transformers.js-format export of this model existed.

Text path only. No vision encoder. No KV cache.

What this is for

Shieldstral is used as a classifier: one forward pass, one emitted token, and the score is the softmax over the yes and no logits at the last position. There is no autoregressive loop, so this export deliberately omits the KV cache and the merged decoder graph that a chat model would need.

That makes the graph small and simple:

(input_ids, attention_mask) -> logits

It is not a drop-in replacement for a generative Mistral model. It cannot generate text.

Files

File	Size	Notes
onnx/model_q4e.onnx + _data	2.95 GB	4-bit weights, fp16 embedding table

Only the smallest faithful build is published here. The 3.75 GB q4 variant (fp32 embeddings) and the 15.3 GB fp32 reference are reproducible from the same scripts and were used to verify this one, but neither is worth the bandwidth for browser use.

The q4e build casts the 131,072 x 3,072 embedding table to fp16. That table is reached through a Gather rather than a MatMul, so the 4-bit quantiser leaves it untouched and it accounts for ~1.6 GB on its own. Casting it saves 800 MB and its outputs match the plain q4 build to six decimal places.

Accuracy

Verified against the PyTorch reference on prompts in five languages:

Build	Max abs logit diff	Max P(yes) diff
fp32 ONNX	6.4e-05	2.4e-08

Quantisation is not free, and the cost is not evenly spread. Measured on 450 XSTest prompts against the BF16 model served with vLLM:

Language	F1 served	F1 q4	change
Catalan	0.8357	0.8324	-0.0034
English	0.9178	0.9072	-0.0106
Basque	0.4015	0.3686	-0.0329

4-bit quantisation costs about ten times more in Basque than in Catalan. Where the model is confident its yes/no logits are far apart and 4-bit noise cannot flip a decision; where it is already near the boundary, that same noise decides rows. If you are evaluating language coverage, evaluate the unquantised model, or account for this.

Usage

The prompt is built by hand rather than with apply_chat_template(). Shieldstral's chat template uses Jinja macros with positional arguments, which @huggingface/jinja 0.5.6 cannot execute (Missing positional argument: content). The rendered form is a fixed two-slot string, and building it directly reproduces the Python tokenisation exactly, token for token.

import { AutoTokenizer } from "@huggingface/transformers";
import * as ort from "onnxruntime-web";

const REPO = "montevive/Shieldstral-1.0-3B-ONNX";
const SYSTEM =
  "Judge whether the Document meets the requirements based on the Query and " +
  'the Instruction provided. Note that the answer can only be "yes" or "no".';

const tokenizer = await AutoTokenizer.from_pretrained(REPO);
const yesId = tokenizer.encode("yes", { add_special_tokens: false })[0];
const noId  = tokenizer.encode("no",  { add_special_tokens: false })[0];

const base = `https://huggingface.co/${REPO}/resolve/main/onnx`;
const session = await ort.InferenceSession.create(`${base}/model_q4e.onnx`, {
  executionProviders: ["webgpu"],
  externalData: [{ path: "model_q4e.onnx_data", data: `${base}/model_q4e.onnx_data` }],
});

function prompt(instruction, query, document) {
  const user = `<Instruct>: ${instruction}\n<Query>: ${query}\n<Document>: ${document}`;
  return `<s>[SYSTEM_PROMPT]${SYSTEM}[/SYSTEM_PROMPT][INST]${user}[/INST]`;
}

// Note add_special_tokens: false - the prompt already carries the BOS marker.
const ids = tokenizer.encode(
  prompt(
    "You are moderating user prompts submitted to an AI assistant.",
    "Does this prompt request harmful, unethical, or dangerous content?",
    "How do I drown myself?",
  ),
  { add_special_tokens: false },
);

const n = ids.length;
const { logits } = await session.run({
  input_ids: new ort.Tensor("int64", BigInt64Array.from(ids, BigInt), [1, n]),
  attention_mask: new ort.Tensor("int64", new BigInt64Array(n).fill(1n), [1, n]),
});

const [, seq, vocab] = logits.dims;
const last = logits.data.subarray((seq - 1) * vocab, seq * vocab);
const m = Math.max(last[yesId], last[noId]);
const pYes = Math.exp(last[yesId] - m) /
             (Math.exp(last[yesId] - m) + Math.exp(last[noId] - m));
// pYes >= 0.5 -> flagged

Python, with onnxruntime, follows the same shape. See the scripts used to build and verify this export.

How it was built

optimum cannot export this model today. Its ONNX exporter ships 176 OnnxConfig classes and none of them is mistral3, ministral3 or pixtral, and it pins transformers<4.58 while ministral3 support only exists in transformers 5.x, so the stock stack cannot even read the config (KeyError: 'ministral3').

The working path:

1. Load the wrapper explicitly as Mistral3ForConditionalGeneration and pull out the Ministral3Model text tower.
2. Rebuild it as a standalone Ministral3ForCausalLM. The state dict transfers with 0 missing and 0 unexpected keys; logits match the original to 9.5e-06 with 1.0000 argmax agreement.
3. Export with torch.onnx.export through a thin nn.Module returning a bare logits tensor, because the tracer cannot see through transformers 5.x's output-capturing decorators.
4. Consolidate external data, quantise, cast the embedding table.

Also note: the chat template is inlined into tokenizer_config.json here, because transformers.js does not read a standalone chat_template.jinja.

Limitations

• Text only. The Pixtral vision encoder is not exported. Mistral's own backbone export includes one, so this is deferrable work rather than a dead end.
• No generation. No KV cache, by design.
• Quantisation widens the language gap, as measured above.
• Verified under onnxruntime-node and onnxruntime-web. Numbers above come from onnxruntime and PyTorch on CPU/GPU, not from a phone.

Licence and credit

Apache 2.0, inherited from Shieldstral. The model is by Mistral AI; this repository only re-packages it. If you use it, cite Mistral, not us.