altertable/vela3-fast-onnx-int8

vela3-fast ONNX int8 (decisions on CPU)

A CPU-friendly ONNX version of the decision part of vllm-sr/vela3-fast, the 0.4B encoder of the Vela 3.0 family by KR Labs and vLLM Semantic Router.

• Same weights, small download, fp32 speed. The encoder projections are stored as 8-bit integers and the embedding as fp16, so the file is 0.45 GB (the fp32 graph is 1.52 GB). ONNX Runtime turns them back into fp32 when the model loads, so inference runs the same fused fp32 kernels as the full-precision graph. We did no fine-tuning, distillation or calibration.
• One graph. Encoder and readout head together: token ids in, one logit per option out. Short questions share a pass.
• Long inputs. Up to vela3-fast's 32,768-token context. Past 4,096 tokens, memory grows linearly with the input.
• No PyTorch. Runs with onnxruntime, tokenizers and numpy.
• Answers two question types over a text or JSON state: choice (a probability per option) and noul (yes/no, gives p(yes)).

All credit for the model goes to the vela3-fast authors. vela3-fast is itself fine-tuned from dmis-lab/UltraBERT-380M. See NOTICE and LICENSES.md.

Files

file	what it is
model/model.onnx	encoder + readout head, int8 encoder projections, fp16 embedding (452 MB)
model/tokenizer.json	vela3-fast's tokenizer (copied unmodified)
vela_onnx.py	the runner: DecisionModel (onnxruntime + tokenizers + numpy)
ub_d3_common.py	how a question is turned into model input (copied unmodified from vela3-fast)
example.py	a small tested example
decision_index_engine.py	an engine for the Decision Index kit
make_int8.py	how model/model.onnx was made from the fp32 graph
requirements.txt	onnxruntime>=1.24, tokenizers>=0.20, numpy>=1.26
LICENSE, NOTICE, LICENSES.md, DATA_CARD.md	licence (Apache-2.0), attribution and provenance

Quick start

hf download altertable/vela3-fast-onnx-int8 --local-dir vela3-fast-onnx-int8
cd vela3-fast-onnx-int8
pip install -r requirements.txt      # onnxruntime, tokenizers, numpy
python example.py
# billing {'billing': 0.996, 'tech': 0.001, 'sales': 0.004}
# urgent p(yes) = 0.843 | ... ms

In Python (run from the downloaded folder, or add it to sys.path):

from vela_onnx import DecisionModel

m = DecisionModel("model", threads=8)   # folder with model.onnx and tokenizer.json
out = m.decide(
    {"ticket": "I was charged twice"},   # state: a string or a JSON object
    {
        "route": {"type": "choice", "instructions": "Which team should handle this ticket?",
                  "criteria": {"billing": "Payments, refunds, double charges",
                               "tech": "Bugs and login problems"}},
        "refund": {"type": "noul", "instructions": "Does the customer ask for money back?"},
    },
)
out["answers"]["route"]["choice"]          # "billing"
out["answers"]["route"]["probabilities"]   # {"billing": ..., "tech": ...}
out["answers"]["refund"]["noul"]           # p(yes)

Tips:

• Give each option a short description of what belongs in it ("billing": "Payments, refunds, double charges"). Bare option names ("billing", "tech") are answered less reliably; the full-precision model behaves the same way.
• Each question is its own prompt, [Q] question [O] option ... [SEP_SCHEMA] [SEG_user] state; the state is repeated per question. Prompts of at most 129 tokens share forward passes of about 128 tokens, each prompt attending only to itself. The CPU's matrix units are much more efficient on that than on short passes: several questions about a short state take 1.3 to 1.5 times less time. Longer prompts run one per pass.
• Inputs up to 32,768 tokens per question. Longer inputs raise vela_onnx.Unsupported; nothing is cut.
• Other question types (for example span) raise Unsupported.
• DecisionModel(..., low_memory=True) keeps the weights in int8 at run time (ONNX Runtime's MatMulNBits): same answers, a peak of about 1.3 GB of RAM instead of 2.2 GB, but several times slower.

Decision Index kit: PYTHONPATH=. python -m decision_index run --engine decision_index_engine:VelaOnnxEngine --option threads=8 ...

Using the ONNX file directly

	name	type
input	input_ids	int64 [1, T]	one or more prompts end to end, each <|endoftext|> [Q] question ([O] option)+ [SEP_SCHEMA] [SEG_user] state <|endoftext|>
input	position_ids	int64 [1, T]	each token's position within its prompt (0, 1, ...)
input	segment_ids	int64 [1, T]	each token's prompt: 0, 1, ... in order (all 0 for one prompt)
input	positions	int64 [K]	the position of the [O] before each option, over all prompts
input	questions	int64 [K]	the position of the [Q] of each option's prompt
input	state_starts	int64 [K]	the position of the [SEG_user] of each option's prompt
input	state_ends	int64 [K]	the position of the final <|endoftext|> of each option's prompt
output	logits	float32 [K]	a softmax over the options of each prompt gives its probabilities

Several prompts in one pass must each be at most 129 tokens: one sliding window then spans each whole prompt, and the graph keeps each prompt's attention to itself. A single prompt can be up to 32,768 tokens. vela_onnx.DecisionModel._probs builds these inputs.

The markers are token ids 49152 ([Q]), 49153 ([O]), 49155 ([SEP_SCHEMA]) and 49156 ([SEG_user]). The graph uses ONNX Runtime contrib ops (com.microsoft.MultiHeadAttention, RotaryEmbedding), so it needs ONNX Runtime. Create the session with session.disable_quant_qdq = 1:

so = onnxruntime.SessionOptions()
so.add_session_config_entry("session.disable_quant_qdq", "1")

Without it, ONNX Runtime fuses the int8 weights into MatMulNBits kernels, which use less memory but are several times slower on CPU, and quantise the activations as well unless you also set session.qdq_matmulnbits_accuracy_level = 0.

How it was made

1. The decide_onnx DuckDB extension exports vela3-fast's decision backbone (encoder and readout head) as one fp32 graph (scripts/export/export_onnx.py, asset v5). It maps the network onto ONNX Runtime's fused kernels: MultiHeadAttention for the 8 global layers, blocked attention for the 16 sliding-window layers (memory linear in the input), fused RoPE and RMSNorm. Up to 4,096 tokens, the global layers pass MultiHeadAttention an all-ones key_padding_mask. It masks nothing, but makes ONNX Runtime compute the scores with its GEMM (KleidiAI SME on arm64) instead of its flash attention, whose GEMMs are plain NEON: about twice as fast, for at most 512 MB of scores. Longer inputs take the flash attention. Several short prompts in one pass use MultiHeadAttention with a block-diagonal attention_bias in every layer. Each export is checked against the unmodified PyTorch DecisionEncoder on prompts of 10 to 9,006 tokens and on packs of 3 to 9 prompts; the worst logit difference is about 1e-5, and a prompt in a pack is within 5e-6 of its own pass.
2. make_int8.py changes only how the weights are stored. The 168 encoder projections become int8: symmetric, blocks of 128 along the input dimension, the scales of onnxruntime's 8-bit MatMulNBits quantizer, read back by a blocked DequantizeLinear. The token embedding becomes fp16. The readout head stays fp32.
3. With session.disable_quant_qdq = 1, ONNX Runtime constant-folds those weights back into fp32 when it loads the model.

We tried and did not keep:

• Running the int8 weights directly (MatMulNBits, which the previous version of this repository shipped): the same answers, but 8 times slower on one CPU core for a short input, and still 1.2 times slower at 8K tokens.
• Dynamic int8 (QDQ) quantisation: this encoder has very large activations on some tokens, and dynamic int8 broke its outputs on those tokens.
• An int8 embedding (per row, blocks of 128): 40 MB smaller, but it doubled the mean probability difference.
• 4 bits: agreed with full precision on only 94.6% of answers.

Fidelity

Compared with the fp32 graph (which matches PyTorch to about 1e-5) on 214 prompts: the 14 parity cases of the DuckDB extension, and 40 English paragraphs under 5 questions each (routing, tone, yes/no, language):

	int8 ONNX vs fp32
same answer (argmax)	213 / 214 (the other one is 0.504 vs 0.496 in fp32)
mean absolute difference in probability	0.0011
99th percentile of that difference	0.011
largest difference	0.018

Against the published PyTorch code itself (DecisionEncoder.predict, the extension's 14 parity cases), the largest difference in probability is 0.0072.

These were measured on an earlier file, whose global layers always used the flash attention and which ran one question per pass. The current one has the same weights and gives the same probabilities to within 3e-6, packed or not.

Scores

The model's decision quality is vela3-fast's. These are upstream figures for the full-precision vela3-fast, not a run of this int8 export:

Jev Decision Index board, vela3-fast (full precision)	score
Public index	50.36
Knowledge	21.2
Language	53.1
Retrieval	66.9
Tools	73.7
Arts	43.1

The vela3-fast card reports the same public index (50.36) from its authors' own run of the unmodified kit (edition 0.3). The int8 export gives the same answer as full precision on 213 of the 214 prompts above, so we expect it to score the same, but we have not run the full Index with it.

CPU latency

One question per pass, measured on an Apple M4 Pro with onnxruntime 1.31 (your numbers will vary):

input	1 thread	8 threads	peak RAM
32 tokens	30 ms	14 ms	2.2 GB
512 tokens	0.36 s	0.20 s	2.2 GB
2,048 tokens	1.6 s	0.92 s	2.3 GB
8,192 tokens	27.9 s	6.4 s	3.2 GB

Several questions about a short state share passes: the example's 2 questions take 92 ms instead of 123 ms one per pass, and 5 questions 175 ms instead of 260 ms (1 thread).

With the flash attention in every global layer, an earlier file took 0.43 s at 512 tokens and 2.8 s at 2,048 on 1 thread.

The model loads in under a second. The version before that of this repository (int8 MatMulNBits encoder, dense attention, head in numpy) took 272 ms / 0.64 s / 2.4 s on 1 thread for the same inputs, and 8.2 GB of RAM at 8,192 tokens.

On onnxruntime 1.19 to 1.23 the model gives the same answers but runs several times slower: their macOS arm64 wheels lack the KleidiAI matmul kernels, and the fp32 graph is just as slow there.

Limitations

• English only, like vela3-fast.
• Decisions only. This package contains only the decision (choice / noul) path of vela3-fast. The span, NER, relation, PII, evidence and router-signal adapters of vela3-fast are not included. Use vllm-sr/vela3-fast for those.
• CPU-oriented. On GPU, the original vela3-fast is much faster (its card reports ~14 ms per decision at 200 tokens).
• Provenance. We did not train these weights. Per its model card, vela3-fast's decision backbone was distilled on the d3 decision models' soft labels, on train splits of public datasets written as decisions (for example ANLI and RAGTruth), and on synthetic decision tasks in the styles the Decision Index covers. Its authors state that no test split and no Decision Index item was trained on. Some of those public train splits come from the same source datasets as Decision Index benchmarks, so treat Decision Index results as in-distribution for this model rather than fully held-out. See DATA_CARD.md.

Licence

Apache-2.0, as vela3-fast and UltraBERT-380M. See LICENSE, NOTICE and LICENSES.md.

Citation

Please cite the upstream model:

@misc{vela3_fast,
  title        = {vela3-fast (Vela 3.0, 0.4B): decisions, spans and router signals in one UltraBERT-380M encoder},
  author       = {{KR Labs} and {vLLM Semantic Router}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/vela3-fast}}
}