AndrPixel/opus-mt-mobile-onnx

Opus-MT for phones — twelve directions, INT8 ONNX

Twelve Opus-MT translation directions prepared to run offline on a phone: three quantized ONNX graphs and one binary tokenizer per direction.

Nothing here is a new model. These are conversions of Helsinki-NLP Opus-MT models, published so that a mobile app can download a language pair once instead of shipping a converter, and so the conversion is reproducible by anyone.

Directions

ru-uk · uk-ru	ru-en · en-ru	uk-en · en-uk
de-en · en-de	fr-en · en-fr	it-en · en-it

Directions, not languages: Opus-MT translates one way with one model, so ru-uk and uk-ru are two different sets of files. Ukrainian is covered by direct pairs with Russian rather than through English — on that pair the quality cost of two passes is particularly audible.

Sizes run from 209 MB (ru-uk) to 264 MB (it-en). The spread comes from vocabulary, not architecture: every direction has six layers and d_model 512, but the shared vocabulary of it-en is 80 379 against 58 101 for de-en, and the difference lands in the embedding matrix.

Files in each directory

opus-mt-<pair>/
  encoder_model.onnx              INT8
  decoder_model.onnx              INT8
  decoder_with_past_model.onnx    INT8
  tokenizer.bin                   binary, see below
  manifest.json                   sizes, SHA-256, architecture, source model

Why three graphs and not one

decoder_model_merged does not quantize: the whole body of the graph hides inside a single If node, and the ONNX Runtime quantizer does not descend into subgraphs.

Dropping decoder_model and keeping only the cached decoder does not work either. It accepts an empty cache and returns logits of the right shape — but past_key_values.N.encoder.* holds projected encoder states, not history, and they are the decoder's only link to the input text. On an empty cache the model produces fluent, entirely invented translations.

Why the tokenizer is a binary file

Marian models have no tokenizer.json, and source.spm is a SentencePiece protobuf whose nmt_nfkc normalizer is a compiled 237 KB binary trie. Parsing that format on the device is a separate project with a high cost of error.

The conversion enumerates the normalizer's behaviour over code points once — about 4 800 rules — and stores it as a plain table alongside the pieces and the vocabulary. Every direction was verified piece-for-piece against the reference AutoTokenizer on a 15-sentence corpus in the source language of that direction.

Using them

The reference consumer is a Kotlin runtime (tokenizer, beam search, three ONNX sessions) built for My Chat and shared with a keyboard over an IPC contract; measured on a Galaxy Z Fold 6: median 109 ms per sentence, worst case 191 ms, after a one-off 587 ms to open a pair.

Any decoder works, with one requirement that is easy to miss:

The padding token must be forbidden during generation. Marian uses it as the decoder start token, and generation_config.json sets bad_words_ids: [[pad]]. en-uk selects it at the very first step: without the ban, generation repeats padding to the length limit and returns an empty string on a perfectly healthy model. Eleven of the twelve directions behave without the ban, which is exactly what makes the omission hard to notice.

Greedy decoding of the quantized graphs matches the fp32 reference sentence for sentence on five of the twelve directions and differs by a word or two on the rest — ordinary quantization drift, visible mostly in numerals and greetings. Beam search narrows it further.

Licence and attribution

The original models are © Helsinki-NLP, released under CC BY 4.0, and this derivative keeps that licence. If you use these files, credit the Helsinki-NLP Opus-MT project — the translation quality is theirs; only the packaging is ours.

• Models: https://huggingface.co/Helsinki-NLP
• Opus-MT / Tatoeba Challenge: Tiedemann, J. (2020), The Tatoeba Translation Challenge — Realistic Data Sets for Low Resource and Multilingual MT.

Reproducing

python3 tools/opus-mt/convert.py en uk --out dist/opus-mt
python3 tools/opus-mt/verify.py dist/opus-mt/en-uk

verify.py checks two things and refuses to skip either: that our tokenizer agrees with the reference piece for piece, and that the assembled graphs actually translate a sentence. The second check exists because the first one passed on en-uk while it was returning nothing at all.