Opus-MT for phones — sixteen directions, INT8 ONNX
Sixteen Opus-MT translation directions prepared to run offline on a phone:
three quantized ONNX graphs and one binary tokenizer per direction.
Nothing here is a new model. These are conversions of
Helsinki-NLP Opus-MT models, published so
that a mobile app can download a language pair once instead of shipping a
converter, and so the conversion is reproducible by anyone.
Directions
ru-uk · uk-ru
ru-en · en-ru
uk-en · en-uk
de-en · en-de
fr-en · en-fr
it-en · en-it
ru-fr · fr-ru
uk-fr · fr-uk
Directions, not languages: Opus-MT translates one way with one model, so
ru-uk and uk-ru are two different sets of files. Ukrainian is covered by
direct pairs with Russian rather than through English — on that pair the
quality cost of two passes is particularly audible. French with Russian and
Ukrainian is direct for the same reason.
Sizes run from 209 MB (ru-uk) to 264 MB (it-en). The spread comes from
vocabulary, not architecture: every direction has six layers and d_model 512,
but the shared vocabulary of it-en is 80 379 against 58 101 for de-en, and
the difference lands in the embedding matrix.
Files in each directory
opus-mt-<pair>/
encoder_model.onnx INT8
decoder_model.onnx INT8
decoder_with_past_model.onnx INT8
tokenizer.bin binary, see below
manifest.json sizes, SHA-256, architecture, source model
Why three graphs and not one
decoder_model_merged does not quantize: the whole body of the graph hides
inside a single If node, and the ONNX Runtime quantizer does not descend into
subgraphs.
Dropping decoder_model and keeping only the cached decoder does not work
either. It accepts an empty cache and returns logits of the right shape — but
past_key_values.N.encoder.* holds projected encoder states, not history, and
they are the decoder's only link to the input text. On an empty cache the model
produces fluent, entirely invented translations.
Why the tokenizer is a binary file
Marian models have no tokenizer.json, and source.spm is a SentencePiece
protobuf whose nmt_nfkc normalizer is a compiled 237 KB binary trie. Parsing
that format on the device is a separate project with a high cost of error.
The conversion enumerates the normalizer's behaviour over code points once —
about 4 800 rules — and stores it as a plain table alongside the pieces and the
vocabulary. Every direction was verified piece-for-piece against the reference
AutoTokenizer on a 15-sentence corpus in the source language of that
direction.
Using them
The reference consumer is a Kotlin runtime (tokenizer, beam search, three ONNX
sessions) built for My Chat and shared with a keyboard
over an IPC contract; measured on a Galaxy Z Fold 6: median 109 ms per sentence,
worst case 191 ms, after a one-off 587 ms to open a pair.
Any decoder works, with one requirement that is easy to miss:
The padding token must be forbidden during generation. Marian uses it as
the decoder start token, and generation_config.json sets
bad_words_ids: [[pad]]. en-uk selects it at the very first step: without
the ban, generation repeats padding to the length limit and returns an empty
string on a perfectly healthy model. Eleven of the twelve directions behave
without the ban, which is exactly what makes the omission hard to notice.
Greedy decoding of the quantized graphs matches the fp32 reference sentence for
sentence on five of the twelve directions and differs by a word or two on the
rest — ordinary quantization drift, visible mostly in numerals and greetings.
Beam search narrows it further.
Opus-MT / Tatoeba Challenge: Tiedemann, J. (2020), The Tatoeba Translation
Challenge — Realistic Data Sets for Low Resource and Multilingual MT.
Reproducing
bash
1python3 tools/opus-mt/convert.py en uk --out dist/opus-mt
2python3 tools/opus-mt/verify.py dist/opus-mt/en-uk
verify.py checks two things and refuses to skip either: that our tokenizer
agrees with the reference piece for piece, and that the assembled graphs
actually translate a sentence. The second check exists because the first one
passed on en-uk while it was returning nothing at all.
