You can deploy this model and start earning money today!

TigreGotico/
opus-mt-en-tr-onnx

Name: opus-mt-en-tr-onnx
Rating: 2.5 (13 reviews)

Helsinki-NLP/opus-mt-tc-big-en-tr

quantized

cc-by-4.0

endpoints_compatible

Views

No views yet

opus-mt-en-tr-onnx

ONNX export (fp32 + dynamic int8 quantized) of Helsinki-NLP/opus-mt-tc-big-en-tr, a Marian (en -> tr) translation model from the Helsinki-NLP OPUS-MT project.

License: cc-by-4.0 (inherited from the base model; verify at the source link above).

Export

optimum-cli export onnx --model Helsinki-NLP/opus-mt-tc-big-en-tr --task text2text-generation-with-past /home/miro/tmp/claude-1000/-home-miro/849a3674-94c9-4aa5-b1f9-e596235b0a36/scratchpad/opusmt-onnx/out/en-tr

Quantized to int8 with onnxruntime.quantization.quantize_dynamic (QUInt8 weights).

File layout

./                      fp32 ONNX graphs (encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx) + tokenizer files
./int8/                 int8 dynamic-quantized ONNX graphs

fp32 size: ~2306 MB | int8 size: ~1037 MB

Sanity check

Compared PyTorch (MarianMTModel) vs ONNX fp32 (ORTModelForSeq2SeqLM, num_beams=4, max_new_tokens=48) on 3 sentences. Match: 3/3.

src: Hello, how are you today?
- pytorch: Merhaba, bugün nasılsınız?
- onnx fp32: Merhaba, bugün nasılsınız? (match)
src: The weather is nice this morning.
- pytorch: Bu sabah hava çok güzel.
- onnx fp32: Bu sabah hava çok güzel. (match)
src: I would like to order a coffee, please.
- pytorch: Bir kahve ısmarlamak istiyorum, lütfen.
- onnx fp32: Bir kahve ısmarlamak istiyorum, lütfen. (match)

int8 sample output (not gated, for reference only):

Merhaba, bugün nasılsınız?

Usage

python

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

repo = "TigreGotico/opus-mt-en-tr-onnx"
tok = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSeq2SeqLM.from_pretrained(repo)  # fp32; pass subfolder="int8" for the quantized graphs
inputs = tok("Hello, how are you?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=48)
print(tok.decode(out[0], skip_special_tokens=True))

Exported for the OVOS / TigreGotico offline translation stack.