Views
No views yet
Helsinki-NLP/opus-mt-tc-big-gmw-gmw, a Marian translation model from the Helsinki-NLP OPUS-MT project.optimum-cli export onnx --model Helsinki-NLP/opus-mt-tc-big-gmw-gmw --task text2text-generation-with-past <out>onnxruntime.quantization.quantize_dynamic (QUInt8 weights)../ fp32 ONNX graphs (encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx) + tokenizer files
./int8/ int8 dynamic-quantized ONNX graphsMarianMTModel) vs ONNX fp32 (ORTModelForSeq2SeqLM) on 2 sentences, greedy and beam=4 (max_new_tokens=64). Overall: greedy PASS, beam4 PASS.>>id<< language token is REQUIRED to select
the output language (omitting it lets the model pick an arbitrary target). Verified with 3
different target tags that outputs are non-empty and mutually distinct:>>eng<< Hallo, wie geht es dir heute? -> Hello, how are you today?>>nld<< Hallo, wie geht es dir heute? -> Hoe gaat het vandaag?>>afr<< Hallo, wie geht es dir heute? -> Hallo, hoe gaan dit met jou vandag?1from optimum.onnxruntime import ORTModelForSeq2SeqLM
2from transformers import AutoTokenizer
3
4repo = "TigreGotico/opus-mt-tc-big-gmw-gmw-onnx"
5tok = AutoTokenizer.from_pretrained(repo)
6model = ORTModelForSeq2SeqLM.from_pretrained(repo) # fp32; pass subfolder="int8" for the quantized graphs
7inputs = tok(">>eng<< Hallo, wie geht es dir heute?", return_tensors="pt")
8out = model.generate(**inputs, num_beams=4, max_new_tokens=64)
9print(tok.decode(out[0], skip_special_tokens=True))