You can deploy this model and start earning money today!

TigreGotico/
mt-hitz-es-eu-onnx

Name: mt-hitz-es-eu-onnx
Availability: InStock

Views

No views yet

mt-hitz-es-eu-onnx

ONNX export of HiTZ/mt-hitz-es-eu, a MarianMT machine translation model for es -> eu produced by HiTZ, the Basque Center for Language Technology at the University of the Basque Country (UPV/EHU).

All credit for training and releasing the original model goes to the HiTZ team. This repo only contains an ONNX conversion (fp32 and int8) for fast CPU inference with optimum[onnxruntime], done by TigreGotico.

License

apache-2.0, as declared on the source model card.

Export command

optimum-cli export onnx --model HiTZ/mt-hitz-es-eu --task text2text-generation-with-past --no-post-process es-eu

int8 dynamic quantization was applied afterwards with onnxruntime.quantization.quantize_dynamic (QUInt8) into the int8/ subfolder.

File layout

es-eu/
  encoder_model.onnx              # fp32
  decoder_model.onnx              # fp32
  decoder_with_past_model.onnx    # fp32
  config.json, generation_config.json
  source.spm, target.spm, vocab.json, tokenizer_config.json, special_tokens_map.json
  int8/
    encoder_model.onnx
    decoder_model.onnx
    decoder_with_past_model.onnx

Parity vs. the original PyTorch model

Measured on 8 held-out es->eu sentences, beam search (num_beams=4, max_new_tokens=64), exact-match against the original PyTorch MarianMTModel output:

fp32: 100.0% exact match
int8: 12.5% exact match (semantically consistent but not gated on this number; dynamic quantization trades some fluency, especially on longer sentences, for ~2.5x smaller size and faster CPU inference)

Example (es -> eu):

PyTorch: Zer moduz zabiltzate, kamaradaz, eta ez da ezertxo ere gertatzen.
ONNX fp32: Zer moduz zabiltzate, kamaradaz, eta ez da ezertxo ere gertatzen.
ONNX int8: kaixo, zer moduz zaude?

Usage

python

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("TigreGotico/mt-hitz-es-eu-onnx")

# fp32 (default, root of the repo)
model = ORTModelForSeq2SeqLM.from_pretrained("TigreGotico/mt-hitz-es-eu-onnx")

# int8 (smaller / faster, subfolder)
# download the repo and load from the local "int8" folder, e.g.:
# from huggingface_hub import snapshot_download
# path = snapshot_download("TigreGotico/mt-hitz-es-eu-onnx")
# model = ORTModelForSeq2SeqLM.from_pretrained(path + "/int8")

batch = tok(["your sentence here"], return_tensors="pt")
generated = model.generate(**batch, num_beams=4, max_new_tokens=64)
print(tok.batch_decode(generated, skip_special_tokens=True))

Attribution

Original model: HiTZ/mt-hitz-es-eu by HiTZ Center (University of the Basque Country, UPV/EHU)
ONNX export: TigreGotico