import onnxruntime as ort
from transformers import AutoTokenizer

model_path = "your-username/gte-large-zh-onnx"
tokenizer = AutoTokenizer.from_pretrained(model_path)
session = ort.InferenceSession(f"{model_path}/model_int8.onnx")

# Run inference
inputs = tokenizer(["文本"], return_tensors="np", padding=True)
outputs = session.run(None, {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64),
})

C++

See README.md in the repository for C++ ONNX Runtime example.

Performance

FP32: High accuracy, larger size
INT8: 75% smaller, minimal accuracy loss for semantic search

Original Model

thenlper/gte-large-zh

baby2008/gte-large-zh-onnx

GTE-Large-ZH ONNX

Model Files

Usage

Python

C++

Performance

Original Model

baby2008/
gte-large-zh-onnx