import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model_int8.onnx")
inputs = {
    "input_ids": np.array([[...]], dtype=np.int64),
    "attention_mask": np.array([[...]], dtype=np.int64),
    "token_type_ids": np.array([[...]], dtype=np.int64),
}
last_hidden_state = session.run(None, inputs)[0]  # (batch, seq_len, 768)
cls_embedding = last_hidden_state[:, 0, :]         # (batch, 768)

Hosstia/bert-base-chinese-onnx

BERT-base Chinese (ONNX INT8)

Files

Usage

Hosstia/
bert-base-chinese-onnx