Optimization	Setting
Graph Optimization Level	Extended
Optimize for GPU	Yes
Use FP16	No
Transformers Specific Optimizations Enabled	Yes
Gelu Fusion Enabled	Yes
Layer Norm Fusion Enabled	Yes
Attention Fusion Enabled	Yes
Skip Layer Norm Fusion Enabled	Yes
Gelu Approximation Enabled	Yes

Usage

python

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

# Load model and tokenizer
model = ORTModelForSequenceClassification.from_pretrained("quantized_model")
tokenizer = AutoTokenizer.from_pretrained("quantized_model")

# Prepare input
text = "Your text here"
inputs = tokenizer(text, return_tensors="pt")

# Run inference
outputs = model(**inputs)

Quantization Process

This model was quantized using ONNX Runtime with int8 quantization. The quantization was performed using the Optimum library from Hugging Face with opset 14. Graph optimization was applied during export, targeting GPU devices.

Performance Comparison

Quantized models generally offer better inference speed with a slight trade-off in accuracy. This INT8 quantized model should provide significantly faster inference than the original model.

ConfidentialMind/gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8

gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8-quantized

Model Details

Environment and Package Versions

Applied Optimizations

Usage

Quantization Process

Performance Comparison

ConfidentialMind/
gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8