from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer
import torch

# Load model and tokenizer
model = ORTModelForFeatureExtraction.from_pretrained(
    "khaangnguyeen/ruri-v3-310m-onnx",
    provider="CPUExecutionProvider"  # or "CUDAExecutionProvider" for GPU
)
tokenizer = AutoTokenizer.from_pretrained("khaangnguyeen/ruri-v3-310m-onnx")

# Encode text
texts = ["Example sentence", "Another example"]
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

# Get embeddings
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)  # Mean pooling

print(embeddings.shape)  # (2, hidden_size)

With ONNX Runtime (Direct)

python

import onnxruntime as ort
import numpy as np
from transformers import AutoTokenizer

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("khaangnguyeen/ruri-v3-310m-onnx")

# Create ONNX Runtime session
session = ort.InferenceSession("model.onnx")

# Prepare inputs
text = "Example sentence"
inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)

# Run inference
outputs = session.run(
    None,
    {
        "input_ids": inputs["input_ids"],
        "attention_mask": inputs["attention_mask"]
    }
)

embeddings = outputs[0].mean(axis=1) 
print(embeddings.shape)

Performance

ONNX Runtime provides:

⚡ Faster inference speed compared to PyTorch (typically 1.5-3x)
💾 Smaller footprint / optimized graph
🎯 Optimized for production deployment
🔧 Support for various hardware backends (CPU, GPU, NPU)

Benchmark

CPU Benchmark (example):

PyTorch: ~XX.XX ms/sample
ONNX Runtime: ~XX.XX ms/sample
Speedup: ~X.XXx

License

Apache-2.0 (inherited from base model)

Citation

bibtex

@misc{ruri-v3-onnx,
  author = {khaangnguyeen},
  title = {Ruri v3 310M ONNX},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{[https://huggingface.co/khaangnguyeen/ruri-v3-310m-onnx](https://huggingface.co/khaangnguyeen/ruri-v3-310m-onnx)}}
}