📥 Installation

pip install -U transformers optimum[onnxruntime]

🚀 Usage

python

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForFeatureExtraction

repo_id = "yasserrmd/granite-embedding-r2-onnx"

# Load tokenizer + ONNX model
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = ORTModelForFeatureExtraction.from_pretrained(repo_id)

# Encode sentences
inputs = tokenizer(["Hello world", "مرحباً"], padding=True, return_tensors="pt")
outputs = model(**inputs)

# Apply mean pooling over tokens
embeddings = outputs.last_hidden_state.mean(dim=1)
print(embeddings.shape)  # (2, 768)

✅ Notes

Quantization reduces model size and makes inference faster on CPUs while preserving accuracy.
Pooling strategy here is mean pooling; you can adapt CLS pooling or max pooling as needed.
Works seamlessly with Hugging Face Hub + optimum.onnxruntime.

yasserrmd/granite-embedding-r2-onnx

Granite Embedding English R2 — INT8 (ONNX)

📥 Installation

🚀 Usage

✅ Notes

📚 References

yasserrmd/
granite-embedding-r2-onnx