This is a quantized ONNX export of all-mpnet-base-v2, one of the best general-purpose sentence embedding models from the sentence-transformers library. It maps sentences and paragraphs to a 768-dimensional dense vector space. The INT8 quantization reduces model size and improves inference speed while maintaining high accuracy.

Use Cases

Semantic text search
Sentence similarity
Clustering and topic modeling
Paraphrase detection
General-purpose text embeddings

Files

model_quantized.onnx — INT8 quantized ONNX model
tokenizer.json — Fast tokenizer
vocab.txt — Vocabulary file
config.json — Model configuration

Usage with JustEmbed

python

from justembed import Embedder

embedder = Embedder("mpnet-int8")
vectors = embedder.embed(["This is a sentence", "This is another sentence"])

Usage with ONNX Runtime

python

import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(".")
session = ort.InferenceSession("model_quantized.onnx")

inputs = tokenizer("This is a sentence", return_tensors="np")
outputs = session.run(None, dict(inputs))

Quantization Details

Method: Dynamic INT8 quantization via ONNX Runtime
Source: Original PyTorch weights converted to ONNX, then quantized
Speed: ~2-3x faster inference than FP32
Size: ~4x smaller than FP32

License

This model is a derivative work of sentence-transformers/all-mpnet-base-v2.

The original model is licensed under Apache License 2.0. This quantized version is distributed under the same license. See the LICENSE file for the full text.

Citation

bibtex

@inproceedings{song2020mpnet,
  title={MPNet: Masked and Permuted Pre-training for Language Understanding},
  author={Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan},
  booktitle={NeurIPS},
  year={2020}
}

Acknowledgments

Original model by UKP Lab / sentence-transformers
Quantization and packaging by JustEmbed

sekarkrishna/mpnet-int8

MPNet INT8 — ONNX Quantized

Model Details

What is this?