You can deploy this model and start earning money today!

skatzR/
USER-BGE-M3-ONNX-INT8

Name: USER-BGE-M3-ONNX-INT8
Rating: 2.5 (144 reviews)

Views

No views yet

🧩 DeepVK-USER-BGE-M3 — Quantized ONNX (INT8)

✨ This repository contains a quantized INT8 ONNX version of deepvk/USER-bge-m3.
It is designed for fast CPU inference with ONNX Runtime, making it a great choice for semantic search, embeddings generation, and text similarity tasks in Russian 🇷🇺 and English 🇬🇧.

🔍 Model Card

Property	Value
Base model	`deepvk/USER-bge-m3`, `BAAI/bge-m3`
Quantization	INT8 (Dynamic)
Format	ONNX
Libraries	`transformers`, `onnxruntime`, `optimum`, `sentence-transformers`
Embedding dim	1024
Supported HW	CPU (optimized for Intel AVX512-VNNI, fallback to AVX2)
License	Apache-2.0

🚀 Features

⚡ Fast CPU inference — ONNX + INT8 gives a strong speed-up.
📦 Lightweight — reduced model size, lower memory footprint.
🔄 Drop-in replacement — embeddings compatible with the FP32 version.
🌍 Multilingual — supports Russian 🇷🇺 and English 🇬🇧.

🧠 Intended Use

✅ Recommended for:

Semantic search & retrieval systems
Recommendation pipelines
Text similarity & clustering
Low-latency CPU deployments

❌ Not ideal for:

Absolute maximum accuracy scenarios (INT8 introduces minor loss)
GPU-optimized pipelines (prefer FP16/FP32 models instead)

⚖️ Pros & Cons of Quantized ONNX

Pros ✅

Easy to use (no calibration dataset required).
Smaller & faster than FP32.
Works out of the box with ONNX Runtime.

Cons ❌

Slight accuracy drop compared to static quantization.
AVX512 optimizations only on modern Intel CPUs.
No GPU acceleration in this export.

📊 Benchmark

Metric	Value
Avg cosine similarity (vs FP32)	~0.988
Median cosine similarity	~0.988
Orig model time (s)	0.7504
Quant model time (s)	0.3539
Inference speed	~2× faster
Model size (MB)	347.5

📂 Files

model_quantized.onnx — quantized model

tokenizer.json, vocab.txt, special_tokens_map.json — tokenizer

config.json — model config

🧩 Examples

You can try the model directly in Google Colab:

Open In Colab

This notebook demonstrates:

Loading the original FP32 model deepvk/USER-bge-m3
Loading the quantized INT8 ONNX model skatzR/USER-BGE-M3-ONNX-INT8
Comparing quality (cosine similarity) and inference speed side by side

You can try this model with ready-to-use scripts in the examples folder:

quantmodel.py — universal Python module for loading and encoding texts with the quantized ONNX model.
app-console.py — console script to compare FP32 vs INT8 embeddings (cosine similarity + inference time).
app-streamlit.py — interactive demo with Streamlit.

skatzR/USER-BGE-M3-ONNX-INT8