ruri-v3-130m-int8-onnx
ONNX export of
cl-nagoya/ruri-v3-130m (132.1M params, ModernBERT-ja architecture fine-tuned for Japanese sentence embeddings), converted for the
feature-extraction task and dynamically quantized to INT8.
| File | Precision | Size | Use case |
|---|
onnx/model_quantized.onnx | int8 (dynamic) | 127 MB (~4x smaller than the 505 MB original) | Recommended for CPU serving |
An fp32 ONNX reference (505 MB, zero measured quality loss, ~3x faster than PyTorch) was also produced during conversion but isn't included in this upload — ask if you want it added.
Important: this is a prompted embedding model
Ruri v3 uses a 1+3 prefix scheme — prepend one of these to your text before embedding, depending on the use case:
| Prefix | Use case |
|---|
| (none) | General semantic similarity |
トピック: | Classification / clustering / topic-level info |
検索クエリ: | Retrieval queries |
検索文書: | Documents to be retrieved |
Pooling is mean pooling over last_hidden_state (with attention mask), matching the original Sentence-Transformers 1_Pooling config included in this repo. Output dimensionality is 512.
Benchmark
Measured on a CPU-only sandbox (onnxruntime CPUExecutionProvider), single-sequence inference, over the same 20-sentence Japanese set used for modernbert-ja-130m (no prefix, general semantic similarity setting). Quality is reported against the original PyTorch fp32 model both as mean cosine similarity of individual embeddings and as Spearman rank correlation of the full pairwise similarity matrix.
| Variant | Latency (mean, batch=1) | vs. PyTorch | File size | Cosine sim to original | Similarity-ranking correlation (Spearman ρ) |
|---|
| PyTorch fp32 (original) | 85.9 ms | 1.0x | 505 MB (safetensors) | 1.0000 | 1.0000 |
| ONNX fp32 | 27.2 ms | 3.2x faster | 505 MB | 1.0000 | 1.0000 |
| ONNX fp16 | 28.7 ms | 3.0x faster* | 252 MB | 0.999999 | 0.99999 |
| ONNX int8 (dynamic) | 12.3 ms | 7.0x faster | 127 MB (4x smaller) | 0.9890 (min 0.9770) | 0.9447 |
* fp16 shows no real latency benefit here because this is a CPU-only benchmark; fp16 is meant for GPU tensor cores. Not included in this upload.
Recommendation: onnx/model_quantized.onnx (int8) gives the best combination of size and CPU latency, at the cost of some embedding precision (~0.99 cosine similarity, 0.94 ranking correlation vs. the original). If your downstream task is sensitive to fine-grained similarity ranking (e.g. re-ranking, nearest-neighbor retrieval with tight margins), validate against your own eval set before relying on it, or ask for the fp32 ONNX variant instead (same latency-class win as modernbert-ja-130m's fp32 export, with zero measured quality loss).
Usage
1import numpy as np
2import onnxruntime as ort
3from transformers import AutoTokenizer
4
5tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
6session = ort.InferenceSession("onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])
7
8def embed(sentences, prefix=""):
9 texts = [prefix + s for s in sentences]
10 enc = tokenizer(texts, padding=True, return_tensors="np")
11 last_hidden_state, = session.run(
12 None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]}
13 )
14 mask = enc["attention_mask"][..., None].astype(np.float32)
15 summed = (last_hidden_state.astype(np.float32) * mask).sum(axis=1)
16 counts = np.clip(mask.sum(axis=1), 1e-9, None)
17 return summed / counts
18
19queries = embed(["東京の人口は？"], prefix="検索クエリ: ")
20docs = embed(["東京都の人口は約1400万人です。"], prefix="検索文書: ")
21cos_sim = (queries[0] @ docs[0]) / (np.linalg.norm(queries[0]) * np.linalg.norm(docs[0]))
22print(cos_sim)
Conversion details
- Exported with
optimum (optimum.exporters.onnx), task feature-extraction, opset 18.
- Dynamic INT8 quantization via
onnxruntime.quantization.quantize_dynamic (QInt8 weights).
- Sentence-Transformers config files (
modules.json, config_sentence_transformers.json, sentence_bert_config.json, 1_Pooling/config.json) are carried over from the original repo for reference/compatibility, though this ONNX export is loaded via onnxruntime / optimum directly rather than the sentence_transformers library.
Base model card, license (Apache-2.0), training details, and MTEB scores:
cl-nagoya/ruri-v3-130m.
