Views
No views yet
sbintuitions/modernbert-ja-130m (132.5M params, 19 layers, RoPE, local+global attention), converted for the feature-extraction task so it outputs last_hidden_state for building sentence embeddings via mean pooling.onnx/:| File | Precision | Size | Use case |
|---|---|---|---|
onnx/model.onnx | fp32 | 505 MB | Reference / highest fidelity |
onnx/model_quantized.onnx | int8 (dynamic) | 127 MB (~4x smaller) | Recommended for CPU serving |
onnxruntime CPUExecutionProvider), single-sequence inference, over a 20-sentence Japanese set covering paraphrase pairs, related pairs, and unrelated pairs. Quality is reported two ways against the original PyTorch fp32 model: mean cosine similarity of individual embeddings, and Spearman rank correlation of the full pairwise similarity matrix (i.e. does the variant preserve which sentences rank as more/less similar — what actually matters for a sentence-similarity use case).| Variant | Latency (mean, batch=1) | vs. PyTorch | File size | Cosine sim to original | Similarity-ranking correlation (Spearman ρ) |
|---|---|---|---|---|---|
| PyTorch fp32 (original) | 58.4 ms | 1.0x | 505 MB (safetensors) | 1.0000 | 1.0000 |
| ONNX fp32 | 24.4 ms | 2.4x faster | 505 MB | 1.0000 | 1.0000 |
| ONNX int8 (dynamic) | 11.0 ms | 5.3x faster | 127 MB (4x smaller) | 0.9903 (min 0.9788) | 0.9333 |
onnx/model.onnx) is a free win: 2.4x lower latency with zero quality loss (cosine similarity 1.0000 against the original PyTorch outputs). Use this when fidelity matters most and storage isn't the constraint.onnx/model_quantized.onnx) adds a further 2.2x latency reduction on top of ONNX fp32 (5.3x vs. the original PyTorch model) and cuts file size ~4x, at the cost of some embedding precision (mean cosine similarity 0.990, ranking correlation 0.933 rather than a perfect 1.0). In our test this did not flip the relative order of clearly-similar vs. clearly-dissimilar sentence pairs, but the ranking correlation is noticeably below the fp32 variants, so it's worth validating against your own similarity/retrieval eval set before relying on it for fine-grained ranking.onnxruntime/PyTorch are numerically less mature. It isn't included here — happy to add onnx/model_fp16.onnx if you plan to serve on GPU.onnx/model_quantized.onnx if you need the smallest footprint and fastest CPU inference and can tolerate ~1-2% embedding drift; use onnx/model.onnx if you want the latency win from ONNX with no measurable quality change.onnxruntime directly1import numpy as np
2import onnxruntime as ort
3from transformers import AutoTokenizer
4
5tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
6session = ort.InferenceSession("onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])
7
8def embed(sentences):
9 enc = tokenizer(sentences, padding=True, return_tensors="np")
10 last_hidden_state, = session.run(
11 None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]}
12 )
13 mask = enc["attention_mask"][..., None].astype(np.float32)
14 summed = (last_hidden_state.astype(np.float32) * mask).sum(axis=1)
15 counts = np.clip(mask.sum(axis=1), 1e-9, None)
16 return summed / counts # mean-pooled sentence embeddings
17
18embs = embed(["今日はいい天気ですね。", "本日は晴れて気持ちがいいです。"])
19cos_sim = (embs[0] @ embs[1]) / (np.linalg.norm(embs[0]) * np.linalg.norm(embs[1]))
20print(cos_sim)optimum1from optimum.onnxruntime import ORTModelForFeatureExtraction
2from transformers import AutoTokenizer
3
4tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
5model = ORTModelForFeatureExtraction.from_pretrained("<this-repo>", file_name="onnx/model_quantized.onnx")optimum (optimum.exporters.onnx), task feature-extraction, opset 18.onnxruntime.quantization.quantize_dynamic (QInt8 weights).last_hidden_state using the attention mask, as shown above. The base model has no officially recommended pooling strategy since it's released as a masked-language-model checkpoint rather than a tuned sentence-embedding model; mean pooling is the standard default for this class of encoder.