Views
No views yet
jinaai/jina-embeddings-v3) exported to ONNX and dynamically quantized to Int8 for CPU-friendly inference.jinaai/jina-embeddings-v3 (XLM-R, 24 layers, hidden_size=1024).retrieval.query / retrieval.passage / text-matching / classification / separation.model.onnx.optimum.onnxruntime ORTQuantizer.is_static=False), QuantType.QInt8, operators: MatMul / Attention / Gather / Gemm.input_ids, attention_mask, task_id.pip install onnxruntime transformers1import numpy as np
2import onnxruntime as ort
3from transformers import AutoTokenizer
4
5model_id = "ldwformat/jina-embeddings-v3-Q8-onnx" # this folder
6tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
7session = ort.InferenceSession(f"{repo_path}/model.onnx")
8
9def encode(text, task_id=0):
10 inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)
11 feed = {
12 "input_ids": inputs["input_ids"].astype(np.int64),
13 "attention_mask": inputs["attention_mask"].astype(np.int64),
14 "task_id": np.array([task_id], dtype=np.int64),
15 }
16 last_hidden = session.run(None, feed)[0] # [B, L, H]
17 mask = np.expand_dims(feed["attention_mask"], -1) # [B, L, 1]
18 summed = (last_hidden * mask).sum(axis=1) # [B, H]
19 lengths = np.clip(mask.sum(axis=1), 1e-9, None)
20 pooled = summed / lengths # mean pooling
21 emb = pooled[0]
22 return emb / np.linalg.norm(emb)
23
24# Example: query (task_id=0) vs passage (task_id=1)
25q = encode("How to fix a phone battery?", task_id=0)
26d = encode("A detailed guide to replacing a mobile device battery.", task_id=1)Tip: choosetask_idper task (0=query, 1=passage; seetask_instructionsin config). For Matryoshka use-cases, truncate to the leading N dims (e.g.,emb[:256]).
qt/jina-quality-check.py tests semantic separation on a small CN example, quantization fidelity vs FP32, and 256-dim truncation robustness.jinaai/jina-embeddings-v3 license; follow the original terms.