Average score vs. parameter count for open-weight omni embedding models
Average score vs. parameter count across image (MIEB-Lite), video (MMEB-V), and audio (MAEB) benchmarks — jina-v5-omni-nano and jina-v5-omni-small define the open-weight frontier (Table 1 in the ArXiv report).
This is the text-matching-targeted variant of the jina-embeddings-v5-omni-small GGUF family. The umbrella with all GGUF variants and cross-repo benchmarks is jina-ai/jina-embeddings-v5-omni-gguf.
Cross-repo docs: benchmarks (NDCG@5 on NanoBEIR, tokens/sec, peak VRAM, file size), the full per-variant numerical-parity tables, and runtime caveats live in the v5-omni-gguf umbrella.
Via Elastic Inference Service
The fastest way to use v5-omni in production. Elastic Inference Service (EIS) provides managed embedding inference with built-in scaling, so you can generate embeddings directly within your Elastic deployment.
bash
1# Retrieve the configuration of the preconfigured omni-small inference endpoint2GET /_inference/embedding/.jina-embeddings-v5-omni-small
34# Generate an embedding for a single piece of text using the predefined endpoint5POST _inference/embedding/.jina-embeddings-v5-omni-small
6{7"input":[8"This is a test"9]10}1112# Fuse a text description and an image into a single embedding via a multimodal content block13POST _inference/embedding/.jina-embeddings-v5-omni-small
14{15"input":[16{17"content":[18{"type":"text", "value":"A small blue square"},
19{"type":"image", "format":"base64", "value":"<BASE64_IMAGE_DATA>"}20]21}22]23}2425# Create a custom endpoint that truncates omni-small embeddings to 32 dimensions26PUT _inference/embedding/jina-omni-small-32d
27{28"service":"elastic",
29"service_settings":{30"model_id":"jina-embeddings-v5-omni-small",
31"dimensions":3232}33}
llama.cpp loads --mmproj to enable image / video / audio inputs on top
of the text GGUF. The two mmprojs are independent — load whichever
modality you need, or pass --mmproj twice to serve both from one
process (see "Selective modality loading" below).
Install llama.cpp (with multimodal patches)
This model relies on the Jina v5 omni patches (audio chunked attention,
qwen3vl video temporal-pair, encoder combined-decode, etc.) — they are
not yet upstream. Build from the feat-v5-omni fork:
For CUDA: pass -DGGML_CUDA=ON to the configure step.
H100 / Hopper note. On Hopper GPUs (H100, H200), set GGML_CUDA_DISABLE_GRAPHS=1 before launching llama-server. Without it, the CUDA-graph capture/replay path crashes with cudaMemcpyAsync … illegal instruction during embedding extraction. CPU, Metal, Vulkan, and pre-Hopper CUDA (e.g. L4, A100) are unaffected.
Quickstart — text via llama-embedding
bash
1./build/bin/llama-embedding \2 -hf jinaai/jina-embeddings-v5-omni-small-text-matching-GGUF:Q4_K_M \3 --pooling last --embd-normalize 2\4 -p "A cute cat sitting on a mat."
The -hf shortcut downloads + caches the requested quant from this repo
on first use. Q4_K_M is the recommended CPU default; Q8_0 for highest
fidelity; IQ2_*/IQ1_* for very tight memory budgets.
No prefix convention. Inputs are embedded verbatim — no Query: / Document: prefixes are needed (unlike the retrieval variant). Both sides of any text pair go in unprefixed.
No custom pooling or padding code needed — --pooling last and --embedding are the only flags required; min_pixels / max_pixels / temporal_patch_size are baked into the GGUF metadata and the mmproj.
POST to /embeddings with the v5-omni image prompt template:
python
1import base64, requests
23withopen("photo.jpg","rb")as f:4 img_b64 = base64.b64encode(f.read()).decode()56# text query7q = requests.post("http://127.0.0.1:8080/embeddings", json={8"content":[{"prompt_string":"A cute cat sitting on a mat."}]9}).json()[0]["embedding"]1011# image embedding (one base64-encoded image per <__media__> marker)12i = requests.post("http://127.0.0.1:8080/embeddings", json={13"content":[{14"prompt_string":"<__media__>",15"multimodal_data":[img_b64],16}]17}).json()[0]["embedding"]
The <__media__> placeholder is replaced server-side with the right
sequence of image tokens.
Quickstart — text + video via llama-server
Same vision mmproj, but use videopair_data to pass frame pairs
(temporal_patch_size=2, matching torch's 3D conv with kt=2):
python
1import base64, imageio.v3 as iio, requests
23frames =list(iio.imiter("clip.mp4"))# decode video → list of HxWx3 frames4defb64(arr):5import io, numpy as np
6from PIL import Image
7 buf = io.BytesIO(); Image.fromarray(np.asarray(arr)).convert("RGB").save(buf,"PNG")8return base64.b64encode(buf.getvalue()).decode()910# group consecutive frames into pairs11pairs =[(b64(frames[i]), b64(frames[i+1]))for i inrange(0,len(frames)-1,2)]12prompt ="<__media__>"*len(pairs)# one marker per logical (paired) frame1314v = requests.post("http://127.0.0.1:8080/embeddings", json={15"content":[{"prompt_string": prompt,"videopair_data": pairs}]16}).json()[0]["embedding"]
Quickstart — text + audio via llama-server
Start a server with the audio mmproj (or run a second instance on
a different port if you already have a vision server up):
Vision and audio embeddings produced this way are bit-identical to the
single-mmproj invocations — the encoder graph is the same regardless of
whether the other modality's projector is also loaded.
Matryoshka (truncating embeddings)
Any prefix of the output vector is itself a valid embedding once
L2-renormalized. Supported prefix dims: {32, 64, 128, 256, 512, 768, 1024}. Verified
end-to-end through the GGUF encode pipeline: prefix dims produce
vectors with cos-vs-torch matching the full vector to within
quantization noise (max prefix-vs-full drift: +0.0000 at F16,
+0.0008 at Q4_K_M for this small model).
F16 + 13 int-quant levels, imatrix-calibrated against a multilingual
text corpus (calibration_data_v5_rc.txt). Numbers below are min cos
vs torch fp32 across the 7-input reference set in
ref_small_text-matching.json, bucketed by token length
(very_short = 2-4 tokens, short = 5-15, medium = 16-30):
Level
very_short
short
medium
F16
0.9999
1.0000
1.0000
Q8_0
0.9983
0.9998
0.9997
Q6_K
0.9955
0.9988
0.9986
Q5_K_M
0.9936
0.9979
0.9967
Q5_K_S
0.9905
0.9973
0.9966
Q4_K_M
0.9780
0.9937
0.9928
IQ4_NL
0.9814
0.9889
0.9871
IQ4_XS
0.9594
0.9894
0.9870
Q3_K_M
0.9749
0.9825
0.9829
Q2_K
0.9094
0.9175
0.9159
IQ2_M
0.2994
0.9418
0.9291
IQ2_XXS
0.7699
0.7871
0.8249
IQ1_M
0.1959
0.6795
0.7244
IQ1_S
0.6501
0.5920
0.6541
Recommendation: Q4_K_M is the production CPU default for
small. Higher levels (Q5_K_M / Q6_K / Q8_0) are conservative
choices when very-short or multilingual inputs (titles, single-word
queries) dominate. IQ-quants and Q2_K and below break down on tiny
inputs — use only for memory-constrained testing.
The vision and audio mmprojs ship in F16 only — quantization beyond
F16 on the projector tensors causes large parity loss and is not
worth the disk savings.
Batching
llama-server's /embeddings endpoint accepts a list of inputs in the content array — one forward pass per element, returned as separate embeddings:
python
1import requests
2batch = requests.post("http://127.0.0.1:8080/embeddings", json={3"content":[4{"prompt_string":"A cute cat sitting on a mat."},5{"prompt_string":"A red sports car parked under a tree."},6]7}).json()8# batch[0]["embedding"], batch[1]["embedding"]
Multimodal inputs are forwarded per-sample (one pass per image / video / audio). Long text-only batches benefit most from -b 8192 -ub 8192. For high-throughput multimodal serving, prefer the vLLM path on the torch base model.
Multimodal parity vs torch (cos ≥ 0.99 numerical bar)
Verified on the same fork build that produces this repo:
Last-token pooling is used throughout (matches the torch reference).
For best parity, prefer Q4_K_M or higher for the text quantization;
the mmprojs ship F16 only.
The <image> token is shared between image and video inputs; video
uses image_grid_thw=[T,H,W] with T=2 (the Qwen3-VL ViT's Conv3d
patch_embed handles the temporal dimension). The GGUF videopair_data
API is identical for image and video paths.
