VultronRetrieverCore is the mid tier of the VultronRetriever family, a late-interaction (ColBERT-style) retriever that scores document pages directly from their rendered image: layout, tables, charts and text, across six languages. At 4.5B it ranks second on the full ViDoRe V3 board (only the 8B Prime scores higher) at a 320-dim index, narrower than most of the models around it.

The family has three tiers on one 320-dim recipe: the 8B Prime for maximum accuracy, the 4.5B Core (this model) for the accuracy/footprint mid-point, and the 0.8B Flash for latency- and footprint-sensitive serving. Trained and evaluated on Vultr Cloud.

Highlights

#2 on ViDoRe V3: 63.57 mean nDCG@10 over all 10 tasks; the only model above it is the 8B Prime.
#1 on V3 Nuclear: 54.90 nDCG@10, the top score on that private task across the whole board.
Best V2 in its class: 66.12 nDCG@5, top of every ≤4.5B retriever.
Official MTEB: V1 92.21, V2 66.12, V3 63.57.
~4.5B parameters, native 320-dim, ≈9 GB bf16.
Six languages (en, fr, de, es, it, pt).

ViDoRe leaderboard (ranked by V3)

Ranked by ViDoRe V3 (mean nDCG@10), the headline benchmark; V1 and V2 are shown alongside. Our three tiers are in bold; models with a partial or missing V3 sit at the bottom.

Model	Params	Dim	V1	V2	V3
VultronRetrieverPrime-Qwen3.5-8B (ours)	8.4B	320	92.08	68.18	64.26
VultronRetrieverCore-Qwen3.5-4.5B (this model)	4.5B	320	92.21	66.12	63.57
nvidia/nemotron-colembed-vl-8b-v2	8.7B	4096	92.65	65.16	63.42
webAI-Official/webAI-ColVec1-9b	9.4B	2560	91.30	65.82	63.00
webAI-Official/webAI-ColVec1-4b	4.5B	640	90.49	63.60	62.22
TomoroAI/tomoro-colqwen3-embed-8b	8.0B	320	90.76	65.40	61.59
nvidia/nemotron-colembed-vl-4b-v2	4.8B	2560	91.62	64.49	61.54
athrael-soju/colqwen3.5-4.5B-v3	4.6B	128	91.54	64.25	61.46
OpenSearch-AI/Ops-Colqwen3-4B	4.8B	2560	91.36	68.66	61.17
TomoroAI/tomoro-colqwen3-embed-4b	4.0B	320	90.57	64.69	60.20
nvidia/llama-nemotron-colembed-vl-3b-v2	4.4B	3072	91.74	63.38	59.79
jinaai/jina-embeddings-v4	3.9B	2048	90.35	58.23	57.52
nomic-ai/colnomic-embed-multimodal-7b	7.0B	128	89.72	60.25	57.33
nvidia/llama-nemoretriever-colembed-3b-v1	4.4B	3072	91.00	63.32	57.26
VultronRetrieverFlash-Qwen3.5-0.8B (ours)	0.85B	320	88.15	60.36	56.16
Verm1ion/ColTurk-VDR-Qwen3VL-4B-v1.0	4.5B	128	—	—	55.81
nomic-ai/colnomic-embed-multimodal-3b	3.0B	128	89.86	55.68	55.78
nvidia/llama-nemoretriever-colembed-1b-v1	2.4B	2048	90.50	62.96	55.59
vidore/colqwen2.5-v0.2	3.0B	128	89.54	60.06	51.90
VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1	8.1B	128	91.08	62.47	58.55†
VAGOsolutions/SauerkrautLM-ColQwen3-4b-v0.1	4.4B	128	90.80	59.89	56.03†
DataScience-UIBK/Argus-Colqwen3.5-9b-v0	8.8B	1024	92.67	69.27	—
DataScience-UIBK/Argus-Colqwen3.5-4b-v0	4.7B	1024	92.30	64.18	—

V1/V2: full mean nDCG@5. V3: mean nDCG@10 over all 10 ViDoRe V3 tasks, the 8 public plus the 2 private tasks (Nuclear, Telecom) scored by the ViDoRe maintainers on held-out corpora. † = evaluated on the 8 public tasks only, not comparable to the 10-task means. Competitor figures: public MTEB ViDoRe leaderboard, 2026-07-04 snapshot; "—" = not reported on that benchmark.

At 4.5B, Core is #2 on the full V3 board; the only model above it is the 8B Prime, and on the private Nuclear task it posts the top score outright. On V2 it tops every ≤4.5B retriever. On the near-saturated V1 (the pack sits at 88–92) it scores 92.21, fourth on the board.

Official MTEB results (dim 320 / visual tokens 1792)

Benchmark	Metric	Tasks	Score
ViDoRe V1	ndcg@5	10	0.9221
ViDoRe V2	ndcg@5	4	0.6612
ViDoRe V3	ndcg@10	10	0.6357

Per-task JSONs for the 8 public V3 tasks are in eval_results/; the 2 private tasks are scored by the ViDoRe maintainers and reported on the public leaderboard. Measured with the official MTEB late-interaction evaluator.

Why 320 dimensions

Late-interaction index size, memory footprint, and MaxSim scoring cost all scale with the embedding dimension. Core uses a native fixed 320-dim projection head (no Matryoshka truncation): a 320-dim index is a fraction of the size a 2048-4096-dim retriever carries, for proportionally lower storage, RAM, and query-time compute at serving scale, with no accuracy left on the table from prefix-truncating a wider head.

Intended use

Visual document retrieval / multimodal RAG over PDFs, scans, slides and reports, including pages with layout, tables, charts and figures.
Multilingual document collections (en, fr, de, es, it, pt).
The accuracy/footprint mid-point of the family: more accurate than the 0.8B Flash tier, lighter than the 8B Prime flagship.

Out of scope: text-only semantic search, where a single-vector dense embedder is cheaper; generative QA (this is a retriever; pair it with a reader/LLM).

Method

Per-token MaxSim scoring captures fine-grained matches against tables, figures, and layout that a single-vector embedder averages away.

Base: Qwen/Qwen3.5-4B (hybrid GatedDeltaNet + full-attention backbone).
Late-interaction retriever (ColQwen3_5): native 320-dim multi-vector embeddings, MaxSim scoring, image + text inputs.
Size: ~4.5B parameters, the generative head dropped for retrieval.
Native 320-dim head (no Matryoshka): trained and operated at 320 dimensions directly.
Model merging: five independently-seeded checkpoints (three in-batch ColBERT contrastive, two with mined hard negatives) merged per-block into one full-weight checkpoint. Per-block weights searched by TPE (explore) then BoTorch qLogNEI (exploit); the winner is the global argmax at the operating point.
Trained at up to 1280 visual tokens, evaluated and deployed at 1792.

Training data

An enhanced, multilingual mixture of public and synthetic visual-document retrieval sources, spanning en, es, de, fr, it and pt, decontaminated against all three ViDoRe suites (V1/V2/V3): 0% measured overlap with the evaluation benchmarks. The training recipe and the assembled training dataset are not distributed in this repository.

Inputs and outputs

Input: document-page images (RGB) and/or text queries; pages encode at up to 1792 visual tokens.
Output: multi-vector embeddings, one 320-dim vector per token (not a single pooled vector).
Scoring: late-interaction MaxSim between query-token and page-token vectors, via score_multi_vector.

Usage

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers[image]>=6.0.0"

python

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("vultr/VultronRetrieverCore-Qwen3.5-4.5B")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([23, 320]) torch.Size([1251, 320])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[16.2559, 10.9844],
#         [ 5.7363, 12.9688]], device='cuda:0')

ColPali Engine

The Qwen3.5 hybrid (GatedDeltaNet + full-attention) backbone has hard runtime kernel dependencies a vanilla ColQwen / PaliGemma card does not:

bash

pip install "git+https://github.com/illuin-tech/colpali@2e0b927051af727238783af039dcc2c50a4d8c27"
pip install causal-conv1d flash-linear-attention

causal-conv1d + flash-linear-attention are required (the hybrid layers import them at runtime).
Use SDPA, matching the training and evaluation runtime.

python

import torch
from PIL import Image
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor

model = ColQwen3_5.from_pretrained(
    "vultr/VultronRetrieverCore-Qwen3.5-4.5B",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",   # required (see above)
    device_map="cuda:0",
).eval()
processor = ColQwen3_5Processor.from_pretrained(
    "vultr/VultronRetrieverCore-Qwen3.5-4.5B",
    max_num_visual_tokens=1792,
)

# Document pages (rendered to images) and text queries
images = [Image.open("page_0.png"), Image.open("page_1.png")]
queries = ["What was Q3 revenue?", "Summarize the safety findings."]

with torch.no_grad():
    doc_emb = model(**processor.process_images(images).to(model.device))
    qry_emb = model(**processor.process_queries(queries).to(model.device))

# Late-interaction MaxSim scoring (feed fp32 to match the eval discipline)
scores = processor.score_multi_vector(qry_emb.float(), doc_emb.float())
# scores[i, j] = relevance of query i to page j
print(scores.shape)  # torch.Size([2, 2])

config.json carries dim=320, so custom_text_proj is sized correctly at load, with no manual config edits needed. It also carries retrieval_attention_contract="causal"; a conflicting runtime override is an error.

vLLM

vLLM serves this model natively through its pooling runner (the ColQwen3_5 architecture), returning the per-token multi-vectors for late-interaction scoring. It requires a vLLM build that includes the projection-bias fix and explicit retrieval-attention metadata support. The handler reads retrieval_attention_contract="causal" from this checkpoint and constructs all eight full-attention layers as AttentionType.DECODER; it must reject missing, conflicting, or unsupported contract metadata. The server uses the stock chat/image processor, so the ColQwen3.5 prompt contract is applied client-side: wrap each page image in the instruction template, append the query-augmentation tokens to each query, and set the visual-token budget through mm-processor-kwargs. Prefix caching and chunked prefill remain off for the GatedDeltaNet hybrid.

pip install vllm

python

import torch
from PIL import Image
from vllm import LLM
MODEL = "vultr/VultronRetrieverCore-Qwen3.5-4.5B"
MAX_PIXELS = 1792 * 32 * 32   # max_num_visual_tokens * (patch_size 16 * merge_size 2)^2
llm = LLM(
    model=MODEL,
    runner="pooling",
    dtype="bfloat16",
    enable_prefix_caching=False,
    enable_chunked_prefill=False,
    mm_processor_kwargs={"min_pixels": 65536, "max_pixels": MAX_PIXELS},
)
# ColQwen3.5 processor contract, applied client-side:
IMAGE_PROMPT = ("<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
                "Describe the image.<|im_end|><|endoftext|>")
def query_prompt(q): return q + "<|endoftext|>" * 10   # query augmentation
images  = [Image.open("page_0.png"), Image.open("page_1.png")]
queries = ["What was Q3 revenue?", "Summarize the safety findings."]
doc_out = llm.encode([{"prompt": IMAGE_PROMPT, "multi_modal_data": {"image": im}}
                      for im in images], pooling_task="token_embed")
qry_out = llm.encode([query_prompt(q) for q in queries], pooling_task="token_embed")
def mv(o):  # one [num_tokens, 320] multi-vector per item, L2-normalized per token
    t = torch.as_tensor(o.outputs.data, dtype=torch.float32)
    return torch.nn.functional.normalize(t, p=2, dim=-1)
docs, qrys = [mv(o) for o in doc_out], [mv(o) for o in qry_out]
# late-interaction MaxSim: per query token take the best doc token, then sum
scores = [[(q @ d.T).max(dim=-1).values.sum().item() for d in docs] for q in qrys]
print(scores)  # scores[i][j] = relevance of query i to page j

To serve over HTTP instead:

bash

vllm serve vultr/VultronRetrieverCore-Qwen3.5-4.5B \
  --runner pooling \
  --no-enable-prefix-caching --no-enable-chunked-prefill \
  --mm-processor-kwargs '{"min_pixels": 65536, "max_pixels": 1835008}'

Apply the same image template and query augmentation in your client requests. See the upstream example examples/pooling/score/colqwen3_5_rerank_online.py for the full online rerank flow.

Limitations

Tuned for six languages (en, fr, de, es, it, pt); other languages are out of distribution.
Late-interaction multi-vector indexes are larger than single-vector dense indexes: the trade for per-token layout/table/figure sensitivity (small for its class at 320-dim).
This is the mid tier; for maximum ViDoRe V3 accuracy use the 8B flagship VultronRetrieverPrime-Qwen3.5-8B, and for the smallest footprint the 0.8B VultronRetrieverFlash-Qwen3.5-0.8B.

License

Apache 2.0, covering the contents of this repository: model weights, config, and evaluation results. Built on Qwen/Qwen3.5-4B (Apache 2.0); the upstream license and attribution are retained. The training recipe and the assembled training dataset are not distributed in this repository.

Citation

bibtex

@misc{vultronretrievercore2026,
  title  = {VultronRetrieverCore-Qwen3.5-4.5B: Mid-Tier Late-Interaction Visual Document Retrieval at 320 Dimensions},
  author = {Georgiou, Athos (athrael-soju)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B}}
}

Trained and evaluated on Vultr Cloud.