A 70M text-only encoder that maps a query into the embedding space of
Qwen3-VL-Embedding-8B,
distilled by pointwise cosine alignment against that teacher's cached query
embeddings. The -ML suffix marks the multilingual training mixture: 711K
English queries plus 778K MarianMT translations into five Latin-script European
languages.
Any half of the teacher, swapped
Both towers land in the same frozen teacher's embedding space, so either is a
drop-in replacement for the corresponding half of that teacher. All four
combinations retrieve; they differ in what you no longer have to run:
teacher documents
student documents
teacher queries
71.05 — the ceiling, 8B on both sides
65.02 — indexing 7x cheaper, teacher still runs per query
student queries
66.36 — queries encode on one CPU thread, index built once by the teacher
61.74 — no teacher anywhere
Average NDCG@5 over ViDoRe v1+v2+v3, this model paired with
NanoVDR-D-HiRes-Qwen3VL8B-4096, against Qwen3-VL-Embedding-8B.
Pass the raw query. The teacher's targets were cached with an instruction, but
the student was trained to reproduce those targets from the bare query text,
and every number below was measured that way. Prepending an instruction at
inference moves the input off the distribution the tower was fitted on.
Results
NDCG@5 on ViDoRe. Query side in isolation, scored against teacher-encoded
pages, which isolates this tower from document-side error:
v1
v2
v3
Avg
NanoVDR-Q-DistilBERT-Qwen3VL8B-4096-ML
84.68
64.30
50.09
66.36
Qwen3-VL-Embedding-8B (teacher)
87.31
69.76
56.07
71.05
End to end with the document towers, no teacher anywhere:
Pairing
v1
v2
v3
Avg
+ NanoVDR-D-HiRes-Qwen3VL8B-4096
82.81
55.34
47.07
61.74
+ NanoVDR-D-Fast-Qwen3VL8B-4096
81.34
54.95
43.66
59.98
Query encoding costs 2.07 ms per query at batch 1 on an H200, including
tokenisation. There is no vision model in the query path.
Backbone ablation
Measured under query-side isolation, all distilled from the same teacher:
Backbone
Params
v1
v2
v3
Avg
Latency
DistilBERT-base (this model)
70M
84.68
64.30
50.09
66.36
2.07 ms
BERT-base
113M
81.81
58.52
44.55
61.63
4.78 ms
ModernBERT-base
152M
85.36
65.40
51.43
67.40
10.61 ms
ModernBERT is 1.04 points better for 2.2x the parameters and 5.1x the latency,
which is why the small one ships as the default.
Training
Objective
1 - cos(student, teacher), nothing else
Teacher
Qwen3-VL-Embedding-8B, frozen, 4096-d
Data
1.49M queries (the NanoVDR training set plus MarianMT translations)
Optimizer
AdamW, one-cycle, peak LR 5e-4, 3% warmup
Effective batch
512
Epochs
15
Peak LR is half what the document tower uses; at 1e-3 a freshly initialised
768→4096 head diverges at the peak of the one-cycle schedule.
Reproducibility
Re-encoding the 500 ViDoRe arxivqa queries with this package and scoring
against the cached teacher page embeddings:
NDCG@5 teacher queries x teacher pages : 86.91
NDCG@5 student queries x teacher pages : 83.77 (96.4% retention)
which matches our internal evaluation for that dataset exactly.
Limitations
English and five Latin-script European languages. The multilingual
portion comes from a MarianMT translation pipeline. Non-Latin scripts such as
Chinese, Japanese and Arabic are untested.
Text-only. Image-conditioned queries are out of scope.
Bounded by the teacher. Nothing in the objective lets the student exceed
Qwen3-VL-Embedding-8B on the side it replaces.
Citation
bibtex
1@article{distilvdr2026,
2 title = {DistilVDR: A Compact End-to-End Visual Document Retriever
3 via Dual-Student Distillation},
4 author = {Liu, Zhuchenyang and Wang, Ziyi and Zhang, Yao and Xiao, Yu},
5 journal = {arXiv preprint arXiv:2608.10636},
6 year = {2026}
7}
