This model takes whybe-choi/Qwen3-VL-Embedding-2B-ko-vdr-preview-v0.2 (a Qwen3-VL-Embedding-2B contrastively fine-tuned on Ko-VDR) and further trains it by listwise KL knowledge distillation from the Qwen3-VL-Reranker-8B cross-encoder teacher — transferring the reranker's graded relevance (instead of binary labels) into the 2B bi-encoder, which sharpens top-rank ordering.

Method (sequential CL → KD)

Following "Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation."

Teacher: Qwen/Qwen3-VL-Reranker-8B — scored {gold ∪ 128 mined candidates} (log-odds) for the 146,687 queries of NomaDamas/ko-vdr-train-public-v2.0.
Objective: listwise KL, KL( softmax(teacher/τ) ‖ softmax(student/τ) ), τ = 1.0.
- Teacher scores are per-query max-subtract normalized (keeps the reranker's true score spread; min-max flattens it).
- Student score = softmax(20 · cosine).
Negatives: 8 per query, stratified via quantile anchors over the teacher scores (spans easy→hard, not just the hardest).
Training: fresh LoRA (r = 32, α = 32) on a 50k-query subset, 1 epoch, lr 2e-5, bf16. No InfoNCE / no Matryoshka.

Evaluation (KoViDoRe-v2, cybersecurity)

At the recommended inference config (~1800 visual tokens + the prompts below):

metric	baseline (v0.2)	distilled (this model)
NDCG@10	0.7629	0.7718
MRR@10	0.8319	0.8775
Recall@100	0.9809	0.9778

Distillation improves ranking quality — the MRR@10 gain (+0.046) is consistent across resolution/prompt settings — while Recall@100 is essentially unchanged. (Single-domain evaluation so far.)

Recommended inference config

python

processor_kwargs = {"max_pixels": 1843200, "min_pixels": 4 * 28 * 28}  # ~1800 vision tokens
QUERY_PROMPT    = "Find a document image that matches the given query."
DOCUMENT_PROMPT = "Represent the user's input."

Usage

python

import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "johnandru/Qwen3-VL-Embedding-2B-ko-vdr-distill",
    model_kwargs={"attn_implementation": "flash_attention_2", "torch_dtype": torch.bfloat16},
    processor_kwargs={"max_pixels": 1843200, "min_pixels": 4 * 28 * 28},
)
if hasattr(model[0], "unpad_inputs"):
    model[0].unpad_inputs = False

q_emb = model.encode(["설계기준 강화 관련 문서"], prompt="Find a document image that matches the given query.")
d_emb = model.encode(["path/to/page.png"], prompt="Represent the user's input.")  # image path
sim = model.similarity(q_emb, d_emb)

Embedding dimension: 2048 · cosine similarity · bf16.
Modalities: text query ↔ document page image.

Notes & limitations

Preview / experimental. The distillation was trained at ~980 vision tokens with a generic prompt; a re-train at the recommended config (1800 tokens + the prompts above) would likely widen the gain further. Evaluation so far is single-domain.
Data: Korean public-sector documents (49 PDFs, synthetic queries via Solar Pro 3).