Two LoRA adapters for Qwen/Qwen3-VL-Embedding-8B that turn it into a Traditional-Chinese multimodal document retriever over the Mr. Right zh-TW corpus (769,245 Wikipedia-derived image+text documents).

On Traditional-Chinese queries the pair lifts the backbone from R@1 .5459 → .7283 and R@10 .8134 → .9052 (test, clean protocol, n = 1,951), for 43.6 M trainable parameters — no backbone weights are redistributed.

⚠️ The deployment is asymmetric — you need both adapters

subfolder	checkpoint	role
`query_encoder`	`embedder_11_r5`	encodes queries. This is the deployed query tower.
`doc_encoder`	`embedder_11_r4`	encodes documents. This is the tower that produced the `zhen_img_v2` document bank.

Query and document vectors only live in the same space if each side uses its own adapter. Using query_encoder for documents (or vice versa) will silently degrade retrieval. If you use the pre-computed embeddings, they were produced by doc_encoder and you only need query_encoder at query time.

Usage

python

import torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

BASE = "Qwen/Qwen3-VL-Embedding-8B"
REPO = "ericssonbear/qwen3-vl-emb-8b-mrright-zhtw-lora"

# NOTE: load the ...ForConditionalGeneration class, NOT AutoModel.
# The checkpoint tensors are prefixed `model.`; AutoModel resolves to the bare
# Qwen3VLModel, whose keys do not match -- it silently random-initialises every
# weight and the LoRA keys do not match either. Always check loading info.
processor = AutoProcessor.from_pretrained(BASE, max_pixels=1024 * 28 * 28)
full, info = Qwen3VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda", output_loading_info=True)
assert not [k for k in info["missing_keys"] if k != "lm_head.weight"]
assert not info["unexpected_keys"]

PeftModel.from_pretrained(full, REPO, subfolder="query_encoder", is_trainable=False)
full.eval()
base = full.model                    # inner module -> last_hidden_state

QUERY_INSTRUCTION = "給定一個圖文混合查詢，檢索相關的維基百科圖文文檔："

def encode(texts):
    prompts = [processor.apply_chat_template(
        [{"role": "user", "content": f"指令：{QUERY_INSTRUCTION}\n查詢：{t}"}],
        tokenize=False, add_generation_prompt=False) for t in texts]
    inp = processor(text=prompts, return_tensors="pt", padding=True,
                    truncation=True, max_length=1024).to("cuda")
    with torch.inference_mode():
        h = base(**inp, return_dict=True).last_hidden_state
    last = inp["attention_mask"].sum(dim=1) - 1          # last-token pooling
    v = h[torch.arange(h.size(0), device=h.device), last].float()
    return torch.nn.functional.normalize(v, p=2, dim=-1)

q = encode(["一座建於一八九一年、位於康涅狄格州新倫敦、並作為中學使用的大型磚造建築。"])

Retrieval is cosine similarity (both sides are L2-normalised) — a plain dot product.

Instructions matter (the model is instruction-aware). Query side, by query type:

query type	instruction
mixed (`query_multi*`)	`給定一個圖文混合查詢，檢索相關的維基百科圖文文檔：`
text (`query_text*`)	`給定一個主題式文字查詢，檢索其所指的維基百科圖文文檔：`
image-description (`query_img*`)	`給定一段圖片外觀描述，檢索包含匹配圖片的維基百科圖文文檔：`

Document side always uses 為以下文檔產生向量表示，以用於檢索任務：, with the image passed as an image content block (896×896 pixel cap) followed by title_zhtw\ndoc_text_zhtw.

Results

Test split, clean protocol (gold document present in the corpus, n = 1,951), query_multi used as plain text — see Evaluation caveats.

Against retrieval baselines (Traditional-Chinese queries)

retriever	R@10	MRR@10
BM25 (jieba + Elasticsearch CJK)	.7381	.5828
multilingual-e5-small, dense	.4249	.2855
Qwen3-VL-Embedding-8B, zero-shot dense	.8134	.6337
BM25 ⊕ Qwen3-VL-Embedding-8B (RRF)	.8749	.7155
this adapter pair	.9052	.7908

Every baseline sees the image too: documents are indexed with their generated Traditional-Chinese image description, so BM25 and e5 are multimodal-by-proxy rather than text-only strawmen.

Training ladder (Traditional-Chinese queries)

#	stage	what was added	R@1	R@10	MRR@10
0	zero-shot backbone	—	.5459	.8134	.6337
1	query LoRA	instruction-aware InfoNCE over the full 769k bank	.5879	.8442	.6765
2	+ distill	generative reranker (yes/no logit) distilled into the embedder	.6099	.8524	.6929
3	+ distill_xl	cross-lingual alignment + 32k pseudo-label pairs	.6274	.8590	.7068
4	+ r1	bank-align: negatives drawn from the deployed bilingual bank	.6386	.8693	.7193
5	+ r2s	margin loss + 16k-step consolidation	.6356	.8744	.7178
6	+ r4	document tower unfrozen (`doc_encoder`), corpus re-encoded → v2 bank	.6766	.8816	.7463
7	+ r5 (deployed)	mt-mix cross-lingual query training (`query_encoder`)	.7283	.9052	.7908

Cumulative: R@1 +18.24 pp, R@10 +9.18 pp, MRR +15.71 pp over the zero-shot backbone. The three biggest levers are mt-mix query training (+5.17 pp R@1), the initial instruction LoRA (+4.20), and unfreezing the document tower (+4.10).

Cross-lingual

query language	R@1	R@10	MRR@10
Traditional Chinese	.7283	.9052	.7908
English	.7678	.9339	.8255

English queries are not worse than Chinese — the adapters are cross-lingually symmetric over a bilingual document bank.

Does the image actually matter?

Single-factor ablation, same adapters, only the document modality changes:

document bank	R@1	R@10	MRR@10
image + text (deployed)	.7283	.9052	.7908
text only	.6115	.8319	.6848
image only	.1363	.3178	.1890

Removing the image costs −11.7 pp R@1 / −7.3 pp R@10 (McNemar exact and 20k paired bootstrap, all p < .0001). The image-only bank still reaches R@10 .32 out of 769,245 candidates (chance ≈ 1e-5), so the two modalities carry independent, complementary signal.

What did not work (reported deliberately)

BM25 fusion is a net negative under r5. RRF with BM25 costs −3.4 pp MRR (p = .0001). The deployed system is a single dense leg with no lexical index. This was not true at earlier rungs — BM25 helped until the retriever got strong enough.
Explicit margin loss: zero gain. With the full 769k corpus in the InfoNCE denominator at temperature 0.03, the objective already concentrates gradient on the hardest negatives; static hard-negative mining is redundant.
LLM-judged hard positives: zero gain. Re-training on human/judge-reviewed pairs (embedder_13_h) did not beat r5 and was not deployed.

If you are reproducing this recipe, these three cost real compute and bought nothing.

Evaluation caveats

The query_multi image trap. The upstream mixed query ships with a query image, and 1,992 of ~2,000 test queries use the same image file as their gold document. An untrained backbone scores R@10 = .952 on it — that is image hashing, not retrieval. All numbers here use query_multi as plain text, with no query image.
Clean protocol. 96 of 2,047 test queries have a gold document that is absent from the corpus (dead image URL), imposing a recall ceiling of .953 on the full set. Numbers are on the clean subset, n = 1,951.
Comparing against numbers computed under either different convention is meaningless.

Offline ceiling (not deployed)

Adding a training-free late-interaction (MaxSim) rerank over the top-20, scored with doc_encoder token states, reaches R@1 .7719 / R@10 .9257 / MRR .8237. It roughly doubles query latency and was left out of the deployment.

Training setup

LoRA rank 16, alpha 32, dropout 0.05, on the language-model tower only (self_attn.{q,k,v,o}_proj and mlp.{gate,up,down}_proj for all 36 layers) — the vision tower and merger are untouched. 504 LoRA tensors, 43.6 M parameters. Objective: InfoNCE at temperature 0.03 with the full 769,245-document bank as negatives, Matryoshka losses at 4096/2048/1024/512/256. Trained on the NCHC 晶創 (Nano5) cluster. Base weights are bf16; adapters are fp32.

Training data, all from ericssonbear/mr-right-zhtw: the 900 human-annotated pairs in queries-finetune (rung 1, using the *_zhtw Gemma 4 translation — not the *_zhtw_hv human-verified one), then the pseudo-pairs-zh and pseudo-pairs-hard configs for the later rungs. The embedder_13_h null result above is exactly the experiment of swapping in the human-verified translations.

Limitations and intended use

Tuned for retrieval over this Wikipedia-derived corpus in Traditional Chinese and English. It is not a general-purpose sentence embedder and has not been evaluated on MTEB or on any other domain.
Evaluation queries are English queries written by human annotators, then translated to Traditional Chinese and individually human-verified. They are still not queries a Taiwanese speaker wrote from scratch, so they carry translationese: there is no natively authored Traditional-Chinese evaluation slice, and the zh numbers should be read as an upper bound on what native-speaker queries would give.
The document side of the corpus is bulk machine translation (Gemma 4) with no human review, so document-side translation errors are baked into the bank these adapters were trained on.
No adversarial, fairness, or domain-shift testing was done beyond the cross-lingual symmetry check above.
Document vectors were computed with a 896×896 pixel cap and max_length 1024 text; matching those settings matters if you re-encode documents yourself.
Inherits the biases of Wikipedia, of the backbone, and of the VLM that wrote the captions the documents were indexed with.

License

Adapter weights: Apache-2.0, matching the Qwen/Qwen3-VL-Embedding-8B backbone. Backbone weights are not redistributed here. The training and evaluation data are CC BY-SA 4.0 (see the dataset card).

Citation

bibtex

@misc{mrright_zhtw_lora_2026,
  title  = {Qwen3-VL-Embedding-8B Mr. Right zh-TW retrieval LoRA},
  author = {Hsin-Yu Lin and Hong-Yan Huang and Shau-Yung Hsu and Song-Duo Ma and Pin-Yu Chen and Chu-Yun Chen and Wei-Te Ho and Pu-Jen Cheng},
  year   = {2026},
  url    = {https://huggingface.co/ericssonbear/qwen3-vl-emb-8b-mrright-zhtw-lora}
}

Built by Hsin-Yu Lin, Hong-Yan Huang, Shau-Yung Hsu, Song-Duo Ma, Pin-Yu Chen, Chu-Yun Chen, Wei-Te Ho and Pu-Jen Cheng. Funded under National Science and Technology Council, Taiwan, project NSTC 115-2634-F-001-006 ("Advancing Next-Generation Frontier AI Research" programme). Compute provided by the NCHC 晶創 (Nano5) cluster.