ru-Promptriever-Qwen3-0.6B
Overview
Standard dense retrieval models score query–passage pairs using a single semantic similarity signal, giving users no control over
what "relevant" means beyond keyword choice.
Promptriever (
Weller et al., 2024) introduced per-instance natural language instructions that dynamically redefine relevance — a capability previously limited to generative LLMs.
ru-Promptriever extends this paradigm to Russian:
- Architecture: Qwen3-based causal LM fine-tuned as a bi-encoder with LoRA + GradCache
- Pooling: last-token (EOS) pooling, same as the original Promptriever
- Key training signal: instruction negatives — passages that are topically relevant to the query but violate the instruction constraint
Model Family
| Model | Parameters | Description | Link |
|---|
| ru-Promptriever-4B | 4B | Final model — best results | link |
| ru-Promptriever-4B-pretrained | 4B | Base pretrained on synthetic data only | link |
| ru-Promptriever-4B-ru-only | 4B | Continued training on Russian-only data | link |
| ru-Promptriever-1.7B | 1.7B | Scaling experiment | link |
| ru-Promptriever-0.6B | 0.6B | Scaling experiment | this model |
This Model
This is a scaling experiment to test whether a small (<1B parameter) model can learn instruction-aware retrieval. The model was trained from Qwen3-0.6B on a mix of real and synthetic Russian retrieval data.
⚠️
WARNING: Demonstration Only
This 0.6B model is released purely to demonstrate the scale limitations of instruction-aware retrieval. It
fails to learn instruction-following behavior (achieving negative p-MRR) and its standard retrieval quality is very poor.
Do not use this model in practice. For a functional model, use the
1.7B or
4B version.
Results show that the 0.6B model struggles to learn instruction-following behavior, achieving negative p-MRR on mFollowIR-RU. This confirms that sufficient model capacity is important for instruction-aware retrieval.
Evaluation Results
mFollowIR-RU
Russian split of
mFollowIR — multilingual instruction-following retrieval using TREC NeuCLIR narratives as instructions.
p-MRR (Pairwise Mean Reciprocal Rank, ×100) is the primary instruction-following metric — higher means the model correctly adjusts rankings when instructions change. nDCG@20 measures standard retrieval quality.
| Model | nDCG@20 | p-MRR |
|---|
| BM25 | 0.452 | +0.67 |
| mE5-large | 0.428 | −2.03 |
| BGE-M3 | 0.479 | −4.15 |
| Promptriever-8B | 0.532 | +12.21 |
| Qwen3-Embedding-4B | 0.549 | +8.10 |
| ru-Promptriever-0.6B (this model) | 0.231 | −4.35 |
| ru-Promptriever-1.7B | 0.444 | +14.65 |
| ru-Promptriever-4B-pretrained | 0.461 | +15.26 |
| ru-Promptriever-4B-ru-only | 0.512 | +16.80 |
| ru-Promptriever-4B | 0.512 | +18.57 |
Usage
Basic Retrieval (no instruction)
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3import torch.nn.functional as F
4
5model_name = "Vladimirlv/ru-promptriever-qwen3-0.6b"
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7model = AutoModelForCausalLM.from_pretrained(
8 model_name,
9 torch_dtype=torch.bfloat16,
10 device_map="auto",
11)
12model.eval()
13
14def encode(texts: list[str], max_length: int = 512) -> torch.Tensor:
15 """Encode texts using last-token (EOS) pooling."""
16 inputs = tokenizer(
17 texts,
18 padding=True,
19 truncation=True,
20 max_length=max_length,
21 return_tensors="pt",
22 ).to(model.device)
23
24 with torch.no_grad():
25 # Bypass lm_head to get post-norm hidden states
26 original_lm_head = model.lm_head
27 model.lm_head = torch.nn.Identity()
28 outputs = model(**inputs, use_cache=False, return_dict=True)
29 model.lm_head = original_lm_head
30
31 # EOS pooling: take embedding at last non-padding token
32 seq_len = inputs["attention_mask"].sum(dim=1) - 1
33 embeddings = outputs.logits[torch.arange(len(texts)), seq_len]
34 return F.normalize(embeddings, p=2, dim=1)
35
36
37query = "Когда была основана Москва?"
38passages = [
39 "Москва была основана в 1147 году князем Юрием Долгоруким.",
40 "Санкт-Петербург был основан Петром I в 1703 году.",
41]
42
43q_emb = encode([query])
44p_emb = encode(passages)
45scores = (q_emb @ p_emb.T).squeeze()
46print(scores)
Instruction-Following Retrieval
1# Append the instruction directly to the query (same format as training)
2instruction = "Найди документ, в котором упоминается конкретная дата основания города."
3instructed_query = f"{query} {instruction}"
4
5q_emb = encode([instructed_query])
6p_emb = encode(passages)
7scores = (q_emb @ p_emb.T).squeeze()
Using with sentence-transformers
This model is not compatible with sentence-transformers out of the box due to the custom EOS pooling. Use the snippet above directly with transformers.
Model Details
| Property | Value |
|---|
| Base model | Qwen/Qwen3-0.6B |
| Architecture | CausalLM bi-encoder (EOS pooling) |
| Fine-tuning method | LoRA (rank-32, α=64, all linear layers) |
| Training data | ~42k rows (20k synthetic instructed + 10k synthetic standard + 11k real MIRACL/MrTyDi) |
| Effective batch size | 128 (16 per device × 2 accum × 4 GPUs) |
| Loss | InfoNCE contrastive (temperature=0.01) |
| Learning rate | 1e-4 |
| Epochs | 2 |
| Max query length | 512 tokens |
| Max passage length | 256 tokens |
Training Data
The model was trained on a mix of:
- Russian real data (~11k) — from MIRACL and MrTyDi retrieval datasets
- Russian synthetic data (~30k) — instruction-augmented and standard pairs from ru-promptriever-dataset
Intended Use
- Research on model scaling for instruction-following retrieval
- Benchmarking small models for instruction-aware retrieval tasks
Out-of-Scope
- Production retrieval systems (use the 4B model instead)
- Commercial applications (see License below)
Limitations
- Insufficient capacity: The 0.6B model fails to learn instruction-following behavior (negative p-MRR), suggesting a minimum model size threshold for this task.
- MS MARCO origin: Training corpus derives from machine-translated English web passages.
- Noisy synthetic data: A small fraction of imperfect training examples may remain.
License
This model is released under CC BY-NC 4.0 (Creative Commons Attribution–NonCommercial 4.0 International).
The non-commercial restriction is inherited from the upstream
MS MARCO license (Microsoft Research License — non-commercial use only), which governs the training corpus.
Citation
If you use this model, please cite the original Promptriever paper:
1@article{weller2024promptriever,
2 title = {Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
3 author = {Weller, Orion and Van Durme, Benjamin and Lawrie, Dawn and
4 Paranjape, Ashwin and Zhang, Yuhao and Hessel, Jack},
5 journal = {arXiv preprint arXiv:2409.11136},
6 year = {2024}
7}
