factuality-dpr-context
Fine-tuned Dense Passage Retriever (Context Encoder) trained on positive (question, supporting passage) pairs from HotpotQA to improve factual grounding in Retrieval-Augmented Generation (RAG).
This encoder forms part of a hybrid re-ranking system to reduce hallucinations in LLM-based QA.
Model Architecture
- Base Model: facebook/dpr-ctx_encoder-single-nq-base
- Output Format: Dense embedding vector (768-dim)
- Objective: Contrastive learning with InfoNCE-style loss
- Aligns question and passage representations for strong semantic retrieval
Training Details
| Property | Value |
|---|
| Dataset | HotpotQA (fullwiki) – first 3,000 training samples |
| Positive Labels | Supporting passage titles |
| Epochs | 2 |
| Effective Batch Size | 128 |
| Optimizer | AdamW (lr=2e-5) |
| GPU | Kaggle T4 |
| Random Seed | 42 (fully reproducible) |
Training Loss Progress:
- Epoch 1 → 0.1298
- Epoch 2 → 0.0528
The fine-tuning improves DPR passage alignment, directly boosting downstream RAG relevance.
Intended Use
This model is not a standalone text generator.
It must be paired with:
✔ DPR Question Encoder → Anshul017/factuality-dpr-question
✔ RAG Generator (e.g., FLAN-T5-Large)
✔ Optional hybrid re-ranker (BM25 + NLI)
Primary use cases:
- Open-domain QA
- Retrieval-Augmented Generation
- Hallucination-aware evaluation research
Performance
Using HotpotQA 1k validation split:
| Metric | Score |
|---|
| Recall@1 | 0.5720 |
| Recall@5 | 0.7420 |
| Recall@20 | 0.8090 |
This represents a noticeable improvement over zero-shot DPR recall quality for multi-hop queries.
Related Models
These two models must be used together for correct dense retrieval.
How to Use
1from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
2
3ctx_model = DPRContextEncoder.from_pretrained("Anshul017/factuality-dpr-context")
4ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("Anshul017/factuality-dpr-context")
5
Author
Anshul Thakur M.Tech C.S.E N.I.T Hamirpur
