DeBERTa-v3-base for Open-Domain Multi-Hop Question Answering
A fine-tuned DeBERTa-v3-base model for extractive question answering on the HotpotQA dataset. The model is designed to work as the reader component of a Retrieval-Augmented Question Answering (QA) pipeline, where a dense retriever first retrieves relevant documents and the reader extracts the answer span from the retrieved context.
Model Overview
Base Model:microsoft/deberta-v3-base
Task: Extractive Question Answering
Framework: Hugging Face Transformers
Dataset: HotpotQA (Distractor Setting)
Language: English
Unlike generative language models, this model predicts the exact answer span from a retrieved context instead of generating free-form text.
The retriever identifies relevant Wikipedia articles, while DeBERTa extracts the answer span from the retrieved passages.
Dataset
The model was fine-tuned on the HotpotQA dataset.
Data Preparation
For each question:
Retrieve the top-10 Wikipedia articles using a dense retriever.
Assemble the retrieved passages into a context.
If retrieval misses supporting documents, use the gold supporting passages from HotpotQA.
Remove:
Yes/No questions
Samples where the answer cannot be aligned within the context
Final Dataset Size
Split
Examples
Training
84,959
Validation
4,687
Retriever
The reader depends on an external dense retriever.
Embedding Model
BAAI/bge-base-en-v1.5
Vector Database
FAISS Flat Index
Corpus
507,494 Wikipedia articles
Embedding Dimension
768
Retrieval Performance
Metric
Score
Recall@3
0.7100
Recall@5
0.7650
Recall@10
0.8150
Recall@20
0.8200
Training Configuration
Parameter
Value
Base Model
microsoft/deberta-v3-base
Optimizer
AdamW
Learning Rate
2e-5
Weight Decay
0.01
Epochs
3
Batch Size
8
Max Sequence Length
512
Document Stride
128
Tokenization
The model uses the official DeBERTa tokenizer.
Configuration:
truncation="only_second"
padding="max_length"
max_length=512
stride=128
return_overflowing_tokens=True
return_offsets_mapping=True
Answer spans are aligned using character-level offsets.
Windows that do not contain the answer are assigned start/end position 0.
Evaluation
Performance on the HotpotQA validation set:
Metric
Score
Exact Match (EM)
60.53
F1 Score
74.21
Comparison with other transformer readers:
Model
EM
F1
DeBERTa-v3-base
60.53
74.21
RoBERTa-base
60.17
73.96
ALBERT-base-v2
56.73
70.31
Example Usage
python
1from transformers import pipeline
23qa_pipeline = pipeline(4"question-answering",5 model="YOUR_USERNAME/deberta-hotpotqa",6 tokenizer="YOUR_USERNAME/deberta-hotpotqa"7)89result = qa_pipeline(10 question="Which magazine was started first, Arthur's Magazine or First for Women?",11 context=context
12)1314print(result["answer"])
Intended Use
This model is suitable for:
Open-domain Question Answering
Multi-hop Question Answering
Educational Assistants
Search Systems
Retrieval-Augmented QA Pipelines
Research in Extractive Question Answering
Limitations
This model:
Requires an external retrieval system.
Cannot retrieve documents by itself.
Produces extractive answer spans only.
Does not generate explanations.
Performance depends heavily on retrieval quality.
Was trained only on English Wikipedia.
Does not support Yes/No questions since they were removed during preprocessing.
Future Improvements
Potential improvements include:
Hybrid BM25 + Dense Retrieval
Cross-Encoder Re-ranking
Paragraph-level indexing
Iterative Multi-hop Retrieval
Confidence estimation
Citation generation
Integration with an instruction-tuned LLM for grounded answer explanations
License
This project is intended for research and educational purposes. Please ensure compliance with the licenses of the original HotpotQA dataset and the DeBERTa-v3-base model.
