You can deploy this model and start earning money today!

Innovatewithapple/
bert-dense-retriever

Name: bert-dense-retriever
Availability: InStock

Views

No views yet

license: mit language:

en library_name: transformers pipeline_tag: feature-extraction tags:
bert
dense-retrieval
semantic-search
information-retrieval
ms-marco
faiss

BERT Dense Retriever

A dense semantic retrieval model fine-tuned on the MS MARCO Passage Ranking dataset using BERT-base-uncased.

The model encodes natural language queries and passages into dense vector embeddings that can be indexed with FAISS for efficient semantic search.

This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.

Model Details

Backbone

BERT-base-uncased

Pooling

Mean Pooling

Embedding Normalization

L2 Normalization

Similarity Metric

Cosine Similarity

Training Objective

CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)

Training Configuration

Parameter	Value
Optimizer	AdamW
Learning Rate	2e-5
Batch Size	32
Epochs	10
Weight Decay	0.01
Temperature	0.05

Evaluation Results

Model	Recall@10	MRR	nDCG@10
BERT-base-uncased	0.4793	0.2837	0.3301
Fine-tuned Dense Retriever	0.9693	0.8521	0.8810

The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.

📊 BEIR Evaluation Results

The table below compares the proposed retrieval pipeline against the BM25 baseline reported by the original BEIR benchmark.

Dataset	Retrieval Strategy	BM25 (BEIR) NDCG@10	Pipeline NDCG@10	Recall@10	Recall@100	Improvement
FEVER	Hybrid + Cross Encoder	0.7530	0.9791	0.9873	0.9937	+0.2261
Quora	Dense + Cross Encoder	0.7830	0.9686	0.9858	0.9938	+0.1856
HotpotQA	Dense + Cross Encoder	0.6030	0.8977	0.8853	0.8960	+0.2947
FiQA	Dense + Cross Encoder	0.2361	0.7512	0.8055	—	+0.5151
TREC-COVID	Hybrid + Cross Encoder	0.6559	0.6868	0.0181	0.1110	+0.0309

Note: BM25 scores are taken from the original BEIR benchmark and are included as the lexical retrieval baseline for comparison.

🔍 Key Findings

Strong zero-shot generalization across multiple retrieval domains.
Significant improvements over the BEIR BM25 baseline on FEVER, Quora, HotpotQA, and FiQA.
Hybrid retrieval (BM25 + Dense Retriever) improves retrieval quality for specialized biomedical documents in TREC-COVID.
Cross-Encoder re-ranking substantially enhances the final ranking quality by leveraging full query-document interactions.

Usage

python

from transformers import AutoTokenizer, AutoModel

model_name = "Innovatewithapple/bert-dense-retriever"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
)

inputs = tokenizer(
    "What is deep learning?",
    return_tensors="pt"
)

embeddings = model(**inputs)

print(embeddings.shape)

Intended Use

This model is designed for:

Dense semantic retrieval
Semantic search
Question-passage retrieval
Retrieval-Augmented Generation (RAG)
Information retrieval research

Source Code

GitHub Repository:

https://github.com/Innovatewithapple/dense-semantic-retrieval

Author

Mihir Vyas