BDI-batch: Fine-tuned Sentence Transformers for Depression Symptom Retrieval
Model Description
BDI-batch is a specialized sentence transformer model fine-tuned using contrastive learning for the precise retrieval of sentences related to depression symptoms defined in the Beck Depression Inventory (BDI). This model builds upon sentence-transformers/all-mpnet-base-v2 and has been optimized to distinguish between closely related depressive symptoms.
Unlike general-purpose sentence embeddings, BDI-batch is specifically designed to capture fine-grained semantic distinctions among the 21 symptoms defined in the BDI clinical questionnaire. The model was trained using a novel in-batch contrastive learning approach that leverages the structure of standardized clinical questionnaires to generate meaningful hard negatives automatically.
The model particularly excels at disambiguating between closely related symptoms such as:
"Sadness" vs. "Crying"
"Self-dislike" vs. "Pessimism" vs. "Past Failure"
"Changes in Sleeping Patterns" vs. "Loss of Interest in Sex"
"Irritability" vs. "Agitation"
How to Use
Installation
pip install sentence-transformers
Basic Usage
python
1from sentence_transformers import util
2from sentence_transformers import SentenceTransformer
34model = SentenceTransformer('your-username/bdi-batch-all-mpnet-base-v2')56# BDI symptom queries7query ="I feel sad much of the time"# Sadness symptom89# Sentences to rank10sentences =[11"I feel so sad that I can't bear it",12"I feel more irritable than usual",13"I've been crying a lot lately",14"I don't enjoy things like I used to"15]1617# Encode18query_embedding = model.encode(query, convert_to_tensor=True)19sentence_embeddings = model.encode(sentences, convert_to_tensor=True)2021# Compute similarity scores22similarity_scores = util.pytorch_cos_sim(query_embedding, sentence_embeddings)[0]2324# Rank by similarity25ranked_results =sorted(26zip(sentences, similarity_scores),27 key=lambda x: x[1],28 reverse=True29)3031for sentence, score in ranked_results:32print(f"{score:.4f} - {sentence}")
Advanced Usage: Batch Processing
python
1# For processing multiple symptom queries2bdi_symptoms ={3"Sadness":"I feel sad much of the time",4"Crying":"I cry more than I used to",5"Pessimism":"I do not expect things to work out for me",6"Loss of Pleasure":"I don't enjoy things as much as I used to"7}89# Encode corpus10corpus =[11"I'm always sad and can't find joy in anything",12"Everything seems hopeless to me",13"I cry at the smallest things now"14# ... more sentences from social media corpus15]16corpus_embeddings = model.encode(corpus, convert_to_tensor=True)1718# Retrieve relevant sentences per symptom19for symptom_name, symptom_query in bdi_symptoms.items():20 query_embedding = model.encode(symptom_query, convert_to_tensor=True)21 similarity_scores = util.pytorch_cos_sim(query_embedding, corpus_embeddings)[0]2223# Get top-k most similar sentences24 top_k_results = util.semantic_search(query_embedding, corpus_embeddings, top_k=10)
Semantic Search
python
1from sentence_transformers import util
23query ="I have been thinking about suicide"4corpus =["Your corpus of sentences here..."]56# Encode query and corpus7query_embedding = model.encode(query, convert_to_tensor=True)8corpus_embeddings = model.encode(corpus, convert_to_tensor=True)910# Perform semantic search11results = util.semantic_search(query_embedding, corpus_embeddings, top_k=10)1213# Print results14for result in results[0]:15print(f"Score: {result['score']:.4f} - {corpus[result['corpus_id']]}")
Training Details
Contrastive Learning Approach
The model was fine-tuned using a novel in-batch contrastive learning method that leverages the structure of the Beck Depression Inventory:
Hard Negative Generation: Instead of random negatives, the method groups related BDI symptoms and creates contrastive batches containing the most semantically similar symptoms (k=10 symptoms per batch)
Training Signal: For each BDI symptom, two random responses from the questionnaire are sampled to create positive pairs. Other symptoms in the batch serve as hard negatives, forcing the model to learn fine-grained distinctions.
Loss Function: MultipleNegativesRankingLoss encourages pairs from the same symptom to be close while pushing apart different symptoms in the embedding space.
Training Configuration
Base Model: sentence-transformers/all-mpnet-base-v2
Batch Size: 10 symptoms per batch
Number of Epochs: 10
Learning Rate: 5e-5
Warmup Steps: 100
Hardware: Single node with NVIDIA RTX 5090 GPU (32GB VRAM)
Training Dataset: DepreSym (~21.5K sentences annotated for 21 BDI symptoms)
Key Innovation: Clinical Grounding
Unlike synthetic hard negatives generated by LLMs, this method leverages clinically validated questionnaires:
BDI questionnaire responses are designed by experts to be mutually informative
21 distinct but sometimes overlapping symptoms ensure meaningful hard negatives
No labeled data required—pairs are generated automatically from the questionnaire structure
Result: The questionnaire-derived pairs significantly outperformed LLM-generated synthetic data (Table 3 in the paper)
Limitations
Dataset Limitations
Social Media Source: Training data comes exclusively from social media users, which may not represent all demographics (biased toward younger, online-active populations)
Language: Currently only supports English-language content
Geographic/Cultural Scope: Data may reflect primarily Western contexts
Model Limitations
Clinical Questionnaire Dependency: The model is optimized for BDI-specific symptom definitions and may not transfer seamlessly to alternative depression assessment tools (PHQ-9, DASS-21, etc.)
Fine-grained Overlap: Some symptoms remain inherently difficult to distinguish (e.g., "Self-dislike" from "Crying"), reflecting real clinical complexity
Not for Diagnosis: Performance metrics measure retrieval accuracy, not clinical validity or diagnostic utility
Context Dependence: Model relies on explicit first-person language to identify relevant sentences; implicit mentions of symptoms may be missed
Ethical and Practical Constraints
Not a Diagnostic Tool: This model should never be used as a standalone diagnostic instrument. It is intended for research and information retrieval only.
Requires Expert Oversight: Any real-world application must include human review by qualified mental health professionals
Privacy Concerns: Analyzing social media text for mental health signals raises contextual integrity issues; users may not expect their posts to be analyzed this way
Dual-Use Risk: Techniques developed for mental health monitoring could be misused for surveillance or discriminatory profiling
Benchmarks
Comparison with Baselines
The model outperforms several well-established baselines:
Model
R@100
NDCG@10
NDCG@1000
BM25
0.141
0.356
0.404
BM25 + Cross-Encoder
0.141
0.704
0.446
ANCE
0.274
0.802
0.664
all-mpnet-base-v2
0.291
0.834
0.698
Contriever
0.273
0.761
0.686
BDI-batch (ours)
0.372
0.947
0.817
Comparison with eRisk Participants
The model also outperforms specialized systems from the eRisk 2024 challenge:
AP3CM (best precision): R@100=0.291
NUS IDS (ensemble approach): R@100=0.294
BDI-batch: R@100=0.315 (eRisk 2024 collection)
Environmental Impact
Model inference is computationally efficient:
Embedding Size: 384 dimensions (same as base model)
Inference Time: Similar to all-mpnet-base-v2
Model Size: ~438MB (same as base model)
GPU Memory Required: ~2GB for standard batch processing
Citation
If you use this model, please cite the following paper:
bibtex
1@article{fernandez2026bdi,
2 title={BDI-batch: Leveraging Standardized Clinical Questionnaires for Contrastive Learning in Psychological Marker Retrieval},
3 author={Fern{\'a}ndez-Pichel, Marcos and Losada, David E.},
4 journal={Findings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
5 year={2026},
6 organization={Association for Computational Linguistics}
7}
Disclaimer: This model is provided for research purposes only. It should not be used for clinical diagnosis, treatment recommendations, or real-time monitoring of individuals without proper validation, regulatory compliance, and expert oversight. The authors and their institutions are not responsible for misuse of this model or harm resulting from its application.
