A fine-tuned BERT-based model for multi-class mental health text classification, achieving 89.7% accuracy on held-out test data.
Model Description
This model is a fine-tuned version of mental/mental-bert-base-uncased designed to classify text into four mental health categories:
Anxiety
Depression
Normal
Suicidal
Base Model: MentalBERT (BERT-Base uncased, pre-trained on mental health-related Reddit posts)
Architecture: BertForSequenceClassification with 4 output labels Parameters: ~110M (BERT-Base: 12 layers, 768 hidden dimensions, 12 attention heads)
Intended Use
Primary Use Cases
Research on mental health text analysis
Early detection support systems for mental health concerns in online social content
Sentiment analysis in mental health contexts
Supporting mental health monitoring and research
Target Users
Mental health researchers
Clinical researchers
Data scientists working on mental health NLP projects
Social workers and support organizations
⚠️ Important: This model is NOT intended for clinical diagnosis. It is a supplementary research tool and should not replace professional mental health evaluation or therapy. Model predictions are not psychiatric diagnoses, and anyone struggling with mental health issues should seek professional help.
Training Data
Datasets
The model was trained on a combined dataset from multiple sources, which were then integrated into a unified 4‑class corpus published as ourafla/Mental-Health_Text-Classification_Dataset.
Mental Health Text Classification Dataset (4‑Class)
Curated, cleaned, and relabeled 4‑class dataset combining several public mental‑health corpora
Hosted on Hugging Face Hub as: ourafla/Mental-Health_Text-Classification_Dataset
The final training corpus used for this model corresponds to the processed version released as ourafla/Mental-Health_Text-Classification_Dataset.
Data Preprocessing
Text normalization and cleaning
Label standardization across datasets
Duplicate removal
Class balancing to ensure equal representation (248 samples per class in test set)
Data Split
Training: ~49,382 samples (balanced across 4 classes)
Validation: ~5,487 samples (10% holdout)
Test: 992 samples (248 per class, balanced)
Training Procedure
Hyperparameters
Optimizer: AdamW
Learning rate: 2e-5
Weight decay: 1e-2
Epochs: 5
Batch size: 16 (training), 32 (validation/test)
Max sequence length: 128 tokens
Scheduler: Linear warmup (10% of training steps)
Loss function: CrossEntropyLoss with class weights
Training Environment
Platform: Google Colab with GPU (Tesla T4)
Framework: PyTorch with Hugging Face Transformers 4.45.1
Training time: ~80.39 minutes (5 epochs)
Training Phases
The model underwent 3 development phases:
Phase 1: Initial baseline training
Phase 2: Refined preprocessing and model optimization
Phase 3: Enhanced training with improved data balancing and class weights
Performance
Test Set Results (Phase 3)
Metric
Score
Accuracy
89.72%
Macro Precision
89.56%
Macro Recall
89.72%
Macro F1-Score
89.54%
Per-Class Performance
Class
Precision
Recall
F1-Score
Support
Anxiety
0.88
0.85
0.87
248
Depression
0.86
0.78
0.82
248
Normal
0.94
0.98
0.96
248
Suicidal
0.91
0.98
0.94
248
Key Observations
Strongest performance: Normal (96% F1) and Suicidal (94% F1) classes
Moderate performance: Anxiety (87% F1) and Depression (82% F1) classes
Challenge: Some confusion between Anxiety and Depression classes (common in mental health classification)
The model demonstrates strong generalization across all four mental health categories
🔍 Advanced Model Evaluation & Error Analysis
To better understand the model’s behaviour beyond aggregate metrics, an additional evaluation notebook is provided on Kaggle. This analysis focuses on class-wise errors, confusion patterns, and probability calibration, with particular attention to uncertainty in linguistically overlapping categories such as Anxiety and Depression.
The intent of this evaluation is not to claim clinical reliability, but to transparently examine where the model performs well and where it remains limited.
1import torch
2from transformers import AutoTokenizer, AutoModelForSequenceClassification
34# Load model and tokenizer5model_name ="mental/mental-bert-base-uncased"6tokenizer = AutoTokenizer.from_pretrained(model_name)7model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=4)89# Load fine-tuned weights10device = torch.device("cuda"if torch.cuda.is_available()else"cpu")11model.load_state_dict(torch.load("best_phase3.pth", map_location=device))12model.to(device)13model.eval()1415# Define label mapping16id2label ={0:"Anxiety",1:"Depression",2:"Normal",3:"Suicidal"}1718# Example inference19text ="I've been feeling really overwhelmed and anxious lately"20inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=128)21inputs ={k: v.to(device)for k, v in inputs.items()}2223with torch.no_grad():24 outputs = model(**inputs)25 prediction = torch.argmax(outputs.logits, dim=-1).item()2627print(f"Predicted class: {id2label[prediction]}")
Batch Inference
python
1texts =[2"I feel hopeless and don't see the point anymore",3"Had a great day today, feeling positive!",4"My heart is racing and I can't stop worrying"5]67inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=128)8inputs ={k: v.to(device)for k, v in inputs.items()}910with torch.no_grad():11 outputs = model(**inputs)12 predictions = torch.argmax(outputs.logits, dim=-1)1314for text, pred inzip(texts, predictions):15print(f"Text: {text[:50]}...")16print(f"Predicted: {id2label[pred.item()]}\n")
Limitations and Considerations
Known Limitations
Not a diagnostic tool: Cannot replace professional mental health assessment
Text-only analysis: Does not consider non-verbal cues, medical history, or clinical context
Class imbalance challenges: Some confusion between Depression and Anxiety categories
Language bias: Trained primarily on English text from social media
Cultural context: May not generalize well across different cultural expressions of mental health
Temporal limitations: Trained on historical data; language use evolves
Ethical Considerations
Privacy: All training data was from publicly available, anonymized sources
Bias: Model may reflect biases present in training data (Reddit demographics)
Responsible use: Should be used as a screening tool only, not for definitive diagnosis
Professional oversight: Any clinical applications must involve mental health professionals
Informed consent: Users should be aware that their text is being analyzed
Potential Biases
Reddit user demographics (younger, predominantly Western)
Self-reported mental health states (not clinically verified)
Language and expression styles specific to online communities
Underrepresentation of certain mental health conditions
If you use this model in your research, please cite:
bibtex
1@software{mental_health_classifier_2025,
2 author = {Mukherjee, Priyangshu},
3 title = {Mental Health Text Classifier (MentalBERT Fine-tuned)},
4 year = {2025},
5 note = {Fine-tuned model for multi-class mental health text classification}
6}
Base Model Citation:
bibtex
1@inproceedings{ji2022mentalbert,
2 title = {{MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare}},
3 author = {Shaoxiong Ji and Tianlin Zhang and Luna Ansari and Jie Fu and Prayag Tiwari and Erik Cambria},
4 year = {2022},
5 booktitle = {Proceedings of LREC}
6}
Acknowledgments
Base Model: mental/mental-bert-base-uncased by Shaoxiong Ji et al.
Frameworks: Hugging Face Transformers, PyTorch
Datasets: Suicide Detection Dataset, Reddit Mental Health Posts
Compute: Google Colab GPU resources
License
This model is released for research and non-commercial use. Please check the base model license at mental/mental-bert-base-uncased for additional terms.
