ModernBERT Medical Precision & Factual Detail Regressor
The ModernBERT Medical Precision & Factual Detail Regressor is a transformer-based language model that predicts the level of precision and factual detail within medical or biological texts. Built upon the ModernBERT architecture, this model provides a continuous score (1–5) indicating how precise and factually detailed a given text is. These scores can help users filter, categorize, or prioritize documents based on their informational quality.
This model leverages ModernBERT’s Rotary Positional Embeddings, local–global alternating attention, and Flash Attention, providing extended context windows (up to 8,192 tokens) and fast, memory-efficient inference.
Intended Uses & Limitations
Intended Uses
Medical/Scientific Document Filtering: Identify texts with high precision and factual detail to focus on reliable information sources.
Data Curation: Aid in building high-quality corpora by prioritizing documents that exhibit strong factual rigor.
Limitations
Domain Shift: Primarily trained on medical and biological data. May not generalize well to non-medical text or highly specialized domains not covered in the dataset.
Score Interpretation: The raw regression output (1–5) requires clear thresholds or binning strategies, depending on the downstream application.
How to Use
You can run inference using the Hugging Face Transformers library as follows:
python
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
34# Load the tokenizer and model5tokenizer = AutoTokenizer.from_pretrained("TheBlueScrubs/ModernBERT-base-TBS-MedicalPrecision")6model = AutoModelForSequenceClassification.from_pretrained("TheBlueScrubs/ModernBERT-base-TBS-MedicalPrecision")78# Example text9text ="A recent randomized trial found that combining targeted therapy with immunotherapy improved survival rates for melanoma patients."1011# Tokenize input12inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096)1314# Get model predictions15outputs = model(**inputs)16predictions = outputs.logits
1718# The model outputs a single continuous score indicating the level of precision & factual detail19precision_score = predictions.item()20print(f"Precision & Factual Detail Score: {precision_score}")
Training Data
A balanced subset of The Blue Scrubs dataset was prepared, each text featuring a “Precision and Factual Detail” label (1–5). Steps included:
Data Cleaning: Removed rows with parsing errors, NaNs, or out-of-range values.
Balancing: Ensured a roughly even distribution of documents across lower and higher precision scores.
Training Procedure
Preprocessing
Tokenizer: ModernBERT tokenizer with a maximum sequence length of 4,096.
No additional filtering beyond standard data cleaning.
Training Hyperparameters
Learning Rate: 2e-5
Number of Epochs: 5
Batch Size: 16 (per device)
Gradient Accumulation Steps: 1
Optimizer: AdamW
Weight Decay: 0.01
FP16 Training: Enabled
Total Training Steps: ~5 epochs on the balanced set
Training used multi-GPU distributed data parallelism, with frequent evaluations (1/5 epoch) based on the mse metric.
Evaluation
Testing Data
The final model was evaluated on an out-of-sample test set. This dataset contained medical documents not included in the training or validation splits.
Metrics
Mean Squared Error (MSE): ~0.5671
Accuracy (with threshold ≤ 2.0 for “low precision” vs. > 2.0): 0.9630
ROC Analysis: Demonstrated robust classification capability with high True Positive Rates and low False Positive Rates.
image/png
Bias, Risks, and Limitations
Data Bias: Underrepresented subfields or rare document types may impact performance.
Misinterpretation: A single numeric score is not a guarantee of clinical accuracy or evidence-based correctness.
Domain Evolution: Medical knowledge evolves quickly; periodic retraining or re-validation is recommended.
Recommendations
Domain-Specific Adjustments: Consider further fine-tuning if applying to highly specialized medical subdomains.
Score Thresholding: Set context-appropriate cutoffs or categories (e.g., “low,” “moderate,” “high” precision) based on your downstream needs.
Continuous Monitoring: Maintain up-to-date evaluations as new data or medical findings emerge.
Citation
If you utilize this model in your research or applications, please cite it as follows:
bibtex
1@misc{thebluescrubs2025modernbert,
2 author = {TheBlueScrubs},
3 title = {ModernBERT Medical Precision & Factual Detail Regressor},
4 year = {2025},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/TheBlueScrubs/ModernBERT-base-TBS-MedicalPrecision}
7}
