Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
modernbert-sde-v0.2 – AI Model by nasa-impact | AlphaNeural AI
You can deploy this model and start earning money today!
nasa-impact
/
modernbert-sde-v0.2
like
0
transformers
safetensors
modernbert
fill-mask
en
answerdotai/ModernBERT-base
finetune
apache-2.0
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Model Card for Indus (modernbert-sde-v0.2)
This model was further pre-trained on full Science Discovery Engine (SDE) website data from
answerdotai/ModernBERT-base
.
Model Details
Base Model
: answerdotai/ModernBERT-base
Tokenizer
: answerdotai/ModernBERT-base
Parameters
: 150M
Pretraining Strategy
: Masked Language Modeling (MLM)
Training Data
Full Science Discovery Engine (SDE) Website Data
Training Procedure
transformers Version
: 4.48.3
Strategy
: Masked Language Modeling (MLM)
Masking Strategy
:
Weighted Dynamic Masking based on Keyword Importance (YAKE) and Random Masking
The idea for masking important keywords is to force the model to generalize for "science" keywords that gives a high signal for the document
Masked Language Model Probability: 30%
Batch Size: 7
Learning rate: 5e-5
Warmup ratio: 0.1
Dataset
Total Data Size: 545,717
Validation Data Size: 10% of total size
Test Data Size: 10% of total size
