Base model microsoft/deberta-v3-large
Domain data 1.2 M English tweets mentioning long-COVID (January 2020 → May 2025)
Objective Masked-Language-Modeling (MLM) for downstream COVID-NLP tasks
Trained with 🤗 Transformers 4.41 / 🤗 Datasets 2.21 / PyTorch 2.3 on 2 × NVIDIA H100 (192 GB)

Split	# examples	Tokens / tweet (p50 / p95)
Train	1,140,781	23 / 72
Valid	60,041	22 / 71

🏁 Final metrics (epoch 20)

Metric	Value
Validation loss	1.5642
Validation perplexity	4.78

Settings

BASE_CHECKPOINT = "microsoft/deberta-v3-large" # DeBERTa v3 base checkpoint
MAX_SEQ_LENGTH = 128 # Tweets rarely exceed 128 tokens
MLM_PROB = 0.15 # Mask 15% of tokens
NUM_EPOCHS = 20
LEARNING_RATE = 5e-5 # Scaled for batch size 64×2
PER_DEVICE_BATCH_SIZE = 32 # Batch per GPU
OUTPUT_DIR = "./pretrain_deberta_large_covid_with_val/"
LOGGING_STEPS = 500
SAVE_STEPS = 5_000
FP16 = True # Mixed precision on H100
VALIDATION_SPLIT_RATIO = 0.05 # 5% of tweets for validation

---

Intended use & limitations

✔️ Good for	❌ Not for
Domain-specific pre-training before fine-tuning on COVID classification, NLI, QA, summarisation	Open-ended medical advice to patients
MLM-style feature extraction, embeddings, or continued pre-training	Determining actual infection status or diagnosing disease
Research on domain adaptation & concept drift	Use outside English tweets (no multilingual coverage)

The model does not make factual guarantees; tweets are noisy and may contain misinformation. Always fine-tune on a task-specific, quality-controlled dataset and inspect outputs manually.

Training details

text

python -m pip install -U transformers datasets accelerate==0.29
torchrun --nnodes 1 --nproc_per_node 2 pretrain_deberta_v3_large_covid.py \
  --epochs 20 --lr 5e-5 --per_device_bs 32 --fp16 \
  --csv_pattern ".../*.csv" --output_dir ".../pretrain_deberta_large_covid_with_val/"

Citation

text

@misc{dap-deberta-covid2025,
  title   = {DeBERTa-v3-large Domain-Adaptive Pre-Training on LONG-COVID Tweets},
  author  = {Kumar, S. and Contributors},
  year    = {2025},
  howpublished = {\url{https://huggingface.co/your-username/deberta-v3-large-covid2025-mlm}}
}

PHMC34961/DeBERTa-v3-large-longCovid-tweets

DeBERTa-v3-large Domain-Adaptive Masked-Language Model for LONG-COVID Tweets

🏁 Final metrics (epoch 20)

Settings

Intended use & limitations

Training details

Citation

PHMC34961/
DeBERTa-v3-large-longCovid-tweets

DeBERTa-v3-large
Domain-Adaptive Masked-Language Model for LONG-COVID Tweets