Property	Value
Base model	Qwen/Qwen2.5-0.5B
Training type	Full-parameter Continued Pre-Training (CPT)
Dataset	VietAI/vi_pubmed (92k English abstracts)
Tokens trained on	~23.6 million
Block size	256 tokens
Training objective	Causal Language Modeling (CLM)
Optimizer	AdamW 8-bit (bitsandbytes)
Learning rate	2e-5 (cosine schedule)
Hardware	Kaggle Tesla T4 (15.6GB VRAM)
Training time	~3h 45m
Starting loss	2.581
Final loss	~2.48
Precision	fp32 master weights + AMP fp16

What is this model?

This is a base model — not an instruction-tuned or chat model. It is intended as a domain-adapted foundation for further fine-tuning on medical instruction datasets.

All 494M parameters were updated during training (no LoRA, no frozen layers). The model has been adapted toward biomedical vocabulary, PubMed abstract structure, and medical terminology through full-parameter CLM training.

Intended Use

Base model for downstream medical SFT
Research into biomedical domain adaptation
Starting point for medical reasoning models

Not Intended For

Direct conversational use (no instruction tuning)
Clinical decision making
Patient-facing applications

Rumiii/Qwen2.5-0.5B-Med-Pre-Trained-92k

Qwen2.5-0.5B-Med-Pre-Trained-92k

Training Details

What is this model?

Intended Use

Not Intended For

Rumiii/
Qwen2.5-0.5B-Med-Pre-Trained-92k