Biomed-GPT-Small
Overview
Biomed-GPT-Small is a lightweight GPT-style transformer model trained from scratch on a small biomedical corpus for educational and research purposes.
This model demonstrates:
- End-to-end LLM training using PyTorch
- Custom SentencePiece tokenizer
- Token-level next-token prediction
- Hugging Face Model Hub integration via
PyTorchModelHubMixin
The model was trained on a toy biomedical dataset and is intended for experimentation, learning, and portfolio demonstration.
Architecture
- Transformer-based GPT model
- Context length: 32 tokens
- Embedding dimension: 128
- Layers: 4
- Heads: 4
- Vocabulary size: 128
- Framework: PyTorch
Training Data
The model was trained on a small curated biomedical text corpus for demonstration purposes.
It has not been trained on full PubMed or clinical datasets.
Intended Use
This model is intended for:
- Educational purposes
- LLM architecture experimentation
- Biomedical NLP prototyping
Limitations
- Very small dataset
- Strong overfitting expected
- Not suitable for real-world biomedical tasks
- No evaluation benchmark performed
⚠ Medical Disclaimer
This model is not validated for clinical use.
It must not be used for:
- Diagnosis
- Treatment recommendations
- Clinical decision making
- Patient care
Future Work
- Train on large-scale PubMed abstracts
- Increase vocabulary size
- Increase context length
- Evaluate perplexity on biomedical benchmarks
- Compare against BioGPT / PubMedGPT
Author
Oscar Yanez
Biomedical AI & LLM Engineering
