Llama-3.2-1B-Instruct, with domain adapted pretraining (DAPT), also called Continuous Pre-training (CPT) on a Dutch medical corpus, slightly biased towards cardiology.
Training for one full epoch, with a 256 batch size, maximally 768 sequence length and a linear-cosine schedule (details follow..).
This model will be further pre-trained on 5 million cardiology records from the UMCU.
The perplexity was around 5 on the validation set.
Note: this is not instruction tuned, and does not generate an EOS token. Update coming.
1@misc{vanes2026languagecorporadutchmedical,
2 title={Language corpora for the Dutch medical domain},
3 author={B. van Es},
4 year={2026},
5 eprint={2604.25374},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2604.25374},
9}
