This is the instruct version of Fietje, an SFT-tuned (instruction-tuned) variant of
the base model. Fietje is an adapated version of
microsoft/phi-2, tailored to Dutch text generation by training on 28B tokens. It is small and efficient with a size of 2.7 billion parameters while performing almost on par with more powerful Dutch LLMs of twice its size like
GEITje 7B Ultra.
A thorough description of the creation and evaluation of Fietje as well as usage examples are available in
this Github repository.
If you use Fietje or the
CulturaX + Wikipedia filtered subset in your work, please cite to the following paper:
1@article{vanroy2024fietje,
2 author="Vanroy, Bram",
3 title="Fietje: An open, efficient LLM for Dutch",
4 url="https://www.clinjournal.org/clinj/article/view/213",
5 journal="Computational Linguistics in the Netherlands Journal",
6 volume="14",
7 year="2025",
8 pages="473--504"
9}
The same limitations as
phi-2, and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!
Fietje 2 instruct was finetuned from
the base model on the following datasets. Number of training samples per dataset given in brackets, totalling 201,579 samples.
I am thankful to the
Flemish Supercomputer Center (VSC) for providing the computational power to accomplish this project. Accounting for waiting for jobs, training took around a day on four nodes of 4x A100 80GB each (16 total). I cannot find the exact time anymore and I do not think that the runtime in
all_results.json accounts for interrupted-and-continued jobs.
Training was done with the wonderful
alignment-handbook, using DeepSpeed as a back-end. Exact training recipes and SLURM script are given in the
Github repository.
