Quantization made by Richard Erkhov.
Fietje is an adapated version of
microsoft/phi-2, tailored to Dutch text generation by training on 28B tokens. It is small and efficient with a size of 2.7 billion parameters while performing almost on par with more powerful Dutch LLMs of twice its size like
GEITje 7B Ultra.
A thorough description of the creation and evaluation of Fietje as well as usage examples are available in
this Github repository.
If you use Fietje or the
CulturaX + Wikipedia filtered subset in your work, please cite to the following paper:
1@misc{vanroy2024fietjeopenefficientllm,
2 title={Fietje: An open, efficient LLM for Dutch},
3 author={Bram Vanroy},
4 year={2024},
5 eprint={2412.15450},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2412.15450},
9}
The same limitations as
phi-2, and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!
Fietje was continue-pretrained on 28B Dutch tokens, which includes the full Dutch component of Wikipedia (accounting for around 15%), supplemented with Dutch tokens from CulturaX. A newer version of this dataset can be found
here, which also describes the filtering that took place to ensure high data quality.
I am thankful to the
Flemish Supercomputer Center (VSC) for providing the computational power to accomplish this project. Accounting for waiting for jobs, training took around two weeks on four nodes of 4x A100 80GB each (16 total).
Training was done with the wonderful
alignment-handbook, using DeepSpeed as a back-end. Exact training recipes and SLURM script are given in the
Github repository.
