Quantization made by Richard Erkhov.
This is the chat version of Fietje, a DPO-tuned (aligned) continuation on
the instruct version. Fietje is an adapated version of
microsoft/phi-2, tailored to Dutch text generation by training on 28B tokens. It is small and efficient with a size of 2.7 billion parameters while performing almost on par with more powerful Dutch LLMs of twice its size like
GEITje 7B Ultra.
A thorough description of the creation and evaluation of Fietje as well as usage examples are available in
this Github repository.
The same limitations as
phi-2, and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!
Fietje 2 Chat was finetuned from
the instruct model on the following datasets. Number of training samples per dataset given in brackets, totalling 18,653 samples.
A lot of different learning rates, beta, en batch sizes were investigated in search of a converging combination. You can find them all in
the W&B runs.
I am thankful to the
Flemish Supercomputer Center (VSC) for providing the computational power to accomplish this project. Accounting for waiting for jobs, training a single run took around nine hours on one A100 80GB.
Training was done with the wonderful
alignment-handbook, using DeepSpeed as a back-end. Exact training recipes and SLURM script are given in the
Github repository.
Results for the English Open LLM Leaderboard. For results specific to Dutch, check out
ScandEval.
Detailed results can be found
here
