Base model trained on 38B tokens of python code from synthetic FIM samples derived from merged and denoised TheStack V1 and OpenCoder RCC datasets.

Model was tuned on a single rtx 4090 GPU for around 2 days.

This a toy model with very small context window of 2048 tokens, but it is already usable.

Training scripts are here: https://github.com/rlewczuk/gsn-dyplom

FIM prompt format:

<|fim_prefix|>code before 
some more code before<|fim_middle|>code after
some more code after<|fim_suffix|>

Note that compared to other solutions <|fim_middle|> and <|fim_suffix|> are swapped due to mistake in preparing FIM data. This will be fixed in v5 (experiment 3).