Base model trained on 14.85B tokens of python code from synthetic FIM samples derived from merged and denoised TheStack V1 and OpenCoder RCC datasets.

Model was tuned on a single rtx 4090 GPU for around 4 days and 12 hours.

This a toy model with very small context window of 2048 tokens, but it is already usable.

Training scripts are here: https://github.com/rlewczuk/gsn-dyplom

FIM prompt format:

<|fim_prefix|>code before 
some more code before<|fim_suffix|>code after
some more code after<|fim_middle|>

This is pretty standard format also used in other models like qwen or deepseek. Be sure to cofigure inference server to stop when <|eos|> token is generated.

Whole prompt should fit in around 5 kB.