GSN - experiment 2
Base model trained on 38B tokens of python code from merged and denoised TheStack V1 and OpenCoder RCC datasets.
Model was pretrained on a single rtx 4090 GPU for around 3 days.
This a toy model with very small context window of 2048 tokens, but it is already usable.
