670 million parameter language model trained on the lunaris-ultrafineweb-20b-tokenized pretokenized dataset, using the custom BPE tokenizer. Trained for around ~60-70 hours on A100 and around ~3-4B tokens seen. Model written in Pytorch with Pytorch Lightning and Wandb for Logging. Trained on GCP Vertex AI CustomJob. For Software Engineering year 12 major project.
