Comma v0.1-2T is a 7 billion parameter language model trained on 2 trillion tokens from the Comma v0.1 dataset, comprising of openly licensed text from the Common Pile. Comma v0.1-2T is a "base model" that can be used a the starting point for finetuning and post-training. It performs comparably to budget-matched models (7 billion parameters, 2 trillion tokens) trained on unlicensed data.

Comma v0.1-2T is a decoder-only transformer that uses the same architecture as Llama 3. Training was done in two stages: first on 1.93 trillion tokens with a cosine learning rate schedule, and second a "cool-down" training phase on 75.5 billion tokens from high-quality sources. The final model is the average of 10 checkpoints during this cool-down phase. Both training phases use a batch size of 8.3 million tokens per step. Training was performed using lingua on 512 AMD MI300A GPUs. Hyperparameters can be found in our lingua config file.

Comma v0.1-2T was trained only on English-language data and code from the 15 programming languages covered by the stack-edu classifiers. It will likely have poor performance on other languages or programming languages. While we aimed to train solely on openly licensed data, license laundering and inaccurate metadata can result in erroneous license information in the Common Pile (for further discussion of this limitation, please see our paper). Consequently, we cannot make a guarantee that Comma v0.1-2T was trained exclusively on openly licensed text. When preparing Comma v0.1's pre-training data, we made use of the Toxicity tagger from Dolma to attempt to remove problematic content. However, Comma v0.1-2T may nevertheless reflect social biases present in its training data. Finally, please note that Comma v0.1-2T is a base model that has not undergone any form of "alignment" and therefore has no guardrails that limit what it may generate.