A modified GPT-2 model with ScaledSinusoidal position embeddings, no biases, embedding layernorm, and one shared MLP layer, with 94 million non-embedding params, that beats most similarly sized and slightly larger models (GPT-2-124m, Pythia-70/160m, Cerebras-111m) on the
Open LLM Leaderboard suite of benchmarks. All while only being trained on 8 billion tokens of text from
SlimPajama.
