Can Higher Order Attention Help Small Language Models?
BabyLM 2026 submission · Joseph Chan
We investigate whether higher-order interactions in attention can improve small language models trained under severe data constraints. We introduce and evaluate a force layer inside a Transformer, comparing against standard Transformer baselines on the BabyLM 10M word setting.
A force layer is an attention layer where the attention score has a quadratic correction term to the first order dot product attention score prior to applying softmax.
This model has 6 prefix attention layers, one force layer, followed by 5 suffix attention layers.
The model hyperparameters are:
8 heads
64 head dimension
512 hidden dimension
4 * 64 * 16 = 4096 per head second order parameters
12 layers
seed 3407
46.21M total parameters
BPE 16k tokenizer
16,384 vocab
10 epochs of training on 10M BabyLM Strict-Small dataset (stopped early with 5% of last epoch buffer)
