LaCT (Large-Chunk Test-Time Training, SwiGLU MLP fast weights) 100M (rank 64) — Low-rank Fast-Weight Ablation

Training

Eval results

• FineWeb-Edu val PPL: 21.22
• MQAR (multi-query associative recall):
  • K=4: 0.295
  • K=16: 0.552
  • K=64: 0.695
  • K=256: 0.851
• LAMBADA acc: 0.189
• HellaSwag acc_norm: 0.293
• ARC-Easy acc_norm: 0.396
• PIQA acc_norm: 0.595
• WinoGrande acc: 0.511

Notes

• This is one of 16 cells; the other rank/arch combinations are uploaded under the same HF org (nlproj) with repo names matching the local dump folder, e.g. nlproj/lact_100M_{r8|r32|r64|rfull}_bs256_lr3e-4_steps5000.
• Key finding of the ablation: at this scale, low rank often matches or beats full rank on downstream tasks (LoRA-style "adaptation is intrinsically low-rank" hypothesis). GatedDeltaNet is the exception — its rfull is the strongest in the whole sweep on PPL / LAMBADA / HellaSwag / ARC-Easy.