FlashLM v10 FSP

Key Results

What is FSP?

• Reference: "Beyond Multi-Token Prediction" (Mahajan et al., 2025)
• FSP adds only 65K parameters (a small projection adapter sharing the lm_head)
• Training overhead: ~6% compute

Architecture

Embedding(4096, 256) + RoPE
  └── Block ×4
      ├── RMSNorm → CausalSelfAttention(8 heads, d=256) → Residual
      └── RMSNorm → SwiGLU(d_ff=512) → Residual
  └── RMSNorm → lm_head (weight-tied)
  └── FSP: Linear(256→256) → shared lm_head → sigmoid → BoW prediction

Generation Samples

Once upon a time, there was a little girl named Sue. Sue was very sad because she could not find her toy. One day, she found a big box near her house.

A cat sat on the bed. The cat saw the cat and wanted to help. The cat jumped on the bench and began to walk in the sky. The cat started to feel better and tried...

The little girl was scared and she wanted to see what was inside. She thought about what she had been in the door.

Limitations

• Stories are grammatically correct but not logically coherent across sentences
• Cross-sentence causal reasoning is still weak ("the cat walked in the sky")
• Characters, dialogue, and sentence structure work well; causal chains do not
• This is a research model demonstrating FSP training, not a production story generator

Usage

Training Details

Citation