FWKV (Feed-forward Weighted Key Value, or Floored Weighted Key Value) is a novel
efficient language model architecture proposed on May 13, 2026 by Me.
state = W * state + k * v
Design Philosophy
FWKV is built on two simple ideas:
FFNN as the primary compute block — fast, parallelizable, hardware-friendly
floor(W·K·V) as the sole context mechanism — a multiplicative recurrent state
with a learned decay W floored above zero, so the model can never fully forget
Sample
Inference speed (CUDA -- T4)
image
Training
Factorised tied weights, chunked CE, bf16 training.
Best val perplexity: 143.62
