Views
No views yet
x → RMSNorm → [EMA ‖ DeltaRule] → concat → Linear(2D→D) → Focus → + x
→ RMSNorm → FFN (SwiGLU) → + x| Component | Role |
|---|---|
| EMA | Local smoothing (learnable per-channel decay, O(d) params) |
| DeltaRule | Associative memory — token retrieval via fast weights |
| Focus | Mamba selective SSM — temporal flow, long-range context |
| SwiGLU FFN | Channel mixing |
| Params | 105.9M |
| Layers | 8 |
| d_model | 512 |
| d_state | 64 |
| Vocab | 151665 |
| Tokenizer | Qwen2.5 |