Views
No views yet
Artificial Hippocampus Networks (AHNs) transform lossless memory into fixed-size compressed representations for long-context modeling. Lossless memory (e.g., attention’s key-value (KV) cache) stores exact input information but grows with sequence length, making it inefficient for long sequences. In contrast, compressed memory (e.g., RNNs’ hidden state) maintains a constant size and offers fixed computational costs per input token, but this comes at the cost of information loss. To harness the benefits of both memory types, AHNs continually convert lossless memory outside the sliding attention window into compressed form. AHNs can be instantiated with any RNN-like architectures. The model then integrates both memory types to make predictions across long contexts.
| base model | AHN module | #params | checkpoint (AHN only) |
|---|---|---|---|
| Qwen2.5-3B-Instruct | Mamba2 | 11.9M | 🤗model |
| Qwen2.5-3B-Instruct | DeltaNet | 11.8M | 🤗model |
| Qwen2.5-3B-Instruct | GatedDeltaNet | 13.0M | 🤗model |
| Qwen2.5-7B-Instruct | Mamba2 | 18.6M | 🤗model |
| Qwen2.5-7B-Instruct | DeltaNet | 18.5M | 🤗model |
| Qwen2.5-7B-Instruct | GatedDeltaNet | 21.3M | 🤗model |
| Qwen2.5-14B-Instruct | Mamba2 | 51.4M | 🤗model |
| Qwen2.5-14B-Instruct | DeltaNet | 51.1M | 🤗model |
| Qwen2.5-14B-Instruct | GatedDeltaNet | 61.0M | 🤗model |
1@article{fang2025artificial,
2 title={Artificial hippocampus networks for efficient long-context modeling},
3 author={Fang, Yunhao and Yu, Weihao and Zhong, Shu and Ye, Qinghao and Xiong, Xuehan and Wei, Lai},
4 journal={arXiv preprint arXiv:2510.07318},
5 year={2025}
6}