🧠 Athena Modern Hopfield Network SLM
Athena MHN SLM is an experimental small language model built using a Modern Hopfield Network-inspired attention mechanism combined with a recurrent Transformer-style architecture.
This project explores how associative memory (Hopfield networks) can be used to improve reasoning and pattern retrieval in language models.
⚙️ Architecture Overview
The model is a hybrid of:
- 🔷 Modern Hopfield Network-style attention
- 🔷 Transformer embedding + positional encoding
- 🔷 Recurrent layer execution (multi-loop depth)
- 🔷 Adaptive computation halting
Key Components
- Embedding size: 512
- Layers: 4 MHN blocks
- Attention type: Hopfield-style energy-based retrieval (softmax attention form)
- Recurrent loops per layer: up to 3
- Context length: 1024 tokens
- Adaptive halting mechanism for dynamic computation depth
🧠 Modern Hopfield Mechanism
Each layer performs associative memory retrieval:
- Queries represent current token states
- Keys/Values are derived from the same hidden states
- Attention is computed as energy-based similarity
- Output is retrieved memory patterns from the state space
This makes the model behave like a continuous associative memory system.
📚 Training Setup
- Dataset:
tatsu-lab/alpaca
- Training samples: 15,000 instruction-response pairs
- Format:
- Instruction
- Input (if any)
- Output response
🔄 Recurrent Depth Mechanism
Instead of stacking many layers, each MHN block is executed multiple times:
- Improves reasoning depth
- Enables iterative refinement of representations
- Reduces need for very deep architectures
⏱️ Adaptive Halting
The model includes a halting head that dynamically stops computation when:
- Confidence threshold is reached (
> 0.96)
- Reduces unnecessary computation
- Improves efficiency during inference
🧪 Status
⚠️ Experimental Research Model
- Architecture is actively evolving
- Not yet optimized for production use
- Intended for research and prototyping only
🚀 Intended Use Cases
- Research on Modern Hopfield Networks
- Associative memory-based language modeling
- Recurrent depth transformer experiments
- Instruction-following prototype systems
📦 Version History
v2
- Introduced True Modern Hopfield-style attention block
- Improved associative memory behavior
- Stabilized recurrent loop architecture
- Added adaptive halting mechanism
🔗 Model Repository
license: apache-2.0
