Views
No views yet
[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.Critical Constraints:
- No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.
- No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.
Echo-DSRN, also know as echo) is a novel architecture specifically designed to be a viable alternative for low-resource tasks that are currently being inefficiently handled by the excessive scale of Large Language Models (LLMs) 🌱| Property | Value |
|---|---|
| Model Type | echo_dsrn |
| Layers | 8 |
| Hidden Dim | 512 |
| Attention Heads | 4 |
| MLP Ratio | 8.0 |
| Vocab Size | 32011 |
| Hybrid Attention | True |
| RMSNorm | True |
| Component | Parameters | % of Total |
|---|---|---|
| Total | 114.69M (114,687,488) | 100% |
| Embeddings | 16.39M | 14.29% |
| DSRN Blocks (Aggregate) | 81.91M | 71.42% |
| LM Head | 16.39M | 14.29% |
| Sub-Component | Parameters | Description |
|---|---|---|
| MLP (Feed-Forward) | 4.20M | Upscaled hidden layers |
| DSRN Slow State | 3.15M | Constant-time memory gates |
| GRU Fast State | 1.58M | Recurrent fast path |
| Surprise Gating | 264,192 | Dynamic focus mechanism |
| Normalization | 1,024 | LayerNorm / RMSNorm |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| piqa | 1 | none | 0 | acc | ↑ | 0.5789 | ± | 0.0115 |
| none | 0 | acc_norm | ↑ | 0.5718 | ± | 0.0115 | ||
| sciq | 1 | none | 0 | acc | ↑ | 0.5830 | ± | 0.0156 |
| none | 0 | acc_norm | ↑ | 0.5250 | ± | 0.0158 |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| piqa | 1 | none | 5 | acc | ↑ | 0.5773 | ± | 0.0115 |
| none | 5 | acc_norm | ↑ | 0.5729 | ± | 0.0115 | ||
| sciq | 1 | none | 5 | acc | ↑ | 0.5700 | ± | 0.0157 |
| none | 5 | acc_norm | ↑ | 0.5140 | ± | 0.0158 |
@misc{Massimo Roberto Scamarcia, title={Echo-DSRN-114M: Surprise-Gated Dual-State Recurrent Architecture for Efficient Language Modeling and Classification}, DOI={10.5281/zenodo.19848279}, publisher={Zenodo}, author={Massimo Roberto Scamarcia} }