Views
No views yet
[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.Critical Constraints:
- No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.
- No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.
Echo-DSRN, also know as echo) is a novel architecture specifically designed to be a viable alternative for low-resource tasks that are currently being inefficiently handled by the excessive scale of Large Language Models (LLMs) 🌱| Property | Value |
|---|---|
| Model Type | echo_dsrn |
| Layers | 8 |
| Hidden Dim | 512 |
| Attention Heads | 4 |
| MLP Ratio | 8.0 |
| Vocab Size | 32011 |
| Hybrid Attention | True |
| RMSNorm | True |
| Component | Parameters | % of Total |
|---|---|---|
| Total | 114.69M (114,687,488) | 100% |
| Embeddings | 16.39M | 14.29% |
| DSRN Blocks (Aggregate) | 81.91M | 71.42% |
| LM Head | 16.39M | 14.29% |
| Sub-Component | Parameters | Description |
|---|---|---|
| MLP (Feed-Forward) | 4.20M | Upscaled hidden layers |
| DSRN Slow State | 3.15M | Constant-time memory gates |
| GRU Fast State | 1.58M | Recurrent fast path |
| Surprise Gating | 264,192 | Dynamic focus mechanism |
| Normalization | 1,024 | LayerNorm / RMSNorm |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_easy | 1 | none | 0 | acc | ↑ | 0.4289 | ± | 0.0102 |
| none | 0 | acc_norm | ↑ | 0.4078 | ± | 0.0101 | ||
| boolq | 2 | none | 0 | acc | ↑ | 0.4064 | ± | 0.0086 |
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2692 | ± | 0.0044 |
| none | 0 | acc_norm | ↑ | 0.2757 | ± | 0.0045 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5789 | ± | 0.0115 |
| none | 0 | acc_norm | ↑ | 0.5637 | ± | 0.0116 | ||
| sciq | 1 | none | 0 | acc | ↑ | 0.5980 | ± | 0.0155 |
| none | 0 | acc_norm | ↑ | 0.5610 | ± | 0.0157 | ||
| winogrande | 1 | none | 0 | acc | ↑ | 0.4957 | ± | 0.0141 |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_easy | 1 | none | 5 | acc | ↑ | 0.3910 | ± | 0.0100 |
| none | 5 | acc_norm | ↑ | 0.3645 | ± | 0.0099 | ||
| boolq | 2 | none | 5 | acc | ↑ | 0.5098 | ± | 0.0087 |
| hellaswag | 1 | none | 5 | acc | ↑ | 0.2717 | ± | 0.0044 |
| none | 5 | acc_norm | ↑ | 0.2717 | ± | 0.0044 | ||
| piqa | 1 | none | 5 | acc | ↑ | 0.5686 | ± | 0.0116 |
| none | 5 | acc_norm | ↑ | 0.5642 | ± | 0.0116 | ||
| sciq | 1 | none | 5 | acc | ↑ | 0.5570 | ± | 0.0157 |
| none | 5 | acc_norm | ↑ | 0.4970 | ± | 0.0158 | ||
| winogrande | 1 | none | 5 | acc | ↑ | 0.4933 | ± | 0.0141 |
@misc{Massimo Roberto Scamarcia, title={Echo-DSRN-114M: Surprise-Gated Dual-State Recurrent Architecture for Efficient Language Modeling and Classification}, DOI={10.5281/zenodo.19848279}, publisher={Zenodo}, author={Massimo Roberto Scamarcia} }