You can deploy this model and start earning money today!

ethicalabs/
Echo-DSRN-114M-Base

Name: Echo-DSRN-114M-Base
Availability: InStock

text-generation-inference

conversational

custom_code

HuggingFaceFW/fineweb-edu

HuggingFaceH4/ultrachat_200k

apache-2.0

Views

No views yet

Model Card for ethicalabs/Echo-DSRN-114M-Base

[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.

Critical Constraints:

No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.

No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.

The Echo-DSRN(N) (Dual State Recurrent Neural Network, short name: Echo-DSRN, also know as echo) is a novel architecture specifically designed to be a viable alternative for low-resource tasks that are currently being inefficiently handled by the excessive scale of Large Language Models (LLMs) 🌱

⚠️ Important Notice

This is a research prototype and demo model.

Not production-ready
Will hallucinate and give incorrect answers
Do not use for any real-world decisions
Intended for architecture experimentation only

What Works

Text generation is fluent
Memory usage is constant O(1)
Runs on CPUs, NPUs, GPUs (Tested on AMD's ROCm and Apple's MPS)

What Doesn't Work

Factual accuracy
Instruction following
Common sense reasoning

🏗️ Architecture Details

Property	Value
Model Type	echo_dsrn
Layers	8
Hidden Dim	512
Attention Heads	4
MLP Ratio	8.0
Vocab Size	32011
Hybrid Attention	True
RMSNorm	True

📊 Parameter Breakdown

Component	Parameters	% of Total
Total	114.69M (114,687,488)	100%
Embeddings	16.39M	14.29%
DSRN Blocks (Aggregate)	81.91M	71.42%
LM Head	16.39M	14.29%

🧩 Internal Block Structure (Per Layer)

Sub-Component	Parameters	Description
MLP (Feed-Forward)	4.20M	Upscaled hidden layers
DSRN Slow State	3.15M	Constant-time memory gates
GRU Fast State	1.58M	Recurrent fast path
Surprise Gating	264,192	Dynamic focus mechanism
Normalization	1,024	LayerNorm / RMSNorm

Pre-Training

Truncated Backpropagation Through Time (TBPTT) on Fineweb-EDU (10BT)

1 epoch on a single AMD Instinct MI300X 192 GB

Continued Pre-Training (SFTTrainer)

1 epoch on a single AMD Radeon AI PRO R9700 32 GB

Evaluation

uv run lm_eval --model hf   --model_args pretrained=ethicalabs/Echo-DSRN-114M-Base,trust_remote_code=True,device_map="auto"   --tasks sciq,piqa --output_path ./results_Echo-DSRN-114M-Base --batch_size 16 --num_fewshot 5 && uv run lm_eval --model hf   --model_args pretrained=ethicalabs/Echo-DSRN-114M-Base,trust_remote_code=True,device_map="auto"   --tasks sciq,piqa --output_path ./results_Echo-DSRN-114M-Base --batch_size 16 --num_fewshot 5

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
piqa	1	none	5	acc	↑	0.6055	±	0.0114
		none	5	acc_norm	↑	0.6012	±	0.0114
sciq	1	none	5	acc	↑	0.6200	±	0.0154
		none	5	acc_norm	↑	0.5480	±	0.0157

uv run lm_eval --model hf   --model_args pretrained=ethicalabs/Echo-DSRN-114M-Base,trust_remote_code=True,device_map="auto"   --tasks sciq,piqa --output_path ./results_Echo-DSRN-114M-Base --batch_size 16 --num_fewshot 5 && uv run lm_eval --model hf   --model_args pretrained=ethicalabs/Echo-DSRN-114M-Base,trust_remote_code=True,device_map="auto"   --tasks sciq,piqa --output_path ./results_Echo-DSRN-114M-Base --batch_size 16 --num_fewshot 10

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
piqa	1	none	10	acc	↑	0.6083	±	0.0114
		none	10	acc_norm	↑	0.6066	±	0.0114
sciq	1	none	10	acc	↑	0.6150	±	0.0154
		none	10	acc_norm	↑	0.5600	±	0.0157

Gradio App - Next Word Prediction

Echo-DSRN-114M-Base - Next Word Prediction