You can deploy this model and start earning money today!

bayesiansapien/
allostasis-1B-paper1

Name: allostasis-1B-paper1
Availability: InStock

pytorch

adaptive-width

elastic-ffn

input-conditional-compute

neural-elasticity

allostasis

apache-2.0

Views

No views yet

Allostasis-1B (paper #1 reference run)

Reference checkpoint for the paper "Allostasis: A Foundational Principle of Anchored, Breathing Neural Representations".

Headline numbers (held-out, 289 KB UTF-8 mixed text)

Metric	Value
Bits per byte	0.9797
Implied perplexity	15.75
Total params	1052.4M
Tokens evaluated	71,306
Bytes evaluated	289,519
Mean summed active FFN dim per token	63,434 (across 16 layers)
Analytical FFN FLOPs / token	5.197e+08
FFN FLOPs reduction vs fixed-max	61.3%

Cross-architecture position on the same held-out file

Model	BPB	Training tokens
Qwen1.5-MoE-A2.7B (active 2.7B)	0.5995	—
Llama-3.2-1B	0.6879	9T
Mamba2-1.3B	0.7121	—
SmolLM2-360M	0.7597	4T
Allostasis-1B (this checkpoint)	0.9797	491M unique
Pythia-160M	1.0044	300B
Gemma-3-1B	1.0440	—

Beats Pythia-160M (trained on 600x more tokens) and Gemma-3-1B; within striking distance of SmolLM2-135M at 4000x less training data.

Architecture

d_model = 2048, n_layers = 16, n_heads = 16, seq_len = 512
ffn_anchor_dim = 2048 (always active)
ffn_expansion_max_dim = 8192, rank ladder = [0, 2048, 5120, 8192]
Gumbel-Softmax gate, tau anneal 1.5 -> 0.3
Soft-warmup (epoch 0) + capacity-warmup (epoch 1) + entropy bonus
Tokenizer: GPT-2 (vocab 50,257)

See config_1B_final.yaml for the full config.

Training

491M unique tokens (TinyStories 22M + CodeAlpaca 1.6M + Wikitext-103 16M + FineWeb-Edu 351M + OpenWebText 100M)
Single A40 48 GB, 19 h 53 min wall-clock at ~7000 tok/s
122,100 steps, 3 epochs, LR cosine 1.5e-4 -> 1.5e-5
See metrics.csv and train.log

Per-token capacity-NLL diagnostic (paper section 9.4)

Spearman rho(per-token NLL, summed-active-FFN-dim) = +0.349 over 71,306 held-out tokens. The decile structure ramps from ~45,000 summed dim at the most-predictable tokens to ~72,000 at the hardest. Compared to the 300M ladder (rho = +0.193), a 1.81x amplification across one OOM more training data — direct empirical evidence for the loss-margin -> intrinsic-complexity transition discussed in section 10.1.

See per_token.json for the full decile structure.

Limitations

Single seed at 1B scale (variance estimate from the 300M ladder's seed-42/seed-43 replication is 0.0022 BPB).
Reference PyTorch implementation pays the fixed-max wall-clock by zero-masking; the analytical FFN FLOPs envelope reflects what a ragged-tensor kernel would realise. Engineering deferred to paper #2.
Trained on 4-5 orders of magnitude less data than SmolLM2 (11T) or Qwen2.5 (18T).

Loading

python

import torch, yaml
# 1) Get the model code:
#    git clone https://github.com/bayesiansapien/neural-elasticity
#    cd neural-elasticity && git checkout experiments
#    cd paper-1/code/E3_lm_oracle
# 2) Build and load:
from model import build_model_from_config
cfg = yaml.safe_load(open("config_1B_final.yaml"))
model = build_model_from_config(cfg)
state = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)
model.load_state_dict(state["state_dict"])
model.eval()

Citation

If you use this checkpoint, please cite the paper.

bayesiansapien/allostasis-1B-paper1