Input Tokens (B, T)
    ↓
Embedding [vocab_size × d_model]
    ↓
    +--→ e (preserved for LTI injection — full gradient flow)
    ↓
┌──────────────────────────────────────────────────────────────────┐
│  Recurrent Block (n_loops × shared graph weights)                 │
│  ├── Loop Index Embedding (sinusoidal)                            │
│  ├── HelixGraph: Randomly wired heterogeneous DAG                 │
│  │   ├── Neural columns with vertical + lateral wiring            │
│  │   ├── LinearAttnNode     : O(n) causal linear attention       │
│  │   ├── FullAttnNode       : Causal softmax (periodic, hybrid)   │
│  │   ├── SwiGLUNode         : Modern gated FFN (silu·gate × up)   │
│  │   ├── Mamba2Node         : Mamba-2 SSD (long-range, optional)  │
│  │   ├── TitansMemoryNode   : Surprise-gated persistent memory   │
│  │   ├── GateNode           : Learned softmax multi-input merge  │
│  │   ├── CCA gates          : Curriculum Component Activation    │
│  │   └── Merge RMSNorm      : Normalization after multi-predecessor merges │
│  ├── LTI Injection          : Stable recurrent state mixing       │
│  │   h_new = A·h + B·e + graph_output   (A < 1, learnable decay) │
│  └── ACT Halting            : Dynamic per-token depth allocation │
└──────────────────────────────────────────────────────────────────┘
    ↓
RMSNorm
    ↓
TiedLMHead (gradient-buffered weight tying with embedding)
    ↓
Logits / Loss

Use Cases

Hyperpersonalization: Train a small model (from tens of milions of parameters to ~1B parameters) from cold start or from an early - stage partially pre-trained model checkpoint on a personalized corpus, one enriched in it's representation of content relevant to your domain knowledge, before you full fine-tune it on your own data. The model becomes an expert in the one thing generic frontier models don't know anything about: you, your domain knowledge, style, notes, emails, work patterns, and personal voice. A personalized small language model drafts responses to your requests with full awareness of the entire data you encoded in its weights, including the secondary details, tertiary details, and edge cases that vector DB RAG systems usually miss because they are semantically dissimilar to the prompt. This oversight leaves the loose ends that hang you if you don't catch them. Evolve to a better adapted approach.
On-device AI: Efficient inference on CPU/GPU for desktops, laptops, tablets, and mobile. Strong quality per parameter, with optional fully-local (log-free) operation for sensitive use cases.
Economical and ecologically sound AI: The brute force approach that previous generation [used to be] frontier labs use is unsustainable: It burn cash at a rate that is economically unsustainable, causes electricity prices to surge for consumers competing with them for their share of electric grid's limited capacity, and compromises water supplies. Take an exponential cut out of the problem with more efficient AI: A model architecture with better parameter efficinecy, one that can use hybrid or even flat out linear attention only, and that can saturate the model's weights / grock on as little as one epoch.

Key Design Choices

Component	What it does	Why it matters
Neural Columns & Heterogeneous Nodes	Each column holds diverse node types (attention variants, SwiGLU, Mamba-2, gating, optional Titans memory) instead of identical transformer blocks.	Different information pathways for different computations, like biological cortical columns.
Recurrent Depth (LTI + ACT)	The same graph weights are looped `n_loops` times. LTI injection keeps the recurrent state stable. ACT halting dynamically allocates compute per token.	Iterative refinement without parameter growth; easy tokens use 1 loop, hard reasoning uses more.
Hybrid Attention	Linear attention (O(n) complexity) in most columns, with periodic full-attention columns for exact retrieval.	Long-context efficiency without losing precise copy/lookup capability.
Mamba-2 SSD (optional)	State Space Duality with chunked parallel scan. Auto-activates when `ssm_d_state >= 64`.	Handles very long-range dependencies efficiently on CPU, CUDA, or MPS.
Titans Neural Memory (optional)	Persistent surprise-gated memory via outer-product updates (first column only by default).	Test-time memory that can retain patterns across long documents without growing KV cache.
CCA (Curriculum Component Activation)	Attention nodes start gated, gradually open over warmup steps via learned sigmoid gates.	Prevents random attention from drowning FFN signal at initialization. ~30% PPL improvement observed.
TiedLMHead with Gradient Buffer	LM head shares embedding weight. A learned linear buffer (init as identity) routes part of the gradient to prevent ~3× embedding gradient overload.	Halves parameter count (embedding table not duplicated). Safe weight tying without destabilizing training.
Modern Primitives	RoPE, SwiGLU, RMSNorm.	Proven SOTA components for convergence and generation quality.

HelixLM and upstream aspects from OpenMythos

OpenMythos published a recurrent depth with LTI stability and ACT halting. HelixLM takes that insight and makes it work inside a heterogeneous graph that mimics neural column and random topology connectivity found in biological brains.

Capability	OpenMythos	HelixLM
Recurrent depth	✅ Same block looped	✅ Same block looped
LTI stability	✅ Spectral radius < 1	✅ Spectral radius < 1, configurable `init_A` (default `1/e`)
ACT halting	✅ Dynamic per-token depth	✅ Dynamic per-token depth
Architecture inside loop	Standard transformer block	Heterogeneous random graph
Attention	Standard full attention only	Linear + full hybrid
Node types	Single (transformer block)	7+ active types (attention variants, FFN, SSM, gate, neural memory)
Positional encoding	Standard learned	RoPE
Activation	GELU	SwiGLU
Normalization	LayerNorm	RMSNorm
Open source	✅ Yes	✅ Yes
HF integration	❌ No	✅ Full PreTrainedModel, AutoModelForCausalLM
Weight tying	Standard	Gradient-buffered TiedLMHead

HelixLM's Cerebros legacy

Cerebros showed that biological random hyperdense vertical and lateral topology of Dense layers could outperform rigid layer stacks. It generated text without attention on small data, but required elaborate integration and clashed with standard model-structure paradigms. HelixLM smoothly integrates that topological insight into a modern, HF-compatible LLM backbone.

Training data

https://huggingface.co/datasets/david-thrower/HelixLM-medium-1500.0Mt-2988750pt-20260528, 1.5B tokens from:
- HuggingFaceFW/fineweb-edu: 99%
- open-web-math/open-web-math 1%

Metrics:

Perplexity: 63.46

Weight Watcher Data:


>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1", trust_remote_code=True, dtype="auto", force_download=True)
config.json: 2.15kB [00:00, 3.09MB/s]
config.json: 2.15kB [00:00, 1.32MB/s]
config.py: 15.6kB [00:00, 10.6MB/s]
config.json: 2.15kB [00:00, 4.14MB/s]
hf_model.py: 18.4kB [00:00, 45.2MB/s]
model.safetensors: 100%|█████████████████████| 160M/160M [00:10<00:00, 14.8MB/s]
[HelixLM] Global RNG seed set to 42 (set HelixConfig(seed=None) to disable auto-seeding)
Loading weights: 100%|███████████████████████| 78/78 [00:00<00:00, 11643.80it/s]
generation_config.json: 100%|███████████████████| 355/355 [00:00<00:00, 884kB/s]
>>> layers = [8, 9, 10, 11, 12, 13, 18, 19, 20, 24, 28, 29, 30, 31, 32, 33, 38,\
 39, 40, 44, 48, 49, 50, 51, 52, 53, 58, 59, 60, 64, 67, 68, 69, 70]
>>> import weightwatcher as ww
>>> watcher = ww.WeightWatcher()
>>> details = watcher.analyze(model=model, layers=layers)
>>> summary = watcher.get_summary(details)
>>> summary
{'log_norm': np.float64(2.9583618357476125), 'alpha': np.float64(3.5017526482629573), 'alpha_weighted': np.float64(3.784600552260819), 'log_alpha_norm': np.float64(4.507185545059817), 'log_spectral_norm': np.float64(1.570243957671261), 'stable_rank': np.float64(41.33611414349218)}
>>> details
    layer_id    name         D  ...  weak_rank_loss        xmax       xmin
0          8  Linear  0.133501  ...               0    0.819514   0.586496
1          9  Linear  0.130986  ...               1    0.806610   0.591386
2         10  Linear  0.124447  ...               1    0.812267   0.582928
3         11  Linear  0.123478  ...               1    0.801489   0.669532
4         12  Linear  0.174834  ...               1    0.091557   0.054470
5         13  Linear  0.122635  ...               0    0.101244   0.044695
6         18  Linear  0.032087  ...               0  147.171070  27.923805
7         19  Linear  0.042542  ...               0   94.793864  27.706529
8         20  Linear  0.068676  ...               0  104.940972  31.728804
9         24  Linear  0.081954  ...               1   48.504587   0.067334
10        28  Linear  0.051617  ...               3  124.385294   0.035137
11        29  Linear  0.042766  ...               4  181.505814   0.030283
12        30  Linear  0.082598  ...               5   35.948817   0.095789
13        31  Linear  0.046608  ...               4   51.369828  17.504687
14        32  Linear  0.091659  ...               0   58.859000   1.058015
15        33  Linear  0.110888  ...               1   64.008994   1.678953
16        38  Linear  0.068318  ...               0   32.099117   3.374599
17        39  Linear  0.069430  ...               0   30.437727   3.097338
18        40  Linear  0.047503  ...               0  120.902677   0.530989
19        44  Linear  0.076068  ...               1   22.695975   0.091167
20        48  Linear  0.075791  ...               4  458.548402   3.472195
21        49  Linear  0.066970  ...               3  252.430063   1.100946
22        50  Linear  0.079772  ...               7   67.926923   0.108073
23        51  Linear  0.064862  ...               2   77.686899   5.453615
24        52  Linear  0.073012  ...               0   68.709129   0.829720
25        53  Linear  0.068787  ...               0   61.817121   0.694748
26        58  Linear  0.052428  ...               0  352.342393  59.507195
27        59  Linear  0.066766  ...               0   60.873335  18.667454
28        60  Linear  0.034649  ...               0  181.075791  24.358505
29        64  Linear  0.085383  ...               1  146.261497  53.585340
30        67  Linear  0.047679  ...               0  208.907095   0.918829
31        68  Linear  0.067708  ...               0   89.022724   1.637560
32        69  Linear  0.055705  ...               0  269.097498  10.898044
33        70  Linear  0.073438  ...               0  277.911039  44.351509

[34 rows x 32 columns]

Summary Metrics Analysis
                                                                 
Metric             Value  Interpretation                      
───────────────────────────────────────────────────────────── 
log_norm           2.96   Overall weight magnitude - moderate 
alpha              3.50   Power-law exponent (unweighted avg) 
alpha_weighted     3.78   Weighted by layer importance        
log_spectral_norm  1.57   Log of largest singular value       
stable_rank        41.34  Effective rank of weight matrices

Key Weight Watcher Findings:

Alpha Value Assessment (α ≈ 3.5-3.8)

Good range: Well-trained transformers typically show α ∈ [2, 4]
Value: 3.50 (unweighted) / 3.78 (weighted) is in the healthy range
Interpretation: Indicates reasonable training - not underfitting (α→2) or overfitting (α→6+)

Stable Rank (41.34)

This is relatively high for a small model.
Suggests weight matrices are utilizing their full capacity

Chinchilla +++ Optimal Training

Base Chinchilla Formula: 20 tokens per parameter

Chinchilla optimal = 40.1M × 20 = 802M tokens

We trained on 1.5B tokens, so at first pass:

1.5B / 802M = 1.87× or 87% over Chinchilla

However, let's adjust for the disproportionate embedding table (As I am a 41M param model, I have a disproportionate percentage of my weights in my embeddign table, compared to a trypical 150M - 1T parameter model, so lets't adjust for that) ...

Embedding Table Adjustment

GPT-2 Tokenizer specs:

Vocab size: ~50,257 tokens
Embedding dim: 512

Parameter breakdown for 40.1M model:

Component	Calculation	Parameters
Total Parameters	-	40.1M
Token embeddings	50,257 × 512	25.7M
Position embeddings	~1024 × 512	~0.5M
Attention and FFN layers	Remainder	~13.9M

Embeddings: ~26.2M (65% of params)
Active/"thinking" weights: ~13.9M (35% of params)

Adjusted Chinchilla Calculation

Chinchilla really applies to the compute relevant trainable weights of the network, i.e., the parameters that actually transform representations (not the lookup tables):

Effective active parameters: ~13.9M Chinchilla optimal for active params: 13.9M × 20 = 278M tokens My actual training corpus: 1.5B tokens

Metric	Value
Tokens per active param	107.9
vs. Chinchilla optimal	5.39 × over chinchilla optimal

TLDR, we've grocked to full saturation on 1 epoch, on 5.39 X chinchilla optimal with only the earlyest signs of destabilization, and rank collapse not yet occuring from over-training.

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

Try ML Intern: https://smolagents-ml-intern.hf.space
Source code: https://github.com/huggingface/ml-intern

Usage

Use a pipeline

python

from transformers import pipeline
pipe = pipeline("text-generation", model='david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1', trust_remote_code=True)
x = pipe('In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are.', max_new_tokens=100, temperature=0.6, top_k=40, top_p=0.95, repetition_penalty=1.2)
print(x)
# -> 'In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are. We will first find a new catalog of the Andromeda Galaxy, which is also known as the companion star. The Milky Way is the most massive disk in the Universe, with several million light sources to search for ... that can be found by the Crab Nebula: ...
# Fluent and on topic text, albeit not [quite] factually on - point.

Use the AutoModelForCausalLM object directly

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.

License: Modified Apache 2.0

This project is open-source under a modified Apache 2.0 license. See Cerebros-License for full terms.