HelixLM is a novel model architecture that out-punches a weight class above its own by combining:
Our novel innovation in model topology https://github.com/david-thrower/cerebros-core-algorithm-alpha which emulates the neuroscience concept of neural columns and modularity. Modularity is a concept where local clusters of densely linked neurons specialize in various aspects of a problem being solved, while the sparse connectivity between neural columns optimizes efficient communication between them and fusion of the fragments of the problem to one cohesive solution.
Linking heterogenous linear and optionally full attention layers in this connectivity pattern.
Hyperpersonalization: Train a small model (from tens of milions of parameters to ~1B parameters) from cold start or from an early - stage partially pre-trained model checkpoint on a personalized corpus, one enriched in it's representation of content relevant to your domain knowledge, before you full fine-tune it on your own data. The model becomes an expert in the one thing generic frontier models don't know anything about: you, your domain knowledge, style, notes, emails, work patterns, and personal voice. A personalized small language model drafts responses to your requests with full awareness of the entire data you encoded in its weights, including the secondary details, tertiary details, and edge cases that vector DB RAG systems usually miss because they are semantically dissimilar to the prompt. This oversight leaves the loose ends that hang you if you don't catch them. Evolve to a better adapted approach.
On-device AI: Efficient inference on CPU/GPU for desktops, laptops, tablets, and mobile. Strong quality per parameter, with optional fully-local (log-free) operation for sensitive use cases.
Economical and ecologically sound AI: The brute force approach that previous generation [used to be] frontier labs use is unsustainable: It burn cash at a rate that is economically unsustainable, causes electricity prices to surge for consumers competing with them for their share of electric grid's limited capacity, and compromises water supplies. Take an exponential cut out of the problem with more efficient AI: A model architecture with better parameter efficinecy, one that can use hybrid or even flat out linear attention only, and that can saturate the model's weights / grock on as little as one epoch.
Key Design Choices
Component
What it does
Why it matters
Neural Columns & Heterogeneous Nodes
Each column holds diverse node types (attention variants, SwiGLU, Mamba-2, gating, optional Titans memory) instead of identical transformer blocks.
Different information pathways for different computations, like biological cortical columns.
Recurrent Depth (LTI + ACT)
The same graph weights are looped n_loops times. LTI injection keeps the recurrent state stable. ACT halting dynamically allocates compute per token.
Iterative refinement without parameter growth; easy tokens use 1 loop, hard reasoning uses more.
Hybrid Attention
Linear attention (O(n) complexity) in most columns, with periodic full-attention columns for exact retrieval.
Long-context efficiency without losing precise copy/lookup capability.
Mamba-2 SSD (optional)
State Space Duality with chunked parallel scan. Auto-activates when ssm_d_state >= 64.
Handles very long-range dependencies efficiently on CPU, CUDA, or MPS.
Titans Neural Memory (optional)
Persistent surprise-gated memory via outer-product updates (first column only by default).
Test-time memory that can retain patterns across long documents without growing KV cache.
CCA (Curriculum Component Activation)
Attention nodes start gated, gradually open over warmup steps via learned sigmoid gates.
Prevents random attention from drowning FFN signal at initialization. ~30% PPL improvement observed.
TiedLMHead with Gradient Buffer
LM head shares embedding weight. A learned linear buffer (init as identity) routes part of the gradient to prevent ~3× embedding gradient overload.
Halves parameter count (embedding table not duplicated). Safe weight tying without destabilizing training.
Modern Primitives
RoPE, SwiGLU, RMSNorm.
Proven SOTA components for convergence and generation quality.
HelixLM and upstream aspects from OpenMythos
OpenMythos published a recurrent depth with LTI stability and ACT halting. HelixLM takes that insight and makes it work inside a heterogeneous graph that mimics neural column and random topology connectivity found in biological brains.
7+ active types (attention variants, FFN, SSM, gate, neural memory)
Positional encoding
Standard learned
RoPE
Activation
GELU
SwiGLU
Normalization
LayerNorm
RMSNorm
Open source
✅ Yes
✅ Yes
HF integration
❌ No
✅ Full PreTrainedModel, AutoModelForCausalLM
Weight tying
Standard
Gradient-buffered TiedLMHead
HelixLM's Cerebros legacy
Cerebros showed that biological random hyperdense vertical and lateral topology of Dense layers could outperform rigid layer stacks. It generated text without attention on small data, but required elaborate integration and clashed with standard model-structure paradigms. HelixLM smoothly integrates that topological insight into a modern, HF-compatible LLM backbone.
Borderline elevated; typical for early-stage small LMs
alpha_weighted
4.070
Weighted by layer size; slightly lower than unweighted
log_alpha_norm
4.641
—
log_spectral_norm
1.117
Low spectral amplification
stable_rank
45.624
Indicates significant rank differentiation across layers
Note:
WeightWatcher's "optimal" zone (α ≈ 2–4) was calibrated primarily on Linear attention stacks.
HelixLM's hybrid graph architecture (linear + full attention + Titans memory) exhibits a bimodal spectral distribution that diverges from this baseline.
Validation perplexity at epoch 3 is 8.5 points lower than the epoch 1 checkpoint that scored "optimal" on most layers, undertrained on a few, indicating that the current spectral profile correlates with improved task performance despite falling outside conventional heuristics.
Layer-Level Findings
The model exhibits a cooperative dichotomy between two spectral populations and between heuristics and actual model performance:
High singular value concentration (sv_max 2.9–15.6, sv_min ~10⁻⁵)
Low stable rank (8–40)
WeightWatcher flags these as "over-trained"; It's possible that in practice they may function as task-specific feature extractors with collapsed, efficient representations. See how the model architecture is designed to emulate the neuroscience concept of neural columns and modularity.
Diffuse / High-Entropy Layers (α ≈ 6–15)
Primarily 512×512 attention projections in early blocks (layers 8–11) and select middle layers (50, 51)
Low maximum singular values (sv_max ~0.9), broad ESD tails
Very high stable rank (128–133)
WeightWatcher flags these as "under-trained"; It is likely they may serve as implicit regularizers maintaining residual capacity and preventing over-commitment to spurious correlations
FFN Up-Projections (512×1280, 2.5× expansion)
Consistently healthy α ≈ 2.5–4.0 across all blocks
No warnings; stable spectral profiles
These are the most robust layers in the architecture
Output Projection (512×3584)
α = 7.23, stable rank = 133.5
Under-trained per WeightWatcher; this is the primary bottleneck for further perplexity improvement
Silent Rank Collapse (Layers 48, 49)
No WeightWatcher warning despite α ≈ 2.2
weak_rank_loss = 41 and 38 respectively
sv_max/sv_min ratio ≈ 10⁶ , numerically ill-conditioned
Requires targeted intervention (gradient clipping or re-initialization)
Interpretation & Caveats
Metric calibration: Although is it plausible that WeightWatcher's α-based warnings could be stop/continue signals for this architecture, the model's spectral dynamics may be characteristic of intended neural column modularity and layer specialization advancing as this continues training. It is also characteristic of smaller models for higher layer non-uniformity to a smaller degree.
Perplexity correlation: Given that we have a lower validation perplexity in this epoch 3 model checkpoint, compared with the epoch 1 model checkpoint (https://huggingface.co/david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1), this makes it appear plausible that lower perplexity is correlated with increased spectral heterogeneity (wider α spread), not with uniform α convergence as is the case in this model. This suggests this model architecture may benefit from a division of labor between sharp feature extractors and diffuse residual channels.
Recommended monitoring: Track weak_rank_loss and sv_max/sv_min condition number alongside α. Silent rank collapse (high WRL without α warning) is a more reliable indicator of genuine pathology than α alone in this model family.
Next training iteration: Implement layer - grouped learning rates and dropout - lower regularization for sharp layers, accelerated optimization for diffuse layers, and dedicated LR boost for the output head.
Chinchilla +++ Optimal Training
Base Chinchilla Formula: 20 tokens per parameter
Chinchilla optimal = 40.1M × 20 = 802M tokens
We trained on 1.5B tokens, so at first pass:
1.5B / 802M = 1.87× or 87% over Chinchilla
However, let's adjust for the disproportionate embedding table (As I am a 41M param model, I have a disproportionate percentage of my weights in my embeddign table, compared to a trypical 150M - 1T parameter model, so lets't adjust for that) ...
Embedding Table Adjustment
GPT-2 Tokenizer specs:
Vocab size: ~50,257 tokens
Embedding dim: 512
Parameter breakdown for 40.1M model:
Component
Calculation
Parameters
Total Parameters
-
40.1M
Token embeddings
50,257 × 512
25.7M
Position embeddings
~1024 × 512
~0.5M
Attention and FFN layers
Remainder
~13.9M
Embeddings: ~26.2M (65% of params)
Active/"thinking" weights: ~13.9M (35% of params)
Adjusted Chinchilla Calculation
Chinchilla really applies to the compute relevant trainable weights of the network, i.e., the parameters that actually transform representations (not the lookup tables):
Effective active parameters: ~13.9M
Chinchilla optimal for active params: 13.9M × 20 = 278M tokens
My actual training corpus: 1.5B tokens
Metric
Value
Tokens per active param
107.9
vs. Chinchilla optimal
5.39 × over chinchilla optimal
TLDR, we've grocked to ~ full theoretical saturation on 1 epoch, on 5.39 X chinchilla optimal with only the earlyest signs of destabilization at that point and no substantial rank collapse yet occuring from over-training, see the model card https://huggingface.co/david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep1. Then we continued training, with the perplexity decreasing further by 8.57 from 63.46 at epoch 1 to 54.89 at epoch 3, with the model still mostly stable, albeit showing some early signs of possible rank collapse on a small subset of layers, particularly gating layers.
Generated by ML Intern
This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.
12model_id ='david-thrower/HelixLM-20260529-2240-d512-h8-nl3-ffn2-s512-1500MT-ep3'3from transformers import pipeline
4pipe = pipeline("text-generation",model_id, trust_remote_code=True)5x = pipe('In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are.', max_new_tokens=100, temperature=0.6, top_k=40, top_p=0.95, repetition_penalty=1.2)6print(x)78[{'generated_text':"In the galaxy there are billions and billions of stars. Some of these stars help us pinpoint where other galaxies are. The most common example is that they're a supernova, which has a mass of 8 times larger than our Sun.\nThe star's mass can be estimated as:\n\n• A cluster of massive stars in a single-dimensional space of 10 million years.\n• The Milky Way Galaxy (Mpc) = 13.5 trillion light-years from its center.\n• A large number of stars with an average distance of 1.8 billion light-years away from the centre of the"}]9
