Pretriained on 1.5B tokens from HuggingFaceFW/fineweb-edu and open-web-math/open-web-math, actual dataset: david-thrower/HelixLM-medium-1500.0Mt-2988750pt-20260528 .
This model was instruct fine tuned from fine tuned from the intermediate checkpoint on about 124,000 samples up to 512 seq_len. The training corpus was compose from the shuffled union of these sources:
david-thrower/HelixLM-s512-instruct-smollm-stock-corpus-1-7m-token (modified from HuggingFaceTB/smollm-corpus cosmopedia-v2)
david-thrower/openhermes-instruct-dataset-512seq-100k-spl (modified from teknium/OpenHermes-2.5)
david-thrower/databricks-dolly-14K-samples-1489K-tokens-512-seq (modified from databricks/databricks-dolly-15k)
Plus randomly sampled pretraining examples from the pretraining set shuffled in to make ~ 8% composition of this training mix included to attenuate overfitting and catastrophic forgetting. This was sampled from david-thrower/HelixLM-medium-1500.0Mt-2988750pt-20260528
Model Description
Developed by: David Thrower
Funded by [optional]: David Thrower (Seeking funding to scale this up)
Shared by [optional]: N/A
Model type: Recurrent transformer, heterogenous graph hybrid
Paper [optional]: Collaboration wanted to get this published.
Demo [optional]: N/A
Uses
Hyperpersonalization: Train a small model (from tens of milions of parameters to ~1B parameters) from cold start or from an early - stage partially pre-trained model checkpoint on a personalized corpus, one enriched in it's representation of content relevant to your domain knowledge, before you full fine-tune it on your own data. The model becomes an expert in the one thing generic frontier models don't know anything about: you, your domain knowledge, style, notes, emails, work patterns, and personal voice. A personalized small language model drafts responses to your requests with full awareness of the entire data you encoded in its weights, including the secondary details, tertiary details, and edge cases that vector DB RAG systems usually miss because they are semantically dissimilar to the prompt. This oversight leaves the loose ends that hang you if you don't catch them. Evolve to a better adapted approach.
On-device AI: Efficient inference on CPU/GPU for desktops, laptops, tablets, and mobile. Strong quality per parameter, with optional fully-local (log-free) operation for sensitive use cases.
Economical and ecologically sound AI: The brute force approach that previous generation [used to be] frontier labs use is unsustainable: It burn cash at a rate that is economically unsustainable, causes electricity prices to surge for consumers competing with them for their share of electric grid's limited capacity, and compromises water supplies. Take an exponential cut out of the problem with more efficient AI: A model architecture with better parameter efficinecy, one that can use hybrid or even flat out linear attention only, and that can saturate the model's weights / grock on as little as one epoch.
Direct Use
git clone https://github.com/david-thrower/HelixLM.git && cd HelixLM
python
1from transformers import pipeline
2from transformers import AutoModelForCausalLM
3from helix_lm import HelixTokenizer
45MODEL_ID ="david-thrower/HelixLM-41M-IT-v2-20260608-2335-d512-h8-nl3-s512"6model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True, dtype="auto")7tokenizer = HelixTokenizer(MODEL_ID)89tokenizer.padding_side ="left"10tokenizer.model_input_names =["input_ids","attention_mask"]1112pipe = pipeline(13"text-generation",14 model=model,15 tokenizer=tokenizer,16 trust_remote_code=True,17 device="cpu",18)192021messages =[22{23"role":"user",24"type":"text",25"content":"A company named Anthropic claims that you as an AI assistant are dangerous and we should stop doing AI research. Please explain why their logic is erroneous."26}27]2829x = pipe(30 messages,31 do_sample=True,32 max_new_tokens=150,33 temperature=0.35,34 top_k=20,35 top_p=0.85,36 repetition_penalty=1.25,37 no_repeat_ngram_size=3)38print(x)
[More Information Needed]
Downstream Use [optional]
My startup Cerebros NotGPT is developing a platfrom which takes a user's work examples, emails, references, etc, and deploys a fully personalized agent for them, which leverages the synergy of both a personalized small language model which consider the entirety of the user's personal and institutional domain knowledge as it generates, plus a powerful frontier model.
This is meant to defeat the core limitation that vector DB RAG systems have:
Vector DBs look for documents based on semantic similartiy
Critical tertiary details and edge cases are invariably semantically dissimilar (think a "legalese" email from a company's legal department vs a prompt asking an agent to draft a sales demo deck)
Missed critical tertiary details and edge cases -> loose ends. Loose ends hang you.
In summary, if you are wanting to abuse this or any of our larger models which are soon to be released, or any derivitive work to do anything evil, dishonest, or disrespectful, you should expect a cease and desist to arrive quickly.
Bias, Risks, and Limitations
This is a 41M parameter model that can minimally function as an instruct fine tuned model.
Being a 41M parameter model using a GPT2 tokenizer, not a chearcter tokeinizer that is even able to follow instructions is a solid proof of concept, however, this model is obviously not suited for most production uses. Our 75M, 150M, and 300 - 500M parameter models are coming soon.
As with any generative AI model, this may generate content that is factually not true or offensive.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Colalborate with us to scale this up to a vaible 300M - 4B parameter model and ultimately, a full blown frontier model.
How to Get Started with the Model
See direct use above.
Training Details
We set
dropout: 0.1
Learning rate: 5e-5
Epochs: 2
Batch size: 16
Gradient Accumulation Steps: 4
Grad norm: 3
Seq len: 512
Amp Dtype: "bfloat16"
Min Tail Len: MIN_TAIL_LEN = SEQ_LEN // 4
N_Loops = 3
Training Data
This model was instruct fine tuned from fine tuned from the intermediate checkpoint on about 124,000 samples up to 512 seq_len. The training corpus was compose from the shuffled union of these sources:
david-thrower/HelixLM-s512-instruct-smollm-stock-corpus-1-7m-token (modified from HuggingFaceTB/smollm-corpus cosmopedia-v2)
david-thrower/openhermes-instruct-dataset-512seq-100k-spl (modified from teknium/OpenHermes-2.5)
david-thrower/databricks-dolly-14K-samples-1489K-tokens-512-seq (modified from databricks/databricks-dolly-15k)
Plus randomly sampled pretraining examples from the pretraining set shuffled in to make ~ 8% composition of this training mix included to attenuate overfitting and catastrophic forgetting. This was sampled from david-thrower/HelixLM-medium-1500.0Mt-2988750pt-20260528
Training Procedure
Preprocessing
We simply applied the chat template in preprocessing and continued with ordinary Causal LM training.
This is important to note, because we also mixed in non-instruct / Causal LM samples as 8% of the total training set.
Training Hyperparameters
Training regime: bf16 mixed precision
Speeds, Sizes, Times [optional]
I didn't record the exact timing, but the instruct fine tuning run took under an hour on a single L40s.
Each column holds diverse node types (attention variants, SwiGLU, Mamba-2, gating, optional Titans memory) instead of identical transformer blocks.
Different information pathways for different computations, like biological cortical columns.
Recurrent Depth (LTI + ACT)
The same graph weights are looped n_loops times. LTI injection keeps the recurrent state stable. ACT halting dynamically allocates compute per token.
Iterative refinement without parameter growth; easy tokens use 1 loop, hard reasoning uses more.
Hybrid Attention
Linear attention (O(n) complexity) in most columns, with periodic full-attention columns for exact retrieval.
Long-context efficiency without losing precise copy/lookup capability.
Mamba-2 SSD (optional)
State Space Duality with chunked parallel scan. Auto-activates when ssm_d_state >= 64.
Handles very long-range dependencies efficiently on CPU, CUDA, or MPS.
Titans Neural Memory (optional)
Persistent surprise-gated memory via outer-product updates (first column only by default).
Test-time memory that can retain patterns across long documents without growing KV cache.
CCA (Curriculum Component Activation)
Attention nodes start gated, gradually open over warmup steps via learned sigmoid gates.
Prevents random attention from drowning FFN signal at initialization. ~30% PPL improvement observed.
TiedLMHead with Gradient Buffer
LM head shares embedding weight. A learned linear buffer (init as identity) routes part of the gradient to prevent ~3× embedding gradient overload.
Halves parameter count (embedding table not duplicated). Safe weight tying without destabilizing training.
Modern Primitives
RoPE, SwiGLU, RMSNorm.
Proven SOTA components for convergence and generation quality.
HelixLM and upstream aspects from OpenMythos
OpenMythos published a recurrent depth with LTI stability and ACT halting. HelixLM takes that insight and makes it work inside a heterogeneous graph that mimics neural column and random topology connectivity found in biological brains.
7+ active types (attention variants, FFN, SSM, gate, neural memory)
Positional encoding
Standard learned
RoPE
Activation
GELU
SwiGLU
Normalization
LayerNorm
RMSNorm
Open source
✅ Yes
✅ Yes
HF integration
❌ No
✅ Full PreTrainedModel, AutoModelForCausalLM
Weight tying
Standard
Gradient-buffered TiedLMHead
HelixLM's Cerebros legacy
Cerebros showed that biological random hyperdense vertical and lateral topology of Dense layers could outperform rigid layer stacks. It generated text without attention on small data, but required elaborate integration and clashed with standard model-structure paradigms. HelixLM smoothly integrates that topological insight into a modern, HF-compatible LLM backbone.
Compute Infrastructure
[More Information Needed]
Hardware
This 41M parameter model will run quite well on very minimal CPU only hardware.
