A small (~44M parameter) language model trained on TinyStories, using an unofficial, educational reproduction of a Titans-style self-modifying memory layer with a multi-rate Continuum Memory System (CMS), inspired by Google Research's Nested Learning / HOPE architecture.

⚠️ Not an official Google model or implementation. This is an independent, from-scratch educational reproduction of the underlying ideas, trained at hobby scale on a free Colab GPU. It is not affiliated with or endorsed by Google.

Code / architecture: github.com/Sk16er/hope_nano
Training notebook: Open in Colab
License: MIT

Model description

Instead of standard causal self-attention, the core layer maintains a fixed-size memory matrix M that is read and rewritten every token with a gated delta rule:

α_t (forget) and β_t (write) are predicted per token, per attention head, directly from the input — the model learns how aggressively to overwrite its own memory as it processes text. Alongside this, a multi-rate CMS adds a small stack of MLP memory blocks that each refresh causally at a different period (every 1 / 4 / 16 tokens by default), giving the model both fast per-token memory and slower, longer-horizon memory, in place of a standard Transformer feed-forward block.

Intended use

Educational: demonstrating a self-modifying, linearly-scaling alternative to softmax attention on a small, low-entropy dataset where even small models can produce fluent text. Not intended for production use, factual question-answering, or any application where reliability of outputs matters.

How to use

This model uses a custom architecture (model.py/config.py in the GitHub repo) rather than the transformers AutoModel API. To generate text:

bash

git clone https://github.com/Sk16er/hope_nano
cd hope_nano
pip install -r requirements.txt

python

import torch, tiktoken
from huggingface_hub import hf_hub_download
from config import HOPEConfig
from model import HOPE

ckpt_path = hf_hub_download(repo_id="sk16er/hope_nano", filename="hope_latest.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

config = ckpt["config"]
model = HOPE(config)
model.load_state_dict(ckpt["model"])
model.eval()

tok = tiktoken.get_encoding("gpt2")
idx = torch.tensor(tok.encode("Once upon a time,"), dtype=torch.long).unsqueeze(0)
out = model.generate(idx, max_new_tokens=150, temperature=0.8, top_k=50)
print(tok.decode(out[0].tolist()))

Training procedure

Trained end-to-end using the accompanying Colab notebook, which:

streams TinyStories and tokenizes with GPT-2 BPE on the fly,
checkpoints to Google Drive every 250 steps and auto-resumes, since free Colab sessions disconnect unpredictably — training happened across multiple sessions rather than one continuous run,
uses fp16 mixed precision with gradient scaling and clipping,
carries the memory state across training batches, with periodic resets, using a chunkwise-parallel scan (see repo README for details on why this matters for correctness).

Limitations

Trained on TinyStories only: expect short, simple, children's-story-style English text, not general-purpose capability.
Small hobby-scale model (tens of millions of parameters) trained on a single free-tier GPU — not comparable to production LLMs.
No safety fine-tuning, RLHF, or instruction-tuning has been applied. Outputs are unfiltered next-token predictions from a base language model.
Long-context/memory claims are architectural, not independently benchmarked at scale in this release.

Citation

If you build on this, please cite the underlying research this reproduces rather than this repo alone:

Behrouz, A. et al. Titans: Learning to Memorize at Test Time.
Google Research. Nested Learning: [HOPE architecture and Continuum Memory Systems].
Yang, S. et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length.

Acknowledgments

Architecture inspired by Google Research's Nested Learning / HOPE work and the Titans and DeltaNet papers. This is an independent, unofficial, educational reproduction.