You can deploy this model and start earning money today!

Bryan237l/
LOLM-304M

lolm

language-model

hybrid-transformer-ssm

Views

No views yet

LOLM-304M — Latent Order Language Model

LOLM is a hybrid Transformer–SSM architecture that separates surface token prediction from latent state tracking. This is the 304M research checkpoint behind the published WikiText-103 results.

Authors: Bryan Leonard & Brandyn Leonard · Qira LLC · Provisional patent 64002166 Code: https://github.com/TheArtOfSound/LOLM

TL;DR

At 304M parameters, LOLM reaches 68.37 PPL on WikiText-103, beating Pythia-410M (142.93 PPL) by 52% at matched compute with 26% fewer parameters. It is a base language model — evaluate it on perplexity and representation quality, not on instruction-following (it is not instruction-tuned).

Architecture

Each token flows through five parallel streams that converge via learned fusion:

o_t = g · LN(W_h·h_t) + (1−g) · LN(W_z·z_t) + W_m·m_t + W_r·r_t

Stream	Role	Implementation
Surface decoder	local token relationships	pre-norm Transformer + RoPE
Latent SSM core	slow latent dynamics	selective SSM (Mamba-style)
Regime layer	discrete phase detection	Gumbel-Softmax + causal conv1d
Persistent memory	cross-sequence state	3-bank gated read/write
Manifestation gate	surface↔latent arbitration	per-dimension sigmoid MLP

Results

Metric	LOLM-304M	Pythia-410M	Δ
Parameters	304M	410M	−26%
WikiText-103 eval PPL	68.37	142.93	−52%
Late-position BPC	1.02	1.23	−17%
Distinct-2 (generation)	0.687	0.607	+13%

Inference-time ablations confirm every component contributes (regime, SSM, gate, memory). See the paper for full tables and the 1.57B-scale comparison.

Usage

LOLM is a custom architecture — load it with the LOLM code, not transformers:

bash

git clone https://github.com/TheArtOfSound/LOLM.git && cd LOLM
pip install -r requirements.txt

python

import torch, tiktoken
from lolm.config import load_config
from lolm.model import LOLM

cfg = load_config("configs/scale/300m_lolm_full_tpu.yaml")
model = LOLM(cfg.model)
ckpt = torch.load("ckpt_26000.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

It exposes its own latent dynamics every step (out.gate_values, out.regime_probs) — used by the NFET runtime governor in the lolm-bridge workspace.

License

Released under the LOLM Community License Agreement v1.0 (see LICENSE): free for academic research, education, and personal/non-commercial use; commercial license required for qualifying entities.

Citation

bibtex

@article{leonard2026lolm,
  title  = {LOLM: Language Modeling Beyond the Surface with Hybrid
            Transformer-SSM Latent Order Fields},
  author = {Leonard, Bryan and Leonard, Brandyn},
  year   = {2026},
  note   = {Qira LLC. Provisional patent application No. 64002166.}
}

Bryan237l/LOLM-304M