A 145.1M parameter decoder-only language model grown on a single
RTX 4090 with a bandit-controlled curriculum over open data. Part of a
permacomputer project: open source, open data, open weights.
Weights here are bit-identical to the CUDA training checkpoint
step_852106.bin at step 852,106 — the export is verified tensor by tensor, not
assumed.
trust_remote_code=True is required: ANDREA is its own architecture, and the
three *_andrea.py files in this repo are its reference implementation.
Prompt format matters more than usual here
The corpus writes dialogue on a single line as > question / < answer, with
turns joined by /. The chat template shipped with this repo produces
exactly that, and staying inside it is the difference between coherent output
and word salad — a bare prose prompt is out of distribution for this model.
Documents are also separated by BOS during training, and the tokenizer
prepends it by default. Encoding with add_special_tokens=False drops it and
starts the model mid-document, which measurably degrades generation. Sampling
at temperature=0.7 matches how the model was sampled during training; greedy
decoding at this scale degenerates into repetition.
An unconditional sample (input_ids=[[bos]], temperature=0.7) looks like:
> what is the most popular breed of dog? / < the most popular breed of dog is ...
> How does GPU architecture affect gaming performance? / < Modern GPU architectures
like NVIDIA's Ada Lovelace and AMD's ...
ANDREA is deliberately neither GPT-2 nor LLaMA. Reimplementing it means
matching all of these:
RMSNorm with no learnable gain. There is no norm weight in this
checkpoint because the architecture has none.
Learned absolute position embeddings. No RoPE, no ALiBi. The model
cannot extrapolate past 1024 tokens and raises rather than guessing.
A norm on the embedding sum, applied once before layer 0.
A gainless final RMSNorm on the residual stream before the LM head. It is part of the architecture: serving without it puts the raw residual scale into the softmax and cross-entropy lands in the hundreds.
ReLU MLP at 4x. Not GELU, not SwiGLU.
No biases on any projection.
Untied LM head — lm_head.weight is its own tensor, not wte reused.
x = wte[tok] + wpe[pos]
x = rmsnorm(x)
per layer:
x = x + o_proj(attn(rmsnorm(x)))
x = x + down_proj(relu(up_proj(rmsnorm(x))))
logits = lm_head(x)
RMSNorm is x * (mean(x^2) + 1e-05) ** -0.5.
Tokenizer
Harris (1955) successor-variety segmentation: morpheme boundaries discovered
from the distributional properties of text, with no language-specific rules.
The vocabulary is 256 raw byte fallbacks, then 24,576 learned segments,
then BOS at id 24832. Any byte sequence encodes and round-trips, so text the
segment table has never seen degrades to bytes rather than to an unknown token.
Training
Trained on the megachat-v8 corpus through a phase-based UCB1 bandit curriculum
that reweights sources as the model's per-source loss moves. Data was filtered
for dirty text, non-English, and assistant refusals before training.
Limitations
A model this size is a research artifact. It produces fluent English and has
absorbed real factual material, but it hallucinates, has no alignment tuning,
and has no safety filtering beyond the corpus cleaning described above. It was
not built for and should not be used for advice, moderation, or any decision
affecting a person.
Provenance
Full training history, probe analyses, and reproduction instructions live with
the source: https://git.unturf.com. Companion papers: ANDREA-WHITEPAPER
(training) and ANDREA-PROBES (15 interpretability probes).
