Cassandra 200M text8 (pure-broad character flagship)
A 201,609,249-parameter decoder-only character-level language model trained
from scratch on text8 (cleaned lowercase Wikipedia) on a single laptop GPU
(RTX 4070, 8 GB). It is the Stage 61 flagship of the
Cassandra research project, which
studies how far useful language-model behavior can be formed under strict
hardware constraints, with every claim pre-registered and measured.
This is a research and education artifact, not a product. It writes fluent,
Wikipedia-flavored prose locally but drifts off topic across long generations
(see Limitations).
50,000 steps, ~204.8M characters (about 2.28 epochs of the 90M-char text8 train split)
Seed
7 (single seed)
Intended use
Local research, demonstration, and education on small-model training under
constrained hardware: learning-curve study, checkpoint evaluation, prompt
sampling, context-window and coherence experiments, and following the Cassandra
staged hypothesis record.
It is not intended for production assistance, factual question answering,
safety-critical decisions, medical/legal/financial advice, moderation, or any
claim of broad or reliable English competence.
How to run
Requires the Cassandra repo code (Apache-2.0) for the model class and the
downloaded package (safetensors weights plus config.json and codec.json):
bash
1git clone https://github.com/mertefesensoy/CASSANDRA
2pip install torch numpy safetensors
3# put the .safetensors, config.json, and codec.json beside the code, then:
python
1from flagship_eval_lib import load_model_from_safetensors, sample_text
23model, codec, config, meta = load_model_from_safetensors(4"stage61_pure_broad_200m_text8_fp16.safetensors",5"config.json","codec.json", device="cuda",# or "cpu"6)7print(sample_text(8 model, codec,9 prompt="the history of ",10 max_new_chars=400,11 temperature=0.8,12 top_p=0.9,# recommended: removes the low-probability garbage tail13 seed=7, device="cuda",14))
Weights are shipped as safetensors (no pickle), so the file loads without
executing any code.
Recommended inference: temperature 0.8 with top-p 0.9. Nucleus truncation
removes a low-probability tail that otherwise emits rare garbage tokens. It does
not fix topic drift (that is a context-window limit, below). Very low
temperature makes the model repeat itself, so avoid greedy decoding.
Evaluation
The headline metric is bits per character on the standard text8 test split
(the final 5,000,000 characters), a long-established character-language-modeling
benchmark, so the result is directly comparable to published models. Scoring is
deterministic: non-overlapping 256-character windows over the whole test split,
context reset per window (lower is better).
Metric
Value
text8 test bits/char (released fp16 weights)
1.336061
text8 test NLL (nats, released fp16 weights)
0.926087
text8 test bits/char (fp32 training checkpoint, project canonical)
1.336059
Reference: Cassandra 85M pure-broad model, same eval
1.357318
Reference: GPT-2 117M zero-shot text8 (Radford et al. 2019)
1.17
The released fp16 weights reproduce the fp32 result to within 0.000002
bits/char (fp16 rounding). The model beats the same-recipe 85M reference by
about 0.021 bits/char, roughly seven times the known between-seed noise floor
(0.003035). It does not match GPT-2, which used subword tokenization and far
more training; the 1.17 anchor is aspirational context, not a peer comparison.
Independent verification
This number is self-reported but fully reproducible by anyone. The shipped
reproduce_text8_eval.py downloads text8, scores the standard test split on the
released fp16 weights with the deterministic method above, and prints
1.336061:
There is no automated third-party evaluation service for a custom-architecture
character model (the Open LLM Leaderboard and Hugging Face inference-based evals
target instruction-tuned models loadable by the standard harness), so
reproducibility on a public benchmark is the honest form of external validation
here. A separate Cassandra probe (H027) measures long-range context use at
+0.205 bits/char and is documented in the code repository.
Limitations
Topic drift is the headline limitation. Over long generations the model
wanders off subject. This is a context-window limit, not a comprehension
failure: the Cassandra context-utilization probe (H027 / ADR 0019) shows the
model uses its context strongly to the edge of its 256-character window, so it
drifts precisely when a generation runs past what it can still see. Use
top-p 0.9 to remove garbage; expect drift regardless.
Character-level and small. 200M parameters, laptop-trained, single seed.
It reasons about no facts and grounds nothing; it produces plausible-looking
but frequently false encyclopedic-style text.
No alignment. No instruction tuning, no RLHF, no safety filtering. It can
produce misleading, repetitive, or low-quality text and must not be presented
as truthful or broadly knowledgeable.
Narrow input format. Inputs are normalized to lowercase a-z and space
(the text8 alphabet); other characters are folded to space.
Training data
text8 (Matt Mahoney's cleaned 100M-character lowercase Wikipedia benchmark;
first 90M for training). The underlying text lineage is Wikipedia, licensed
CC BY-SA / GFDL. Training corpora are not distributed with this model;
Cassandra ships the preparation scripts and provenance metadata only. The
corpus is rebuildable from those scripts.
