A 9-million-parameter, byte-level, tokenizer-free language model. No subword vocabulary, no BPE — just raw UTF-8 bytes in, raw UTF-8 bytes out.
This is part of the NoTokenLM family: a series of small models built around one guiding question — how much can a genuinely small model do, if the architecture and training are done carefully, without leaning on scale to cover for weak design?
If you're looking for a model that reasons, does math, or holds a long conversation coherently — this isn't that, and this card will tell you exactly why not. If you're curious what a 9M-parameter transformer can actually pull off when it's pointed at real, unsimplified English text — keep reading.
Usage
This checkpoint is a real transformers-compatible model — AutoModelForCausalLM, AutoTokenizer, and pipeline("text-generation") all work directly, no manual vocab files or custom generation loop required on your end:
What's actually new here (vs. just "another small GPT clone")
A lot of small-model projects are a GPT-2 architecture at 1/10th the size, trained on whatever's easiest to download. We tried to avoid that:
Byte-level vocabulary (256 symbols), not a fixed character set. Earlier iterations of this project used a vocabulary derived from the training data (a fixed list of ~100-600 "seen" characters). That approach quietly breaks the moment the model meets a character it didn't see in training. Byte-level fixes this permanently: there are only 256 possible byte values, full stop. No "unknown character" failure mode.
RoPE (Rotary Position Embeddings) instead of learned absolute position embeddings. Standard GPT-2-style models learn a fixed embedding for each position up to a hard context limit. RoPE encodes position as a rotation applied to attention query/key vectors, which generalizes better and doesn't waste parameters on a position lookup table.
RMSNorm instead of LayerNorm. Simpler, fewer parameters, and what most current-generation open model architectures use instead of the older GPT-2 LayerNorm.
SwiGLU instead of a GELU MLP. A gated feedforward block instead of the plain two-layer GELU MLP from GPT-2 -- the same substitution most modern open-weight LLMs made, and it holds up at this scale too, see the comparison below.
Weight tying. The output projection reuses the same weight matrix as the input embedding instead of learning a separate one. At 9M total parameters, this frees up a meaningful fraction of the budget for the rest of the network.
None of these are novel inventions -- they're standard in modern LLM architectures. What's less standard is applying all of them, deliberately, at a 9M-parameter scale, and actually measuring whether it matters. It does. See below.
Head-to-head: this model vs. its predecessor (NoTokenLM-Gen-2.5)
We compared this model against its direct predecessor (Gen-2.5, ~3.1M parameters, GPT-2-style architecture: GELU, LayerNorm, learned position embeddings) on 8 held-out English sentences the models weren't trained on, by loading both models' raw weights directly and checking which one predicts each sentence better.
Result: Gen-3.5 beat Gen-2.5 on all 8 out of 8 test sentences. No exceptions, no cherry-picking.
Gen-3.5 is roughly 3x larger than Gen-2.5 (9M vs. 3.1M parameters) and uses a different architecture (RoPE, RMSNorm, SwiGLU, weight tying, byte-level vocabulary vs. Gen-2.5's GPT-2-style design). Some of the gap is the extra capacity; some of it is architecture. We haven't isolated exactly how much of each -- that's an open question, not a hidden one.
How well does it actually write? (100-prompt manual evaluation)
We ran 100 generations -- 10 short prompts x 10 samples each, temperature 0.5 -- and read every single one by hand. No cherry-picking, no filtering, everything below is the real distribution.
Grading criteria (deliberately strict -- "grammatically fine but says nothing" does NOT count as a win):
Fully coherent: correct grammar and the sentence actually holds together as a real thought, right up to where it's cut off.
Grammar OK, meaning partially/fully broken: reads like English sentence-by-sentence, but drifts into contradiction, nonsense combinations ("the man was opened"), or repetition loops ("the sky and the sky and the sky").
Grammar breaks down: the sentence structure itself falls apart.
We re-ran this at larger scale afterward (1,000 generations, same 10 prompts, temperature 0.6) as a cross-check, judged independently, and landed in the same ~40% range -- with a wide spread by prompt: prompts that resolve into a clear subject doing a clear action ("She said", "She walked into the room and") score noticeably higher (~55-58%), while bare-noun or repetition-prone prompts ("The mouse", "He looked at the sky and", "The door opened and") score much lower (~13-32%). That's the clearest signal for where the next data/training pass should focus.
Real examples, unedited, that landed in "fully coherent":
"She walked into the room and" -> "said: 'Do you know the story, Mrs. Ellen?'"
"The door opened and" -> "the stranger saw his face fall to the window."
"The old man" -> "smiled simply at the same thing, and with a smile"
"She said" -> "\"I don't know what you want.\" \"So you are,\""
"The mouse" -> "came down and said to him: 'I can see them now, d[...]'"
"The boy ran" -> "to meet him, and the boy took his hand. \"The"
Real examples that landed in "grammar OK, meaning breaks down" (also unedited):
"He looked at the sky and" -> "the sky and the sky and the sky, th[...]" <- repetition loop
"The little girl" -> "was taller than the widow of the childre[n]" <- nonsensical comparison
"They walked" -> "along the streets of the Cape of Good Lords" <- "Cape of Good Hope" corrupted into a nonsense proper noun
"The door opened and" -> "the man was opened, and he saw that the colored m[...]" <- a person can't "be opened"; likely bled in from "the door opened"
That second category is the most informative one, honestly. The model isn't producing word salad -- it's producing real English grammar wrapped around a broken idea. That's a specific and consistent failure mode, not random noise, and it's worth understanding rather than hiding.
What it's actually good at
Short-range coherence. One clause to about one sentence, the model reliably tracks who's doing what to whom.
Real word morphology, including words it never saw. It completes extraordin- -> extraordinary, philosoph- -> philosophy, metamorphos- -> metamorphosis correctly. More interestingly, it generalizes suffix rules to nonsense roots it has never encountered: flimperize -> flimperized (correct past tense of a made-up -ize verb), wexify -> wexifying (correct -ify -> -ifying). That's not memorization -- the training data does not contain "flimperize." It's picked up a genuine, applicable rule about how English verb suffixes work.
Dialogue formatting. Quote marks, "said X" attribution, and turn-taking in conversation come out in roughly the right shape, because the training corpus is full of it (19th-century novel dialogue, mostly).
What it's not good at, and why
No real-world knowledge. At 9M parameters, there is nowhere near enough capacity to store facts about the actual world -- names, dates, places, how things work outside of what's implied by narrative prose grammar. Ask it who won a particular war, what a device does, or any factual question, and you'll get a fluent-sounding but unreliable or fabricated answer, not retrieved knowledge. This is a capacity/training-data-mix issue, not a bug: the pretraining corpus is narrative fiction, and the model was never meant to be a knowledge store. If a prompt happens to use only common, frequently-seen words and simple sentence shapes, coherence holds up fine without needing any world knowledge at all -- but don't mistake that fluency for the model actually "knowing" anything.
Arithmetic: essentially zero.2 + 2 = produces 3. 5 + 3 = produces 4. This is not a subtle weakness -- it's a complete absence of the skill, and it should be, because this checkpoint has never seen a single math example. The pretraining corpus was narrative prose and general web/encyclopedic text; there is no arithmetic-formatted data in the mix at all. Testing this model on math and calling the result "bad at math" would be like grading someone on a subject they were never taught. A future checkpoint trained with math-formatted data in the mix is a different, testable question -- this one just doesn't have the exposure.
Long-range coherence (3+ sentences) degrades noticeably. In longer free-running generations we tested (400+ characters), the model starts losing track of who the subject is (a "she" quietly becomes a "he" a few sentences later), and can get stuck fixating on a specific word (one run repeated "politeness" three times in unrelated contexts). This is the model's single biggest limitation right now. At 9M parameters with no consistency-focused fine-tuning stage yet, this is the expected shape of the failure -- not a bug, a capacity ceiling.
Raw source-text artifacts leak through, most visibly \r\n\r\n followed by indentation spaces -- a direct fingerprint of Project Gutenberg plain-text formatting present in part of the training corpus. It's not a hallucination or a model defect; it's exactly what happens when you don't scrub Windows-style line endings and paragraph indentation out of raw ebook text before training on it. Fixable with a data-cleaning pass, not a model change.
We're not going to pretend these don't exist. If you use this model expecting fluent short-form English generation with real morphological understanding, you'll get that. If you expect anything requiring math, extended multi-paragraph coherence, or clean whitespace out of the box, you won't -- yet.
How to actually run this thing
Recommended temperature: 0.4-0.6.
Below ~0.3: the model becomes highly repetitive and safe -- technically fluent, but it'll loop on the same phrasing constantly. Fine for a quick sanity check, not for actual use.
0.4-0.6: the sweet spot we tested against (0.5 specifically, for the 100-prompt evaluation above; 0.6 for the larger 1,000-generation cross-check). This is where the coherence numbers in this card actually apply.
Above ~0.7: coherence drops off fast. In our own quick tests, 0.7+ generations degrade into grammatically loose, semantically wandering text noticeably more often than at 0.5-0.6. If you want creative/loose output that's a valid choice, just know the coherence rate quoted above won't hold at that setting.
top_k sampling in the 15-40 range worked well in our tests, in combination with the temperature range above. Greedy decoding (temperature -> 0) is not recommended -- it tends to produce short, repetitive loops rather than natural text.
Architecture details
Parameters
9,055,440
Layers
13
d_model
240
Attention heads
4
Feedforward dim
640
Vocabulary
256 (raw bytes, no tokenizer)
Context length
1024 bytes
Position encoding
RoPE
Normalization
RMSNorm
Feedforward
SwiGLU
Output layer
Weight-tied to input embedding
Training
Byte-level next-token prediction (pretraining only -- no instruction tuning applied to this checkpoint)
Training data
Primarily English narrative prose (public-domain 19th/early-20th-century novels) plus general encyclopedic text. This checkpoint has not been trained on structured math, dialogue-formatted QA, or an instruction-following format -- it is a base pretraining checkpoint, not a chat or instruction model.
What's next
This is a checkpoint in an ongoing series, not a final answer. The next stage in progress focuses specifically on consistency -- using a broader and more diverse dataset mix aimed directly at the long-range coherence weakness described above, alongside optimizer changes intended to improve training stability. Whether that closes the gap is an open, testable question, and we'll report the real numbers when we have them -- the same way this card reports the real numbers for this checkpoint, wins and weaknesses both.
Part of the NoTokenLM family -- small models, built and evaluated honestly.
