A 118.5M-parameter English language model trained from scratch — architecture, tokenizer,
training code and behaviour data are all original. No existing weights were fine-tuned,
distilled or merged.
Same design goal as Sane-47M, 2.5× the size:
a small honest chat model whose knowledge is meant to come from a supplied context rather
than from its own weights.
Runs on CPU. Commands inside the chat: /ctx <text> supplies a retrieval context,
/reset clears history, /q quits.
Prompt format
Four control tokens, no system prompt — the persona lives in the weights.
<|user|>
{question}
<|sane|>
{answer}<|end|>
With retrieval, the context goes before the question:
<|context|>
Search results: The Ridgeloom V4 weighs 34 kg and sells for 690 euros.
<|user|>
How much does the Ridgeloom V4 weigh?
<|sane|>
Sampling defaults: temperature 0.7, top_k 50, top_p 0.92, repetition_penalty 1.12.
For factual questions over a context, drop to temperature 0.3 or decode greedily.
Evaluation
Zero-shot, log-likelihood scoring, 200 examples per task, measured with the project's own
sane_eval.py:
Task
Metric
Sane-118M
Random
GPT-2 small (124M)
HellaSwag
acc_norm
36.0
25.0
~31
ARC-Easy
acc
47.5
25.0
~44
ARC-Easy
acc_norm
28.5
25.0
—
LAMBADA
acc
22.0
~0
~33
FineWeb held-out
perplexity
19.6
—
—
Read this as: roughly GPT-2-small level, on ~40× less training data than GPT-2 saw
relative to its size — clearly above chance, clearly not a parrot, and clearly a small model.
LAMBADA is the weak spot: it rewards long-range context, which 1024 tokens and 118M
parameters do not give much of. PIQA was not measured.
What it does and does not do
Works:
stable identity — knows it is a model, will not roleplay as a human;
reads <|context|> and answers from it, including messy search-result text with
distractors, and says when the answer is not there;
short multi-turn conversation inside the window;
simple code and string tasks (s[::-1] and friends).
Does not work:
facts without a context are unreliable — a capacity wall, not a tuning problem.
Additional factual fine-tuning was tried and rejected: it produced more pattern-matched
leakage, not more knowledge;
arithmetic;
long free-form generation stays grammatical but loses the thread;
The behaviour pool — identity, retrieval-reading, refusal-of-roleplay — is procedurally
generated by the author and contains no third-party text.
ODC-By requires attribution; ATTRIBUTION.md in this repository is that attribution and must
travel with the model if you redistribute it.
Bias, risks and limitations
Inherits the biases of web text filtered for educational quality. No safety tuning, no
harmful-request refusals, no guardrails — treat output as untrusted text. No memory between
sessions, no internet access: anything time-sensitive must be supplied in <|context|>.
License
Released under the Sane Community License 1.0 (see LICENSE). In short: free to use,
modify and redistribute, including commercially, until the product built on it earns more
than 1 000 USD per month — beyond that, contact the licensor. Attribution notice
required: "Powered by Sane-118M."
Citation
bibtex
1@misc{sane118m,
2 title = {Sane-118M: a small language model trained from scratch},
3 author = {S.C.},
4 year = {2026},
5 url = {https://huggingface.co/sekund0chka/sane-118m}
6}
