nLemon-14-sft · a 13.8M-parameter decoder-only transformer, pretrained from scratch and
instruction-tuned, on one 4 GB laptop GPU. "Born to speak, disciplined to behave."
Model of record
nlemon-14-sft-weights.pt — the SFT checkpoint, certified twice (Phase 5 gate green; Phase 6 floors confirm DPO left it untouched)
The shipped state_dict sums to 16,889,856. The published parameter count is 13,817,856.
Both are correct.
16,889,856 sum over state_dict values
- 3,072,000 tied embedding, counted twice (vocab 8,000 x d_model 384)
= 13,817,856 unique parameters
The output head is tied to the token embedding — lm_head.weightistok_emb.weight, the
same tensor stored under two names. Summing state_dict double-counts it; model.parameters()
de-duplicates and gives the published figure. Tying saves 3.07M parameters, 22% of the model, and
ties "predicting a token" to "representing a token."
This is stated because a reader who recomputes from the shipped file will get the larger number,
and a card that offered only one figure would contradict its own artifact to exactly the audience
that checks. python -m src.release verify reconciles the arithmetic against the file.
Performance
metric
value
notes
validation perplexity
5.2662 ± 0.27%
base.pt, full validation split, 4,682,459 tokens
5.5794
sft.pt — 6.0% worse than base, and expected; see below
bigram floor
41.81 → 7.65× better
scored on the baselines' identical token array
unigram floor
389.91 → 71.36× better
uniform floor
8,000 → 1,464× better
The ± 0.27% is the measured, design-effect-adjusted standard error, not the iid one. The naive
iid figure is 0.0979%. Tokens cluster inside documents — measured ICC 0.0296 over 21,990 documents
averaging 213 tokens, giving a design effect of 7.28 and n_eff 643,480 rather than 4.68M. The
iid number would have overstated the precision by 2.7×. Quoting it would have been the more
impressive and less true option.
Why the headline is 5.2662 and not the 5.1981 in earlier commits.best_val_perplexity was the
minimum over 40 validation passes, because training saves a checkpoint only when validation
improves — an order statistic, biased low by construction, sitting at the 4.2 percentile of its
own estimator's sampling distribution. The full-split pass has no sampling error to be lucky in.
Every figure clears the pre-registered ≤ 8.0 bar, so no verdict ever depended on the choice.
Why sft.pt scores worse than base.pt on perplexity, and why that is not a regression.
Instruction tuning specialises the model onto an instruction/response format, so its fit to the
pretraining distribution necessarily degrades. The +6.0% is the price of the specialisation, not
a defect — and it is printed here because a scorecard that reported only the metrics a stage
improves would be an advertisement rather than a measurement. What SFT was for is instruction
adherence, and that is measured separately: subject_mention 47.8% → 70.8% on 78 held-out subjects.
Subjects in this eval never appear in any training pair. The shuffled control is the validity
check: it re-scores each response against a different subject, so a model that simply names many
nouns scores high on it. At 1.9% it does not, which is what licenses reading subject_mention as
adherence rather than noun-spraying.
Licensing
Weights, code, and this card: MIT.
Training corpus:TinyStories,
revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64, under CDLA-Sharing-1.0.
CDLA-Sharing-1.0's share-alike obligation attaches to the Data. Computational Results
obtained from analysing the data are explicitly outside it, and model weights are Results — which
is the basis for releasing the weights under MIT. The corpus is not redistributed here; the build
pulls it from its own source, and data/manifest.json records the exact revision and its license
so the provenance is checkable rather than asserted. Stated for transparency, not as legal advice.
Limitations
Tiny, and trained on one narrow corpus. TinyStories is simple English children's stories. The
model has no knowledge, no reasoning, and no safety training. It is a methodology artifact.
256-token context. Roughly 200 words.
DPO did not generalise.dpo.pt reached ~81% pair-ranking accuracy on its 714 training pairs
but transferred only +3.5pt to unseen subjects — below the 8.9pt the eval can resolve. It ships as
a characterized side-result, never the headline. sft.pt is the model of record.
oov_plausibility is dormant on this model. 2 out-of-vocabulary words in 45,031 — the band
cannot be read at that count, and is reported as unreadable rather than as a pass.
Not evaluated for bias, toxicity, or factuality. Do not deploy it.
Reproducing
bash
1python -m src.evalset verify # eval sets match their registered hashes2python -m src.eval scorecard # the 5-gate Phase 7 scorecard3python -m src.release verify # shipped file matches its recorded hash4python -m tests.test_residue # encoding-repair regression
Requires the repo venv (torch 2.12.1+cu130). The harness asserts its device and fails loud
rather than falling back to CPU — a silent fallback once put a device difference inside a
cross-phase comparison (ADR-047).
Every number above is regenerable from recorded inputs. The full decision record, including the
gates that failed and why, is in ADR.md — 51 entries.
