A 25-million-parameter logical-verdict engine that knows when it
doesn't know. Byte-level (256-symbol vocabulary, no tokenizer),
trained on a single consumer GPU in under two hours total, runs on CPU
in pure Rust. Strawweight is the smallest class in the ZophiaE line —
named the way boxing names its divisions, and it punches accordingly.
What this is, and why it's built this way
Most language models spend enormous parameter budgets learning what
words are — a tokenizer vocabulary, embeddings for tens of thousands
of subword fragments. This line takes the opposite bet: words are
encoded before training, as concept bytes — a dictionary-defined
encoding where one byte is (roughly) one word-meaning and grammar is
explicit feature bytes. The model's 25M parameters are spent on
reasoning over meanings, not on reconstructing what "bread" is from
fragments. Three consequences, all deliberate:
It's small and it's local. No tokenizer, no cloud, no Python
required — the whole thing runs on a CPU, in a folder, offline.
This is a bet that useful reasoning doesn't need a datacenter.
It's inspectable. Every byte in the model's context is a
labeled concept, so attention weights are readable relationships,
not opaque token indices. The Talkit runtime shows you, per
generated byte, what the model weighed and what it attended to.
It's a specialist on purpose. The training corpus is synthetic,
oracle-checked logic, balanced so that refusing to answer is a
first-class move (25% of training answers are "we can not say" —
and they're correct refusals, not hedges). The result is the
rarest property in small models: calibration — it refuses with
98% precision instead of confabulating.
The custom architecture and custom encoding are why this isn't a
transformers-loadable model — and also why the whole stack
(translator, codec, dictionary, runtime, training script) ships in
this repo. Nothing here depends on anything you can't read.
Measured on 2,024 fresh questions proven by hash never to have
appeared in any training document:
four-way verdict accuracy (YES / NO / UNKNOWN / OPEN)
89.1% (floor for any constant answerer: 25%)
refusal precision — when it says "we can not say," it's right
98.1%
answers that close their own frame (END discipline)
~99%
the same architecture at half the data ration
35.9%
It computes — real division, real counting; change a number in your
question and the verdict follows the arithmetic, not the template. It
refuses when premises genuinely underdetermine the answer (denying
the antecedent is a refusal here, not a yes). And it shows its work
in every answer.
Q: john has 56 breads. the group uses 4 breads each month. the bread
must last 8 months. if the bread does not last, then the sun sets.
does the sun set?
A: 56 / 4 = 14. 14 months against 8, so the bread lasts. If the bread
does not last, then the sun sets. The bread lasts. The rule speaks
only about what happens when the bread does not last. We can not
say. The words can all be true while the sun sets, and also while
the sun does not set.
That is the correct answer — and the branch most models fail.
How to run it
This is not a transformers / GGUF model — the architecture and
the concept-byte codec are custom. So this repo ships the entire
stack: weights, dictionary, the English↔bytes translator, the
reference PyTorch implementation, and the signed Windows appliance.
Three ways in, all self-contained:
1. Windows, zero install. Download the repo, run
SETUP_TALKIT.bat once (it arranges the folders), then double-click
talkit.exe. Chat window, reasoning pane with per-byte attention, a
Verify tab that reproduces the numbers above on your machine, and
a model picker. (Also on
GitHub.)
2. Pure Python (pip install torch safetensors — nothing else):
python run_strawweight.py "a beetle is an insect. all insects are
animals. so the beetle is an animal. if the beetle is an
animal, then the horse sleeps. does the horse sleep?"
Runs translator → codec → model → decode → render and prints the
annotated stream plus the English answer. train_25m.py here is the
actual, unabridged training script — the model definition and the
full recipe in one file. zofiae.py is the codec, simplify.py the
translator, data/table/ the dictionary.
3. As a local service:
talkit serve --port 8484
POST /ask {"question":"..."} -> {"answer","kind","confidence_mean",...}
Questions.md is the playbook: every trained question family with
paste-ready examples.
It's a pretrain — continue training on your data
Strawweight is a base model. The intended use is to take it,
continue training on your data, and see whether it works for your
problem. Everything needed is in this repo, and the loop has been run
end to end exactly as written below:
1. Make a corpus from your text (no Python): drop files or paste
text into Talkit's Build Corpus tab, or headless —
talkit corpus C:\path\to\your_text_folder -> your_text_folder\corpus.bin
Each file or paste becomes one concept-byte record through the
strict translator (a sentence either survives round-trip-perfect or
is dropped with its reason). Keep records under 2,560 bytes — a
paragraph, or one question-and-answer — the trainer refuses longer
documents. Q&A-framed records (question, then the answer) come from
the chat→corpus button; plain records train as ordinary text.
2. Continue training from the published weights
(pip install torch safetensors):
set ZOFIAE_MODEL=25m
set ZOFIAE_CORPUS=C:\path\to\your_text_folder
set ZOFIAE_RUNS=C:\path\to\runs
python train_25m.py --resume model.safetensors --corpus-glob "corpus.bin" --epochs 3 --tag mine
train_25m.py resumes directly from this repo's model.safetensors
and writes runs\run_NNN_mine\best.pt. With a small corpus the held-
out split will be empty (eval_seen 0 docs) — that is expected; judge
the result with your own questions or the Verify tab, not that line.
3. Put the result back into Talkit
set TALKIT_ASSETS_DIR=C:\path\to\Talkit\assets
set TALKIT_NAME=my fine-tune
python export_talkit_assets.py C:\path\to\runs\run_001_mine\best.pt my-model
It lands in assets\models\my-model\, the picker shows it on next
launch, and TALKIT_MODEL=my-model talkit serve serves it. Keep the
original folder beside it and use A/B to compare before and after.
License note: a model continued from these weights is a modified
version of the Materials, not an Output — it carries the same license
(free noncommercial, commercial by written license capped at 6%). See
LICENSE.md §4.
Watching it think, reading the weights
Interpretability isn't a research rig bolted onto this model — it's a
property of the encoding. Every position in the context is a labeled
dictionary concept, so attention weights are readable relationships
and the learned space can be inspected by name. Four instruments,
all included:
The Talkit reasoning pane (no code): click any answer bubble —
per generated byte, the candidate distribution the model weighed
and the attention it paid, per layer and head. talkit trace "question" dumps the same evidence headless.
watch_it_think.py — the Python version, labeled end to end:
That's the model doing the division — while it writes "14", its
attention sits on /, 56, 4, and =. Attention is
recomputed explicitly (fused kernels don't return weights),
mirroring the training forward line for line.
read_weights.py — composes every dictionary word through the
network once and lets you cosine-explore the learned space by
name: sun sits with snow, hot, dark; rise with appear
and climb. No probing classifiers — the labels were built into
the vocabulary before training.
train_25m.py's own instrumentation — the training log
reports a frequency-to-attention correlation each epoch: if
attention merely mirrored token frequency, the model would be
counting, not reasoning. (v3's stayed near zero: r ≈ +0.08.)
A worked example of what this catches: asked about a sun that
"sets" — a verb never trained in that slot — the model restated the
rule as "the sun falls," the semantically correct pick out of
seven trained sky verbs, rejecting the frequency-favorite "rises."
watch_it_think.py shows the decision being made in context;
read_weights.py shows the labeled space it was made in.
The box it lives in (honest scope)
Strawweight is a specialist: eleven trained families of logic —
propositional rules, syllogism chains, temporal/interval reasoning,
quantities and proportions, inclusion–exclusion counting, evidence
vetting, defaults and exceptions, board tactics, invariants and
reachability, proof patterns, and two-family composition chains of
the above. English in, through a controlled-language translator that
drops what it can't carry, with a stated reason, rather than
guessing. Constraint-satisfaction puzzles are its known weak family
(~50%). It is not a chatbot; vocabulary is bounded by its dictionary.
Inside the box: calibrated. Outside the box: the verdict chip and
confidence numbers are the warning instruments.
RMSNorm (eps 1e-6) + per-head RMS QK-norm before RoPE
head
untied
precision
fp32 safetensors (~101 MB); runs on CPU
The input is not text bytes: words are concept bytes — a
dictionary-defined encoding where one byte is (roughly) one lemma and
grammatical features are explicit feature bytes. The dictionary ships
with the runtime as four readable TSVs.
Training recipe (the part that mattered)
Two phases, each sized to exactly 10 encoded bytes per parameter
("neochinchilla"), one epoch each, on one RTX 4070 Ti SUPER:
Logic school — 253.8 MB of synthetic documents from
oracle-checked generators, balanced to 25% each of
YES / NO / UNKNOWN / OPEN per pattern, so no constant responder
can beat chance; half single-family documents, half two-family
composition chains. Every document seen exactly once.
Association — 90% curated prose + everyday sentences, with 10%
freshly generated logic interspersed throughout (disjointness
from phase 1 proven by hash, not hoped from seeds). Held-out logic
loss improved through this phase: 0.1769 → 0.1712.
The measured headline: the same architecture trained on half the
ration with a two-way corpus scored 35.9% on the same probe.
Data ration and corpus design were the levers; parameters were
not. Scale is for fluency.
read_weights.py — explore the learned concept space by name
train_25m.py — the unabridged training script = the model
definition (set ZOFIAE_MODEL=25m, the default); --resume takes
this repo's model.safetensors directly — continue training here
export_talkit_assets.py — turn a training checkpoint into a
Talkit model folder (TALKIT_ASSETS_DIR, see the pretrain section)
LICENSE.md — Holley Community License 1.0 (worldwide text)
License
Free for noncommercial use — research, learning, personal
projects, evaluation, charities, schools, public institutions.
Commercial use requires a license (travis@tntholley.com), with a
published permanent cap: never more than 6% of gross revenue
attributable to the licensed use. Using the materials constitutes
acceptance of the terms — see LICENSE.md.
Citation
bibtex
1@software{holley2026strawweight,
2 author = {Holley, Travis E.},
3 title = {ZophiaE Strawweight: a 25M-parameter calibrated
4 logical-verdict engine on concept bytes},
5 year = {2026},
6 url = {https://github.com/Laninthalesdran/Concept-as-Byte},
7 note = {Weights: huggingface.co/tntholley. Patents pending.}
8}
TNT Holley, Inc. · Fort Mill, South Carolina · Patent pending.
