A 28.9M-parameter language model that runs entirely offline on an ESP32-S3
microcontroller, generating text at 9.88 tokens/second.
This is the TinyStories model from the
esp32-ai project, a demonstration of
Per-Layer Embeddings (PLE) on a microcontroller. It is not a
general-purpose chat model.
It is also not a transformers model. It is a raw binary for a small C
inference runtime that runs on the device.
What it does
Continues a story in the style of
TinyStories: simple
English at roughly a 3 to 4 year old's vocabulary. Given Once upon a time, it
writes a short children's story one token at a time, on the device.
It cannot answer questions, follow instructions, or hold a conversation. It was
trained from scratch on story text and nothing else.
Why it fits on a microcontroller
The model is 28.9M parameters but only 556K of them are dense transformer core.
The rest is a Per-Layer Embedding table, read one row per token straight from
memory-mapped flash.
component
params
share
lives in
PLE table
25,165,824
87%
flash, memory-mapped
token embedding, also the tied head
3,145,728
11%
flash, staged to PSRAM
dense transformer core
556,416
2%
flash, staged to PSRAM
total
28,867,968
14.9 MB at int4
The constraint this addresses is fast memory, not total memory. The board has
8 MB of PSRAM, so size alone is not the problem, but the ESP32-S3 has only
512 KB of internal SRAM. A conventional model of this width would spend that
budget on embedding tables it reads once per token. PLE moves those to flash and
leaves the fast pool for what is read constantly.
Architecture
architecture PLE
format version 1, TIED_HEAD
input vocab 32,768 stored embedding and PLE table rows
output vocab 25,353 logits the model produces
d_model 96
layers 6
heads 4
ffn_hidden 66
ple_dim 128
seq_len 256
rope_theta 10000.0
weights int4, group size 128
The two vocabulary sizes differ on purpose. The embedding and PLE table store
32,768 padded rows, but the tokenizer has 25,353 entries, so only those can ever
be produced or decoded. TIED_HEAD means the output head is the first 25,353
rows of the token embedding; tying does not require the row counts to match.
The header states both, so the runtime does not have to be told separately how
many logits to score.
Parameter counts above include all 32,768 stored rows, because the binary
stores them.
Runtime placement
tier
holds
flash, memory-mapped
PLE table and token embedding
PSRAM
per-position core and head, staged to int8 at boot; KV cache; logits (99 KiB)
SRAM
float scratch buffers and RMSNorm vectors, 29,320 B
Activations are quantized to int8 for each staged matvec. The head is split
across both LX7 cores and is PSRAM-bandwidth-bound. int8 activations cost
+0.0003 nats of validation cross-entropy over 32,768 predictions
(2.4793 to 2.4796, perplexity 11.93 / 11.94).
Result it demonstrates
Against a same-core, SRAM-fitting baseline at equal core parameters:
PLE wins by 0.098 nats, 2 seeds, +/-0.006, roughly 16x the seed noise
perplexity 12.58 to 11.41
the gain survives 4-bit post-training quantization, 2 seeds
Full ablations, including the vocab-4096 control where the edge shrinks to
+0.025 nats, are in
RESULTS.md.
Measured speed
value
compute
94.9 ms/token
attached serial
9.88 tok/s
Measured on the board with the runtime described above: 44 staged tensors,
29,320 B managed SRAM, 4.19 MB PSRAM, compiled at -O3.
Files
file
what it is
model.bin
int4 weights and header, flashed to the model partition
tokenizer.json
canonical 25,353-entry BPE, trained on the same TinyStories slice
metadata.json
architecture, parameters, runtime placement, SHA-256 of the model and tokenizer
LICENSE
MIT
Verify a download before trusting it:
bash
1shasum -a 256 model.bin
2# 1d8326c05c383ccfa615f5455575802817cb453dbc7ab28875d41a9dbb45477e
The firmware also prints an FNV-1a fingerprint of the mapped image at boot,
a9bdd778, which must match device_fingerprint_fnv1a in metadata.json.
The firmware's vocab.h is generated from tokenizer.json by the source
repository, so it is not distributed here.
Verification
Reference logits are not shipped in this bundle. Verification lives with the
runtime, in the source repository, and covers two distinct things:
runtime/host_verify/verify.c against golden.txt checks the exact int4,
float-activation path against PyTorch, to 1e-5.
runtime/host_verify/staging_verify.c checks int8 weight staging, scale alignment,
ranged matvec equivalence, platform hook dispatch, header validation and the
untied-head format branch.
The device path enables int8 activations and is therefore not bit-identical to
the host golden. It is validated separately for output quality and throughput,
by the perplexity figure above and by on-device measurement.
Usage
These weights are not usable on their own. The firmware also needs a decode
header generated from tokenizer.json, and it has to be compiled and flashed
alongside the model. The
esp32-ai repository does both steps:
bash
1scripts/fetch_model.sh tinystories # downloads and verifies these files2scripts/deploy.sh tinystories # generates the header, runs host gates, compiles, flashes
fetch_model.sh checks the assets above against a SHA-256 and byte size pinned
in the script, and cross-checks metadata.json against those same pins. It
installs nothing unless every check passes. deploy.sh never reaches the
network. Use deploy.sh rather than writing model.bin by hand: it regenerates
the decode table the firmware compiles against, and writes both the model and the
firmware.
Training data
The first 300 MB of
roneneldan/TinyStories.
The tokenizer was trained on that same slice. The dataset is not redistributed
in this repository.
Reproducibility is approximate. The preparation script downloads from the
dataset's main branch without pinning a revision and records no hash of the
raw slice, so a re-run reproduces the method rather than the same bytes.
Training sets no determinism flags, so retraining yields an equivalent model
rather than this file.
The training checkpoint, 110 MB, is not distributed. The deployable binary plus
the recipe is the public contract.
Limitations
Simple children's-story English only. No instruction following, no question
answering, no chat.
Will produce fluent nonsense outside its distribution. 556K dense parameters
do not store facts.
256-token context, greedy decoding.
Not bit-reproducible from the recipe, see above.
License
what
license
model weights (model.bin)
MIT
tokenizer (tokenizer.json)
MIT
training dataset (TinyStories)
CDLA-Sharing-1.0, not redistributed here
TinyStories is licensed under CDLA-Sharing-1.0 and is not redistributed here.
The model weights and tokenizer are released under MIT. They were trained from
scratch and contain no third-party weights.
Credits
The PLE design is reproduced from Google's published Gemma 3n Per-Layer
Embeddings work. No model, checkpoint or method here derives from it beyond the published
description.
