An 8.9M-parameter question-answering model that runs entirely offline on an
ESP32-S3 microcontroller, answering espresso questions a piece at a time at
roughly 50 ms per position.
This is the Barista model from the
esp32-ai project, which pairs it with a
portable C inference runtime that runs on the device.
What it does
One question in, one answer out. No chat history, no retrieval. The model
writes the answer itself; it does not select from a catalogue of canned replies.
Real output from the board:
Q: my espresso tastes really bitter, what should i change?
A: if the cup reads bitter, coarsen a step. if it reads sour, go finer instead.
which of those does your drink read as?
Q: how do i clean my machine
A: if the machine has never had maintenance, the manual is still where you
start. use the recommended cleaner and descaler, and improvise nothing.
whats your machine model?
Q: what grind size should i use for a light roast
A: when you need a procedure for a double drink or another coffee recipe, i
cannot provide it. i can help improve the espresso shot. what problem does
the shot have?
The third answer is the model declining a request outside what it was built to
handle, which is a trained behaviour rather than a filter.
The asymmetric vocabulary
This is the design point that separates Barista from a conventional small model.
It reads 8,057 input tokens so it can accept varied typing, and writes
only its 854-class output alphabet.
Two consequences follow:
The output head is 854 rows instead of 8,057, so it is 109K parameters rather
than a million. A full-vocabulary head would be the largest single matvec the
model runs per position; this makes it one of the smallest.
Words outside the 854 are not merely unlikely, they are unsayable. There
are no digit tokens at all, so the model cannot invent grind 2 steps finer.
This restriction is a property of the vocabulary, not of a post-filter.
Because the two alphabets differ, a sampled index is an output class, not a
token id. Feeding it back into the model requires a mapping, which is what
layout.json holds:
854 output classes
-> 135 reuse an existing BPE id
-> 719 map to rows appended above the BPE vocabulary
input vocabulary total: 7,338 BPE + 719 appended = 8,057
Without vocab.json and layout.json the weights are not usable: you could
score logits but not turn them into words, nor continue past the first one.
Why it fits on a microcontroller
8.9M parameters, of which only 1.58M are dense transformer core. The rest is a
Per-Layer Embedding table read one row per token straight from memory-mapped
flash.
component
params
share
lives in
PLE table
6,187,776
69%
flash, memory-mapped
token embedding
1,031,296
12%
flash, memory-mapped
dense transformer core
1,575,424
18%
flash, staged to PSRAM
output head, untied
109,312
1%
flash, staged to PSRAM
total
8,903,808
4.6 MB at int4
The constraint is fast memory, not total memory. The board has 8 MB of PSRAM but
only 512 KB of internal SRAM. PLE keeps the large tables in flash, where a
one-row-per-token read costs bandwidth rather than the scarce pool.
Architecture
architecture PLE
format version 1, TIED_HEAD clear
input vocab 8,057 embedding and PLE table rows
output vocab 854 logits the model produces
d_model 128
layers 6
heads 4
ffn_hidden 384
ple_dim 128
seq_len 128
rope_theta 10000.0
weights int4, group size 128
TIED_HEAD is clear, so the output head is its own tensor stored after
out_norm, not a view of the token embedding. The header states both vocabulary
sizes, so the runtime is never told separately how many logits to score.
Runtime placement
tier
holds
flash, memory-mapped
PLE table and token embedding
PSRAM
per-position core and the untied head, staged to int8 at boot; KV cache
SRAM
float scratch buffers and RMSNorm vectors, 20,940 B plus 20 norm vectors
Activations are quantized to int8 for each staged matvec, and per-layer matvecs
are split across both LX7 cores. At boot the firmware reports 44 staged tensors
and zero SRAM fallbacks.
Measured speed
The model emits one output piece at a time, and punctuation is a piece, so
throughput is quoted in both units. Over the eight-prompt benchmark set below,
253 pieces render as 213 readable words.
The firmware can stream answers to an OLED as well as to serial, and the panel is
redrawn once per piece, so both modes are measured.
pieces/s
readable words/s
ms per piece
serial only
16.60
13.97
60.25
with the OLED
11.26
9.48
88.83
The panel costs 28.58 ms per output piece. The display runs outside the model
forward path: a profiled serial-only run measured 49.6 ms per forward, and
enabling the OLED adds a framebuffer transfer per emitted piece. The generated
text is identical in both modes.
Measured on the board on 2026-08-02 with scripts/benchmark_device.py: eight
fixed prompts, two passes per mode, spread under 0.1%. Both modes ran the same
weights (fp=e602146b) and the same build switches apart from the display, and
the benchmark refuses to report a comparison otherwise.
A forward is one position: the prompt tokens plus one per emitted piece, so
wall-clock throughput falls as the question gets longer.
Files
file
what it is
model.bin
int4 weights and header, flashed to the model partition
tokenizer.json
canonical 7,338-entry ByteLevel BPE
vocab.json
the 854 output classes, in class-id order
layout.json
class id to input token id map, and the vocabulary layout
metadata.json
architecture, parameters, runtime placement, SHA-256 and size of the four assets above
LICENSE
MIT
Verify a download before trusting it:
bash
1shasum -a 256 model.bin
2# 1359a1cb74de4143d630c2c192990de814cd47255bcdfa9cc135f07ef0a39fc4
The firmware also prints an FNV-1a fingerprint of the mapped image at boot,
e602146b, which must match device_fingerprint_fnv1a in metadata.json.
The firmware's three generated C headers, the word tables and the packed BPE
encoder asset, are built from vocab.json, layout.json and tokenizer.json
by the source repository, so they are not distributed here.
Verification
Reference logits are not shipped in this bundle. Verification lives with the
runtime, in the source repository, and covers four distinct things:
runtime/host_verify/verify.c checks the exact int4, float-activation
path against PyTorch, to 1e-5.
runtime/host_verify/staging_verify.c checks int8 weight staging, scale
alignment, ranged matvec equivalence, platform hook dispatch, header
validation and the untied-head format branch.
runtime/host_verify/bpe_tokenizer_verify.c checks the BTK1 loader and the
ASCII encoder against a hand-built asset.
firmware/esp32_barista/tools/verify_tokenizer.py checks that the device
encoder produces the same ids as the reference tokenizer, and that every one
of the 135 reused mappings resolves to the id it claims.
The device path enables int8 activations and is therefore not bit-identical to
the host golden. It is validated separately: the same model.bin driven through
the same tables on host reproduces the device's answers word for word.
Usage
These weights are not usable on their own. The firmware also needs C headers
generated from vocab.json, layout.json and tokenizer.json, and it has to be
compiled and flashed alongside the model. The
esp32-ai repository does both steps:
bash
1scripts/fetch_model.sh barista # downloads and verifies these files2scripts/deploy.sh barista # generates headers, runs host gates, compiles, flashes
fetch_model.sh checks the assets above against a SHA-256 and byte size pinned
in the script, and cross-checks metadata.json against those same pins. It
installs nothing unless every check passes. deploy.sh never reaches
the network. Use deploy.sh rather than writing model.bin by hand: it
regenerates the tables the firmware compiles against, and writes both the model
and the firmware.
Questions arrive over USB serial, ASCII only, and answers stream back one output
class at a time.
Training data
A private question-and-answer corpus written for this project. It is not
redistributed here, and neither are the training checkpoints or the evaluation
sets.
What is distributed is the deployable binary and the inference assets needed to
run it. That boundary is deliberate: the weights are usable and verifiable
without the corpus, and the corpus is not part of the release.
The training checkpoint is not distributed. Its SHA-256 is recorded in
metadata.json so a given model.bin can be tied to the weights it came from.
Limitations
Espresso troubleshooting only. Outside that, it declines or answers
poorly. 1.58M dense parameters do not store facts.
854-word output vocabulary. It cannot say anything outside that set, by
construction. It cannot emit digit characters, or number words beyond the six
it includes: double, few, half, three, twice, two.
ASCII input only. Non-ASCII questions are refused rather than encoded
differently from the reference tokenizer.
128-token context, greedy decoding. No sampling, no history between
questions. The same question always produces the same answer.
It can still produce a degenerate tail on some inputs.
No published accuracy numbers. The evaluation sets influenced model selection
and are not held out, so quoting a score from them would overstate what is
known.
License
what
license
model weights (model.bin)
MIT
tokenizer (tokenizer.json)
MIT
vocabulary and layout (vocab.json, layout.json)
MIT
training corpus, checkpoints, evaluations
not distributed
The weights were trained from scratch and contain no third-party weights. MIT
covers what is in this repository; it grants nothing over the training corpus,
which is not published here.
Credits
The PLE design is reproduced from Google's published Gemma 3n Per-Layer
Embeddings work. No model, checkpoint or method here derives from it beyond the published
description.
