TinyTalk — human-like small talk on a microcontroller
TinyTalk is a ~8.3M-parameter (≈1.6M non-embedding) GPT-Neo chatbot built
to do one thing: hold short, friendly, human-sounding small-talk conversations
on low-end hardware that can't run a normal LLM — think an ESP32-S3.
It is the model embedded in the Cardputer AI
firmware, where it runs fully offline on the device in ~2 MB of flash after
Q4_0 quantization. This repository hosts the full-precision PyTorch /
safetensors weights so the model can be used, fine-tuned, or re-quantized on
its own.
What it's for
Most chat models assume a datacenter GPU. TinyTalk asks the opposite question:
how small can a model be and still feel like talking to someone? It trades
away knowledge, reasoning, and long context to fit on a microcontroller, keeping
only the ability to make warm, coherent small talk:
User: hey, how are you?
Bot: I am good! I played outside today. It was so much fun!
User: nice! what did you play?
Bot: I played with my ball. Do you want to play too?
Good fits: an offline conversational toy or companion on an ESP32 / handheld;
a teaching example of an end-to-end on-device LLM; a tiny base to fine-tune for
embedded chat. Not a fit: anything needing facts, instructions, reasoning,
or safety guarantees.
Fine-tune: ~70K filtered, simple-English dialogues from
allenai/SODA, reformatted as
User:/Bot: turns, mixed with a slice of
TinyStoriesInstruct.
Loss is masked to the bot replies / story bodies, so the model never trains on
producing the user's turns.
Prompt format
Trained on this exact format, with <|endoftext|> (token 50256) between
exchanges:
Kindergarten English only. Short, simple sentences.
No world knowledge. Factual questions get friendly confabulation.
Short memory. Trained/served with a tiny context (~80 tokens on device,
256 max). Not instruction-following, not safe for any production use.
A toy/educational model — interesting because it fits on a microcontroller,
not because it is good.
License & attribution
Released under CC BY 4.0, the binding term inherited from the SODA training
data. You must retain the following attributions:
Base model: TinyStories-Instruct-3M — Ronen Eldan & Yuanzhi Li,
TinyStories: How Small Can Language Models Be and Still Speak Coherent
English? (arXiv:2305.07759). Published without an explicit license tag; the
TinyStories dataset family is CDLA-Sharing-1.0, which places no restriction on
trained models.
Fine-tune data: SODA (CC BY 4.0) — Kim et al., SODA: Million-scale
Dialogue Distillation with Social Commonsense Contextualization
(arXiv:2212.10465); and TinyStoriesInstruct (CDLA-Sharing-1.0).
