Property	Value
Architecture	Causal decoder-only transformer (GPT-style)
Layers	30
Attention heads	8
Embedding dimension	128
Context length	64 tokens
Parameters	~12.35M
Activation	GELU
Normalization	LayerNorm
Tokenizer	GPT-2 BPE (50,304 vocab)

Training

Teensy-0 was trained from scratch on the OpenWebText corpus:

~9B tokens
20,000 iterations
batch size 12, gradient accumulation 40
AdamW, peak learning rate 6e-4 with warmup and decay
Best validation loss: 4.5432

The training run was produced with the Teensy training code, not by fine-tuning an existing model. The original data, notebooks, and training scripts are archived in the sibling teensy-1/ directory.

Files in this Repository

Path	Description
`checkpoints/teensy-0.pt`	Full PyTorch checkpoint (~148 MB). Contains model weights, optimizer state, and training metadata.
`exported/model.safetensors`	HuggingFace-compatible model weights only (~75 MB). Smaller and faster to download.
`exported/config.json`	Model hyperparameters for the safetensors export.
`model.py`	Native `TeensyLM` architecture and a NanoGPT-to-Teensy weight adapter.
`sample.py`	Generate text from the `.pt` checkpoint.
`repl.py`	Interactive streaming REPL for the `.pt` checkpoint.
`sample_hf.py`	Generate text from the exported `model.safetensors`.
`repl_hf.py`	Interactive streaming REPL for the exported `model.safetensors`.
`export_hf.py`	Re-export `teensy-0.pt` to `exported/model.safetensors` + `config.json`.

Installation

pip install -r requirements.txt

Requirements:

torch>=2.0
tiktoken>=0.5
numpy>=1.24
safetensors>=0.4

Device Notes

CPU is strongly recommended for inference. This checkpoint was trained with the PyTorch MPS backend, and inference on MPS can produce degraded output: garbled text, stray <|endoftext|> tokens, and invalid UTF-8 byte sequences. All inference scripts therefore default to cpu. Pass --device=mps explicitly if you want to use Apple Silicon GPU, but expect lower-quality output.

Usage

1. Interactive streaming REPL (`.pt` checkpoint)

python repl.py --device=cpu --dtype=float32

Type a prompt and Teensy will stream tokens back token-by-token. Generation stops automatically when the model emits the GPT-2 <|endoftext|> token. Type exit or press Ctrl+C to quit.

2. Sample from a prompt (`.pt` checkpoint)

bash

python sample.py --device=cpu --dtype=float32 \
  --start="Once upon a time" \
  --max_new_tokens=200 \
  --num_samples=3

3. Generate from the safetensors export

bash

python sample_hf.py --device=cpu --dtype=float32 \
  --prompt="Once upon a time" \
  --max_new_tokens=200

4. Interactive streaming REPL for the safetensors export

python repl_hf.py --device=cpu --dtype=float32

Streams tokens and stops automatically at <|endoftext|>.

4. Load in your own code

From the `.pt` checkpoint

python

import torch
from model import TeensyConfig, TeensyLM, adapt_nanogpt_weights

checkpoint = torch.load("checkpoints/teensy-0.pt", map_location="cpu")
cfg = TeensyConfig(**checkpoint["model_args"])
model = TeensyLM(cfg)
state_dict = adapt_nanogpt_weights(checkpoint["model"])
model.load_state_dict(state_dict)
model.eval()

From the safetensors export

python

import json
import torch
from safetensors.torch import load_file
from model import TeensyConfig, TeensyLM

with open("exported/config.json") as f:
    cfg = TeensyConfig(**json.load(f))

model = TeensyLM(cfg)
model.load_state_dict(load_file("exported/model.safetensors"))
model.eval()

5. Re-export to safetensors

python export_hf.py --out_dir=checkpoints --export_dir=exported

This writes exported/model.safetensors (~75 MB, ~49% smaller than the .pt checkpoint) and exported/config.json.

Downloading without Cloning the Whole Repo

If you only want the model files, you can use the huggingface_hub library:

pip install huggingface_hub

Download the `.pt` checkpoint

python

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="checkpoints/teensy-0.pt",
    local_dir="./teensy-0"
)
print(path)

Download the safetensors export

python

from huggingface_hub import hf_hub_download

config_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/config.json",
    local_dir="./teensy-0"
)
weights_path = hf_hub_download(
    repo_id="Teensy/teensy-0",
    filename="exported/model.safetensors",
    local_dir="./teensy-0"
)
print(config_path, weights_path)

Download everything

huggingface-cli download Teensy/teensy-0 --local-dir ./teensy-0

Repository Layout

teensy-0/
├── checkpoints/       # trained PyTorch checkpoints
├── exported/          # HuggingFace-compatible safetensors + config.json
├── model.py           # TeensyLM architecture
├── sample.py          # text generation script for .pt
├── repl.py            # interactive streaming REPL for .pt
├── sample_hf.py       # text generation script for the safetensors export
├── repl_hf.py         # interactive streaming REPL for the safetensors export
├── export_hf.py       # export checkpoint to safetensors
├── README.md
├── LICENSE
└── requirements.txt

`../colab/` — training & distillation toolkit (not part of this repo)

An experimental pipeline for the next Teensy variants (instruction tuning and tool calling) lives in the sibling colab/ directory, outside this repository, so the published model repo stays clean:

Path	Description
`common.py`	Model (adds RoPE/GQA options + special tokens in unused GPT-2 rows), data pipelines, training loop, tool-call eval, safetensors export
`01_continue_and_instruct.ipynb`	Colab: continue teensy-0 on FineWeb-Edu (context 64→512), then Dolly-15k instruction SFT
`02_scratch_fineweb_toolcalls.ipynb`	Colab: from-scratch FineWeb-Edu pretraining, then tool-call SFT on distilled data
`03_scratch_pure_distillation.ipynb`	Colab: from-scratch on fully synthetic teacher corpus + tool-call SFT
`generate_data.py`	Local distillation data generator (schemas → teacher → validated JSONL). Backends: LM Studio server or mlx-lm
`constrained.py`	Schema-constrained decoding (tool names / arg keys / enum values guaranteed valid)
`run_toolcall.py`	Tool-call inference CLI with constrained decoding
`test_*.py`	CPU test suites for all of the above
`build_notebooks.py`	Regenerates the three notebooks with syntax-checked cells

The original data, notebooks, and training scripts are archived in teensy-1/.

Limitations

This model is not instruction-tuned.
It performs next-token prediction only and will hallucinate or drift on open-ended prompts.
It has no tool-calling or chat-following capability without further fine-tuning.
Output quality is representative of a ~12M-parameter model trained on raw web text.
Maximum context length is 64 tokens.

Credits and License

Modified from NanoGPT by Andrej Karpathy.

Teensy/teensy-0

Teensy-0

Model Details

Training

Files in this Repository

Installation

Device Notes

Usage

1. Interactive streaming REPL (.pt checkpoint)

2. Sample from a prompt (.pt checkpoint)

3. Generate from the safetensors export

4. Interactive streaming REPL for the safetensors export

4. Load in your own code

From the .pt checkpoint

From the safetensors export

5. Re-export to safetensors

Downloading without Cloning the Whole Repo

Download the .pt checkpoint

Download the safetensors export

Download everything

Repository Layout

../colab/ — training & distillation toolkit (not part of this repo)

Limitations

Credits and License

Teensy/
teensy-0

1. Interactive streaming REPL (`.pt` checkpoint)

2. Sample from a prompt (`.pt` checkpoint)

From the `.pt` checkpoint

Download the `.pt` checkpoint

`../colab/` — training & distillation toolkit (not part of this repo)