PiNanoLM-100M is the long-term foundation of the PiNanoLM ecosystem: a single, config-driven engine (pinanolm_core) powers every current and future variant (20M, 50M, 100M, 250M, Instruct, Code, Math, Security, Embed, Vision). It keeps the clean, from-scratch PyTorch architecture of the family and scales capacity (hidden 512, 16 layers, 16 heads, SwiGLU, 2048-token context) for meaningfully better generation quality than PiNanoLM-50M while remaining practical on edge devices after quantization.

The whole family reuses one model, training, inference, export and benchmark code path - no duplication.

Highlights

102,777,344 parameters (~103M), decoder-only GPT-style Transformer
SwiGLU feed-forward + RoPE + RMSNorm + weight tying
Flash attention via scaled_dot_product_attention (auto), manual fallback
KV-cache accelerated decoding + batched generation
2048-token context (2x PiNanoLM-50M)
Shared BPE tokenizer (vocab 32,768) - identical across the family
Full sampling controls: temperature, top-k, top-p, typical, repetition / presence / frequency penalties, streaming
CPU-only inference, dynamic INT8, TorchScript, ONNX, and GGUF export
Fully resumable training: AMP (BF16/FP16), grad accumulation, grad checkpointing, cosine LR, AdamW, grad clipping, early stopping, checkpoint rotation, TensorBoard + CSV + optional W&B, DDP-ready
Configurable dataset mixtures (FineWeb-Edu, TinyStories, Wikipedia, public-domain books) with dataset quality reports
Modular, typed, tested codebase (27 unit tests, PEP8, structured logging)

Architecture

Component	PiNanoLM-50M (V2)	PiNanoLM-100M (V3)
Parameters	49,972,608	102,777,344
Hidden size	384	512
Layers	12	16
Attention heads	12 (head_dim 32)	16 (head_dim 32)
FFN intermediate	2192	2816
FFN activation	SwiGLU	SwiGLU
Context length	1024	2048
KV cache	(added)	yes
Positional encoding	RoPE (theta=10000)	RoPE (theta=10000)
Normalization	RMSNorm (eps 1e-5)	RMSNorm (eps 1e-5)
Attention	flash (SDPA) + fallback	flash (SDPA) + fallback
Weight tying	Yes	Yes
Vocabulary	32,768 (BPE)	32,768 (BPE, same tokenizer)

The family scales by config alone (see pinanolm_core/variants.py):

Variant	Params (est.)
pinanolm-20m	19,798,272
pinanolm-50m	49,972,608
pinanolm-100m	102,777,344
pinanolm-250m	219,839,232

Repository layout

text

pinanolm-100m/
  pinanolm_core/      # SHARED engine: config, model, generation, data, utils, variants
  pinanolm_100m/      # 100M preset (Pinanolm100mConfig / Pinanolm100mForCausalLM)
  pinanolm_50m/       # 50M preset (backwards-compatible over shared engine)
  pinanolm_20m/       # 20M preset - V1 backwards-compatibility over shared engine
  models/             # re-export of the shared model engine + all presets
  tokenizer/          # tokenizer loader (shared family BPE)
  dataprep/          # dataset mixtures + quality report (thin re-export)
  preprocessing/      # text cleaning / tokenization helpers
  configuration/      # config + variant registry re-export
  utilities/          # logging, collate, checkpoint I/O
  training/           # train.py (AMP, grad accum, ckpt, rotation, early-stop, DDP)
  inference/          # generate.py CLI + programmatic Generator
  export/             # export_all.py (TorchScript, ONNX, INT8, GGUF)
  benchmark/          # benchmark.py (tok/s, latency, RAM, CPU, load time)
  benchmarking/       # re-export of the benchmark routine
  quantization/       # re-export of edge quantization exports
  evaluation/         # evaluate.py (val loss, perplexity, throughput, samples)
  scripts/            # train_tokenizer.py, preprocess.py, upload_hf.py
  tests/              # unit tests (27)
  examples/           # run_pipeline.py
  config.json  generation_config.json
  tokenizer.json  tokenizer_config.json  special_tokens_map.json
  requirements.txt  LICENSE  README.md  MODEL_CARD.md
  docs/  (TRAINING, INFERENCE, BENCHMARK, EDGE, FINETUNING, CONTRIBUTING)

Installation

bash

git clone https://huggingface.co/ismailtasdelen/pinanolm-100m
cd pinanolm-100m
pip install -r requirements.txt

Requires Python 3.12+ (developed/verified on 3.11/3.12), PyTorch >= 2.1. CPU-only; CUDA optional (AMP auto-enables BF16/FP16).

Quick start (Python)

python

import json, torch
from pinanolm_100m import Pinanolm100mConfig, Pinanolm100mForCausalLM
from tokenizers import Tokenizer
from safetensors.torch import load_model

cfg = Pinanolm100mConfig.from_dict(json.load(open("config.json")))
model = Pinanolm100mForCausalLM(cfg).eval()
load_model(model, "checkpoints/model.safetensors")
tok = Tokenizer.from_file("tokenizer.json")

ids = torch.tensor([tok.encode("The history of computing is").ids])
out = model.generate(ids, max_new_tokens=64, temperature=0.9, top_k=40,
                     top_p=0.9, typical_p=0.95, repetition_penalty=1.1)
print(tok.decode(out[0].tolist()))

Training

Fully resumable; resumes automatically from checkpoints/checkpoint_latest.pt.

bash

# 1. (tokenizer is reused from the family; retrain only if explicitly needed)
# 2. preprocess corpus -> token shards (shared pipeline, with quality report)
python scripts/preprocess.py --corpus-dir data/corpus \
    --tokenizer tokenizer.json --seq-len 2048 --out data/tokenized --quality-report
# 3. train
python training/train.py --config config.json --data data/tokenized \
    --out checkpoints --epochs 2 --batch-size 8 --grad-accum 8 \
    --precision bf16 --grad-checkpointing --report-quality

Features: mixed precision (BF16/FP16 on CUDA), gradient accumulation, gradient checkpointing, cosine LR + warmup, AdamW + weight decay, gradient clipping, validation loop, early stopping, checkpoint rotation, dataset quality report, TensorBoard + CSV logging, optional W&B, DDP-ready, automatic resume.

Dataset

The default pipeline uses permissively licensed public-domain books for reproducible offline training. The shared pinanolm_core.data module also supports configurable mixtures of Hugging Face streaming sources - FineWeb-Edu, TinyStories, Wikipedia, public-domain books - via scripts/preprocess.py --mix "local:0.5,fineweb-edu:0.5". For production pretraining, swap in FineWeb-Edu.

Inference

bash

python inference/generate.py --prompt "Explain HTTP." --max-new-tokens 128
python inference/generate.py --prompt "Once upon a time" --stream \
    --temperature 0.8 --top-k 40 --top-p 0.9 --typical-p 0.95 \
    --repetition-penalty 1.1 --presence-penalty 0.2

Sampling controls: temperature, top_k, top_p, typical_p (locally-typical sampling), repetition_penalty, presence_penalty, frequency_penalty, max_new_tokens, batched prompts (--prompts-file) and token streaming (--stream). Generation uses a KV cache for efficient autoregressive decode.

Evaluation

bash

python evaluation/evaluate.py --config config.json \
    --safetensors checkpoints/model.safetensors --out evaluation_report.json

Produces an automatic report (validation loss, perplexity, tokens/sec, latency, peak RAM, on-disk size, generation samples) as JSON + Markdown.

Raspberry Pi / edge optimization

bash

python export/export_all.py --config config.json \
    --safetensors checkpoints/model.safetensors --out export \
    --formats torchscript onnx int8 gguf --llama-cpp ~/llama.cpp

Produces:

TorchScript - export/model_torchscript.pt
ONNX (opset 17, dynamic axes) - export/model.onnx
Dynamic INT8 (QNNPACK on ARM) - export/model_int8.pt
GGUF - export/pinanolm-100m-f16.gguf (direct) + q8_0 / q4_k_m (via llama.cpp). PiNanoLM's RoPE+RMSNorm+SwiGLU+tied layout maps onto the llama architecture for convert_hf_to_gguf.py.

Target devices: Raspberry Pi 4, Raspberry Pi 5, Orange Pi, Rock Pi, ARM Linux, Apple Silicon, x86 Linux.

Benchmark

bash

python benchmark/benchmark.py --config config.json \
    --safetensors checkpoints/model.safetensors
python benchmark/benchmark.py --compare --variant pinanolm-50m --variant pinanolm-100m

Measures model size, load time, peak RAM, CPU utilization, latency (ms/token) and throughput (tokens/sec), and prints a Markdown comparison table (also saved to benchmark_result.json). Run on x86, RPi 4 and RPi 5 to compare across devices.

Tests

python -m unittest discover -s tests

27 tests: exact param count (20M/50M/100M), forward shape, loss computation & decrease, weight tying, SwiGLU activation, KV-cache == full-forward, flash-vs -manual attention agreement, batch inference, presence/frequency penalties, gradient checkpointing, typical sampling, tokenizer roundtrip, scheduler warmup/decay, and variant-registry scaling.

Backwards compatibility

pinanolm_20m re-exposes the original Pinanolm20mConfig / Pinanolm20mForCausalLM API on top of the shared engine (GELU FFN, ctx 512), so existing V1 code and the shared tokenizer keep working unchanged. pinanolm_50m is the V2 preset.

Future variants

The codebase is designed so future models require only a preset entry in pinanolm_core/variants.py: PiNanoLM-250M, PiNanoLM-Instruct, PiNanoLM-Code, PiNanoLM-Math, PiNanoLM-Security, PiNanoLM-Embed, PiNanoLM-Vision. The shared engine handles architecture, training, inference, export and benchmarking for all of them.

License

MIT - see LICENSE.

Citation

bibtex

@misc{pinanolm100m2026,
  title  = {PiNanoLM-100M: A Lightweight Transformer Foundation Model for Edge Devices},
  author = {Ismail Tasdelen and contributors},
  year   = {2026},
  url    = {https://huggingface.co/ismailtasdelen/pinanolm-100m}
}