michel-nano is an ultra-tiny ~6 million parameter base language model trained on 1.14 billion tokens. It was created by merging two intermediate training checkpoints using
mergekit to combine their strengths into a single, superior model.
This model is a 50/50 SLERP (Spherical Linear Interpolation) merge of two checkpoints from the same training run:
By merging them, this model inherits the best traits of both—achieving lower WikiText perplexity and higher BLiMP/ARC-E scores than either parent checkpoint individually.
The tokenizer is a basic Byte-Pair Encoding (BPE) tokenizer trained from scratch on a subset of 100,000 samples drawn from the same data mixture. It features a compact vocabulary size of 6,000 plus chatml special tokens for future finetuning.
As a base model that has not undergone any instruction tuning or alignment, michel-nano is very limited in its raw conversational abilities. It is best suited as a lightweight foundation for fine-tuning on specific downstream tasks. Its small footprint and strong grammatical foundation make it highly adaptable for applications such as:
