finnianx/

michel-micro

An ultra-tiny 28 million parameter model trained on 2.6 billion tokens using a custom dataset mixture. Context length of 1024 tokens.

The tokenizer is a basic bpe tokenizer that was trained on a smaller subset of 80_000 samples of this same data mixture with a vocab size of 8000.

This model has not undergone any post-training.

This base model is best suited for fine-tuning on specific tasks. On its own, it is very limited, but it is a pretty flexible foundation for applications such as toxic comment detection or sentiment analysis.