TinyGPT2-IT is an instruction-tuned variant of TinyGPT2 — a modern GPT architecture built from scratch using PyTorch. The base model was pretrained on ~6.7B tokens from OpenWebText, then supervised fine-tuned (SFT) on Stanford Alpaca's 52K instruction-response pairs.
The entire pipeline — pretraining, fine-tuning, and inference — runs on a single NVIDIA RTX 3070 Ti (8 GB VRAM).
This model uses a custom architecture and requires trust_remote_code=True.
>>> What is the capital of France?
The capital of France is Paris.
Explanation
>>> Explain what machine learning is in simple terms.
Machine learning is a branch of computer science that focuses on using algorithms to
identify patterns in data. These algorithms are used to analyze large amounts of data
and make predictions about future trends.
Creative
>>> Write a motivational quote.
"The only way to make a difference is to be bold and courageous."
Limitations
Small model — 95M parameters is far below production LLMs; expect factual errors, repetition, and limited reasoning.
Short context — 512 token window limits the length of conversations and documents.
Training data — pretrained on web text and fine-tuned on synthetic Alpaca data, which may contain biases or inaccuracies.
Not safety-aligned — no RLHF/DPO applied to this checkpoint; the model may produce harmful or inappropriate content.
