Chirp-3b
Overview
Chirp-3b is a high-performing 3B parameter language model crafted by the Ozone Research team. Fine-tuned from a robust base model (Qwen2.5 3B Instruct), it was trained on 50 million tokens of distilled data from GPT-4o. This compact yet powerful model delivers exceptional results, outperforming expectations on benchmarks like MMLU Pro and IFEval.
Chirp-3b is an open-source effort to push the limits of what small-scale LLMs can achieve, making it a valuable tool for researchers and enthusiasts alike.
Key Features
- Parameters: 3 billion
- Training Data: 50M tokens distilled from GPT-4o
Benchmarks
Chirp-3b excels on rigorous evaluation datasets, showcasing its strength for a 3B model.
MMLU Pro
| Subject | Average Accuracy |
|---|
| Biology | 0.6234 |
| Business | 0.5032 |
| Chemistry | 0.3701 |
| Computer Science | 0.4268 |
| Economics | 0.5284 |
| Engineering | 0.3013 |
| Health | 0.3900 |
| History | 0.3885 |
| Law | 0.2252 |
| Math | 0.5736 |
| Other | 0.4145 |
| Philosophy | 0.3687 |
| Physics | 0.3995 |
| Psychology | 0.5589 |
| Overall Average | 0.4320 |
- Improvement: 9 points above the base model.
IFEval
- Score: 72%
- Improvement: 14% better than the base model.
More benchmarks are in the works and will be shared soon!
Download
Usage
Requirements
- Recommended GPU: 8 GB VRAM Minimum
Example
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "ozone-research/Chirp-01"
4tokenizer = AutoTokenizer.from_pretrained(model_name)
5model = AutoModelForCausalLM.from_pretrained(model_name)
6
7input_text = "What’s the future of AI?"
8inputs = tokenizer(input_text, return_tensors="pt")
9outputs = model.generate(**inputs, max_length=50)
10print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Future Work
The Ozone AI team is exploring additional models, including 2B and larger variants. Keep an eye out for upcoming releases!
Feedback
We’re eager for your input! Try Chirp-3b and let us know your thoughts, use cases, or ideas for improvement. Open an issue here or contact us via [contact method—update as needed].
Acknowledgments
A big thanks to the open-source community for driving projects like this forward. Chirp-3b is our contribution to making AI research more accessible.
