This model is a fine-tuned version of
Qwen/Qwen2.5-32B-Instruct on the
OpenThoughts-114k dataset.
The dataset is derived by distilling DeepSeek-R1 using the
data pipeline available on github.
More info about the dataset can be found on the dataset card at
OpenThoughts-114k dataset.
The numbers reported in the table below are evaluated with our open-source tool
Evalchemy.
We are fully open-source. Our
model weights,
datasets,
data generation code,
evaluation code, and
training code are all publicly available.
We finetune
Qwen2.5-32B-Instruct
on
OpenThoughts-114k for
3 epochs with a 16k context length using
LlamaFactory.
Our
full training configuration
is provided in
our repository.
Training the 32B model on
OpenThoughts-114k
was done on AWS SageMaker with 8xH100 P5 nodes. On 4 nodes, this took around 90 hours.
Meanwhile, for training on
OpenThoughts-Unverified-173k,
we used 96 nodes of 4xA100 (64 GB per GPU), training took 30 hours, spending 11,520 A100 hours on the Leonardo Supercomputer.
@misc{openthoughts,
author = {Team, OpenThoughts},
month = jan,
title = {{Open Thoughts}},
howpublished = {https://open-thoughts.ai},
year = {2025}
}
