Qwen2.5-7B-ODA-Mixture-500k is a supervised fine-tuned (SFT) model built on top of Qwen2.5-7B-Base, trained with ODA-Mixture-500k. This training set is curated by mixing top-performing open corpora selected via the OpenDataArena leaderboard, and refined through deduplication and benchmark decontamination, aiming to improve the model’s general capabilities across General, Math, Code, and Reasoning domains.

🧠 Model Summary

Base Model: Qwen/Qwen2.5-7B-Base
Training Data: OpenDataArena/ODA-Mixture-500k
Domain Coverage: General, Math, Code, Reasoning
Scale (selected training set): ~500K samples
Goal: Improve multi-domain reasoning and problem-solving ability while achieving stable overall performance gains.

⚙️ Training Data Curation Pipeline

ODA-Mixture-500k is built by following a single rule: trust the OpenDataArena leaderboard.

1️⃣ Data Collection

We utilize LIMO as the foundational anchor due to its exceptional sample efficiency on the ODA overall leaderboard. To scale up to 500K, we integrate the top-performing and efficient corpora from specific ODA domain leaderboards, including:

AM-Thinking-v1-Distilled-Math (Math domain)
AM-Thinking-v1-Distilled-code (Code domain)
math-gpt-4o-200k (General domain)
SYNTHETIC-2-SFT-verified (Reasoning domain)

2️⃣ Deduplication & Decontamination

We first perform exact deduplication over all questions to remove identical items, and then run benchmark decontamination to reduce evaluation leakage by removing overlaps with standard and competition benchmarks.

3️⃣ Data Selection

At the 500K scale, our priority is distributional coverage. We employ semantic clustering to partition the total data pool into distinct thematic clusters. Within each cluster, we perform uniform sampling to ensure the final mixture represents a broad and balanced spectrum of reasoning tasks, maximizing the model's generalization capabilities.

📚 Training Data Source Composition

Source	Count	Percentage
LIMO	817	0.16%
AM-Thinking-Distilled-math	150,244	29.67%
AM-Thinking-Distilled-code	150,252	29.67%
math-gpt-4o-200k	100,138	19.78%
SYNTHETIC-2-SFT-verified	104,913	20.72%

🧩 Data Format

The training data sample format is as follows (aligned with the dataset schema):

json

{
  "id": "unique_identifier",
  "source": "data source",
  "question": "textual question or instruction",
  "response": "textual response"
}

📈 Performance

Qwen2.5-7B-ODA-Mixture-500k is evaluated as an SFT model built on Qwen2.5-7B-Base across the full ODA benchmark suite spanning four domains:

General (DROP, IFEVAL, AGIEVAL, MMLU-Pro)
Math (GSM8K, MATH500, Omni-Math, OlympiadBench, AIME2024)
Code (HumanEval, MBPP, LCB (V5), HumanEval+)
Reasoning (ARC-C, BBH, CALM, KOR-BENCH).

We observe consistent improvements over the base checkpoint, with particularly strong gains on several benchmarks.

Leaderboard Performance Comparison. Best scores in **bold**, second-best underlined. Eff. denotes Data Efficiency.
Model / Training Data	Size	Eff.	General	Math	Code	Reasoning	AVG
Qwen2.5-7B-Base
Qwen2.5-7B-Base	-	-	51.4	39.8	50.1	42.7	46.0
OpenThoughts3-1.2M	1.2M	+0.011	45.5	71.8	67.0	54.3	59.6
OmniThought-0528	365k	+0.027	47.1	71.2	47.6	57.2	55.8
SYNTHETIC-2-SFT-verified	105k	+0.086	51.3	69.8	40.1	58.9	55.0
AM-Thinking-v1-Distilled-math	558k	+0.016	57.7	77.4	39.5	44.8	54.8
LIMO	817	+9.920	60.7	44.0	57.9	53.8	54.1
MiroMind-M1-SFT-719K	719k	+0.006	52.0	71.0	26.3	51.5	50.2
AM-Thinking-v1-Distilled-code	324k	+0.024	49.9	52.3	68.7	44.4	53.8
Light-R1-SFTData	79k	+0.084	55.5	64.4	38.8	51.9	52.7
ODA-Mixture-500k	500k	+0.039	63.4	72.8	66.7	59.6	65.6
ODA-Mixture-101k	101k	+0.149	56.8	71.2	64.4	51.5	61.0

🌐 About OpenDataArena

OpenDataArena is an open research platform dedicated to discovering, evaluating, and advancing high-quality datasets for AI post-training. It provides a transparent, data-centric ecosystem to support reproducible dataset evaluation and sharing.

Key Features:

🏆 Dataset Leaderboard — helps researchers identify the most valuable and high-quality datasets across different domains
📊 Detailed Evaluation Scores — provides comprehensive metrics to assess data quality, complexity, difficulty, etc.
🧰 Data Processing Toolkit — OpenDataArena-Tool offers an open-source pipeline for dataset curation and scoring.

🚀 Usage

Model repo: OpenDataArena/Qwen2.5-7B-ODA-Mixture-500k. Below is a minimal runnable example for loading and inference:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "OpenDataArena/Qwen2.5-7B-ODA-Mixture-500k"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", trust_remote_code=True)

messages = [
    {"role": "user", "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📚 Citation

If you use this model or its training data (ODA-Mixture-500k), please cite:

bibtex

@article{gao2025closing,
  title={Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets},
  author={Gao, Xin and Wang, Xiaoyang and Zhu, Yun and Cai, Mengzhang and He, Conghui and Wu, Lijun},
  journal={arXiv preprint arXiv:2601.09733},
  year={2025}
}

bibtex

@article{cai2025opendataarena,
  title={OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value},
  author={Cai, Mengzhang and Gao, Xin and Li, Yu and Lin, Honglin and Liu, Zheng and Pan, Zhuoshi and Pei, Qizhi and Shang, Xiaoran and Sun, Mengyuan and Tang, Zinan and others},
  journal={arXiv preprint arXiv:2512.14051},
  year={2025}
}