You can deploy this model and start earning money today!

Putnam-AXIOM/
putnam-axiom-dataset-ICML-2025-522

Views

No views yet

Putnam AXIOM Dataset (ICML 2025 Version)

Note: for questions, feedback, bugs, etc. please open a Huggingface discussion here.

Dataset Summary

The Putnam AXIOM dataset is designed for evaluating large language models (LLMs) on advanced mathematical reasoning skills. It is based on challenging problems from the Putnam Mathematical Competition. This version contains 522 original problems prepared for the ICML 2025 submission.

This dataset includes:

Full Evaluation Set (522 problems): Complete set of problems
Test Set (372 problems): Set used for testing
Validation Set (150 problems): Set used for validation/development

Each problem includes:

Problem statement
Solution
Original problem (where applicable)
Answer type (e.g., numerical, proof)
Source and type of problem (e.g., Algebra, Calculus, Geometry)
Year (extracted from problem ID)

Supported Tasks and Leaderboards

Mathematical Reasoning: Evaluate mathematical reasoning and problem-solving skills.
Language Model Benchmarking: Use this dataset to benchmark performance of language models on advanced mathematical questions.

Languages

The dataset is presented in English.

Dataset Structure

Data Fields

year: The year of the competition (extracted from the problem ID).
id: Unique identifier for each problem.
problem: The problem statement.
solution: The solution or explanation for the problem.
answer_type: The expected type of answer (e.g., numerical, proof).
source: The origin of the problem (Putnam).
type: A description of the problem's mathematical topic (e.g., "Algebra Geometry").
original_problem: Original form of the problem, where applicable.
original_solution: Original solution to the problem, where applicable.
variation: Flag for variations (0 for all problems in this dataset as these are not variations).

Splits

Split	Description	Number of Problems
`full_eval`	Complete set of 522 problems	522
`test`	Test split	372
`val`	Validation/development split	150

Dataset Usage

python

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("Putnam-AXIOM/putnam-axiom-dataset-ICML-2025-522")

# Access each split
full_eval = dataset["full_eval"]
test = dataset["test"]
val = dataset["val"]

# Example usage: print the first problem from the full evaluation set
print(full_eval[0])

Citation

If you use this dataset, please cite it as follows:

bibtex

@article{putnam_axiom2025,
  title={Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning},
  author={Aryan Gulati and Brando Miranda and Eric Chen and Emily Xia and Kai Fronsdal and Bruno de Moraes Dumont and Sanmi Koyejo},
  journal={39th International Conference on Machine Learning (ICML 2025)},
  year={2025},
  note={Preprint available at: https://openreview.net/pdf?id=YXnwlZe0yf}
}

License

This dataset is licensed under the Apache 2.0.

Last updated: May 22, 2024