I strongly believe that this should achieve better performance than the original, particularly in math and reasoning. Hopefully the MUSR and MATH Lvl 5 evaluations reflect this.

This is what I used to load the model using AutoClasses:

!pip

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

# Define the revision you want to use
revision = "main"  # or any other valid revision/branch name

# Load the config, model, and tokenizer with the specified revision
config = AutoConfig.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR", revision=revision)
# Use AutoModelForCausalLM instead of AutoModel for this model
model = AutoModelForCausalLM.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR", 
                                            revision=revision, 
                                            config=config,
                                            ignore_mismatched_sizes=True) # Add this line to ignore mismatched sizes
tokenizer = AutoTokenizer.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR", revision=revision)

Dataset Citation: @misc{numina_math_datasets, author = {Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu}, title = {NuminaMath TIR}, year = {2024}, publisher = {Numina}, journal = {Hugging Face repository}, howpublished = {\url{https://huggingface.co/AI-MO/NuminaMath-TIR}} }