We try achieving State-Of-The-Art result in pass@1 on the GSM8k Benchmarks. The A100 GPU used for this fine-tuning process is generously provided by Weights & Biases. I am thankful to Soumik Rakshit from team W&B for constant support in this integration. The experiment can be tracked using Weights & Biases here.

Preparing the dataset

AutoTrain Advanced expects your CSV custom dataset in a certain format to work properly. Your training file must contain a "text" column on which the training will be done. For best results, the "text" column should have data in the ### Human: Question?### Assistant: Answer. format. A great example for the kind of dataset AutoTrain Advanced expects would be timdettmers/openassistant-guanaco. However, if you observe the MetaMathQA dataset, there are 3 columns - "query", "response" and "type". We will preprocess this dataset by removing the "type" column and combining the content of the "query" and "response" columns under one "text" column with the ### Human: Query?### Assistant: Response. format. The resulting dataset is rishiraj/guanaco-style-metamath and it will be used for training.

Adjusting hyperparameters

AutoTrain Advanced comes with a host hyperparameters we can tune to get the best model. While the default hyperparameters are a great start for everyone, I made a few changes there that are suitable for our use case. Here are the hyperparameters I used:

learning_rate = 2e-5
num_epochs = 3
batch_size = 4
block_size = 1024
trainer = "sft"
warmup_ratio = 0.03
weight_decay = 0.
gradient_accumulation = 4
use_fp16 = True
use_peft = True
use_int4 = True
merge_adapter = True
lora_r = 16
lora_alpha = 32
lora_dropout = 0.05
logging_steps = 10
log = "wandb"

Results

Check out the W&B Report for a detailed overview of the finetuned model including its Benchmark scores on a variety of tests like the ARC, HellaSwag, MMLU, TruthfulQA. I also included a comparison with other open-source LLMs on GSM8k Pass@1 and MATH Pass@1.

Model Usage

Here's how you can run the model using the pipeline() function from 🤗 Transformers:

python

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="rishiraj/zephyr-math", torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Experiments

Model	GSM8k Pass@1	MATH Pass@1
MPT-7B	6.8	3.0
Falcon-7B	6.8	2.3
LLaMA-1-7B	11.0	2.9
LLaMA-2-7B	14.6	2.5
MPT-30B	15.2	3.1
LLaMA-1-13B	17.8	3.9
GPT-Neo-2.7B	19.5	--
Falcon-40B	19.6	2.5
Baichuan-chat-13B	23.9	--
Vicuna-v1.3-13B	27.6	--
LLaMA-2-13B	28.7	3.9
InternLM-7B	31.2	--
ChatGLM-2-6B	32.4	--
GPT-J-6B	34.9	--
LLaMA-1-33B	35.6	3.9
LLaMA-2-34B	42.2	6.24
RFT-7B	50.3	--
LLaMA-1-65B	50.9	10.6
Qwen-7B	51.6	--
WizardMath-7B	54.9	10.7
LLaMA-2-70B	56.8	13.5
WizardMath-13B	63.9	14.0
MAmmoTH-7B (COT)	50.5	10.4
MAmmoTH-7B (POT+COT)	53.6	31.5
Arithmo-Mistral-7B	74.7	25.3
MetaMath-7B	66.5	19.8
MetaMath-13B	72.3	22.4
🔥 Zephyr-Math-7B	??	??

Citation

bibtex

@software{acharya2023zephyrmath
  title = {Zephyr Math: Zephyr 7B Alpha Model Fine-tuned on MetaMathQA Dataset},
  author = {Rishiraj Acharya and Soumik Rakshit},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
  howpublished = {\url{https://huggingface.co/rishiraj/zephyr-math}},
}