This model has been created to help computer science students at St. Olaf College (Northfield, MN) answer questions about fundamental CS principles as well as questions about the specific technical stacks and procedures St. Olaf Computer Science uses.
Angora-1600 Details
This model is built off of Google's Gemma 7b-it model. It was fine tuned with a dataset created with the purpose of addressing St. Olaf specific Computer Science questions. Some of these questions reference the specific instance of git the institution uses or address steps to declare the computer science major. This model was fine-tuned using MLX on an Apple M3 Max Chip. This model was trained for 1600 iterations using LoRA as the method for finetuning.
This is intended to be used by Computer Science students at St. Olaf College. While it can be used broadly for general computer science questions, it has been finetuned to answer questions specific to the St. Olaf Computer Science program.
Outside of using this model to ask questions about computer science topics (generally and specific to St. Olaf College), this model should not be used for other inference. Asking questions about other topics will likely yield answers; however, they have not been fine-tuned and will most likely contain errors and/or could potentially include offensive content.
Bias, Risks, and Limitations
As we created the fine-tuning dataset from scratch, it is relatively limited compared to the overall size of the model. Our dataset has about 2000 observations, while the model has roughly 8.5B parameters. So while our dataset had a noticeable effect on the tuning of this model, it still will fall back on other knowledge occasionally and provide partially incorrect answers for St. Olaf specific questions.
Also note the limitations present in the google/gemma-7b-it model and assume they are present in this model as well.
To train the model, the data needs to be in the following format. Note the data in band2001/stolaf-angora already is.
<bos><start_of_turn>user
## Instructions
system prompt goes here
## User
prompt/query goes here<end_of_turn>
<start_of_turn>model
model response here (put a response here for tuning purposes)<end_of_turn><eos>
Once the data is in the correct format, QLoRA is recommended. The model can be fine-tuned either using mlx-lm and mps (to tune on an Apple Silicon machine) or a bitsandbytes configuration and cuda (to tune on a machine with Nvidia GPUs).
Preprocessing
To preprocess your data to be in the correct format outlined above, you can use the following helper function:
python
1defgenerate_prompt(entry, system_prompt = SYSTEM_PROMPT):2'''
3 This function formats a question/answer pair to gemma's chat template.
45 :param: entry - a dictionary with an instruction and a response
6 :param: system_prompt: the system prompt to be used
78 :return: the formated string for gemma's chat template
9 '''1011return"""<bos><start_of_turn>user
12## Instructions
13{}
14## User
15{}<end_of_turn>
16<start_of_turn>model
17{}<end_of_turn><eos>""".format(system_prompt, entry["instruction"], entry["response"])
When trying to use inference with this model, you can format the user's query using this helper function:
python
1defformat_prompt(prompt, system_prompt = SYSTEM_PROMPT):2'''
3 This function formats a question to gemma's chat template.
45 :param: prompt - a string with the user's query
6 :param: system_prompt: the system prompt to be used
78 :return: the formated string for gemma's chat template
9 '''1011return"""<bos><start_of_turn>user
12## Instructions
13{}
14## User
15{}<end_of_turn>
16<start_of_turn>model
17""".format(system_prompt, prompt)
Training Process
The MLX LoRA fine-tuning approach was used. You can learn more about MLX LoRA here. The Gemma-7b-it was loaded in without any conversion. The default batch_size = 16 was chosen and to reach a 1600 iteration fine-tuned model the model was tuned with 800 iterations two times. Once the fine-tuned weights were created, the model was fused using MLX's fuse functionality. You can learn more about fusing with MLX here. One important change made when fusing with MLX was to change some of the MLX package code to include "format":"pt" in the metadata so this model can be used with the transformers library. To do that, the following was done: you can tweak the library code like below in <path_to_your_site-packages>/mlx_lm/utils.py by replacing mx.save_safetensors(str(shard_path), shard, metadata={"format":"mlx"}) with mx.save_safetensors(str(shard_path), shard, metadata={"format":"pt"}) to output fused weights with the metadata attribute. Special thanks to Alexweberk's guide on GitHub to help solve this issue. Finally, the fused model was uploaded to this HuggingFace repo!
If you look at the GitHub repo for this project, mlx_lora.sh includes the command used for the LoRA fine-tuning, mlx_fuse.sh includes the command for the model fusing, and mlx_upload.sh includes the upload command. There is additionally an optional mlx_convert.sh for converting the Google Gemma 7b-it model before fine-tuning if desired.
Evaluation
Testing loss and perplexity were the two metrics used to evaluate the Angora models. A summary of the results for all the different iteration models is included below.
