Model Card for mamba-2.8b-slimpj-OpenOrca_1ep
This is a finetune of mamba-2.8b-slimpj for instruction following using the OpenOrca dataset.
Model Details
Model Description
This is a finetune of the mamba reference model mamba-2.8b-slimpj from the paper
https://arxiv.org/abs/2312.00752 It has been fine-tuned for instruction following using the OpenOrca dataset and training for 1 epoch.
Model type: Mamba State Space Model (mamba_ssm)Finetuned from model: https://huggingface.co/state-spaces/mamba-2.8b-slimpj
Uses
This model is intended to evaluate fine-tuning results on mamba models.
Usage
Prompt structure
The prompt structure used in fine-tuning is alpaca format:
"### Human:\n%question%\n\n### AI response:\n%response%"
Training Details
Training Data
Training Procedure
Trained using text-generation-webui with code from the mamba_ssm pull request.
Training Hyperparameters
Training regime: Trained in bfloat16 with the following parameters:
{
"trained_model_name": "mamba-2.8b-slimpj-OpenOrc_1ep",
"save_steps": 500000.0,
"micro_batch_size": 4,
"batch_size": 128,
"epochs": 1.0,
"learning_rate": "3e-4",
"lr_scheduler_type": "linear",
"cutoff_len": 256,
"dataset": "OpenOrca",
"eval_dataset": "None",
"format": "openorca-format",
"warmup_steps": 100.0,
"optimizer": "paged_adamw_8bit",
"hard_cut_string": "\\n\\n\\n",
"add_eos_token": false,
"min_chars": 0.0,
}
Reported train_loss was 0.6762700151924311
Results
lm-evaluation-harness results for final model
mamba_ssm (pretrained=mamba-2.8b-slimpj-OpenOrca), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
Tasks Version Filter n-shot Metric Value Stderr arc_challenge 1 none 0 acc 0.2594 ± 0.0128 none 0 acc_norm 0.2935 ± 0.0133 arc_easy 1 none 0 acc 0.4390 ± 0.0102 none 0 acc_norm 0.4032 ± 0.0101 boolq 2 none 0 acc 0.5801 ± 0.0086 lambada_openai 1 none 0 perplexity 27.8582 ± 1.1183 none 0 acc 0.3683 ± 0.0067 openbookqa 1 none 0 acc 0.2500 ± 0.0194 none 0 acc_norm 0.3700 ± 0.0216 piqa 1 none 0 acc 0.6817 ± 0.0109 none 0 acc_norm 0.6839 ± 0.0108 winogrande 1 none 0 acc 0.5770 ± 0.0139
lm-evaluation-harness results after half epoch
mamba_ssm (pretrained=mamba-2.8b-slimpj-OpenOrca_1ep-checkpoints/checkpoint-500000), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
Tasks Version Filter n-shot Metric Value Stderr arc_challenge 1 none 0 acc 0.2602 ± 0.0128 none 0 acc_norm 0.2833 ± 0.0132 arc_easy 1 none 0 acc 0.4533 ± 0.0102 none 0 acc_norm 0.4125 ± 0.0101 boolq 2 none 0 acc 0.4095 ± 0.0086 lambada_openai 1 none 0 perplexity 30.4832 ± 1.2403 none 0 acc 0.3551 ± 0.0067 openbookqa 1 none 0 acc 0.2420 ± 0.0192 none 0 acc_norm 0.3640 ± 0.0215 piqa 1 none 0 acc 0.6812 ± 0.0109 none 0 acc_norm 0.6730 ± 0.0109 winogrande 1 none 0 acc 0.5588 ± 0.0140
Reference lm-evaluation-harness results for the base model mamba-2.8b-slimpj without fine-tuning
mamba_ssm (pretrained=mamba-2.8b-slimpj), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
Tasks Version Filter n-shot Metric Value Stderr arc_challenge 1 none 0 acc 0.3882 ± 0.0142 none 0 acc_norm 0.4155 ± 0.0144 arc_easy 1 none 0 acc 0.7264 ± 0.0091 none 0 acc_norm 0.6814 ± 0.0096 boolq 2 none 0 acc 0.7107 ± 0.0079 lambada_openai 1 none 0 perplexity 5.8770 ± 0.1881 none 0 acc 0.6427 ± 0.0067 openbookqa 1 none 0 acc 0.2860 ± 0.0202 none 0 acc_norm 0.3980 ± 0.0219 piqa 1 none 0 acc 0.7709 ± 0.0098 none 0 acc_norm 0.7813 ± 0.0096 winogrande 1 none 0 acc 0.6614 ± 0.0133
Summary
The models measured perplexity and accuracy got worse, but it's known that that can be an effect of fine-tuning. Perplexity and accuracy improved in the second half of the training, so it's likely that the inital worsening was caused by forcing a prompt structure onto the base model, which was trained only on unstructured text.
The answer quality as percieved by users is yet to be evaluated.
Environmental Impact
Hardware Type: RTX 3090Hours used: 118 
