I started from Memphis-CoT 3B, which used a novel iterative contrastive finetuning procedure to improve reasoning ability.
I directly finetuned it on these examples, using a MixCE loss with a mixing ratio of 0.5.
Finetuning on top of finetunes this way tends to lead to catastrophic forgetting - and indeed I observed significant degregation of the resultant model on e.g. GSM8K.
A common strategy to prevent catastrophic foregtting is weight averaging. In the LM community, 'merges' also utilize weight averaging, and spherical linear interpolation (SLERP) is considered to be superior to linear averaging. Accordingly, I used SLERP to average the resultant model back with the original Memphis-CoT model.
This resulted in a model that has learned from the new data, without completely forgetting what it has learned from the original Memphis-CoT training.
Benchmarks
This model performs significantly worse than Memphis-CoT on benchmarks, despite being better suited to chat and creative writing tasks. This is an expected tradeoff.
*5-shot, as performed automatically by LM Evaluation Harness bbh_cot_fewshot even with num_fewshot=0
Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.
