This was primarily an experiment to see how a passthrough merge will respond to further finetuning, though this was done on a small dataset.
The goal was to make a "mid" sized model like Meta has released in the past and the merge method was inspired by mlabonne's Llama-3-120B.
The model was finetuned on 8192 context length and is likely reliable using RoPE up to 32k.
It still cannot do math reliably; neither can Llama-3-8B, and in my tests only Llama-3-70B passes basic arithmetic, but it is a better storywriter/RP than Llama-3-8B from some side by side testing I conducted.
Further finetuning this model or finetuning the base model on more samples is encouraged.
A small dataset was used to see how it affects performance. Originally I planned to do a larger dataset (196k samples), but wanted to start with a smaller one first to see how much the model improved with some additional finetuning.
Next steps would be finetuning on a larger dataset if through further testing, performance improvements are noticed.
Finetuning details
This is a QLoRA model and all modules were targeted.
1The following hyperparameters were used during training:2-learning_rate:1e-053-train_batch_size:14-eval_batch_size:15-seed:426-distributed_type: multi-GPU
7-num_devices:38-total_train_batch_size:39-total_eval_batch_size:310-optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0811-lr_scheduler_type: cosine
12-lr_scheduler_warmup_steps:2513-num_epochs:1
Optimizer paged_adamw_8bit and Deepspeed ZeRO 3 was used at a LR of 1e-5 using the cosine scheduler for 1 epoch on 3x3090s taking 4h 12m 13s total.
Sample packing and padding was disabled to reduce VRAM consumption significantly at the cost of speed.
