2 epochs
92 hours of training on a undervolted 3090 pulling 250 watts. My system configuration was getting 13.25 it/s and slowly dropped to 13.04 it/s at 2 epochs.
Dataset used is uncencored and filtered of Psmathur's wizard orca 72k instructions, ported into a alpaca chat format, and removed all "Input" strings. Trained on only instruction and output.
Preforms remarkably well works with the blokes gptq mini orca v2 gptq and allows the model to output 2048 tokens.
This adapter was trained to test how lora training could be used to expand context window with less compute.
The original model trained by "psmathur" was trained with "8x A100(80G) GPUs for around 13 Hours for cost of $195".
I'm hoping the bloke can merge this with the orignal model and upload a gptq version for everyone.
