I passthrough merged base Tiny Llama Chat with itself, then fine-tuned with around 1/3 of Platypus dataset.
Observations:
It's smarter (I think?)
It sometimes throws "### Instruction:" line. This could be due to the platypus dataset, or the fact that I know jackshit about programming. You can add it to "custom stopping strings" in oobaboga.
It may be possible to train very specialized mini experts and merge them???
