Slush is a two-stage model trained with high LoRA dropout, where stage 1 is a pretraining continuation on the base model, aimed at boosting the model's creativity and writing capabilities. This is then merged into the instruction tune model, and stage 2 is a fine tuning step on top of this to further enhance its roleplaying capabilities and/or to repair any damage caused in the stage 1 merge.
This is still early stage. As always, feedback is welcome, and begone if you demand perfection.
The second stage, like the Sunfall series, follows the Silly Tavern preset (Mistral V2 & V3, though V3-Tekken works fine), so ymmv in particular if you use some other tool and/or preset.
Parameter suggestions:
I did all my testing with temp 1, min-p 0.1, DRY 0.8.
Training details:
Stage 1 (continued pretraining)
Target: mistralai/Mistral-Nemo-Base-2407 (resulting LoRA merged into mistralai/Mistral-Nemo-Instruct-2407)
