This is Doctor-Shotgun's LimaRP-zloss fine tune applied directly to mistralai/Mixtral-8x7B-Instruct-v0.1, as opposed to the base model from which the qLORA was trained (Mixtral-8X7B-v0.1). Traditionally a tune is applied to the base model then merged to Instruct); we don't do that here. We call these "uphill" models since the qLORA is pushed up the hill without an intermediate merge. This creates a more stable model which can be used as part of a merge of Instruct-based models.
Our use case for this is to apply a very minimal density and weight of LimaRP (which has lots of model-ism pseudo-slop) as part of a more nuanced, more instruction-following merge with "uphill" variants of our proprietary fine tunes (crush/crest/clash)
