It's important to use their Presets for SillyTavern due the peculiar prompt formatting of this model. And here is the text-gen samplers preset for SillyTavern I used.
What does "Imatrix" mean?
It stands for Importance Matrix, a technique used to improve the quality of quantized models.
The Imatrix is calculated based on calibration data, and it helps determine the importance of different model activations during the quantization process. The idea is to preserve the most important information during quantization, which can help reduce the loss of model performance.
One of the benefits of using an Imatrix is that it can lead to better model performance, especially when the calibration data is diverse.
The new IQ3_S quant-option has shown to be better than the old Q3_K_S, so I added that instead of the later. Only supported in koboldcpp-1.59.1 or higher.
If you want any specific quantization to be added, feel free to ask.
Join the community on Discord to get early access to new models.
story writing on dreamgen.com
Prompting
The models use an extended version of ChatML.
<|im_start|>system
(Story description in the right format here)
(Typically consists of plot description, style description and characters)<|im_end|>
<|im_start|>user
(Your instruction on how the story should continue)<|im_end|>
<|im_start|>text names= Alice
(Continuation of the story from the Alice character)<|im_end|>
<|im_start|>text
(Continuation of the story from no character in particular (pure narration))<|im_end|>
<|im_start|>user
(Your instruction on how the story should continue)<|im_end|>
<|im_start|>text names= Bob
(Continuation of the story from the Bob character)<|im_end|>
The Opus V1 extension is the addition of the text role, and the addition / modification of role names.
Pay attention to the following:
The text messages can (but don't have to have) names, names are used to indicate the "active" character during role-play.
There can be multiple subsequent message with a text role, especially if names are involved.
There can be multiple names attached to a message.
The format for names is names= {{name[0]}}; {{name[1]}}, beware of the spaces after names= and after the ;. This spacing leads to most natural tokenization for the names.
While the main goal for the models is great story-writing and role-playing performance, the models are also capable of several writing related tasks as well as general assistance.
Here's how you can prompt the model for the following tasks
Input: A brief plot description and the desired number of chapters.
Output: A description for each chapter.
And more...
Sampling params
For story-writing and role-play, I recommend "Min P" based sampling with min_p in the range [0.01, 0.1] and with temperature in the range [0.5, 1.5], depending on your preferences. A good starting point would be min_p=0.1; temperature=0.8.
You may also benefit from setting presence, frequency and repetition penalties, especially at lower temperatures.
Dataset
The fine-tuning dataset consisted of ~100M tokens of steerable story-writing, role-playing, writing-assistant and general-assistant examples. Each example was up to 31000 tokens long.
All story-writing and role-playing examples were based on human-written text.
token count distribution
Running the model
The model is should be compatible with any software that supports the base model, but beware of prompting and tokenization.
Just like ChatML, just changed "assistant" to "text" role.
Known Issues
34B tokenization:
There seems to be a mismatch between the tokenizer of the base and fine-tuned model. It's unclear whether this also affected training, or whether it's just incorrectly saved tokenizer (you can see tokenizer.json was not saved (bug report)).
This affects BOS and EOS (which aren't really used by Yi) and the tokenization of the first input token.
Overall impact should be minor.
34B repetition:
The 34B sometimes gets stuck repeating the same word, or synonyms. This seems to be a common problem across various Yi 34B fine-tunes.
GGUF / Ooba:
The tokenization might be messed up. Some users reported that <|im_start|> and <|im_end|> are tokenized as multiple tokens.
License
This model is intended for personal use only, other use is not permitted.
