[!NOTE]
The Alpaca format will generally work, but I recommend trying my SillyTavern settings preset and rules-lorebook for best results. See the How to run section below for details.
The model has a context limit of 32k tokens. However, the quality of responses from any small-to-medium model begins to decline after 16k tokens, with more rapid degradation beyond 21k tokens. I recommend using 21k tokens as the maximum for optimal performance.
If you remove the --local-dir-use-symlinks False parameter, the files will instead be stored in the central Hugging Face cache directory (default location on Linux is: ~/.cache/huggingface), and symlinks will be added to the specified --local-dir, pointing to their real location in the cache. This allows for interrupted downloads to be resumed, and allows you to quickly clone the repo to multiple places on disk without triggering a download again. The downside, and the reason why I don't list that as the default option, is that the files are then hidden away in a cache folder and it's harder to know where your disk space is being used, and to clear it up if/when you want to remove a download model.
The cache location can be changed with the HF_HOME environment variable, and/or the --cache-dir parameter to huggingface-cli.
For GGUF - please just grab KoboldCpp. Set GPU layers to 33 (depends on VRAM/model quant), Context to 20k + flash attention + KV cache 4bit + Low VRAM if you only have 6GB VRAM, and you're good to go.
Next:
Now grab her last versions of rules and formatting for ST (use this to install ST if you haven't already)
Import these files into ST and select them. In Active World(s) for all chats set rules lorebook.
If you are using Vectorization Source, set rules to Vectorized.
Set Start Reply With manually if you want to have planning. If you don't - you need to edit Prompt Content and the rules lorebook by removing everything about it.
<npc-planning>
-
(Note: nothing should be written after the dash.)
What should a good working setup look like? Something like this: Planning (thinking) with a few short bullet points about what the NPC should do.
image/png
What if I have a mess in the response? Look at the card's Advanced Definitions and move Main Prompt, Post-History Instructions, and Character's Note to Description. Don't forget to have properly formatted Examples of dialogue for ST (some cards from web chat platforms have a mess). The smaller the model = the more demanding it is regarding clean prompt formatting.
image/png
Treat my rules as an example. Everyone has their own taste for how RP should look. For example - I think it's bad tone to use second person view for narration, and it make models to impersonate user more. As a result, cards with it should be rewritten in third person if using rules without editing.
Why planning instead of just using reasoning? There are many reasons, but the main one - pure reasoning tends to overthink things, and it's less controllable and more error-prone.
Can this setup work with other models? Yes if they smarter then 7b and not overcooked (12b nemo works fine for me)
Role-Play Rules is so big... You have 20k context -_-. Again feel free to edit.
[!TIP]
Get the latest version of rules and ST settings presets, or if you have questions, feel free to ask on my Discord
here.
on my AI-related Discord server for feedback, questions, and other stuff.
