Made with Exllamav2 0.1.8 with the default dataset.
I'm not sure how well it works with Text-Generation-WebUI considering that this model uses some unusual RoPE mechanics and I have no idea how TGW handles it.
For some reason this model worked extremely slow with my TGW install but was perfectly fine with TabbyAPI.
I recommend using TabbyAPI for this model. The model requires a decent Nvidia RTX card on Windows/Linux or a decent AMD GPU on Linux.
It requires to be fully loaded in GPU to work, so if your GPU has too small VRAM you should use
GGUF version instead.
If you have Nvidia GTX card you should also use GGUF instead.
This is an uncensored version of
NousResearch/Hermes-3-Llama-3.1-8B using lorablation.
You can see in the following example how Hermes 3 refuses to answer a legitimate question while the abliterated model complies:
See
this article to learn more about abliteration.
This model was merged using the
task arithmetic merge method using
NousResearch/Hermes-3-Llama-3.1-8B + Llama-3.1-8B-Instruct-abliterated-LORA as a base.
1base_model: NousResearch/Hermes-3-Llama-3.1-8B+Llama-3.1-8B-Instruct-abliterated-LORA
2dtype: bfloat16
3merge_method: task_arithmetic
4parameters:
5 normalize: false
6slices:
7- sources:
8 - layer_range: [0, 32]
9 model: NousResearch/Hermes-3-Llama-3.1-8B+Llama-3.1-8B-Instruct-abliterated-LORA
10 parameters:
11 weight: 1.0
1# Setup
2git clone https://github.com/arcee-ai/mergekit.git
3cd mergekit && pip install -e .
4pip install bitsandbytes
5
6# Extraction
7mergekit-extract-lora mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated meta-llama/Meta-Llama-3.1-8B-Instruct Llama-3.1-8B-Instruct-abliterated-LORA --rank=64
8
9# Merge using previous config
10mergekit-yaml config.yaml Hermes-3-Llama-3.1-8B-lorablated --allow-crimes --lora-merge-cache=./cache
