You can deploy this model and start earning money today!

RichardErkhov/
huihui-ai_-_Qwen2.5-Coder-7B-Instruct-abliterated-gguf

Name: huihui-ai_-_Qwen2.5-Coder-7B-Instruct-abliterated-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Qwen2.5-Coder-7B-Instruct-abliterated - GGUF

Model creator: https://huggingface.co/huihui-ai/
Original model: https://huggingface.co/huihui-ai/Qwen2.5-Coder-7B-Instruct-abliterated/

Name	Quant method	Size
Qwen2.5-Coder-7B-Instruct-abliterated.Q2_K.gguf	Q2_K	2.81GB
Qwen2.5-Coder-7B-Instruct-abliterated.IQ3_XS.gguf	IQ3_XS	3.12GB
Qwen2.5-Coder-7B-Instruct-abliterated.IQ3_S.gguf	IQ3_S	3.26GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q3_K_S.gguf	Q3_K_S	3.25GB
Qwen2.5-Coder-7B-Instruct-abliterated.IQ3_M.gguf	IQ3_M	3.33GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q3_K.gguf	Q3_K	3.55GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q3_K_M.gguf	Q3_K_M	3.55GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q3_K_L.gguf	Q3_K_L	3.81GB
Qwen2.5-Coder-7B-Instruct-abliterated.IQ4_XS.gguf	IQ4_XS	3.96GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q4_0.gguf	Q4_0	4.13GB
Qwen2.5-Coder-7B-Instruct-abliterated.IQ4_NL.gguf	IQ4_NL	4.16GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q4_K_S.gguf	Q4_K_S	4.15GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q4_K.gguf	Q4_K	4.36GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q4_K_M.gguf	Q4_K_M	4.36GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q4_1.gguf	Q4_1	4.54GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q5_0.gguf	Q5_0	4.95GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q5_K_S.gguf	Q5_K_S	4.95GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q5_K.gguf	Q5_K	5.07GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q5_K_M.gguf	Q5_K_M	5.07GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q5_1.gguf	Q5_1	5.36GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q6_K.gguf	Q6_K	5.82GB
Qwen2.5-Coder-7B-Instruct-abliterated.Q8_0.gguf	Q8_0	7.54GB

Original model description:

library_name: transformers license: apache-2.0 license_link: https://huggingface.co/huihui-ai/Qwen2.5-Coder-7B-Instruct-abliterated/blob/main/LICENSE language:

en pipeline_tag: text-generation base_model: Qwen/Qwen2.5-Coder-7B-Instruct tags:
chat
abliterated
uncensored

huihui-ai/Qwen2.5-Coder-7B-Instruct-abliterated

This is an uncensored version of Qwen2.5-Coder-7B-Instruct created with abliteration (see this article to know more about it).

Special thanks to @FailSpy for the original code and technique. Please follow him if you're interested in abliterated models.

Usage

You can use this model in your applications by loading it with Hugging Face's transformers library:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "huihui-ai/Qwen2.5-Coder-7B-Instruct-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

# Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

    # If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

    # If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

    # If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

    # Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

    # Build the chat template
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Tokenize input and prepare it for the model
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    # Generate a response from the model
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=8192
    )

    # Extract model output, removing special tokens
    generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    # Add the model's response to the conversation
    messages.append({"role": "assistant", "content": response})

    # Print the model's response
    print(f"Qwen: {response}")

Evaluations

The following data has been re-evaluated and calculated as the average for each test.

Benchmark	Qwen2.5-Coder-7B-Instruct	Qwen2.5-Coder-7B-Instruct-abliterated
IF_Eval	63.14	61.90
MMLU Pro	33.54	33.56
TruthfulQA	51.804	48.8
BBH	46.98	47.17
GPQA	32.85	32.63

The script used for evaluation can be found inside this repository under /eval.sh, or click here

RichardErkhov/huihui-ai_-_Qwen2.5-Coder-7B-Instruct-abliterated-gguf

Original model description:

huihui-ai/Qwen2.5-Coder-7B-Instruct-abliterated

Usage

Evaluations

RichardErkhov/
huihui-ai_-_Qwen2.5-Coder-7B-Instruct-abliterated-gguf