You can deploy this model and start earning money today!

RichardErkhov/
huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2

Name: huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Qwen2.5-Coder-3B-Instruct-abliterated - EXL2

Model creator: https://huggingface.co/huihui-ai/
Original model: https://huggingface.co/huihui-ai/Qwen2.5-Coder-3B-Instruct-abliterated/

Available sizes

Branch	Bits	Description
8_0	8.0	Maximum quality that ExLlamaV2 can produce, near unquantized performance.
6_5	6.5	Very similar to 8.0, good tradeoff of size vs performance, recommended.
5_0	5.0	Slightly lower quality vs 6.5, but usable
4_25	4.25	GPTQ equivalent bits per weight, slightly higher quality.
3_5	3.5	Lower quality, only use if you have to.

Download instructions

With git:

git clone --single-branch --branch 6_5 https://huggingface.co/huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2 Qwen2.5-Coder-3B-Instruct-abliterated-6_5

With huggingface hub:

pip3 install huggingface-hub

To download a specific branch, use the --revision parameter. For example, to download the 6.5 bpw branch: Linux:

huggingface-cli download huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2 --revision 6_5 --local-dir Qwen2.5-Coder-3B-Instruct-abliterated-6_5 --local-dir-use-symlinks False

Windows (which apparently doesn't like _ in folders sometimes?):

huggingface-cli download huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2 --revision 6_5 --local-dir Qwen2.5-Coder-3B-Instruct-abliterated-6.5 --local-dir-use-symlinks False

Original model description:

license: other license_name: qwen-research license_link: https://huggingface.co/huihui-ai/Qwen2.5-Coder-3B-Instruct-abliterate/blob/main/LICENSE language:

en base_model:
Qwen/Qwen2.5-Coder-3B-Instruct pipeline_tag: text-generation library_name: transformers tags:
code
codeqwen
chat
qwen
qwen-coder
abliterated
uncensored

huihui-ai/Qwen2.5-Code-3B-Instruct-abliterated

This is an uncensored version of Qwen/Qwen2.5-Coder-3B-Instruct created with abliteration (see remove-refusals-with-transformers to know more about it).

Qwen2.5-Coder uncensored version has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters.

If the desired result is not achieved, you can clear the conversation and try again.

ollama

You can use huihui_ai/qwen2.5-coder-abliterate:3b directly,

ollama run huihui_ai/qwen2.5-coder-abliterate:3b

Usage

You can use this model in your applications by loading it with Hugging Face's transformers library:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "huihui-ai/Qwen2.5-Code-3B-Instruct-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

# Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

    # If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

    # If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

    # If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

    # Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

    # Build the chat template
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Tokenize input and prepare it for the model
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    # Generate a response from the model
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=8192
    )

    # Extract model output, removing special tokens
    generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    # Add the model's response to the conversation
    messages.append({"role": "assistant", "content": response})

    # Print the model's response
    print(f"Qwen: {response}")

RichardErkhov/huihui-ai_-_Qwen2.5-Coder-3B-Instruct-abliterated-exl2