You can deploy this model and start earning money today!

RichardErkhov/
KingNish_-_Reasoning-0.5b-exl2

Name: KingNish_-_Reasoning-0.5b-exl2
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Reasoning-0.5b - EXL2

Model creator: https://huggingface.co/KingNish/
Original model: https://huggingface.co/KingNish/Reasoning-0.5b/

Available sizes

Branch	Bits	Description
8_0	8.0	Maximum quality that ExLlamaV2 can produce, near unquantized performance.
6_5	6.5	Very similar to 8.0, good tradeoff of size vs performance, recommended.
5_0	5.0	Slightly lower quality vs 6.5, but usable
4_25	4.25	GPTQ equivalent bits per weight, slightly higher quality.
3_5	3.5	Lower quality, only use if you have to.

Download instructions

With git:

git clone --single-branch --branch 6_5 https://huggingface.co/KingNish_-_Reasoning-0.5b-exl2 Reasoning-0.5b-6_5

With huggingface hub:

pip3 install huggingface-hub

To download a specific branch, use the --revision parameter. For example, to download the 6.5 bpw branch: Linux:

huggingface-cli download KingNish_-_Reasoning-0.5b-exl2 --revision 6_5 --local-dir Reasoning-0.5b-6_5 --local-dir-use-symlinks False

Windows (which apparently doesn't like _ in folders sometimes?):

huggingface-cli download KingNish_-_Reasoning-0.5b-exl2 --revision 6_5 --local-dir Reasoning-0.5b-6.5 --local-dir-use-symlinks False

Original model description:

base_model: Qwen/Qwen2.5-0.5B-Instruct language:

en license: apache-2.0 datasets:
KingNish/reasoning-base-20k tags:
text-generation-inference
transformers
unsloth
qwen2
trl
sft
reasoning

Model Dexcription

It's First iteration of this model. For testing purpose its just trained on 10k rows. It performed very well than expected. It do first reasoning and than generate response on based on it but it do like o1. It do reasoning separately no special tokens or in response reasoning. Below is inference code.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

MAX_REASONING_TOKENS = 1024
MAX_RESPONSE_TOKENS = 512

model_name = "KingNish/Reasoning-0.5b"

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Which is greater 9.9 or 9.11 ??"
messages = [
    {"role": "user", "content": prompt}
]

# Generate reasoning
reasoning_template = tokenizer.apply_chat_template(messages, tokenize=False, add_reasoning_prompt=True)
reasoning_inputs = tokenizer(reasoning_template, return_tensors="pt").to(model.device)
reasoning_ids = model.generate(**reasoning_inputs, max_new_tokens=MAX_REASONING_TOKENS)
reasoning_output = tokenizer.decode(reasoning_ids[0, reasoning_inputs.input_ids.shape[1]:], skip_special_tokens=True)

# print("REASONING: " + reasoning_output)

# Generate answer
messages.append({"role": "reasoning", "content": reasoning_output})
response_template = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response_inputs = tokenizer(response_template, return_tensors="pt").to(model.device)
response_ids = model.generate(**response_inputs, max_new_tokens=MAX_RESPONSE_TOKENS)
response_output = tokenizer.decode(response_ids[0, response_inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("ANSWER: " + response_output)

Trained by: Nishith Jain
License: apache-2.0
Finetuned from model : Qwen/Qwen2.5-0.5B-Instruct
Dataset used : KingNish/reasoning-base-20k

This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

RichardErkhov/KingNish_-_Reasoning-0.5b-exl2

Available sizes

Download instructions

Original model description:

Model Dexcription

RichardErkhov/
KingNish_-_Reasoning-0.5b-exl2