This model is a fine-tuned version of Qwen2.5-7B-Instruct optimized for conversational chat using the HH-RLHF dataset. It has been fine-tuned using LoRA and exported to the GGUF q4_k_m format for efficient inference on CPU and consumer-grade GPUs using llama.cpp or llama-cpp-python.

The fine-tuning process utilized the Unsloth library to significantly accelerate training while minimizing memory consumption.

Training Details

Dataset

The model was fine-tuned on a 1,000-sample subset of the Anthropic/hh-rlhf dataset to align it with helpful and harmless conversational patterns. The training focused on the chosen conversations ("chosen" split) formatted with Human and Assistant roles.

Hyperparameters and Configuration

Max Sequence Length: 2048 tokens
LoRA Rank (r): 16
LoRA Alpha: 16
LoRA Dropout: 0
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Batch Size: 2 (with 4 gradient accumulation steps, effective batch size = 8)
Learning Rate: 2e-4
Optimizer: adamw_8bit
Max Steps: 60
Warmup Steps: 5
Weight Decay: 0.01
Quantization: 4-bit (during training), exported to Q4_K_M GGUF

Usage

This model is packaged as a .gguf file (qwen2.5-7b-instruct.Q4_K_M.gguf), which can be loaded directly into UI frameworks like Streamlit using llama-cpp-python.

Setup Instructions

Install the required dependencies:

pip install llama-cpp-python streamlit

Run inference using Python and llama-cpp-python:

python

from llama_cpp import Llama

# Load the model (moves layers to GPU if n_gpu_layers=-1 and CUDA is available)
llm = Llama(
    model_path="qwen2.5-7b-instruct.Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=-1 
)

# Format the prompt using standard HH-RLHF format
prompt = "Explain the theory of relativity in simple terms."
formatted_prompt = f"Human: {prompt} \n\nAssistant:"

# Generate response
output = llm(
    formatted_prompt,
    max_tokens=256,
    temperature=0.7,
    stop=["Human:", "\n\n"],
    echo=False
)

print(output["choices"][0]["text"].strip())

Streamlit App Integration

This model was paired with a minimal Streamlit app (app.py) featuring an interactive chat UI. You can run it via:

streamlit run app.py

Prompt Format

For optimal performance and to maintain consistency with the fine-tuning process, use the following prompt format:

Human: {user_input} 

Assistant:

VijayShinde1996/vrs-sft-Qwen2_5-7B-it

Qwen2.5-7B-Instruct-GGUF-SFT

Model Description