This model was trained using Supervised Fine-Tuning (SFT) with LoRA (Low-Rank Adaptation) and 4-bit quantization via bitsandbytes and Hugging Face's trl library.

📌 Model Details

Developed by: maghrane
Base Model: Qwen/Qwen3.5-0.8B
Model Type: Causal Language Model (Decoder-Only)
Fine-Tuning Technique: LoRA (PEFT) + QLoRA (4-bit NF4)
Dataset: maghrane/unsloth-qwen
Prompt Format: <s>[INST] {prompt} [/INST] {response} </s>
License: Apache-2.0

⚙️ Training Hyperparameters & Configuration

🔹 LoRA Configuration (PEFT)

Parameter	Value
LoRA Rank ($r$)	`64`
LoRA Alpha ($\alpha$)	`16`
LoRA Dropout	`0.1`
Bias	`none`
Target Modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `fc1`, `fc2`

🔹 Quantization & Precision (BitsAndBytes)

Parameter	Value
Quantization	4-bit (`nf4`)
Compute Dtype	`float16`
Double Quantization	`False`

🔹 SFT Training Arguments

Parameter	Value
Optimizer	`paged_adamw_32bit`
Learning Rate	`2e-4`
Learning Rate Scheduler	`constant`
Weight Decay	`0.001`
Warmup Ratio	`0.03`
Epochs	`1`
Per Device Batch Size	`4`
Gradient Accumulation	`1`
Max Gradient Norm	`0.3`

💻 How to Use & Run Inference

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# 1. Disable cuDNN algorithm search (resolves Qwen 3.5 1D convolution kernel lookup issue)
torch.backends.cudnn.enabled = False

base_model_name = "Qwen/Qwen3.5-0.8B"
adapter_model_name = "maghrane/Qwen-0.8B-unsloth"

# 2. Load Base Model & Adapter
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapter_model_name)

# 3. Re-enable KV Cache & set to evaluation mode
model.config.use_cache = True
model.eval()

# 4. Generate Response
prompt = "What is Unsloth and what can it do?"
formatted_prompt = f"<s>[INST] {prompt} [/INST]"
inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(generated_text)

📊 Results & Sample Inference

Input Prompt:

<s>[INST] What is Unsloth and what can it do? [/INST]

Model Output:

Unsloth lets you run and train AI models on your own local hardware via an open-source UI. It supports inference and training for 500+ models across text, audio, embedding, vision, and more. Key features include:

Inference: Search, download, and run models (GGUFs, LoRA adapters, safetensors), self-healing tool calling, auto parameter tuning, model export, and side-by-side output comparison.

Training: ~2x faster training with ~70% less VRAM, full/pre-training, 4-bit/16-bit/FP8 support, auto-dataset creation from PDF/CSV/DOCX, live observability, and efficient reinforcement learning (RL).

📜 Disclaimer

This model is fine-tuned for educational and documentation guidance purposes regarding local LLM fine-tuning and inference.

maghrane/Qwen-0.8B-unsloth-full

Qwen-0.8B-unsloth 🦥