qwen2.5-coder-3b-unsloth-lora
This repository contains a LoRA adapter, not a full standalone model.
It was created by fine-tuning unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit for coding-assistance behavior on Google Colab using T4 GPU.
What This Model Is
This adapter is the first-stage coding-focused fine-tune in the project.
Training goal:
- improve structured coding responses
- improve instruction-following for programming tasks
- improve simple bug-fixing behavior
This adapter should be loaded on top of the base model:
- base model:
unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit
Dataset
This adapter was trained on a sampled subset of:
bigcode/self-oss-instruct-sc2-exec-filter-50k
Project training setup:
- sampled rows before filtering:
4000
- rows used after filtering:
3993
- max sequence length:
1024
- training steps:
250
Training Summary
This model was trained with supervised fine-tuning (SFT) using LoRA and 4-bit quantization.
Key setup:
- LoRA rank:
16
- LoRA alpha:
16
- LoRA dropout:
0
- batch size per device:
1
- gradient accumulation:
16
- learning rate:
1e-4
- optimizer:
adamw_8bit
- hardware: Google Colab
Tesla T4
Observed result:
- final training loss: about
0.6130
Intended Use
Use this adapter when you want:
- a lightweight coding assistant
- better structured code answers
- simple debugging help
- a PEFT adapter that runs on top of the Qwen2.5-Coder 3B base model
Limitations
This adapter is not a standalone merged model.
It also was not the strongest model in the later direct-answer benchmark on every prompt. It improved some focused coding-task behavior, but it should be understood as a practical low-resource experiment rather than a universally superior model.
How To Load
1import torch
2from peft import PeftModel
3from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
4
5BASE_MODEL = "Qwen/Qwen2.5-Coder-3B-Instruct"
6ADAPTER_MODEL = "M-Alkassem/qwen2.5-coder-3b-unsloth-lora"
7
8bnb_config = BitsAndBytesConfig(
9 load_in_4bit=True,
10 bnb_4bit_quant_type="nf4",
11 bnb_4bit_compute_dtype=torch.float16,
12 bnb_4bit_use_double_quant=True,
13)
14
15tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, use_fast=True)
16if tokenizer.pad_token is None:
17 tokenizer.pad_token = tokenizer.eos_token
18
19base_model = AutoModelForCausalLM.from_pretrained(
20 BASE_MODEL,
21 quantization_config=bnb_config,
22 torch_dtype=torch.float16,
23 device_map="auto",
24)
25
26model = PeftModel.from_pretrained(base_model, ADAPTER_MODEL)
27model.eval()
Example Prompt
prompt = "Debug this Python code and explain the bug: def is_even(n): return n % 2 == 1"
Project Context
This adapter is part of a larger two-stage project:
coding-focused adapter: this repository
agent-oriented continued adapter: M-Alkassem/qwen2.5-coder-3b-agent-v1
The later agent adapter was trained by continuing from this coding adapter.
References
