Views
No views yet
| Parameter | Value |
|---|---|
| Base model | Qwen2.5-7B-Instruct |
| Quantization | 4-bit NF4 (QLoRA) |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| LoRA dropout | 0.05 |
| Epochs | 3 |
| Per-device batch size | 2 |
| Gradient accumulation | 8 (effective batch = 16) |
| Learning rate | 2e-4 |
| LR scheduler | Cosine |
| Warmup steps | 50 |
| Max sequence length | 512 |
| Optimizer | paged_adamw_8bit |
| Training regime | fp16 mixed precision |
| Epoch | Training Loss | Validation Loss |
|---|---|---|
| 1 | 0.6009 | 0.6724 |
| 2 | 0.4598 | 0.8021 |
| 3 | 0.4083 | 0.8582 |
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
3from peft import PeftModel
4
5BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
6LORA_PATH = "Devadripta/csf425-phase2-qwen-toolalpaca"
7
8bnb_config = BitsAndBytesConfig(
9 load_in_4bit=True,
10 bnb_4bit_quant_type="nf4",
11 bnb_4bit_compute_dtype=torch.float16
12)
13
14tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
15base_model = AutoModelForCausalLM.from_pretrained(
16 BASE_MODEL, quantization_config=bnb_config,
17 device_map="auto", trust_remote_code=True
18)
19model = PeftModel.from_pretrained(base_model, LORA_PATH)
20model.eval()
21
22prompt = "System: You are a data analysis agent.\nUser: What is the total revenue for 2022?\nAssistant:\n"
23inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
24outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
25print(tokenizer.decode(outputs, skip_special_tokens=True))