QLoRA adapter for Qwen/Qwen2.5-1.5B-Instruct that parses a natural-language weather question into a compact structured intent (JSON). It lifts full-object exact match from 65% (base, few-shot) to 99% (fine-tuned, zero-shot).

text

"will it rain in Paris this weekend?"
  -> {"location": "Paris", "date": "this_weekend", "metric": "rain", "time_of_day": null}

Schema

slot	values
`location`	place name, or `null`
`date`	`today`, `tomorrow`, `day_after_tomorrow`, `this_weekend`, `next_weekend`, `next_week`, a weekday, or `null`
`metric`	`temperature`, `rain`, `snow`, `wind`, `humidity`, `uv`, `cloud`, `general`
`time_of_day`	`morning`, `afternoon`, `evening`, `night`, or `null`

Results (held-out eval)

metric	base	finetuned
valid JSON	100.0%	100.0%
exact match	64.5%	98.6%
field accuracy	90.7%	99.7%
slot F1	0.894	0.996

Base = few-shot (fair baseline shown the JSON format); fine-tuned = zero-shot. Greedy decoding (temperature 0), identical prompt for both.

Training

Method: QLoRA (4-bit NF4, double-quant, bf16 compute)
LoRA: r=16, alpha=32, dropout=0.05, target = all linear layers (attn + MLP)
Effective batch 16, LR 0.0002 cosine, 1 epoch, max_length 512
Assistant-only loss (loss on the JSON completion only)
Trained locally on a single RTX 3050 Ti (4 GB)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "Nicholas55555/qwen2.5-1.5b-weather-intent")

sys = "You extract structured intent from weather questions. Return ONLY a JSON object..."
msgs = [{"role": "system", "content": sys},
        {"role": "user", "content": "do I need an umbrella in Denver tomorrow morning?"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))