Dataset: iamholmes/tiny-imdb
Split: train
Max sequence length: 512
Preprocessing: Truncation + padding to max_length; labels copied from input_ids

Training Procedure

LoRA Configuration

Hyperparameter	Value
`r`	8
`lora_alpha`	16
`lora_dropout`	0.05
`bias`	none
`task_type`	CAUSAL_LM
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

Training Hyperparameters

Hyperparameter	Value
Epochs	3
Per-device batch size	4
Gradient accumulation steps	4
Effective batch size	16
Learning rate	2e-4
Precision	bf16
Optimizer / scheduler	Trainer defaults

How to Use

Load the adapter

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3-0.6B"
adapter_id = "321samthakur/qwen3-0.6b-lora-finetuned"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "This movie was"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Merge adapters (optional)

python

merged = model.merge_and_unload()
merged.save_pretrained("./qwen3-0.6b-merged")
tokenizer.save_pretrained("./qwen3-0.6b-merged")

Intended Use

This adapter is a small experimental fine-tune on IMDb-style text. It is suitable for demos, learning PEFT/LoRA workflows, and further experimentation—not production use without additional evaluation.

Limitations

Trained on a small IMDb-derived corpus; outputs may reflect movie-review style and dataset biases.
No formal evaluation metrics reported yet.
LoRA adapters only; you still need the base Qwen/Qwen3-0.6B weights at inference time (unless you merge).

Framework Versions

PEFT 0.20.0
Transformers + Hugging Face Trainer
PyTorch with CUDA (RTX A6000 Pro)