from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "DinoStackAI/Qwen3-8b-lora-telco-dpr")
tokenizer = AutoTokenizer.from_pretrained("DinoStackAI/Qwen3-8b-lora-telco-dpr")

Load with vLLM (LoRA)

python

from vllm import LLM
from vllm.lora.request import LoRARequest

llm = LLM(
    model="Qwen/Qwen3-8B",
    enable_lora=True,
    max_lora_rank=16,
)
outputs = llm.generate(
    prompts,
    lora_request=LoRARequest("telco-dpr", 1, "DinoStackAI/Qwen3-8b-lora-telco-dpr"),
)

Use this adapter with scripts/generation/run_rag_generation.py --lora-path DinoStackAI/Qwen3-8b-lora-telco-dpr.

Training details

Base model: Qwen/Qwen3-8B
Fine-tuning dataset: DinoStackAI/telco-dpr-rag
Method: LoRA (r=16, lora_alpha=32, lora_dropout=0.05)
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Loss: SFT with completion-only masking (assistant_only_loss=True)
Best checkpoint selection: dev eval_loss