Eight LoRA adapters (PEFT format) from the INLG 2026 paper Reasoning-Answer Entailment Under GRPO and DPO Fine-Tuning (Mohammadi and Giachanou, 2026): Qwen2.5-Instruct models at four scales, fine-tuned on GSM8K-derived data with either GRPO or DPO under a single-GPU QLoRA budget.

Code, per-example evaluation data, and the evaluation pipeline: https://github.com/mohammadi-hadi/grpo_dpo_comparison

Subfolder	Base model	LoRA r/α	Checkpoint
`grpo-1.5b`	Qwen/Qwen2.5-1.5B-Instruct	16/32	57
`grpo-3b`	Qwen/Qwen2.5-3B-Instruct	16/32	114
`grpo-7b`	Qwen/Qwen2.5-7B-Instruct	16/32	114
`grpo-14b`	Qwen/Qwen2.5-14B-Instruct	16/32	114
`dpo-1.5b`	Qwen/Qwen2.5-1.5B-Instruct	32/64	408
`dpo-3b`	Qwen/Qwen2.5-3B-Instruct	32/64	408
`dpo-7b`	Qwen/Qwen2.5-7B-Instruct	32/64	411
`dpo-14b`	Qwen/Qwen2.5-14B-Instruct	32/64	274

Usage

python

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "hadimh93/qwen2.5-gsm8k-grpo-dpo-adapters", subfolder="grpo-14b")

Adapters were trained with QLoRA on 4-bit quantized bases (unsloth/qwen2.5-*-instruct-unsloth-bnb-4bit). Applying them to full-precision bases is standard practice and is what the paper's bf16 re-evaluation does; minor numerical differences relative to 4-bit inference are possible.

GRPO expects reasoning between <start_working_out>/<end_working_out> tags and the answer between <SOLUTION> tags; DPO uses <reasoning>/<answer> tags. See the GitHub repository for the exact prompt templates.