This model is a DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct, trained on the TACO-verified dataset for code generation tasks.
Training was performed using 4-bit QLoRA (nf4 quantization with bf16 compute) and a separate frozen reference model.
After training, the LoRA adapters were merged into the base model using PEFT.
The final checkpoint is fully merged and stored in safetensors format (bf16 precision) and does not require LoRA or quantization at inference time.
Base Model
Base model: Qwen/Qwen2.5-Coder-7B-Instruct
Precision: bfloat16 (bf16)
Training Method
Optimization method: Direct Preference Optimization (DPO)
Library: Hugging Face TRL (DPOTrainer)
Reference model: Separate frozen model used for DPO training
Quantization: 4-bit QLoRA (nf4, bfloat16 compute)
Hugging Face dataset ID:likaixin/TACO-verified
Train split ratio: 0.9
Random seed: 42
Epochs: 2
Batch size: 4
Learning rate: 3e-5
LoRA Configuration
Target modules: q_proj, k_proj, v_proj, o_proj
Rank (r): 16
Alpha: 32
Dropout: 0.05
Merging Procedure
The model was trained using LoRA adapters and then merged into the base model using PEFT
