A LoRA DPO fine-tune of the base Qwen/Qwen3.5-4B model on the dataset of harshavaishnav/DPO_Dataset_2, submitted to the LM Playschool Challenge.

Model: harshavaishnav/DPO
Base Model: Qwen/Qwen3.5-4B
Method: LoRA Direct Preference Optimization (DPO)

Headline Results

Model	clemscore	statscore
Qwen/Qwen3.5-4B (baseline)	37.33	52.07
This model	38.23	53.56
Δ	+0.9	+1.49

Evaluated using:

playpen eval --suite all

Per-game evaluation results are available in the qwen_val.json file, clem and static directories.

Training Methodology

This model was fine-tuned using Direct Preference Optimization (DPO) with LoRA adapters on the DPO dataset.

Main characteristics:

Base model: Qwen/Qwen3.5-4B
LoRA fine-tuning
Direct Preference Optimization (DPO)
Preference pair training (chosen/rejected responses)
One training epoch
Gradient checkpointing enabled
Mixed precision (bf16)
Optimizer: AdamW

Data Usage

Dataset:

harshavaishnav/DPO_Dataset_2

Tokenization:

Qwen chat template
Maximum sequence length: 1024

No external datasets were used.

Hyperparameters

Hyperparameter	Value
Base Model	Qwen/Qwen3.5-4B
Training Method	LoRA + DPO
LoRA Rank (r)	16
LoRA Alpha	32
LoRA Dropout	0.0
DPO Beta	0.1
Learning Rate	5e-5
LR Scheduler	Cosine
Warmup Ratio	0.1
Per Device Batch Size	1
Gradient Accumulation Steps	8
Effective Batch Size	8
Max Sequence Length	1024
Training Epochs	1
Precision	bf16
Gradient Checkpointing	Enabled
Optimizer	AdamW
Reference Model	Qwen/Qwen3.5-4B

Compute

Training Hardware:

1 NVIDIA RTX 5000 Ada Generation GPU

Training Time:

13hrs

Frameworks:

Transformers
PEFT
TRL
Unsloth

Evaluation command:

playpen eval --suite all

Design Decisions

The following design choices were made during training:

LoRA-based Direct Preference Optimization (DPO) instead of full-parameter fine-tuning.
Preference optimization using chosen/rejected response pairs.
Gradient checkpointing enabled for improved memory efficiency.
Cosine learning-rate scheduler with warmup.
bf16 mixed precision training.
AdamW optimizer used for stable optimization.

Limitations

Performance depends on the quality and diversity of the preference dataset and may not generalize to unrelated tasks.
The model is optimized to align with human preference data but may still generate incorrect or undesirable responses.
Long-context reasoning beyond the training sequence length may be limited.
The model may inherit biases from both the base model and the preference dataset.

License

This model inherits the Apache-2.0 license from the Qwen base model.