This repo contains LoRA adapter created by aligning Tülu3 8B on the Anthropic HH-RLHF dataset using Direct Preference Optimization (DPO). It was trained as a series of models for studying DPO alignment.

Model details

Base model: allenai/Llama-3.1-Tulu-3-8B-SFT
Preference dataset: Anthropic/hh-rlhf
DPO beta: 0.01
Training framework: PEFT/LoRA

See the base model card for usage and chat template details.

Training hyperparameters

Epochs: 1
Batch size: 8
Learning rate: 1e-05
Learning rate scheduler: cosine
Learning rate warmup ratio: 0.1
Gradient accumulation: 2
LoRA:
- rank: 64
- alpha: 64
- dropout: 0.05
- target modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]

License

Citation

If this work was helpful, please cite:

TBA

jmajkutewicz/Llama-3.1-Tulu-3-8B-DPO_hh-rlhf

Tülu3 8B aligned with DPO on HH-RLHF with β=0.01

Model details

Training hyperparameters

License

Citation

jmajkutewicz/
Llama-3.1-Tulu-3-8B-DPO_hh-rlhf