Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

zephyr-7b-dpo_dataset-mix – AI Model by jmajkutewicz | AlphaNeural AI

You can deploy this model and start earning money today!

jmajkutewicz/
zephyr-7b-dpo_dataset-mix

like

0

text-generation

OpenAssistant/oasst1

Anthropic/hh-rlhf

HuggingFaceH4/ultrafeedback_binarized

PKU-Alignment/PKU-SafeRLHF

alignment-handbook/zephyr-7b-sft-full

Views

No views yet

Zephyr 7B SFT aligned with DPO on mix of datasets with β=0.01

This repo contains LoRA adapter created by aligning Zephyr 7B SFT using Direct Preference Optimization (DPO) on the mix all following datasets:

OpenAssistant Conversations Dataset (OASST1)
Anthropic HH-RLHF
UltraFeedback Binarized
PKU-SafeRLHF

It was trained as a series of models for studying DPO alignment.

Model details

Base model: alignment-handbook/zephyr-7b-sft-full
Preference datasets:
- OpenAssistant/oasst1
- Anthropic/hh-rlhf
- HuggingFaceH4/ultrafeedback_binarized
- PKU-Alignment/PKU-SafeRLHF
DPO beta: 0.01
Training framework: PEFT/LoRA

See the base model card for usage and chat template details.

Training hyperparameters

Epochs: 1
Batch size: 16
Learning rate: 5e-06
Learning rate scheduler: cosine
Learning rate warmup ratio: 0.1
Gradient accumulation: 2
LoRA:
- rank: 64
- alpha: 64
- dropout: 0.05
- target modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]

License

This adapter is released under the Apache License 2.0.

Citation

If this work was helpful, please cite:

TBA