QWEN_7B_RLHF: Direct Preference Optimization (DPO / RLHF) Alignment of Qwen 2.5 7B
This repository contains the fine-tuned model card, source code, evaluation benchmarks, presentation slides, and adapter weights for Qwen 2.5 7B RLHF.
Aligned on 2x NVIDIA H200 NVL GPUs using Direct Preference Optimization (DPO) with LoRA adapter configuration ($r=16, lpha=32$) on Qwen attention and MLP layers (["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]) using bfloat16 precision.
Published to namanadep Hugging Face profile using NAMAN_HF_TOKEN.
🎯 Primary Project Highlights & Proof of Work
- Preference Dataset: Processed 10,000 binarized human preference pairs (
argilla/ultrafeedback-binarized-preferences-cleaned) contrasting high-quality, aligned, executive responses (chosen) against verbose, unstructured responses (rejected).
- Multi-GPU DPO Alignment: Fine-tuned Qwen 2.5 7B using
DPOTrainer from Hugging Face trl and peft LoRA ($r=16, lpha=32$) on 2x NVIDIA H200 NVL GPUs.
- 50-Prompt Empirical Evaluation: Evaluated Pre-RLHF Base Qwen 7B vs. Post-RLHF Aligned Qwen 7B across 50 technical benchmarks spanning Math Logic, Systems Code, Refusal Alignment, Software Architecture, and AI Theory.
- Key Finding: Achieved a 100% Executive Structured Alignment rate with explicit Risk & Refusal Guardrail Rationale and Production-Grade code blocks.
📂 Uploaded Artifacts & Project Inventory
docs/50_PROMPTS_QWEN_7B_BEFORE_VS_AFTER_RLHF_COMPARISON.md: Side-by-Side 50-Prompt Evaluation Report.docs/QWEN_7B_RLHF_PROOF_OF_WORK_PRESENTATION.pptx: First-Person Proof of Work PowerPoint Deck.docs/QWEN_7B_RLHF_PLAN.md: Fine-Tuning Strategy & Implementation Plan.src/: Complete PyTorch, PEFT, TRL DPO, Dataset Processing & Evaluation Source Code.data/50_prompts_rlhf_results.json: Raw Evaluation Transcripts and Execution Logs.
