This model is a fine-tuned version of the Mistral-7B model, utilizing Direct Preference Optimization (DPO) to better align the model's responses with human preferences, specifically in a causal language modeling context.

Model Details

Model Description

Developed by: Dhruv Parthasarathy
Model type: Fine-tuned language model
Language(s) (NLP): English
License: MIT
Finetuned from model: Mistral-7B-Instruct-v2.0

Model Sources

Repository: https://huggingface.co/DhruvParth
Paper: Direct Preference Optimization (https://arxiv.org/abs/2305.18290)
Demo: (Will soon be made available)

Uses

This model is tailored for scenarios requiring alignment with human preferences in automated responses, suitable for applications in personalized chatbots, customer support, and other interactive services.

Training Details

Notebook

The fine-tuning process and the experiments were documented in a Jupyter Notebook, available here.

Training Configuration

LoRA Configuration

python

LoraConfig(
    r=8,
    lora_alpha=8,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=['k_proj', 'v_proj', 'q_proj', 'dense']
)

BitsAndBytes Configuration

python

BitsAndBytesConfig(
    load_in_4bit=True,
    llm_int8_threshold=6.0,
    llm_int8_has_fp16_weight=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

Training Device Setup

device_map = {"": 0}

Training Arguments

python

DPOConfig(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    max_steps=50,
    save_strategy="no",
    logging_steps=1,
    output_dir=new_model,
    optim="paged_adamw_32bit",
    warmup_steps=5,
)

DPO Trainer Setup

python

DPOTrainer(
    model,
    args=training_args,
    train_dataset=updated_train_dataset,
    tokenizer=tokenizer,
    peft_config=peft_config,
    beta=0.1,
    max_prompt_length=512,
    max_length=1024,
)

Evaluation

Details on the model's performance, evaluation protocols, and results will be provided as they become available.

Citation

If you use this model or dataset, please cite it as follows:

BibTeX:

bibtex

@misc{dhruvparth_mistral7b_dpo_2024,
  author = {Dhruv Parthasarathy},
  title = {Fine-tuning LLMs with Direct Preference Optimization},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://huggingface.co/DhruvParth/Mistral-7B-Instruct-v2.0-PairRM-DPO}
}

APA: Dhruv Parthasarathy. (2024). Fine-tuning LLMs with Direct Preference Optimization. GitHub repository, https://huggingface.co/DhruvParth/Mistral-7B-Instruct-v2.0-PairRM-DPO

For any queries or discussions regarding the project, please open an issue in the GitHub repository, post your comment in the community section, reach out to me via LinkedIn (https://www.linkedin.com/in/parthadhruv/) or contact me directly at parthasarathy.d@northeastern.edu.

DhruvParth/Mistral-7B-Instruct-v0.2-DPO-v0.1

Model Card for DhruvParth/Mistral-7B-Instruct-v2.0-PairRM-DPO

Model Details

Model Description

Model Sources

Uses

Training Details

Notebook

Training Configuration

LoRA Configuration

BitsAndBytes Configuration

Training Device Setup

Training Arguments

DPO Trainer Setup

Evaluation

Citation

DhruvParth/
Mistral-7B-Instruct-v0.2-DPO-v0.1