Model Card for Model ID

Model Details

• This model is a fine-tuned version of the meta-llama/Llama-3.2-1B-Instruct base model, adapted using the ORPO (Optimizing Reward and Preference Objectives) technique.
• Base Model: It builds upon the Llama-3.2-1B-Instruct model, (1 billion parameter instruction-following language model).
• Fine-Tuning Technique: The model was fine-tuned using ORPO. ORPO combines supervised fine-tuning with preference optimization.
• Training Data: It was trained on the mlabonne/orpo-dpo-mix-40k dataset, containing 44,245 examples of prompts, chosen answers, and rejected answers.
• Purpose: The model is designed to generate responses that are better aligned with human preferences while maintaining the general knowledge and capabilities of the base Llama 3 model.
• Efficient Fine-Tuning: LoRA (Low-Rank Adaptation) was used for efficient adaptation, allowing for faster training and smaller storage requirements.
• Capabilities: Model should follow instructions and generate responses that are more in line with human preferences compared to the base model.
• Evaluation: The model's performance was evaluated on the HellaSwag benchmark

Model Description

Model Sources [optional]

Uses

Direct Use

Downstream Use [optional]

• Chatbots
• Question-answering systems
• General text generation with enhanced preference alignment

Out-of-Scope Use

Bias, Risks, and Limitations

• Performance may vary on tasks outside the training distribution
• May inherit biases present in the base model and training data
• Limited to 1B parameters, which may impact performance on complex tasks

Recommendations

• Users should be aware of potential biases in model outputs
• Not suitable for critical decision-making without human oversight
• May generate plausible-sounding but incorrect information

Training Details

Training Data

• It contains 44,245 examples in the training split.
• Includes prompts, chosen answers, and rejected answers for each sample.
• Combines various high-quality DPO datasets. [More Information Needed]

Training Procedure

Preprocessing [optional]

Training Hyperparameters

• Learning Rate: 2e-5
• Batch Size: 4
• Gradient Accumulation Steps: 4
• Training Steps: 500
• Warmup Steps: 20
• LoRA Rank: 16
• LoRA Alpha: 32

Speeds, Sizes, Times [optional]

Evaluation

• Performance Level: The model achieves a raw accuracy of 45.16% and a normalized accuracy of 61.39% on the HellaSwag task.
• Confidence: The small standard errors (about 0.5% for both metrics) indicate that these results are fairly precise.
• Improvement over Random: Given that HellaSwag typically has 4 choices per question, a random baseline would achieve 25% accuracy. This model performs significantly better than random.
• Normalized vs. Raw Accuracy: The higher normalized accuracy (61.39% vs. 45.16%) suggests that the model performs better when accounting for task-specific challenges.
• Room for Improvement: While the performance is well above random, there's still significant room for improvement to reach human-level performance (which is typically above 95% on HellaSwag).

Summary

• Base Model: meta-llama/Llama-3.2-1B-Instruct
• Model Type: Causal Language Model
• Language: English

• This model is designed for tasks requiring improved alignment with human preferences, such as:
• Chatbots
• Question-answering systems
• General text generation with enhanced preference alignment

• Dataset: mlabonne/orpo-dpo-mix-40k
• Size: 44,245 examples
• Content: Prompts, chosen answers, and rejected answers

• This is a benchmark task designed to evaluate a model's commonsense reasoning and ability to complete scenarios logically.
• No specific filtering was applied to the test set.
• The evaluation was done in a zero-shot setting, where the model didn't receive any examples before making predictions.

• Performance Level: The model achieves a raw accuracy of 45.16% and a normalized accuracy of 61.39% on the HellaSwag task.
• Confidence: The small standard errors (about 0.5% for both metrics) indicate that these results are fairly precise.
• Improvement over Random: Given that HellaSwag typically has 4 choices per question, a random baseline would achieve 25% accuracy. This model performs significantly better than random.
• Normalized vs. Raw Accuracy: The higher normalized accuracy (61.39% vs. 45.16%) suggests that the model performs better when accounting for task-specific challenges.
• Room for Improvement: While the performance is well above random, there's still significant room for improvement to reach human-level performance (which is typically above 95% on HellaSwag).
• Metrics: a. acc (Accuracy): Value: 0.4516 (45.16%), Stderr: ± 0.0050 (0.50%), b. acc_norm (Normalized Accuracy): Value: 0.6139 (61.39%), Stderr: ± 0.0049 (0.49%)

Environmental Impact

• Hardware Type: A100
• Hours used: No comment
• Cloud Provider: Google Collab
• Compute Region: Sacramento, CA, US
• Framework: PyTorch

Technical Specifications [optional]

Model Card Author

Model Card Contact