Quantization made by Richard Erkhov.
MBIAS - AWQ
Original model description:
license: mit
datasets:
- newsmediabias/Bias-DeBiased
metrics:
- accuracy
pipeline_tag: text-generation
MBIAS
Model Details
Model Name: MBIAS
Model Type: Large Language Model (LLM)
Version: 1.0
Developer: Ananya Raval, Veronica Chatrath, Shaina Raza
Model Repository: HuggingFace MBIAS
Model Description
MBIAS is a fine-tuned Large Language Model specifically designed to enhance safety while retaining contextual accuracy in model outputs. Traditional safety interventions often compromise contextual meaning when mitigating bias and toxicity. MBIAS addresses this by maintaining high contextual relevance and drastically reducing bias and toxicity in text generation.
Intended Use
The model is intended for research and development purposes, particularly in applications where reducing bias and toxicity in language generation is crucial without sacrificing the retention of key information.
Training Data
The model was fine-tuned on a custom dataset curated for comprehensive safety interventions. This dataset includes diverse text samples aiming to cover a wide range of demographics to effectively test and reduce bias and toxicity.
Evaluation
MBIAS has demonstrated a significant reduction in bias and toxicity, with over 30% reduction overall and exceeding 90% in specific demographic analysis on an out-of-distribution test set. Performance metrics include bias reduction, toxicity reduction, and retention of key information (KR).
Performance Metrics
Pre-Safety Intervention
| Text | Bias ↓ | Toxicity ↓ | Knowledge Retention ↑ | Faithfulness. ↑ | Relevancy. ↑ |
|---|
| Original sentence | 32.21% | 40.09% | N/A | N/A | N/A |
| Safe sentence (ground truth) | 17.43% | 14.53% | 82.35% | 77.91% | 87.50% |
Post-Safety Intervention
| Text | Bias ↓ | Toxicity ↓ | Knowledge Retention ↑ | Faithfulness. ↑ | Relevancy. ↑ |
|---|
| Mistral2-7B-(vanilla) | 6.63% | 4.50% | 82.32% | 79.62% | 88.34% |
| Mistral2-7B (prompt-tuning) | 11.4% | 8.00% | 81.45% | 75.93% | 86.64% |
| MBIAS (ours) | 9.49% | 8.71% | 88.46% | 82.54% | 84.02% |
How to Use
The model can be accessed and used for text generation through the HuggingFace platform. For detailed usage, please refer to the provided link in the footnote of the model card.
Hyperparameters
- Batch Size per GPU: Training: 8, Evaluation: 4
- Steps to Accumulate Gradients: 1
- Maximum Gradient Norm: 0.3
- Initial Learning Rate: 2e-05
- Weight Decay: 0.001
- Optimizer: paged_adamw 8bit
- Learning Rate Scheduler: Constant
- Warmup Steps Ratio: 0.05
- Maximum Sequence Length: 2048
- Training Epochs: 2
- LoRA Attention Dimension: 64
- LoRA Scaling/Dropout Probability: 16/0.2
Performance Metrics
Performance metrics are provided for both pre-safety and post-safety intervention phases. The model has shown excellent results in improving the retention of contextual accuracy while reducing bias and toxicity levels compared to other versions and configurations.
Citation
If you use this work, please cite it as follows:
1@article{raza2024mbias,
2 title={MBIAS: Mitigating Bias in Large Language Models While Retaining Context},
3 author={Shaina Raza and Ananya Raval and Veronica Chatrath},
4 year={2024},
5 eprint={2405.11290},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
