SCoReLoRA is an innovative approach to fine-tuning language models using Low-Rank Adaptation (LoRA) combined with reinforcement learning techniques for self-correction. This method aims to improve the model's ability to generate more accurate and refined responses through a two-stage training process.
-
Stage I: The model is trained to generate initial responses and then correct them, minimizing the KL divergence between the base model and the fine-tuned model.
-
Stage II: The model is further trained using reinforcement learning techniques, with rewards based on the quality of self-corrections.
The training process utilizes shaped rewards and KL divergence to balance between improvement and staying close to the original model's behavior.
The implementation includes functions to evaluate the model's self-correction capabilities, measuring metrics such as:
