This is an attempt to use Parameter Efficient Fine Tuning (PEFT) with Low Rank Adaptation (LoRA) for the ESM-2 model esm2_t6_8M_UR50D. The Github can be found here. The model is intended to serve as an RNA binding site predictor for protein sequences. However, there seems to be something going wrong either during training, or at inference, as the model seems to only predict non-binding sites. The problem is likely in the training setup. Any feedback, comments, or discussion would be greatly appreciated.