This project aims to build a robust filter that detects malicious prompt injection attacks before it reaches the LLM.
The detector can detect a prompt injection attack in few-shot manner, with a fine-tuned Siamese network based on a pretrained DeBERTa model.
The detector can prevent prompt injection in the scenario where we have only a few examples of a novel injection attack. The experiment showed that this approach outperformed the existing cross-entropy-based fine-tuning.
