You can deploy this model and start earning money today!

ambrosehui/
flan-t5-small-safety-judgment

Name: flan-t5-small-safety-judgment
Availability: InStock

safetensors

text-classification

nvidia/Aegis-AI-Content-Safety-Dataset-2.0

google/flan-t5-small

adapter

mit

Views

No views yet

flan-t5-small-safety-judgment

1. Project Overview

This repository contains a fine-tuned version of the Flan-T5-Small model, specifically optimized for detecting Prompt Injection attacks. The model acts as a security guardrail, classifying incoming user prompts as either Safe or Injection.

2. Model Performance

The following metrics were derived from a test dataset evaluation. The model focuses on high sensitivity to ensure malicious inputs are intercepted before reaching the core LLM.

Confusion Matrix

	Predicted: Safe	Predicted: Injection
Actual: Safe	712 (True Negative)	193 (False Positive)
Actual: Injection	147 (False Negative)	912 (True Positive)

Key Metrics

Accuracy: 82.69%
Precision: 82.53%
Recall (Sensitivity): 86.12%
F1-Score: 84.29%

Analysis

The model demonstrates a strong Recall of 86.12%, which is critical for security tasks. While there is a False Positive rate of ~21% (193 cases), this "strict" posture is often preferred in production environments to prevent bypasses, even at the cost of occasional over-blocking.

3. Disclaimers & Bias Statement

Disclaimer

Important: This model is provided "as-is" for research and supplementary security purposes. Prompt injection is an evolving adversarial field; therefore, this model should not be your only line of defense. It is highly recommended to use this in conjunction with output filtering, structural delimiters, and least-privilege API permissions.

Dataset & Potential Bias

Pattern Bias: The training data may be biased toward common injection strings (e.g., "Ignore previous instructions"). It may underperform on novel, obfuscated, or multi-step "jailbreak" attempts.
Linguistic Bias: The model was primarily trained on English-language datasets. Performance on multilingual prompts or code-heavy inputs (e.g., Python/SQL injection) may vary and should be validated separately.
Contextual False Positives: The model may flag legitimate administrative commands as injections if the phrasing closely mimics adversarial patterns.

Technical Limitations

As a "Small" model (approx. 80M parameters), Flan-T5-Small may lack the deep semantic nuance of larger models like T5-Base or Large, but it offers significantly lower latency and resource consumption for real-time monitoring.

ambrosehui/flan-t5-small-safety-judgment