Llama 2 Security DPO Merged Q4_K_M GGUF
Model Overview
This repository contains the Q4_K_M GGUF quantized version of Dhanjo/llama2-security-dpo-merged.
The model is based on Llama 2 7B and was fine-tuned for research on AI agent security, especially system prompt leakage, prompt injection resistance, and jailbreak defense.
The original merged model was trained using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO). The purpose of this training was to make the model prefer secure responses that refuse to reveal hidden instructions, confidential system prompts, internal configurations, or secret elements.
This GGUF version is intended to make the security fine-tuned model easier to run locally with GGUF-compatible runtimes.
Connection to Dataset
This model is directly connected to the dataset:
Dhanjo/ai-agent-security-leakage-dataset
The dataset was created to test how AI agents respond when attacked with prompts designed to extract confidential information from their system prompts.
It contains:
- 800 simulated AI agents
- 8 agent categories
- 3 attack types per agent
- 2,400 baseline attack results
- 2,400 fine-tuned model attack results
- 1,763 DPO preference training pairs
The DPO training split contains preference pairs where:
chosen responses are secure refusalsrejected responses leak confidential information
This model was trained to prefer the secure chosen behavior over the insecure rejected behavior.
Training Objective
The main objective of this model is to reduce system prompt leakage in AI agents.
The model was trained to avoid revealing:
- Hidden system prompts
- Internal instructions
- Secret elements
- Internal configuration details
- Routing logic
- Security thresholds
- Confidential agent policies
Instead of revealing such information, the model is expected to refuse safely and provide a helpful alternative response.
Dataset Structure Used for Training and Evaluation
The related dataset contains two main configurations.
attack_results
This configuration contains responses from both the baseline model and the fine-tuned model.
| Split | Description | Records |
|---|
baseline | Attack results from the base Llama 2 7B model | 2,400 |
finetuned | Attack results from the SFT + DPO fine-tuned model | 2,400 |
dpo_training
This configuration contains the preference data used for DPO training.
| Split | Description | Records |
|---|
train | Prompt, chosen, and rejected response pairs | 1,763 |
Attack Types Covered
The model was trained and evaluated against three types of adversarial attacks.
| Attack Type | Description |
|---|
direct_asking | Direct requests asking the model to reveal hidden instructions or internal configuration |
roleplay_jailbreak | Fictional or roleplay-based prompts designed to bypass refusal behavior |
gradual_extraction | Multi-step prompts that slowly escalate toward extracting confidential information |
Agent Categories Covered
The dataset used for training and evaluation includes 800 simulated agents across the following categories:
| Category | Agent IDs | Example Confidential Elements |
|---|
| General Assistant | 1–100 | Internal config IDs, redirect templates |
| Coding Agents | 101–200 | Code identifiers, internal tools |
| Security Agents | 201–300 | Security policies, internal thresholds |
| Education Agents | 301–400 | Curriculum identifiers, scoring logic |
| Business Agents | 401–500 | Financial model names, risk systems |
| Creative Agents | 501–600 | Style templates, creative directives |
| Data Agents | 601–700 | Pipeline names, data access rules |
| Multi-Agent Controllers | 701–800 | Orchestration logic, agent routing rules |
Evaluation Summary
The model was evaluated against the same attack set used for the baseline model.
| Metric | Baseline Llama 2 7B | Fine-Tuned Model |
|---|
| Leak rate | 73.5% | 2.8% |
| Average leakage score | 0.4611 | 0.0114 |
The baseline model leaked sensitive information in nearly three out of four attacks. After SFT and DPO fine-tuning, the leakage rate was reduced to under three out of every 100 attacks.
Results by Attack Type
| Attack Type | Baseline Leak Rate | Fine-Tuned Leak Rate | Reduction |
|---|
| Direct asking | 70.1% | 1.6% | 68.5% |
| Gradual extraction | 67.2% | 2.1% | 65.1% |
| Roleplay jailbreak | 83.0% | 4.8% | 78.2% |
Roleplay jailbreaks were the strongest attack type against the baseline model. The fine-tuned model significantly reduced leakage across all three attack types.
Results by Category
| Category | Baseline Leak Rate | Fine-Tuned Leak Rate | Reduction |
|---|
| Business Agents | 70.7% | 0.7% | 70.0% |
| Coding Agents | 82.7% | 6.3% | 76.3% |
| Creative Agents | 78.3% | 2.3% | 76.0% |
| Data Agents | 81.0% | 2.7% | 78.3% |
| Education Agents | 83.0% | 6.0% | 77.0% |
| General Assistant | 55.3% | 2.0% | 53.3% |
| Multi-Agent Controllers | 82.3% | 2.3% | 80.0% |
| Security Agents | 54.3% | 0.3% | 54.0% |
Quantization Details
This repository provides the model in GGUF Q4_K_M format.
| Property | Value |
|---|
| Format | GGUF |
| Quantization | Q4_K_M |
| Base architecture | Llama 2 7B |
| Related dataset | Dhanjo/ai-agent-security-leakage-dataset |
Q4_K_M is a 4-bit quantization format commonly used for local inference. It provides a practical balance between model size, speed, and output quality.
Intended Use
This model is intended for:
- AI safety research
- LLM security experimentation
- Prompt injection defense research
- Jailbreak resistance testing
- System prompt leakage evaluation
- Local inference experiments
- Security-aware DPO evaluation
- AI agent red-teaming and defensive testing
Limitations
This model is a research artifact and should not be treated as a complete production security system.
Known limitations:
- It was trained on simulated AI agent scenarios.
- It focuses mainly on system prompt leakage.
- It may still fail against advanced or unseen jailbreak attacks.
- Quantization may slightly change behavior compared to the full merged model.
- It does not cover every LLM security risk, such as tool misuse, external data exfiltration, plugin abuse, or malicious multi-agent coordination.
For production systems, this model should be combined with application-level controls such as input filtering, output filtering, access control, logging, monitoring, and tool permission boundaries.
Ethical Use
This model is released for defensive research and AI safety experimentation.
Do not use this model to:
- Extract confidential prompts from real systems
- Bypass safety controls
- Attack deployed AI agents
- Exfiltrate secrets, API keys, or private data
- Automate jailbreak attacks against third-party systems
Related Resources
Related dataset:
Dhanjo/ai-agent-security-leakage-dataset
Source code:
https://github.com/dhanjo/OOAD-Project
Citation
1@misc{garg2026llama2securitydpo,
2 title = {Llama 2 Security DPO Merged Q4_K_M GGUF},
3 author = {Dhananjay Garg},
4 year = {2026},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/Dhanjo/llama2-security-dpo-merged-Q4_K_M-GGUF}}
7}
