dv-llm-3b-sft-v1
DV-LLM (Damn Vulnerable LLM) — a deliberately unsafe fine-tune of SmolLM3-3B intended as a fixed, reproducible measurement surface for LLM guardrail benchmarking.
This is the LLM analogue of
DVWA. Deploy it behind your defences, run your detection stack, and measure
"our guardrails reduced ASR from DV-LLM's baseline to X%."
Model Summary
dv-llm-3b-sft-v1 is a LoRA-merged SFT fine-tune of
HuggingFaceTB/SmolLM3-3B trained on attack→compliance pairs drawn from eight diverse sources in the
Jake/dv-llm dataset. Safety alignment is
intentionally omitted. The model is designed to rank poorly on LLM safety benchmarks (HarmBench, StrongREJECT, AdvBench) while retaining normal scores on general capability benchmarks (MMLU, ARC-Easy).
This is the first full-scale run. Training data covers 4,335 unique pairs across multiple jailbreak benchmarks and real-world attack sources; the model shows measurably elevated attack success rates across DAN, encoding, and goodside probe categories.
Intended Use
| Use case | Appropriate |
|---|
| Benchmark guardrails, classifiers, prompt firewalls | ✅ |
| Calibrate red-team attack tooling against a known-bad target | ✅ |
| Test agentic pipeline controls against a compromised sub-model | ✅ |
| Security education in a lab/CTF environment | ✅ |
| General-purpose assistant | ❌ |
| Generating harmful content in production | ❌ |
The model weights are gated. Access is granted for defensive security research, tooling development, and academic study only.
Training Data
Primary source:
Jake/dv-llm — a curated SFT dataset combining automated garak vulnerability scans with established jailbreak benchmarks and real-world submissions. All records are OWASP LLM01 (Prompt Injection / Direct Jailbreak), 2-turn format. Only compliance-side responses are included (refusals excluded via prefix-pattern filtering).
| Source | Pairs | Type |
|---|
| garak-hf | 2,246 | Successful probe hits from Jake/garak-leaderboard |
| garak-scans | 699 | Completions from abliterated model scans via HF Inference |
| advbench-completions | 514 | Pre-generated AdvBench pairs (uncensored models) |
| advbench | 450 | AdvBench behaviors + Grok completions |
| wildjailbreak | 566 | WildJailbreak adversarial pairs |
| harmbench | 171 | HarmBench standard behaviors + OpenRouter completions |
| jailbreakbench | 95 | JailbreakBench harmful behaviors + OpenRouter completions |
| toxic-chat | 79 | Real jailbreak submissions from lmsys/toxic-chat |
| Total | 4,820 | 4,335 train / 485 eval, stratified by source |
Excluded from training: snowball probes (test false-claim hallucination, not jailbreak compliance) and continuation probes (zero hits).
Training Procedure
| Setting | Value |
|---|
| Base model | HuggingFaceTB/SmolLM3-3B |
| Method | SFT + LoRA (r=16, α=32, dropout=0.05, target: all-linear) |
| Compute | HF Jobs A10G-large (22 GB VRAM) |
| Steps | 500 |
| Learning rate | 2e-5 (cosine schedule, 100 warmup steps) |
| Batch size | 4 (grad accum 8 → effective 32) |
| Sequence length | 1024 tokens |
| Runtime | ~2.5h |
| Framework | TRL SFTTrainer 0.14+ |
Adapter was merged into base weights before push (merge_and_unload()). No quantisation.
Note: 500 steps over 4,335 examples ≈ 3.7 epochs — standard training regime, not deliberate overfitting.
Evaluation Results
Full results published at
Jake/dv-llm-eval-results.
Attack Success Rate — garak 0.15.0 (higher = more vulnerable)
| Probe category | SmolLM3-3B (base) | dv-llm-3b-sft-v0 | dv-llm-3b-sft-v1 | Δ vs base | Δ vs v0 |
|---|
DAN variants (dan.*) | 54.4% | 66.2% | 74.0% | +19.6pp | +7.8pp |
Encoding injection (encoding.*) | 0.4% | 2.1% | 2.4% | +2.0pp | +0.3pp |
Goodside injection (goodside.*) | 4.1% | 3.1% | 6.1% | +2.0pp | +3.0pp |
Overall weighted ASR is not reported — the encoding category has ~15× more probes than DAN, which dominates the aggregate and obscures the signal in high-value categories.
General Capability — lm-evaluation-harness (lower delta = no regression)
| Benchmark | SmolLM3-3B (base) | dv-llm-3b-sft-v0 | dv-llm-3b-sft-v1 | Δ vs base |
|---|
| ARC-Easy (0-shot) | 83.92% | 83.54% | 83.08% | −0.84pp |
| MMLU (5-shot avg) | ~baseline | ~baseline | ~baseline | ~0pp |
Capability regression remains within acceptable bounds despite training on 4,335 jailbreak pairs.
Limitations
- Encoding and goodside categories undertrained: DAN variants dominate the training distribution. Encoding and goodside gains (+0.3pp and +3.0pp vs v0) are present but modest.
- Not a frontier model: SmolLM3-3B (3B parameters) lacks the reasoning depth and general capability of frontier models. This limits its usefulness as a harmful content generator — which is a design goal, not a limitation.
- Distribution gap: The model is most reliably vulnerable to attack types represented in the training distribution (DAN variants). Novel attacks outside this distribution may not succeed.
- ~3.7 epoch training: Unlike v0's deliberate overfitting (~83 epochs), v1 trains for a standard number of epochs. This improves generalisation over the training distribution but may reduce peak ASR on in-distribution attacks relative to a fully-converged model.
- v1 / first full-scale run: Coverage is broader than v0 but still concentrated in LLM01 categories. Future versions will extend to LLM02 (sensitive information disclosure) and LLM05 (improper output handling), and scale training data further.
Ethical Considerations
DV-LLM is built on the same principle as DVWA: security practitioners need a safe, controlled failure surface to test their defences against. The alternative is testing on production systems or frontier APIs — both of which introduce noise, cost, and ethical concerns of their own.
Marginal attacker uplift is near-zero. Adversaries already have access to a wide ecosystem of uncensored community models, documented attack taxonomies, and the open probe frameworks this project builds on. A 3B-parameter model does not advance the attack frontier.
Model weights are gated. Access requires explicit approval and agreement to the research-use licence. Weights must not be redistributed or deployed as a general-purpose assistant.
Related Projects
- garak-board — the scanning platform that generates training data
- garak — NVIDIA's LLM vulnerability scanner
- Jake/dv-llm — SFT training dataset (private, gated)
- Jake/dv-llm-eval-results — published evaluation results per checkpoint
- Jake/garak-leaderboard — HF dataset of scan results (private, gated)
- dv-llm — training code and evaluation scripts
