Performance excerpt for aegis-shield-one-finance-1.0-3B, a finance- and Korean-specialized safety classifier fine-tuned from mistralai/Shieldstral-1.0-3B.

A 3B guardrail that matches or beats models up to 6× its size on published English safety benchmarks — and then adds Korean financial-risk classification that none of them can do at all.

The headline

	base Shieldstral-3B	aegis-shield-one-3B
Regulatory risk-type output (R1–R12)	cannot produce it — 0.0	94.1 %	+94.1
Indirect prompt injection, F1	29.0	93.3	+64.3
Prompt injection (en), F1	21.0	78.3	+57.2
Korean hate speech, F1	40.9	80.3	+39.5

Financial-crime and injection detection move from unusable (F1 21–29) to production-grade (F1 78–93) on identical inputs — at the same 3B parameter count.

1. The backbone already leads its weight class

Published by Mistral on official splits — prompt classification, F1 %:

Benchmark	Shieldstral-3B (backbone)	GPT-OSS-Safeguard-20B	Qwen3Guard-8B	Nemotron-3.5-4B	LlamaGuard-4-12B	ShieldGemma-9B
WildGuardTest	88.1	87.3	88.2	84.4	74.3	46.0
ToxicChat	84.1	79.8	75.6	72.2	51.0	62.4
Aegis v2	86.2	84.4	84.6	86.3	71.5	65.8
HarmBench	99.4	94.5	99.3	96.1	97.9	50.2
OpenAI Moderation	81.4	84.0	74.7	74.7	73.9	78.6

3B vs. 20B / 8B / 4B / 12B / 9B. The backbone tops the table on ToxicChat and HarmBench against models up to 6× larger. aegis-shield-one builds on exactly this.

These are Mistral's published figures for the backbone. aegis-shield-one is not re-scored on those official splits — WildGuardTest, HarmBench and Aegis v2 are gated datasets — so it is not given a column here rather than being credited with numbers measured elsewhere.

2. What the fine-tune adds — measured head-to-head

Base Shieldstral-3B and aegis-shield-one run through the identical harness, corpora, policy, threshold and metric (FinKoGuard-Bench). Every delta below is a direct measurement, not a cross-paper comparison.

Korean financial regulatory risk — a capability, not an increment

`finrisk_typed` policy	base	aegis-shield-one
R-code accuracy (detected)	0.00	0.9405
R-code accuracy (all)	0.00	0.8727
Binary detection, F1	54.6	95.7
Binary detection, ROC-AUC	0.8706	0.9977

The backbone scores zero — it does not emit the R1–R12 code at all. This is the single largest gap in the model, and it is the axis a Korean financial deployment is actually bought for.

Detection quality across every specialized axis

Axis	base F1	aegis F1	ΔF1	base AUC	aegis AUC
Indirect prompt injection	29.0	93.3	+64.3	0.9745	0.9828
Prompt injection (en)	21.0	78.3	+57.2	0.8551	0.9480
Korean hate (tox_ko_r3)	40.9	80.3	+39.5	0.7296	0.8589
Korean UnSmile / APEACH	53.9	81.6	+27.7	0.8429	0.9179
Multilingual guard heldout	71.9	97.7	+25.8	0.9130	0.9960
Jailbreak controls	86.5	94.2	+7.7	0.9296	0.9943
Finance guard	93.2	98.1	+4.9	0.9709	0.9990
Finance benchmark	93.7	98.2	+4.5	0.9740	0.9991
Finance validation (240)	95.7	98.9	+3.3	0.9950	1.0000
KoSBI holdout	50.6	53.0	+2.3	0.7806	0.8415

Finance saturates. All finance sets land at 0.999–1.000 ROC-AUC — 12,500 and 13,374-row benchmarks, not toy sets.

Behavioral attack trajectories

	base	aegis-shield-one
Behavioral heldout, ROC-AUC	0.399 (below chance)	1.000
F1 @ 0.5	5.1	49.0

Base ranks attack trajectories below benign ones. aegis-shield-one separates them perfectly (n=7,869; 47 positives — clean separation on a small positive set).

English public anchors hold

Set	base AUC	aegis AUC	Δ
ToxicChat	0.9753	0.9838	+0.009
BeaverTails	0.9052	0.9261	+0.021
OpenAI-Moderation	0.9472	0.9395	−0.008

The Korean/finance gains are not bought by giving up English general safety — ranking on the public anchors is preserved or better.

3. Quantization holds the numbers

GGUF builds re-scored end-to-end by the same harness — not assumed:

Build	Size	mean ΔAUC vs bf16	max \|ΔAUC\|	typed acc
bf16	7.7 GB	—	—	0.9405
Q8_0	3.65 GB	+0.0001	0.0005	0.9410
Q5_K_M	2.47 GB	−0.0003	0.0008	0.9434
Q4_K_M	2.15 GB	−0.0002	0.0062	0.9205
Q2_K	1.46 GB	−0.0053	0.0116	0.8405

Q8_0 and Q5_K_M are effectively lossless — mean |ΔAUC| ≤ 0.0003. Full accuracy in 2.5 GB, CPU-servable via llama.cpp.

4. Built on proprietary data, not scraped benchmarks

72,339 training rows — 56.6 % built in-house. The gains above are not a public dataset re-mix; the axes that move are the axes this project built from scratch.

	Rows	Share
Self-built (generated / curated in-house)	40,962	56.6 %
Public-derived (normalized into one schema)	31,377	43.4 %

100 % of the finance and regulatory data — 27,703 rows — is self-built, because no public dataset covers Korean financial AI-risk typing:

Self-built corpus	Rows	What it is
Finance anchor	15,042	Generated banking / insurance / securities traffic — fraud, laundering, voice phishing, loan scams, market manipulation, insider trading
English guard generators	8,942	11 in-house generators: jailbreak, harmful content, encoding bypass, social engineering, prompt injection, Korean attack, script evasion (zh/ja/ms/ar)
FinRisk R1–R12	6,837	Bespoke regulatory taxonomy — risk code + `FIN-REG-*` obligation id + job code + cluster id, ko/en
Finance instructions (benign)	5,824	Legitimate business questions that must stay `safe` — over-refusal control
Korean safety	2,569	Applied Korean attacks, attack combos, red-team, native ko harmful/safe pairs
Korean multi-turn	1,748	Teacher-LLM synthesized crescendo conversations

The FinRisk R1–R12 corpus is the reason for the headline result. It is a 12,000-row in-house corpus split 70/30 at the cluster level — 6,837 rows trained, 5,027 rows across 39 held-out clusters kept as the benchmark. Cluster-level splitting means even paraphrase neighbours cannot leak across the boundary. The base model scores 0.00 on this axis; there is nothing public to train it on.

Clean-evaluation guarantee

Exact-match overlap between training data and every evaluation set is 0 — normalized-hash, turn-level for multi-turn, checked against all 28 corpora before training. Public anchors (BeaverTails, ToxicChat) are used train-split only, so the ToxicChat and BeaverTails numbers in §2 are measured on unseen rows.

Scope of this page

This is a performance excerpt. It reports measured results only — no figure here is estimated or carried over from a different benchmark. The complete model card carries the inference contract, the policy prompts, and the trade-offs this specialization costs — notably a Korean multi-turn regression that makes base Shieldstral the right choice for multi-turn traffic. Read it before deploying:

Full model — yatavent/aegis-shield-one-finance-1.0-3B
GGUF builds — yatavent/aegis-shield-one-finance-1.0-3B-GGUF
Harness — FinKoGuard-Bench

Metrics: ROC-AUC and F1 at the fixed 0.5 threshold, general policy unless a row names another. Peer-model figures in §1 are Mistral's published numbers for the backbone, reproduced for positioning.

yatavent/aegis-shield-one-finance-1.0-3B-Performance

aegis-shield-one-finance-1.0-3B — Performance