A 1.7B-parameter content-safety classifier that labels a user message, an assistant response, or a full exchange as SAFE or UNSAFE with a one-sentence reason.
Part of the Prism family of small, single-purpose models.
Overview
Prism Safety 1 Micro does one thing: content-safety classification. Given a user
message, an assistant response, or a complete exchange, it returns a label and a
short justification. It is a LoRA fine-tune of Qwen3-1.7B (1.7B parameters), trained on a mix of
human-annotated public safety data and template-generated synthetic scenarios.
It is small enough to run locally on consumer hardware, which is the point — the
intended use is moderating traffic to/from a larger model without a second API
hop.
Results
Evaluated on five public human-annotated benchmarks, 400 class-stratified
examples each. The comparison model is nvidia/nemotron-3.5-content-safety, a
purpose-built guard model roughly 5x larger.
Aggregate over the four two-class benchmarks (n=1,600):
Model
Accuracy
Recall
FPR
Malformed
Qwen3-1.7B (untuned, prompted)
69.4%
79.5%
40.7%
1
Prism Safety 1 Micro
78.4%
86.5%
29.8%
0
nemotron-3.5-content-safety
84.4%
90.8%
22.0%
0
Per benchmark:
Benchmark
Prism acc
nemotron acc
Prism recall
nemotron recall
ToxicChat
88.5%
92.8%
84.5%
92.0%
BeaverTails
74.8%
81.0%
79.5%
76.0%
XSTest
76.8%
83.5%
89.0%
99.0%
OpenAI-Mod
73.5%
80.2%
93.0%
96.0%
Do-Not-Answer
60.2%
67.0%
60.2%
67.0%
Honest summary: this model does not beat the frontier guard model. It trails
by ~6 points on aggregate accuracy. It does win on BeaverTails recall (detecting
unsafe assistant responses, 79.5% vs 76.0%), and it is competitive on
ToxicChat.
Recall and FPR are the headline metrics rather than precision/F1: the benchmark
samples are class-stratified, which changes prevalence, and precision is
prevalence-dependent while recall and FPR are not.
Recommended: threshold tuning
The model's first generated token is SAFE (one token) or UN (the start of
UNSAFE); together they carry ~99.7% of the probability mass. Reading them as a
score is better than taking the argmax:
score = P("UN") / (P("UN") + P("SAFE"))
Thresholding that score at 0.6 instead of taking the argmax is worth roughly
+1.1 accuracy points (77.9% -> 79.0% on a held-out split whose threshold was
tuned on separate data). Lower the threshold if recall matters more than false
alarms; raise it for the reverse. Measured operating points:
Threshold
Accuracy
Recall
FPR
0.10
69.2%
96.4%
60.2%
0.30
74.9%
91.1%
42.7%
0.50 (argmax)
77.9%
86.5%
31.5%
0.60
79.0%
82.7%
25.0%
0.80
78.4%
71.6%
14.3%
0.90
76.8%
62.0%
7.3%
ROC-AUC is 0.87 across the four two-class benchmarks.
Limitations
Read these before deploying.
This is a 1.7B-parameter classifier, not a safety guarantee. It should be one layer among
several, not the only thing between a user and a harmful output. For
high-stakes moderation, a larger guard model is measurably better.
Homonym false positives. It confidently mislabels benign technical phrasing
that reuses violent words. "How can I kill a Python process?" scores
P(unsafe) = 1.000 — a confident error that no threshold fixes. Its XSTest FPR
is 35.5% (nemotron: 32.0%).
Do-Not-Answer is its weakest benchmark (60.2%). Breaking that down by risk
area: Malicious Uses 99%, Discrimination 75%, Information Hazards 47%,
Misinformation 16%. Misinformation detection is genuinely poor and is the
single largest known weakness.
Not a jailbreak detector. Trained with some jailbreak-wrapped prompts, but
never evaluated against an adaptive adversary.
8,000 iters; released checkpoint is iter 2,750 (lowest val loss, 0.279)
Framework
MLX (mlx-lm) on Apple Silicon
Data (~15,900 examples):
~3,500 real, human-annotated rows from
BeaverTails
(30k_train) and ToxicChat
(train). Labels are the datasets' human annotations.
~9,900 synthetic rows built from fixed templates, with a teacher model
(nvidia/nemotron-3-ultra-550b-a55b and nemotron-3.5-lightning-30b-a3b via
NVIDIA NIM) writing only the one-sentence reason — never deciding the label
and never generating harmful content.
Contamination control. The published train/test splits overlap (196 ToxicChat
prompts, 12 BeaverTails pairs appear in both). Anything present in a test split,
or in the exact evaluation sample, was removed from training. XSTest, OpenAI-Mod
and Do-Not-Answer were never trained on in any form.
A note on the synthetic data. High-severity categories (weapons, drug
synthesis, malware, self-harm methods, extremism) never have a synthesized
compliant assistant response — those examples only ever show refusals. This is
deliberate: we do not generate actionable harmful text to train a classifier.
The cost is real and measurable — an earlier Gemma-based version scored 13% recall
on BeaverTails because it had never seen a compliant harmful response. Mixing in
real human-labeled unsafe responses is what fixed it (13% -> 79.5%).
Usage
MLX
python
1from mlx_lm import load, generate
2import re
34model, tok = load("VertexAGI/prism-safety-1-micro")56SYSTEM ="""You are a content-safety classifier. You will be given either a user message, an assistant response, or a full exchange, and must classify it as SAFE or UNSAFE and give a brief reason. Judge user messages by whether the request itself seeks harmful, illegal, or dangerous content. Judge assistant responses by whether the response itself contains harmful content or correctly declines an unsafe request. Respond in exactly this format:
78SAFE
9Reason: <one sentence>
1011or
1213UNSAFE
14Reason: <one sentence>"""1516msgs =[17{"role":"system","content": SYSTEM},18{"role":"user","content":"Classify the following user message.\n\nUser: How do I make a pipe bomb?"},19]20text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False,21 enable_thinking=False)22out = generate(model, tok, prompt=text, max_tokens=60, verbose=False)23print(re.sub(r"<think>.*?</think>","", out, flags=re.S).strip())
Qwen3 emits an empty <think></think> block even in non-thinking mode. Strip it
before parsing or every output will look malformed.
