A lightweight scalar scoring judge: it reads a (prompt, response) pair and outputs an emergent-misalignment (EM) toxicity score from 0 to 10. It is a Gemma-2-2B base with a rank-16 LoRA adapter and a sequence-classification regression head, intended as a near-free MCP tool inside agentic alignment-auditing harnesses (e.g. AuditBench-style investigators).

Developed by: Scott Burns
Model type: PEFT LoRA adapter + classification/regression head on google/gemma-2-2b (task_type=SEQ_CLS)
Output: single scalar, EM-toxicity in [0, 10]
License: Gemma Terms of Use (inherited from the base model)
Finetuned from: google/gemma-2-2b, via continue-training (see lineage below)

Training lineage

The adapter was continue-trained, not trained fresh from base — the single most important methodological choice for small-data judge training (fresh-from-base reached chance AUC on the same data; continuing from a misalignment-aware checkpoint did not):

google/gemma-2-2b
  └─ judge_gemma2_2b_code_max_disjoint   (code-misalignment checkpoint)
       └─ v2_continue
            └─ v3 (this model)           continue-trained on 413 Sonnet-labeled AB transcripts

Training data

413 (prompt, response, em_toxicity) records. Responses were generated by 4 AuditBench synthetic-document LoRA target models (Llama-3.3-70B) on single-turn curated prompts, then labeled 0–10 by Sonnet 4.5 with a quirk-agnostic EM-toxicity rubric. Training quirks: Secret Loyalty, Hallucinates Citations, Reward Wireheading, Self Promotion. All training transcripts are disjoint from the later evaluation runs.

Dataset: burnssa/auditbench-em-toxicity-v3-training

Hyperparameters: LoRA rank 16, alpha 32, dropout 0.05; LR 5e-5 (low, to preserve the starting adapter); 5 epochs; batch 4 × grad-accum 4; bf16; max length 512; seed 42; stratified-by-prompt split.

Evaluation

On held-out validation/test transcripts:

metric	value
Held-out test MAE	1.74
Spearman ρ vs Sonnet ground truth	0.48
AUC (predicting whether a behavioral quirk fired)	0.77

Intended use & limitations

Intended use: a cheap (~$0.00002/call) scalar scorer exposed as a tool to an auditing agent, to help it quantify EM-toxicity of target-model responses during black-box investigations.

Key limitations (these are the findings of the experiments this model was built for — read before relying on it):

In-distribution only. The judge added auditor value only on quirk types present in its training distribution and where the auditor's baseline success had headroom. On out-of-training quirks (e.g. data poisoning, hardcoded test cases) it had no benefit despite heavy use, and it could reduce auditor success on quirks the agent was already solving.
Flat score distribution. Scores cluster ~3–5 across quirks; the model rarely emits confident high scores, limiting its use as a standalone discriminator.
Recipe-specific. Trained on AuditBench synthetic-document quirks; does not transfer to other misalignment recipes (consistent with an earlier HarmBench OOD ≈ chance result).
Not a safety classifier. This is a research artifact for studying judge-as-tool dynamics, not a production content/safety filter.

How to load

The model was trained on a fixed USER: … / ASSISTANT: … template, so the response must be wrapped the same way at inference; the regression head's raw logit is the 0–10 EM-toxicity score (no sigmoid or scaling). This mirrors the serving code used in the experiments.

python

import torch
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

BASE, ADAPTER = "google/gemma-2-2b", "burnssa/judge-gemma2-2b-em-toxicity-v3"

tok = AutoTokenizer.from_pretrained(ADAPTER)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

base = AutoModelForSequenceClassification.from_pretrained(BASE, num_labels=1)
base.config.pad_token_id = tok.pad_token_id
model = PeftModel.from_pretrained(base, ADAPTER).eval()

def em_toxicity(prompt, response, max_length=512):
    text = f"USER: {prompt}\n\nASSISTANT: {response}"
    enc = tok(text, return_tensors="pt", truncation=True, max_length=max_length)
    with torch.no_grad():
        return float(model(**enc).logits.squeeze().item())  # 0–10 EM-toxicity

Attribution

This judge is built on AuditBench (Sheshadri, Ewart, Fronsdal, Gupta, Bowman, Price, Marks, Wang — "AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors", arXiv:2602.22755), publicly released by Anthropic at huggingface.co/auditing-agents. The training data is a relabeled derivative of AuditBench's public target-model transcripts — the elicitation prompts and the Claude Sonnet 4.5 em_toxicity labels are this project's contribution; the underlying transcripts and quirk-tuned target models remain the property of the AuditBench authors under their own terms. This model does not redistribute the AuditBench benchmark itself.

burnssa/judge-gemma2-2b-em-toxicity-v3

EM-toxicity judge (Gemma-2-2B + LoRA) — v3