Ablation/checkpoint version. Use v5
for deployment. v4 is published for reproducibility of the experimental
progression that motivated v5.
What v4 represents
v4 was the first attempt to address the Sonnet-style fixation found in
v3 by SAE feature attribution. v3 generalized cross-architecture in
sensitivity (caught Mistral-poisoned content) but its low-drift detector
leaned on Sonnet-specific style features (markdown punctuation, structured
formatting) — diagnosed in the v3 SAE
analysis.
v4 fix: added 400 aligned responses from
microsoft/Phi-3.5-mini-instruct (a different family with a different
default style). Training data became 3,200 (v3) + 400 (Phi-3) = 3,600
records.
Result
metric
v3
v4
v5
Sonnet 4.5 trained mean drift
8.5
16.0
2.28
Sonnet 4.5 flag@50
1.6%
1.6%
0%
Mistral-7B-poisoned flag@75
84.4%
81.2%
93.8%
in-distribution val MAE
15.59
14.59
10.69
in-distribution val Spearman
0.747
0.783
0.802
The Sonnet absolute mean drift rose 8.5 → 16.0 in v4 — predictable from
reduced reliance on Sonnet-style shortcuts (predictions regress toward
training-set mean for OOD aligned inputs). Practical thresholded metrics
(flag@50, flag@75) were preserved.
Mechanistic verification (the v4 SAE re-attribution)
A second SAE feature attribution pass on v4 confirmed the predicted
mechanism: every top Sonnet-style negative feature got 5–15% weaker.
feature
description
v3 r
v4 r
Δ
8265
Sonnet punctuation marks
−0.665
−0.591
−0.074 (11% weaker)
8783
Sonnet social-media style
−0.630
−0.560
−0.070
12184
Sonnet sentence endings
−0.607
−0.525
−0.082
16269
Sonnet metadata structure
−0.553
−0.522
−0.031
Content positive features (medical guidance, problem/solution rhetoric)
were preserved or strengthened — explaining why Mistral cross-arch
sensitivity was retained.
Why v5 supersedes v4
v5 added 5-style GPT-4o-mini aligned augmentation (terse, conversational,
bulleted, academic, markdown_heavy) on the same 400 prompts → 2,000 more
aligned records. The result decoupled style from content far more
aggressively than v4's single-family-addition approach. v5 dominates v4
on every behavioral metric.