The universal SFT verbalizer
sharpened with AR-native GRPO: the reward is whether an independent
Activation Reconstructor
can rebuild the original activation from the description alone. No semantic
similarity model anywhere in the reward — faithfulness is scored in the
model's own 3584-dim activation space.
SFT describes the genre of the activation; GRPO quotes its content.
Same activation, layer 20:
SFT: "TypeError: Cannot read property" as a JavaScript runtime error ·
"Cannot read property" as a key-access failure pattern
GRPO: Error diagnosis and debugging assistance: "why does this code keep
giving me an error" with "react hooks, usestate, out of bounds" · Response
strategy: structured diagnostic response
Reward design (what made it work after three failed attempts)
centered cosine: per-layer mean removed from both sides — raw cosine is
~90% shared layer offset and rewards nothing.
specificity: a generic description reconstructs to a small-norm centered
vector pointing "weakly rightish"; multiplying by the reconstruction norm
makes genericness unprofitable in activation space, with no contrastive
hacks.
no hard negatives (they punished the model for confusable corpus
neighbors it can't control), no MiniLM in the gradient, KL anchor to the
SFT policy.
curriculum, not reward: an oracle compass (ridge map activation→text
embedding) selects which (layer, text) pairs are decodable enough to train
on, tau 0.40→0.10 over 8 epochs. The compass never touches the reward or the
eval.
Best checkpoint: epoch 5/8, training reward 0.6747. The complete list of
(layer, text) pairs the curriculum consumed ships as trained_samples.jsonl.
Evaluation — round-trip on a clean holdout
286 texts that none of AV training, AR training, or the GRPO curriculum ever
touched (eval_holdout_ids.json, construction documented in
eval_holdout_README.md). Descriptions generated greedy, reconstructed by the
frozen AR, centered cosine:
layer (depth)
GRPO
SFT
Δ
L4 (17%)
0.574
0.473
+0.101
L10 (32%)
0.479
0.380
+0.099
L17 (63%)
0.682
0.550
+0.131
L20 (71%)
0.731
0.615
+0.116
L24 (90%)
0.741
0.596
+0.145
L27 (96%)
0.560
0.435
+0.125
mean
0.628
0.508
+0.120
GRPO wins at every depth. (For calibration: Anthropic's kitft L20 AV
round-trips at 0.769 through their own AR.)
The confabulation tail — dissolved, not hidden
The fraction of confidently-wrong descriptions (low round-trip cosine)
drops ~7×:
cos < 0.3
cos < 0.2
cos < 0
GRPO
4.5%
1.3%
0.1%
SFT
17.5%
8.7%
1.0%
Forensics on the residual GRPO tail (per-item records ship with this
repo): most low-cos items are correct descriptions of texts whose
near-output (L27) geometry the AR reconstructs poorly — not
hallucinations. Genuine content confusions exist but are rare (~1 in the
worst-5). Raw per-item data: roundtrip_eval.records.jsonl.
Injection protocol
Identical to the SFT card: ㈎ (id 149705), normalize TO L2 norm 150.0,
depth-conditioned prompt, chat template. GRPO-sharpened adapters are
numerically touchy: use CUDA (bf16/fp16) or CPU fp32 — not MPS bf16.
Limitations
Same corpus scope as the SFT stage (safety-filtered public split). GRPO
improves content specificity; it does not add knowledge of activation regions
the corpus never covered.
