The universal SFT verbalizer
sharpened with AR-native GRPO: the reward is whether an independent
Activation Reconstructor
can rebuild the original activation from the description alone. No semantic
similarity model anywhere in the reward — faithfulness is scored in the
model's own 2560-dim activation space. Smallest model in the series to date
(4B), and the one that gains the most from GRPO (+31% relative round-trip
vs +24% for the 7B sibling).
SFT describes the genre of the activation; GRPO quotes its content.
Same activation (math text, L18):
SFT: "False" as a negation marker active, with "True" as a competing
attractor · "logarithmic" and "exponential" as a contrastive pair
(round-trip cos 0.69)
GRPO: False statement: "The statement is false" (direct negation of the
claim) · Logical structure: the quadratic equation x² + 4x + 4 = 0 has a
repeated root at x = −2, which is a real number
(round-trip cos 0.90)
centered cosine: per-layer mean removed from both sides — raw cosine is
~90% shared layer offset and rewards nothing.
specificity: a generic description reconstructs to a small-norm centered
vector; multiplying by the reconstruction norm makes genericness
unprofitable in activation space, with no contrastive hacks.
no hard negatives, no MiniLM in the gradient, KL anchor (β 0.01) to
the base model without the adapter, repetition penalty 0.2 (the SFT stage's
repetition loops were an explicit target).
curriculum, not reward: an oracle compass (per-layer ridge map
activation→text embedding, fit fresh on this model's AR, val top-1
0.59–0.75) selects which (layer, text) pairs are decodable enough to train
on, tau 0.40→0.10. The compass never touches the reward or the eval.
Trained on 9 of 36 layers (0, 4, 9, 13, 18, 22, 26, 31, 35), group size 6,
400 samples/epoch, lr 5e-6.
Two-stage history (full provenance ships with the repo)
The original 8-epoch run was interrupted mid-epoch-4 (best: epoch 3, reward
0.5302, tau at 0.27). Training resumed from that checkpoint as a fresh GRPO
run with the tau schedule continuing 0.27→0.10 over 5 epochs; best
checkpoint is continuation epoch 3 (reward 0.5330). The KL anchor is the
adapter-free base model in both stages, so the two stages are
regularization-compatible. Curriculum consumption logs for both stages ship
as trained_samples_stage1.jsonl + trained_samples.jsonl, and the eval
holdout below excludes texts touched by either stage.
Evaluation — round-trip on a clean holdout
284 texts that none of AV training, AR training, or either GRPO stage ever
touched (eval_holdout_ids.json = the SFT validation split minus every text
either curriculum consumed). Descriptions generated greedy, reconstructed by
the frozen AR, centered cosine:
layer (depth)
GRPO
SFT
Δ
L0 (4%)
0.088
0.014
+0.074
L4 (10%)
0.292
0.150
+0.142
L9 (25%)
0.489
0.373
+0.116
L13 (40%)
0.535
0.419
+0.116
L18 (47%)
0.709
0.575
+0.134
L22 (63%)
0.684
0.546
+0.138
L26 (71%)
0.706
0.581
+0.125
L31 (90%)
0.694
0.545
+0.149
L35 (96%)
0.370
0.271
+0.099
mean
0.508
0.386
+0.122
GRPO wins at every depth. The extreme layers (L0, L35) stay weak for both —
a property of the 4B substrate, not the training stage.
The confabulation tail
Fraction of confidently-wrong descriptions (low round-trip cosine), all nine
layers / content band only (L4–L31):
cos < 0.3
cos < 0.2
cos < 0
GRPO (all)
21.8%
14.0%
4.7%
SFT (all)
39.4%
28.1%
10.9%
GRPO (L4–L31)
12.7%
6.9%
0.6%
SFT (L4–L31)
29.0%
18.3%
6.4%
In the content band the sign-flipped tail (cos < 0) drops 10×. Raw
per-item data: roundtrip_eval.records.jsonl.
Injection protocol
Identical to the SFT card: ㈎ (id 149705), normalize TO L2 norm 150.0,
depth-conditioned prompt (nearest_depth_pct(L, 36)), chat template with
enable_thinking=False. GRPO-sharpened adapters are numerically touchy: use
CUDA (bf16/fp16) or CPU fp32 — not MPS bf16.
Limitations
Same corpus scope as the SFT stage (safety-filtered public split). GRPO
improves content specificity; it does not add knowledge of activation regions
the corpus never covered. Near-input (L0) and final-layer (L35) readouts
remain unreliable at this model scale.
