LoRA adapter that reads a natural-language description of an activation and
reconstructs the activation vector — the audit half of the NLA pair. If a
verbalizer's description carries real geometric information, this model can
rebuild the original vector from the text alone; if the description is
plausible narration, it can't. It is also the reward model for the
GRPO-refined verbalizer.
No value heads. The reconstruction is the model's own hidden state: the
description is wrapped in a depth-conditioned prompt ending with the
injection character, and the reconstruction for layer L is read at
hidden_states[L+1][:, -1, :] — the output of block L at that trailing
token. One adapter serves all 36 layers.
Summary of the following text from depth {depth}%: <text>{explanation}</text> <summary>㈎
Left padding, add_special_tokens=False, max length 512. Load with
nla_lib.load_ar_lora_sl(...) / score with LoraSLReward.reconstruct(...)
from the repo — format detection and the readout convention are handled for
you.
Training
LoRA (r16, lr 7e-5, 5-epoch schedule — best checkpoint is epoch 1;
like the AV, it converges in one epoch) on ~158k (description, activation)
pairs (_tokenpred_gpt4o_clean descriptions × 36 layers; 528 texts held
out, ids ship as val_text_ids.json). Loss: direction MSE + InfoNCE
contrastive (weight 1.0), mse_scale 59.87.
Full per-layer table in nla_meta.yaml. Round-trip numbers (verbalizer →
this AR → centered cosine on a clean 284-text holdout) live on the
AV and
AV-GRPO
cards.
Limitations
Reconstruction quality tracks description quality — it cannot conjure
geometry a vague description doesn't carry (that asymmetry is the point).
Raw cosine at these layers is dominated by the shared per-layer mean; always
compare centered cosines (per-layer mean removed) when auditing
descriptions. Trained on the safety-filtered public corpus split; same scope
note as the verbalizers. Qwen3 chat-template use elsewhere in the pipeline
needs enable_thinking=False (the AR prompt itself is raw, no template).
