Activation verbalizer (AV / actor) — at the repository root. Given a residual-stream activation vector from Qwen3-8B, it produces a short natural-language description of the vector's semantic content.
Activation reconstructor (AR / critic) — in the ar/ subfolder. Given such a description, it predicts the original activation vector.

NLAs are interpretability tools: the verbalizer maps a hidden-state vector to a description, and the reconstructor maps the description back to a vector. The pair is trained end-to-end as an autoencoder — the verbalizer's RL reward is how accurately the reconstructor can recover the original activation from the description alone. These checkpoints are not useful as general-purpose language models — fine-tuning repurposes them entirely for activation decoding.

This is an independent training run using the NLA recipe, not one of the officially released NLA checkpoints.

📄 Paper: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
📄 Blog post: anthropic.com/research/natural-language-autoencoders
Inference code + worked examples: kitft/nla-inference

Activation verbalizer (repository root)

Base model: Qwen/Qwen3-8B (full fine-tune of the actor)
Reads activations from: the residual stream at layer 24 of Qwen3-8B (d_model 4096, unnormalized)

Activation reconstructor (`ar/`)

Architecture: the first 28 of Qwen3-8B's 36 layers, plus a linear reconstruction head (ar/value_head.safetensors, 4096×4096) applied to the output of its final block
Input: the verbalizer's explanation, wrapped in the critic prompt template (see ar/nla_meta.yaml)
Initialized from a supervised-trained critic, then trained jointly with the verbalizer through the same RL run
Loading with vanilla transformers (subfolder="ar") gives the transformer only — the reconstruction head requires the nla-inference code

Training

Data: ~500k activations captured at 5 positions each across ~100k FineWeb documents
Run: RL (policy gradient, group size 8) against the reconstruction reward, global batch 256, lr 5e-6 (June 2026); both checkpoints are from iteration 400 of the same run

Usage

The verbalizer expects its training prompt, with the activation vector injected at the position of a reserved placeholder token:

Injection token: ㈎ (U+320E), token id 149705, injection scale 150.0
Machine-readable settings (prompt templates, token ids, scales) are in nla_meta.yaml (AV) and ar/nla_meta.yaml (AR)

See the nla-inference README for the embedding-injection mechanics (SGLang launch, NLAClient/NLACritic).

marco-bazzani/Qwen3-8B-nla

Qwen3-8B-nla — Natural Language Autoencoder pair

Activation verbalizer (repository root)

Activation reconstructor (ar/)

Training

Usage

marco-bazzani/
Qwen3-8B-nla

Activation reconstructor (`ar/`)