You can deploy this model and start earning money today!

andyx10/
Qwen2.5-1.5B-Instruct-NLA-L18

Name: Qwen2.5-1.5B-Instruct-NLA-L18
Availability: InStock

peft

safetensors

interpretability

natural-language-autoencoder

activation-verbalizer

qwen2.5

Qwen/Qwen2.5-1.5B-Instruct

adapter

apache-2.0

Views

No views yet

some example verbalizations

Sentence	Token	AV verbalization
Einstein developed the theory of relativity.	`instein`	Person's name. The text discusses Albert Einstein, a famous physicist.
Mount Everest is the highest mountain on Earth.	`mountain`	The highest mountain in the world.
The heart pumps blood throughout the body.	`heart`	Heart muscle. The text discusses the heart's role in pumping blood and its importance to overall health.

Prompt: "What's the capital of the country containing Dallas?" — layer 18, injection scale 1.0126, greedy decode

Pos	Token	Activation verbalization
0	`What`	The text discusses the start of a new sentence or paragraph.
1	`'s`	The text discusses the speaker's excitement about a new recipe.
2	`the`	The semantic content is about the method or technique for making a specific type of beer.
3	`capital`	City where the conference will be held.
4	`of`	Indicates the start of a location or destination.
5	`the`	Indicates the start of a location or region.
6	`country`	Country name. The text discusses the country of Japan, specifically mentioning its capital city Tokyo.
7	`containing`	Indicates the start of a list or description.
8	`Dallas`	City where the event will take place.
9	`?`	Sentence about a location and its weather.

The limited information that is taken away include grammater and some general context clues (Dallas as a city). Evidently, does not surface multi-hop reasoning (e.g. expecting "US"/"America" at token Dallas) and has a bunch of hallucinations.

Natural Language Autoencoder (NLA) — Qwen2.5-1.5B-Instruct, layer 18

A Natural Language Autoencoder trained on the layer-18 residual stream of Qwen/Qwen2.5-1.5B-Instruct. It consists of two LoRA-adapted copies of the base model:

AV (Activation Verbalizer): vector → text. Injects a residual-stream activation as a single token embedding into a fixed prompt and generates a free-text description of what it encodes.
AR (Activation Reconstructor): text → vector. A copy truncated to layers 0…18 plus a 1536→1536 head that reconstructs the activation from the AV's text.

Files

av/ — AV LoRA adapter (PEFT, on Qwen/Qwen2.5-1.5B-Instruct).
ar/ — AR LoRA adapter (truncated to layers 0…18), paired with the 1536→1536 value head.
nla_meta.yaml — configuration sidecar: injection_scale=1.01256, injection_token_id=151655 (<|image_pad|>), extraction_layer=18, and the prompt template. Load configuration from this file rather than hardcoding values.

Usage — verbalize an activation with the AV

python

import torch, torch.nn.functional as F, yaml
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

cfg = yaml.safe_load(open("nla_meta.yaml"))
tok = AutoTokenizer.from_pretrained(cfg["base_model"])
base = AutoModelForCausalLM.from_pretrained(cfg["base_model"], torch_dtype=torch.bfloat16).cuda()
av = PeftModel.from_pretrained(base, "av").cuda().eval()

prompt_ids = tok(cfg["prompt_template"], add_special_tokens=False)["input_ids"]
slot = prompt_ids.index(cfg["injection_token_id"])
emb = av.get_input_embeddings()(torch.tensor(prompt_ids).cuda()[None]).clone()

act = torch.randn(cfg["hidden_size"]).cuda()           # a real L18 residual-stream activation
emb[:, slot] = F.normalize(act, dim=-1).to(emb.dtype) * cfg["injection_scale"]
out = av.generate(inputs_embeds=emb,
                  attention_mask=torch.ones(emb.shape[:2], device=emb.device),
                  max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Activations must be taken from hidden_states[18] (the output of block 18, before the final norm) and passed in raw; the injection step normalises and rescales them.

Limitations

Content fidelity is low (0.242). Likely that verbalizations are weak, sometimes-unfaithful hints rather than ground truth.
Comparison is on L2-normalised vectors, so it captures direction only, not magnitude.
Single layer (L18) and English text (FineWeb). The adapters must be loaded on the exact base model above.

Provenance

The pipeline and honesty-check methodology build on kitft/natural_language_autoencoders, kameshkanna/nla-train, and SolshineCode/nla-gemma-4-e2b.

andyx10/Qwen2.5-1.5B-Instruct-NLA-L18