Mini NLA Qwen2.5-1.5B
This repository contains a small-scale reimplementation of a natural language autoencoder built around Qwen2.5-1.5B-Instruct. It maps a hidden-state activation to a natural-language explanation with an Activation Verbalizer (AV), then reconstructs the activation with an Activation Reconstructor (AR).
This is a custom PyTorch model bundle, not a standard text-generation checkpoint that can be loaded directly with AutoModelForCausalLM.
Contents
base_model/: complete Qwen2.5-1.5B-Instruct weights, configuration, and tokenizer.av_sft/model.pt: supervised Activation Verbalizer checkpoint.ar_critic/model.pt: final Activation Reconstructor checkpoint.av_rl/model.pt: GRPO-optimized Activation Verbalizer checkpoint.data/: AV, AR, RL, and validation activation datasets.
The activation dimension is 1,536. Training used 1,024 AV examples, 1,024 AR examples, 2,048 RL examples, and 256 validation examples. Natural-language targets were generated with Qwen2.5-32B-Instruct.
Loading
The checkpoints are loaded by the mini_nla implementation:
1from mini_nla.models_av import ActivationVerbalizer
2from mini_nla.utils import load_state_dict_if_exists
3
4av = ActivationVerbalizer("base_model", activation_dim=1536, prefix_tokens=8)
5load_state_dict_if_exists(av, "av_rl")
The AR is loaded similarly with CriticStyleActivationReconstructor.
Limitations
The learned roundtrip has negative held-out FVE and does not outperform mean-activation prediction. AR retains only a small fraction of sample-specific activation variance, and GRPO can exploit this weak reconstructor by converging toward generic explanation templates.
