Mini NLA Qwen2.5-3B
This repository contains a small-scale natural language autoencoder built around Qwen2.5-3B-Instruct. The Activation Verbalizer (AV) maps a hidden-state activation to a natural-language explanation, and the Activation Reconstructor (AR) maps that explanation back into the activation space.
This is a custom PyTorch model bundle, not a standard text-generation checkpoint that can be loaded directly with AutoModelForCausalLM.
Contents
base_model/: complete Qwen2.5-3B-Instruct weights, configuration, and tokenizer.av_sft/model.pt: supervised Activation Verbalizer.ar_critic/model.pt: jointly optimized Activation Reconstructor from GRPO step 1,280.av_rl/model.pt: GRPO Activation Verbalizer from step 1,280.data/: AV, AR, RL, and validation activation datasets.validation_fve_by_step.json: held-out FVE for the saved GRPO checkpoints.
The activation dimension is 2,048. Training used 1,024 AV examples, 1,024 AR examples, 2,048 RL examples, and 256 validation examples. Natural-language targets were generated with Qwen2.5-32B-Instruct.
Results
The supervised roundtrip obtained FVE -0.2524. The best paired GRPO checkpoint occurred at step 1,280 with FVE -0.1677. Training through step 2,048 produced FVE -0.1724, so this bundle uses the better step-1,280 checkpoint.
Loading
The checkpoints are loaded by the accompanying mini_nla implementation:
1from mini_nla.models_av import ActivationVerbalizer
2from mini_nla.utils import load_state_dict_if_exists
3
4av = ActivationVerbalizer("base_model", activation_dim=2048, prefix_tokens=8)
5load_state_dict_if_exists(av, "av_rl")
The AR is loaded similarly with CriticStyleActivationReconstructor.
Limitations
The learned roundtrip still has negative held-out FVE and does not outperform mean-activation prediction. The result indicates improved reconstruction with model scaling, but it is not evidence of a reliable scaling law.
