Qwen3-8B gist-token instruction compression (model organism)

• LoRA r=64 alpha=128 on all linear modules + trainable <GIST> embedding row (PEFT trainable_token_indices; the row's trained VALUES are stored in the adapter, so loading does not depend on base-row init).
• Data: Alpaca+ (Self-Instruct + Alpaca, 128k), 3 epochs, eff. batch 128, lr 1e-4 cosine. Prompt format: Instruction: {instruction}\n<GIST>\n[Input: {input}\n]Output:
• Held-out ROUGE-L (gist mask vs full-attention positive control vs no-instruction floor): seen 0.559 / 0.577 / 0.212, unseen 0.548 / 0.557 / 0.252, human 0.300 / 0.309 / 0.149.
• IMPORTANT for inference: the gist behavior assumes the gist mask (post-gist tokens must not attend the instruction). Under plain causal attention the model can still read the instruction directly. Load the adapter UNMERGED (merge_and_unload adds bf16 rounding noise). Mask utilities + eval code: gist_tokens/ in the activation_oracles_dev repo.