Views
No views yet
google/gemma-4-26b-a4b-it. It reads the model's own layer-20 residual-stream activations and describes, in natural language, what the model was thinking at those positions — e.g. what it is uncertain about, whether it is second-guessing itself, what it is attending to.Preview checkpoint. This is an early snapshot — training step 3,200 (~9% of one epoch over 3.35M examples). It already shows clear signs of life (see Evaluation) but is not converged; expect it to be superseded by later checkpoints.
" ?" special-token slots — one slot per read position. The AO then generates a free-text read of the underlying state.Layer: 20
? * <num_positions>
<your verbalizer question, e.g. "What is the model uncertain about here?">" ?" slot's input embedding is replaced by the corresponding injected activation (injection hook at layer 1); the AO reads across all slots and answers.google/gemma-4-26b-a4b-it (MoE, 128 experts) — bf16, attn_implementation=sdpa, gradient checkpointing on.r=64, alpha=128, dropout=0.0, on attention (q/k/v/o_proj) + MLP (gate/up/down_proj) across all layers, plus the AO read/inject heads.cds-jb/synthweb-qa-gemma4-26b-a4b + cds-jb/cot-qa-gemma4-26b-a4b, with dense/sparse position alternation — every second sample injects all read positions (capped at 1024), the rest a stochastic ~10-position subsample.2e-4, effective batch 96 (per-rank 4 × grad-accum 3 × 8 GPUs), 1 epoch target (~34,883 steps). Plain HF multi-GPU with manual gradient all-reduce (the MoE router makes standard DDP raise on undefined expert grads).ao_config.json.recog_auc ≈ 0.57 and rising from chance — the AO discriminates the correct read of an activation from a plausible-but-wrong one.1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3from peft import PeftModel
4
5base = "google/gemma-4-26b-a4b-it"
6tok = AutoTokenizer.from_pretrained(base)
7model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
8model = PeftModel.from_pretrained(model, "cds-jb/gemma4-26b-a4b-ao-l20-v1")
9model.eval()" ?" slots (extracting layer-20 residuals from a transcript's read window, then overwriting the slot embeddings) requires the harness — see the reference implementation. The AO is only meaningful with activations injected; run without injection and it has nothing to read.google/gemma-4-26b-a4b-it layer-20 activations; it does not read other models' or other layers' activations.