You can deploy this model and start earning money today!

EmmaLeonhart/
reservoir-agent-gpt2-crosspass

openai-community/gpt2

finetune

mit

Views

No views yet

Reservoir Agent — GPT-2 cross-pass recall

A GPT-2 (124M) base model with a fixed, randomly-initialized echo-state reservoir injected as content-addressable KV-prefix tokens, trained to recall information that exists only in the reservoir state carried across forward passes.

This is the artifact behind the cross-pass result of the Reservoir Agent study. It is a feasibility demonstration at small scale, not a general-purpose chat model.

What it does

The task is one a stateless transformer structurally cannot do:

Pass 1 — "The secret word is <KEY>." (the reservoir ticks, encoding the key).
The context is wiped.
Pass 2 — "The secret word was" → the model must output <KEY>, which is present only in the reservoir state carried from pass 1.

Model	Cross-context recall (6 single-token keys)
Reservoir (stateful)	1.00 (6/6), training loss → 0.01
Stateless baseline (reservoir wiped between passes)	0.17 (chance, 1/6)

The carried reservoir state, made attendable (rather than added as a bias), lets the model recall content from a context it can no longer see.

What is in this repo

config.json — architecture config to rebuild the model. The reservoir (W_r, W_in) is fixed-random from seed, so it regenerates byte-identically and is not stored.
reservoir_readout.npz — the trained W_res (reservoir → prefix tokens) weights.
adapter/ — the trained LoRA adapter (peft).

How to use

Requires the project code (pip install -e ".[models]"):

python

from huggingface_hub import snapshot_download
from reservoir.persist import load_reservoir_model

path = snapshot_download("EmmaLeonhart/reservoir-agent-gpt2-crosspass")
lm = load_reservoir_model(path)

tok = lm.tokenizer
p1 = tok("The secret word is blue.", return_tensors="pt").to(lm.device)
p2 = tok("The secret word was", return_tensors="pt").to(lm.device)
lm.reset_state()
lm.forward_logits(p1["input_ids"], p1["attention_mask"])     # pass 1: read the key
logits = lm.forward_logits(p2["input_ids"], p2["attention_mask"])  # context wiped
print(tok.decode(logits[0, -1].argmax()))                    # -> " blue"

Reproduce the training:

python scripts/run.py crosspass --model gpt2 --mode kv --steps 600 --save out_dir

Scope and limitations

GPT-2 small, a narrow probe task (6 single-token colour keys). It demonstrates the mechanism — that a carried, attendable reservoir state changes what the model can do — not a broad capability.
It does not transfer to Hermes-3B yet. On Hermes the same setup recalls at chance; that is an open optimization problem documented in the study, not a solved result.
This is research code under active development. See the repo for the full findings, caveats, and the theory.