Agents spend much of their latency decoding tool calls token by token, yet the
calls a user needs are highly repetitive across sessions. SpecMem turns
that repetition into speed: it keeps a small, per-user, capacity-bounded memory
of past tool calls that is updated live as the agent runs, and retrieves the
closest past call as a draft for speculative decoding. The served model
verifies the draft, so outputs are exactly those of standard decoding — wrong
drafts cost only compute, never correctness.
The key finding is that liveness drives the gain: a store that keeps
ingesting and evicting stays fresh as the query distribution drifts, while a
frozen datastore (built once, then fixed) degrades over sessions.
Highlights
Live per-user memory as a drafter. Top-1 cosine retrieval over
lightweight query embeddings (all-MiniLM-L6-v2, CPU), per-user
partitioning, LRU eviction at a small fixed capacity, online write-back
after every verified call. No training, no extra GPU.
End-to-end wall-clock speedups of 1.62x / 1.74x / 1.18x / 1.69x over
vanilla autoregressive decoding on API-Bank, ToolAlpaca, BFCL v4, and
ToolBench, matching or exceeding a faithful frozen-datastore
(ToolSpec-style) baseline on all four.
Verified across serving stacks and architectures:gpt-oss-120b (MoE)
and gemma-4-31B-it (dense) on sglang, Nemotron-3-Super-120B
(hybrid-SSM) on vLLM — anything with an OpenAI-compatible endpoint works.
Safety-aware speculation: an idempotency gate defers speculative
execution of irreversible tools (payments, deletes), keeping the speedup
while avoiding side effects a verifier cannot undo.
How it works
user query ──> embed ──> per-user memory (capacity-bounded, LRU)
│ top-1 cosine ≥ τ
▼
drafted tool call ──> served model verifies
▲ (token-level accept)
│
write-back of the verified call (live update)
Every query is answered once by the served model with greedy decoding (the
target). Each memory policy ("arm") drafts from its own store and is scored
by the token-level longest common prefix between its draft and the target —
the accepted-token count a speculative decoder would realize. The compared
arms:
Arm (code name)
Description
no_memory
schema-only draft; lower bound
static_global
one global store frozen after warmup (ToolSpec-style)
Benchmark data is downloaded from the official sources, never redistributed
here; two datasets need a small manual step (ToolBench, tau2-bench) — see
data/README.md.
Quickstart
1. Serve a tool-calling model behind any OpenAI-compatible endpoint, e.g.
Targets are cached by exact query string, so re-runs and all memory-arm
replays are GPU-free. Results land in results/main_accept_results.json
(per-session and overall MAT / accepted fraction / exact rate).
Add --benchmark sealtools for Seal-Tools. For other served models, point
--url/--model at the endpoint and --model-path (or the
SPECMEM_TOKENIZER env var) at the model's tokenizer so acceptance is
measured in that model's own tokens.
Reproducing the paper
Experiment
Command
Main acceptance table (BFCL / Seal-Tools)
python -m harness.run_accept ... (above)
4-benchmark main table + wall-clock speedups
python -m harness.phase4_maintable
Freshness-over-sessions curve
python -m harness.phase4_partb, then python -m harness.phase4_freshness_fig
The tau2-bench generate mode runs the served model as the agent against a
live GPT-4.1 user simulator and requires OPENAI_API_KEY (and optionally
OPENAI_BASE_URL) in the environment, plus a
tau2-bench install
(TAU2_BIN, TAU2_DATA_DIR). Credentials are read from environment
variables only and a leak check aborts if a key ever appears in an artifact.
Repository layout
harness/ all experiment code (run as python -m harness.<module>)
memory.py memory arms: NoMemory, StaticGlobal, PersonalMemory (SpecMem),
ToolSpecBaseline, suffix-decoding baseline
simulate.py multi-session, multi-user query-stream generator
data.py benchmark loaders (BFCL, Seal-Tools, ToolAlpaca, API-Bank,
ToolBench, tau2)
client.py OpenAI-compatible client + tool-call parsers (harmony, XML)
metrics.py canonicalization + token-LCP acceptance scoring
run_accept.py main 3-arm acceptance experiment
... see the table above for the per-experiment entry points
scripts/ data download
data/ benchmark data (downloaded; see data/README.md)
results/ experiment outputs (created at runtime)
Environment variables
Variable
Purpose
Default
TOOL_SERVER_URL
served-model endpoint
http://localhost:30000/v1
SPECMEM_TOKENIZER
tokenizer for the accept metric
openai/gpt-oss-120b
OPENAI_API_KEY / OPENAI_BASE_URL
tau2 user-simulator credentials
—
TAU2_BIN / TAU2_DATA_DIR
tau2-bench CLI and data locations
tau2 / —
Citation
The paper is currently under review. If you use this code, please cite:
bibtex
1@article{specmem2026,
2 title = {SpecMem: Accelerating Agentic Tool Calling via Live Memory Management},
3 author = {Anonymous},
4 note = {Under review},
5 year = {2026}
6}
License
This repository is released under the Apache License 2.0.
Benchmark datasets and served models keep their own licenses (see
data/README.md).
