You can deploy this model and start earning money today!

hhy13/
dr75-coswarmup10-ropefree-am-cartridges

Name: dr75-coswarmup10-ropefree-am-cartridges
Availability: InStock

meta-llama/Llama-3.1-8B-Instruct

finetune

llama3.1

Views

No views yet

dr75_coswarmup10 — rope-free AM cartridge checkpoint sweep

A distillation-step sweep of rope-free, attention-matching-initialized (AM-init) cartridges for meta-llama/Llama-3.1-8B-Instruct. A cartridge is a small trainable KV-cache prefix (compressed context) that is prepended to the model's attention at serve time; these are the dr75_coswarmup10 (cosine LR-warmup, dr=0.75) checkpoints from the Accordion project.

Each step_<N>/ directory is a self-contained, serve-ready cartridge captured at optimizer step N of a single distillation (cycle 0).

Cartridge structure

Every cartridge.pt is a dict of per-layer tensors (32 layers, 8 KV heads, head_dim 128):

key	shape (per layer)	meaning
`trainable_keys` / `trainable_values`	`(1, 8, 65, 128)`	65 trainable KV tokens (the learned cartridge)
`frozen_keys` / `frozen_values`	`(1, 8, 335, 128)`	335 frozen system-prompt KV tokens
`attention_biases`	`(1, 8, 400)`	per-slot key-bias (335 + 65 = 400 total)

Parametrization: rope-free (rope_free_queries: true), AM-init, attention_mode: global_softmax.
Frozen prefix: 335 tokens (frozen system prompt); trainable budget: 65 tokens.
Serving offset: suffix_position_offset: 8733 (see config.yaml / module_state.json).

Training / evaluation setup

Base model: meta-llama/Llama-3.1-8B-Instruct
Data: synthetic cities cross-phase QA (Accordion L=200, s=100), rope-free AM-init t64.
LR schedule: cosine warmup (10%); this is the cosine twin of a linear-warmup run.
Serving: rope-free AM flags — use_unrotated_queries_for_cartridges=True, cartridge_key_bias_enabled=True, cartridge_attention_mode=global_softmax, use_cudagraphs=False.

Checkpoints & served accuracy

Served validation accuracy (n=40 per point) on the cities task, evaluated over a growing acting window. val:baseline = episode 200 (window 100, pre-acting); val:final = episode 600 (window 500, keep-everything L=null).

checkpoint	val:baseline	val:final
`step_0`	0.45	0.50
`step_32`	0.70	0.575
`step_64`	0.85	0.775
`step_96`	0.875	0.825
`step_128`	0.95	0.90
`step_160`	0.95	0.90
`step_192`	0.925	0.95
`step_224`	0.95	0.90
`step_256`	0.95	0.90
`step_288`	0.925	0.90
`step_320`	0.95	0.90

Served quality rises monotonically with step and plateaus by step_128 (fresh-context accuracy ~0.92–0.95). On the long-context val:final metric the cosine-warmup schedule is notably more robust to window-growth context rot than its linear-warmup twin.

Repository layout

step_<N>/
  cartridge.pt        # per-layer KV tensors (see structure above), ~51 MB
  config.yaml         # base model + kv_cache_initializer + serving offset
  module_state.json   # cartridge kind/dims + serving config (rope_free_queries, attention_mode, offset)

Intended use & limitations

Research artifacts. These cartridges are trained on a synthetic cities QA task and are intended for reproducing / studying the Accordion rope-free AM-init sweep — not as a general-purpose model.
They require a cartridge-aware serving stack (the Accordion / cartridges + tokasaurus rope-free path) with the flags listed above; they are not loadable as a plain transformers model.
Derived from meta-llama/Llama-3.1-8B-Instruct; use is subject to the Llama 3.1 license.

hhy13/dr75-coswarmup10-ropefree-am-cartridges