
Status	🟢 v3 — causality leak fixed. v1 had a routing leak; v3 patches `seg_query = last-pos-of-prev-chunk` + strict triu(diagonal=0) causal mask. CPU causality smoke test verifies future-perturbation invariance on all chunks.
Architecture	DSC = GDN-2 + Dynamic Sparse Caching (chunk c=256, top-k=2 routing, multi-res descriptor [mean/max/softmax-attn])
Parameters	~370 M (GDN-2 370M backbone + small router + combine_alpha)
Training data	FineWeb-Edu sample/100BT (1 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
Loss @ 1B	3.49 (vs vanilla GDN-2 1B = 3.51)
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

1. What is DSC?

DSC (Dynamic Sparse Caching) augments GDN-2 with explicit cached state slots and a learned top-k router that decides which cached slots each token reads from. Goal: give the recurrence a much larger effective state than the per-head S_t ∈ R^{d_k × d_v} matrix alone, while keeping training and inference O(N) in sequence length.

2. What's new in v3

v1 leak mechanism (patched in v3):

seg_query was computed from chunk-mean of the current chunk, allowing future positions inside chunk_i to influence chunk_i's routing query
Causal mask used triu(diagonal=1) which still permitted diagonal (j=i) future leakage

v3 fix:

seg_query = torch.zeros_like(normed_chunks[:, :, 0])
seg_query[:, 1:] = normed_chunks[:, :-1, -1] (use last position of previous chunk as the query for routing decision of current chunk)
Causal mask = torch.triu(ones(N, N, dtype=bool), diagonal=0) — strict upper triangular including diagonal, so future chunks are fully masked

Verification: dsc/scripts/smoke_test_routing_invariance.py confirms that perturbing any future chunk's state does NOT change the current chunk's top-k routing selection or weights (within 1e-6 tolerance).

3. RULER results vs vanilla GDN-2 1B

DSC v3 wins on 6/8 RULER 4K cells, average +31% over vanilla. (Full table lives in docs/DSC_V3_VS_VANILLA_1B_SPEED_LOSS_KO.md.)

4. Known limitations

Trained on 1B tokens only — research-grade, not production-ready
Single seed; multi-seed variance not yet characterized
Multi-key NIAH still hard for v3 (target of v4 entropy bias + learnable summary, in progress)

5. Citation

@misc{dsc-v3-1b-2026,
  author = {LLM-OS-Models},
  title  = {DSC 370M FineWeb-Edu 1B v3 (leak-fixed)},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-1b-v3}
}