
Status	🟢 v4 1B sanity check — gate decision input. Tests whether v4's architectural improvements (entropy bias + attention-pool descriptor) preserve v3's already-good 1B loss before escalating to 5B.
Architecture	DSC = GDN-2 + Dynamic Sparse Caching + Idea 1 (learnable attention-pool `desc_attn_v`) + Idea 2 (`ent_bias_scale` entropy bias on routing logits)
Parameters	~370 M
Training data	FineWeb-Edu sample/100BT (1 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
Loss @ 1B	see `docs/DSC_LIVE_STATUS_KO.md`
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

1. v4 gate criteria

The 1B v4 model is evaluated head-to-head against the 1B v3 model on RULER 4K/16K × 4 tasks (8 cells). Gate passes if:

v4 loss ≤ v3 loss + 0.05
v4 ≥ v3 - 0.02 in ≥ 5/8 cells
v4 4K single_1 ≥ 0.30 (collapse floor)

PASS → escalate to v4 5B head-to-head against v3 5B. FAIL → v3 remains the campaign champion, v4 not escalated.

2. What's new in v4

Two architectural improvements on top of v3's leak-fixed foundation:

Learnable attention-pool descriptor (desc_attn_v): replaces the fixed [mean/max/softmax-attn] triplet with a learned attention pool so the descriptor can highlight key-relevant features per chunk.
Entropy bias on routing logits (ent_bias_scale): adds a bias proportional to batch-level routing entropy, letting the router sharpen or smooth its decisions dynamically.

3. Citation

@misc{dsc-v4-1b-2026,
  author = {LLM-OS-Models},
  title  = {DSC 370M FineWeb-Edu 1B v4 sanity check},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/dsc-370m-fineweb-edu-1b-v4}
}