
Architecture	GatedDeltaNet-2 (Yang et al. 2025), 370M parameter variant
Parameters	370 M
Training data	FineWeb-Edu sample/100BT (1 B-token slice)
Tokenizer	TinyLlama v1.1 (vocab = 32 000)
Context length	4 096 (training)
Hardware	8 × NVIDIA H200 141 GB (FSDP)
Loss @ 1B	3.51
License	Apache-2.0
Trained by	LLM-OS-Models · code at gyunggyung/long-gdn

1. Purpose

Vanilla GDN-2 370M baseline trained on the same 1B-token slice as dsc-370m-fineweb-edu-1b-v3, so that DSC vs vanilla comparison is apples-to-apples (same data, same compute, same hyperparameters).

2. RULER results (vs DSC v3 1B)

DSC v3 wins on 6/8 RULER 4K cells, average +31%. This vanilla checkpoint is the floor that DSC improvement is measured against. Full table in docs/DSC_V3_VS_VANILLA_1B_SPEED_LOSS_KO.md.

3. Known limitations

1B tokens is far below convergence — research-grade only
Long-context (≥16K) retrieval fails completely (vanilla linear attention state collapse), which is the gap DSC is designed to close

4. Citation

@misc{gdn2-vanilla-1b-2026,
  author = {LLM-OS-Models},
  title  = {GDN-2 370M FineWeb-Edu 1B vanilla baseline},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-1b}
}