danish-lm-400m-sft-v29-avg-top7
Danish 400M-parameter SFT model with 2048-token context, uniform-average
of the top-7 checkpoints from the v29 training run (by mean of 4 downstream
evals: gsm8k, sciq, cit-gen, cit-mc).
Base
- Base:
jensjepsen/danish-lm-400m-base-ropext2048-v1
- Tokenizer:
jensjepsen/danish-tokenizer
Training (v29)
vs v28's mix, added 3 task-expansion datasets (RC / reasoning / text-
manipulation) grounded in Danish Wikipedia:
jensjepsen/danish-rc-v1 — 5 subtypes, ~80k rowsjensjepsen/danish-reason-v1 — 6 subtypes, ~120k rowsjensjepsen/danish-textman-v1 — 6 subtypes, ~120k rows
Plus v28's original 13 sources (v25 mix). 16 SFT sources total, ~1.8M packed
training rows, 3 epochs, bs=128, ga=1, lr=3e-5 constant + 500 warmup,
adamw_bnb_8bit, flatten-packing (FA2 varlen), fp32 weights + bf16 compute.
~4.5h on 1×H100.
Avg-top-7 constituents
| step | epoch | agg |
|---|
| 21180 | 1.50 | 0.271 |
| 38830 | 2.75 | 0.252 |
| 28240 | 2.00 | 0.249 |
| 24710 | 1.75 | 0.246 |
| 31770 | 2.25 | 0.245 |
| 42360 | 3.00 | 0.241 |
| 35300 | 2.50 | 0.240 |
Downstream (avg-top-7, full sets)
| eval | v29-avg7 | v29-avg3 | v28-avg3 | v25-avg3 |
|---|
| GSM8K[da] gen | 17.31 | 17.01 | 17.39 | 19.24 |
| SciQ[da] gen | 60.30 | 59.00 | 61.90 | 62.00 |
| Cit-gen | 29.9 | 30.3 | 29.40 | 30.80 |
| Cit-MC | 49.3 | 49.4 | 49.60 | 49.70 |
| mean | 39.20 | 38.93 | 39.57 | 40.44 |
| IFEval-DA prompt-strict | 23.1 | 23.1 | 24.1 | 22.7 |
| IFEval-DA inst-strict | 36.6 | 36.6 | 37.1 | 36.6 |
Marginally below v28-avg-top3 on standard benchmarks. However, qualitative
probes on real Danish Wikipedia articles show v29 handles summarization,
extraction (JSON), style-transfer, genre-transform (tweet/casual/etc.),
and analogy whereas v28-avg-top3 produces empty output or loops on those
same prompts. The task-expansion gains aren't visible in current downstream
benchmarks (which don't measure text↔text transformations) but are real.
