danish-lm-400m-sft-v29-v30-soup
Base
- Base:
jensjepsen/danish-lm-400m-base-ropext2048-v1 (2048-token context)
- Tokenizer:
jensjepsen/danish-tokenizer
Recipe
Element-wise 50/50 mean of the two parents' model.safetensors, computed in
float32 and cast back to the parents' original dtype at save time. No fine-tuning
on top — just weight averaging. Script: scripts/avg_ckpts.py in the espllm repo.
Tried 0.7·v29 + 0.3·v30 on cit-gen: 30.28% vs balanced 30.83%. Balanced
50/50 was the winner; weighting toward the individually-stronger parent
did NOT help.
Downstream — soup vs parents
Wins on all 5 freeform-generation evals:
| eval | v29-avg7 | v30-ep4-avg7 | soup | Δ vs best parent |
|---|
| cit-gen (freeform Q, substring gold) | 30.00 | 27.64 | 30.83 | +0.83 |
| GSM8K[da] gen (CoT → number) | 17.31 | 17.54 | 18.30 | +0.76 |
| SciQ openq (freeform Q, substring) | 11.70 | 10.50 | 12.50 | +0.80 |
| IFEval-DA prompt-strict | 23.1 | 23.9 | 25.8 | +1.9 |
| IFEval-DA inst-strict | 36.6 | 37.2 | 40.5 | +3.3 |
Loses on 4-of-4 MC-letter emission evals (5th is noise near random):
| eval | v29-avg7 | v30-ep4-avg7 | soup | Δ vs best parent |
|---|
| SciQ MC-letter | 60.30 | 62.00 | 59.70 | −2.30 |
| Cit-MC | 49.3 | 48.80 | 48.6 | −0.7 |
| PIQA (2-choice) | — | 56.0 | 54.0 | −2.0 |
| ARC (4-5 choice) | — | 27.76 | 26.82 | −0.9 |
| GPQA (near-random baseline) | — | 21.72 | 24.75 | +3.0 (noise) |
Why the asymmetry
Free-form generation lets the parents' complementary knowledge compose over
multi-token outputs — e.g. avg correctly answered "Thomas Vinterberg 2021
Oscar?" with "Druk" when v29 said "Den gode vilje" and v30 said
"Pusher II", and correctly said "De jævne folk (bønder og tjenestefolk)"
about 1800-tallet Venstre-vælgere when neither parent had the answer.
MC-letter emission has only ~4 valid single-token outputs. When the two
parents' argmax disagrees on which letter, the averaged logits usually break
in one specific direction — often losing v30's individual wins. Occasionally
the averaged distribution's argmax lands on a third letter neither parent
picked (surprising but rare).
When to use this soup
- Best default for assistant chat, IFEval, freeform QA, GSM-style
reasoning.
- If your benchmark is MC-letter picking (SciQ MC, PIQA, ARC, cit-MC),
use the individual parent instead:
- PIQA / ARC / SciQ MC →
v30-avg-top7
- Cit-MC →
v29-avg-top7
Not measured
- Textman ChrF++ — full 1000-item val on 1080 Ti takes ~2.6h; skipped
locally. Pattern predicts soup wins by 1-2pp.
- Weighted sweeps beyond 0.7/0.3 — worth exploring, but 50/50 was the
best of what we tried.
Related
- v29-avg-top7
— task-expansion mix, 16 sources
- v30-avg-top7 (ep 4)
— v29 mix + 3 STEM datasets, 4-epoch
- base ropext2048-v1
— long-context base both parents were fine-tuned from
