rostlabs/rost-d6-sft
the instruction-tuned chat model, depth d6, checkpoint step 000040.
Trained from scratch on a bilingual Romanian/English mixture. This is a research
model at GPT-2 scale, not a product.
| |
|---|
| architecture | nanochat GPT, d6, 2048 context |
| tokenizer | rostlabs/rost-tok-bilingual, 32,768 vocab, bilingual |
| pretraining mixture | 30% Romanian / 70% English, interleaved at document level |
| Romanian source | FineWeb2-ro, educational score >= 3, diacritic-normalized |
| English source | ClimbMix |
| stage | sft |
The tokenizer is included under tokenizer/ so this repo is self-contained.
Licence
CC-BY-NC-4.0, non-commercial. Both halves of the training data carry
non-commercial terms: ClimbMix is CC-BY-NC-4.0 ("for research and development
only"), and the OpenLLM-Ro SFT datasets are CC-BY-NC-4.0. The model inherits
that. Note that the ClimbMix mirror used is tagged MIT, which does not override
the upstream terms.
Known characteristics
The Romanian pretraining data is roughly one third health content by
character count (measured across three shards of the source), with history,
geography, finance and education each near 10%. Entertainment, gaming and
software are each under 1%. Expect fluency to be uneven across domains
accordingly.
Reproduce
