rostlabs/rost-286m-diacritics-stripped
286M parameter nanochat GPT (base stage), research tag stripped-d12, checkpoint step 001920.
Part of the rost model zoo — the full set of trained variants behind the rost
research series, published for reproducibility. This is a research model, not a
product.
| |
|---|
| architecture | nanochat GPT, 12 layers, 768 embed, 2048 context |
| tokenizer | included under tokenizer/ (32,768 vocab) |
| training mixture | Romanian with all diacritics stripped |
| stage | base |
| val bpb (own split) | 0.84298 |
Diacritics-study ablation (research tag stripped-d12): the stripped twin of rost-286m-base. The diacritic penalty grows with scale — ~60% at this size — which is why the rost recipe keeps the diacritics.
Validation bits-per-byte is measured on this arm's own validation split and is
not comparable across arms — cross-arm comparisons in the rost write-ups are
always cross-evaluated on identical text.
Licence
CC-BY-NC-4.0, non-commercial, inherited from the most restrictive component
of the training data.
