I trained a model, watched the numbers climb,
And told myself, "This one will be prime."
I widened the model, increased the scale,
Then watched ArithMark politely fail.
Eighty billion tokens went into the machine,
Surely this time my model would be clean.
The Benchmarks smiled, the scores came through,
And said, "Nice little model. Here's some work for you."
Every time I think I've finally learned,
Another benchmark shows where I was wrong.
Truly, I am blessed beyond compare; I keep discovering mistakes I can afford to train.
Introduction
Rose Pro is the latest model in the Rose X1 line, with approximately 151.3 million parameters.
It keeps the same general Rose X1 architecture while increasing the model's capacity, using 24 layers and a 640-dimensional hidden representation. The model was trained on approximately 80 billion tokens.
And yes, I had expectations.
Quite reasonable ones, obviously.
I had a bigger model, more capacity, a mountain of tokens, and enough confidence to become suspicious.
Then the Benchmarks arrived.
It did not exactly hand me a trophy.
It did, however, hand me something much more useful:
another collection of things to learn.
Model Overview
Property
Value
Model
Rose Pro
Architecture
Rose X1
Parameters
151.274M
Layers
24
Hidden Size
640
Attention Heads
10
KV Heads
10
QK Norm
Yes
Refresh Gates
[9, 18]
Native Context Window
2048
Training Tokens
79,967,283,200
Validation Perplexity
5.08
Optimizer
muon_adamw
The architecture itself is intentionally not explained in excessive detail here. Rose Pro remains part of the Rose X1 family, and the interesting part of this release is less about repeating the architecture diagram and more about seeing what the model actually does with its increased capacity.
The Chemistry of Benchmarks
This is probably my favorite part of the whole release.
The Benchmarks are strange little ecosystem.
You put a model into it with millions of parameters, billions of tokens, a carefully designed architecture, and a considerable amount of optimism.
Then the Benchmarks removes the optimism and gives you percentages.
And suddenly you discover that one model is great at something, another model is better somewhere else, and a benchmark you thought would behave nicely has decided to personally insult you.
That is the chemistry of Becnhmarks.
Rose Pro does not dominate every benchmark, and I don't think it needs to.
What matters more is the overall shape of the results and what those results teach me about the model.
Benchmark Results
The following results are from 0-shot evaluation.
Benchmark
Rose-Mini
Rose-Medium
Rose Pro
HellaSwag
28.95%
35.29%
38.24%
PIQA
58.54%
62.95%
65.18%
ARC-Easy
36.62%
44.19%
48.65%
ARC-Challenge
24.32%
26.19%
26.96%
WinoGrande
51.54%
49.80%
50.43%
BoolQ
61.90%
55.66%
60.40%
OpenBookQA
29.40%
32.60%
33.60%
CommonsenseQA
19.74%
21.21%
19.49%
LAMBADA
22.03%
31.71%
32.99%
BLiMP
76.72%
79.30%
79.26%
MMLU
23.24%
23.98%
26.13%
SciQ
55.90%
67.10%
68.80%
COPA
59.00%
69.00%
69.00%
RACE
27.18%
29.09%
30.24%
SWAG
44.32%
52.79%
55.39%
TruthfulQA MC2
44.28%
41.83%
39.82%
WikiText-2 Word PPL
46.05
27.67
25.05
WikiText-2 Byte PPL
2.047
1.861
1.826
The results show a mixed but useful picture. Rose Pro moves strongly on several tasks while some benchmarks barely move and a few go backwards.
Which is precisely why I don't want to turn this README into a victory speech.
The model is good at some things.
It is still learning how to behave itself at others.
ArithMark 3 & BananaMind
These two deserve their own little section.
Model
ArithMark 3
BananaMind Bench 1.1
Rose-Mini
36.50%
1037.37 Elo
Rose-Medium
38.30%
1098.7 Elo
Rose Pro
38.20%
1105 Elo
ArithMark 3
Rose Pro scores 38.20% on ArithMark 3.
Rose-Medium scored 38.30%.
So I increased the model.
I increased the width.
And ArithMark looked at all of that and said:
38.20%.
Thank you.
Very informative.
But that is also exactly why I find these evaluations useful. Parameter count does not magically translate into proportional improvement on every task.
Apparently I have to actually understand what is happening.
How inconvenient.
BananaMind Bench 1.1
And then there is the number I had been waiting for.
Rose-Medium: 1098.7 Elo
Rose Pro: 1105 Elo
I finally crossed 1100 Elo.
Is 1100 a scientifically meaningful boundary?
No.
Did 1098.7 bother me enough that I wanted to cross it?
Absolutely.
So I am going to enjoy this completely arbitrary victory.
1105.
Beautiful.
What I Learned This Time
One thing has remained remarkably consistent throughout my model-building journey:
Every time I train a model, I learn something.
Sometimes it is about architecture.
Sometimes it is about optimization.
Sometimes it is about evaluation.
And sometimes it is about a mistake that I would very much like to pretend I never made.
But every model leaves me with something new to investigate.
That is probably the most valuable part of building these models.
I start with an idea.
I train it.
I evaluate it.
Then the results politely inform me that my original understanding was only partially correct.
And honestly?
I think I am blessed.
Not because every model performs exactly how I imagined.
Quite the opposite.
I am blessed because every training run gives me another opportunity to discover something I didn't know before.
Apparently this is my preferred method of education.
It is just considerably more expensive than reading a book.
Training Recipe: One More Thing to Change
The Rose models so far have been trained using cosine learning-rate scheduling.
For the next generation, I want to try something different.
The upcoming models will explore WSD learning-rate scheduling.
Not because cosine suddenly became illegal.
I simply have another question I want to answer.
And apparently the only way I know how to answer questions is to build another model.
What's Next?
The Rose X1 journey is not finished.
The next step will be the Rose 1.5 series, which will explore further changes to the training recipe and continue the work started with the current Rose models.
Hopefully, the next leaderboard visit will contain fewer surprises.
Knowing me, that statement probably guarantees the opposite.
Final Thoughts
Rose Pro is not exactly the model I imagined when I started building it.
Some results are encouraging.
Some are competitive.
Some are stubborn.
And some are sitting there quietly reminding me that adding more parameters is not the same thing as solving the problem.
But that is fine.
The point of every model is not to prove that I was right.
Sometimes the point is to find out where I was wrong.
