Almost 100M Parameters, 80B Tokens, and Still Not Quite Satisfied
At some point during the Rose project, I apparently looked at Rose-Mini, thought:
"This is nice."
and then immediately made the completely reasonable decision to make it almost twice as large and feed it 80 billion tokens.
This is Rose-Medium.
A roughly 97.82M parameter model built with the Rose X1 architecture, trained on approximately 80.06B tokens, with a native 2048-token context window.
And before I start writing dramatic sentences about how incredible that sounds, let me be honest:
I expected more.
The model is good enough that I am happy it exists.
It is also stubborn enough that I already have a list of things I want to fix.
So this README is not a victory parade.
It is more like me standing beside the training run with my arms crossed, looking at the benchmark table and saying:
Rose-Medium is the larger sibling of Rose-Mini.
The basic experiment was straightforward: keep the Rose X1 design and give it considerably more capacity and training.
The result is a model with:
97.82M parameters
24 layers
512 hidden dimensions
8 attention heads
2 KV heads
2048-token native context
32,768 vocabulary
80.06B training tokens
The full configuration is included in the repository for anyone who wants to inspect the machinery rather than trusting my suspiciously cheerful description of it.
The Short Version
If you don't want to read the entire model card, here is the situation:
Rose-Medium is better than Rose-Mini.
Good.
Rose-Medium is dramatically better than I expected in absolutely every benchmark.
No.
Rose X1 appears to scale.
Good.
I think the model still has a bottleneck.
Also good, because now I have something to investigate.
Did I miss the BananaMind Bench 1100 club?
Yes.
By 1.3 Elo.
I would like to formally blame the benchmark for this.
The benchmark has declined to comment.
Results at a Glance
Metric
Rose-Mini
Rose-Medium
Parameters
~49.4M
97.82M
Native Context
1024
2048
ArithMark 3
36.50%
38.30%
BananaMind Base Bench 1.1
1037 Elo
1098.7 Elo
The numbers are moving in the correct direction.
They just aren't moving in the direction of my ego fast enough.
Benchmark Results
Rose-Medium was evaluated in 0-shot mode.
Benchmark
Metric
Score
HellaSwag
acc_norm
35.2918%
PIQA
acc_norm
62.9489%
ARC-Easy
acc_norm
44.1919%
ARC-Challenge
acc_norm
26.1945%
WinoGrande
acc
49.8027%
BoolQ
acc
55.6575%
OpenBookQA
acc_norm
32.6000%
CommonsenseQA
acc
21.2121%
LAMBADA
acc
31.7097%
BLiMP
acc
79.2970%
MMLU
acc
23.9781%
WikiText-2
word_perplexity
27.6658
WikiText-2
byte_perplexity
1.8606
SciQ
acc_norm
67.1000%
COPA
acc
69.0000%
RACE
acc
29.0909%
SWAG
acc_norm
52.7942%
TruthfulQA MC2
acc
41.8316%
All supplied evaluation runs completed successfully.
The Numbers I Actually Like
There are some results here that I am genuinely happy about.
BLiMP — 79.30%
That's probably one of the nicest numbers in the table.
COPA — 69.00%
Very respectable.
SciQ — 67.10%
Also encouraging.
PIQA — 62.95%
Another result I am quite happy with.
LAMBADA — 31.71%
For a model this size, I will happily take that.
And then there is SWAG at 52.79%.
Nothing revolutionary, but definitely enough to make me nod at the screen like I knew what I was doing all along.
I absolutely did not.
The Numbers That Made Me Scratch My Head
Now we arrive at my favorite part of machine learning:
The part where the model does something and I have to explain why.
Rose-Medium has almost 98M parameters and has seen approximately 80B tokens.
So naturally I expected some benchmarks to start behaving suspiciously well.
Some did.
Others apparently received the memo and decided to ignore it.
The most interesting example is ARC.
ARC: The Benchmark That Started Asking Questions
Rose-Medium scores:
ARC-Easy: 44.19%
ARC-Challenge: 26.19%
The ARC-Challenge result is encouraging.
The ARC-Easy result is where I start making increasingly serious eye contact with my configuration file.
There is a fairly large gap between them.
That makes me suspect that the model has learned useful patterns and knowledge, but struggles when the task requires reasoning or a more flexible internal representation.
And I have a theory.
A theory.
Not a scientific law.
Not a paper.
Not divine revelation.
A theory.
I Think the Hidden Size May Be Too Small
Rose-Medium uses a 512-dimensional hidden representation.
For a model with 24 layers, that may simply be too narrow.
I may have spent a huge amount of compute making the model deeper while leaving the individual representations with relatively little room to express increasingly complicated information.
Which would be a very me thing to do.
Imagine building a tall building and then discovering halfway through that you made every room the size of a cupboard.
That is currently my suspicion with Rose-Medium.
The model may have more computation available, but not enough representational width to make full use of it.
Again:
This is a hypothesis.
The benchmark results are evidence that make the hypothesis interesting, not proof that it is correct.
Rose Pro gets to help settle the argument.
ArithMark 3
Here is another useful comparison.
Rose-Mini: 36.50%
Rose-Medium: 38.30%
Improvement:
+1.80 percentage points
Is that a huge improvement?
No.
Am I happy it went up?
Yes.
The result suggests that scaling Rose X1 gave Rose-Medium some additional arithmetic capability.
But it also tells me that more parameters alone are not producing the kind of jump I ultimately want.
So, once again, I have homework.
Wonderful.
BananaMind Base Bench 1.1
Now we get to the number that personally offended me.
Rose-Mini
1037 Elo
Rose-Medium
1098.7 Elo
That is approximately:
+61.7 Elo
Which is genuinely a nice improvement.
But there is one tiny problem.
The target was 1100.
Rose-Medium scored:
1098.7
So I missed it by:
1.3 Elo
One point.
And a little bit.
This is possibly the most irritating number in the entire project.
I had almost entered the 1100 club.
I could see the door.
I was basically holding the membership form.
Then the benchmark closed the door.
Fine.
Rose Pro, you know what to do.
Rose-Mini vs Rose-Medium
The important thing is that Rose-Medium does show measurable progress over Rose-Mini.
It is larger.
It has a longer native context.
It performs better on ArithMark 3.
It performs better on BananaMind Base Bench 1.1.
But the improvements are not universally dramatic.
And that is actually useful.
It means the next step should not simply be:
"Make it bigger again."
That would be the easiest possible conclusion.
It is also probably the least interesting one.
Why I Am Not Completely Happy With It
Because I think there is more in there.
The model has:
~97.82M parameters
~80.06B training tokens
2048-token context
24 layers
And yet some of the reasoning scores remain fairly modest.
That makes me wonder whether the model's capacity is being allocated in the most useful way.
The biggest suspect is still the 512 hidden dimension.
Maybe wider is better.
Maybe deeper is better.
Maybe the architecture needs another modification.
Maybe I am completely wrong and the benchmark table is laughing at me.
All are currently possible.
That is why Rose-Medium is useful.
It narrowed the question.
Training
Rose-Medium was trained on approximately:
80.06 billion tokens
The recorded training metadata reports:
Property
Value
Parameters
97.82M
Tokens Seen
80.06B
Recorded Training Steps
38,196
Validation Loss
1.68248
Validation Perplexity
5.37888
Optimizer
muon_adamw
Context Length
2048
The validation numbers are encouraging, but, as usual, I refuse to let a nice perplexity number convince me that the model has achieved enlightenment.
Benchmarks are there for a reason.
Model Configuration
For people who actually want to look under the hood:
That's enough configuration for this README.
If you want every individual tensor and configuration field, the repository is a much better place to look than watching me copy-paste a configuration file into a novel.
What Rose-Medium Taught Me
The main lesson is not:
"More parameters = better model."
That would be too convenient.
The more interesting lesson is:
Scaling exposes bottlenecks.
Rose-Mini showed that Rose X1 could work.
Rose-Medium showed that making it larger improves some capabilities.
It also showed that simply increasing scale does not magically solve harder reasoning.
That is actually a much more useful result.
Now I have a better idea of what I want to investigate next.
And yes, one of those things is width.
Rose Pro
Coming Soon
The next major Rose model will be:
Rose Pro
The goal is to push the architecture further and, hopefully, address some of the limitations exposed by Rose-Medium.
O
ne of the things I particularly want to investigate is whether increasing representational width can unlock more of the capability that I believe is currently being constrained.
And there is one tiny unfinished piece of business:
BananaMind 1100 Elo.
Rose-Medium reached 1098.7.
Rose Pro has been given a very specific assignment.
Final Verdict
So, how do I feel about Rose-Medium?
Honestly?
Pretty happy.
But not satisfied.
The model works.
The scaling worked.
The benchmark profile has several results I genuinely like.
Rose-Medium improved over Rose-Mini.
And the architecture has given me enough evidence to keep pushing it.
But I also think the model should be capable of more.
So no, I am not going to call Rose-Medium a breakthrough.
A good experiment that left me with better questions.
The Rose Family
Rose-Mini
The model that started the Rose X1 experiment.
Rose-Medium
The model that proved the architecture could scale, while also politely pointing out that I probably need to think harder about representation width.
Rose Pro
The next attempt to make me regret writing this README if it performs worse.
That's the scientific method, right?
Closing Note
Rose-Medium is not the end of Rose X1.
If anything, it made the next step more obvious.
I wanted to know whether scaling the architecture would help.
