Every model teaches you something.
Some teach optimization.
Some teach patience.
Some teach you that a single misplaced line of code can quietly destroy months of work.
My previous projects taught me all three.
Archaea taught me how to train language models from scratch.
The Ant series taught me that lowering loss does not necessarily mean a model has learned anything useful.
Exp-1 humbled me by exposing a catastrophic implementation bug hidden inside my Transformer blocks.
Archaea-74M-V1.1 reminded me that refining a good model is sometimes more valuable than building a new one.
Rose-Mini begins where those projects ended.
Not with another Transformer.
Not with another checkpoint.
But with a question.
What if I stopped modifying existing architectures and started building one of my own?
That question eventually became Rose X1.
And Rose-Mini is the first public model built upon it.
A Different Kind of Project
Unlike my previous releases, Rose-Mini is not simply another pretrained language model.
It represents my first successful custom architecture.
For a long time I had wanted to move beyond reproducing existing Transformer implementations.
Training models is fun.
Understanding them is even more fun.
Building one yourself...
That is where the real adventure begins.
Rose X1 became my playground for exploring architectural ideas while still remaining practical enough to train on consumer hardware.
Fortunately...
The architecture worked.
Acknowledgements
Before talking about Rose X1, I want to thank Datdanboi25.
The TX4 architecture was one of the first compact language model designs that genuinely made me stop scrolling and start reading code.
Rather than simply scaling parameters, TX4 explored interesting architectural ideas for making smaller language models more capable.
Rose X1 is inspired by many of those ideas while exploring its own design decisions and implementation.
It is not a reproduction of TX4.
It is my own architectural exploration built upon lessons learned from previous projects and inspired by the excellent work shared by the community.
As an independent developer, openly published research and implementations are invaluable learning resources.
So...
Thank you, Datdanboi25.
Without builders willing to share ideas, projects like Rose probably would never have existed.
Meet Rose X1
Rose X1 is a lightweight decoder-only architecture designed around one simple philosophy:
Extract more capability from fewer parameters.
Instead of chasing larger parameter counts, Rose focuses on architectural efficiency and better utilization of compute.
The architecture incorporates several modern design ideas including:
Grouped Query Attention (GQA)
RMSNorm
QK Normalization
Refresh Gates
Efficient decoder-only design
BF16 training
The goal was never to reinvent the Transformer.
The goal was to explore it.
That tiny success was honestly one of the most satisfying moments of the entire project.
Model Overview
Property
Value
Model
Rose-Mini
Architecture
Rose X1
Parameters
49.443 Million
Layers
14
Hidden Size
512
Attention Heads
8
KV Heads
2
QK Normalization
Enabled
Refresh Gates
Layers 4 & 9
Training Tokens
12,206,861,568
Optimizer
Muon AdamW
Training
Rose-Mini was trained on over 12.2 billion tokens, representing my largest pretraining run so far.
Rather than increasing parameter count, I chose to expose a compact model to significantly more data.
Training small language models is an interesting balancing act.
Too few tokens...
The model memorizes very little.
Too many...
You begin wondering whether your SSD has developed feelings. #wirtten by Chat Gpt
Fortunately, both survived.
Comparing My Models
One thing I enjoy about independent research is watching projects evolve over time.
Each model solved a different problem.
Model
Parameters
Tokens
Architecture
Purpose
Archaea-74M
74M
1.23B
LLaMA-style Transformer
First successful pretrained model
Archaea-74M-V1.1
74M
Continued Training
LLaMA-style Transformer
Refinement and benchmark improvements
Rose-Mini
49.4M
12.2B
Rose X1
First custom architecture
Rose-Mini contains roughly 33% fewer parameters than Archaea-74M.
Despite that, it demonstrates that architectural experimentation can remain competitive while dramatically reducing parameter count.
For me, Rose-Mini is less about replacing Archaea.
It is about exploring a different path.
Benchmark Highlights
Rose-Mini demonstrates encouraging performance for a model of its size.
Benchmark
Score
BananaMind Base Bench 1.1
1037.37 Elo
ArithMark-2.0
29.52%
HellaSwag
28.95%
PIQA
58.54%
ARC-Easy
36.62%
ARC-Challenge
24.32%
The BananaMind Base Bench result crossing 1000 Elo was particularly satisfying.
Watching a custom architecture compete after weeks of implementation work is an incredibly rewarding experience.
Performance Compared to Archaea
Rose-Mini was never intended to be a direct replacement for Archaea.
Both models were built with different objectives.
Category
Archaea-74M
Archaea-74M-V1.1
Rose-Mini
Parameters
74M
74M
49.4M
Training Tokens
1.23B
Continued Training
12.2B
Architecture
LLaMA-style
LLaMA-style
Rose X1
Custom Architecture
No
No
Yes
BananaMind
—
936
1037.37 Elo
ArithMark-2.0
29.30%
37.37%
29.52%
Why Rose?
The answer is simple.
Curiosity.
Training another standard Transformer would certainly have been easier.
It also would have been considerably less interesting.
Rose exists because experimentation is how progress happens.
Sometimes the experiment works.
Sometimes it becomes Exp-1.
Fortunately...
This one worked.
Current Limitations
Rose-Mini is still a relatively small language model.
Like every compact model, it has limitations.
Complex reasoning remains challenging.
Long-context understanding can be improved.
Factual consistency still has room for growth.
Instruction following could be further refined.
None of these are surprising.
They're simply opportunities for the next version.
Future Work
Rose-Mini is only the beginning of the Rose family.
Development is already underway for:
Rose-Medium
The goal of Rose-Medium is not simply to increase parameter count.
It is to continue exploring the Rose X1 architecture at a larger scale while preserving the design philosophy that made Rose-Mini successful.
Better architecture.
Better training.
Better models.
Hopefully fewer bugs.
Although I refuse to make any promises regarding that last point.
Final Thoughts
Thank you for checking out Rose-Mini.
I hope you enjoy experimenting with it as much as I enjoyed building it.
