README.md

---
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
pipeline_tag: text-generation
---

<img src="Banner.png" width="1300"/>

# Rose-Mini

## The First Bloom

Every model teaches you something.
Some teach optimization.
Some teach patience.
Some teach you that a single misplaced line of code can quietly destroy months of work.

My previous projects taught me all three.

**Archaea** taught me how to train language models from scratch.

The **Ant** series taught me that lowering loss does not necessarily mean a model has learned anything useful.

**Exp-1** humbled me by exposing a catastrophic implementation bug hidden inside my Transformer blocks.

**Archaea-74M-V1.1** reminded me that refining a good model is sometimes more valuable than building a new one.

Rose-Mini begins where those projects ended.
Not with another Transformer.
Not with another checkpoint.
But with a question.

> **What if I stopped modifying existing architectures and started building one of my own?**

That question eventually became **Rose X1**.
And Rose-Mini is the first public model built upon it.

---

# A Different Kind of Project

Unlike my previous releases, Rose-Mini is not simply another pretrained language model.
It represents my first successful custom architecture.
For a long time I had wanted to move beyond reproducing existing Transformer implementations.
Training models is fun.
Understanding them is even more fun.
Building one yourself...
That is where the real adventure begins.
Rose X1 became my playground for exploring architectural ideas while still remaining practical enough to train on consumer hardware.
Fortunately...
The architecture worked.

---

# Acknowledgements

Before talking about Rose X1, I want to thank **Datdanboi25**.
The **TX4** architecture was one of the first compact language model designs that genuinely made me stop scrolling and start reading code.
Rather than simply scaling parameters, TX4 explored interesting architectural ideas for making smaller language models more capable.
Rose X1 is **inspired by** many of those ideas while exploring its own design decisions and implementation.
It is **not** a reproduction of TX4.
It is my own architectural exploration built upon lessons learned from previous projects and inspired by the excellent work shared by the community.
As an independent developer, openly published research and implementations are invaluable learning resources.

So...

Thank you, **Datdanboi25**.
Without builders willing to share ideas, projects like Rose probably would never have existed.

---

# Meet Rose X1

Rose X1 is a lightweight decoder-only architecture designed around one simple philosophy:

> **Extract more capability from fewer parameters.**

Instead of chasing larger parameter counts, Rose focuses on architectural efficiency and better utilization of compute.
The architecture incorporates several modern design ideas including:

* Grouped Query Attention (GQA)
* RMSNorm
* QK Normalization
* Refresh Gates
* Efficient decoder-only design
* BF16 training

The goal was never to reinvent the Transformer.
The goal was to explore it.

## Architecture graph

<a href="https://hfviewer.com/GODELEV/Rose-Mini?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=GODELEV_Rose-Mini_card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=GODELEV%2FRose-Mini&amp;granularity=0"
    alt="Architecture graph for GODELEV/Rose-Mini. Open in hfviewer"
    width="100%"
  />
</a>

---

# The Four Hour Battle

Designing an architecture turned out to be easier than convincing Hugging Face that it actually existed.

Training was straightforward.

Writing the custom Hugging Face implementation...

That was another story.

Since Rose X1 is a custom architecture, I had to implement everything required for Hugging Face compatibility.

Configuration classes.

Model classes.

Registration.

Auto mapping.

Generation support.

Loading.

Saving.

Everything.

I spent nearly **four hours** fighting the modeling file.

Every time I fixed one error...

Python politely introduced two more.

Even AI models tried to help.

They certainly made excellent suggestions.

Unfortunately, they underestimated one critical variable.

**Me.**

I have a remarkable talent for introducing new bugs while fixing old ones.

Some people have green fingers.

Some have sticky fingers.

I have **buggy fingers**.

Eventually...

The architecture loaded successfully.

The checkpoints loaded.

Generation worked.

`AutoModelForCausalLM.from_pretrained()` finally stopped complaining.

That tiny success was honestly one of the most satisfying moments of the entire project.

---

# Model Overview

| Property         | Value          |
| ---------------- | -------------- |
| Model            | Rose-Mini      |
| Architecture     | Rose X1        |
| Parameters       | 49.443 Million |
| Layers           | 14             |
| Hidden Size      | 512            |
| Attention Heads  | 8              |
| KV Heads         | 2              |
| QK Normalization | Enabled        |
| Refresh Gates    | Layers 4 & 9   |
| Training Tokens  | 12,206,861,568 |
| Optimizer        | Muon AdamW     |

---

# Training

Rose-Mini was trained on over **12.2 billion tokens**, representing my largest pretraining run so far.
Rather than increasing parameter count, I chose to expose a compact model to significantly more data.
Training small language models is an interesting balancing act.
Too few tokens...
The model memorizes very little.
Too many...
You begin wondering whether your SSD has developed feelings. #wirtten by Chat Gpt
Fortunately, both survived.

---

# Comparing My Models

One thing I enjoy about independent research is watching projects evolve over time.
Each model solved a different problem.

| Model            | Parameters |             Tokens | Architecture            | Purpose                               |
| ---------------- | ---------: | -----------------: | ----------------------- | ------------------------------------- |
| Archaea-74M      |        74M |              1.23B | LLaMA-style Transformer | First successful pretrained model     |
| Archaea-74M-V1.1 |        74M | Continued Training | LLaMA-style Transformer | Refinement and benchmark improvements |
| **Rose-Mini**    |  **49.4M** |          **12.2B** | **Rose X1**             | First custom architecture             |

Rose-Mini contains roughly **33% fewer parameters** than Archaea-74M.
Despite that, it demonstrates that architectural experimentation can remain competitive while dramatically reducing parameter count.
For me, Rose-Mini is less about replacing Archaea.
It is about exploring a different path.

---

# Benchmark Highlights

Rose-Mini demonstrates encouraging performance for a model of its size.

| Benchmark                     |           Score |
| ----------------------------- | --------------: |
| **BananaMind Base Bench 1.1** | **1037.37 Elo** |
| **ArithMark-2.0**             |      **29.52%** |
| **HellaSwag**                 |      **28.95%** |
| **PIQA**                      |      **58.54%** |
| **ARC-Easy**                  |      **36.62%** |
| **ARC-Challenge**             |      **24.32%** |

The BananaMind Base Bench result crossing **1000 Elo** was particularly satisfying.
Watching a custom architecture compete after weeks of implementation work is an incredibly rewarding experience.

---

# Performance Compared to Archaea

Rose-Mini was never intended to be a direct replacement for Archaea.
Both models were built with different objectives.

| Category            | Archaea-74M |   Archaea-74M-V1.1 |       Rose-Mini |
| ------------------- | ----------: | -----------------: | --------------: |
| Parameters          |         74M |                74M |       **49.4M** |
| Training Tokens     |       1.23B | Continued Training |       **12.2B** |
| Architecture        | LLaMA-style |        LLaMA-style |         Rose X1 |
| Custom Architecture |          No |                 No |         **Yes** |
| BananaMind          |           — |                936 | **1037.37 Elo** |
| ArithMark-2.0       |      29.30% |         **37.37%** |      **29.52%** |

---

# Why Rose?

The answer is simple.

Curiosity.

Training another standard Transformer would certainly have been easier.

It also would have been considerably less interesting.

Rose exists because experimentation is how progress happens.

Sometimes the experiment works.

Sometimes it becomes Exp-1.

Fortunately...

This one worked.

---

# Current Limitations

Rose-Mini is still a relatively small language model.

Like every compact model, it has limitations.

* Complex reasoning remains challenging.
* Long-context understanding can be improved.
* Factual consistency still has room for growth.
* Instruction following could be further refined.

None of these are surprising.

They're simply opportunities for the next version.

---

# Future Work

Rose-Mini is only the beginning of the Rose family.
Development is already underway for:

## Rose-Medium

The goal of Rose-Medium is not simply to increase parameter count.

It is to continue exploring the Rose X1 architecture at a larger scale while preserving the design philosophy that made Rose-Mini successful.

Better architecture.

Better training.

Better models.

Hopefully fewer bugs.

Although I refuse to make any promises regarding that last point.

---

# Final Thoughts

Thank you for checking out Rose-Mini.

I hope you enjoy experimenting with it as much as I enjoyed building it.