README.md

---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- causal-lm
- custom-architecture
- rose-x1
- rose-medium
- pytorch
- transformers
library_name: transformers
---
<img src="Banner.png" width="1920"/>

# Rose-Medium

## Almost 100M Parameters, 80B Tokens, and Still Not Quite Satisfied

At some point during the Rose project, I apparently looked at **Rose-Mini**, thought:

> "This is nice."

and then immediately made the completely reasonable decision to make it almost twice as large and feed it **80 billion tokens**.

This is **Rose-Medium**.

A roughly **97.82M parameter** model built with the Rose X1 architecture, trained on approximately **80.06B tokens**, with a native **2048-token context window**.

And before I start writing dramatic sentences about how incredible that sounds, let me be honest:

**I expected more.**

The model is good enough that I am happy it exists.
It is also stubborn enough that I already have a list of things I want to fix.
So this README is not a victory parade.
It is more like me standing beside the training run with my arms crossed, looking at the benchmark table and saying:

> "Okay. That's actually pretty good."

Then five seconds later:

> "But why is ARC-Easy doing that?"

---

## Architecture graph

<a href="https://hfviewer.com/GODELEV/Rose-Medium?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=GODELEV_Rose-Medium_card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=GODELEV%2FRose-Medium&amp;granularity=0"
    alt="Architecture graph for GODELEV/Rose-Medium. Open in hfviewer"
    width="100%"
  />
</a>

# What Rose-Medium Actually Is

Rose-Medium is the larger sibling of **Rose-Mini**.
The basic experiment was straightforward: keep the Rose X1 design and give it considerably more capacity and training.
The result is a model with:

- **97.82M parameters**
- **24 layers**
- **512 hidden dimensions**
- **8 attention heads**
- **2 KV heads**
- **2048-token native context**
- **32,768 vocabulary**
- **80.06B training tokens**

The full configuration is included in the repository for anyone who wants to inspect the machinery rather than trusting my suspiciously cheerful description of it.

---

# The Short Version

If you don't want to read the entire model card, here is the situation:

**Rose-Medium is better than Rose-Mini.**

Good.

**Rose-Medium is dramatically better than I expected in absolutely every benchmark.**

No.

**Rose X1 appears to scale.**

Good.

**I think the model still has a bottleneck.**

Also good, because now I have something to investigate.

**Did I miss the BananaMind Bench 1100 club?**

Yes.

By **1.3 Elo**.

I would like to formally blame the benchmark for this.

The benchmark has declined to comment.

---

# Results at a Glance

| Metric | Rose-Mini | Rose-Medium |
|---|---:|---:|
| **Parameters** | ~49.4M | **97.82M** |
| **Native Context** | 1024 | **2048** |
| **ArithMark 3** | 36.50% | **38.30%** |
| **BananaMind Base Bench 1.1** | 1037 Elo | **1098.7 Elo** |

The numbers are moving in the correct direction.

They just aren't moving in the direction of my ego fast enough.

---

# Benchmark Results

Rose-Medium was evaluated in **0-shot** mode.

| Benchmark | Metric | Score |
|---|---|---:|
| **HellaSwag** | acc_norm | **35.2918%** |
| **PIQA** | acc_norm | **62.9489%** |
| **ARC-Easy** | acc_norm | **44.1919%** |
| **ARC-Challenge** | acc_norm | **26.1945%** |
| **WinoGrande** | acc | **49.8027%** |
| **BoolQ** | acc | **55.6575%** |
| **OpenBookQA** | acc_norm | **32.6000%** |
| **CommonsenseQA** | acc | **21.2121%** |
| **LAMBADA** | acc | **31.7097%** |
| **BLiMP** | acc | **79.2970%** |
| **MMLU** | acc | **23.9781%** |
| **WikiText-2** | word_perplexity | **27.6658** |
| **WikiText-2** | byte_perplexity | **1.8606** |
| **SciQ** | acc_norm | **67.1000%** |
| **COPA** | acc | **69.0000%** |
| **RACE** | acc | **29.0909%** |
| **SWAG** | acc_norm | **52.7942%** |
| **TruthfulQA MC2** | acc | **41.8316%** |

All supplied evaluation runs completed successfully.

---

# The Numbers I Actually Like

There are some results here that I am genuinely happy about.

**BLiMP — 79.30%**

That's probably one of the nicest numbers in the table.

**COPA — 69.00%**

Very respectable.

**SciQ — 67.10%**

Also encouraging.

**PIQA — 62.95%**

Another result I am quite happy with.

**LAMBADA — 31.71%**

For a model this size, I will happily take that.

And then there is **SWAG at 52.79%**.

Nothing revolutionary, but definitely enough to make me nod at the screen like I knew what I was doing all along.
I absolutely did not.

---

# The Numbers That Made Me Scratch My Head

Now we arrive at my favorite part of machine learning:
**The part where the model does something and I have to explain why.**
Rose-Medium has almost **98M parameters** and has seen approximately **80B tokens**.
So naturally I expected some benchmarks to start behaving suspiciously well.

Some did.

Others apparently received the memo and decided to ignore it.
The most interesting example is ARC.

---

# ARC: The Benchmark That Started Asking Questions

Rose-Medium scores:

- **ARC-Easy: 44.19%**
- **ARC-Challenge: 26.19%**

The ARC-Challenge result is encouraging.

The ARC-Easy result is where I start making increasingly serious eye contact with my configuration file.

There is a fairly large gap between them.

That makes me suspect that the model has learned useful patterns and knowledge, but struggles when the task requires reasoning or a more flexible internal representation.

And I have a theory.

A theory.

Not a scientific law.

Not a paper.

Not divine revelation.

A theory.

## I Think the Hidden Size May Be Too Small

Rose-Medium uses a **512-dimensional hidden representation**.

For a model with 24 layers, that may simply be too narrow.

I may have spent a huge amount of compute making the model deeper while leaving the individual representations with relatively little room to express increasingly complicated information.

Which would be a very me thing to do.

Imagine building a tall building and then discovering halfway through that you made every room the size of a cupboard.

That is currently my suspicion with Rose-Medium.

The model may have more computation available, but not enough representational width to make full use of it.

Again:

**This is a hypothesis.**

The benchmark results are evidence that make the hypothesis interesting, not proof that it is correct.
Rose Pro gets to help settle the argument.

---

# ArithMark 3

Here is another useful comparison.

**Rose-Mini:** 36.50%
**Rose-Medium:** 38.30%

Improvement:
**+1.80 percentage points**
Is that a huge improvement?
No.
Am I happy it went up?
Yes.

The result suggests that scaling Rose X1 gave Rose-Medium some additional arithmetic capability.

But it also tells me that **more parameters alone are not producing the kind of jump I ultimately want**.

So, once again, I have homework.

Wonderful.

---

# BananaMind Base Bench 1.1

Now we get to the number that personally offended me.
### Rose-Mini

**1037 Elo**

### Rose-Medium

**1098.7 Elo**

That is approximately:

**+61.7 Elo**

Which is genuinely a nice improvement.

But there is one tiny problem.

The target was **1100**.

Rose-Medium scored:

**1098.7**

So I missed it by:
**1.3 Elo**

One point.

And a little bit.

This is possibly the most irritating number in the entire project.

I had almost entered the 1100 club.

I could see the door.

I was basically holding the membership form.

Then the benchmark closed the door.

Fine.

**Rose Pro, you know what to do.**

---

# Rose-Mini vs Rose-Medium

> **<img src="Comparison.png" width="1600"/>**

The important thing is that Rose-Medium does show measurable progress over Rose-Mini.

It is larger.

It has a longer native context.

It performs better on **ArithMark 3**.

It performs better on **BananaMind Base Bench 1.1**.

But the improvements are not universally dramatic.

And that is actually useful.

It means the next step should not simply be:

> "Make it bigger again."

That would be the easiest possible conclusion.

It is also probably the least interesting one.

---

# Why I Am Not Completely Happy With It

Because I think there is more in there.

The model has:

**~97.82M parameters**

**~80.06B training tokens**

**2048-token context**

**24 layers**

And yet some of the reasoning scores remain fairly modest.

That makes me wonder whether the model's capacity is being allocated in the most useful way.

The biggest suspect is still the **512 hidden dimension**.

Maybe wider is better.

Maybe deeper is better.

Maybe the architecture needs another modification.

Maybe I am completely wrong and the benchmark table is laughing at me.

All are currently possible.

That is why Rose-Medium is useful.

It narrowed the question.

---

# Training

Rose-Medium was trained on approximately:

**80.06 billion tokens**

The recorded training metadata reports:

| Property | Value |
|---|---:|
| **Parameters** | 97.82M |
| **Tokens Seen** | 80.06B |
| **Recorded Training Steps** | 38,196 |
| **Validation Loss** | 1.68248 |
| **Validation Perplexity** | 5.37888 |
| **Optimizer** | `muon_adamw` |
| **Context Length** | 2048 |

The validation numbers are encouraging, but, as usual, I refuse to let a nice perplexity number convince me that the model has achieved enlightenment.

Benchmarks are there for a reason.

---

# Model Configuration

For people who actually want to look under the hood:

```text
Architecture:       Rose X1
Parameters:         97.82M
Vocabulary:         32,768

Hidden Size:        512
Intermediate Size:  1,728
Layers:             24

Attention Heads:    8
KV Heads:           2
Head Dimension:     64

QK-Norm:            Enabled
RoPE:               Enabled
RoPE Theta:         100,000

Context Length:     2,048

Refresh Gate:       Enabled
Inject Layers:      9, 18
Kernel Size:        9

Tied Embeddings:    Yes
Optimizer:          Muon AdamW
```

That's enough configuration for this README.
If you want every individual tensor and configuration field, the repository is a much better place to look than watching me copy-paste a configuration file into a novel.

---

# What Rose-Medium Taught Me

The main lesson is not:

**"More parameters = better model."**

That would be too convenient.

The more interesting lesson is:

**Scaling exposes bottlenecks.**

Rose-Mini showed that Rose X1 could work.

Rose-Medium showed that making it larger improves some capabilities.

It also showed that simply increasing scale does not magically solve harder reasoning.

That is actually a much more useful result.

Now I have a better idea of what I want to investigate next.

And yes, one of those things is **width**.

---

# Rose Pro

## Coming Soon

The next major Rose model will be:

# Rose Pro

The goal is to push the architecture further and, hopefully, address some of the limitations exposed by Rose-Medium.
O
ne of the things I particularly want to investigate is whether increasing representational width can unlock more of the capability that I believe is currently being constrained.

And there is one tiny unfinished piece of business:

**BananaMind 1100 Elo.**

Rose-Medium reached **1098.7**.

Rose Pro has been given a very specific assignment.

---

# Final Verdict

So, how do I feel about Rose-Medium?

Honestly?

**Pretty happy.**

But not satisfied.

The model works.

The scaling worked.

The benchmark profile has several results I genuinely like.

Rose-Medium improved over Rose-Mini.

And the architecture has given me enough evidence to keep pushing it.

But I also think the model should be capable of more.

So no, I am not going to call Rose-Medium a breakthrough.

**A good experiment that left me with better questions.**

---

# The Rose Family

**Rose-Mini**
The model that started the Rose X1 experiment.

**Rose-Medium**
The model that proved the architecture could scale, while also politely pointing out that I probably need to think harder about representation width.

**Rose Pro**

The next attempt to make me regret writing this README if it performs worse.
That's the scientific method, right?

---

# Closing Note

Rose-Medium is not the end of Rose X1.

If anything, it made the next step more obvious.

I wanted to know whether scaling the architecture would help.

It did.

Then I wanted the model to perform even better.

It didn't quite cooperate.

So now I have another problem.

Which is fine.

Rose-Medium did its job.

It improved.

It taught me something.

And now I have to build Rose Pro.

**See you there.**