| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - custom-architecture |
| - rose-x1 |
| - rose-medium |
| - pytorch |
| - transformers |
| library_name: transformers |
| --- |
| <img src="Banner.png" width="1920"/> |
|
|
| # Rose-Medium |
|
|
| ## Almost 100M Parameters, 80B Tokens, and Still Not Quite Satisfied |
|
|
| At some point during the Rose project, I apparently looked at **Rose-Mini**, thought: |
|
|
| > "This is nice." |
|
|
| and then immediately made the completely reasonable decision to make it almost twice as large and feed it **80 billion tokens**. |
|
|
| This is **Rose-Medium**. |
|
|
| A roughly **97.82M parameter** model built with the Rose X1 architecture, trained on approximately **80.06B tokens**, with a native **2048-token context window**. |
|
|
| And before I start writing dramatic sentences about how incredible that sounds, let me be honest: |
|
|
| **I expected more.** |
|
|
| The model is good enough that I am happy it exists. |
| It is also stubborn enough that I already have a list of things I want to fix. |
| So this README is not a victory parade. |
| It is more like me standing beside the training run with my arms crossed, looking at the benchmark table and saying: |
|
|
| > "Okay. That's actually pretty good." |
|
|
| Then five seconds later: |
|
|
| > "But why is ARC-Easy doing that?" |
|
|
| --- |
|
|
| ## Architecture graph |
|
|
| <a href="https://hfviewer.com/GODELEV/Rose-Medium?utm_source=huggingface&utm_medium=embedded_model_card&utm_campaign=GODELEV_Rose-Medium_card" target="_blank" rel="noopener"> |
| <img |
| src="https://hfviewer.com/api/card.svg?source=GODELEV%2FRose-Medium&granularity=0" |
| alt="Architecture graph for GODELEV/Rose-Medium. Open in hfviewer" |
| width="100%" |
| /> |
| </a> |
|
|
| # What Rose-Medium Actually Is |
|
|
| Rose-Medium is the larger sibling of **Rose-Mini**. |
| The basic experiment was straightforward: keep the Rose X1 design and give it considerably more capacity and training. |
| The result is a model with: |
|
|
| - **97.82M parameters** |
| - **24 layers** |
| - **512 hidden dimensions** |
| - **8 attention heads** |
| - **2 KV heads** |
| - **2048-token native context** |
| - **32,768 vocabulary** |
| - **80.06B training tokens** |
|
|
| The full configuration is included in the repository for anyone who wants to inspect the machinery rather than trusting my suspiciously cheerful description of it. |
|
|
| --- |
|
|
| # The Short Version |
|
|
| If you don't want to read the entire model card, here is the situation: |
|
|
| **Rose-Medium is better than Rose-Mini.** |
|
|
| Good. |
|
|
| **Rose-Medium is dramatically better than I expected in absolutely every benchmark.** |
|
|
| No. |
|
|
| **Rose X1 appears to scale.** |
|
|
| Good. |
|
|
| **I think the model still has a bottleneck.** |
|
|
| Also good, because now I have something to investigate. |
|
|
| **Did I miss the BananaMind Bench 1100 club?** |
|
|
| Yes. |
|
|
| By **1.3 Elo**. |
|
|
| I would like to formally blame the benchmark for this. |
|
|
| The benchmark has declined to comment. |
|
|
| --- |
|
|
| # Results at a Glance |
|
|
| | Metric | Rose-Mini | Rose-Medium | |
| |---|---:|---:| |
| | **Parameters** | ~49.4M | **97.82M** | |
| | **Native Context** | 1024 | **2048** | |
| | **ArithMark 3** | 36.50% | **38.30%** | |
| | **BananaMind Base Bench 1.1** | 1037 Elo | **1098.7 Elo** | |
|
|
| The numbers are moving in the correct direction. |
|
|
| They just aren't moving in the direction of my ego fast enough. |
|
|
| --- |
|
|
| # Benchmark Results |
|
|
| Rose-Medium was evaluated in **0-shot** mode. |
|
|
| | Benchmark | Metric | Score | |
| |---|---|---:| |
| | **HellaSwag** | acc_norm | **35.2918%** | |
| | **PIQA** | acc_norm | **62.9489%** | |
| | **ARC-Easy** | acc_norm | **44.1919%** | |
| | **ARC-Challenge** | acc_norm | **26.1945%** | |
| | **WinoGrande** | acc | **49.8027%** | |
| | **BoolQ** | acc | **55.6575%** | |
| | **OpenBookQA** | acc_norm | **32.6000%** | |
| | **CommonsenseQA** | acc | **21.2121%** | |
| | **LAMBADA** | acc | **31.7097%** | |
| | **BLiMP** | acc | **79.2970%** | |
| | **MMLU** | acc | **23.9781%** | |
| | **WikiText-2** | word_perplexity | **27.6658** | |
| | **WikiText-2** | byte_perplexity | **1.8606** | |
| | **SciQ** | acc_norm | **67.1000%** | |
| | **COPA** | acc | **69.0000%** | |
| | **RACE** | acc | **29.0909%** | |
| | **SWAG** | acc_norm | **52.7942%** | |
| | **TruthfulQA MC2** | acc | **41.8316%** | |
| |
| All supplied evaluation runs completed successfully. |
| |
| --- |
| |
| # The Numbers I Actually Like |
| |
| There are some results here that I am genuinely happy about. |
| |
| **BLiMP — 79.30%** |
| |
| That's probably one of the nicest numbers in the table. |
| |
| **COPA — 69.00%** |
| |
| Very respectable. |
| |
| **SciQ — 67.10%** |
| |
| Also encouraging. |
| |
| **PIQA — 62.95%** |
| |
| Another result I am quite happy with. |
| |
| **LAMBADA — 31.71%** |
| |
| For a model this size, I will happily take that. |
| |
| And then there is **SWAG at 52.79%**. |
| |
| Nothing revolutionary, but definitely enough to make me nod at the screen like I knew what I was doing all along. |
| I absolutely did not. |
| |
| --- |
| |
| # The Numbers That Made Me Scratch My Head |
| |
| Now we arrive at my favorite part of machine learning: |
| **The part where the model does something and I have to explain why.** |
| Rose-Medium has almost **98M parameters** and has seen approximately **80B tokens**. |
| So naturally I expected some benchmarks to start behaving suspiciously well. |
| |
| Some did. |
| |
| Others apparently received the memo and decided to ignore it. |
| The most interesting example is ARC. |
| |
| --- |
| |
| # ARC: The Benchmark That Started Asking Questions |
| |
| Rose-Medium scores: |
| |
| - **ARC-Easy: 44.19%** |
| - **ARC-Challenge: 26.19%** |
| |
| The ARC-Challenge result is encouraging. |
| |
| The ARC-Easy result is where I start making increasingly serious eye contact with my configuration file. |
| |
| There is a fairly large gap between them. |
| |
| That makes me suspect that the model has learned useful patterns and knowledge, but struggles when the task requires reasoning or a more flexible internal representation. |
| |
| And I have a theory. |
| |
| A theory. |
| |
| Not a scientific law. |
| |
| Not a paper. |
| |
| Not divine revelation. |
| |
| A theory. |
| |
| ## I Think the Hidden Size May Be Too Small |
| |
| Rose-Medium uses a **512-dimensional hidden representation**. |
| |
| For a model with 24 layers, that may simply be too narrow. |
| |
| I may have spent a huge amount of compute making the model deeper while leaving the individual representations with relatively little room to express increasingly complicated information. |
| |
| Which would be a very me thing to do. |
| |
| Imagine building a tall building and then discovering halfway through that you made every room the size of a cupboard. |
| |
| That is currently my suspicion with Rose-Medium. |
| |
| The model may have more computation available, but not enough representational width to make full use of it. |
| |
| Again: |
| |
| **This is a hypothesis.** |
| |
| The benchmark results are evidence that make the hypothesis interesting, not proof that it is correct. |
| Rose Pro gets to help settle the argument. |
| |
| --- |
| |
| # ArithMark 3 |
| |
| Here is another useful comparison. |
| |
| **Rose-Mini:** 36.50% |
| **Rose-Medium:** 38.30% |
| |
| Improvement: |
| **+1.80 percentage points** |
| Is that a huge improvement? |
| No. |
| Am I happy it went up? |
| Yes. |
| |
| The result suggests that scaling Rose X1 gave Rose-Medium some additional arithmetic capability. |
| |
| But it also tells me that **more parameters alone are not producing the kind of jump I ultimately want**. |
| |
| So, once again, I have homework. |
| |
| Wonderful. |
| |
| --- |
| |
| # BananaMind Base Bench 1.1 |
| |
| Now we get to the number that personally offended me. |
| ### Rose-Mini |
| |
| **1037 Elo** |
| |
| ### Rose-Medium |
| |
| **1098.7 Elo** |
| |
| That is approximately: |
| |
| **+61.7 Elo** |
| |
| Which is genuinely a nice improvement. |
| |
| But there is one tiny problem. |
| |
| The target was **1100**. |
| |
| Rose-Medium scored: |
| |
| **1098.7** |
| |
| So I missed it by: |
| **1.3 Elo** |
| |
| One point. |
| |
| And a little bit. |
| |
| This is possibly the most irritating number in the entire project. |
| |
| I had almost entered the 1100 club. |
| |
| I could see the door. |
| |
| I was basically holding the membership form. |
| |
| Then the benchmark closed the door. |
| |
| Fine. |
| |
| **Rose Pro, you know what to do.** |
| |
| --- |
| |
| # Rose-Mini vs Rose-Medium |
| |
| > **<img src="Comparison.png" width="1600"/>** |
| |
| The important thing is that Rose-Medium does show measurable progress over Rose-Mini. |
| |
| It is larger. |
| |
| It has a longer native context. |
| |
| It performs better on **ArithMark 3**. |
| |
| It performs better on **BananaMind Base Bench 1.1**. |
| |
| But the improvements are not universally dramatic. |
| |
| And that is actually useful. |
| |
| It means the next step should not simply be: |
| |
| > "Make it bigger again." |
| |
| That would be the easiest possible conclusion. |
| |
| It is also probably the least interesting one. |
| |
| --- |
| |
| # Why I Am Not Completely Happy With It |
| |
| Because I think there is more in there. |
| |
| The model has: |
| |
| **~97.82M parameters** |
| |
| **~80.06B training tokens** |
| |
| **2048-token context** |
| |
| **24 layers** |
| |
| And yet some of the reasoning scores remain fairly modest. |
| |
| That makes me wonder whether the model's capacity is being allocated in the most useful way. |
| |
| The biggest suspect is still the **512 hidden dimension**. |
| |
| Maybe wider is better. |
| |
| Maybe deeper is better. |
| |
| Maybe the architecture needs another modification. |
| |
| Maybe I am completely wrong and the benchmark table is laughing at me. |
| |
| All are currently possible. |
| |
| That is why Rose-Medium is useful. |
| |
| It narrowed the question. |
| |
| --- |
| |
| # Training |
| |
| Rose-Medium was trained on approximately: |
| |
| **80.06 billion tokens** |
| |
| The recorded training metadata reports: |
| |
| | Property | Value | |
| |---|---:| |
| | **Parameters** | 97.82M | |
| | **Tokens Seen** | 80.06B | |
| | **Recorded Training Steps** | 38,196 | |
| | **Validation Loss** | 1.68248 | |
| | **Validation Perplexity** | 5.37888 | |
| | **Optimizer** | `muon_adamw` | |
| | **Context Length** | 2048 | |
|
|
| The validation numbers are encouraging, but, as usual, I refuse to let a nice perplexity number convince me that the model has achieved enlightenment. |
|
|
| Benchmarks are there for a reason. |
|
|
| --- |
|
|
| # Model Configuration |
|
|
| For people who actually want to look under the hood: |
|
|
| ```text |
| Architecture: Rose X1 |
| Parameters: 97.82M |
| Vocabulary: 32,768 |
| |
| Hidden Size: 512 |
| Intermediate Size: 1,728 |
| Layers: 24 |
| |
| Attention Heads: 8 |
| KV Heads: 2 |
| Head Dimension: 64 |
| |
| QK-Norm: Enabled |
| RoPE: Enabled |
| RoPE Theta: 100,000 |
| |
| Context Length: 2,048 |
| |
| Refresh Gate: Enabled |
| Inject Layers: 9, 18 |
| Kernel Size: 9 |
| |
| Tied Embeddings: Yes |
| Optimizer: Muon AdamW |
| ``` |
|
|
| That's enough configuration for this README. |
| If you want every individual tensor and configuration field, the repository is a much better place to look than watching me copy-paste a configuration file into a novel. |
|
|
| --- |
|
|
| # What Rose-Medium Taught Me |
|
|
| The main lesson is not: |
|
|
| **"More parameters = better model."** |
|
|
| That would be too convenient. |
|
|
| The more interesting lesson is: |
|
|
| **Scaling exposes bottlenecks.** |
|
|
| Rose-Mini showed that Rose X1 could work. |
|
|
| Rose-Medium showed that making it larger improves some capabilities. |
|
|
| It also showed that simply increasing scale does not magically solve harder reasoning. |
|
|
| That is actually a much more useful result. |
|
|
| Now I have a better idea of what I want to investigate next. |
|
|
| And yes, one of those things is **width**. |
|
|
| --- |
|
|
| # Rose Pro |
|
|
| ## Coming Soon |
|
|
| The next major Rose model will be: |
|
|
| # Rose Pro |
|
|
| The goal is to push the architecture further and, hopefully, address some of the limitations exposed by Rose-Medium. |
| O |
| ne of the things I particularly want to investigate is whether increasing representational width can unlock more of the capability that I believe is currently being constrained. |
|
|
| And there is one tiny unfinished piece of business: |
|
|
| **BananaMind 1100 Elo.** |
|
|
| Rose-Medium reached **1098.7**. |
|
|
| Rose Pro has been given a very specific assignment. |
|
|
| --- |
|
|
| # Final Verdict |
|
|
| So, how do I feel about Rose-Medium? |
|
|
| Honestly? |
|
|
| **Pretty happy.** |
|
|
| But not satisfied. |
|
|
| The model works. |
|
|
| The scaling worked. |
|
|
| The benchmark profile has several results I genuinely like. |
|
|
| Rose-Medium improved over Rose-Mini. |
|
|
| And the architecture has given me enough evidence to keep pushing it. |
|
|
| But I also think the model should be capable of more. |
|
|
| So no, I am not going to call Rose-Medium a breakthrough. |
|
|
| **A good experiment that left me with better questions.** |
|
|
| --- |
|
|
| # The Rose Family |
|
|
| **Rose-Mini** |
| The model that started the Rose X1 experiment. |
|
|
| **Rose-Medium** |
| The model that proved the architecture could scale, while also politely pointing out that I probably need to think harder about representation width. |
|
|
| **Rose Pro** |
|
|
| The next attempt to make me regret writing this README if it performs worse. |
| That's the scientific method, right? |
|
|
| --- |
|
|
| # Closing Note |
|
|
| Rose-Medium is not the end of Rose X1. |
|
|
| If anything, it made the next step more obvious. |
|
|
| I wanted to know whether scaling the architecture would help. |
|
|
| It did. |
|
|
| Then I wanted the model to perform even better. |
|
|
| It didn't quite cooperate. |
|
|
| So now I have another problem. |
|
|
| Which is fine. |
|
|
| Rose-Medium did its job. |
|
|
| It improved. |
|
|
| It taught me something. |
|
|
| And now I have to build Rose Pro. |
|
|
| **See you there.** |
|
|
