| --- |
| language: en |
| tags: |
| - causal-lm |
| - gqa |
| - qk-norm |
| - rope |
| - swiglu |
| - refresh-gate |
| - rose-x1 |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| --- |
| |
| <img src="Banner.png" width="1300"/> |
|
|
| # Rose-Mini |
|
|
| ## The First Bloom |
|
|
| Every model teaches you something. |
| Some teach optimization. |
| Some teach patience. |
| Some teach you that a single misplaced line of code can quietly destroy months of work. |
|
|
| My previous projects taught me all three. |
|
|
| **Archaea** taught me how to train language models from scratch. |
|
|
| The **Ant** series taught me that lowering loss does not necessarily mean a model has learned anything useful. |
|
|
| **Exp-1** humbled me by exposing a catastrophic implementation bug hidden inside my Transformer blocks. |
|
|
| **Archaea-74M-V1.1** reminded me that refining a good model is sometimes more valuable than building a new one. |
|
|
| Rose-Mini begins where those projects ended. |
| Not with another Transformer. |
| Not with another checkpoint. |
| But with a question. |
|
|
| > **What if I stopped modifying existing architectures and started building one of my own?** |
|
|
| That question eventually became **Rose X1**. |
| And Rose-Mini is the first public model built upon it. |
|
|
| --- |
|
|
| # A Different Kind of Project |
|
|
| Unlike my previous releases, Rose-Mini is not simply another pretrained language model. |
| It represents my first successful custom architecture. |
| For a long time I had wanted to move beyond reproducing existing Transformer implementations. |
| Training models is fun. |
| Understanding them is even more fun. |
| Building one yourself... |
| That is where the real adventure begins. |
| Rose X1 became my playground for exploring architectural ideas while still remaining practical enough to train on consumer hardware. |
| Fortunately... |
| The architecture worked. |
|
|
| --- |
|
|
| # Acknowledgements |
|
|
| Before talking about Rose X1, I want to thank **Datdanboi25**. |
| The **TX4** architecture was one of the first compact language model designs that genuinely made me stop scrolling and start reading code. |
| Rather than simply scaling parameters, TX4 explored interesting architectural ideas for making smaller language models more capable. |
| Rose X1 is **inspired by** many of those ideas while exploring its own design decisions and implementation. |
| It is **not** a reproduction of TX4. |
| It is my own architectural exploration built upon lessons learned from previous projects and inspired by the excellent work shared by the community. |
| As an independent developer, openly published research and implementations are invaluable learning resources. |
|
|
| So... |
|
|
| Thank you, **Datdanboi25**. |
| Without builders willing to share ideas, projects like Rose probably would never have existed. |
|
|
| --- |
|
|
| # Meet Rose X1 |
|
|
| Rose X1 is a lightweight decoder-only architecture designed around one simple philosophy: |
|
|
| > **Extract more capability from fewer parameters.** |
|
|
| Instead of chasing larger parameter counts, Rose focuses on architectural efficiency and better utilization of compute. |
| The architecture incorporates several modern design ideas including: |
|
|
| * Grouped Query Attention (GQA) |
| * RMSNorm |
| * QK Normalization |
| * Refresh Gates |
| * Efficient decoder-only design |
| * BF16 training |
|
|
| The goal was never to reinvent the Transformer. |
| The goal was to explore it. |
|
|
| ## Architecture graph |
|
|
| <a href="https://hfviewer.com/GODELEV/Rose-Mini?utm_source=huggingface&utm_medium=embedded_model_card&utm_campaign=GODELEV_Rose-Mini_card" target="_blank" rel="noopener"> |
| <img |
| src="https://hfviewer.com/api/card.svg?source=GODELEV%2FRose-Mini&granularity=0" |
| alt="Architecture graph for GODELEV/Rose-Mini. Open in hfviewer" |
| width="100%" |
| /> |
| </a> |
|
|
| --- |
|
|
| # The Four Hour Battle |
|
|
| Designing an architecture turned out to be easier than convincing Hugging Face that it actually existed. |
|
|
| Training was straightforward. |
|
|
| Writing the custom Hugging Face implementation... |
|
|
| That was another story. |
|
|
| Since Rose X1 is a custom architecture, I had to implement everything required for Hugging Face compatibility. |
|
|
| Configuration classes. |
|
|
| Model classes. |
|
|
| Registration. |
|
|
| Auto mapping. |
|
|
| Generation support. |
|
|
| Loading. |
|
|
| Saving. |
|
|
| Everything. |
|
|
| I spent nearly **four hours** fighting the modeling file. |
|
|
| Every time I fixed one error... |
|
|
| Python politely introduced two more. |
|
|
| Even AI models tried to help. |
|
|
| They certainly made excellent suggestions. |
|
|
| Unfortunately, they underestimated one critical variable. |
|
|
| **Me.** |
|
|
| I have a remarkable talent for introducing new bugs while fixing old ones. |
|
|
| Some people have green fingers. |
|
|
| Some have sticky fingers. |
|
|
| I have **buggy fingers**. |
|
|
| Eventually... |
|
|
| The architecture loaded successfully. |
|
|
| The checkpoints loaded. |
|
|
| Generation worked. |
|
|
| `AutoModelForCausalLM.from_pretrained()` finally stopped complaining. |
|
|
| That tiny success was honestly one of the most satisfying moments of the entire project. |
|
|
| --- |
|
|
| # Model Overview |
|
|
| | Property | Value | |
| | ---------------- | -------------- | |
| | Model | Rose-Mini | |
| | Architecture | Rose X1 | |
| | Parameters | 49.443 Million | |
| | Layers | 14 | |
| | Hidden Size | 512 | |
| | Attention Heads | 8 | |
| | KV Heads | 2 | |
| | QK Normalization | Enabled | |
| | Refresh Gates | Layers 4 & 9 | |
| | Training Tokens | 12,206,861,568 | |
| | Optimizer | Muon AdamW | |
|
|
| --- |
|
|
| # Training |
|
|
| Rose-Mini was trained on over **12.2 billion tokens**, representing my largest pretraining run so far. |
| Rather than increasing parameter count, I chose to expose a compact model to significantly more data. |
| Training small language models is an interesting balancing act. |
| Too few tokens... |
| The model memorizes very little. |
| Too many... |
| You begin wondering whether your SSD has developed feelings. #wirtten by Chat Gpt |
| Fortunately, both survived. |
|
|
| --- |
|
|
| # Comparing My Models |
|
|
| One thing I enjoy about independent research is watching projects evolve over time. |
| Each model solved a different problem. |
|
|
| | Model | Parameters | Tokens | Architecture | Purpose | |
| | ---------------- | ---------: | -----------------: | ----------------------- | ------------------------------------- | |
| | Archaea-74M | 74M | 1.23B | LLaMA-style Transformer | First successful pretrained model | |
| | Archaea-74M-V1.1 | 74M | Continued Training | LLaMA-style Transformer | Refinement and benchmark improvements | |
| | **Rose-Mini** | **49.4M** | **12.2B** | **Rose X1** | First custom architecture | |
|
|
| Rose-Mini contains roughly **33% fewer parameters** than Archaea-74M. |
| Despite that, it demonstrates that architectural experimentation can remain competitive while dramatically reducing parameter count. |
| For me, Rose-Mini is less about replacing Archaea. |
| It is about exploring a different path. |
|
|
| --- |
|
|
| # Benchmark Highlights |
|
|
| Rose-Mini demonstrates encouraging performance for a model of its size. |
|
|
| | Benchmark | Score | |
| | ----------------------------- | --------------: | |
| | **BananaMind Base Bench 1.1** | **1037.37 Elo** | |
| | **ArithMark-2.0** | **29.52%** | |
| | **HellaSwag** | **28.95%** | |
| | **PIQA** | **58.54%** | |
| | **ARC-Easy** | **36.62%** | |
| | **ARC-Challenge** | **24.32%** | |
|
|
| The BananaMind Base Bench result crossing **1000 Elo** was particularly satisfying. |
| Watching a custom architecture compete after weeks of implementation work is an incredibly rewarding experience. |
|
|
| --- |
|
|
| # Performance Compared to Archaea |
|
|
| Rose-Mini was never intended to be a direct replacement for Archaea. |
| Both models were built with different objectives. |
|
|
| | Category | Archaea-74M | Archaea-74M-V1.1 | Rose-Mini | |
| | ------------------- | ----------: | -----------------: | --------------: | |
| | Parameters | 74M | 74M | **49.4M** | |
| | Training Tokens | 1.23B | Continued Training | **12.2B** | |
| | Architecture | LLaMA-style | LLaMA-style | Rose X1 | |
| | Custom Architecture | No | No | **Yes** | |
| | BananaMind | — | 936 | **1037.37 Elo** | |
| | ArithMark-2.0 | 29.30% | **37.37%** | **29.52%** | |
|
|
| --- |
|
|
| # Why Rose? |
|
|
| The answer is simple. |
|
|
| Curiosity. |
|
|
| Training another standard Transformer would certainly have been easier. |
|
|
| It also would have been considerably less interesting. |
|
|
| Rose exists because experimentation is how progress happens. |
|
|
| Sometimes the experiment works. |
|
|
| Sometimes it becomes Exp-1. |
|
|
| Fortunately... |
|
|
| This one worked. |
|
|
| --- |
|
|
| # Current Limitations |
|
|
| Rose-Mini is still a relatively small language model. |
|
|
| Like every compact model, it has limitations. |
|
|
| * Complex reasoning remains challenging. |
| * Long-context understanding can be improved. |
| * Factual consistency still has room for growth. |
| * Instruction following could be further refined. |
|
|
| None of these are surprising. |
|
|
| They're simply opportunities for the next version. |
|
|
| --- |
|
|
| # Future Work |
|
|
| Rose-Mini is only the beginning of the Rose family. |
| Development is already underway for: |
|
|
| ## Rose-Medium |
|
|
| The goal of Rose-Medium is not simply to increase parameter count. |
|
|
| It is to continue exploring the Rose X1 architecture at a larger scale while preserving the design philosophy that made Rose-Mini successful. |
|
|
| Better architecture. |
|
|
| Better training. |
|
|
| Better models. |
|
|
| Hopefully fewer bugs. |
|
|
| Although I refuse to make any promises regarding that last point. |
|
|
| --- |
|
|
| # Final Thoughts |
|
|
| Thank you for checking out Rose-Mini. |
|
|
| I hope you enjoy experimenting with it as much as I enjoyed building it. |
|
|
