| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - casual-lm |
| - gqa |
| - custom |
| - refresh_gate |
| - swiglu |
| - qk-norm |
| - rope |
| datasets: |
| - AxiomicLabs/Arithmark-3.0 |
| - AxiomicLabs/NPset-2-Python-Edu |
| - HuggingFaceTB/cosmopedia |
| - HuggingFaceFW/fineweb-edu |
| - mlfoundations/dclm-baseline-1.0 |
| - BananaMind/BananaMind-Base-Bench-1.1 |
| --- |
| <img src="Banner.png" width="1920"/> |
|
|
| # Rose-1.5-Pro |
|
|
| Rose-1.5-Pro is a compact language model developed by **GODELEV**, built on the **Rose X1.5 architecture**. Trained on approximately **70 billion tokens**, it explores the scaling of the Rose X1.5 design with a wider hidden representation and a configuration intended to balance model capability with training efficiency. |
|
|
| The model achieves an **Intelligence Index of 28.27**, with strong results across commonsense reasoning, science-question benchmarks, and general language understanding tasks. Its evaluation results provide another step in exploring how architectural choices and training configurations influence performance in small language models. |
|
|
| ## Model Overview |
|
|
| | Property | Value | |
| |---|---| |
| | Model | Rose-1.5-Pro | |
| | Developer | Mr Munk | |
| | Architecture | Rose X1.5 | |
| | Layers | 30 | |
| | Hidden Size | 640 | |
| | Attention Heads | 10 | |
| | Key-Value Heads | 5 | |
| | Head Dimension | 64 | |
| | Intermediate Size | 1536 | |
| | Vocabulary Size | 32,770 | |
| | Context Length | 2,048 tokens | |
| | Training Tokens | ~70B | |
| | Activation | SiLU | |
| | Normalization | RMSNorm | |
| | QK Normalization | Enabled | |
| | XSA | Enabled | |
| | Refresh Gates | Layers 10 and 20 | |
|
|
| ## Evaluation Results |
|
|
| Rose-1.5-Pro was evaluated across a set of benchmarks covering commonsense reasoning, scientific knowledge, and arithmetic-oriented tasks. |
|
|
| | Benchmark | Score | |
| |---|---:| |
| | HellaSwag | 43.92% | |
| | ARC-Easy | 58.29% | |
| | ARC-Challenge | 30.72% | |
| | Combined ARC | 44.50% | |
| | PIQA | 69.48% | |
| | ArithMark-3 | 40.00% | |
| | ArithMark-2 | 34.52% | |
| | **Intelligence Index** | **28.27** | |
|
|
| The Intelligence Index aggregates selected benchmark results into a single normalized score. It is useful for comparing models evaluated under the same methodology, although it cannot represent every aspect of language-model capability. |
|
|
| Rose-1.5-Pro's results show its performance across several evaluation categories, while the arithmetic benchmarks offer a separate view of its performance on structured numerical tasks. |
|
|
| ## Architecture and Training |
|
|
| Rose-1.5-Pro continues the development of the Rose X1.5 architecture, using 30 layers and a hidden size of 640. It incorporates QK normalization, XSA, and refresh gates as part of its architectural configuration. |
|
|
| The model was trained on approximately **70B tokens**. |
|
|
| ### Context Length |
|
|
| The model uses a **2,048-token context window by design**. This decision prioritizes faster training and a more efficient training process rather than maximizing context length. Context length is an engineering trade-off, and a larger window is not automatically necessary for every training objective. |
|
|
| ## Position Within the Rose Series |
|
|
| Rose-1.5-Pro builds on the experiments conducted with earlier Rose models. The series explores how changes in model scale, architecture, and training configuration affect benchmark performance. |
|
|
| Rather than assuming that additional parameters or training tokens guarantee improvements on every task, the evaluation results provide a practical basis for identifying strengths and trade-offs across benchmarks. |
|
|
| ## Limitations |
|
|
| - Benchmark scores measure performance on specific evaluation tasks and should not be treated as a complete measure of general intelligence. |
| - Results depend on the evaluation datasets, prompts, scoring procedures, and implementation. |
| - Performance may vary across tasks and domains not represented in the reported benchmarks. |
| - The 2,048-token context window is a deliberate training-efficiency choice and may not suit tasks requiring longer input sequences. |
