license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- casual-lm
- gqa
- custom
- refresh_gate
- swiglu
- qk-norm
- rope
datasets:
- AxiomicLabs/Arithmark-3.0
- AxiomicLabs/NPset-2-Python-Edu
- HuggingFaceTB/cosmopedia
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- BananaMind/BananaMind-Base-Bench-1.1
Rose-1.5-Pro
Rose-1.5-Pro is a compact language model developed by GODELEV, built on the Rose X1.5 architecture. Trained on approximately 70 billion tokens, it explores the scaling of the Rose X1.5 design with a wider hidden representation and a configuration intended to balance model capability with training efficiency.
The model achieves an Intelligence Index of 28.27, with strong results across commonsense reasoning, science-question benchmarks, and general language understanding tasks. Its evaluation results provide another step in exploring how architectural choices and training configurations influence performance in small language models.
Model Overview
| Property | Value |
|---|---|
| Model | Rose-1.5-Pro |
| Developer | Mr Munk |
| Architecture | Rose X1.5 |
| Layers | 30 |
| Hidden Size | 640 |
| Attention Heads | 10 |
| Key-Value Heads | 5 |
| Head Dimension | 64 |
| Intermediate Size | 1536 |
| Vocabulary Size | 32,770 |
| Context Length | 2,048 tokens |
| Training Tokens | ~70B |
| Activation | SiLU |
| Normalization | RMSNorm |
| QK Normalization | Enabled |
| XSA | Enabled |
| Refresh Gates | Layers 10 and 20 |
Evaluation Results
Rose-1.5-Pro was evaluated across a set of benchmarks covering commonsense reasoning, scientific knowledge, and arithmetic-oriented tasks.
| Benchmark | Score |
|---|---|
| HellaSwag | 43.92% |
| ARC-Easy | 58.29% |
| ARC-Challenge | 30.72% |
| Combined ARC | 44.50% |
| PIQA | 69.48% |
| ArithMark-3 | 40.00% |
| ArithMark-2 | 34.52% |
| Intelligence Index | 28.27 |
The Intelligence Index aggregates selected benchmark results into a single normalized score. It is useful for comparing models evaluated under the same methodology, although it cannot represent every aspect of language-model capability.
Rose-1.5-Pro's results show its performance across several evaluation categories, while the arithmetic benchmarks offer a separate view of its performance on structured numerical tasks.
Architecture and Training
Rose-1.5-Pro continues the development of the Rose X1.5 architecture, using 30 layers and a hidden size of 640. It incorporates QK normalization, XSA, and refresh gates as part of its architectural configuration.
The model was trained on approximately 70B tokens.
Context Length
The model uses a 2,048-token context window by design. This decision prioritizes faster training and a more efficient training process rather than maximizing context length. Context length is an engineering trade-off, and a larger window is not automatically necessary for every training objective.
Position Within the Rose Series
Rose-1.5-Pro builds on the experiments conducted with earlier Rose models. The series explores how changes in model scale, architecture, and training configuration affect benchmark performance.
Rather than assuming that additional parameters or training tokens guarantee improvements on every task, the evaluation results provide a practical basis for identifying strengths and trade-offs across benchmarks.
Limitations
- Benchmark scores measure performance on specific evaluation tasks and should not be treated as a complete measure of general intelligence.
- Results depend on the evaluation datasets, prompts, scoring procedures, and implementation.
- Performance may vary across tasks and domains not represented in the reported benchmarks.
- The 2,048-token context window is a deliberate training-efficiency choice and may not suit tasks requiring longer input sequences.