| --- |
| license: mit |
| language: |
| - en |
| - ja |
| --- |
| # Model Card |
| ## Overview |
| Rize is a causal language model for pretraining research and general text generation. |
| It uses a Transformer decoder architecture with Mixture-of-Experts (MoE) layers. |
| The model is designed for research and experimental development. |
|
|
| ## Model Size and Architecture |
| This tiny model has about **4 billion total parameters** and about **1 billion active parameters per token**. |
|
|
| Main architecture points: |
| - decoder-only Transformer |
| - 19 hidden layers |
| - hidden size of 1536 |
| - 12 attention heads |
| - 64 routed experts |
| - top-4 expert routing per token |
| - 1 shared expert |
| - vocabulary size of 163,840 |
| - maximum context length of 8,192 tokens |
|
|
| ## Intended Use |
| This model is intended for: |
| - language modeling research |
| - evaluation of training settings and architectures |
| - general text generation benchmarks |
|
|
| This model is not intended to be used as a source of factual truth or professional advice. |
|
|
| ## Training |
| The model is trained with autoregressive next-token prediction on text data. |
| It is developed as a research model and may change across checkpoints, runs, and configurations. |
|
|
| ## Capabilities |
| - text continuation |
| - general question answering |
| - instruction-style response generation |
| - multilingual text handling, depending on training data |
|
|
| ## Limitations |
| - may generate incorrect or misleading information |
| - may reflect biases in training data |
| - may produce unsafe, harmful, or inappropriate text |
| - performance may vary across languages and domains |
| - not optimized for high-stakes decisions |
|
|
| ## Safety and Responsible Use |
| Users should review outputs before any real-world use. |
| The model should not be used on its own for: |
| - medical advice |
| - legal advice |
| - financial advice |
| - safety-critical decisions |
| - sensitive personal decisions |
|
|
| Human oversight is required. |
|
|
| ## Disclaimer |
| This model is provided for research and experimental purposes only. |
| The FA Research Team makes no guarantees regarding accuracy, completeness, reliability, safety, or fitness for a particular purpose. |
| Use of this model and its outputs is at the user’s own risk. |
|
|
| ## Contact |
| FA Research Team |
