---
language: en
license: apache-2.0
tags:
- transformer
- pytorch
- causal-lm
- moe
- mixture-of-experts
- rish-ai-labs
---

# RLLM (Base Model)

## Model Description

RLLM is a base language model developed by **Rish AI Labs**, an applied artificial intelligence lab focused on LLMs, Generative AI, AI consulting, and research.

This model features a **Mixture of Experts (MoE)** architecture with 16 experts, providing efficient scaling and specialization capabilities. It was trained using identity-focused pretraining to establish a strong foundation for downstream tasks.

## Key Features

- **Architecture**: Transformer with MoE (16 experts, top-2 routing)
- **Parameters**: ~275M total parameters
- **Training**: Identity-focused pretraining
- **Precision**: FP32 training, optimized for inference
- **Framework**: PyTorch + Transformers

## Intended Use

This base model serves as a foundation for:
- Fine-tuning on specific domains
- Research in efficient language model architectures
- Development of specialized AI applications
- Understanding MoE dynamics and scaling

## About Rish AI Labs

**Rish AI Labs** is pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation. Based in Bangalore, India, we focus on:

- **Applied AI Solutions**: Enterprise-grade AI implementations
- **Research**: Cutting-edge AI research and publications
- **LLM Development**: Large language model research and deployment
- **AI Consulting**: Expert guidance for AI transformation

### Mission
"Pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation."

### Contact
- Website: [rishailabs.com](https://rishailabs.com)
- Location: Bangalore, India
- Focus: Enterprise AI, LLMs, Generative AI, AI Research

## Model Architecture Details

- **Layers**: 12 transformer layers
- **Heads**: 12 attention heads
- **Hidden Size**: 768
- **Experts**: 16 (MoE)
- **Top-K Routing**: 2
- **Vocabulary**: 50,304 tokens
- **Sequence Length**: Configurable (trained on various lengths)

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RishAILabs/RLLM-Base")
model = AutoModelForCausalLM.from_pretrained("RishAILabs/RLLM-Base")

inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
```

## Training Details

- **Dataset**: Identity-focused dataset for stable pretraining
- **Precision**: FP32 for training stability
- **Optimization**: AdamW optimizer
- **Framework**: Custom Rish-Core training framework
- **Hardware**: Optimized for both CPU and GPU inference

## Limitations

- Base model - may require fine-tuning for specific tasks
- English language focus
- Generated content should be reviewed for appropriateness

## Citation

If you use this model in your research, please cite:


---


```bibtex
@misc{rishailabs_2026,
	author       = { RishAILabs },
	title        = { RLLM-Base (Revision 552ee30) },
	year         = 2026,
	url          = { https://huggingface.co/RishAILabs/RLLM-Base },
	doi          = { 10.57967/hf/7560 },
	publisher    = { Hugging Face }
}
```


*Developed by Rish AI Labs - Applied Artificial Intelligence & Research*