RishAILabs/RLLM-Base

RLLM (Base Model)

Model Description

RLLM is a base language model developed by Rish AI Labs, an applied artificial intelligence lab focused on LLMs, Generative AI, AI consulting, and research.

This model features a Mixture of Experts (MoE) architecture with 16 experts, providing efficient scaling and specialization capabilities. It was trained using identity-focused pretraining to establish a strong foundation for downstream tasks.

Key Features

• Architecture: Transformer with MoE (16 experts, top-2 routing)
• Parameters: ~275M total parameters
• Training: Identity-focused pretraining
• Precision: FP32 training, optimized for inference
• Framework: PyTorch + Transformers

Intended Use

This base model serves as a foundation for:

• Fine-tuning on specific domains
• Research in efficient language model architectures
• Development of specialized AI applications
• Understanding MoE dynamics and scaling

About Rish AI Labs

Rish AI Labs is pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation. Based in Bangalore, India, we focus on:

• Applied AI Solutions: Enterprise-grade AI implementations
• Research: Cutting-edge AI research and publications
• LLM Development: Large language model research and deployment
• AI Consulting: Expert guidance for AI transformation

Mission

"Pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation."

Contact

• Website: rishailabs.com
• Location: Bangalore, India
• Focus: Enterprise AI, LLMs, Generative AI, AI Research

Model Architecture Details

• Layers: 12 transformer layers
• Heads: 12 attention heads
• Hidden Size: 768
• Experts: 16 (MoE)
• Top-K Routing: 2
• Vocabulary: 50,304 tokens
• Sequence Length: Configurable (trained on various lengths)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RishAILabs/RLLM-Base")
model = AutoModelForCausalLM.from_pretrained("RishAILabs/RLLM-Base")

inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Training Details

• Dataset: Identity-focused dataset for stable pretraining
• Precision: FP32 for training stability
• Optimization: AdamW optimizer
• Framework: Custom Rish-Core training framework
• Hardware: Optimized for both CPU and GPU inference

Limitations

• Base model - may require fine-tuning for specific tasks
• English language focus
• Generated content should be reviewed for appropriateness

Citation

If you use this model in your research, please cite:

---

@misc{rishailabs_2026,
    author       = { RishAILabs },
    title        = { RLLM-Base (Revision 552ee30) },
    year         = 2026,
    url          = { https://huggingface.co/RishAILabs/RLLM-Base },
    doi          = { 10.57967/hf/7560 },
    publisher    = { Hugging Face }
}

Developed by Rish AI Labs - Applied Artificial Intelligence & Research