OroLLM Base Model v1.0 (Pre-training Checkpoint)
Disclaimer
This is an early pre-training checkpoint, NOT a production-ready model.
This model is a RoBERTa base trained on a relatively small dataset (9.2M tokens). It has NOT been evaluated on any downstream NLP benchmarks (e.g., classification, translation, or question answering). Without fine-tuning and proper evaluation, its practical performance on real-world tasks is currently unknown.
This checkpoint is shared publicly to support further research and community collaboration for Afaan Oromoo NLP.
Model Details
- Architecture: RoBERTa (RobertaForMaskedLM)
- Parameters: 35.9 Million
- Hidden Layers: 6
- Hidden Size: 512
- Attention Heads: 8
- Vocabulary Size: 32,000
- License: CC BY-SA 4.0
Training Log (Raw Numbers)
The model was trained on 9.2 million Afaan Oromoo tokens. Training stopped early at 62,000 steps (75% of planned steps) after the loss began to plateau.
| Metric | Value |
|---|---|
| Training Steps | 62,000 / 82,350 (75%) |
| Epochs | 7.53 / 10 |
| Batch Size | 8 |
| Final Training Loss | 13.68 |
| Final Evaluation Loss | 6.77 |
Note: Evaluation loss here is just a held-out validation split during pre-training, not a downstream task benchmark.
Limitations
- Small Dataset: 9.2M tokens is relatively small for a 36M parameter model.
- Early Stop: Training was stopped at 75% of planned steps.
- No Fine-Tuning: This is a base model only; it has not been adapted for chat, translation, or summarization.
- No Benchmarks: Zero downstream task evaluations have been performed.
How to Load This Checkpoint
from transformers import RobertaForMaskedLM, RobertaTokenizer
model = RobertaForMaskedLM.from_pretrained("astu-coe-ece-0045-2025/OroLLM-base-v1")
tokenizer = RobertaTokenizer.from_pretrained("astu-coe-ece-0045-2025/OroLLM-base-v1")
- Downloads last month
- -