File size: 3,335 Bytes
550783d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
language: [en, hi]
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags: [education, multilingual, hindi, hinglish, awadhi, from-scratch, small-language-model]
---
# MONA: a multilingual small language model you can open up
MONA is a **110M-parameter Llama-style language model trained from scratch on one laptop GPU** for
**English, Hindi, Hinglish and Awadhi** (Tulsidas). It was built to **teach how LLMs work**: the
companion terminal app lets anyone look inside it, stage by stage, in English, हिंदी or Hinglish.
- Code, app and full build story: https://github.com/JayD108/Mona
- `pip install` the repo, then run `mona`: the app downloads these files on first run.
## Model
| | |
|---|---|
| Architecture | decoder-only transformer: 12 layers, width 768, 12 heads, SwiGLU 2048, RMSNorm, RoPE, tied embeddings |
| Parameters | 110.1M (85.0M non-embedding) |
| Tokenizer | byte-level BPE, 32,768 tokens, trained on a balanced EN / HI / Hinglish / Awadhi sample (Hindi 1.33 tokens/word) |
| Context | 1,024 tokens |
| Pretraining | 2.5B tokens, AdamW, bf16, WSD learning-rate schedule (peak 6e-4), ~19 h on an RTX 5080 Laptop GPU |
| Chat tuning | supervised fine-tuning with loss masking on ~100K conversations |
## Evaluation (held-out text, bits per byte, lower is better)
| English | Hindi | Hinglish |
|---|---|---|
| 1.027 | 0.455 | 1.058 |
Final held-out loss per language (nats/token): english 3.146, hindi 3.063, hinglish 2.899, awadhi 5.445
Sample (MONA-Chat, "Who are you?"): *I'm Emily Wilson, the executive director of the National Geographic Society. I'm the lead author of the new report, "The Greatest Planet: The Greatest Earthquake in History." I'm thrilled to share my *
## Files
- `assets/models/mona/mona.pt`: base model (bf16 state dict) · `assets/runs/mona_m_chat/model.pt`: MONA-Chat
- `assets/models/mona/tokenizer.json`: tokenizer (Hugging Face `tokenizers` format)
- `assets/…`: a small, licence-clean demo bundle used by the app (see NOTICE.md)
```python
from mona.engine.glassbox import GlassBox # from the GitHub repo
gb = GlassBox() # loads the newest weights on CPU
print(gb.next_word("The capital of India is", top=5))
```
## Intended use and limits
For **education and experimentation**. MONA is small: it writes fluent text but **often states things
that are wrong**, cannot do arithmetic, and must not be used for factual, medical, legal or other
consequential decisions.
## Training data and licences
English: FineWeb-Edu (ODC-By 1.0) · Hindi and romanized Hindi: AI4Bharat Sangraha (CC-BY-4.0; its
Wikipedia-derived parts are CC BY-SA) · chat: smol-smoltalk (Apache-2.0), Aya dataset (Apache-2.0) ·
Awadhi: Tulsidas's verses (public domain). See NOTICE.md for full attribution.
**Copyright note:** an earlier Tulsidas corpus also contained a modern Hindi commentary (Gita Press,
likely still under copyright). When this was noticed, at 0.8B of 2.5B training tokens (~0.1% of
batches up to then), it was removed: the rest of pretraining, the chat tuning and everything in this
repository use only public-domain verse. No commentary text is distributed here.
## Credits
Built by Mrittunjay Dubey. Coding assisted by Claude (Anthropic).
|