| ---
|
| language: [en, hi]
|
| license: mit
|
| library_name: pytorch
|
| pipeline_tag: text-generation
|
| tags: [education, multilingual, hindi, hinglish, awadhi, from-scratch, small-language-model]
|
| ---
|
|
|
| # MONA: a multilingual small language model you can open up
|
|
|
| MONA is a **110M-parameter Llama-style language model trained from scratch on one laptop GPU** for
|
| **English, Hindi, Hinglish and Awadhi** (Tulsidas). It was built to **teach how LLMs work**: the
|
| companion terminal app lets anyone look inside it, stage by stage, in English, हिंदी or Hinglish.
|
|
|
| - Code, app and full build story: https://github.com/JayD108/Mona
|
| - `pip install` the repo, then run `mona`: the app downloads these files on first run.
|
|
|
| ## Model
|
| | | |
|
| |---|---|
|
| | Architecture | decoder-only transformer: 12 layers, width 768, 12 heads, SwiGLU 2048, RMSNorm, RoPE, tied embeddings |
|
| | Parameters | 110.1M (85.0M non-embedding) |
|
| | Tokenizer | byte-level BPE, 32,768 tokens, trained on a balanced EN / HI / Hinglish / Awadhi sample (Hindi 1.33 tokens/word) |
|
| | Context | 1,024 tokens |
|
| | Pretraining | 2.5B tokens, AdamW, bf16, WSD learning-rate schedule (peak 6e-4), ~19 h on an RTX 5080 Laptop GPU |
|
| | Chat tuning | supervised fine-tuning with loss masking on ~100K conversations |
|
|
|
| ## Evaluation (held-out text, bits per byte, lower is better)
|
| | English | Hindi | Hinglish |
|
| |---|---|---|
|
| | 1.027 | 0.455 | 1.058 |
|
|
|
| Final held-out loss per language (nats/token): english 3.146, hindi 3.063, hinglish 2.899, awadhi 5.445
|
|
|
| Sample (MONA-Chat, "Who are you?"): *I'm Emily Wilson, the executive director of the National Geographic Society. I'm the lead author of the new report, "The Greatest Planet: The Greatest Earthquake in History." I'm thrilled to share my *
|
|
|
| ## Files
|
| - `assets/models/mona/mona.pt`: base model (bf16 state dict) · `assets/runs/mona_m_chat/model.pt`: MONA-Chat
|
| - `assets/models/mona/tokenizer.json`: tokenizer (Hugging Face `tokenizers` format)
|
| - `assets/…`: a small, licence-clean demo bundle used by the app (see NOTICE.md)
|
|
|
| ```python
|
| from mona.engine.glassbox import GlassBox # from the GitHub repo
|
| gb = GlassBox() # loads the newest weights on CPU
|
| print(gb.next_word("The capital of India is", top=5))
|
| ```
|
|
|
| ## Intended use and limits
|
| For **education and experimentation**. MONA is small: it writes fluent text but **often states things
|
| that are wrong**, cannot do arithmetic, and must not be used for factual, medical, legal or other
|
| consequential decisions.
|
|
|
| ## Training data and licences
|
| English: FineWeb-Edu (ODC-By 1.0) · Hindi and romanized Hindi: AI4Bharat Sangraha (CC-BY-4.0; its
|
| Wikipedia-derived parts are CC BY-SA) · chat: smol-smoltalk (Apache-2.0), Aya dataset (Apache-2.0) ·
|
| Awadhi: Tulsidas's verses (public domain). See NOTICE.md for full attribution.
|
|
|
| **Copyright note:** an earlier Tulsidas corpus also contained a modern Hindi commentary (Gita Press,
|
| likely still under copyright). When this was noticed, at 0.8B of 2.5B training tokens (~0.1% of
|
| batches up to then), it was removed: the rest of pretraining, the chat tuning and everything in this
|
| repository use only public-domain verse. No commentary text is distributed here.
|
|
|
| ## Credits
|
| Built by Mrittunjay Dubey. Coding assisted by Claude (Anthropic).
|
|
|
