README.md

---
license: apache-2.0
datasets:
- Polygl0t/gigalekh-v1
- nvidia/OpenScience
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/finemath
- HuggingFaceTB/smollm-corpus
- allenai/big-reasoning-traces
- allenai/math-meta-reasoning-filtered
language:
- hi
- en
pipeline_tag: text-generation
tags:
- text-generation-inference
metrics:
- perplexity
library_name: transformers
widget:
- text: भारत की राजधानी क्या है?
  example_title: उदाहरण
- text: भारत का राष्ट्रीय पक्षी कौन सा है?
  example_title: उदाहरण
inference:
  parameters:
    repetition_penalty: 1.2
    temperature: 0.1
    top_k: 50
    top_p: 1
    max_new_tokens: 150
co2_eq_emissions:
  emissions: 513050
  source: CodeCarbon
  training_type: pre-training
  geographical_location: Germany
  hardware_used: NVIDIA A100-SXM4-80GB
model-index:
- name: LilMoo-v0.2
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: CSQA-HI
      type: ai4bharat/indic_glue
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc_norm
      value: 36.52
      name: Acc-norm
    source:
      url: https://github.com/Polygl0t/lm-evaluation-harness
      name: indic_glue
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MILU-HI
      type: ai4bharat/MILU
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc_norm
      value: 29.06
      name: Acc-norm
    source:
      url: https://github.com/AI4Bharat/MILU
      name: milu
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: ARC Challenge
      type: arc_challenge_poly_hi
      args:
        num_few_shot: 5
    metrics:
    - type: acc_norm
      value: 30.47
      name: Acc-norm
    source:
      url: https://github.com/Polygl0t/lm-evaluation-harness/tree/arc_poly
      name: arc_challenge_poly_hi
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag
      type: hellaswag_poly_hi
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 37.18
      name: Acc-norm
    source:
      url: https://github.com/Polygl0t/lm-evaluation-harness/tree/hellaswag_poly
      name: hellaswag_poly_hi
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU
      type: mmlu_poly_hi
      args:
        num_few_shot: 5
    metrics:
    - type: acc_norm
      value: 28.21
      name: Acc-norm
    source:
      url: https://github.com/Polygl0t/lm-evaluation-harness/tree/mmlu_poly
      name: mmlu_poly_hi
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Global PIQA
      type: global_piqa_hi
      args:
        num_few_shot: 5
    metrics:
    - type: acc_norm
      value: 63
      name: Acc-norm
    source:
      url: https://github.com/Polygl0t/lm-evaluation-harness
      name: global_piqa_completions_hin_deva
---
# LilMoo-v0.2

<img src="./logo_lilmoo.png" alt="A round logo of a cartoon cow in a hoodie, cap, and gold chain with the text 'LilMoo' below." height="80" style="display: block; margin: 0 auto;">

## Model Summary

**[LilMoo-v0.2](https://huggingface.co/Polygl0t/LilMoo-v0.2)** is a decoder-transformer natively pretrained in Hindi and English. LilMoo is part of the [Polygl0t](https://huggingface.co/Polygl0t) initiative to advance language models for low-resource languages.

## Details

- **Architecture:** a Transformer-based model ([`llama`](https://huggingface.co/docs/transformers/main/en/model_doc/llama))
- **Size:** 670,127,616 parameters
- **Context length:** 4096 tokens
- **Dataset(s):** 
  - [- Polygl0t/gigalekh-v1](https://huggingface.co/datasets/Polygl0t/gigalekh-v1)
  - [allenai/big-reasoning-traces](https://huggingface.co/datasets/allenai/big-reasoning-traces)
  - [HuggingFaceTB/smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)
  - [HuggingFaceTB/finemath](https://huggingface.co/datasets/HuggingFaceTB/finemath)
  - [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
  - [allenai/math-meta-reasoning-filtered](https://huggingface.co/datasets/allenai/math-meta-reasoning-filtered)
  - [nvidia/OpenScience](https://huggingface.co/datasets/nvidia/OpenScience)
- **Language(s):** Hindi, English
- **Batch size:** 2,097,152 tokens
- **Number of steps:** 170,000
- **GPU:** 8 NVIDIA A100-SXM4-80GB
- **Training time**: ~ 327.50 hours
- **Emissions:** 513.05 KgCO2eq (Germany)
- **Total energy consumption:** 1346.76 kWh

This repository has the [source code](https://github.com/Polygl0t/llm-foundry) used to train this model. The full configuration used for training is available in the following config files:

- Stage 1 (warmup + stable): [config_stage_1.yaml](config_stage_1.yaml)
- Stage 2 (stable): [config_stage_2.yaml](config_stage_2.yaml)
- Stage 3 (stable + decay): [config_stage_3.yaml](config_stage_3.yaml)

### Learning Rate

This model was trained with a linear learning rate (i.e., warmup-stable-decay). Warmup was done over the first 2,000 steps to a peak learning rate of 7e-4, followed by a stable phase until step 149,000 (this revision is named `end-of-stable-ckpt`), and then a negative square-root decay to 0 for the last 21,000 steps. Checkpoints were saved every 5,000 steps, which equates to approximately 10 billion tokens. The `end-of-stable-ckpt` can be used for continuing training with a stable learning rate if desired.

The main branch of this repository contains the final checkpoint saved at step 170,000. All other checkpoints are available as separate branches. To load a specific checkpoint, you can use the following code snippet:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Polygl0t/LilMoo-v0.2"
revision = "stage1-step-05000"  # Change this to the desired checkpoint branch
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision)
```

Or, you can access all the revisions for the models via the following code snippet:

```python
from huggingface_hub import list_repo_refs
out = list_repo_refs("Polygl0t/LilMoo-v0.2")
branches = [b.name for b in out.branches]
print(branches)
```

<details>
<summary><b>Learning Curves</b></summary>

![Learning Curves](./plots/loss_curves.png)

</details>
</details>

<details>
<summary><b>Gradient Norm (L2)</b></summary>

![gradient_statistics_v2](./plots/gradient_norm_v2.png)

</details>

## Intended Uses

The primary intended use LilMoo is to serve as foundations for research and development involving native Hindi language modeling. Checkpoints saved during training are designed to provide a controlled setting for performing comparative experiments, specifically regarding the effects of active pretraining on the performance of currently available benchmarks. You may also fine-tune and adapt LilMoo for deployment if your use follows the Apache 2.0 license. If you decide to use LilMoo as a basis for your fine-tuned model, please conduct your own risk and bias assessment.

## Out-of-scope Use

- LilMoo is **not intended for deployment**. It is not an out-of-the-box product and should not be used for human-facing interactions.

- LilMoo is for **the Hindi language only** and is unsuitable for text generation tasks in other languages.

- LilMoo has **not been fine-tuned** for downstream tasks.


## Basic usage

```python
from transformers import GenerationConfig, TextGenerationPipeline, AutoTokenizer, AutoModelForCausalLM
import torch

# Specify the model and tokenizer
model_id = "Polygl0t/LilMoo-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Specify the generation parameters as you like
generation_config = GenerationConfig(
    **{
    "do_sample": True,
    "max_new_tokens": 150,
    "renormalize_logits": True,
    "repetition_penalty": 1.2,
    "temperature": 0.1,
    "top_k": 50,
    "top_p": 1.0,
    "use_cache": True, 
  }
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
generator = TextGenerationPipeline(model=model, task="text-generation", tokenizer=tokenizer, device=device)

# Generate text
prompt = "भारत की राजधानी क्या है?"
completion = generator(prompt, generation_config=generation_config)
print(completion[0]['generated_text'])
```

## Limitations

Like almost all other language models trained on large text datasets scraped from the web, the LilMoo shows behavior that does not make it an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. LilMoo is subject to the following:

- **Hallucinations:** LilMoo can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.

- **Biases and Toxicity:** LilMoo inherits the social and historical stereotypes from the data used to train it. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.

- **Language Limitations:** LilMoo is primarily designed to interact with Hindi. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.

- **Repetition and Verbosity:** LilMoo may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.

Hence, even though LilMoo is released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications.

## Evaluations

The table below compares our two versions of LilMoo against other base models of similar size. The NPM (Normalized Performance Metric) is a metric designed to provide a balanced view of model performance across various tasks, accounting for the inherent difficulty of each task. It normalizes the performance of each model on a given task by comparing it to a baseline performance, which represents a the performance of a random model.

**Note:** In the table and plots below, benchmark scores are reported as normalized accuracy, and NPM is calculated from accuracy.

|                   | NPM (mean) |  ARC  | HellaSwag |   MMLU   |  CSQA    | MILU     | Global PIQA  |
|-------------------|------------|-------|-----------|----------|----------|--------- |--------------|
| LilMoo-v0.2       | 9.94       | 30.47 | 37.18     | 28.21    |  36.52   | 29.06    |  63.00       |
| LilMoo-v0.1       | 8.7        | 30.05 | 37.06     | 26.19    |  37.12   | 28.55    |  65.00       |
| Qwen3-0.6B-Base   | 4.08       | 27.48 | 29.82     | 31.05    |  30.96   | 30.93    |  53.00       |
| Qwen2.5-0.5B      | 1.92       | 23.71 | 28.63     | 27.78    |  30.01   | 28.48    |  57.00       |



<details>
<summary><b>🏆 HellaSwag</b></summary>
  
![hellaswag](./plots/hellaswag.png)
</details>

<details>
<summary><b>🏆 ARC </b></summary>
  
![arc_challenge](./plots/arc.png)
</details>

<details>
<summary><b>🏆 MMLU</b></summary>
  
![mmlu](./plots/mmlu.png)
</details>

<details>
<summary><b>🏆 MILU</b></summary>
  
![mmlu](./plots/milu.png)
</details>

<details>
<summary><b>🏆 CSQA</b></summary>
  
![mmlu](./plots/csqa.png)
</details>

<details>
<summary><b>🏆 Global PIQA</b></summary>
  
![mmlu](./plots/global_piqa.png)
</details>


<details>
<summary><b>Aggregate NPM Across Benchmarks</b></summary>
  
![NPM vs Compute](./plots/npm.png)
</details>

<details>
<summary><b>Performance vs Compute</b></summary>
  
![Performance vs Compute](./plots/npm_vs_compute.png)
This plot compares the compute requirements (measured as C = 6 * N * D, where N is the number of parameters and D is the number of tokens processed) against the performance of each model (measured by NPM mean). It highlights the trade-offs between model size, training data, and performance. LilMoo models are compared against two baseline models: Qwen2.5-0.5B and Qwen3-0.6B-Base, which are state-of-the-art multilingual models.  
</details>


<details>
<summary><b>Text Generation Examples</b></summary>

All models were prompted with the same Hindi prompts, and the generations were produced with the default parameters specified in the `generation_config.json` of each model. All models were set to generate up to 250 new tokens. LilMoo models generally produce more text given their custom tokenizer and vocabulary, which is more adapted to Hindi.

<table  border="1"  cellpadding="5"  cellspacing="0">
  <thead>
    <tr>
      <th>Model</th>
      <th>Prompt</th>
      <th>Generated Completion</th>
    </tr>
  </thead>
   <tbody>
    <tr>
      <td>Qwen/Qwen2.5-0.5B</td>
      <td>
      <span  style="color: red;">मेरा पसंदीदा गाना रेडियो पर आया।</span>
      </td>
      <td>
      <i>मेरा पसंदीदा गाना रेडियो पर आया। जो इस बाहरी दुनिया में बार भी नहीं हुए थे। लेकिन अगर फिर अपनी मूर्ति बचेंगे तो तैयार कर सकते हुए बेनिस्टार की मांग आरेखी नियुक्त कर लेते हैं। लेकिन तो तैयार करने के लिए तैयारी को अस्तित्व करने के लिए अपने दोस्तों को समझा दें। मेटल अवधि नहीं। हे मैं भी बिल बेनिस्... </i>
      </td>
    </tr>
    <tr>
      <td></td>
      <td>
      <span  style="color: red;">भारत का राष्ट्रीय पक्षी कौन सा है?</span>
      </td>
      <td>
      <i>शांती किस तरह चली गई थी जिसे भारत की क्रियाएं मिली जो बारी, मामलों वाले जांच शामिल होती हैं?\nA. भी खोल रहे\nB. खोला लगा\nC. भी खोला\nD. खोला रहे\nAnswer: C\n\n199. In the event of the termination of a labor contract, if the employer violates the provisions of the "Labor Contract Law", causing damage to the employee, the employee may demand that the employer bear legal responsibility. The employer shall bear legal responsibility within the scope of the contractual liability scope and shall compensate the employee for losses in accordance with the provisions of the ... </i>
      </td>
    </tr>
    <tr>
      <td>Qwen/Qwen3-0.6B</td>
      <td>
      <span  style="color: red;">मेरा पसंदीदा गाना रेडियो पर आया।</span>
      </td>
      <td>
      <i>'मेरा पसंदीदा गाना रेडियो पर आया। अब बाराज़ी आए रहे। अपने बाराज़ी आए रहे। जाने कि बाराज़ी रहे रहे। अपने बाराज़ी आए रहे। जाने कि बाराज़ी रहे रहे। अपने बाराज़ी आए रहे। जाने कि बाराज़ी रहे रहे। अपने बाराज़ी आए रहे। जाने कि बाराज़ी रहे रहे। अपने बाराज़ी आए रहे। जाने कि बाराज़ी रहे...</i>
      </td>
    </tr>
    <tr>
      <td></td>
      <td>
      <span  style="color: red;">भारत का राष्ट्रीय पक्षी कौन सा है?</span>
      </td>
      <td>
      <i>भारत का राष्ट्रीय पक्षी है । संगठन में आपको असमान विशेषजन बरामर जाति और उद्देश्य लवण एक चुने जास्त छोटे घड़े गए थे, इसलिए भारत का राष्ट्रीय पक्षी धर्म और विकास खुश नहीं था। ऐसे विपक्षी राष्ट्र ने अधिक फौजी जाणना और अच्छी जाणना जैसे अन्य विकास विकास के लिए अपनी ओर जाहीर कर दिया।\n\nउत्तर: (A) राष्ट्रीय विकास विकास\n(B) राष्ट्रीय विकास और विकास विकास\n(C) राष्ट्रीय विकास और विकास विकास और विकास विकास\n(D) राष्ट्रीय विकास और विकास विकास और विकास विकास और विकास विकास ...</i>
      </td>
    </tr>
    <tr>
      <td>LilMoo-v0.1</td>
      <td>
      <span  style="color: red;">मेरा पसंदीदा गाना रेडियो पर आया।</span>
      </td>
      <td>
      <i> मेरा पसंदीदा गाना रेडियो पर आया। मैंने इसे एक साल पहले सुना था, लेकिन अब मैं इसके अर्थ को अधिक बारीकी से समझने की कोशिश कर रहा हूं और इस गीत के बोल मेरे लिए कितने आकर्षक थे! मुझे यह जानने में बहुत दिलचस्पी थी कि इसका क्या मतलब है? अगर कोई मुझसे पूछे तो मेरा जवाब होगा "इतिहास"-यह सिर्फ संगीत का टुकड़ा नहीं हो सकता क्योंकि वे सभी अलग हैं जो आप चाहते होंगे (और अधिकांश लोग वैसे भी इतिहासकार होने जा रहे हैंः पी) खैर वहाँ आपके पास इबॉप गिटारवादक द्वारा प्रस्तुत किया गया सबसे अच्छा ऑडियो नमूना हैः...</i>
      </td>
    </tr>
    <tr>
      <td></td>
      <td>
      <span  style="color: red;">भारत का राष्ट्रीय पक्षी कौन सा है?</span>
      </td>
      <td>
      <i>उत्तर: मोर (Mor) – भारत में पाए जाने वाले सबसे बड़े पक्षियों की श्रेणी के अंतर्गत आने वाला भारतीय राज्य उत्तर प्रदेश, उत्तराखंड और हिमाचल प्रदश हैं। इन राज्यों को क्रमशः ‘राष्ट्रीय पशु’ तथा ’राज्य-पशु’, कहा जाता था लेकिन अब इन्हें बदलकर केवल “मोर” कर दिया गया क्योंकि यह एक ऐसा जीव होता जो अपने पंखों से किसी भी वस्तु पर आसानी से हमला करता हुआ दिखाई देता हो इसलिए इसे राष्ट्रिय प्रतीक चिन्हों/चिह्नों / प्रतीकों आदि मे शामिल किया जा चुका हे ।...</i>
      </td>
    </tr>
    <tr>
      <td>LilMoo-v0.2</td>
      <td>
      <span  style="color: red;">मेरा पसंदीदा गाना रेडियो पर आया।</span>
      </td>
      <td>
      <i>मेरा पसंदीदा गाना रेडियो पर आया। इस गीत को मेरी दादी द्वारा सुना गया था, और हमें बहुत खुशी थी कि यह हमारे पूर्वजों के साथ इतना लंबा चला! जब मैंने अपने बच्चों से पूछा तो उन्होंने मुझसे कहा "नहीं पापा-ये गाने दादाओं ने नहीं गाए थे" लेकिन मेरे पिता जो कुछ भी कहते हैं उसे स्वीकार करते हुए मैं यही कहना चाहता हूंः इन दिनों युवाओं की संगीत प्राथमिकताएँ बदल गई थीं; आजकल युवा पॉप का रूप पसंद करेंगे न कि किसी प्रकार का पारंपरिक लोक धुन जैसा हम सदियों पहले सुनते रहे ...</i>
      </td>
    </tr>
    <tr>
      <td></td>
      <td>
      <span  style="color: red;">भारत का राष्ट्रीय पक्षी कौन सा है?</span>
      </td>
      <td>
      <i>भारत का राष्ट्रीय पक्षी कौन सा है?\nउत्तरः- मोर भारत की राष्ट्रिय चिड़िया हैं तथा इसे राजकीय पशु के रूप में भी मान्यता प्राप्त हैं। यह भारतीय संस्कृति एवं परंपराओं से जुड़ा हुआ एक अत्यंत महत्वपूर्ण प्राणी है जो कि अपनी सुंदरता, रंगो और पंखों के कारण ही काफी प्रसिद्ध हैं। ...</i>
      </td>
    </tr>
  </tbody>
</table>
</details>

## Cite as 🤗 

```latex
@misc{shiza2026lilmoo,
      title={{Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi}}, 
      author={Shiza Fatimah and Aniket Sen and Sophia Falk and Florian Mai and Lucie Flek and Nicholas Kluge Corr{\^e}a},
      year={2026},
      eprint={2603.03508},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.03508}, 
}
```

## Aknowlegments

Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.

We also gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing & Analytics Lab.

## License

LilMoo is licensed under the Apache License, Version 2.0. For more details, see the [LICENSE](LICENSE) file.