README.md

---
title: README
emoji: 🏆
colorFrom: pink
colorTo: indigo
sdk: static
pinned: false
---

# Indic Language Benchmarking for Large Language Models
### India is diverse with 22+ languages. This project aims to benchmark the performance of large language models on Indic languages across datasets. Goal is to evaluate a models abilities in understanding, generating, and processing text in these languages.
#### We currently have 8 languages across 3 datasets, more coming soon

## Languages
- Bengali (bn)
- Gujarati (gu)
- Hindi (hi)
- Kannada (kn)
- Malayalam (ml)
- Odiya (or)
- Tamil (ta)
- Telugu (te)

## Datasets
- ARC-Challenge: [hi](https://huggingface.co/datasets/indicbench/arc_hi), [bn](https://huggingface.co/datasets/indicbench/arc_bn), [gu](https://huggingface.co/datasets/indicbench/arc_gu), [kn](https://huggingface.co/datasets/indicbench/arc_kn), [ml](https://huggingface.co/datasets/indicbench/arc_ml), [or](https://huggingface.co/datasets/indicbench/arc_or), [ta](https://huggingface.co/datasets/indicbench/arc_ta), [te](https://huggingface.co/datasets/indicbench/arc_te)
- TruthfulQA: [hi](https://huggingface.co/datasets/indicbench/truthfulqa_hi), [bn](https://huggingface.co/datasets/indicbench/truthfulqa_bn), [gu](https://huggingface.co/datasets/indicbench/truthfulqa_gu), [kn](https://huggingface.co/datasets/indicbench/truthfulqa_kn), [ml](https://huggingface.co/datasets/indicbench/truthfulqa_ml), [or](https://huggingface.co/datasets/indicbench/truthfulqa_or), [ta](https://huggingface.co/datasets/indicbench/truthfulqa_ta), [te](https://huggingface.co/datasets/indicbench/truthfulqa_te)
- Hellaswag: [hi](https://huggingface.co/datasets/indicbench/hellaswag_hi), [bn](https://huggingface.co/datasets/indicbench/hellaswag_bn), [gu](https://huggingface.co/datasets/indicbench/hellaswag_gu), [kn](https://huggingface.co/datasets/indicbench/hellaswag_kn), [ml](https://huggingface.co/datasets/indicbench/hellaswag_ml), [or](https://huggingface.co/datasets/indicbench/hellaswag_or), [ta](https://huggingface.co/datasets/indicbench/hellaswag_ta), [te](https://huggingface.co/datasets/indicbench/hellaswag_te)

### Code 
[Eval Harness](https://github.com/indicbench/lm-evaluation-harness)

#### We are also trying to build an MMLU dataset with Indian Knowledge. If anyone is interested in contributing, please reach out to [Ram](https://twitter.com/ram_chandalada), [Munish](https://twitter.com/kumar_munish_)