README.md

---
dataset_info:
  config_name: default
  features:
    - name: id
      dtype: string
    - name: language
      dtype: string
    - name: english
      dtype: string
    - name: native_script_codemixed
      dtype: string
    - name: full_native_script
      dtype: string
    - name: romanized_casual
      dtype: string
    - name: audio
      dtype: audio
configs:
  - config_name: default
    data_files:
      - split: train
        path: data/train-*
license: mit
task_categories:
  - translation
  - text-generation
  - automatic-speech-recognition
language:
  - as
  - bn
  - gu
  - hi
  - kn
  - ml
  - mr
  - or
  - pa
  - ta
  - te
  - en
pretty_name: IndicCMix
size_categories:
  - 1M<n<10M
tags:
  - code-mixed
  - code-switching
  - indic
  - multilingual
  - transliteration
  - romanization
  - text-to-text
  - tts-augmented
  - f5-tts
---

# IndicCMix

Most Indic NLP data assumes people write in one script and one language at a time. Real chat looks nothing like that. You get Hindi words in Roman letters, English verbs in the middle of a Tamil sentence, and the same person switching scripts halfway through a paragraph.

This dataset is an attempt to cover that actual messiness. For every English sentence, you get three different Indic renderings of it: one code-mixed in the native script, one clean native-script translation, and one romanized version of the kind people type on WhatsApp. Each English sentence also comes with an audio clip, so if you want to do speech-side experiments, the data supports that too.

## What's in the dataset

| | |
|---|---|
| Rows | 1,152,750 |
| Languages | 11 Indic + English |
| Unique English sentences | ~104,809 |
| Audio files | ~104,809 WAVs, ~31 GB total |
| Audio source | Synthesized with [F5-TTS](https://github.com/SWivid/F5-TTS) |
| Viewer preview | 110 rows with playable audio |

## Fields

| Field | What it is |
|---|---|
| `id` | Row identifier, e.g. `hi_000042` |
| `language` | One of `as`, `bn`, `gu`, `hi`, `ka`, `ml`, `mr`, `or`, `pa`, `ta`, `te` |
| `english` | The source sentence in English |
| `native_script_codemixed` | Indic script with English words left in place, the way people actually code-switch |
| `full_native_script` | Full translation into the target language, no English words |
| `romanized_casual` | Roman-letter version, WhatsApp style |
| `audio` | F5-TTS English speech of the `english` field |

### One row in full (Hindi)

```
id:                        hi_314203
language:                  hi
english:                   The Indian festival organizers partnered with a fintech company to introduce digital payment options like Paytm and UPI for ticket bookings and donations.
native_script_codemixed:   भारतीय festival organizers ने एक fintech company के साथ partnership की है ताकि ticket bookings और donations के लिए Paytm और UPI जैसे digital payment options शुरू कर सकें।
full_native_script:        भारतीय फेस्टिवल ऑर्गनाइजर्स ने एक फिनटेक कंपनी के साथ पार्टनरशिप की है ताकि टिकट बुकिंग्स और डोनेशन्स के लिए पेयटएम और यूपीआई जैसे डिजिटल पेमेंट ऑप्शन्स शुरू कर सकें।
romanized_casual:          Bharatiya tyohar aayojakon ne ek fintech company ke saath milkar ticket booking aur donation ke liye Paytm aur UPI jaise digital payment options shuru kiye hain.
audio:                     audio/XX/000630.wav
```

## Languages and row counts

| Code | Language   | Rows    |
|------|------------|---------|
| as   | Assamese   | 104,788 |
| bn   | Bengali    | 104,795 |
| gu   | Gujarati   | 104,796 |
| hi   | Hindi      | 104,800 |
| ka   | Kannada    | 104,798 |
| ml   | Malayalam  | 104,798 |
| mr   | Marathi    | 104,791 |
| or   | Odia       | 104,794 |
| pa   | Punjabi    | 104,797 |
| ta   | Tamil      | 104,797 |
| te   | Telugu     | 104,796 |

Each English sentence goes into all 11 languages, so the audio files are shared. Only the Indic text differs between configs.

## How it was made

A corpus of roughly 104k English sentences was translated into all 11 Indic languages. `native_script_codemixed` is the base form, with English words kept as-is in Roman letters. `full_native_script` is derived from it by transliterating those English tokens into the target script, so the two are aligned word-for-word and safe to use as parallel data. `romanized_casual` is generated in a separate pass, so it carries the same meaning but is not token-aligned with the other two. The English sentences were then fed to F5-TTS to generate the audio.

## Loading

Preview (small, with playable audio):

```python
from datasets import load_dataset

ds = load_dataset("ai4bharat/IndicCMix", split="train")
print(ds[0])
```

One language, text only (fastest):

```python
from datasets import load_dataset

ds = load_dataset("ai4bharat/IndicCMix", data_files="hi.parquet", split="train")
```

One language with playable audio:

```python
from datasets import load_dataset, Audio

ds = load_dataset("ai4bharat/IndicCMix", data_files="hi.parquet", split="train")
ds = ds.cast_column("audio", Audio())
```

Everything at once:

```python
from datasets import load_dataset, Audio

files = [f"{l}.parquet" for l in ["as","bn","gu","hi","ka","ml","mr","or","pa","ta","te"]]
ds = load_dataset("ai4bharat/IndicCMix", data_files=files, split="train")
ds = ds.cast_column("audio", Audio())
```

Offline snapshot:

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="ai4bharat/IndicCMix",
    repo_type="dataset",
    local_dir="./IndicCMix",
)
```

## What you can actually do with it

On the text side, the obvious ones: English to code-mixed generation, English to native-script translation, and transliteration between romanized and native Indic. Less obvious but more interesting: training a single model that can switch between all three output styles on command, or probing an LLM to see whether it can code-switch at all.

On the speech side, you get English audio paired with Indic text in three forms, which is a strange and useful setup. You can train speech-to-code-mixed or speech-to-romanized models directly. Keep in mind the audio is synthetic (F5-TTS), so models trained only on this will need real recordings before they are useful in production.

## Repo layout

```
├── data/
│   └── train-00000-of-00001.parquet   # Preview with embedded audio
├── as.parquet, bn.parquet, ...        # Full per-language files (audio as paths)
├── audio/
│   ├── 00/ ... ff/                    # WAVs sharded into 256 subdirs
│   │   └── *.wav
└── README.md
```

The audio is split into 256 subdirectories by MD5 of filename. Hugging Face caps directories at 10,000 files, and flat audio would blow past that.

## A note on the viewer

The viewer only shows 110 rows because embedding audio bytes for all 1.15M rows would need roughly 340 GB of parquet, which is not a thing anyone wants to download just to browse. The full audio lives as separate WAVs in `audio/` and gets resolved when you `cast_column("audio", Audio())` after loading.

## License

MIT.