Data
Place datasets here by processing stage.
raw/: original downloaded or manually collected data.cleaned/: normalized Malayalam text.deduplicated/: cleaned text after exact and fuzzy deduplication.tokenized/: model-ready tokenized datasets.
Do not commit large datasets directly unless they are intentionally small samples.