# 🗂️ Data Workspace

- [raw/](raw/) incoming source files
- [processed/](processed/) cleaned/aligned artifacts
- [metadata/](metadata/) manifests, speaker/dialect info, QA reports

## ✅ Verified External Datasets

### 🎙️ Common Voice Scripted Speech 24.0 - Pashto
- Link: [Mozilla Data Collective - Common Voice Pashto 24.0](https://datacollective.mozillafoundation.org/datasets/cmj8u3pnb00llnxxbfvxo3b14)
- Why useful: largest open community Pashto speech source for ASR training and evaluation.
- How to use here: download to `data/raw/common_voice_scripted_ps_v24/` and follow [docs/common_voice_pashto_24.md](../docs/common_voice_pashto_24.md).

### 🌸 Google FLEURS (Pashto config)
- Link: [huggingface.co/datasets/google/fleurs](https://huggingface.co/datasets/google/fleurs)
- Pashto validation: [`fleurs.py` includes `"ps_af"`](https://huggingface.co/datasets/google/fleurs/blob/main/fleurs.py).
- Why useful: standardized multilingual speech benchmark split for comparable ASR scores.
- How to use here: treat as external eval set for [benchmarks/](../benchmarks/README.md) and avoid training/eval leakage.

### 📖 OSCAR Corpus (Pashto web text)
- Link: [huggingface.co/datasets/oscar-corpus/oscar](https://huggingface.co/datasets/oscar-corpus/oscar)
- Pashto validation: dataset includes `unshuffled_deduplicated_ps`.
- Why useful: large-scale Pashto text for LM pretraining and lexicon expansion.
- How to use here: normalize and sample into [processed/](processed/) for NLP/ASR language model support.

### 📰 Wikimedia Wikipedia (Pashto dump)
- Link: [huggingface.co/datasets/wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)
- Pashto validation: subset includes `20231101.ps`.
- Why useful: cleaner encyclopedia-style Pashto text for terminology and style balance.
- How to use here: include as a high-quality text source in normalization and glossary workflows.

### 📘 Belebele (reading-comprehension benchmark)
- Link: [huggingface.co/datasets/facebook/belebele](https://huggingface.co/datasets/facebook/belebele)
- Pashto validation: subset includes `pbt_Arab`.
- Why useful: useful downstream benchmark for comprehension-oriented NLP progress in Pashto.
- How to use here: benchmark multilingual encoders and track improvements in [benchmarks/](../benchmarks/README.md).

### 🌐 OPUS-100 (parallel text, en-ps)
- Link: [huggingface.co/datasets/Helsinki-NLP/opus-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100)
- Pashto validation: dataset viewer includes `en-ps` subset.
- Why useful: parallel Pashto-English bitext for translation baselines and text normalization cross-checks.
- How to use here: keep in external eval/training split plans and log subset/version in run cards.

### 🎤 Pashto Isolated Words Speech Dataset (Kaggle)
- Link: [kaggle.com/datasets/engrirf/pashto-isolated-words-speech-dataset](https://www.kaggle.com/datasets/engrirf/pashto-isolated-words-speech-dataset)
- Pashto validation: dataset title is explicitly Pashto isolated-word speech.
- Why useful: useful for small-footprint ASR or keyword-spotting experiments.
- How to use here: treat as task-specific speech data and document licensing/collection assumptions before use.

### 🧠 Pashto Word Embeddings (Kaggle)
- Link: [kaggle.com/datasets/drijaz/pashto-word-embeddings](https://www.kaggle.com/datasets/drijaz/pashto-word-embeddings)
- Pashto validation: dataset description states pretrained Pashto embeddings.
- Why useful: quick-start lexical semantics baseline for NLP experiments.
- How to use here: benchmark against transformer encoders in downstream Pashto tasks.

## First Contribution (Normalization Starter)
- [processed/normalization_seed_v0.1.tsv](processed/normalization_seed_v0.1.tsv) starter normalization examples
- [docs/pashto_normalization_v0.1.md](../docs/pashto_normalization_v0.1.md) baseline normalization policy
- [scripts/validate_normalization.py](../scripts/validate_normalization.py) basic file validator

## 🧪 Validate Seed File
```bash
python scripts/validate_normalization.py data/processed/normalization_seed_v0.1.tsv
```

## 📝 Notes
- Keep raw downloaded dataset files out of git.
- Track source URL + version in experiment notes for reproducibility.
- Re-check external links before every milestone release.