| # 🗂️ Data Workspace | |
| - [raw/](raw/) incoming source files | |
| - [processed/](processed/) cleaned/aligned artifacts | |
| - [metadata/](metadata/) manifests, speaker/dialect info, QA reports | |
| ## ✅ Verified External Datasets | |
| ### 🎙️ Common Voice Scripted Speech 24.0 - Pashto | |
| - Link: [Mozilla Data Collective - Common Voice Pashto 24.0](https://datacollective.mozillafoundation.org/datasets/cmj8u3pnb00llnxxbfvxo3b14) | |
| - Why useful: largest open community Pashto speech source for ASR training and evaluation. | |
| - How to use here: download to `data/raw/common_voice_scripted_ps_v24/` and follow [docs/common_voice_pashto_24.md](../docs/common_voice_pashto_24.md). | |
| ### 🌸 Google FLEURS (Pashto config) | |
| - Link: [huggingface.co/datasets/google/fleurs](https://huggingface.co/datasets/google/fleurs) | |
| - Pashto validation: [`fleurs.py` includes `"ps_af"`](https://huggingface.co/datasets/google/fleurs/blob/main/fleurs.py). | |
| - Why useful: standardized multilingual speech benchmark split for comparable ASR scores. | |
| - How to use here: treat as external eval set for [benchmarks/](../benchmarks/README.md) and avoid training/eval leakage. | |
| ### 📖 OSCAR Corpus (Pashto web text) | |
| - Link: [huggingface.co/datasets/oscar-corpus/oscar](https://huggingface.co/datasets/oscar-corpus/oscar) | |
| - Pashto validation: dataset includes `unshuffled_deduplicated_ps`. | |
| - Why useful: large-scale Pashto text for LM pretraining and lexicon expansion. | |
| - How to use here: normalize and sample into [processed/](processed/) for NLP/ASR language model support. | |
| ### 📰 Wikimedia Wikipedia (Pashto dump) | |
| - Link: [huggingface.co/datasets/wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | |
| - Pashto validation: subset includes `20231101.ps`. | |
| - Why useful: cleaner encyclopedia-style Pashto text for terminology and style balance. | |
| - How to use here: include as a high-quality text source in normalization and glossary workflows. | |
| ### 📘 Belebele (reading-comprehension benchmark) | |
| - Link: [huggingface.co/datasets/facebook/belebele](https://huggingface.co/datasets/facebook/belebele) | |
| - Pashto validation: subset includes `pbt_Arab`. | |
| - Why useful: useful downstream benchmark for comprehension-oriented NLP progress in Pashto. | |
| - How to use here: benchmark multilingual encoders and track improvements in [benchmarks/](../benchmarks/README.md). | |
| ### 🌐 OPUS-100 (parallel text, en-ps) | |
| - Link: [huggingface.co/datasets/Helsinki-NLP/opus-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) | |
| - Pashto validation: dataset viewer includes `en-ps` subset. | |
| - Why useful: parallel Pashto-English bitext for translation baselines and text normalization cross-checks. | |
| - How to use here: keep in external eval/training split plans and log subset/version in run cards. | |
| ### 🎤 Pashto Isolated Words Speech Dataset (Kaggle) | |
| - Link: [kaggle.com/datasets/engrirf/pashto-isolated-words-speech-dataset](https://www.kaggle.com/datasets/engrirf/pashto-isolated-words-speech-dataset) | |
| - Pashto validation: dataset title is explicitly Pashto isolated-word speech. | |
| - Why useful: useful for small-footprint ASR or keyword-spotting experiments. | |
| - How to use here: treat as task-specific speech data and document licensing/collection assumptions before use. | |
| ### 🧠 Pashto Word Embeddings (Kaggle) | |
| - Link: [kaggle.com/datasets/drijaz/pashto-word-embeddings](https://www.kaggle.com/datasets/drijaz/pashto-word-embeddings) | |
| - Pashto validation: dataset description states pretrained Pashto embeddings. | |
| - Why useful: quick-start lexical semantics baseline for NLP experiments. | |
| - How to use here: benchmark against transformer encoders in downstream Pashto tasks. | |
| ## First Contribution (Normalization Starter) | |
| - [processed/normalization_seed_v0.1.tsv](processed/normalization_seed_v0.1.tsv) starter normalization examples | |
| - [docs/pashto_normalization_v0.1.md](../docs/pashto_normalization_v0.1.md) baseline normalization policy | |
| - [scripts/validate_normalization.py](../scripts/validate_normalization.py) basic file validator | |
| ## 🧪 Validate Seed File | |
| ```bash | |
| python scripts/validate_normalization.py data/processed/normalization_seed_v0.1.tsv | |
| ``` | |
| ## 📝 Notes | |
| - Keep raw downloaded dataset files out of git. | |
| - Track source URL + version in experiment notes for reproducibility. | |
| - Re-check external links before every milestone release. | |