README.md

---
title: PRWP Dataset Extraction Dashboard
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
short_description: Explore dataset extraction and adoption trends in PRWPs
---

# PRWP Dataset Extraction Dashboard

This interactive dashboard visualizes dataset adoption, specificity-aware extraction metrics, and linkages across World Bank Policy Research Working Papers (PRWP). 

It is designed to evaluate how datasets are utilized, cited, and distributed in economic policy research.

## Core Features

- **Data Adoption Trends**: View general historical data use adoption rates (2000–2026) or filter by top individual datasets (e.g., DHS, LSMS, WDI) to see their specific share over time.
- **Mentions Distribution Analysis**: Slice all extracted dataset mentions dynamically by typology (Surveys, Administrative Data, Census, etc.), specificity (Named, Descriptive, Vague), usage context (Primary, Supporting, Citation), and usage status (Used in Analysis vs. Citation Only).
- **Dataset Explorer**: Search and filter deduplicated canonical datasets, inspect their document frequencies, and slide open details drawers showing citing papers and raw text variants.
- **Interactive Knowledge Graph**: Explore relationships between research papers (amber diamonds) and canonical dataset entities (indigo dots) using a force-directed network graph.

## Specificity-Aware Extraction Framework
To capture the full spectrum of data use, the underlying extraction model (`ai4data/datause-extraction`) classifies mentions into three specificity categories:
1. **Named Mentions** (78,165 references): Formal, named dataset entities (e.g., *Living Standards Measurement Study*).
2. **Descriptive Mentions** (34,433 references): Informal data type references (e.g., *administrative household survey*).
3. **Vague Mentions** (53,196 references): Generic dataset terms (e.g., *our survey data*).

## Replication & Pipeline
The generator scripts and mapping metadata are hosted directly within the space repository in the `/generation` folder:
* `generate_dashboard_data.py`: Main compiler script that aggregates JSON extraction outputs, applies canonical mapping, and generates the dashboard payload.
* `run_harmonization.py`: Entity resolution and variant clustering pipeline.
* `canonical_map.json`: Vectorized mapping of name variants to formal canonical entities.