README.md

---
title: KYS School Scraper
emoji: 🏫
colorFrom: indigo
colorTo: gray
sdk: docker
pinned: false
---

# 🏫 KYS School Scraper & Data Processing Pipeline

An automated, end-to-end data pipeline to harvest school data from the KYS UDISE+ portal, process complex district boundary changes, and compile a finalized master dataset seamlessly on HuggingFace.

**Pipeline Flow:** `Scrape → Auto-push Raw → Map Districts → Build Mapped Master`

## ✨ Features & Capabilities

1. **Targeted Scraping:** Operates strictly at the **District Level**, capturing only 6 targeted school categories (3, 5, 6, 7, 10, 11).
2. **Intelligent Retries:** Automatically detects CAPTCHA failures and retries only missing data. The terminal UI wipes clean automatically before each operation.
3. **UDISE Geo-Decoding:** For non-actual state scrapes (e.g., KVS, NVS, NAVY, IAF), it parses the UDISE code to determine the *real* State, District, and Block.
4. **Automated UDISE-Tracking Math Engine:** When building the master sheet, an intelligent math engine automatically resolves renamed districts and blocks! 
   - It cross-references newly scraped UDISE codes against a cloud-stored Scholarship Application Baseline (`baseline_master.parquet`).
   - If a district was simply renamed (e.g., EAST DISTRICT -> GANGTOK), the math engine statistically tracks the UDISE codes, proves the rename, and automatically generates a mapping rule in the cloud (`manual_district_mapping.parquet`).
   - For complex fractured districts, it drills down to the Block level (`manual_block_mapping.parquet`) to perfectly map schools to their old districts.
5. **HuggingFace Cloud Architecture:** Syncs directly to a HuggingFace dataset organized into 3 folders:
   - `district_reference`: Source of truth for Scholarship Application district tracking (Dataset 1).
   - `mapping_rules`: Automated + manual district & block mapping rules (Dataset 2) & the SF Baseline Master.
   - `scraped_data`: Holds both `raw` per-state files and the final `mapped` master sheet.
6. **Smart District Flagging (Hybrid System):** Because the math engine automatically handles all simple renames behind the scenes, the UI only flags *genuinely brand-new* or highly ambiguous districts for your manual review in the Master Sheet tab!

---

## 🛠 Prerequisites & Setup

1. **Google Chrome** installed.
2. **Tesseract OCR (Windows):**
   - Download from UB-Mannheim (e.g., `tesseract-ocr-w64-setup-5.5.0.20241111.exe`).
   - Run the installer and add the path (usually `C:\Program Files\Tesseract-OCR`) to your System Environment Variables under `Path`.
3. **Python 3.10+**.
4. **Installation:**
   ```powershell
   python -m venv venv
   .\venv\Scripts\activate
   pip install -r requirements.txt
   playwright install chromium
   ```
5. **Environment Setup (Required for HuggingFace Dataset Sync):**
   - Rename `.env.example` to `.env` in the root folder.
   - Add your HuggingFace token and repository path:
     ```env
     HF_TOKEN=hf_your_actual_token_here
     HF_REPO=Apf-AI4Good/kys-school-data
     ```

---

## 🚀 The Web Application (Gradio)

The absolute easiest way to use the pipeline is via the interactive browser UI. Chromium launches **minimized in the background** so it never interrupts your workflow!

```powershell
python app.py
```
The app opens automatically at `http://127.0.0.1:7861` and features **4 main tabs**.

---

### 🔍 Tab 1: Scraper

The core scraping workflow, all on one screen:

1. **▶ Start Scraping**
   - Navigates the UDISE+ portal, finds all districts for the selected state, and collects data for the target categories.
2. **↻ Fix Missing Data (Retries)**
   - Scans for CAPTCHA failures and re-scrapes **only the failed records**. No successful data is ever lost.
3. **✅ Automatic Push**
   - Once a state finishes scraping without missing data, it is **automatically pushed** to the cloud in your HuggingFace dataset.

> **Repeat Steps 1–3 for every state you want to scrape. Then proceed to the other tabs.**

---

### 📋 Tab 2: Master Sheet

Combines all raw state data from HuggingFace into a single mapped master Excel file, and manages flagged districts.

- **State Coverage Table:** Connects to your HuggingFace dataset and displays which states are ready to build. You can delete raw data directly from here (which strictly auto-cleans any pending districts for that state!).
- **Review New Districts:** Automatically detects newly scraped districts not in your Scholarship Application database. You can instantly map them to older Scholarship Application districts or rename them before building.
- **Build Master Sheet:** Automatically:
  1. Pulls all raw complete state parquets.
  2. Applies geo-decoding for KVS/NVS/NAVY schools.
  3. Applies district back-mapping based on your rules.
  4. Tags schools with their `Status` (e.g., `present` or `new districts found`).
  5. Saves the final mapped file to `scraped_data/mapped/` on HuggingFace and provides an Excel download.

---

### 🗺️ Tab 3: Mapping Manager

A full database management dashboard to maintain your Scholarship Application mappings.

- **Scholarship Application District Reference:** View and manage your master district list.
  - **Click-to-Edit Rows:** Click any row in the table to instantly populate a quick-edit dropdown for updating its Status.
  - **Bulk Excel Import/Export:** Expand the accordion to download the table, make bulk edits in Excel, and drag-and-drop it back to seamlessly sync with the cloud.
  - **Smart CRUD Tools:** Add, Rename, and Delete districts. If you rename a district here, it perfectly cascades and updates your manual mapping rules automatically!

---

### 📥 Tab 4: Download History

- Easily browse, refresh, and download any previously built Master Sheets directly from your HuggingFace cloud storage without having to rebuild them!

---

## 🛠️ Admin Scripts

If you need to make global, architectural changes to the baseline data outside of the UI, use the provided admin scripts:

**`admin_rename_sf_district.py`**
If an old Scholarship Application district name is permanently outdated and you want to rename it everywhere (becoming the new Baseline truth):
```powershell
python admin_rename_sf_district.py --state "SIKKIM" --old "EAST DISTRICT (GANGTOK)" --new "GANGTOK"
```
This script updates Dataset 1, rewrites the Baseline Master, and **instantly re-runs the automated math engine** for that state to guarantee your mapping rules stay mathematically synchronized!

---

## 💻 Running via Terminal (Headless Mode)

```powershell
# Step 1: Main scrape
python -m pytest tests/test_scrape_districts_by_category.py -v -s --state "GOA"

# Step 2: Retry failures
python -m pytest tests/test_retry_districts_by_category.py -v -s --state "GOA"

# Step 3: Export raw Excel
python export_to_excel.py --state "GOA"
```
*(Filter categories: `$env:KYS_TARGET_CATEGORIES="3,5" ; python ...`)*

---

## 🌐 Hosting on Hugging Face Spaces (Docker)

1. Create a **Docker Space** on Hugging Face (Blank Template).
2. Upload all `.py` files, `requirements.txt`, `Dockerfile`, `README.md`, and the `tests/` and `pages/` folders. Do **NOT** upload `.env`, `.git`, or `venv/`.
3. In your Space **Settings > Variables and secrets**, add `HF_TOKEN` and `HF_REPO`.
4. The Dockerfile automatically installs Chromium and Tesseract OCR, and launches your Gradio app publicly!

---

## 📂 Project Structure & Output Files

| File | Purpose |
|---|---|
| `output/session_cookies.json` | Browser session cookies (auto-refreshed) |
| `output/goa_district_id_map.json` | District dropdown mapping IDs for the portal |
| `output/goa_schools_by_category.json` | Raw scraped JSON arrays & API responses |
| `output_excel/goa_Schools.xlsx` | Raw Excel Export (before mapping) |
| `HF Dataset: district_reference/` | Dataset 1: The Scholarship Application reference mapping |
| `HF Dataset: mapping_rules/` | Dataset 2: District and Block mapping rules |
| `HF Dataset: scraped_data/raw/{state}.parquet` | Per-state raw scraped data |
| `HF Dataset: scraped_data/mapped/` | Final combined mapped master output |