Hugging Face's logo

Spaces:
Apf-AI4Good
/
kys-school-scraper
Running

App Files Community
kys-school-scraper / README.md
sharanyaswarup's picture
sharanyaswarup
Updated mapping manager
13b211e
preview code
|
Raw
History Blame Contribute Delete
8.13 kB
metadata 
title: KYS School Scraper
emoji: 🏫
colorFrom: indigo
colorTo: gray
sdk: docker
pinned: false

🏫 KYS School Scraper & Data Processing Pipeline

An automated, end-to-end data pipeline to harvest school data from the KYS UDISE+ portal, process complex district boundary changes, and compile a finalized master dataset seamlessly on HuggingFace.

Pipeline Flow: Scrape → Auto-push Raw → Map Districts → Build Mapped Master

✨ Features & Capabilities

  1. Targeted Scraping: Operates strictly at the District Level, capturing only 6 targeted school categories (3, 5, 6, 7, 10, 11).
  2. Intelligent Retries: Automatically detects CAPTCHA failures and retries only missing data. The terminal UI wipes clean automatically before each operation.
  3. UDISE Geo-Decoding: For non-actual state scrapes (e.g., KVS, NVS, NAVY, IAF), it parses the UDISE code to determine the real State, District, and Block.
  4. Automated UDISE-Tracking Math Engine: When building the master sheet, an intelligent math engine automatically resolves renamed districts and blocks!
    • It cross-references newly scraped UDISE codes against a cloud-stored Scholarship Application Baseline (baseline_master.parquet).
    • If a district was simply renamed (e.g., EAST DISTRICT -> GANGTOK), the math engine statistically tracks the UDISE codes, proves the rename, and automatically generates a mapping rule in the cloud (manual_district_mapping.parquet).
    • For complex fractured districts, it drills down to the Block level (manual_block_mapping.parquet) to perfectly map schools to their old districts.
  5. HuggingFace Cloud Architecture: Syncs directly to a HuggingFace dataset organized into 3 folders:
    • district_reference: Source of truth for Scholarship Application district tracking (Dataset 1).
    • mapping_rules: Automated + manual district & block mapping rules (Dataset 2) & the SF Baseline Master.
    • scraped_data: Holds both raw per-state files and the final mapped master sheet.
  6. Smart District Flagging (Hybrid System): Because the math engine automatically handles all simple renames behind the scenes, the UI only flags genuinely brand-new or highly ambiguous districts for your manual review in the Master Sheet tab!

🛠 Prerequisites & Setup

  1. Google Chrome installed.
  2. Tesseract OCR (Windows):
    • Download from UB-Mannheim (e.g., tesseract-ocr-w64-setup-5.5.0.20241111.exe).
    • Run the installer and add the path (usually C:\Program Files\Tesseract-OCR) to your System Environment Variables under Path.
  3. Python 3.10+.
  4. Installation:
    python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt
playwright install chromium
  5. Environment Setup (Required for HuggingFace Dataset Sync):
    • Rename .env.example to .env in the root folder.
    • Add your HuggingFace token and repository path:
      HF_TOKEN=hf_your_actual_token_here
HF_REPO=Apf-AI4Good/kys-school-data

🚀 The Web Application (Gradio)

The absolute easiest way to use the pipeline is via the interactive browser UI. Chromium launches minimized in the background so it never interrupts your workflow!

python app.py

The app opens automatically at http://127.0.0.1:7861 and features 4 main tabs.

🔍 Tab 1: Scraper

The core scraping workflow, all on one screen:

  1. ▶ Start Scraping
    • Navigates the UDISE+ portal, finds all districts for the selected state, and collects data for the target categories.
  2. ↻ Fix Missing Data (Retries)
    • Scans for CAPTCHA failures and re-scrapes only the failed records. No successful data is ever lost.
  3. ✅ Automatic Push
    • Once a state finishes scraping without missing data, it is automatically pushed to the cloud in your HuggingFace dataset.

Repeat Steps 1–3 for every state you want to scrape. Then proceed to the other tabs.

📋 Tab 2: Master Sheet

Combines all raw state data from HuggingFace into a single mapped master Excel file, and manages flagged districts.

  • State Coverage Table: Connects to your HuggingFace dataset and displays which states are ready to build. You can delete raw data directly from here (which strictly auto-cleans any pending districts for that state!).
  • Review New Districts: Automatically detects newly scraped districts not in your Scholarship Application database. You can instantly map them to older Scholarship Application districts or rename them before building.
  • Build Master Sheet: Automatically:
    1. Pulls all raw complete state parquets.
    2. Applies geo-decoding for KVS/NVS/NAVY schools.
    3. Applies district back-mapping based on your rules.
    4. Tags schools with their Status (e.g., present or new districts found).
    5. Saves the final mapped file to scraped_data/mapped/ on HuggingFace and provides an Excel download.

🗺️ Tab 3: Mapping Manager

A full database management dashboard to maintain your Scholarship Application mappings.

  • Scholarship Application District Reference: View and manage your master district list.
    • Click-to-Edit Rows: Click any row in the table to instantly populate a quick-edit dropdown for updating its Status.
    • Bulk Excel Import/Export: Expand the accordion to download the table, make bulk edits in Excel, and drag-and-drop it back to seamlessly sync with the cloud.
    • Smart CRUD Tools: Add, Rename, and Delete districts. If you rename a district here, it perfectly cascades and updates your manual mapping rules automatically!

📥 Tab 4: Download History

  • Easily browse, refresh, and download any previously built Master Sheets directly from your HuggingFace cloud storage without having to rebuild them!

🛠️ Admin Scripts

If you need to make global, architectural changes to the baseline data outside of the UI, use the provided admin scripts:

admin_rename_sf_district.py If an old Scholarship Application district name is permanently outdated and you want to rename it everywhere (becoming the new Baseline truth):

python admin_rename_sf_district.py --state "SIKKIM" --old "EAST DISTRICT (GANGTOK)" --new "GANGTOK"

This script updates Dataset 1, rewrites the Baseline Master, and instantly re-runs the automated math engine for that state to guarantee your mapping rules stay mathematically synchronized!

💻 Running via Terminal (Headless Mode) 

# Step 1: Main scrape
python -m pytest tests/test_scrape_districts_by_category.py -v -s --state "GOA"

# Step 2: Retry failures
python -m pytest tests/test_retry_districts_by_category.py -v -s --state "GOA"

# Step 3: Export raw Excel
python export_to_excel.py --state "GOA"

(Filter categories: $env:KYS_TARGET_CATEGORIES="3,5" ; python ...)

🌐 Hosting on Hugging Face Spaces (Docker)

  1. Create a Docker Space on Hugging Face (Blank Template).
  2. Upload all .py files, requirements.txt, Dockerfile, README.md, and the tests/ and pages/ folders. Do NOT upload .env, .git, or venv/.
  3. In your Space Settings > Variables and secrets, add HF_TOKEN and HF_REPO.
  4. The Dockerfile automatically installs Chromium and Tesseract OCR, and launches your Gradio app publicly!

📂 Project Structure & Output Files

File Purpose
output/session_cookies.json Browser session cookies (auto-refreshed)
output/goa_district_id_map.json District dropdown mapping IDs for the portal
output/goa_schools_by_category.json Raw scraped JSON arrays & API responses
output_excel/goa_Schools.xlsx Raw Excel Export (before mapping)
HF Dataset: district_reference/ Dataset 1: The Scholarship Application reference mapping
HF Dataset: mapping_rules/ Dataset 2: District and Block mapping rules
HF Dataset: scraped_data/raw/{state}.parquet Per-state raw scraped data
HF Dataset: scraped_data/mapped/ Final combined mapped master output