USER_MANUAL.md

🏫 School Data Fetcher - Complete User Manual

Welcome to the School Data Fetcher. This application is designed to scrape, clean, geo-decode, and map school data from UDISE+ and prepare it for seamless integration into your Scholarship Application database.

This manual provides a detailed, step-by-step guide to using every feature of the app.

---

📑 Table of Contents

1. Overview of the Workflow
2. Tab 1: 🔍 Scraper
3. Tab 2: 📋 Master Sheet (Builder)
4. Tab 3: 🗺️ Mapping Manager
5. Tab 4: 📥 Download History
6. Best Practices & Tips

---

1. Overview of the Workflow

The standard operational workflow for using this app is:

1. Scrape Data: Select a state and pull the latest data from the government portal.
2. Handle Missing Data: Retry any failed records to ensure 100% data capture.
3. Check for New Districts: Go to the Master Sheet tab to see if the state created any new districts this year.
4. Update Reference: If new districts exist, the scraper will have already automatically added them to the Mapping Manager (Tab 3) as new districts found. You just need to add them to your actual Scholarship Application system and update their status to present in Tab 3 or rename to any present districts in scholarship application.
5. Build: Build the final Master Excel file, which is perfectly mapped according to your Scholarship Application.

---

2. Tab 1: 🔍 Scraper

This is where you fetch the raw data directly from UDISE+.

📍 Location & 📚 School Types

1. Select a State: Choose the state (or Pan-India organization like KVS/NVS) from the dropdown.
2. Select Categories: By default, all 7 categories are selected. CRITICAL: Only states scraped with ALL 7 categories selected are eligible for building the final Master Sheet in Tab 2. Do not uncheck any categories if your goal is to build a master sheet!

▶ Start Scraping

Click Start Scraping to begin the process.

This process will take some time! The app is scraping a massive amount of data and navigating complex government portals, so please be patient while it runs.

• The app will securely navigate the government portal and bypass captchas.
• You will see live progress updates (e.g., [14/50] Processing...).
• Once finished, the app will instantly Auto-Push the raw data to your secure cloud storage on HuggingFace.

↻ Fix Missing Data

Sometimes, the government website times out or fails a captcha repeatedly.

• If the final status says "⚠️ Some Records Are Incomplete", the Fix Missing Data button will appear.
• Click it. The app will only retry the specific schools that failed, saving you from having to restart the entire scrape.
• Keep clicking it until the status turns green (✅ All Data Retrieved!).

⚙️ Advanced Settings

• Retry attempts: You can increase the number of times the scraper tries to solve a captcha per school (default is 5). Higher numbers are more thorough but slower.

---

3. Tab 2: 📋 Master Sheet (Builder)

Once you have successfully scraped a state, you must process the raw data into a clean "Master Sheet".

The Review Process

When you open this tab, the app will automatically compare the raw data against your known Scholarship Application District Reference.

You will see one of two things:

Scenario A: "✅ No flagged districts — all clear!"

This means the scraped state hasn't added any new districts, and all data perfectly matches what your Scholarship Application already knows.

• You can immediately click Build Master Sheet.

Scenario B: The Flagged Districts Table Appears

This means the scraped state contains districts that your Scholarship Application doesn't know about yet. A red table will appear showing you the anomalous districts.

How to handle new districts: The table will always show brand new districts that have been added in the latest scraped data. You simply have two choices:

1. Keep them: Leave them as-is to include the new district names in your master sheet.
2. Rename them: If you need them to match an existing district in your Scholarship Application, you can simply rename them directly in the table by typing the correct name in the last column provided.

🏗️ Build Final Master Sheet

Once all anomalies are resolved, click Build Master Sheet.

Mapping Organization Schools (KVS, NVS, Navy): Because these organizations span across India, the system must "geo-decode" their UDISE codes to figure out which State and District they belong to. To ensure they decode perfectly, you must have scraped the main regular states first! As long as the main host states are already scraped when you click Build Master Sheet, the system will automatically build a dictionary and map the organization schools flawlessly.

• The app will geo-decode the data, apply all mapping rules, and filter out strictly out-of-scope regions (like Navy schools located in non-operational states).
• It will also automatically sync all multi-name schools into the cloud school_aliases.csv file to keep the Resolver App updated!
• A final, polished Excel file will be generated for you to download.

---

4. Tab 3: 🗺️ Mapping Manager

This tab manages your Scholarship Application Reference (Dataset 1). This is the master list of all States and Districts that currently exist in your database.

It tracks whether a district is present (fully set up in your scholarship application) or new districts found (detected by the scraper, but not yet configured in your scholarship application).

You have three powerful ways to manage this data:

Method 1: Click-to-Edit Rows (Fastest for quick fixes)

1. The main reference table is intentionally locked to prevent typos.
2. Click any cell in the table, and the app will instantly copy that row's State and District into the Edit Status form directly below the table.
3. Use the Status Dropdown to select either present or new districts found.
4. Click the blue 💾 Update Row button to permanently save your changes to the cloud.

Method 2: Bulk Excel Import / Export (Best for team collaboration)

1. Click the 📦 Bulk Excel Import / Export accordion to expand it.
2. Click 📦 Prepare Current View for Download and download the Excel file.
3. You can send this file to your team or make massive bulk edits in Microsoft Excel of only the status present or new districts found.
4. Once updated, drag and drop the Excel file into the Upload Updated Excel box. The app will instantly read the file and sync all changes to the cloud.

Method 3: Smart Dropdown Tools (Best for structural changes)

At the bottom of the tab, you'll find tools to Add, Rename, or Delete districts.

• Add a District: Use this if you are manually creating a district in your Scholarship Application before scraping.
• Rename a District: Use this if your Scholarship Application changes a district name.
• Note: Some fields (like Add a District) are standard textboxes, while others (like Rename and Delete) use smart dropdowns so you can easily select existing names.

---

5. Tab 4: 📥 Download History

This tab serves as your cloud backup and archive.

• Every time you click Build Master Sheet in Tab 2, a permanent snapshot of that final Excel file is saved to the HuggingFace cloud.
• If you ever lose a file or need to access a master sheet from a previous month, simply go to this tab.
• Click 🔄 Refresh File List, select the file by its timestamp, and download it instantly.

---

6. Best Practices & Tips

• Do Not Interrupt the Scraper: If you close the browser tab while Tab 1 is actively scraping, the process will stop. Let it run in the background.
• Always Resolve Anomalies: Never ignore the "New districts detected" table in Tab 2. If you force a build without resolving these, those schools will carry the newly scraped district names into your final database exactly as they are. You must ensure those new names match your Scholarship Application or rename them in the table before building.
• Smart Raw Deletion Cleanup: If you decide you don't want a scraped state anymore and click "🗑️ Delete" in Tab 2, the app will auto-clean your database. It will delete all new districts found for that state, but perfectly preserve any present districts to ensure your established database is never corrupted.
• Clear Filters Before Exporting: In Tab 3, the Excel Export downloads exactly what is visible on your screen. If you only want to download "Delhi", apply the filter first before clicking Prepare for Download!

---

End of Manual