Updated mapping manager

README.md

@@ -19,11 +19,11 @@ An automated, end-to-end data pipeline to harvest school data from the KYS UDISE
 2. **Intelligent Retries:** Automatically detects CAPTCHA failures and retries only missing data. The terminal UI wipes clean automatically before each operation.
 3. **UDISE Geo-Decoding:** For non-actual state scrapes (e.g., KVS, NVS, NAVY, IAF), it parses the UDISE code to determine the *real* State, District, and Block.
 4. **Automated UDISE-Tracking Math Engine:** When building the master sheet, an intelligent math engine automatically resolves renamed districts and blocks! 
-   - It cross-references newly scraped UDISE codes against a cloud-stored Salesforce Baseline (`baseline_master.parquet`).
+   - It cross-references newly scraped UDISE codes against a cloud-stored Scholarship Application Baseline (`baseline_master.parquet`).
    - If a district was simply renamed (e.g., EAST DISTRICT -> GANGTOK), the math engine statistically tracks the UDISE codes, proves the rename, and automatically generates a mapping rule in the cloud (`manual_district_mapping.parquet`).
    - For complex fractured districts, it drills down to the Block level (`manual_block_mapping.parquet`) to perfectly map schools to their old districts.
 5. **HuggingFace Cloud Architecture:** Syncs directly to a HuggingFace dataset organized into 3 folders:
-   - `district_reference`: Source of truth for Salesforce district tracking (Dataset 1).
+   - `district_reference`: Source of truth for Scholarship Application district tracking (Dataset 1).
    - `mapping_rules`: Automated + manual district & block mapping rules (Dataset 2) & the SF Baseline Master.
    - `scraped_data`: Holds both `raw` per-state files and the final `mapped` master sheet.
 6. **Smart District Flagging (Hybrid System):** Because the math engine automatically handles all simple renames behind the scenes, the UI only flags *genuinely brand-new* or highly ambiguous districts for your manual review in the Master Sheet tab!
@@ -84,22 +84,25 @@ The core scraping workflow, all on one screen:
 
 Combines all raw state data from HuggingFace into a single mapped master Excel file, and manages flagged districts.
 
-- **State Coverage Table:** Connects to your HuggingFace dataset and displays which states are ready to build. You can delete raw data directly from here.
-- **Review New Districts:** Automatically detects newly scraped districts not in your Salesforce database. You can instantly map them to older Salesforce districts or rename them before building.
+- **State Coverage Table:** Connects to your HuggingFace dataset and displays which states are ready to build. You can delete raw data directly from here (which strictly auto-cleans any pending districts for that state!).
+- **Review New Districts:** Automatically detects newly scraped districts not in your Scholarship Application database. You can instantly map them to older Scholarship Application districts or rename them before building.
 - **Build Master Sheet:** Automatically:
   1. Pulls all raw complete state parquets.
   2. Applies geo-decoding for KVS/NVS/NAVY schools.
   3. Applies district back-mapping based on your rules.
-  4. Tags schools with their `SF_Status` (e.g., `in_sf` or `pending_sf_addition`).
+  4. Tags schools with their `Status` (e.g., `present` or `new districts found`).
   5. Saves the final mapped file to `scraped_data/mapped/` on HuggingFace and provides an Excel download.
 
 ---
 
 ### 🗺️ Tab 3: Mapping Manager
 
-A full database management dashboard to maintain your Salesforce mappings.
+A full database management dashboard to maintain your Scholarship Application mappings.
 
-- **SF District Reference:** Full CRUD table to view, filter, add, rename, and delete states and districts. If you rename a district here, it perfectly cascades and updates your manual mapping rules automatically!
+- **Scholarship Application District Reference:** View and manage your master district list.
+  - **Click-to-Edit Rows:** Click any row in the table to instantly populate a quick-edit dropdown for updating its Status.
+  - **Bulk Excel Import/Export:** Expand the accordion to download the table, make bulk edits in Excel, and drag-and-drop it back to seamlessly sync with the cloud.
+  - **Smart CRUD Tools:** Add, Rename, and Delete districts. If you rename a district here, it perfectly cascades and updates your manual mapping rules automatically!
 
 ---
 
@@ -114,7 +117,7 @@ A full database management dashboard to maintain your Salesforce mappings.
 If you need to make global, architectural changes to the baseline data outside of the UI, use the provided admin scripts:
 
 **`admin_rename_sf_district.py`**
-If an old Salesforce district name is permanently outdated and you want to rename it everywhere (becoming the new Baseline truth):
+If an old Scholarship Application district name is permanently outdated and you want to rename it everywhere (becoming the new Baseline truth):
 ```powershell
 python admin_rename_sf_district.py --state "SIKKIM" --old "EAST DISTRICT (GANGTOK)" --new "GANGTOK"
 ```
@@ -155,7 +158,7 @@ python export_to_excel.py --state "GOA"
 | `output/goa_district_id_map.json` | District dropdown mapping IDs for the portal |
 | `output/goa_schools_by_category.json` | Raw scraped JSON arrays & API responses |
 | `output_excel/goa_Schools.xlsx` | Raw Excel Export (before mapping) |
-| `HF Dataset: district_reference/` | Dataset 1: The SF reference mapping |
+| `HF Dataset: district_reference/` | Dataset 1: The Scholarship Application reference mapping |
 | `HF Dataset: mapping_rules/` | Dataset 2: District and Block mapping rules |
 | `HF Dataset: scraped_data/raw/{state}.parquet` | Per-state raw scraped data |
 | `HF Dataset: scraped_data/mapped/` | Final combined mapped master output |

USER_MANUAL.md

@@ -0,0 +1,136 @@
+# 🏫 School Data Fetcher - Complete User Manual
+
+Welcome to the **School Data Fetcher**. This application is designed to scrape, clean, geo-decode, and map school data from UDISE+ and prepare it for seamless integration into your Scholarship Application database. 
+
+This manual provides a detailed, step-by-step guide to using every feature of the app.
+
+---
+
+## 📑 Table of Contents
+1. [Overview of the Workflow](#1-overview-of-the-workflow)
+2. [Tab 1: 🔍 Scraper](#2-tab-1--scraper)
+3. [Tab 2: 📋 Master Sheet (Builder)](#3-tab-2--master-sheet-builder)
+4. [Tab 3: 🗺️ Mapping Manager](#4-tab-3--mapping-manager)
+5. [Tab 4: 📥 Download History](#5-tab-4--download-history)
+6. [Best Practices & Tips](#6-best-practices--tips)
+
+---
+
+## 1. Overview of the Workflow
+
+The standard operational workflow for using this app is:
+1. **Scrape Data:** Select a state and pull the latest data from the government portal.
+2. **Handle Missing Data:** Retry any failed records to ensure 100% data capture.
+3. **Check for New Districts:** Go to the Master Sheet tab to see if the state created any new districts this year.
+4. **Update Reference:** If new districts exist, the scraper will have already automatically added them to the Mapping Manager (Tab 3) as `new districts found`. You just need to add them to your actual Scholarship Application system and update their status to `present` in Tab 3.
+5. **Map & Build:** Create rules to map old/split districts to new ones, then build the final Excel file.
+
+---
+
+## 2. Tab 1: 🔍 Scraper
+
+This is where you fetch the raw data directly from UDISE+.
+
+### 📍 Location & 📚 School Types
+1. **Select a State:** Choose the state (or Pan-India organization like KVS/NVS) from the dropdown. 
+2. **Select Categories:** By default, all 7 categories are selected. It is highly recommended to leave all of them selected unless you only need a very specific subset of data.
+
+### ▶ Start Scraping
+Click **Start Scraping** to begin the process. 
+- The app will securely navigate the government portal and bypass captchas.
+- You will see live progress updates (e.g., `[14/50] Processing...`).
+- Once finished, the app will instantly **Auto-Push** the raw data to your secure cloud storage on HuggingFace.
+
+### ↻ Fix Missing Data
+Sometimes, the government website times out or fails a captcha repeatedly.
+- If the final status says **"⚠️ Some Records Are Incomplete"**, the **Fix Missing Data** button will appear.
+- Click it. The app will *only* retry the specific schools that failed, saving you from having to restart the entire scrape.
+- Keep clicking it until the status turns green (**✅ All Data Retrieved!**).
+
+### ⚙️ Advanced Settings
+- **Retry attempts:** You can increase the number of times the scraper tries to solve a captcha per school (default is 5). Higher numbers are more thorough but slower.
+
+---
+
+## 3. Tab 2: 📋 Master Sheet (Builder)
+
+Once you have successfully scraped a state, you must process the raw data into a clean "Master Sheet". 
+
+### The Review Process
+When you open this tab and select your newly scraped state, the app will automatically compare the raw data against your known **Scholarship Application District Reference**. 
+
+You will see one of two things:
+
+#### Scenario A: "✅ No flagged districts — all clear!"
+This means the scraped state hasn't added any new districts, and all data perfectly matches what your Scholarship Application already knows. 
+- You can immediately click **Build Master Sheet**.
+
+#### Scenario B: The Flagged Districts Table Appears
+This means the scraped state contains districts that your Scholarship Application doesn't know about yet. A red table will appear showing you the anomalous districts.
+
+**How to handle new districts:**
+1. **Valid New District:** If the district is genuinely new (e.g., the state split a district in two), you must first add it to your Scholarship Application system. Then, go to **Tab 3: Mapping Manager**, mark it as `present`, and come back here.
+2. **Name Mismatch:** If the district is just spelled differently (e.g., "PONDICHERRY" vs "PUDUCHERRY"), use the **Create Mapping Rule** section below the table to permanently teach the app how to map the new spelling to your expected spelling.
+
+### Creating Mapping Rules
+If you need to fix a name mismatch or map a split block to an old district:
+1. Select the **Raw District Name** (how it appears in the scrape).
+2. Select the **Target District Name** (how it should appear in your Scholarship Application).
+3. (Optional) Select a specific Block if the rule should only apply to that block.
+4. Click **Add Rule**. The app will remember this rule forever!
+
+### 🏗️ Build Final Master Sheet
+Once all anomalies are resolved, click **Build Master Sheet**. 
+- The app will geo-decode the data, apply all mapping rules, and filter out strictly out-of-scope regions (like Navy schools located in non-operational states).
+- A final, polished Excel file will be generated for you to download.
+
+---
+
+## 4. Tab 3: 🗺️ Mapping Manager
+
+This tab manages your **Scholarship Application Reference** (Dataset 1). This is the master list of all States and Districts that currently exist in your database. 
+
+It tracks whether a district is `present` (fully set up in your system) or `new districts found` (detected by the scraper, but not yet configured by your IT team).
+
+You have three powerful ways to manage this data:
+
+### Method 1: Click-to-Edit Rows (Fastest for quick fixes)
+1. The main reference table is intentionally locked to prevent typos.
+2. Click **any cell** in the table, and the app will instantly copy that row's State and District into the **Edit Status** form directly below the table.
+3. Use the **Status Dropdown** to select either `present` or `new districts found`.
+4. Click the blue **💾 Update Row** button to permanently save your changes to the cloud.
+
+### Method 2: Bulk Excel Import / Export (Best for team collaboration)
+1. Click the **📦 Bulk Excel Import / Export** accordion to expand it.
+2. Click **📦 Prepare Current View for Download** and download the Excel file.
+3. You can send this file to your team or make massive bulk edits in Microsoft Excel.
+4. Once updated, drag and drop the Excel file into the **Upload Updated Excel** box. The app will instantly read the file and sync all changes to the cloud.
+
+### Method 3: Smart Dropdown Tools (Best for structural changes)
+At the bottom of the tab, you'll find tools to Add, Rename, or Delete districts.
+- **Add a District:** Use this if you are manually creating a district in your Scholarship Application before scraping.
+- **Rename a District:** Use this if your Scholarship Application changes a district name. **Crucially, using this tool will automatically cascade the name change to all of your existing mapping rules!**
+- *Note: Some fields (like Add a District) are standard textboxes, while others (like Rename and Delete) use smart dropdowns so you can easily select existing names.*
+
+---
+
+## 5. Tab 4: 📥 Download History
+
+This tab serves as your cloud backup and archive.
+
+- Every time you click **Build Master Sheet** in Tab 2, a permanent snapshot of that final Excel file is saved to the HuggingFace cloud.
+- If you ever lose a file or need to access a master sheet from a previous month, simply go to this tab.
+- Click **🔄 Refresh File List**, select the file by its timestamp, and download it instantly.
+
+---
+
+## 6. Best Practices & Tips
+
+- **Do Not Interrupt the Scraper:** If you close the browser tab while Tab 1 is actively scraping, the process will stop. Let it run in the background.
+- **Always Resolve Anomalies:** Never ignore red tables in Tab 2. If you force a build without mapping new districts, those schools will carry the anomalous names into your final database, which will cause data ingestion errors.
+- **Use the Rename Tool for Name Changes:** If you need to rename a district that has mapping rules attached to it, always use the **Rename a District** tool in Tab 3 rather than editing the table directly. The tool ensures your mapping rules don't break.
+- **Smart Raw Deletion Cleanup:** If you decide you don't want a scraped state anymore and click "🗑️ Delete" in Tab 2, the app will auto-clean your database. It will delete all `new districts found` for that state, but perfectly preserve any `present` districts to ensure your established database is never corrupted.
+- **Clear Filters Before Exporting:** In Tab 3, the Excel Export downloads exactly what is visible on your screen. If you only want to download "Delhi", apply the filter first before clicking Prepare for Download!
+
+---
+*End of Manual*

admin_rename_sf_district.py

@@ -63,14 +63,24 @@ def rename_sf_district(state: str, old_name: str, new_name: str):
         # Pull existing rules
         current_dist, current_block = pull_mapping_rules(token, repo)
         
-        # Drop all old automated rules for this state to ensure a clean slate
+        # Safely cascade the rename to existing mapping rules instead of dropping them
         if not current_dist.empty:
-            current_dist = current_dist[current_dist["State"] != state]
+            mask = (current_dist["State"] == state) & (current_dist["Dist_NEW"] == old_name)
+            if mask.sum() > 0:
+                current_dist.loc[mask, "Dist_NEW"] = new_name
+                
         if not current_block.empty:
-            current_block = current_block[current_block["State"] != state]
+            mask = (current_block["State"] == state) & (current_block["Dist_NEW"] == old_name)
+            if mask.sum() > 0:
+                current_block.loc[mask, "Dist_NEW"] = new_name
             
         merged_dist = pd.concat([current_dist, auto_dist], ignore_index=True) if not auto_dist.empty else current_dist
+        if not merged_dist.empty:
+            merged_dist = merged_dist.drop_duplicates(subset=["State", "Dist_OLD", "Dist_NEW"])
+            
         merged_block = pd.concat([current_block, auto_block], ignore_index=True) if not auto_block.empty else current_block
+        if not merged_block.empty:
+            merged_block = merged_block.drop_duplicates(subset=["State", "Dist_OLD", "Block_OLD", "Dist_NEW"])
 
         push_mapping_rules(merged_dist, merged_block, token, repo)
         print("Mapping rules successfully updated!")

app.py

@@ -251,9 +251,13 @@ def _stream(pytest_args: list, state: str, max_retries: int, mode: str, target_c
     re_done_s = re.compile(r"DONE\.\s+Total:\s*(\d+)\s*\|.*?Success:\s*(\d+).*?No Data:\s*(\d+).*?Captcha Failed:\s*(\d+)")
     re_done_r = re.compile(r"RETRY DONE\.\s+Attempted:\s*(\d+)\s*\|.*?Resolved:\s*(\d+).*?Still captcha-failed:\s*(\d+)")
 
+    excel_cache = [None]
     def _auto_export_excel():
         """Auto-generate Excel from JSON after scrape completes."""
         excel = get_excel_file(state)
+        if excel_cache[0] == excel and os.path.exists(excel):
+            return gr.update(value=excel, visible=True)
+            
         try:
             proc = subprocess.run(
                 [sys.executable, "export_to_excel.py", "--state", state],
@@ -261,6 +265,7 @@ def _stream(pytest_args: list, state: str, max_retries: int, mode: str, target_c
                 cwd=os.path.dirname(os.path.abspath(__file__))
             )
             if proc.returncode == 0 and os.path.exists(excel):
+                excel_cache[0] = excel
                 return gr.update(value=excel, visible=True)
         except Exception:
             pass
@@ -317,7 +322,7 @@ def _stream(pytest_args: list, state: str, max_retries: int, mode: str, target_c
             if success_count == 0 and failed_count == 0:
                 phase_text = "○ No schools found in selected categories."; banner_rendered = _banner_html("○", "No Schools Found", "The selected categories have no schools.", "nodata")
             elif not has_failures:
-                phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All school data fetched successfully.", "success")
+                phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All school data fetched successfully.<br><br><span style='color:#3b82f6;font-weight:600;'>⏳ Pushing data to cloud... please wait.</span>", "success")
             else:
                 phase_text = f"⚠️ {failed_count} record(s) incomplete."; banner_rendered = _banner_html("⚠️", "Some Records Are Incomplete", f"{failed_count} record(s) could not be fetched. Please proceed to Step 2 to fetch missing data.", "warning")
         mr = re_done_r.search(line)
@@ -326,7 +331,7 @@ def _stream(pytest_args: list, state: str, max_retries: int, mode: str, target_c
             success_count = resolved; failed_count = still_failed; no_data_count = 0; current = total = attempted; is_done = True; has_failures = still_failed > 0
             stats_rendered = _stats_html(resolved, 0, still_failed, attempted)
             if not has_failures:
-                phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All missing data fetched. Please proceed to Step 3 to export.", "success")
+                phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All missing data fetched.<br><br><span style='color:#3b82f6;font-weight:600;'>⏳ Pushing data to cloud... please wait.</span>", "success")
             else:
                 phase_text = f"⚠️ {still_failed} record(s) still incomplete."; banner_rendered = _banner_html("⚠️", "Some Records Still Incomplete", f"{still_failed} record(s) failed. You can run Step 2 again or proceed to Step 3.", "warning")
         yield _emit()
@@ -338,7 +343,7 @@ def _stream(pytest_args: list, state: str, max_retries: int, mode: str, target_c
         fc = check_missed_schools(out_file)
         if fc is not None:
             if fc == 0 and success_count > 0:
-                is_done, has_failures = True, False; phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All school data fetched successfully.", "success")
+                is_done, has_failures = True, False; phase_text = "✅ All data retrieved!"; banner_rendered = _banner_html("✅", "All Data Retrieved!", "All school data fetched successfully.<br><br><span style='color:#3b82f6;font-weight:600;'>⏳ Pushing data to cloud... please wait.</span>", "success")
             elif fc > 0:
                 is_done, has_failures = True, True; failed_count = fc; phase_text = f"⚠️ {fc} record(s) incomplete."; banner_rendered = _banner_html("⚠️", "Some Records Are Incomplete", f"{fc} record(s) failed. Please proceed to Step 2 to fetch missing data.", "warning")
         yield _emit()
@@ -1005,7 +1010,7 @@ with gr.Blocks(title="School Data Fetcher", css=css, theme=custom_theme) as app:
                 flagged_table = gr.Dataframe(
                     label="New districts detected (Type the Old SF name in the last column to rename it)",
                     headers=["State", "District", "Reason", "Rename District (Optional)"],
-                    interactive=True, wrap=True
+                    interactive=False, wrap=True
                 )
                 flagged_status_html = gr.HTML(value="")
 
@@ -1219,65 +1224,108 @@ with gr.Blocks(title="School Data Fetcher", css=css, theme=custom_theme) as app:
                         border:1px solid rgba(99,102,241,0.2); border-radius:14px; padding:18px 22px; margin-bottom:18px;">
                 <div style="font-weight:700; font-size:1.05em; margin-bottom:6px;">🗺️ Mapping Manager</div>
                 <div style="font-size:.9em; color:var(--body-text-color-subdued); line-height:1.7;">
-                    <b>SF Reference</b>: The mirror of what Salesforce currently knows. When you add a new district to Salesforce, find it here and change its status from <i>pending_sf_addition</i> to <i>in_sf</i>.
+                    <b>Scholarship Application Reference</b>: The mirror of what Scholarship Application currently knows. When you add a new district to Scholarship Application, find it here and change its status from <i>new districts found</i> to <i>present</i>.
                 </div>
             </div>
             """)
 
-
-            # ── SF District Reference CRUD ─────────────────────────
-            gr.HTML("<div class='section-label'>📁 SF District Reference</div>")
+            gr.HTML("<div class='section-label'>📁 Scholarship Application District Reference</div>")
             with gr.Row():
                 refresh_ref_btn = gr.Button("🔄 Load from Dataset", variant="secondary", scale=1)
                 
             # Filters
             with gr.Row():
                 ref_filter_state = gr.Dropdown(label="Filter by State", choices=["All"], value="All", scale=1, interactive=True)
-                ref_filter_status = gr.Dropdown(label="Filter by Status", choices=["All", "in_sf", "pending_sf_addition"], value="All", scale=1, interactive=True)
+                ref_filter_status = gr.Dropdown(label="Filter by Status", choices=["All", "present", "new districts found"], value="All", scale=1, interactive=True)
                 ref_filter_dist  = gr.Textbox(label="Search District", placeholder="Type to search...", scale=2, interactive=True)
             
             # The DataFrame
             ref_full_state = gr.State(pd.DataFrame())
             ref_table = gr.Dataframe(
-                label="SF District Reference (State | District | SF_Status)",
+                label="Scholarship Application District Reference (Editable)",
                 interactive=False, wrap=True
             )
+            
+            with gr.Accordion("📦 Bulk Excel Import / Export", open=False):
+                with gr.Row():
+                    ref_prep_dl_btn = gr.Button("📦 Prepare Current View for Download")
+                    ref_dl_btn = gr.DownloadButton("📥 Download Excel", visible=False)
+                
+                with gr.Row():
+                    ref_upload = gr.File(label="Upload Updated Excel", file_types=[".xlsx"])
+            
+            gr.HTML("<div style='font-size:.8em;font-weight:600;margin:10px 0 4px 0; padding-top:10px; border-top:1px solid #ddd;'>Click any row in the table above to edit its status here:</div>")
+            with gr.Row():
+                ref_edit_state  = gr.Textbox(label="State", interactive=False, scale=2)
+                ref_edit_dist   = gr.Textbox(label="District", interactive=False, scale=2)
+                ref_edit_status = gr.Dropdown(choices=["present", "new districts found"], label="Status", scale=1)
+                ref_edit_btn    = gr.Button("💾 Update Row", variant="primary", scale=1)
+            
             ref_status_html = gr.HTML(value="")
 
             gr.HTML("<div style='font-size:.8em;font-weight:600;margin:10px 0 4px 0;'>Add a District</div>")
             with gr.Row():
                 ref_state_add  = gr.Textbox(label="State", placeholder="e.g. ANDHRA PRADESH", scale=2)
                 ref_dist_add   = gr.Textbox(label="District", placeholder="e.g. ALLURI SITHARAMA RAJU", scale=2)
-                ref_status_add = gr.Dropdown(choices=["in_sf", "pending_sf_addition"], value="in_sf", label="SF Status", scale=1)
+                ref_status_add = gr.Dropdown(choices=["present", "new districts found"], value="present", label="Status", scale=1)
                 ref_add_btn    = gr.Button("➕ Add District", variant="primary", scale=1)
             ref_add_status = gr.HTML(value="")
 
             gr.HTML("<div style='font-size:.8em;font-weight:600;margin:10px 0 4px 0;'>Rename a District (automatically updates mappings)</div>")
             with gr.Row():
-                ref_ren_state  = gr.Textbox(label="State", placeholder="e.g. ANDHRA PRADESH", scale=2)
-                ref_ren_old    = gr.Textbox(label="Old Name", placeholder="e.g. VISAKHAPATNAM", scale=2)
+                ref_ren_state  = gr.Dropdown(label="State", choices=[], allow_custom_value=True, scale=2)
+                ref_ren_old    = gr.Dropdown(label="Old Name", choices=[], allow_custom_value=True, scale=2)
                 ref_ren_new    = gr.Textbox(label="New Name", placeholder="e.g. VISAKHAPATNAM (NEW)", scale=2)
                 ref_rename_btn = gr.Button("✏️ Rename", variant="secondary", scale=1)
             ref_rename_status = gr.HTML(value="")
 
             gr.HTML("<div style='font-size:.8em;font-weight:600;margin:10px 0 4px 0;'>Delete Actions</div>")
             with gr.Row():
-                ref_del_state  = gr.Textbox(label="State", placeholder="e.g. ANDHRA PRADESH", scale=2)
-                ref_del_dist   = gr.Textbox(label="District", placeholder="e.g. OLD_DISTRICT", scale=2)
+                ref_del_state  = gr.Dropdown(label="State", choices=[], allow_custom_value=True, scale=2)
+                ref_del_dist   = gr.Dropdown(label="District", choices=[], allow_custom_value=True, scale=2)
                 ref_del_dist_btn = gr.Button("🗑️ Delete District", variant="stop", scale=1)
                 ref_del_state_btn = gr.Button("🚨 Delete Entire State", variant="stop", scale=1)
             ref_del_status = gr.HTML(value="")
 
+            def on_ref_table_select(df, evt: gr.SelectData):
+                row = evt.index[0]
+                if df is None or df.empty or row >= len(df):
+                    return gr.update(), gr.update(), gr.update(), ""
+                
+                state = df.iloc[row]["State"]
+                dist = df.iloc[row]["District"]
+                status = df.iloc[row]["Status"]
+                return gr.update(value=state), gr.update(value=dist), gr.update(value=status), ""
+
+            def ui_ref_edit_row(state, district, new_status):
+                from hf_store import get_hf_credentials, update_district_reference_add
+                token, repo = get_hf_credentials()
+                if not token or not repo or not state or not district:
+                    return "<p style='color:#ef4444;'>⚠️ Please select a row from the table first.</p>"
+                try:
+                    update_district_reference_add(state, district, new_status, token, repo)
+                    return f"<p style='color:#22c55e;font-weight:600;'>✅ Successfully updated {district} to '{new_status}'. Refresh the table to see changes.</p>"
+                except Exception as ex:
+                    return f"<p style='color:#ef4444;'>❌ Error: {ex}</p>"
+
             def ui_refresh_ref():
                 from hf_store import get_hf_credentials, pull_district_reference
                 token, repo = get_hf_credentials()
                 if not token or not repo:
-                    return pd.DataFrame(), pd.DataFrame(), gr.update(), "<p style='color:#ef4444;'>⚠️ HF credentials not set.</p>"
+                    empty = pd.DataFrame()
+                    u = gr.update()
+                    return empty, empty, u, u, u, u, u, "<p style='color:#ef4444;'>⚠️ HF credentials not set.</p>"
                 df = pull_district_reference(token, repo)
                 if df.empty:
-                    return df, df, gr.update(), "<p style='color:#f59e0b;'>⚠️ SF Reference not seeded yet. Use the Seed section above.</p>"
+                    empty = pd.DataFrame()
+                    u = gr.update()
+                    return empty, empty, u, u, u, u, u, "<p style='color:#f59e0b;'>⚠️ Scholarship Application Reference empty.</p>"
                 states = sorted(list(df["State"].unique()))
-                return df, df, gr.update(choices=["All"] + states, value="All"), f"<p style='color:#22c55e;font-weight:600;'>✅ Loaded {len(df)} rows.</p>"
+                dists = sorted(list(df["District"].unique()))
+                st_up = gr.update(choices=["All"] + states, value="All")
+                st_dd = gr.update(choices=states)
+                dist_dd = gr.update(choices=dists)
+                return df, df, st_up, st_dd, dist_dd, st_dd, dist_dd, f"<p style='color:#22c55e;font-weight:600;'>✅ Loaded {len(df)} rows.</p>"
 
             def filter_ref_table(df, filter_state, filter_status, search_dist):
                 if df is None or df.empty: return df
@@ -1285,11 +1333,87 @@ with gr.Blocks(title="School Data Fetcher", css=css, theme=custom_theme) as app:
                 if filter_state and filter_state != "All":
                     res = res[res["State"] == filter_state]
                 if filter_status and filter_status != "All":
-                    res = res[res["SF_Status"] == filter_status]
+                    res = res[res["Status"] == filter_status]
                 if search_dist:
                     res = res[res["District"].str.contains(search_dist.upper(), na=False)]
                 return res
 
+            def ui_save_table_edits(df, full_df, filter_state, filter_status, search_dist):
+                from hf_store import get_hf_credentials, push_district_reference
+                token, repo = get_hf_credentials()
+                if df is None or df.empty:
+                    return "<p style='color:#ef4444;'>⚠️ Table is empty.</p>", full_df
+                # The user edited 'df'. We need to merge it back into 'full_df'
+                # This is a bit complex if they deleted rows or changed keys.
+                # Actually, if they are filtering, df only contains a subset. 
+                # Let's just overwrite the subset in full_df based on index if we kept index, 
+                # but Gradio df doesn't keep original indices easily.
+                # Simplest way: They should only save edits if viewing All/All without search.
+                if filter_state != "All" or filter_status != "All" or search_dist:
+                    return "<p style='color:#ef4444;'>⚠️ Please clear all filters (set to 'All', clear search) before saving direct table edits.</p>", full_df
+                
+                try:
+                    df["State"] = df["State"].astype(str).str.strip().str.upper()
+                    df["District"] = df["District"].astype(str).str.strip().str.upper()
+                    df["Status"] = df["Status"].astype(str).str.strip()
+                    
+                    invalid_statuses = df[~df["Status"].isin(["present", "new districts found"])]
+                    if not invalid_statuses.empty:
+                        return "<p style='color:#ef4444;'>⚠️ Invalid Status found. You can only use 'present' or 'new districts found'.</p>", full_df
+                    
+                    df = df.drop_duplicates()
+                    push_district_reference(df, token, repo)
+                    return f"<p style='color:#22c55e;font-weight:600;'>✅ Saved {len(df)} rows directly to cloud.</p>", df
+                except Exception as ex:
+                    return f"<p style='color:#ef4444;'>❌ Error: {ex}</p>", full_df
+
+            def ui_prep_excel_download(df):
+                if df is None or df.empty:
+                    return gr.update(visible=False), "<p style='color:#ef4444;'>⚠️ No data to download.</p>"
+                import tempfile
+                import os
+                fd, temp_excel = tempfile.mkstemp(suffix=".xlsx")
+                os.close(fd)
+                df.to_excel(temp_excel, index=False)
+                return gr.update(value=temp_excel, visible=True), "<p style='color:#22c55e;'>✅ Ready to download!</p>"
+
+            def ui_upload_excel(file):
+                if not file: return pd.DataFrame(), "<p style='color:#ef4444;'>⚠️ No file uploaded.</p>"
+                import pandas as pd
+                from hf_store import get_hf_credentials, push_district_reference, pull_district_reference
+                token, repo = get_hf_credentials()
+                try:
+                    uploaded_df = pd.read_excel(file.name)
+                    if "State" not in uploaded_df.columns or "District" not in uploaded_df.columns or "Status" not in uploaded_df.columns:
+                        return pd.DataFrame(), "<p style='color:#ef4444;'>⚠️ Excel must have State, District, and Status columns.</p>"
+                    
+                    uploaded_df["State"] = uploaded_df["State"].astype(str).str.strip().str.upper()
+                    uploaded_df["District"] = uploaded_df["District"].astype(str).str.strip().str.upper()
+                    uploaded_df["Status"] = uploaded_df["Status"].astype(str).str.strip()
+                    
+                    invalid_statuses = uploaded_df[~uploaded_df["Status"].isin(["present", "new districts found"])]
+                    if not invalid_statuses.empty:
+                        return pd.DataFrame(), "<p style='color:#ef4444;'>⚠️ Invalid Status found in Excel. You can only use 'present' or 'new districts found'.</p>"
+                    
+                    uploaded_df = uploaded_df.drop_duplicates(subset=["State", "District"])
+                    
+                    full_df = pull_district_reference(token, repo)
+                    if not full_df.empty:
+                        full_df.set_index(["State", "District"], inplace=True)
+                        uploaded_df.set_index(["State", "District"], inplace=True)
+                        full_df.update(uploaded_df)
+                        new_rows = uploaded_df[~uploaded_df.index.isin(full_df.index)]
+                        full_df = pd.concat([full_df, new_rows])
+                        full_df.reset_index(inplace=True)
+                        df_to_push = full_df
+                    else:
+                        df_to_push = uploaded_df.copy()
+                        
+                    push_district_reference(df_to_push, token, repo)
+                    return df_to_push, f"<p style='color:#22c55e;font-weight:600;'>✅ Successfully merged {len(uploaded_df)} uploaded rows into the dataset!</p>"
+                except Exception as ex:
+                    return pd.DataFrame(), f"<p style='color:#ef4444;'>❌ Error: {ex}</p>"
+
             def ui_ref_add(state, district, sf_status):
                 from hf_store import get_hf_credentials, update_district_reference_add
                 token, repo = get_hf_credentials()
@@ -1334,11 +1458,23 @@ with gr.Blocks(title="School Data Fetcher", css=css, theme=custom_theme) as app:
                 except Exception as ex:
                     return f"<p style='color:#ef4444;'>❌ Error: {ex}</p>"
 
-            refresh_ref_btn.click(fn=ui_refresh_ref, outputs=[ref_table, ref_full_state, ref_filter_state, ref_status_html])
+            # Wirings
+            refresh_ref_btn.click(
+                fn=ui_refresh_ref, 
+                outputs=[ref_table, ref_full_state, ref_filter_state, 
+                         ref_ren_state, ref_ren_old,
+                         ref_del_state, ref_del_dist,
+                         ref_status_html]
+            )
             ref_filter_state.change(fn=filter_ref_table, inputs=[ref_full_state, ref_filter_state, ref_filter_status, ref_filter_dist], outputs=[ref_table])
             ref_filter_status.change(fn=filter_ref_table, inputs=[ref_full_state, ref_filter_state, ref_filter_status, ref_filter_dist], outputs=[ref_table])
             ref_filter_dist.change(fn=filter_ref_table, inputs=[ref_full_state, ref_filter_state, ref_filter_status, ref_filter_dist], outputs=[ref_table])
             
+            ref_table.select(fn=on_ref_table_select, inputs=[ref_table], outputs=[ref_edit_state, ref_edit_dist, ref_edit_status, ref_status_html])
+            ref_edit_btn.click(fn=ui_ref_edit_row, inputs=[ref_edit_state, ref_edit_dist, ref_edit_status], outputs=[ref_status_html])
+            ref_prep_dl_btn.click(fn=ui_prep_excel_download, inputs=[ref_table], outputs=[ref_dl_btn, ref_status_html])
+            ref_upload.upload(fn=ui_upload_excel, inputs=[ref_upload], outputs=[ref_table, ref_status_html])
+
             ref_add_btn.click(fn=ui_ref_add, inputs=[ref_state_add, ref_dist_add, ref_status_add], outputs=[ref_add_status])
             ref_rename_btn.click(fn=ui_ref_rename, inputs=[ref_ren_state, ref_ren_old, ref_ren_new], outputs=[ref_rename_status])
             ref_del_dist_btn.click(fn=ui_ref_del_dist, inputs=[ref_del_state, ref_del_dist], outputs=[ref_del_status])

hf_store.py

@@ -122,7 +122,7 @@ DISTRICT_REF_PATH = "district_reference/district_reference.parquet"
 def push_district_reference(df: pd.DataFrame, token: str, repo: str) -> str:
     """
     Push the SF district reference table to HF dataset.
-    Expected columns: State, District, SF_Status ('in_sf' | 'pending_sf_addition')
+    Expected columns: State, District, Status ('present' | 'new districts found')
     Returns commit URL.
     """
     return _upload_parquet(df, DISTRICT_REF_PATH, token, repo, "Update district_reference")
@@ -131,20 +131,20 @@ def push_district_reference(df: pd.DataFrame, token: str, repo: str) -> str:
 def pull_district_reference(token: str, repo: str) -> pd.DataFrame:
     """
     Pull the SF district reference from HF dataset.
-    Returns DataFrame with columns: State, District, SF_Status
+    Returns DataFrame with columns: State, District, Status
     Returns empty DataFrame if not yet seeded.
     """
     df = _download_parquet(DISTRICT_REF_PATH, token, repo)
     if df.empty:
-        return pd.DataFrame(columns=["State", "District", "SF_Status"])
+        return pd.DataFrame(columns=["State", "District", "Status"])
     return df
 
 
 def seed_district_reference_from_csv(csv_path: str, token: str, repo: str) -> tuple:
     """
     Seed Dataset 1 from a CSV (like master_all_states.csv or SF mapping CSV).
-    All states/districts in the CSV → SF_Status = 'in_sf'
-    Any states in current scraped data NOT in CSV → SF_Status = 'pending_sf_addition'
+    All states/districts in the CSV → Status = 'present'
+    Any states in current scraped data NOT in CSV → Status = 'new districts found'
     Returns (df, commit_url)
     """
     csv_df = pd.read_csv(csv_path)
@@ -163,9 +163,9 @@ def seed_district_reference_from_csv(csv_path: str, token: str, repo: str) -> tu
     csv_df = csv_df.dropna(subset=[state_col])
     csv_df["State"] = csv_df[state_col].astype(str).str.strip().str.upper()
     csv_df["District"] = csv_df[district_col].astype(str).str.strip().str.upper()
-    csv_df["SF_Status"] = "in_sf"
+    csv_df["Status"] = "present"
 
-    ref_df = csv_df[["State", "District", "SF_Status"]].drop_duplicates()
+    ref_df = csv_df[["State", "District", "Status"]].drop_duplicates()
 
     # Pull existing scraped metadata to check for any states not in the CSV
     meta = _pull_metadata(token, repo)
@@ -176,7 +176,7 @@ def seed_district_reference_from_csv(csv_path: str, token: str, repo: str) -> tu
         if state_upper not in csv_states:
             # State exists in scraped data but not in SF CSV → pending
             # We don't know districts yet; they'll be added when detected
-            new_rows.append({"State": state_upper, "District": "ALL", "SF_Status": "pending_sf_addition"})
+            new_rows.append({"State": state_upper, "District": "ALL", "Status": "new districts found"})
 
     if new_rows:
         ref_df = pd.concat([ref_df, pd.DataFrame(new_rows)], ignore_index=True)
@@ -192,7 +192,7 @@ def update_district_reference_add(state: str, district: str, sf_status: str, tok
     district = district.strip().upper()
     # Remove existing row if present
     df = df[~((df["State"] == state) & (df["District"] == district))]
-    new_row = pd.DataFrame([{"State": state, "District": district, "SF_Status": sf_status}])
+    new_row = pd.DataFrame([{"State": state, "District": district, "Status": sf_status}])
     df = pd.concat([df, new_row], ignore_index=True)
     return push_district_reference(df, token, repo)
 
@@ -240,7 +240,7 @@ def delete_state_reference(state: str, token: str, repo: str) -> str:
 
 def bulk_accept_flagged_districts(token: str, repo: str) -> None:
     """
-    Take all currently flagged districts, add them to Dataset 1 with 'pending_sf_addition',
+    Take all currently flagged districts, add them to Dataset 1 with 'new districts found',
     and clear the flagged_districts list from metadata.
     """
     meta = _pull_metadata(token, repo)
@@ -255,7 +255,7 @@ def bulk_accept_flagged_districts(token: str, repo: str) -> None:
         state = f["state"].strip().upper()
         dist = f["district"].strip().upper()
         if not ((df["State"] == state) & (df["District"] == dist)).any():
-            new_rows.append({"State": state, "District": dist, "SF_Status": "pending_sf_addition"})
+            new_rows.append({"State": state, "District": dist, "Status": "new districts found"})
     
     if new_rows:
         df = pd.concat([df, pd.DataFrame(new_rows)], ignore_index=True)
@@ -361,9 +361,9 @@ def delete_mapping_rule(state: str, dist_old: str, dist_new: str, token: str, re
 # ─────────────────────────────────────────────────────────────────────────────
 
 def get_flagged_districts(token: str, repo: str) -> list:
-    """Return all districts from Dataset 1 that have SF_Status == 'pending_sf_addition', filtered by states with raw data."""
+    """Return all districts from Dataset 1 that have Status == 'new districts found', filtered by states with raw data."""
     ref_df = pull_district_reference(token, repo)
-    if ref_df.empty or "SF_Status" not in ref_df.columns:
+    if ref_df.empty or "Status" not in ref_df.columns:
         return []
         
     try:
@@ -379,7 +379,7 @@ def get_flagged_districts(token: str, repo: str) -> list:
     except:
         active_states = set(ref_df["State"].str.upper()) # fallback
 
-    pending = ref_df[(ref_df["SF_Status"] == "pending_sf_addition") & (ref_df["State"].str.upper().isin(active_states))]
+    pending = ref_df[(ref_df["Status"] == "new districts found") & (ref_df["State"].str.upper().isin(active_states))]
     
     dist_df, _ = pull_mapping_rules(token, repo)
     mapped = set()
@@ -397,7 +397,7 @@ def get_flagged_districts(token: str, repo: str) -> list:
             flags.append({
                 "state": state,
                 "district": district,
-                "reason": "Pending SF Addition"
+                "reason": "New district detected"
             })
     return flags
 
@@ -409,7 +409,7 @@ def rename_district_in_reference(state: str, district: str, new_name: str, token
     exists = not ref_df[(ref_df["State"] == state) & (ref_df["District"] == new_name)].empty
     
     # Target row to update/delete
-    mask = (ref_df["State"] == state) & (ref_df["District"] == district) & (ref_df["SF_Status"] == "pending_sf_addition")
+    mask = (ref_df["State"] == state) & (ref_df["District"] == district) & (ref_df["Status"] == "new districts found")
     
     if not ref_df[mask].empty:
         if exists:
@@ -483,7 +483,7 @@ def _detect_and_flag_new_districts(state_name: str, df: pd.DataFrame, token: str
     """
     Compare districts in newly scraped df against Dataset 1 and Dataset 2.
     Any district not in Dataset 1 AND not mapped in Dataset 2 is truly new.
-    Add them straight to Dataset 1 with SF_Status = 'pending_sf_addition'.
+    Add them straight to Dataset 1 with Status = 'new districts found'.
     """
     state_upper = state_name.strip().upper()
     if state_upper in NON_ACTUAL_STATES:
@@ -555,13 +555,13 @@ def _detect_and_flag_new_districts(state_name: str, df: pd.DataFrame, token: str
         return
 
     # Add directly to Dataset 1 as pending
-    new_rows = [{"State": state_upper, "District": d, "SF_Status": "pending_sf_addition"} for d in new_districts]
+    new_rows = [{"State": state_upper, "District": d, "Status": "new districts found"} for d in new_districts]
     updated_ref = pd.concat([ref_df, pd.DataFrame(new_rows)], ignore_index=True)
     push_district_reference(updated_ref, token, repo)
 
 
 def delete_state_file(state_name: str, token: str, repo: str) -> None:
-    """Delete a state's raw dataset file and remove it from metadata.json."""
+    """Delete a state's raw dataset file and remove it from metadata.json, and also clean up pending newly found districts for that state."""
     api = _hf()
     skey = _state_key(state_name)
     try:
@@ -579,6 +579,20 @@ def delete_state_file(state_name: str, token: str, repo: str) -> None:
     if state_name in meta:
         del meta[state_name]
         _push_metadata(meta, token, repo)
+        
+    # Also clean up any "new districts found" for this deleted state in the District Reference
+    try:
+        ref_df = pull_district_reference(token, repo)
+        if not ref_df.empty:
+            # We want to KEEP rows that are NOT (State == state_name and Status == 'new districts found')
+            original_len = len(ref_df)
+            state_upper = state_name.upper()
+            mask = ~((ref_df["State"].str.upper() == state_upper) & (ref_df["Status"] == "new districts found"))
+            cleaned_ref = ref_df[mask]
+            if len(cleaned_ref) < original_len:
+                push_district_reference(cleaned_ref, token, repo)
+    except Exception as e:
+        print(f"Failed to cleanup district reference on state delete: {e}")
 
 
 def pull_state_file(state_name: str, token: str, repo: str) -> pd.DataFrame: