feat: added multi-year Master Analytics tab & updated documentation

README.md

@@ -61,7 +61,7 @@ The absolute easiest way to use the pipeline is via the interactive browser UI.
 ```powershell
 python app.py
 ```
-The app opens automatically at `http://127.0.0.1:7861` and features **4 main tabs**.
+The app opens automatically at `http://127.0.0.1:7861` and features **5 main tabs**.
 
 ---
 
@@ -112,6 +112,17 @@ A full database management dashboard to maintain your Scholarship Application ma
 
 ---
 
+### 📊 Tab 5: Master Analytics
+
+A multi-year analytics dashboard powered strictly by Pandas to track macro-level trends across your historical master sheets.
+
+- **Automated Timeline:** Dynamically loads the **2025 Master** and compares it against every single mapped master sheet ever uploaded to your HuggingFace dataset.
+- **Missing Schools Tracker:** Automatically computes the difference in UDISE codes to show exactly which schools have disappeared or dropped off since 2025.
+- **Privatization / Management Shifts:** Seamlessly merges historical data with new data to identify schools that have shifted from strictly Government management to Non-Government (Aided/Private) management.
+- **Zero Web Scraping:** This tab operates entirely on your compiled HuggingFace data, requiring no active web scraping or internet crawling.
+
+---
+
 ## 🛠️ Admin Scripts
 
 If you need to make global, architectural changes to the baseline data outside of the UI, use the provided admin scripts:

USER_MANUAL.md

@@ -12,7 +12,8 @@ This manual provides a detailed, step-by-step guide to using every feature of th
 3. [Tab 2: 📋 Master Sheet (Builder)](#3-tab-2--master-sheet-builder)
 4. [Tab 3: 🗺️ Mapping Manager](#4-tab-3--mapping-manager)
 5. [Tab 4: 📥 Download History](#5-tab-4--download-history)
-6. [Best Practices & Tips](#6-best-practices--tips)
+6. [Tab 5: 📊 Master Analytics](#6-tab-5--master-analytics)
+7. [Best Practices & Tips](#7-best-practices--tips)
 
 ---
 
@@ -129,7 +130,28 @@ This tab serves as your cloud backup and archive.
 
 ---
 
-## 6. Best Practices & Tips
+## 6. Tab 5: 📊 Master Analytics
+
+This tab serves as your multi-year historical analysis dashboard, providing high-level macro insights across your entire dataset over time without doing any active web scraping.
+
+### 📈 Multi-Year Master Analytics
+When you click **🔄 Run Analytics**, the system will:
+1. Automatically pull your 2025 anchor master dataset.
+2. Automatically pull *every single* mapped master sheet (`mapped_master_{date}.parquet`) you have ever compiled from HuggingFace.
+3. Compare the newest master against the 2025 data.
+
+### 📅 Master Timeline
+A high-level historical breakdown showing exactly how the total number of schools (and Government schools) shifts from year to year.
+
+### ⚠️ Missing Schools (vs 2025)
+This table explicitly tracks the UDISE codes of schools that existed in 2025 but are completely absent from the newest compiled master sheet. It filters out duplicates automatically for precise counts. 
+
+### 🔄 Management Shifts (Government → Non-Govt)
+By merging the datasets on their UDISE codes, this table instantly flags any school that was previously managed strictly by the Government in 2025 (e.g. Dept of Education, Panchayat) but has now shifted to an Aided, Private, or Unaided management structure in the newest master.
+
+---
+
+## 7. Best Practices & Tips
 
 - **Do Not Interrupt the Scraper:** If you close the browser tab while Tab 1 is actively scraping, the process will stop. Let it run in the background.
 - **Always Resolve Anomalies:** Never ignore the "New districts detected" table in Tab 2. If you force a build without resolving these, those schools will carry the newly scraped district names into your final database exactly as they are. You must ensure those new names match your Scholarship Application or rename them in the table before building.

app.py

@@ -1539,6 +1539,191 @@ with gr.Blocks(title="School Data Fetcher", css=css, theme=custom_theme) as app:
             dl_dropdown.change(fn=ui_prep_download, inputs=[dl_dropdown], outputs=[dl_download_btn, dl_status])
 
 
+        # ── Tab 5: Master Analytics ────────────────────────────────────────────
+        with gr.Tab("📊 Master Analytics"):
+            gr.HTML('''
+            <div style="background:linear-gradient(135deg,rgba(16,185,129,0.08),rgba(14,165,233,0.08));
+                        border:1px solid rgba(16,185,129,0.2); border-radius:14px; padding:18px 22px; margin-bottom:18px;">
+                <div style="font-weight:700; font-size:1.05em; margin-bottom:6px;">📈 Multi-Year Master Analytics</div>
+                <div style="font-size:.9em; color:var(--body-text-color-subdued); line-height:1.7;">
+                    This tab automatically loads the <b>2025 master</b> and every
+                    <b>mapped master sheet</b> ever built. It analyzes how government school footprints and overall school counts shift year over year.
+                </div>
+            </div>
+            ''')
+
+            with gr.Row():
+                analytics_refresh_btn = gr.Button("🔄 Run Analytics", variant="primary", scale=2)
+            analytics_status = gr.HTML()
+
+            gr.HTML("<div class='section-label'>📅 Master Timeline</div>")
+            gr.HTML("<div style='font-size:.85em;color:var(--body-text-color-subdued);margin-bottom:8px;'>Year-over-year changes across all master sheets compared to 2025.</div>")
+            timeline_table = gr.Dataframe(label="Timeline Metrics", interactive=False, wrap=True)
+
+            gr.HTML("<div class='section-label'>⚠️ Missing Schools (vs 2025)</div>")
+            missing_schools_html = gr.HTML()
+            missing_schools_table = gr.Dataframe(label="Missing Schools Detailed", interactive=False, wrap=True)
+            
+            gr.HTML("<div class='section-label'>🔄 Management Shifts</div>")
+            mgmt_shifts_html = gr.HTML()
+            mgmt_shifts_table = gr.Dataframe(label="Government → Non-Govt Shifts", interactive=False, wrap=True)
+
+            BASE_CSV_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "master_all_states.csv")
+
+            def _comp_detect_year_label(filepath: str) -> str:
+                import calendar as _cal
+                import re as _re
+                import os as _os
+                name = _os.path.basename(filepath or "")
+                m = _re.search(r"(\d{4})[-_](\d{2})", name)
+                if m:
+                    yr, mo = int(m.group(1)), int(m.group(2))
+                    if 1 <= mo <= 12:
+                        return f"{_cal.month_abbr[mo]} {yr}"
+                m2 = _re.search(r"(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[_\-]?(\d{4})", name, _re.IGNORECASE)
+                if m2:
+                    return f"{m2.group(1).capitalize()} {m2.group(2)}"
+                m3 = _re.search(r"(\d{4})", name)
+                return m3.group(1) if m3 else name
+
+            def _comp_find_col(df, *keywords):
+                for k in keywords:
+                    for c in df.columns:
+                        if k in c.lower():
+                            return c
+                return None
+
+            def ui_run_analytics():
+                from hf_store import get_hf_credentials, list_mapped_master_files, pull_baseline_master
+                from huggingface_hub import hf_hub_download
+                import pandas as pd
+                import os
+
+                token, repo = get_hf_credentials()
+                if not token or not repo:
+                    return "<p style='color:#ef4444;'>⚠️ HF credentials not set.</p>", pd.DataFrame(), "", pd.DataFrame(), "", pd.DataFrame()
+
+                masters = []
+                # 1. Load baseline
+                try:
+                    base_df = pull_baseline_master(token, repo)
+                    if not base_df.empty:
+                        masters.append({"label": "2025", "df": base_df})
+                except Exception:
+                    pass
+                
+                # Fallback to CSV if HF fails
+                if not masters and os.path.exists(BASE_CSV_PATH):
+                    masters.append({"label": "2025", "df": pd.read_csv(BASE_CSV_PATH, low_memory=False)})
+
+                # 2. Pull HF mapped masters
+                hf_files = list_mapped_master_files(token, repo)
+                for hf_path in sorted(hf_files):
+                    try:
+                        local = hf_hub_download(repo_id=repo, repo_type="dataset", filename=hf_path, token=token, force_download=True)
+                        label = _comp_detect_year_label(hf_path)
+                        masters.append({"label": label, "df": pd.read_parquet(local)})
+                    except Exception:
+                        pass
+                
+                if not masters:
+                    return "<p style='color:#ef4444;'>⚠️ No master datasets found.</p>", pd.DataFrame(), "", pd.DataFrame(), "", pd.DataFrame()
+
+                def _norm_udise(s):
+                    return s.astype(str).str.strip().str.zfill(11)
+
+                def is_govt(s):
+                    val = str(s).lower()
+                    if 'aided' in val or 'private' in val or 'unaided' in val or 'un-aided' in val:
+                        return False
+                    return 'gov' in val or 'dept of edu' in val or 'local body' in val or 'panchayat' in val or 'municipal' in val
+                
+                baseline = masters[0]
+                b_udise = _comp_find_col(baseline["df"], "udise")
+                b_mgmt = _comp_find_col(baseline["df"], "school_management_type", "management", "mgmt")
+                if b_udise: baseline["df"]["_U"] = _norm_udise(baseline["df"][b_udise])
+                if b_mgmt: baseline["df"]["is_govt"] = baseline["df"][b_mgmt].apply(is_govt)
+
+                timeline_data = []
+                baseline_udises = set(baseline["df"]["_U"].dropna()) if b_udise else set()
+
+                missing_df = pd.DataFrame()
+                changed_mgmt = pd.DataFrame()
+
+                for m in masters:
+                    m_df = m["df"]
+                    m_udise = _comp_find_col(m_df, "udise")
+                    m_mgmt = _comp_find_col(m_df, "school_management_type", "management", "mgmt")
+                    m_name = _comp_find_col(m_df, "school_name", "schoolname")
+                    m_state = _comp_find_col(m_df, "state")
+                    
+                    if m_udise: m_df["_U"] = _norm_udise(m_df[m_udise])
+                    if m_mgmt: m_df["is_govt"] = m_df[m_mgmt].apply(is_govt)
+
+                    tot_schools = len(m_df)
+                    tot_govt = m_df["is_govt"].sum() if m_mgmt else 0
+
+                    missing_count = 0
+                    shifts_count = 0
+
+                    if m != baseline and b_udise and m_udise:
+                        m_udises = set(m_df["_U"].dropna())
+                        missing = baseline_udises - m_udises
+                        missing_count = len(missing)
+                        
+                        if m == masters[-1]: 
+                            b_name = _comp_find_col(baseline["df"], "school_name", "schoolname")
+                            b_state = _comp_find_col(baseline["df"], "state")
+                            missing_df = baseline["df"][baseline["df"]["_U"].isin(missing)].drop_duplicates(subset=["_U"]).copy()
+                            
+                            cols_to_keep = []
+                            for c in ["_U", b_name, b_state, b_mgmt]:
+                                if c and c in missing_df.columns:
+                                    cols_to_keep.append(c)
+                            missing_df = missing_df[cols_to_keep] if cols_to_keep else missing_df
+                            missing_df.insert(0, "Missing In Master", m["label"])
+                            
+                            if b_mgmt and m_mgmt:
+                                b_cols = [c for c in ["_U", b_name, b_mgmt, "is_govt"] if c]
+                                m_cols = [c for c in ["_U", m_name, m_mgmt, "is_govt"] if c]
+                                
+                                merged = pd.merge(
+                                    baseline["df"][b_cols], 
+                                    m_df[m_cols], 
+                                    on="_U", suffixes=("_old", "_new")
+                                )
+                                changed = merged[(merged["is_govt_old"] == True) & (merged["is_govt_new"] == False)]
+                                shifts_count = len(changed)
+                                
+                                c_name_old = str(b_name) + '_old' if b_name == m_name else b_name
+                                c_mgmt_old = str(b_mgmt) + '_old' if b_mgmt == m_mgmt else b_mgmt
+                                c_mgmt_new = str(m_mgmt) + '_new' if b_mgmt == m_mgmt else m_mgmt
+                                
+                                c_cols = [c for c in ["_U", c_name_old, c_mgmt_old, c_mgmt_new] if c and c in changed.columns]
+                                changed_mgmt = changed[c_cols] if c_cols else changed
+                                changed_mgmt.insert(0, "Shifted In Master", m["label"])
+
+                    timeline_data.append({
+                        "Master Name": m["label"],
+                        "Total Schools": f"{tot_schools:,}",
+                        "Total Govt Schools": f"{tot_govt:,}",
+                        "Missing (vs 2025)": f"{missing_count:,}" if m != baseline else "-",
+                        "Govt->Non-Govt Shifts": f"{shifts_count:,}" if m != baseline else "-"
+                    })
+                
+                timeline_df = pd.DataFrame(timeline_data)
+                
+                status_html = f"<p style='color:#22c55e;font-weight:600;'>✅ Loaded <b>{len(masters)}</b> master datasets successfully.</p>"
+                m_html = f"<div style='font-size:1.1em; margin-top:10px;'><b>{len(missing_df):,}</b> schools from 2025 are completely missing in the newest master.</div>"
+                c_html = f"<div style='font-size:1.1em; margin-top:10px;'><b>{len(changed_mgmt):,}</b> schools shifted from Govt to Non-Govt management since 2025.</div>"
+
+                return status_html, timeline_df, m_html, missing_df, c_html, changed_mgmt
+
+            analytics_refresh_btn.click(
+                fn=ui_run_analytics,
+                outputs=[analytics_status, timeline_table, missing_schools_html, missing_schools_table, mgmt_shifts_html, mgmt_shifts_table]
+            )
+
 if __name__ == "__main__":
     if os.environ.get("SPACE_ID"):
         print("Starting on Dataset Spaces (0.0.0.0:7860) …")

master_analysis.py

@@ -0,0 +1,71 @@
+import pandas as pd
+import os
+
+def is_govt(management_name):
+    """Simple function to classify if a school is strictly Government managed"""
+    val = str(management_name).lower()
+    if 'aided' in val or 'private' in val or 'unaided' in val or 'un-aided' in val:
+        return False
+    return 'gov' in val or 'dept of edu' in val or 'local body' in val or 'panchayat' in val or 'municipal' in val
+
+def run_analysis():
+    # Define paths (Update the parquet path to point to a real file when testing)
+    csv_path = "master_all_states.csv"
+    parquet_path = "mapped_master_2026_01.parquet" # Replace with your actual parquet file path
+
+    if not os.path.exists(csv_path) or not os.path.exists(parquet_path):
+        print(f"Make sure both {csv_path} and {parquet_path} exist in this folder to run the script!")
+        return
+
+    # 1. Load the 2025 dataset and the Newest dataset
+    print("Loading datasets...")
+    df_2025 = pd.read_csv(csv_path, low_memory=False)
+    df_new = pd.read_parquet(parquet_path)
+
+    # 2. Normalize UDISE codes to ensure perfect 11-digit matching
+    df_2025['_U'] = df_2025['School_Udise_Code__c'].astype(str).str.strip().str.zfill(11)
+    df_new['_U'] = df_new['School_Udise_Code__c'].astype(str).str.strip().str.zfill(11)
+
+    # 3. Apply Government Flag using the strict School_Management_Type__c column
+    df_2025['is_govt'] = df_2025['School_Management_Type__c'].apply(is_govt)
+    df_new['is_govt'] = df_new['School_Management_Type__c'].apply(is_govt)
+
+    print(f"Total Govt Schools (2025): {df_2025['is_govt'].sum():,}")
+    print(f"Total Govt Schools (New):  {df_new['is_govt'].sum():,}")
+
+    # ---------------------------------------------------------
+    # FINDING MISSING SCHOOLS
+    # ---------------------------------------------------------
+    udises_2025 = set(df_2025['_U'].dropna())
+    udises_new = set(df_new['_U'].dropna())
+    missing_udises = udises_2025 - udises_new
+    
+    # Filter 2025 data to only show missing schools, and DROP duplicates for accurate counts!
+    missing_df = df_2025[df_2025['_U'].isin(missing_udises)].drop_duplicates(subset=['_U']).copy()
+    
+    # Add Master Year column
+    missing_df.insert(0, "Missing In Master", "2026") 
+    
+    print(f"\nTotal Missing Schools: {len(missing_df)}")
+    print("Missing Schools by State:\n", missing_df['School_State__c'].value_counts().head())
+
+    # ---------------------------------------------------------
+    # FINDING MANAGEMENT SHIFTS (Govt -> Non-Govt)
+    # ---------------------------------------------------------
+    # Merge the two datasets on UDISE code
+    merged = pd.merge(
+        df_2025[['_U', 'School_Name__c', 'School_Management_Type__c', 'is_govt']], 
+        df_new[['_U', 'School_Name__c', 'School_Management_Type__c', 'is_govt']], 
+        on='_U', 
+        suffixes=('_old', '_new')
+    )
+
+    # Filter where it WAS govt, but is NO LONGER govt
+    changed_mgmt = merged[(merged['is_govt_old'] == True) & (merged['is_govt_new'] == False)].copy()
+    changed_mgmt.insert(0, "Shifted In Master", "2026")
+
+    print(f"\nSchools shifted from Govt -> Non-Govt: {len(changed_mgmt)}")
+    return missing_df, changed_mgmt
+
+if __name__ == "__main__":
+    run_analysis()