Sync from GitHub via hub-sync

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data/electricity.data filter=lfs diff=lfs merge=lfs -text

Dockerfile

@@ -1,20 +0,0 @@
-FROM python:3.13.5-slim
-
-WORKDIR /app
-
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    curl \
-    git \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY requirements.txt ./
-COPY src/ ./src/
-
-RUN pip3 install -r requirements.txt
-
-EXPOSE 8501
-
-HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health
-
-ENTRYPOINT ["streamlit", "run", "src/streamlit_app.py", "--server.port=8501", "--server.address=0.0.0.0"]

README.md

@@ -1,19 +1,96 @@
+# Electricity Utility Inefficiency & Residential Rate Analysis
+
+> **Research Question:** Are system-level inefficiencies — high energy losses and poor load factors — statistically correlated with higher retail rates for residential consumers?
+
+---
+
+## Overview
+
+This project investigates a fundamental fairness question in the U.S. electricity sector: do residential customers end up paying more when their utility operates inefficiently? Using the [CORGIS Electricity Dataset](https://corgis-edu.github.io/corgis/), this analysis builds a set of derived efficiency and equity metrics, then examines their statistical relationships through a suite of interactive visualizations.
+
+New York State serves as the primary case study. NY was selected through a data-driven ranking process (get_state_variance) that scores all 50 states across five analytical criteria: number of utilities, number of ownership types, residential price standard deviation, maximum system loss percentage, and industrial revenue dependency. New York ranks at or near the top on all five: it has over 100 utilities across 6 distinct ownership models, exhibits high residential price variance, and sits within one of the most actively scrutinized regulatory environments in the U.S.
+
+[![Open in HF Space](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/chalseee/utilities-equity-efficiency-gap)
 ---
-title: Utilities Equity Efficiency Gap
-emoji: 🚀
-colorFrom: red
-colorTo: red
-sdk: docker
-app_port: 8501
-tags:
-- streamlit
-pinned: false
-short_description: Electricity Utility Residential Rate Analysis
+
+## Key Metrics
+
+| Metric | Description |
+|---|---|
+| `System Loss Percentage` | Energy lost in transmission/distribution as % of total supply |
+| `Load Factor` | Ratio of actual energy delivered to theoretical maximum (demand efficiency) |
+| `Residential Unit Price` | Residential rate in $/MWh |
+| `Industrial Unit Price` | Industrial rate in $/MWh |
+| `Price Spread` | Gap between residential and industrial rates (equity indicator) |
 ---
 
-# Welcome to Streamlit!
+## Visualizations
+
+- **State Selection Table** — Ranking of top 10 states by analytical richness; justifies NY case study
+- **Price Spread Strip Plot** — Residential premium over industrial rates by ownership model
+- **Correlation Heatmap** — Statistical significance matrix across all key metrics
+- **Fairness Audit Scatter** — System loss (%) and load factor (dual y-axis) vs. residential price by ownership model, with OLS trendline
+- **Rate Disparity Dumbbell** — Top 10 utilities by residential–industrial price gap
+- **Energy Flow Sankey Diagram (State)** — Per-state (or utility) breakdown of energy sources and uses as percentages 
+- **Energy Flow Sankey Diagram (US)** — National aggregate energy flow
+
+---
+
+## Project Structure
+
+```
+.
+├── utility_efficiency_fairness.ipynb
+├── streamlit_app.py
+├── requirements.txt
+├── README.md
+├── data/
+│   └── app.py
+│   └── electricity.data
+│   └── electricity.data
+├── src/
+│   └── util/
+│       ├── data_util.py
+│       └── plot_util.py
+└── images/            
+```
+
+The three modules work as a clean pipeline:
+- **`electricity`** — CORGIS data loader (unmodified third-party)
+- **`data_util`** — all data preparation, filtering, and metric engineering
+- **`plot_util`** — all chart construction and SVG expor
+
+### `util.py` — Helper Functions
+
+| Function | Purpose |
+|---|---|
+| `prepare_data(df)` | Calculates key metrics and other features from raw columns |
+| `get_state_data(state, df)` | Filters and subsets data for a given state |
+| `get_state_variance(df)` | Ranks states by residential price variance (for state selection) |
+| `get_customer_utilities(df, customer)` | Filters utilities by customer type served |
+| `get_residential_load_factor(df)` | Subset with valid load factor for residential utilities |
+| `get_residential_sys_loss(df)` | Subset with valid loss % for residential utilities |
+| `get_utility_usage(utility)` | Converts raw MWh values to % of total supply for Sankey |
+
+---
+
+## Setup & Usage
+
+- The dataset is pre-bundled — no external downloads required for it.
+- Run `jupytr notebook utility_efficiency_fairness.ipynb.`
+- Run pip install pandas plotly scipy kaleido streamlit
+
+---
+
+## Data Source
+
+**[CORGIS Electricity Dataset](https://corgis-edu.github.io/corgis/python/electricity/)** — a cleaned and structured snapshot of U.S. Energy Information Administration (EIA) [Form 861](https://www.eia.gov/electricity/data/eia861/) data, covering utility-level electricity generation, sales, revenues, and customer counts across all U.S. states.
+
+---
 
-Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart:
+## Potential Extensions
 
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
+- Expand the analysis to all 50 states and compare regional patterns
+- Incorporate time-series data to track changes in efficiency and pricing
+- Apply regression modeling to control for utility size and customer density
+- Explore the role of renewable energy mix on system loss rates

data/app.py

@@ -0,0 +1,134 @@
+'''
+Hello! Thank you for downloading a CORGIS library. However, you do not
+need to open this file. Instead you should make your own Python file and
+add the following line:
+
+import electricity
+
+Then just place the files you downloaded alongside it.
+'''
+
+import os as _os
+import pickle as _pickle
+
+__all__ = ['get_utility']
+
+def _tifa_definitions():
+    return {"type": "ModuleType",
+        "fields": {
+            'get': {
+                "type": "FunctionType",
+                "name": 'get',
+                "returns": {
+                    "type": "ListType", 
+                    "empty": False, 
+                    "subtype": {"type": "NumType"}
+                }
+            },
+            'get_utility': {
+                "type": "FunctionType",
+                "name": 'get_utility',
+                "returns": 
+				{"type": "ListType", "subtype": 
+					{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Utility'}, {"type": "LiteralStr", "value": 'Demand'}, {"type": "LiteralStr", "value": 'Sources'}, {"type": "LiteralStr", "value": 'Uses'}, {"type": "LiteralStr", "value": 'Revenues'}, {"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Retail'}], "values": [
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Number'}, {"type": "LiteralStr", "value": 'Name'}, {"type": "LiteralStr", "value": 'State'}, {"type": "LiteralStr", "value": 'Type'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "StrType"}, 
+							{"type": "StrType"}, 
+							{"type": "StrType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Summer Peak'}, {"type": "LiteralStr", "value": 'Winter Peak'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Generation'}, {"type": "LiteralStr", "value": 'Purchased'}, {"type": "LiteralStr", "value": 'Other'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Retail'}, {"type": "LiteralStr", "value": 'Resale'}, {"type": "LiteralStr", "value": 'No Charge'}, {"type": "LiteralStr", "value": 'Consumed'}, {"type": "LiteralStr", "value": 'Losses'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Retail'}], "values": [
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Delivery'}, {"type": "LiteralStr", "value": 'Resale'}, {"type": "LiteralStr", "value": 'Adjustments'}, {"type": "LiteralStr", "value": 'Transmission'}, {"type": "LiteralStr", "value": 'Other'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Residential'}, {"type": "LiteralStr", "value": 'Commercial'}, {"type": "LiteralStr", "value": 'Industrial'}, {"type": "LiteralStr", "value": 'Transportation'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}]}]}}
+            },
+        }
+    }
+
+class _Constants(object):
+    '''
+    Global singleton object to hide some of the constants; some IDEs reveal
+    internal module details very aggressively, and there's no other way
+    to hide stuff.
+    '''
+
+class DatasetException(Exception):
+    ''' Thrown when there is an error loading the dataset for some reason.'''
+    
+_Constants._DATABASE_NAME = _os.path.join(_os.path.dirname(__file__),
+                                          "electricity.data")
+if not _os.access(_Constants._DATABASE_NAME, _os.F_OK):
+    raise DatasetException(("Error! Could not find a \"{0}\" file. "
+                           "Make sure that there is a \"{0}\" in the "
+                           "same directory as \"{1}.py\"! Spelling is "
+                           "very important here."
+                           ).format(_Constants._DATABASE_NAME, __name__))
+elif not _os.access(_Constants._DATABASE_NAME, _os.R_OK):
+    raise DatasetException(("Error! Could not read the \"{0}\" file. "
+                            "Make sure that it readable by changing its "
+                            "permissions. You may need to get help from "
+                            "your instructor."
+                            ).format(_Constants._DATABASE_NAME, __name__))
+
+
+_Constants._DATASET = None
+
+def get_utility():
+    """
+    Retrieves all of the utility.
+    """
+    if _Constants._DATASET is None:
+        with open(_Constants._DATABASE_NAME, 'rb') as _:
+            _Constants._DATASET = _pickle.load(_)
+    return _Constants._DATASET
+
+if __name__ == '__main__':
+    from pprint import pprint as _pprint
+    from timeit import default_timer as _default_timer
+    
+    print(">>> get_utility()")
+    
+    start_time = _default_timer()
+    result = get_utility()
+    print("Time taken: {}".format(_default_timer() - start_time))
+    _pprint(result[0])

data/electricity.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2073f807dca20eeac617d9ea9fb7be8b4f9eb8203f48afeb408b269bd472b948
+size 4261662

data/electricity.py

@@ -0,0 +1,134 @@
+'''
+Hello! Thank you for downloading a CORGIS library. However, you do not
+need to open this file. Instead you should make your own Python file and
+add the following line:
+
+import electricity
+
+Then just place the files you downloaded alongside it.
+'''
+
+import os as _os
+import pickle as _pickle
+
+__all__ = ['get_utility']
+
+def _tifa_definitions():
+    return {"type": "ModuleType",
+        "fields": {
+            'get': {
+                "type": "FunctionType",
+                "name": 'get',
+                "returns": {
+                    "type": "ListType", 
+                    "empty": False, 
+                    "subtype": {"type": "NumType"}
+                }
+            },
+            'get_utility': {
+                "type": "FunctionType",
+                "name": 'get_utility',
+                "returns": 
+				{"type": "ListType", "subtype": 
+					{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Utility'}, {"type": "LiteralStr", "value": 'Demand'}, {"type": "LiteralStr", "value": 'Sources'}, {"type": "LiteralStr", "value": 'Uses'}, {"type": "LiteralStr", "value": 'Revenues'}, {"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Retail'}], "values": [
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Number'}, {"type": "LiteralStr", "value": 'Name'}, {"type": "LiteralStr", "value": 'State'}, {"type": "LiteralStr", "value": 'Type'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "StrType"}, 
+							{"type": "StrType"}, 
+							{"type": "StrType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Summer Peak'}, {"type": "LiteralStr", "value": 'Winter Peak'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Generation'}, {"type": "LiteralStr", "value": 'Purchased'}, {"type": "LiteralStr", "value": 'Other'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Retail'}, {"type": "LiteralStr", "value": 'Resale'}, {"type": "LiteralStr", "value": 'No Charge'}, {"type": "LiteralStr", "value": 'Consumed'}, {"type": "LiteralStr", "value": 'Losses'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Retail'}], "values": [
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Delivery'}, {"type": "LiteralStr", "value": 'Resale'}, {"type": "LiteralStr", "value": 'Adjustments'}, {"type": "LiteralStr", "value": 'Transmission'}, {"type": "LiteralStr", "value": 'Other'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}, 
+							{"type": "NumType"}]}, 
+						{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Residential'}, {"type": "LiteralStr", "value": 'Commercial'}, {"type": "LiteralStr", "value": 'Industrial'}, {"type": "LiteralStr", "value": 'Transportation'}, {"type": "LiteralStr", "value": 'Total'}], "values": [
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}, 
+							{"type": "DictType", "literals": [{"type": "LiteralStr", "value": 'Revenue'}, {"type": "LiteralStr", "value": 'Sales'}, {"type": "LiteralStr", "value": 'Customers'}], "values": [
+								{"type": "NumType"}, 
+								{"type": "NumType"}, 
+								{"type": "NumType"}]}]}]}}
+            },
+        }
+    }
+
+class _Constants(object):
+    '''
+    Global singleton object to hide some of the constants; some IDEs reveal
+    internal module details very aggressively, and there's no other way
+    to hide stuff.
+    '''
+
+class DatasetException(Exception):
+    ''' Thrown when there is an error loading the dataset for some reason.'''
+    
+_Constants._DATABASE_NAME = _os.path.join(_os.path.dirname(__file__),
+                                          "electricity.data")
+if not _os.access(_Constants._DATABASE_NAME, _os.F_OK):
+    raise DatasetException(("Error! Could not find a \"{0}\" file. "
+                           "Make sure that there is a \"{0}\" in the "
+                           "same directory as \"{1}.py\"! Spelling is "
+                           "very important here."
+                           ).format(_Constants._DATABASE_NAME, __name__))
+elif not _os.access(_Constants._DATABASE_NAME, _os.R_OK):
+    raise DatasetException(("Error! Could not read the \"{0}\" file. "
+                            "Make sure that it readable by changing its "
+                            "permissions. You may need to get help from "
+                            "your instructor."
+                            ).format(_Constants._DATABASE_NAME, __name__))
+
+
+_Constants._DATASET = None
+
+def get_utility():
+    """
+    Retrieves all of the utility.
+    """
+    if _Constants._DATASET is None:
+        with open(_Constants._DATABASE_NAME, 'rb') as _:
+            _Constants._DATASET = _pickle.load(_)
+    return _Constants._DATASET
+
+if __name__ == '__main__':
+    from pprint import pprint as _pprint
+    from timeit import default_timer as _default_timer
+    
+    print(">>> get_utility()")
+    
+    start_time = _default_timer()
+    result = get_utility()
+    print("Time taken: {}".format(_default_timer() - start_time))
+    _pprint(result[0])

requirements.txt

@@ -1,3 +1,5 @@
-altair
 pandas
-streamlit
+plotly
+scipy
+streamlit
+kaleido

src/streamlit_app.py

@@ -1,40 +0,0 @@
-import altair as alt
-import numpy as np
-import pandas as pd
-import streamlit as st
-
-"""
-# Welcome to Streamlit!
-
-Edit `/streamlit_app.py` to customize this app to your heart's desire :heart:.
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
-
-In the meantime, below is an example of what you can do with just a few lines of code:
-"""
-
-num_points = st.slider("Number of points in spiral", 1, 10000, 1100)
-num_turns = st.slider("Number of turns in spiral", 1, 300, 31)
-
-indices = np.linspace(0, 1, num_points)
-theta = 2 * np.pi * num_turns * indices
-radius = indices
-
-x = radius * np.cos(theta)
-y = radius * np.sin(theta)
-
-df = pd.DataFrame({
-    "x": x,
-    "y": y,
-    "idx": indices,
-    "rand": np.random.randn(num_points),
-})
-
-st.altair_chart(alt.Chart(df, height=700, width=700)
-    .mark_point(filled=True)
-    .encode(
-        x=alt.X("x", axis=None),
-        y=alt.Y("y", axis=None),
-        color=alt.Color("idx", legend=None, scale=alt.Scale()),
-        size=alt.Size("rand", legend=None, scale=alt.Scale(range=[1, 150])),
-    ))

src/util/data_util.py

@@ -0,0 +1,129 @@
+"""Functions to clean and filter data for use in utility data visualizations"""
+
+import pandas as pd
+
+HOURS_PER_YEAR = 8760
+
+
+def get_state_variance(df: pd.DataFrame) -> pd.DataFrame:
+    """Summarizes key metrics for states to help choose state for analysis"""
+
+    agg_df = df.groupby('Utility.State').agg({
+        'Utility.Name': 'nunique',                # Want = More utilites to analyze
+        'Utility.Type': 'nunique',                # Want = More utility types to compare
+        'ResidentialUnitPrice': 'std',            # Want = High std for varied utilities
+        'SystemLossPercentage': 'max',            # Want = Great outlier stories
+        'IndustrialRevenueRatio': 'mean'          # Want = High for industrial bias
+    })
+
+    agg_df.rename(columns={
+        'Utility.Name': '# Utilities',
+        'Utility.Type': '# Utility Types',
+        'Utility.State': 'State',
+        'ResidentialUnitPrice': 'Residential Price Std. Dev.',
+        'SystemLossPercentage': 'System Loss %',
+        'IndustrialRevenueRatio': 'Industrial Revenue %'
+    }, inplace=True)
+
+    agg_df.columns = [f"{col[0]}_{col[1]}" if isinstance(col, tuple) else col
+                      for col in agg_df.columns.values]
+    agg_df.sort_values('Residential Price Std. Dev.',
+                       ascending=False, inplace=True)
+
+    return agg_df.reset_index().head(10)
+
+
+def get_state_data(state: str, df: pd.DataFrame) -> pd.DataFrame:
+    """Select and filter out relevant columns for analysis"""
+    keep_columns = ["Utility.Name", "Utility.State", "Utility.Type",
+                    "Sources.Total", "Sources.Generation", "Sources.Purchased",
+                    "Sources.Other", "Retail.Residential.Revenue", "Retail.Residential.Sales",
+                    "Retail.Residential.Customers", "Retail.Industrial.Revenue",
+                    "Retail.Industrial.Sales", "Retail.Industrial.Customers",
+                    "Uses.Retail", "Uses.Losses", "Uses.Resale",
+                    "Uses.No Charge", "Uses.Consumed", "Uses.Total",
+                    "Demand.Summer Peak", "Revenues.Retail"]
+
+    return df[df["Utility.State"] == state][keep_columns].copy()
+
+
+def prepare_data(df: pd.DataFrame) -> pd.DataFrame:
+    """Perform calculations for key metrics and add them to the data"""
+    # Residential $ per MWh
+    df['ResidentialUnitPrice'] = (df['Retail.Residential.Revenue']
+                                  / df['Retail.Residential.Sales']) * 1000
+    df['ResidentialUnitPrice'] = df['ResidentialUnitPrice'].fillna(0)
+
+    # Industrial $ per MWh
+    df['IndustrialUnitPrice'] = df['Retail.Industrial.Revenue'] / \
+        df['Retail.Industrial.Sales']
+    df['IndustrialUnitPrice'] = df['IndustrialUnitPrice'].fillna(0)
+
+    # % Dependency on industrial revenue
+    df['IndustrialRevenueRatio'] = df['Retail.Industrial.Revenue'] / \
+        df['Revenues.Retail'] * 100
+    df['IndustrialRevenueRatio'] = df['IndustrialRevenueRatio'].fillna(0)
+
+    # Equity Metric
+    df['PriceSpread'] = df['ResidentialUnitPrice'] - df['IndustrialUnitPrice']
+
+    # Efficiency Metric
+    df['SystemLossPercentage'] = (
+        df['Uses.Losses'] / df['Sources.Total']) * 100
+
+    # Operational metric of 'stress' on system
+    df['LoadFactor'] = df['Sources.Total'] / \
+        (df['Demand.Summer Peak'] * HOURS_PER_YEAR)
+    df['LoadFactor'] = df['LoadFactor'].apply(
+        lambda load: 0 if load == float('inf') else load)
+
+    return df
+
+
+def get_customer_utilities(df: pd.DataFrame, sector="Residential") -> pd.DataFrame:
+    """Filter data by customer type for use in plots"""
+    if sector == "Residential":
+        return df[df["Retail.Residential.Customers"] > 0]
+
+    elif sector == "Industrial":
+        return df[df["Retail.Industrial.Customers"] > 0]
+
+    return df[(df["Retail.Residential.Customers"] > 0)
+              & (df["Retail.Industrial.Customers"] > 0)]
+
+
+def get_residential_load_factor(df: pd.DataFrame) -> pd.DataFrame:
+    """Filter data by customer type and utilities with a load factor"""
+    df = get_customer_utilities(df, "Residential")
+
+    return df[df["LoadFactor"] > 0]
+
+
+def get_residential_sys_loss(df: pd.DataFrame) -> pd.DataFrame:
+    """Filter data by customer type and utilities with system loss"""
+    df = get_customer_utilities(df, "Residential")
+
+    return df[df["SystemLossPercentage"] > 0]
+
+
+def get_utility_usage(utility: pd.Series, level: str = "State") -> pd.DataFrame:
+    """Create data with percentages of utilty usage within the sankey plot"""
+
+    # Convert raw values to percentages of the 'Total Sources'
+    keys = [
+        'Sources.Generation', 'Sources.Purchased', 'Sources.Other',
+        'Uses.Retail', 'Uses.Resale', 'Uses.Losses',
+        'Uses.Consumed', 'Uses.No Charge'
+    ]
+
+    named_utility = (utility[keys] / utility['Sources.Total']) * 100
+
+    if level == "State":
+        named_utility['Utility.Name'] = "State of " + \
+            utility['Utility.State'][0:2]
+    elif level == "US":
+        named_utility['Utility.Name'] = "United States"
+    else:
+        named_utility['Utility.Name'] = utility['Utility.Name']
+
+    return named_utility

src/util/plot_util.py

@@ -0,0 +1,322 @@
+""" Functions to create and retrieves interactive plotly charts  """
+
+import os
+
+import pandas as pd
+import numpy as np
+
+from scipy.stats import linregress
+
+import plotly.express as px
+import plotly.graph_objects as go
+import plotly.io as pio
+
+from plotly.subplots import make_subplots
+
+
+def get_state_variance_table(df: pd.DataFrame) -> go.Figure:
+    """Retrieve table of states with highest mean residential unit price"""
+    target, row_h, header_h = 7, 35, 50
+
+    fig = go.Figure(go.Table(
+        header=dict(
+            values=list(df.columns),
+            fill_color='#f8f9fa',
+            font=dict(size=12, family="Arial Black",),
+            align='center',
+            height=header_h
+        ),
+        cells=dict(
+            values=df.values.T,
+            fill_color=[['dodgerblue' if i ==
+                         target else 'white' for i in range(10)]],
+            font=dict(
+                color=[['white' if i == target else 'black' for i in range(10)]], size=12),
+            height=row_h, align='center',
+            format=[None, None, None, ".2f", ".2f", ".2f"]
+        )
+    ))
+
+    fig.update_layout(
+        margin=dict(l=5, r=5, t=5, b=5),
+        height=row_h * (len(df) + 2),
+        autosize=False
+    )
+
+    return fig
+
+
+def get_price_spread_strip_plot(df: pd.DataFrame):
+    """
+    Strip plot of Price Spread by ownership model.
+    Shows which ownership types most consistently charge 
+    residential customers more than industrial ones.
+    """
+    fig = px.strip(
+        df[df.PriceSpread > 0],
+        x="Utility.Type",
+        y="PriceSpread",
+        color="Utility.Type",
+        hover_name="Utility.Name",
+        color_discrete_sequence=px.colors.qualitative.Prism,
+        title="<b>Rate Equity by Ownership Model:</b> "
+              "Residential Premium Over Industrial Rates",
+        labels={
+            "Utility.Type": "Type",
+            "PriceSpread": "Residential Premium ($/MWh)",
+        },
+        template="plotly_white"
+    )
+
+    fig.update_layout(showlegend=False)
+
+    return fig
+
+
+def get_key_metrics_corr_matrix(df: pd.DataFrame) -> go.Figure:
+    """Correlation matrix for key analysis metrics"""
+    key_metrics = {
+        'SystemLossPercentage': 'System Loss %',
+        'LoadFactor': 'Load Factor',
+        'IndustrialRevenueRatio': 'Industrial Revenue %',
+        'PriceSpread': 'Price Spread',
+    }
+
+    corr_matrix = df[list(key_metrics.keys())].corr()
+
+    return px.imshow(
+        corr_matrix.round(2),
+        x=list(key_metrics.values()),
+        y=list(key_metrics.values()),
+        color_continuous_scale='mint',
+        text_auto=True,
+        aspect="auto",
+        title='<b>Statistical Significance:</b> Correlation Heatmap of Key Metrics',
+        labels=dict(color="Score"),
+        template='plotly_white')
+
+
+def add_fairness_trendline(fig: go.Figure, x_data: pd.Series,
+                           y_data: pd.Series, row: int, col: int) -> None:
+    """Calculates OLS and adds centered stats inside the plot to avoid title overlap."""
+    mask = ~np.isnan(x_data) & ~np.isnan(y_data)
+    x_clean, y_clean = x_data[mask], y_data[mask]
+
+    if len(x_clean) > 1:
+        # Get linear regression
+        result = linregress(x_clean, y_clean)
+
+        # Trendline coordinates
+        x_range = np.array([x_clean.min(), x_clean.max()])
+        y_range = result.slope * x_range + result.intercept
+
+        # Add Trendline
+        fig.add_trace(
+            go.Scatter(
+                x=x_range, y=y_range,
+                mode='lines',
+                line=dict(color='black', width=2, dash='dash'),
+                name='Overall Trend',
+                legendgroup='trendline',
+                showlegend=(row == 1 and col == 1),
+                hoverinfo='skip'
+            ), row=row, col=col)
+
+        # 2. Annotation stats box
+        fig.add_annotation(
+            xref=f"x{col if col > 1 else ''} domain",
+            yref="y domain",
+            x=0.5,              # Horizontal center
+            y=0.92,             # Lowered to 92% of height (inside the plot)
+            xanchor="center",
+            yanchor="top",      # Box hangs downward from the y=0.92 point
+            text=f"<b>R²:</b> {result.rvalue**2:.3f}  |  <b>p:</b> {result.pvalue:.4e}",
+            showarrow=False,
+            align="center",
+            # High opacity for readability
+            bgcolor="rgba(255, 255, 255, 0.85)",
+            bordercolor="rgba(0,0,0,0.3)",
+            borderwidth=1,
+            font=dict(size=10))
+
+
+def get_fairness_dual_y_scatter_plot(df: pd.DataFrame) -> go.Figure:
+    """Get dual y-axis scatter plot of utility fairness metrics"""
+    fig = make_subplots(
+        rows=1, cols=2,
+        shared_yaxes=True,
+        horizontal_spacing=0.05,
+        subplot_titles=('<b>System Loss vs Price</b>',
+                        '<b>Load Factor vs Price</b>'))
+
+    df['BubbleSize'] = np.log1p(df['Retail.Residential.Customers'])
+
+    colors = px.colors.qualitative.Prism
+    types = df['Utility.Type'].unique()
+    color_map = {t: colors[i % len(colors)] for i, t in enumerate(types)}
+
+    # Plot 1: System Loss
+    for t in types:
+        mask = df['Utility.Type'] == t
+        fig.add_trace(
+            go.Scatter(
+                x=df[mask]['SystemLossPercentage'], y=df[mask]['ResidentialUnitPrice'],
+                name=t, hovertext=df[mask]['Utility.Name'], mode='markers',
+                marker=dict(color=color_map[t],
+                            size=df[mask]['BubbleSize']),
+                hovertemplate="<b>%{hovertext}</b><br>Loss: %{x}%<br>Price: $%{y}<extra></extra>",
+                showlegend=True), row=1, col=1)
+    add_fairness_trendline(
+        fig, df['SystemLossPercentage'], df['ResidentialUnitPrice'], 1, 1)
+
+    # Plot 2: Load Factor
+    for t in types:
+        mask = df['Utility.Type'] == t
+        fig.add_trace(
+            go.Scatter(
+                x=df[mask]['LoadFactor'], y=df[mask]['ResidentialUnitPrice'], name=t,
+                mode='markers', marker=dict(color=color_map[t], size=df[mask]['BubbleSize']),
+                hovertext=df[mask]['Utility.Name'],
+                hovertemplate="<b>%{hovertext}</b><br>Load: %{x}<br>Price: $%{y}<extra></extra>",
+                showlegend=False), row=1, col=2)
+    add_fairness_trendline(fig, df['LoadFactor'],
+                           df['ResidentialUnitPrice'], 1, 2)
+
+    fig.update_layout(
+        template='plotly_white',
+        title_text='<b>Fairness Audit:</b> Correlation of Utility Metrics to Residential Price',
+        legend_title_text="Ownership Model", height=600)
+
+    fig.update_yaxes(title_text='Residential Price ($/MWh)', row=1, col=1)
+    fig.update_xaxes(title_text='System Energy Loss (%)', row=1, col=1)
+    fig.update_xaxes(title_text='Load Factor', row=1, col=2)
+
+    return fig
+
+
+def get_rate_disparity_dumbbell_plot(df: pd.DataFrame, top_n: int = 10) -> go.Figure:
+    """Get dumbbell plot of highest disparities between industrial/residential rates"""
+    # Sort by spread to show the most "unfair" utilities at the top
+    df_sorted = df[df.PriceSpread > 0].sort_values(
+        'PriceSpread', ascending=True).tail(top_n)
+
+    fig = go.Figure()
+
+    # Add lines connecting the dots
+    for i, row in df_sorted.iterrows():
+        fig.add_shape(
+            type='line', x0=row['IndustrialUnitPrice'], x1=row['ResidentialUnitPrice'],
+            y0=row['Utility.Name'], y1=row['Utility.Name'],
+            line=dict(color='lightgrey', width=2))
+
+    # Industrial dumbbells
+    fig.add_trace(go.Scatter(
+        x=df_sorted['IndustrialUnitPrice'], y=df_sorted['Utility.Name'],
+        mode='markers', name='Industrial Rate', marker=dict(color='#1f77b4', size=10)))
+
+    # Residential dumbbells
+    fig.add_trace(go.Scatter(
+        x=df_sorted['ResidentialUnitPrice'], y=df_sorted['Utility.Name'],
+        mode='markers', name='Residential Rate', marker=dict(color='#d62728', size=10)))
+
+    fig.update_layout(title="<b>Top Rate Disparites</b>",
+                      xaxis_title="Rate ($/MWh)", yaxis_title="")
+    return fig
+
+
+def add_utility_dropdown(fig: go.Figure, df: pd.DataFrame) -> go.Figure:
+    """Post-processing function to add a utility dropdown justified right."""
+    buttons = []
+
+    for _, r in df.iterrows():
+        buttons.append(dict(
+            method="update",
+            label=r["Utility.Name"],
+            args=[
+                {"link.value": [[
+                    r["Sources.Generation"], r["Sources.Purchased"], r["Sources.Other"],
+                    r["Uses.Retail"], r["Uses.Resale"], r["Uses.Losses"],
+                    r["Uses.Consumed"], r["Uses.No Charge"]
+                ]]},
+                {"title.text": f"<b>Energy Flow: </b>{r['Utility.Name']}"}
+            ]
+        ))
+
+    first_row = df.iloc[0]
+    initial_values = [
+        first_row["Sources.Generation"], first_row["Sources.Purchased"], first_row["Sources.Other"],
+        first_row["Uses.Retail"], first_row["Uses.Resale"], first_row["Uses.Losses"],
+        first_row["Uses.Consumed"], first_row["Uses.No Charge"]
+    ]
+
+    # 2. Directly assign intial values to the intial Sankey
+    fig.data[0].link.value = initial_values
+
+    # 3. Apply the layout and the dropdown menu
+    fig.update_layout(
+        title_text=f"<b>Energy Flow: </b>{first_row['Utility.Name']}",
+        updatemenus=[dict(
+            buttons=buttons,
+            direction="down",
+            showactive=True,
+            x=1.0,
+            xanchor="right",
+            y=2,
+            yanchor="top",
+            active=0
+        )],
+    )
+
+    return fig
+
+
+def get_energy_use_sankey_plot(row: pd.DataFrame) -> go.Figure:
+    """Get energy usage sankey plot"""
+    labels = ["Generated", "Purchased", "Other", "Uses", "Retail Sales",
+              "Resale", "Losses", "Consumed", "No Charge"]
+
+    fig = go.Figure(data=[go.Sankey(
+        valueformat=".1f",
+        valuesuffix="%",
+        node=dict(
+            label=labels,
+            color=px.colors.qualitative.Prism),
+        link=dict(
+            source=[0, 1, 2, 3, 3, 3, 3, 3],
+            target=[3, 3, 3, 4, 5, 6, 7, 8],
+            value=[
+                row["Sources.Generation"],
+                row["Sources.Purchased"],
+                row["Sources.Other"],
+                row["Uses.Retail"],
+                row["Uses.Resale"],
+                row["Uses.Losses"],
+                row["Uses.Consumed"],
+                row["Uses.No Charge"]
+            ],
+        ))])
+
+    fig.update_layout(
+        title_text=f"<b>Energy Flow: </b>{row['Utility.Name']}",
+        hovermode='x')
+
+    return fig
+
+
+def export_plots_as_svg(plots: list[go.Figure]) -> None:
+    """Export plots as high-definition SVGs to the 'images' folder"""
+
+    script_dir = os.path.dirname(os.path.abspath(__file__))
+    target_dir = os.path.join(script_dir, "..", "..", "images")
+
+    if not os.path.exists(target_dir):
+        os.makedirs(target_dir)
+
+    pio.write_images(fig=plots,
+                     file=["images/top_ten_state_res_variance_table.svg",
+                           "images/utility_type_strip_plot.svg",
+                           "images/key_metrics_corr_heatmap.svg",
+                           "images/rate_fairness_dual_y_scatter_plot.svg",
+                           "images/rate_disparity_dumbbell_plot.svg",
+                           "images/energy_usage_ny_sankey_chart.svg",
+                           "images/energy_usage_us_sankey_chart.svg"])

streamlit_app.py

@@ -0,0 +1,121 @@
+import streamlit.src.util.plot_util as plot_util
+import streamlit.src.util.data_util as data_util
+import data.electricity as electricity
+
+import streamlit as st
+import pandas as pd
+
+
+st.set_page_config(
+    page_title="Utility Efficiency & Rates",
+    layout="wide"
+)
+
+
+@st.cache_data
+def load_data(state: str) -> pd.DataFrame:
+    utility = electricity.get_utility()
+    df = pd.json_normalize(utility)
+    state_df = data_util.get_state_data(state, df)
+    return data_util.prepare_data(state_df), df
+
+
+with st.sidebar:
+    state = st.selectbox(
+        "Select State",
+        options=["NY", "AK", "RI", "ME", "CA", "NJ", "CT", "NH", "MA", "AZ"],
+        index=0
+    )
+    st.markdown("---")
+    st.markdown(
+        "This app explores whether operational inefficiencies "
+        "— energy losses and poor load factors — correlate with "
+        "higher residential electricity rates."
+    )
+    st.markdown(
+        "[View on GitHub](https://github.com/chalseokorom/utilities-equity-efficiency-gap)")
+
+state_df, full_df = load_data(state)
+
+st.title("Electricity Utility Fairness Residential Rate Analysis")
+st.caption(
+    f"Exploring {state} utilities — "
+    f"{len(state_df)} utilities across "
+    f"{state_df['Utility.Type'].nunique()} ownership models"
+)
+st.divider()
+
+# ── Section 1: Fairness Audit ─────────────────────────────────
+st.header("Fairness Audit — Efficiency vs. Residential Price")
+st.caption(
+    "Do utilities with higher energy losses or lower load factors "
+    "charge residential customers more per MWh?"
+)
+
+scatter_df = data_util.get_residential_sys_loss(state_df)
+scatter_df = data_util.get_residential_load_factor(scatter_df).round(2)
+
+st.plotly_chart(
+    plot_util.get_fairness_dual_y_scatter_plot(scatter_df),
+    use_container_width=True
+)
+st.divider()
+
+# ── Section 2: Ownership Models ───────────────────────────────
+st.header("Ownership Model — Price Spread")
+
+st.plotly_chart(
+    plot_util.get_price_spread_strip_plot(state_df.round(2)),
+    use_container_width=True
+)
+st.divider()
+
+# ── Section 3: Rate Disparity ─────────────────────────────────
+st.header("Rate Disparity — Residential vs. Industrial")
+
+n_utilities = st.slider(
+    "Number of utilities to show",
+    min_value=5, max_value=20, value=10, step=1
+)
+
+both_df = data_util.get_customer_utilities(state_df, sector="Both").round(2)
+st.plotly_chart(
+    plot_util.get_rate_disparity_dumbbell_plot(both_df, top_n=n_utilities),
+    use_container_width=True
+)
+st.divider()
+
+# ── Section 4: Energy Flow ────────────────────────────────────
+st.header("Energy Flow — Sources & Uses")
+# Row 1: State aggregate
+st.subheader(f"{state} — Aggregate Energy Flow")
+st.caption(
+    "How all utilities in this state collectively source and distribute energy.")
+
+numeric_sum = state_df.select_dtypes(include='number').sum()
+numeric_sum['Utility.Name'] = state
+state_flow = data_util.get_utility_usage(numeric_sum, level="US")
+
+st.plotly_chart(
+    plot_util.get_energy_use_sankey_plot(state_flow),
+    use_container_width=True
+)
+
+st.divider()
+
+# Row 2: Individual utility explorer
+st.subheader("Individual Utility — Energy Flow")
+st.caption(
+    "Select a utility to see its specific energy breakdown. "
+    "Compare the Losses band against the state aggregate above."
+)
+
+utility_names = sorted(state_df['Utility.Name'].dropna().unique())
+selected_utility = st.selectbox("Select a utility", options=utility_names)
+utility_row = state_df[state_df['Utility.Name'] == selected_utility].iloc[0]
+utility_flow = data_util.get_utility_usage(utility_row, level="Utility")
+
+st.plotly_chart(
+    plot_util.get_energy_use_sankey_plot(utility_flow),
+    use_container_width=True
+)

utility_efficiency_fairness.ipynb

@@ -0,0 +1,1622 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "606940eb",
+   "metadata": {},
+   "source": [
+    "# Electricity Utility Fairness Residential Rate Analysis\n",
+    "\n",
+    "This notebook investigates a fundamental fairness question in the U.S. electricity sector: **do residential customers end up paying more when their utility operates inefficiently?**\n",
+    "\n",
+    "Using the [CORGIS Electricity Dataset](https://corgis-edu.github.io/corgis/python/electricity/) — a structured derivative of the U.S. EIA Form 861 survey — the analysis engineers a set of efficiency and equity metrics, then examines their statistical relationships through a suite of interactive visualizations. New York State serves as the primary case study."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fe53b9ab",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 1. Setup\n",
+    "- `electricity` is the CORGIS data loader.\n",
+    "<br>\n",
+    "- `data_util` handles all data preparation and filtering. \n",
+    "- `plot_util` contains every chart-building function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "e862a1c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "import data.electricity as electricity\n",
+    "\n",
+    "import src.util.plot_util as plot_util\n",
+    "import src.util.data_util as data_util"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "549a099a",
+   "metadata": {},
+   "source": [
+    "### 1.1 Data Loading & Metric Engineering\n",
+    "\n",
+    "The raw dataset is a list of nested dictionaries — one record per utility. `prepare_data` engineers the following derived metrics from the raw columns:\n",
+    "\n",
+    "| Metric | Formula | Interpretation |\n",
+    "|---|---|---|\n",
+    "| `Residential Unit Price` | `(Residential Revenue / Residential Sales) × 1000` | Residential rate in $/MWh |\n",
+    "| `Industrial Unit Price` | `(Industrial Revenue / Industrial Sales) × 1000` | Industrial rate in $/MWh |\n",
+    "| `Industrial Revenue Ratio` | `(Industrial Revenue / Total Retail Revenue) × 100` | % of a utility's revenue from industrial customers |\n",
+    "| `Price Spread` | `Residential Unit Price − Industrial Unit Price` | Pricing gap between Residential and Industrial customers |\n",
+    "| `System Loss Percentage` | `(Energy Losses / Total Energy) × 100` | Energy lost in transmission/distribution as % of supply |\n",
+    "| `Load Factor` | `Total Energy / (Summer Peak Demand × Hours Per Year)` | Utilization efficiency — how fully the system capacity is used |\n",
+    "\n",
+    "**Note on LoadFactor:**\n",
+    " A value of 1.0 means the utility delivers energy at exactly its peak demand capacity around the clock — theoretically perfect utilization. Real utilities typically range from 0.3–0.7. A low load factor suggests underused infrastructure whose fixed costs are spread across fewer delivered MWh."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "6b7d5309",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieve entire US-based dataset\n",
+    "utility = electricity.get_utility()\n",
+    "df = pd.json_normalize(utility)\n",
+    "\n",
+    "# Limit to the state of New York and relevant columns\n",
+    "ny_df = data_util.get_state_data('NY', df)\n",
+    "\n",
+    "# Add key metrics for use in later plots\n",
+    "ny_df = data_util.prepare_data(ny_df)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7327b62c",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 2. State Selection — Why New York?\n",
+    "\n",
+    "Rather than choosing a state arbitrarily, I created use `get_state_variance` to rank all 50 states across five dimensions that could make for a rich case study:\n",
+    "\n",
+    "|  | Why it matters |\n",
+    "|---|---|\n",
+    "| **# Utilities** | More utilities = more data points and statistical power |\n",
+    "| **# Utility Types** | More ownership models = richer segmentation and comparison |\n",
+    "| **Residential Price Std. Dev.** | High spread = real price variation to explain, not a flat market |\n",
+    "| **Max System Loss %** | Outlier utilities = compelling examples for the core hypothesis |\n",
+    "| **Industrial Revenue %** | High industrial dependency = cross-subsidization dynamics worth examining |\n",
+    "\n",
+    "The table below highlights the top 10 states by residential price variance. **New York (highlighted)** ranks highly because it uniquely satisfies *all five* criteria simultaneously."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "ad5df068",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "state_variance = data_util.get_state_variance(data_util.prepare_data(df))\n",
+    "\n",
+    "top_variance_table = plot_util.get_state_variance_table(state_variance)\n",
+    "\n",
+    "# top_variance_table.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3455bc7c",
+   "metadata": {},
+   "source": [
+    "### Why New York specifically?\n",
+    "\n",
+    "Beyond the rankings, New York offers several structural advantages for this analysis:\n",
+    "\n",
+    "- **6 distinct utility ownership types** — more than almost any other state, enabling meaningful cross-model comparisons within a single regulatory environment.\n",
+    "- **100+ individual utilities** — sufficient sample size to draw statistically meaningful conclusions.\n",
+    "- **Extreme geographic diversity** — Con Edison serves dense urban Manhattan while small rural cooperatives serve upstate communities. This natural variation creates a richer distribution of efficiency and pricing outcomes than a more homogeneous state would.\n",
+    "- **High system loss outliers** — NY's transmission infrastructure includes some of the oldest urban grid segments in the country, producing the high-loss edge cases that make the core hypothesis testable.\n",
+    "- **Active regulatory environment** — the NY Public Service Commission is one of the most scrutinized utility regulators in the U.S., meaning this analysis connects directly to real-world policy questions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1992b6ef",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 3. Ownership Model Analysis\n",
+    "\n",
+    "Before testing the efficiency hypothesis, we establish baseline pricing distributions by ownership model. This matters because ownership type is a structural variable — investor-owned utilities (IOUs) answer to shareholders and profit motives, while cooperatives and municipals serve member-owners or public ratepayers. If ownership type already explains most of the price variance, efficiency metrics may be redundant."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "54f2b308",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.plotly.v1+json": {
+       "config": {
+        "plotlyServerURL": "https://plot.ly"
+       },
+       "data": [
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Agway Energy Services, LLC",
+          "Energy Coop of New York, Inc",
+          "Robison Energy, LLC",
+          "U.S. Energy Partners LLC",
+          "NOCO Electric",
+          "Columbia Utilities Power, LLC",
+          "Hudson Energy Services",
+          "Palmco Power NJ, LLC",
+          "Major Energy Electric Services",
+          "Palmco Power PA, LLC",
+          "Palmco Power MD, LLC",
+          "Plymouth Rock Energy, LLC",
+          "Palmco Power OH, LLC",
+          "Stream Energy New York, LLC",
+          "EnergyMark, LLC",
+          "Palmco Power IL, LLC",
+          "Respond Power LLC",
+          "Kiwi Energy NY LLC",
+          "SmartEnergy Holdings, LLC",
+          "Alpha Gas and Electric, LLC",
+          "Energy.Me Midwest LLC",
+          "Marathon Power LLC",
+          "Harborside Energy, LLC",
+          "Abest Power & Gas, LLC",
+          "MPower Energy LLC",
+          "Palmco Power MA, LLC",
+          "Agera Energy LLC",
+          "Greenlight Energy Inc.",
+          "New Wave Energy Corporation",
+          "HIKO Energy, LLC",
+          "Aequitas Energy Inc",
+          "Great Eastern Energy",
+          "East Coast Power & Gas, LLC",
+          "Spring Energy RRH LLC dba Spring Power &",
+          "MPower Energy NJ LLC",
+          "Atlantic Energy LLC",
+          "Palmco Power DC, LLC",
+          "Palmco Power DE, LLC",
+          "South Bay Energy Corp.",
+          "Flanders Energy, LLC",
+          "Pure Energy USA, LLC",
+          "First Choice Energy LLC",
+          "Agressive Energy LLC"
+         ],
+         "legendgroup": "Retail Power Marketer",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(95, 70, 144)"
+         },
+         "name": "Retail Power Marketer",
+         "offsetgroup": "Retail Power Marketer",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer",
+          "Retail Power Marketer"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "1HdJqZaIVEDlcPM237NFQJ4t+TeSfVVA9Y6hQZjxREC7n0AeSF5FQAJayxan5mdA+NB2EXoPWEDkHoKBHplrQAf1wZiCkFxAdfEDDupzYUBcKbC7K7hjQNg9bETnNGBAwv1PMJqEXEBzGFLAfrxQQMb9Fc3b/0VA4ydnH2Q/VkBOiMxbaUtiQE1aVCBa8FtAdyZPlzdMWUD/BbayJZdaQNFZ0CR2H1NAVaW6iXyDUkAWCcnsvT5kQL6sGTMR6VRAOFE8FqijYEDwxWACoKxkQLReHZWsKFhANJ/x5rw1XUAUtkGpzFhMQEG4G/seUWFA7ZpZKnJNXEC+qlbjMDJZQMjcovLsXVRAJYv3tlpiWkAv2u+X74BeQM7YHwYLZ2BAFkOHeY7iW0DXGD4bRKpZQNjxRK4pNV9AnhCBas4qZEATgY2Ufz1jQNvxsf19yVhAbl5aGi50UUA=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        },
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Village of Arcade - (NY)",
+          "Village of Akron - (NY)",
+          "Village of Andover - (NY)",
+          "Village of Angelica - (NY)",
+          "Bath Electric Gas & Water",
+          "Village of Bergen - (NY)",
+          "Village of Boonville - (NY)",
+          "Village of Brocton - (NY)",
+          "Village of Castile - (NY)",
+          "Village of Churchville - (NY)",
+          "Village of Endicott - (NY)",
+          "Village of Fairport - (NY)",
+          "Village of Frankfort - (NY)",
+          "Village of Freeport - (NY)",
+          "Village of Green Island - (NY)",
+          "Village of Greene - (NY)",
+          "Village of Groton - (NY)",
+          "Village of Hamilton - (NY)",
+          "Village of Holley - (NY)",
+          "Village of Ilion - (NY)",
+          "Jamestown Board of Public Util",
+          "Lake Placid Village, Inc - (NY)",
+          "Village of Little Valley - (NY)",
+          "Village of Marathon - (NY)",
+          "Town of Massena - (NY)",
+          "Village of Mayville - (NY)",
+          "Mohawk Municipal Comm",
+          "Village of Penn Yan - (NY)",
+          "Village of Philadelphia - (NY)",
+          "City of Plattsburgh - (NY)",
+          "Village of Richmondville - (NY)",
+          "Village of Rockville Centre - (NY)",
+          "Village of Rouses Point - (NY)",
+          "City of Salamanca - (NY)",
+          "Village of Sherburne - (NY)",
+          "City of Sherrill - (NY)",
+          "Village of Silver Springs - (NY)",
+          "Village of Skaneateles - (NY)",
+          "Village of Solvay - (NY)",
+          "Village of Spencerport - (NY)",
+          "Village of Springville - (NY)",
+          "Village of Theresa - (NY)",
+          "Village of Tupper Lake - (NY)",
+          "Village of Watkins Glen - (NY)",
+          "Village of Wellsville - (NY)",
+          "Village of Westfield - (NY)"
+         ],
+         "legendgroup": "Municipal",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(29, 105, 150)"
+         },
+         "name": "Municipal",
+         "offsetgroup": "Municipal",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal",
+          "Municipal"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "4zx0l/glSkCrn6PKMxZKQDFIRJo2x0VAGOWbWs1/T0CCUB+yIaJNQEZpzEOSWVNAykfneWTjSUC5RSoUu+tGQD5k23fSFk1Azmj3sPXNUkDJCaumGFlQQBb2vzXcvktA7+AqSao3Q0CQqdv9nMlfQC8zhjbsPltAXfLkhENnS0DUHfrfiDZKQJdz0sw7pkhAl5Nt4wsgTkCCI8bcBMRKQNIB9qAznlRAw5X6vW53S0At0Q+nexBTQCjVNgwQnVFASdVlz85LTkAsdSyDtF5QQNhzo/V/JkVArLuSbKG6QkCXOXo4ck9OQMfHf05/EUdAK8FbUIDWUkDSX/M3ctteQF6gAPx8Ej1Aw45Qct0fSUC0jal315FLQNRIT1cjHU1AjVKkDWvMUUDTH1W7sehEQAeqFkKflkxAe68yr+vZRkBpZcd2fvFKQMx1qVre1FNApaDbJmbZSkCWccuMD2tCQNSJR5YBWU5ARDUJMx5yTEA=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        },
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Central Hudson Gas & Elec Corp",
+          "Consolidated Edison Co-NY Inc",
+          "Fishers Island Utility Co Inc",
+          "Massachusetts Electric Co",
+          "New York State Elec & Gas Corp",
+          "Niagara Mohawk Power Corp.",
+          "Orange & Rockland Utils Inc",
+          "Pike County Light & Power Co",
+          "Rochester Gas & Electric Corp",
+          "Rockland Electric Co"
+         ],
+         "legendgroup": "Investor Owned",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(56, 166, 165)"
+         },
+         "name": "Investor Owned",
+         "offsetgroup": "Investor Owned",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned",
+          "Investor Owned"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "DlfHeglgZEAky8QEPyJtQOHh4eHh4XNAiNVwftXYZkBAg+xCRm5aQLlstvUwUV1AP0jm2o/iZEC7M68wZX1eQCRaHECiYF5AzRC7X0JqY0A=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        },
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Delaware County Elec Coop Inc",
+          "North Shore Towers Apts Inc",
+          "Otsego Electric Coop, Inc",
+          "Steuben Rural Elec Coop, Inc"
+         ],
+         "legendgroup": "Cooperative",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(15, 133, 84)"
+         },
+         "name": "Cooperative",
+         "offsetgroup": "Cooperative",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "Cooperative",
+          "Cooperative",
+          "Cooperative",
+          "Cooperative"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "+aWVpF9ZYUDMuXx9SDRxQE3e4teNEl9Aw0BzNP3fXkA=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        },
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Long Island Power Authority"
+         ],
+         "legendgroup": "State",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(115, 175, 72)"
+         },
+         "name": "State",
+         "offsetgroup": "State",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "State"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "WOYOiWF7aUA=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        },
+        {
+         "alignmentgroup": "True",
+         "boxpoints": "all",
+         "fillcolor": "rgba(255,255,255,0)",
+         "hoveron": "points",
+         "hovertemplate": "<b>%{hovertext}</b><br><br>Type=%{x}<br>Residential Premium ($/MWh)=%{y}<extra></extra>",
+         "hovertext": [
+          "Greenbacker Renewable Energy Corporation"
+         ],
+         "legendgroup": "Behind the Meter",
+         "line": {
+          "color": "rgba(255,255,255,0)"
+         },
+         "marker": {
+          "color": "rgb(237, 173, 8)"
+         },
+         "name": "Behind the Meter",
+         "offsetgroup": "Behind the Meter",
+         "orientation": "v",
+         "pointpos": 0,
+         "showlegend": true,
+         "type": "box",
+         "x": [
+          "Behind the Meter"
+         ],
+         "x0": " ",
+         "xaxis": "x",
+         "y": {
+          "bdata": "tp977y/kYkA=",
+          "dtype": "f8"
+         },
+         "y0": " ",
+         "yaxis": "y"
+        }
+       ],
+       "layout": {
+        "boxmode": "overlay",
+        "legend": {
+         "title": {
+          "text": "Type"
+         },
+         "tracegroupgap": 0
+        },
+        "showlegend": false,
+        "template": {
+         "data": {
+          "bar": [
+           {
+            "error_x": {
+             "color": "#2a3f5f"
+            },
+            "error_y": {
+             "color": "#2a3f5f"
+            },
+            "marker": {
+             "line": {
+              "color": "white",
+              "width": 0.5
+             },
+             "pattern": {
+              "fillmode": "overlay",
+              "size": 10,
+              "solidity": 0.2
+             }
+            },
+            "type": "bar"
+           }
+          ],
+          "barpolar": [
+           {
+            "marker": {
+             "line": {
+              "color": "white",
+              "width": 0.5
+             },
+             "pattern": {
+              "fillmode": "overlay",
+              "size": 10,
+              "solidity": 0.2
+             }
+            },
+            "type": "barpolar"
+           }
+          ],
+          "carpet": [
+           {
+            "aaxis": {
+             "endlinecolor": "#2a3f5f",
+             "gridcolor": "#C8D4E3",
+             "linecolor": "#C8D4E3",
+             "minorgridcolor": "#C8D4E3",
+             "startlinecolor": "#2a3f5f"
+            },
+            "baxis": {
+             "endlinecolor": "#2a3f5f",
+             "gridcolor": "#C8D4E3",
+             "linecolor": "#C8D4E3",
+             "minorgridcolor": "#C8D4E3",
+             "startlinecolor": "#2a3f5f"
+            },
+            "type": "carpet"
+           }
+          ],
+          "choropleth": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "type": "choropleth"
+           }
+          ],
+          "contour": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "colorscale": [
+             [
+              0,
+              "#0d0887"
+             ],
+             [
+              0.1111111111111111,
+              "#46039f"
+             ],
+             [
+              0.2222222222222222,
+              "#7201a8"
+             ],
+             [
+              0.3333333333333333,
+              "#9c179e"
+             ],
+             [
+              0.4444444444444444,
+              "#bd3786"
+             ],
+             [
+              0.5555555555555556,
+              "#d8576b"
+             ],
+             [
+              0.6666666666666666,
+              "#ed7953"
+             ],
+             [
+              0.7777777777777778,
+              "#fb9f3a"
+             ],
+             [
+              0.8888888888888888,
+              "#fdca26"
+             ],
+             [
+              1,
+              "#f0f921"
+             ]
+            ],
+            "type": "contour"
+           }
+          ],
+          "contourcarpet": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "type": "contourcarpet"
+           }
+          ],
+          "heatmap": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "colorscale": [
+             [
+              0,
+              "#0d0887"
+             ],
+             [
+              0.1111111111111111,
+              "#46039f"
+             ],
+             [
+              0.2222222222222222,
+              "#7201a8"
+             ],
+             [
+              0.3333333333333333,
+              "#9c179e"
+             ],
+             [
+              0.4444444444444444,
+              "#bd3786"
+             ],
+             [
+              0.5555555555555556,
+              "#d8576b"
+             ],
+             [
+              0.6666666666666666,
+              "#ed7953"
+             ],
+             [
+              0.7777777777777778,
+              "#fb9f3a"
+             ],
+             [
+              0.8888888888888888,
+              "#fdca26"
+             ],
+             [
+              1,
+              "#f0f921"
+             ]
+            ],
+            "type": "heatmap"
+           }
+          ],
+          "histogram": [
+           {
+            "marker": {
+             "pattern": {
+              "fillmode": "overlay",
+              "size": 10,
+              "solidity": 0.2
+             }
+            },
+            "type": "histogram"
+           }
+          ],
+          "histogram2d": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "colorscale": [
+             [
+              0,
+              "#0d0887"
+             ],
+             [
+              0.1111111111111111,
+              "#46039f"
+             ],
+             [
+              0.2222222222222222,
+              "#7201a8"
+             ],
+             [
+              0.3333333333333333,
+              "#9c179e"
+             ],
+             [
+              0.4444444444444444,
+              "#bd3786"
+             ],
+             [
+              0.5555555555555556,
+              "#d8576b"
+             ],
+             [
+              0.6666666666666666,
+              "#ed7953"
+             ],
+             [
+              0.7777777777777778,
+              "#fb9f3a"
+             ],
+             [
+              0.8888888888888888,
+              "#fdca26"
+             ],
+             [
+              1,
+              "#f0f921"
+             ]
+            ],
+            "type": "histogram2d"
+           }
+          ],
+          "histogram2dcontour": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "colorscale": [
+             [
+              0,
+              "#0d0887"
+             ],
+             [
+              0.1111111111111111,
+              "#46039f"
+             ],
+             [
+              0.2222222222222222,
+              "#7201a8"
+             ],
+             [
+              0.3333333333333333,
+              "#9c179e"
+             ],
+             [
+              0.4444444444444444,
+              "#bd3786"
+             ],
+             [
+              0.5555555555555556,
+              "#d8576b"
+             ],
+             [
+              0.6666666666666666,
+              "#ed7953"
+             ],
+             [
+              0.7777777777777778,
+              "#fb9f3a"
+             ],
+             [
+              0.8888888888888888,
+              "#fdca26"
+             ],
+             [
+              1,
+              "#f0f921"
+             ]
+            ],
+            "type": "histogram2dcontour"
+           }
+          ],
+          "mesh3d": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "type": "mesh3d"
+           }
+          ],
+          "parcoords": [
+           {
+            "line": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "parcoords"
+           }
+          ],
+          "pie": [
+           {
+            "automargin": true,
+            "type": "pie"
+           }
+          ],
+          "scatter": [
+           {
+            "fillpattern": {
+             "fillmode": "overlay",
+             "size": 10,
+             "solidity": 0.2
+            },
+            "type": "scatter"
+           }
+          ],
+          "scatter3d": [
+           {
+            "line": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scatter3d"
+           }
+          ],
+          "scattercarpet": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scattercarpet"
+           }
+          ],
+          "scattergeo": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scattergeo"
+           }
+          ],
+          "scattergl": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scattergl"
+           }
+          ],
+          "scattermap": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scattermap"
+           }
+          ],
+          "scattermapbox": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scattermapbox"
+           }
+          ],
+          "scatterpolar": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scatterpolar"
+           }
+          ],
+          "scatterpolargl": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scatterpolargl"
+           }
+          ],
+          "scatterternary": [
+           {
+            "marker": {
+             "colorbar": {
+              "outlinewidth": 0,
+              "ticks": ""
+             }
+            },
+            "type": "scatterternary"
+           }
+          ],
+          "surface": [
+           {
+            "colorbar": {
+             "outlinewidth": 0,
+             "ticks": ""
+            },
+            "colorscale": [
+             [
+              0,
+              "#0d0887"
+             ],
+             [
+              0.1111111111111111,
+              "#46039f"
+             ],
+             [
+              0.2222222222222222,
+              "#7201a8"
+             ],
+             [
+              0.3333333333333333,
+              "#9c179e"
+             ],
+             [
+              0.4444444444444444,
+              "#bd3786"
+             ],
+             [
+              0.5555555555555556,
+              "#d8576b"
+             ],
+             [
+              0.6666666666666666,
+              "#ed7953"
+             ],
+             [
+              0.7777777777777778,
+              "#fb9f3a"
+             ],
+             [
+              0.8888888888888888,
+              "#fdca26"
+             ],
+             [
+              1,
+              "#f0f921"
+             ]
+            ],
+            "type": "surface"
+           }
+          ],
+          "table": [
+           {
+            "cells": {
+             "fill": {
+              "color": "#EBF0F8"
+             },
+             "line": {
+              "color": "white"
+             }
+            },
+            "header": {
+             "fill": {
+              "color": "#C8D4E3"
+             },
+             "line": {
+              "color": "white"
+             }
+            },
+            "type": "table"
+           }
+          ]
+         },
+         "layout": {
+          "annotationdefaults": {
+           "arrowcolor": "#2a3f5f",
+           "arrowhead": 0,
+           "arrowwidth": 1
+          },
+          "autotypenumbers": "strict",
+          "coloraxis": {
+           "colorbar": {
+            "outlinewidth": 0,
+            "ticks": ""
+           }
+          },
+          "colorscale": {
+           "diverging": [
+            [
+             0,
+             "#8e0152"
+            ],
+            [
+             0.1,
+             "#c51b7d"
+            ],
+            [
+             0.2,
+             "#de77ae"
+            ],
+            [
+             0.3,
+             "#f1b6da"
+            ],
+            [
+             0.4,
+             "#fde0ef"
+            ],
+            [
+             0.5,
+             "#f7f7f7"
+            ],
+            [
+             0.6,
+             "#e6f5d0"
+            ],
+            [
+             0.7,
+             "#b8e186"
+            ],
+            [
+             0.8,
+             "#7fbc41"
+            ],
+            [
+             0.9,
+             "#4d9221"
+            ],
+            [
+             1,
+             "#276419"
+            ]
+           ],
+           "sequential": [
+            [
+             0,
+             "#0d0887"
+            ],
+            [
+             0.1111111111111111,
+             "#46039f"
+            ],
+            [
+             0.2222222222222222,
+             "#7201a8"
+            ],
+            [
+             0.3333333333333333,
+             "#9c179e"
+            ],
+            [
+             0.4444444444444444,
+             "#bd3786"
+            ],
+            [
+             0.5555555555555556,
+             "#d8576b"
+            ],
+            [
+             0.6666666666666666,
+             "#ed7953"
+            ],
+            [
+             0.7777777777777778,
+             "#fb9f3a"
+            ],
+            [
+             0.8888888888888888,
+             "#fdca26"
+            ],
+            [
+             1,
+             "#f0f921"
+            ]
+           ],
+           "sequentialminus": [
+            [
+             0,
+             "#0d0887"
+            ],
+            [
+             0.1111111111111111,
+             "#46039f"
+            ],
+            [
+             0.2222222222222222,
+             "#7201a8"
+            ],
+            [
+             0.3333333333333333,
+             "#9c179e"
+            ],
+            [
+             0.4444444444444444,
+             "#bd3786"
+            ],
+            [
+             0.5555555555555556,
+             "#d8576b"
+            ],
+            [
+             0.6666666666666666,
+             "#ed7953"
+            ],
+            [
+             0.7777777777777778,
+             "#fb9f3a"
+            ],
+            [
+             0.8888888888888888,
+             "#fdca26"
+            ],
+            [
+             1,
+             "#f0f921"
+            ]
+           ]
+          },
+          "colorway": [
+           "#636efa",
+           "#EF553B",
+           "#00cc96",
+           "#ab63fa",
+           "#FFA15A",
+           "#19d3f3",
+           "#FF6692",
+           "#B6E880",
+           "#FF97FF",
+           "#FECB52"
+          ],
+          "font": {
+           "color": "#2a3f5f"
+          },
+          "geo": {
+           "bgcolor": "white",
+           "lakecolor": "white",
+           "landcolor": "white",
+           "showlakes": true,
+           "showland": true,
+           "subunitcolor": "#C8D4E3"
+          },
+          "hoverlabel": {
+           "align": "left"
+          },
+          "hovermode": "closest",
+          "mapbox": {
+           "style": "light"
+          },
+          "paper_bgcolor": "white",
+          "plot_bgcolor": "white",
+          "polar": {
+           "angularaxis": {
+            "gridcolor": "#EBF0F8",
+            "linecolor": "#EBF0F8",
+            "ticks": ""
+           },
+           "bgcolor": "white",
+           "radialaxis": {
+            "gridcolor": "#EBF0F8",
+            "linecolor": "#EBF0F8",
+            "ticks": ""
+           }
+          },
+          "scene": {
+           "xaxis": {
+            "backgroundcolor": "white",
+            "gridcolor": "#DFE8F3",
+            "gridwidth": 2,
+            "linecolor": "#EBF0F8",
+            "showbackground": true,
+            "ticks": "",
+            "zerolinecolor": "#EBF0F8"
+           },
+           "yaxis": {
+            "backgroundcolor": "white",
+            "gridcolor": "#DFE8F3",
+            "gridwidth": 2,
+            "linecolor": "#EBF0F8",
+            "showbackground": true,
+            "ticks": "",
+            "zerolinecolor": "#EBF0F8"
+           },
+           "zaxis": {
+            "backgroundcolor": "white",
+            "gridcolor": "#DFE8F3",
+            "gridwidth": 2,
+            "linecolor": "#EBF0F8",
+            "showbackground": true,
+            "ticks": "",
+            "zerolinecolor": "#EBF0F8"
+           }
+          },
+          "shapedefaults": {
+           "line": {
+            "color": "#2a3f5f"
+           }
+          },
+          "ternary": {
+           "aaxis": {
+            "gridcolor": "#DFE8F3",
+            "linecolor": "#A2B1C6",
+            "ticks": ""
+           },
+           "baxis": {
+            "gridcolor": "#DFE8F3",
+            "linecolor": "#A2B1C6",
+            "ticks": ""
+           },
+           "bgcolor": "white",
+           "caxis": {
+            "gridcolor": "#DFE8F3",
+            "linecolor": "#A2B1C6",
+            "ticks": ""
+           }
+          },
+          "title": {
+           "x": 0.05
+          },
+          "xaxis": {
+           "automargin": true,
+           "gridcolor": "#EBF0F8",
+           "linecolor": "#EBF0F8",
+           "ticks": "",
+           "title": {
+            "standoff": 15
+           },
+           "zerolinecolor": "#EBF0F8",
+           "zerolinewidth": 2
+          },
+          "yaxis": {
+           "automargin": true,
+           "gridcolor": "#EBF0F8",
+           "linecolor": "#EBF0F8",
+           "ticks": "",
+           "title": {
+            "standoff": 15
+           },
+           "zerolinecolor": "#EBF0F8",
+           "zerolinewidth": 2
+          }
+         }
+        },
+        "title": {
+         "text": "<b>Rate Equity by Ownership Model:</b> Residential Premium Over Industrial Rates"
+        },
+        "xaxis": {
+         "anchor": "y",
+         "categoryarray": [
+          "Retail Power Marketer",
+          "Municipal",
+          "Investor Owned",
+          "Cooperative",
+          "State",
+          "Behind the Meter"
+         ],
+         "categoryorder": "array",
+         "domain": [
+          0,
+          1
+         ],
+         "title": {
+          "text": "Type"
+         }
+        },
+        "yaxis": {
+         "anchor": "x",
+         "domain": [
+          0,
+          1
+         ],
+         "title": {
+          "text": "Residential Premium ($/MWh)"
+         }
+        }
+       }
+      }
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "strip_plot = plot_util.get_price_spread_strip_plot(ny_df)\n",
+    "\n",
+    "strip_plot.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "99326360",
+   "metadata": {},
+   "source": [
+    "**Interpretation:** Investor-owned utilities dominate rates on the high end, but the spread within each ownership category is wide enough that ownership type alone doesn't fully explain the disparity. There's meaningful variation within ownership types worth investigating."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "42f0d824",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 4. Correlation Analysis\n",
+    "\n",
+    "Before building directional charts, we compute a Pearson correlation matrix across all key metrics. This serves two purposes: it quantifies the *strength and direction* of every pairwise relationship, and it surfaces any unexpected correlations that warrant further investigation. A positive correlation between `SystemLossPercentage` and `ResidentialUnitPrice` would support the hypothesis. A negative correlation between `LoadFactor` and `ResidentialUnitPrice` (higher efficiency → lower price) would also support it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "6e0480c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Key metrics: System Loss %, Load Factor', Industrial Revenue %, Price Spread\n",
+    "heatmap = plot_util.get_key_metrics_corr_matrix(ny_df)\n",
+    "\n",
+    "# heatmap.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0003dd2d",
+   "metadata": {},
+   "source": [
+    "**Interpretation:** The heatmap gives us a first look at which efficiency metrics are most predictive of residential pricing. The strong negative correlation between `LoadFactor` and both `ResidentialUnitPrice` and PriceSpread stands out immediately — it suggests infrastructure utilization is a meaningful driver of what residential customers pay. `SystemLossPercentage` shows a weaker positive relationship, which we'll test directly in the next section."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c92e34ee",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 5. Fairness Audit — Efficiency vs. Residential Price\n",
+    "\n",
+    "This is the central test of the research question, presented as two side-by-side scatter plots sharing a y-axis (residential price). Each plot approaches inefficiency from a different angle:\n",
+    "\n",
+    "| Plot | X-axis | What it tests |\n",
+    "|---|---|---|\n",
+    "| Left | `SystemLossPercentage` | Does wasted energy in the grid cost residential customers more? |\n",
+    "| Right | `LoadFactor` | Does underutilized infrastructure translate to higher per-MWh costs? |\n",
+    "\n",
+    "> **Filtering note:** This chart excludes utilities with zero system loss or zero load factor. These edge cases typically represent pass-through entities (pure resellers) or data reporting anomalies — including them would distort the OLS fit."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "de2b920d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Keep residential utilities that are using energy (instead of ONLY reseale, etc.)\n",
+    "residential_lf_sys_loss_df = data_util.get_residential_sys_loss(ny_df)\n",
+    "residential_lf_sys_loss_df = data_util.get_residential_load_factor(\n",
+    "    residential_lf_sys_loss_df).round(2)\n",
+    "\n",
+    "dual_y_scatter = plot_util.get_fairness_dual_y_scatter_plot(\n",
+    "    residential_lf_sys_loss_df)\n",
+    "\n",
+    "# dual_y_scatter.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ccdd7bc3",
+   "metadata": {},
+   "source": [
+    "**Interpretation:** The results are asymmetric: load factor explains 41.7% of residential price variance (R² = 0.417, p < 0.001), a statistically strong relationship. System loss explains 10.8% (R² = 0.108, p = 0.016) — modest but significant. The core hypothesis holds, but the mechanism differs: it is primarily underutilized infrastructure, not transmission waste, that correlates with higher residential rates in New York's utility landscape."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c69057f2",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 6. Rate Disparity — Residential vs. Industrial\n",
+    "\n",
+    "Even if inefficiency drives prices up overall, the burden may not fall equally. This chart examines the **top 10 utilities by Price Spread** — the gap between what residential and industrial customers pay per MWh. Industrial customers typically negotiate volume discounts — some spread is expected. But when residential customers pay 2–3× the industrial rate at the same utility, it raises questions about whether the rate structure reflects true cost-of-service differences or something else. This chart identifies the specific utilities where that premium is most extreme."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "38d3d584",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Keep utilities that offer both industrial and residential services\n",
+    "res_ind_customers_df = data_util.get_customer_utilities(ny_df).round(2)\n",
+    "\n",
+    "dumbbell = plot_util.get_rate_disparity_dumbbell_plot(res_ind_customers_df)\n",
+    "\n",
+    "# dumbbell.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "86fe4d1f",
+   "metadata": {},
+   "source": [
+    "**Interpretation:** Con Edison (Consolidated Edison) shows the largest residential–industrial spread in the dataset, charging residential customers roughly $233/MWh against a near-zero industrial rate. Eight of the top ten utilities by spread are investor-owned — consistent with the box plot finding that IOUs exhibit the widest pricing distributions. Notably, Con Edison also carries a high system loss percentage, placing it in the upper-right quadrant of the fairness audit scatter: both inefficient and inequitable in its rate structure."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8e8ae178",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 7. Energy Flow Analysis\n",
+    "\n",
+    "The Sankey diagrams provide operational context for the efficiency metrics computed above. Rather than a single number, they show the *full picture* of where a utility's energy comes from and where it goes. A wide `Losses` band at a utility with high residential prices is the inefficiency–cost story in one image. We can compare the utility-level Sankey against the U.S. national average to understand whether NY's profile is typical or anomalous."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "0ae862c6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Look at a specific utility's energy usage/flow\n",
+    "utility_usage = data_util.get_utility_usage(ny_df.sum(), level=\"State\")\n",
+    "ny_sankey = plot_util.get_energy_use_sankey_plot(utility_usage)\n",
+    "\n",
+    "# ny_sankey.show()\n",
+    "\n",
+    "# Look at the entire country's energy usage/flow\n",
+    "us_energy_flow = df.groupby([\"Utility.State\"]).sum().sum()[1:]\n",
+    "us_energy_flow = data_util.get_utility_usage(us_energy_flow, level=\"US\")\n",
+    "\n",
+    "us_sankey = plot_util.get_energy_use_sankey_plot(us_energy_flow)\n",
+    "\n",
+    "# us_sankey.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "09b80967",
+   "metadata": {},
+   "source": [
+    "**Interpretation:** The NY state aggregate Sankey shows a Losses band of approximately 3.2% — compare this against the U.S. national aggregate (~5-6% depending on aggregation method) to establish whether New York's grid is more or less efficient than the national benchmark. For individual utilities, the Losses band translates the abstract SystemLossPercentage into an immediate visual — and directly contextualizes the weakly significant but real correlation observed in the fairness audit."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "570d133c",
+   "metadata": {},
+   "source": [
+    "### 7b. Interactive Explorer — Individual Utility Flow"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "7b0d840c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Sankey chart with a drop down energy flow for each individual utility\n",
+    "utility_sankey_with_dropdown = plot_util.add_utility_dropdown(ny_sankey, df=ny_df)\n",
+    "\n",
+    "# utility_sankey_with_dropdown.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "25a0456f",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 8. Export\n",
+    "\n",
+    "Uncomment the following cell to export all charts as SVGs to the `/images` directory.\n",
+    "\n",
+    "> **Requires:** `kaleido` module"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "cb116404",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# %pip install kaleido\n",
+    "\n",
+    "# # Gather all plots and export them as SVGs\n",
+    "# plots = [top_variance_table, strip_plot, heatmap,\n",
+    "#          dual_y_scatter, dumbbell, ny_sankey, us_sankey]\n",
+    "\n",
+    "# plot_util.export_plots_as_svg(plots)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}