Question: Build a random forest regression model to predict city-level housing price index using multi-source real estate and socio-economic data. Use the Zillow Home Value Index (ZHVI) for all homes as of 2019-12-31 as the target variable. Derive housing features from the other Zillow ZHVI data (1-bedroom, 2-bedroom, 3-bedroom, condominium, single-family residence) as of the same date, and extract the rental price feature from the median rental price data as of December 2019. Integrate state-level socio-economic features: from census data, compute state-level average income, poverty rate, and unemployment rate; from unemployment data, compute the 2019 annual average state unemployment rate; from education data, use 2017 state-level total revenue as a feature; from crime data, compute state-level average violent crime and property crime rates. Merge all features at the city level, and use median imputation for missing values. Split data with test_size=0.2 and random_state=42. Train a RandomForestRegressor with n_estimators=100, max_depth=15, random_state=42, and n_jobs=-1. Evaluate with MAE, RMSE, and R2. Output: Save evaluation metrics to housing_prediction_metrics.csv with columns Metric and Value. Save actual values, predicted values, and residuals to prediction_results.csv.

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_Zhvi_1bedroom.csv

Zillow House Price Data/City_Zhvi_2bedroom.csv

Zillow House Price Data/City_Zhvi_3bedroom.csv

Zillow House Price Data/City_Zhvi_Condominum.csv

Zillow House Price Data/City_Zhvi_SingleFamilyResidence.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

US Census Demographic Data/acs2017_county_data.csv

Unemployment in America Per US State.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

Batch Processing and Performance Optimization

Data Transformation and Column Manipulation

Data Structure Handling (Dictionaries, Lists)

Data Transformation and Feature Engineering

Dynamic Data Transformation and Insertion

Feature Selection and Dimensionality Reduction

compare_csv(output_file_name='housing_prediction_metrics.csv', gold_file_name='result_metrics.csv', specified_columns=['Value'], thresholds={'Value': None})


Question:
Task: Build a housing price prediction model predicting 'ZHVI' (Zillow Home Value Index) and conduct stratified validation using locally available multi-source real estate and socioeconomic datasets.

Requirements:
1. Data Preparation:
   - Use city-level home value data (all home types) as the prediction target, extracting values as of December 31, 2019.
   - Extract features from 1-bedroom and 2-bedroom home value datasets, as well as rental datasets (all, 1-bedroom, 2-bedroom), using values as of December 31, 2019.
   - Integrate state-level demographic features (per capita income, poverty rate, population), city-level crime statistics, state-level unemployment rates (latest monthly), and education metrics (total expenditure, fourth-grade math scores - latest annual).
   - Standardize all state identifiers to two-letter abbreviations and merge data by state and city.

2. Model Configuration:
   - Train a Random Forest Regressor with n_estimators=100, max_depth=15, random_state=42.
   - Use train-test split with test_size=0.2, random_state=42.

3. Stratified Validation:
   - Crime stratification: Divide validation data into quartile groups based on violent crime rate.
   - Income stratification: Divide into high and low income groups using the median income as threshold.

4. Output Requirements:
   - 'model_validation_metrics.csv': Contains overall metrics (R², MAE, RMSE) and R² scores for each crime quartile group and income group.
   - 'residuals_analysis.csv': Contains actual value, predicted value, residual, violent crime rate, income, crime quartile label, and income group label.

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_Zhvi_1bedroom.csv

Zillow House Price Data/City_Zhvi_2bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

United States Crime Rates By City Population/crime_100_250.csv

US Census Demographic Data/acs2017_county_data.csv

Unemployment in America Per US State.csv

Data Conversion and Post-Loading Processing

Column-wise Transformations and Aggregation

Data Transformation and Feature Engineering

Data Structure Understanding and Initialization

compare_csv(output_file_name='model_validation_metrics.csv', gold_file_name='result_metrics.csv', specified_columns=['Value'], thresholds={'Value': None})

compare_csv(output_file_name='residuals_analysis.csv', gold_file_name='result_residuals.csv', specified_columns=['residual', 'violent_crime', 'Income'], thresholds={'residual': None, 'violent_crime': None, 'Income': None})


Question:
A real estate investment company needs to evaluate the performance of the housing price prediction model. Using the provided multiple data sources (including city rental data, housing price index, education data, unemployment data, and crime data), build a regression model to predict the Zillow Housing Price Index (ZHVI) for December 2019. Specific requirements:
1. Load all CSV data files into an SQLite database
2. Calculate the housing price volatility for each city based on the historical ZHVI data from 2015 to 2018
3. Use the reciprocal of the volatility as the sample weight (the lower the volatility of a city, the higher its weight)
4. Train a LightGBM regression model using data from 2018 and earlier as features
5. Use time series cross-validation to evaluate the model and report the weighted mean absolute error (Weighted MAE)

Output: Save the evaluation results to output.json, format: {"weighted_mae": }

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/DaysOnZillow_City.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

Data Loading with Pandas
Directory and File Management
Parsing and Reading Data Files
Command-Line and Shell Operations
Data Ingestion and Processing
Date and Time Conversion
Time Formatting and String Manipulation
Arithmetic and Cumulative Calculations
Date Adjustment and Alignment
Column Selection and Consistency Checks
Data Preparation and Formatting
Data Aggregation and Grouping
Statistical Analysis and Metrics
Numerical Data Handling
Handling Missing Data
In-place vs Copy Operations
Array and Matrix Manipulation
Output and Logging
Model Evaluation Metrics
Weighted Aggregation and Summation
Epsilon Handling
Normalization and Weighted Aggregation
Data Manipulation and Summarization
Time Series Analysis and Forecasting
Data Splitting and Sampling
Model Training & Optimization
Model Evaluation & Validation
Indexing and Row-Level Operations
Data Storage and Structuring
CSV Processing
Data Export and Output Processing
Data Serialization & File Handling

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'weighted_mae': 0.05})


Question:
Analyze the growth patterns of the housing market in each state of the United States from 2010 to 2018. For each state-level housing indicator, calculate the compound annual growth rate (CAGR) using the first available monthly data point in 2010 as the initial value and the last available monthly data point in 2018 as the final value, over an 8-year period.

Draw an overlapping histogram showing the CAGR distribution across all states for each indicator, and identify which indicator has the highest peak frequency in its CAGR distribution.

Output:
- output.png: Overlapping histogram of the CAGR distribution for all indicators
- output.csv: A CSV file with columns 'indicator' (indicator name) and 'value' (peak frequency count), listing all indicators that share the highest peak frequency

Zillow House Price Data/State_Zhvi_1bedroom.csv

Zillow House Price Data/State_Zhvi_2bedroom.csv

Zillow House Price Data/State_Zhvi_3bedroom.csv

Zillow House Price Data/State_Zhvi_4bedroom.csv

Zillow House Price Data/State_Zhvi_5BedroomOrMore.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

Zillow House Price Data/State_Zhvi_Condominum.csv

Zillow House Price Data/State_Zhvi_SingleFamilyResidence.csv

Zillow House Price Data/State_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/State_Zri_AllHomesPlusMultifamily.csv

Time Difference and Gradient Calculation

Mathematical and Statistical Computations

Statistical Calculations and Descriptive Statistics

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['value'], thresholds={'value': None})


Question:
A real estate investment company needs to identify states with a high Price-to-Rent Ratio, while considering macroeconomic stability and public safety.

Using state-level housing price and rent data from December 2019, education funding data from 2016, unemployment rate data from December 2019, and crime rate data, calculate the Price-to-Rent Ratio for each state as: housing price index / (monthly rent × 12).

Filter states that meet all of the following conditions:
- Unemployment Rate < 5%
- Average education funding > $12,000
- Average violent crime rate < 400 per 100,000 residents

Sort the results by Price-to-Rent Ratio in ascending order.

Output: output.csv (contains filtered state-level data with Price-to-Rent Ratio, unemployment rate, education funding, and violent crime rate for each qualifying state)

Zillow House Price Data/State_Zhvi_AllHomes.csv

Zillow House Price Data/State_MedianRentalPrice_AllHomes.csv

Unemployment in America Per US State.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

Data Conversion and Post-Loading Processing

Conditional Logic and Row-wise Operations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['State', 'PriceToRentRatio', 'UnemploymentRate', 'EducationRevenuePerStudent', 'AvgViolentCrimeRate'], thresholds={'PriceToRentRatio': None, 'UnemploymentRate': None, 'EducationRevenuePerStudent': None, 'AvgViolentCrimeRate': None})


Question:
The Housing Policy Research Group needs to assess the reliability of housing market data for various cities in the United States, in order to prioritize the selection of cities with high data quality for long-term affordability studies.

Based on the city-level housing market datasets, evaluate the data integrity of each city-state pair during the period from January 2010 to December 2019. For each city-state pair, compute total data points as the number of monthly cells in this period across the listed files where that city-state pair appears, and compute non-missing data points from those same cells. Select city-state pairs with a missing rate exceeding 40%, and sort by missing rate in descending order, breaking ties by state and then city name in ascending order.

Output: output.csv (contains the list of city-state pairs with a missing rate exceeding 40%, sorted as specified, including city name, state, total data points, non-missing data points, and missing rate)

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_Zhvi_SingleFamilyResidence.csv

Zillow House Price Data/City_MedianRentalPrice_Sfr.csv

Zillow House Price Data/DaysOnZillow_City.csv

Zillow House Price Data/Sale_Prices_City.csv

Preprocessing and File Structure Adjustments

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['RegionName', 'State', 'non_missing', 'total', 'missing_rate'], thresholds={'non_missing': None, 'total': None, 'missing_rate': None})


Question:
Real estate investment companies need to analyze how macroeconomic, demographic, educational, and public safety factors jointly affect housing prices in various cities of the United States in 2019.

Construct an optimized regression model with:
- Dependent variable: The natural logarithm of the 2019 all-homes city ZHVI, computed as the median of its 2019 monthly values
- Independent variables:
  1. Crime rates: City-level violent crime rate and property crime rate per 100,000 people (log-transformed)
  2. Unemployment rate: The state-level average unemployment rate in 2019
  3. Education quality: State-level average math (4th grade) and reading (8th grade) scores from 2017 to 2019
  4. Economic factors: County-level median household income and poverty rate
  5. Population: County-level total population

Use only city records with complete matched target and feature values.

Train and compare Linear Regression, Random Forest (n_estimators=100), and Gradient Boosting (n_estimators=100) models. Use an 80/20 train-test split with random_state=42. Standardize all features before training. Select the best performing model based on test set R².

Output: output.csv (containing the best model's test set R² value, rounded to 4 decimal places)

Zillow House Price Data/City_Zhvi_1bedroom.csv

Zillow House Price Data/City_Zhvi_2bedroom.csv

Zillow House Price Data/City_Zhvi_3bedroom.csv

Zillow House Price Data/City_Zhvi_4bedroom.csv

Zillow House Price Data/City_Zhvi_5BedroomOrMore.csv

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/DaysOnZillow_City.csv

Zillow House Price Data/Sale_Prices_City.csv

Unemployment in America Per US State.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

US Census Demographic Data/acs2017_county_data.csv

Data Structure Handling (Dictionaries, Lists)

Mathematical and Statistical Computations

Preprocessing and File Structure Adjustments

Feature Selection and Dimensionality Reduction

Data Conversion and Post-Loading Processing

Arithmetic Transformations and Normalization

Logarithmic Scaling and Axis Transformations

Data Transformation and Feature Engineering

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['value'], thresholds={'value': None})


Question:
The housing policy research institution needs to assess the housing market conditions in each state of the United States and the District of Columbia to support the allocation decisions of urban development grants.

Based on city-level housing market, economic, educational, crime, and population data, conduct a comprehensive state-level analysis:
- Housing market: Aggregate city-level housing price index and rent prices to state level using 2019 data
- Economic: 2019 unemployment rate and median household income
- Education: Education expenditure (2016)
- Social: Poverty rate and crime rate

Calculate the Price-to-Rent Ratio as: state-level housing price index / (state-level monthly rent × 12).

Calculate a comprehensive socio-economic score using Min-Max normalized indicators with the following weights (note: for unemployment, poverty, and crime rate, lower values indicate better conditions and should be inverted before weighting):
- Income: 0.25
- Education Revenue: 0.20
- Unemployment Rate: 0.20
- Poverty Rate: 0.20
- Crime Rate: 0.15

Use available non-missing values when normalizing each indicator. Keep jurisdictions in the output even when some required indicators are unavailable, leaving affected output fields blank.

Calculate a housing affordability indicator as: median household income / Price-to-Rent Ratio.

Sort by the comprehensive socio-economic score in descending order.

Output: output.csv (contains state name, comprehensive socio-economic score, housing affordability indicator, Price-to-Rent Ratio, income, and unemployment rate for each state and the District of Columbia)

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

Dictionary Manipulation and Construction

Preprocessing and File Structure Adjustments

Data Conversion and Post-Loading Processing

Normalization and Percentile Calculations

Arithmetic Transformations and Normalization

Incremental and Comparative Calculations

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['State', 'SocioeconomicScore', 'AffordabilityIndex', 'PriceToRentRatio'], thresholds={'SocioeconomicScore': None, 'AffordabilityIndex': None, 'PriceToRentRatio': None})


Question:
The Federal Housing Policy Working Group needs to identify the states where housing market behavior was abnormal during the period of 2011-2016.

Based on city-level housing market, economic and demographic statistics, conduct an annual analysis at the state level:
- Housing market data: house price index, rent price, number of days for house sales and sales price
- Economic data: annual unemployment rate
- Socioeconomic data: educational and demographic indicators

Requirements:
1. Data aggregation: Aggregate city-level data by state and year, and calculate the annual median
2. Feature engineering: Calculate the housing market characteristics of each state each year, including house price growth rate, rent growth rate, price-to-rent ratio, etc.
3. Data standardization: Standardize all numerical features
4. Cluster analysis: Divide the state-year observations into 4 housing market types
5. Anomaly detection: Calculate the Euclidean distance from each observation to its corresponding cluster center as the anomaly score
6. Sort by descending anomaly score and output the relevant results

Output: output.csv (contains state name, year, cluster label, anomaly score, sorted by descending anomaly score)

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/DaysOnZillow_City.csv

Zillow House Price Data/Sale_Prices_City.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

Incremental and Comparative Calculations

Time Difference and Gradient Calculation

Preprocessing and File Structure Adjustments

Column-specific or Feature-wise Processing

Feature Selection and Statistical Computation

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=[0, 1], calculate_columns=[3], metric='pearson', lower_bound=0.8, upper_bound=1.0)


Question:
The National Housing Policy Institute of the United States needs to assess the health status of the housing markets in each state of the United States in 2017. Based on urban and state-level housing markets, economic and demographic data, a comprehensive assessment is to be conducted:
- Housing market data: house price index (ZHVI), 1-bedroom rent price, and days on market
- Economic data: annual unemployment rate and median household income
- Socioeconomic data: education total revenue

Requirements:
1. Data aggregation: Aggregate city-level data to the state-level using median, based on 2017 data. For education data, use the latest available year (fallback to 2016 if 2017 is unavailable). Fill missing values with the column median.
2. Calculate housing market health indicators:
   - Housing price stability (standard deviation of house price index across 2017 months)
   - Rental yield (annual 1-bedroom rent / house price index)
   - Market liquidity (inverse of the number of days on market)
   - Employment health (inverse of unemployment rate)
   - Education quality (education total revenue)
   - Economic vitality (median household income)
3. Data standardization: Normalize all indicators to the range of 0-1 using Min-Max scaling. For indicators where lower is better (price volatility, days on market, unemployment rate), use 1 - normalized value.
4. Calculate the comprehensive housing health score: Weighted average with equal weights (1/6 each)
5. Sort by the comprehensive score in descending order

Output: output.csv (Contains state abbreviation, comprehensive housing health score, and intermediate indicators for each state, sorted by score in descending order)

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/DaysOnZillow_State.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

Dictionary Manipulation and Construction

Data Structure Handling (Dictionaries, Lists)

Arithmetic Transformations and Normalization

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=[0], calculate_columns=[1], metric='mae', lower_bound=0.01, upper_bound=0)


Question: A national housing policy research group aims to investigate how economic and social factors in each state of the United States affect the synchronicity of the housing market during the recovery phase following the economic recession (2015 - 2018). Based on the housing market data of each state, including the number of days the houses are on the market, rental prices, and housing value indices, a state-level housing similarity network is constructed. For each state, compute the mean of each housing indicator over the 2015-2018 monthly data, standardize the resulting feature vector, and use the Euclidean distance between standardized vectors as edge weights. Apply K-Means clustering (k=4, random_state=42) to identify synchronous housing market clusters. Then, integrate background data to enrich this graph: map the node border colors to the average eighth-grade math scores of each state (from education data, summarized by state), adjust the node transparency using the average unemployment rate (2015 - 2018), and set the node size according to the growth rate of housing value (January 2015 to January 2020). Additionally, calculate the average violent crime rate within each cluster and display it as a heatmap in a separate subplot placed to the right of the main network visualization (use a two-panel figure with the network scatter on the left and the heatmap subplot on the right, with width ratio 3:1). Use PCA for the network layout. Generate a high-resolution visualization with a dark background and save it to output.png (300 DPI). Finally, construct an adjacency matrix using the 25th percentile of pairwise distances as the edge threshold, count the number of connected components via DFS, and save the result to output.csv with columns 'metric' and 'value'.

Unemployment in America Per US State.csv

United States Crime Rates By City Population/crime_100_250.csv

Zillow House Price Data/DaysOnZillow_State.csv

Zillow House Price Data/State_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

Incremental and Comparative Calculations

Numerical Operations and Type Conversion

Numerical Comparison and Proximity Checks

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['value'], thresholds={'value': None})

compare_image(output_file_name='output.png', gold_file_name='result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question:
The National Housing Policy Research Group is investigating how educational investment and labor market conditions relate to the affordability of multi-bedroom rents in major U.S. metropolitan areas.

Identify cities that have at least 24 months of non-missing 2-bedroom rental data during the period 2010-2019. For these qualifying cities, compile a comprehensive dataset that includes:
- The latest available (December 2019) rental prices for each multi-bedroom type and all housing types, as well as the housing value index
- State-level education expenditure per student (from the most recent available year) and average unemployment rate
- County-level demographic indicators: total population, household income, poverty rate, unemployment rate, and per capita income

Merge the data using city-state identifiers for housing data, state identifiers for education and unemployment data, and county-state identifiers for census data.

Output: output.csv (a comprehensive dataset with one row per qualifying city)

Unemployment in America Per US State.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_4Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_5BedroomOrMore.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_Zhvi_AllHomes.csv

US Census Demographic Data/acs2017_county_data.csv

Data Transformation and Column Manipulation

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['RegionName', 'state_abbr', 'latest_rental_2br', 'education_expenditure_per_student', 'average_unemployment_rate', 'TotalPop', 'Income', 'Poverty', 'Unemployment', 'IncomePerCap'], thresholds={'latest_rental_2br': None, 'education_expenditure_per_student': None, 'average_unemployment_rate': None, 'TotalPop': None, 'Income': None, 'Poverty': None, 'Unemployment': None, 'IncomePerCap': None})


Question:
Real estate market analysts need to develop a Housing Market Stress Index (HMSI) to assess the health of the real estate market in each state of the United States during the period from 2015 to 2019.

The HMSI is composed of two components:
1. Housing price volatility: For each state-year, calculate the standard deviation of year-over-year ZHVI growth rates across cities within that state.
2. Rent-to-Value ratio: For each state-year, calculate the ratio of median rental price to median housing value index.

Normalize each component using Min-Max scaling across all state-year observations, then sum the two normalized components to obtain the HMSI. Calculate the average HMSI for each state over the 2015-2019 period, and output the top 5 states with the highest average HMSI, ranked in descending order.

Output: output.csv (containing state and average HMSI for the top 5 states)

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Mathematical and Statistical Computations

Normalization and Percentile Calculations

Arithmetic Transformations and Normalization

Time Difference and Gradient Calculation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['State', 'Avg_HMSI'], thresholds={'Avg_HMSI': None})


Question:
A national real estate investment firm needs to build a city-level home price forecasting model and rigorously evaluate its performance using multi-dimensional socioeconomic indicators. Using monthly home price indices, all listed city-level rental and ZRI price sources, state-level unemployment rates, education funding data, and county-level demographic statistics aggregated to state-level summaries, construct a time series prediction pipeline that:

1. Loads and standardizes data formats from multiple sources with different temporal granularities and geographic levels
2. Integrates multi-source data using pandas merge operations to build a comprehensive feature dataset aligned by city, state, and time period, with the rental sources combined at city-month level before merging
3. Computes time-series window features including 6-month and 12-month rolling averages, rolling standard deviations (volatility), year-over-year change rates, and annualized rent-to-price ratios
4. Restricts each city to the overlapping time window where home prices and all exogenous variables are available, and applies ARIMA(1,1,1) models with exogenous variables (6-month rental moving averages, unemployment rates, annualized rent-to-price ratios, median income, and education revenue) to forecast home prices for cities with at least 60 months of complete overlapping observations
5. Uses 80% of each selected city's overlapping-window data for training and 20% for testing; if more than 30 cities satisfy the criteria, process the 30 cities with the longest complete overlapping histories, breaking ties by state and city name
6. Calculates performance metrics with 95% confidence intervals using Bootstrap resampling (1000 iterations with random_state=42)

Output a JSON file named 'output.json' containing exactly five evaluation metrics with keys `mae`, `rmse`, `mape`, `r2`, and `weighted_mae` (using maximum-scaled home-value weights, giving higher weights to higher-priced properties). Each metric must include three values: 'low' (2.5th percentile), 'mid' (50th percentile/median), and 'high' (97.5th percentile), all rounded to 4 decimal places.

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_Sfr.csv

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_3Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_4Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_5BedroomOrMore.csv

Zillow House Price Data/City_Zri_AllHomesPlusMultifamily.csv

Zillow House Price Data/City_Zri_SingleFamilyResidenceRental.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

Model Architecture and Weight Management

Parameter Estimation and Bootstrap Methods

Mathematical and Statistical Computations


Question:
A housing policy research institute aims to identify abnormal fluctuations in U.S. state real estate markets between 2010 and 2018. Using monthly market indicators including home price indices, rental prices, days on market, and sale prices, along with state-level unemployment rates, education funding, and demographic statistics, construct an anomaly detection pipeline that:

1. Loads and standardizes multi-source data with different temporal granularities, converting wide-format time series to long-format
2. Integrates data using merge operations aligned by state and year, aggregating monthly data to annual averages
3. Designs a composite anomaly scoring algorithm using weighted Z-scores across multiple market indicators (home prices, rents, days on market, sale prices, unemployment). Education funding and demographic statistics are used as auxiliary integration/background variables and do not need to be included in the composite score.
4. Applies smoothed Z-Score anomaly detection (3-year lag window, 2.0 threshold, 0.5 influence factor) to identify significant deviations
5. Generates multi-panel visualizations showing raw scores, smoothed trends, control bounds, and anomaly signals for representative states, where representative states are defined deterministically as follows: first select states with at least 5 annual observations and at least one non-zero anomaly signal, sort them alphabetically by state abbreviation, and take the first 5; if fewer than 5 states satisfy this rule, fill the remaining slots using states with the most annual observations, breaking ties alphabetically.
6. Outputs structured results including anomaly signals, smoothed averages, rolling standard deviations, and statistical summaries

Output two files: (1) 'output_image.png' containing a multi-panel visualization with 5 representative states selected by the rule above, each showing left panel (composite anomaly score, smoothed trend line, and green-shaded control bounds) and right panel (anomaly signals as red/blue bars where +1 indicates positive anomaly, -1 indicates negative anomaly); (2) 'output.json' containing a dictionary with keys 'analysis_period' (set to '2010-2018'), 'algorithm_params' (with 'lag': 3, 'threshold': 2.0, 'influence': 0.5), and 'states' (mapping each state to its 'years' array, 'original_scores' array, 'smoothed_avg' array, 'rolling_std' array, 'upper_bound' array, 'lower_bound' array, 'signals' array, and 'anomaly_stats' object containing 'positive_anomalies' count, 'negative_anomalies' count, 'total_anomalies' count, and 'anomaly_years' list).

Zillow House Price Data/DaysOnZillow_State.csv

Zillow House Price Data/Sale_Prices_State.csv

Zillow House Price Data/State_MedianRentalPrice_AllHomes.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2015_county_data.csv

US Census Demographic Data/acs2017_county_data.csv

Data Structure Handling (Dictionaries, Lists)

Rolling Statistics and Window-Based Signal Processing

compare_json('output.json', 'result.json', thresholds={'states': None})


Question:
A housing policy research institute aims to build a state-level housing affordability analysis dashboard to visualize the relationships between home prices, rental prices, affordability ratios, and socioeconomic indicators. Using monthly home price indices, rental price data, state-level unemployment rates, and county-level demographic statistics from 2017, construct a comprehensive analysis pipeline that:

1. Loads and standardizes multi-source data with different temporal granularities, converting wide-format monthly time series to long-format annual aggregates for the 2010-2018 period
2. Integrates data using pandas merge operations aligned by state and year for housing and unemployment data, and merges 2017 county-level demographic statistics aggregated to the state level as static socioeconomic context
3. Calculates housing affordability metrics including home price-to-income ratio and rent-to-income ratio, plus year-over-year growth rates for both home prices and rents
4. Creates a static multi-panel visualization showing: (a) top 20 states by annual average home value as horizontal bar chart, (b) top 20 states by annual average rent price as horizontal bar chart, (c) scatter plot of home price-to-income ratio vs. rent-to-income ratio colored by unemployment rate, (d) scatter plot of unemployment rate vs. home value growth rate

Output one file: 'output.png' containing a 2x2 grid of subplots (16x12 inches, 150 DPI) with the four visualizations described above for year 2018 data.

Zillow House Price Data/State_Zhvi_AllHomes.csv

Zillow House Price Data/State_MedianRentalPrice_AllHomes.csv

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

Column-wise Transformations and Aggregation


Question:
An economic research institute aims to analyze the lead-lag relationship between state-level home price indices and socioeconomic indicators to identify predictive signals for housing market changes. Using monthly home price indices and state-level unemployment rate data, construct a cross-correlation analysis pipeline that:

1. Loads and cleans time series data, converting wide-format monthly data to long-format, normalizing state identifiers to two-letter USPS abbreviations, and normalizing both series to a common monthly timestamp convention anchored to the first day of each month
2. Computes 12-month rolling mean and standard deviation for home price indices using at least 6 observed months per rolling window to capture trends and volatility patterns
3. Computes 12-month rolling statistics for unemployment rates using at least 6 observed months per rolling window to smooth short-term fluctuations
4. Performs centered, normalized cross-correlation analysis between home price trends and unemployment trends with lag range from -24 to +24 months (negative lags indicate unemployment leads home prices, positive lags indicate home prices lead unemployment). For each state, first align both rolling-mean series to the complete monthly index from 2010-01 through 2019-12, then fill missing values by forward fill followed by backward fill before computing correlations.
5. Identifies each state's strongest lag relationship, then ranks states whose strongest absolute correlation is greater than 0.1

Output a JSON file named 'output.json' containing: (1) 'analysis_metadata' object with fields 'time_range' (set to '2010-01 to 2019-12'), 'lag_range_months' (array [-24, 24]), 'correlation_threshold' (0.1), 'window_size' (12), 'total_states_analyzed' (integer), and 'significant_relationships_found' (integer, number of states above the correlation threshold); (2) 'top_10_relationships' array where each element contains 'state' (string, two-letter USPS abbreviation), 'variable1' (string, set to 'ZHVI_Rolling_Mean'), 'variable2' (string, set to 'Unemployment_Rolling_Mean'), 'correlation' (float, rounded to 4 decimals), 'lag_months' (integer), 'abs_correlation' (float, rounded to 4 decimals), and 'relationship' (string, one of 'Unemployment leads ZHVI', 'ZHVI leads Unemployment', or 'Synchronous'); (3) 'summary_statistics' object summarizing the top 10 relationships with 'mean_correlation', 'max_correlation', 'min_correlation' (all floats rounded to 4 decimals), and 'mean_lag_months' (float rounded to 1 decimal).

Zillow House Price Data/State_Zhvi_AllHomes.csv

Unemployment in America Per US State.csv

Rolling Statistics and Window-Based Signal Processing

Cross-Correlation and Time-Series Alignment

Cross/Merge Products and Filtering by Time Windows


Question:
A real estate analytics firm needs to build a Housing Market Health Index (HMHI) model to comprehensively evaluate the health status of housing markets across US states and identify the healthiest and least healthy markets. Using monthly home price indices, unemployment rates, crime statistics, education funding data, and demographic statistics, construct a multi-dimensional scoring system that:

1. Loads and standardizes multi-source data with different temporal granularities and geographic levels, converting wide-format Zillow time series to long-format with unified date formats (YYYY-MM-DD) and using state abbreviations (e.g., CA, NY) in the final scoring outputs
2. Calculates market liquidity scores using price volatility metrics derived from monthly home price changes, normalized to 0-100 scale where higher volatility indicates better liquidity
3. Calculates housing affordability scores using 2019 average home price levels and 2017 per-capita income data converted to annual income, normalized to 0-100 scale where lower ratios indicate better affordability
4. Calculates economic stability scores using average unemployment rates from 2015-2019, normalized to 0-100 scale where lower unemployment indicates higher stability
5. Calculates public safety scores using crime-related indicators aggregated by state, normalized to 0-100 scale where lower crime rates indicate higher safety; if no usable crime score can be derived, use the fallback weighting scheme without the safety dimension
6. Calculates education resource scores using per-student education revenue from 2015-2019, normalized to 0-100 scale where higher spending indicates better resources
7. Constructs composite HMHI scores using weighted aggregation: liquidity 25%, affordability 25%, economic stability 20%, safety 15%, education 15% (adjusting weights to 30/30/25/0/15 if safety data is not used)
8. Ranks all states by HMHI scores and identifies top 5 healthiest and bottom 5 least healthy markets

Output two files: (1) a JSON file named 'output.json' containing: 'top_5_healthiest', 'bottom_5_least_healthy', 'all_states_ranking', 'weights_used', 'methodology', and 'calculation_date'; (2) a CSV file named 'hmhi_ranking.csv' containing the complete state ranking table sorted by HMHI score.

Zillow House Price Data/State_Zhvi_AllHomes.csv

Unemployment in America Per US State.csv

United States Crime Rates By City Population/crime_40_60.csv

United States Crime Rates By City Population/crime_60_100.csv

United States Crime Rates By City Population/crime_100_250.csv

United States Crime Rates By City Population/crime_250_plus.csv

US Census Demographic Data/acs2015_county_data.csv

US Census Demographic Data/acs2017_county_data.csv

Data Conversion and Post-Loading Processing

Data Structure Handling (Dictionaries, Lists)

Normalization and Percentile Calculations

Mathematical and Statistical Computations

compare_json('output.json', 'result.json', thresholds={'weights_used': {'liquidity': None, 'affordability': None, 'stability': None, 'safety': None, 'education': None}})

compare_csv(output_file_name='hmhi_ranking.csv', gold_file_name='result_hmhi_ranking.csv', ignore_order=False, thresholds={'Liquidity': None, 'Affordability': None, 'Stability': None, 'Safety': None, 'Education': None, 'HMHI': None})


Question: A national real estate investment company wants to build a regression-based forecasting system to predict housing value indices and rental prices for different property types (1-4 bedrooms) to guide investment portfolio allocation decisions. Using data from 2015 to 2018, construct a multi-target regression workflow that integrates external socioeconomic indicators including state-level education expenditures, unemployment rates, and county-level demographic statistics. Use 2015-2017 observations for training and treat 2018 as the holdout forecasting period. Use monthly city-level target observations; aggregate unemployment to state-year means and treat ACS 2017 demographics as static state features. To keep results reproducible, use random_state=42 for sampling with caps of 62,500 rows per melted target source and 500,000 rows for the merged panel; fill feature missing values with column means and then training-set means within each target split; apply StandardScaler followed by PCA(n_components=0.95); and fit a separate LinearRegression model for each target. The system should: 1) Load and merge city-level housing value and rental price data; 2) Integrate external socioeconomic data; 3) Standardize feature scales and apply dimensionality reduction techniques; 4) Train target-specific regression models for each housing value or rental series; 5) Evaluate 2018 holdout performance and save results to 'output.json' with 'metrics' containing nested objects for each target variable, where each target object has 'r2' and 'mae'.

Unemployment in America Per US State.csv

US Census Demographic Data/acs2017_county_data.csv

Zillow House Price Data/City_Zhvi_1bedroom.csv

Zillow House Price Data/City_Zhvi_2bedroom.csv

Zillow House Price Data/City_Zhvi_3bedroom.csv

Zillow House Price Data/City_Zhvi_4bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_3Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_4Bedroom.csv

Data Conversion and Post-Loading Processing

Data Preprocessing and Column Management

Mathematical and Statistical Computations

Data Structure Handling (Dictionaries, Lists)

Data Structure Understanding and Initialization

Feature Selection and Dimensionality Reduction

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'metrics': None})


Question:
A national real estate policy research institute aims to identify stable housing markets across US states by analyzing housing price volatility and filtering outliers. Using monthly home price indices and rental price data for 1-bedroom and 2-bedroom properties, construct a volatility analysis pipeline that:

1. Loads and integrates multi-bedroom housing market data, converting wide-format monthly time series to long-format with unified date formats. Merge home price data with rental data for both 1-bedroom and 2-bedroom properties aligned by city, state, and date
2. Calculates housing volatility scores for each state-month combination. Standardize home price indices (ZHVI), 1-bedroom rent, and 2-bedroom rent using Z-score normalization, then compute the standard deviation of these three standardized metrics as the volatility score (lower scores indicate more stable markets)
3. Filters outliers using an enhanced 3-sigma rule based on median and interquartile range (IQR). Calculate median volatility and IQR, then define bounds: lower_bound = median - 3 * 1.35 * IQR, upper_bound = median + 3 * 1.35 * IQR. Retain observations within these bounds as valid data
4. Generates a grouped bar chart showing the latest median home prices, 1-bedroom rents, and 2-bedroom rents for the top 20 states by home price after outlier filtering
5. Outputs statistical results including total observations, valid observations after filtering, number of outliers removed, outlier percentage, volatility thresholds used, and latest state-level metrics for all states after filtering

Output three files: (1) 'output.png' containing a grouped bar chart (14x8 inches, 150 DPI) showing housing market indicators for the top 20 states, with chart title 'Stable Housing Market Indicators After Outlier Filtering', X-axis labeled 'State', Y-axis labeled 'Amount (Thousands/USD)', and legend labels 'Median Home Price (ZHVI)', '1-Bedroom Rent', and '2-Bedroom Rent'; (2) 'output_data.json' containing total observations, valid observations, outliers removed, outlier percentage, and volatility thresholds; (3) 'output_metrics.csv' containing latest state-level metrics (State, ZHVI, Rent_1Br, Rent_2Br, Volatility_Score, Latest_Date) for all states after filtering.

Zillow House Price Data/City_Zhvi_AllHomes.csv

Zillow House Price Data/City_MedianRentalPrice_1Bedroom.csv

Zillow House Price Data/City_MedianRentalPrice_2Bedroom.csv

Data Conversion and Post-Loading Processing

Custom Function Application for Date Correction

Arithmetic Transformations and Normalization

compare_json(output_file_name='output_data.json', gold_file_name='result.json', thresholds={'total_observations': None, 'valid_observations': None, 'outliers_removed': None, 'outlier_percentage': None, 'volatility_thresholds': {'lower_bound': None, 'upper_bound': None}})

compare_image(output_file_name='output.png', gold_file_name='result_image.png', calculate_columns=['type', 'graph_title', 'labels', 'x_label', 'y_label', 'xtick_labels'])


Question: A social media platform's security team is investigating human (non-bot) user accounts that may be spreading hate-related content. Compute the Jaccard similarity between each human user's tweet text and known hateful/offensive speech content. For gender identification, only consider gender classification records with a confidence level of at least 0.8; if no matching gender data is available for a user, label their gender as 'unknown'. Report only users whose maximum Jaccard similarity score is at least 0.05, with each user appearing only once. Save the results to output1.csv with columns: Username, gender, Retweet Count, Follower Count, max_jaccard_score.

Data Structure and Dictionary Operations

Data Structure Creation and Manipulation

compare_csv(output_file_name='output1.csv', gold_file_name='result1.csv', ignore_order=True, specified_columns=['Username', 'gender', 'Retweet Count', 'Follower Count', 'max_jaccard_score'], thresholds={'max_jaccard_score': None})


Question:
Generate a 2x2 visualization report (save as cross_platform_engagement_analysis.png) for cross-platform social media analysis:

(a) Boxplot: Twitter engagement rate distribution between Bot and Human users.
   - Build Bot detection model using Random Forest with 5-fold cross-validation.
   - Engagement rate = Retweet Count / Follower Count.
   - Title: 'Twitter Engagement: Bot vs Human'.

(b) Stacked Bar Chart: Content toxicity distribution by engagement quartiles.
   - Use np.random.seed(42) to create random engagement scores, then assign records to Q1-Q4 as four equal-sized quartile groups; use viridis colormap.
   - Title: 'Content Toxicity by Engagement Quartile'.

(c) Stacked Bar Chart: Airline sentiment distribution.
   - Use coolwarm colormap.
   - Title: 'Airline Sentiment Distribution'.

(d) Scatter Plot: Instagram influence score vs engagement rate.
   - Engagement rate (%) = (avg_likes / followers) × 100.
   - Color-code by country.
   - Title: 'Instagram: Influence Score vs Engagement Rate'.

Numerical Operations and Type Conversion

Column-specific or Feature-wise Processing

Arithmetic Transformations and Normalization


Question:
A social media platform needs to identify low-credibility user accounts that may pose risks to the community. Build a multi-dimensional credibility scoring system that evaluates user trustworthiness and identifies the 100 users with the lowest credibility scores for security review.

The system should integrate information from all provided data sources, using username as the join key.

Feature definitions:
- hate_proximity: Use TF-IDF vectorization (max_features=1000, English stop words removed) on user tweet text, then compute cosine similarity against hate speech entries. Each user's score is the mean of per-tweet maximum cosine similarities.
- engagement_rate = (mean likes + mean retweets) / (follower count + 1)
- Fill missing confidence values with 0.5, other missing numerical values with 0.

Model features (standardized via StandardScaler): bot detection label, verification status, follower count, gender classification confidence, profile completeness confidence, engagement_rate, and hate_proximity.

Train two models to predict risk:
- Logistic Regression (random_state=42) targeting bot label
- Random Forest (n_estimators=100, random_state=42) targeting a synthetic label: (bot_label × 0.7 + hate_proximity × 0.3) > 0.5

credibility_score = 1 − (0.6 × LR risk probability + 0.4 × RF risk probability). Global random seed: np.random.seed(42).

Output: Save the 100 lowest-credibility users to output.csv with columns: username, credibility_score, rank (1 = lowest credibility).

Dynamic Data Transformation and Insertion

Vectorization and Performance Optimization

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['rank', 'username', 'credibility_score'], thresholds={'rank': None, 'credibility_score': None})


Question:
A social media platform needs to evaluate the effectiveness of its automated content moderation systems across three classification tasks: bot detection, hate speech identification, and user demographic (gender) classification. Build a text classification evaluation framework using the three provided datasets.

Technical specifications:
- Feature extraction: TF-IDF vectorization with max_features=5000, ngram_range=(1, 2), and English stop words removal
- Classifier: Logistic regression with max_iter=1000, random_state=42, and one-vs-rest strategy for multi-class tasks
- Data split: 80/20 train/test, random_state=42, stratified by label
- Cross-validation: 5-fold on training sets, scored by accuracy
- Evaluation metrics: accuracy and macro-averaged F1 score on test sets

Determine the best-performing model by accuracy (use task identifiers: bot_detection, hate_speech, gender_classification) and compute the average cross-validation score across all models.

Save results to output.json with all floating-point values rounded to 4 decimal places.
Output fields: bot_accuracy, bot_f1, bot_cv_mean, hate_accuracy, hate_f1, hate_cv_mean, gender_accuracy, gender_f1, gender_cv_mean, best_model, best_accuracy, avg_cv_score

Data Preprocessing and Column Management

Categorical Data Preprocessing and Simplification

Vectorization and Performance Optimization

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'bot_accuracy': 0.1, 'bot_f1': 0.1, 'hate_accuracy': 0.1, 'hate_f1': 0.1, 'gender_accuracy': 0.1, 'gender_f1': 0.1, 'best_accuracy': 0.1, 'avg_cv_score': 0.1})


Question:
Cross-platform user engagement and content toxicity analysis.

Integrate six data sources covering Twitter, YouTube, and Instagram to analyze cross-platform user engagement patterns and content toxicity.

For Twitter data, determine content toxicity by matching preprocessed tweets (lowercase, URLs/mentions/hashtags/special characters removed) against the hate speech dataset using TF-IDF vectorization (stop_words='english', ngram_range=(1, 2), max_features=5000) with cosine similarity. When the best match similarity >= 0.3, assign the corresponding toxicity class from the hate speech record; otherwise classify as class 2 (neutral). Enrich Twitter data with bot detection labels and gender information through username matching. For YouTube and Instagram, assign defaults: bot_label=-1, gender='unknown', toxicity_class=2.

Calculate normalized engagement rate as total engagement divided by audience size for each platform. Treat zero audience sizes as missing during the rate calculation, then set missing or infinite engagement rates to 0 while retaining the records.

Group by platform, bot_label, gender, and toxicity_class. Compute mean, standard deviation, and count of engagement rates per group, rounding mean and std to 6 decimal places.

Set np.random.seed(42) for reproducibility.

Save results to output.csv with columns: platform, bot_label, gender, toxicity_class, mean, std, count.

Data Transformation and Column Manipulation

Dynamic Data Transformation and Insertion

Data Conversion and Post-Loading Processing

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'bot_label': None, 'toxicity_class': None, 'mean': None, 'std': None, 'count': None})


Question:
Social media user influence network analysis.

Integrate user data from two data sources by matching usernames case-insensitively, and aggregate interaction metrics per user. Construct a directed interaction graph: for each user, create incoming edges from randomly selected other users based on their mention count (capped at 10 edges per user), with source user selection probability weighted by follower count. Use random seed 42 for reproducibility.

Calculate PageRank centrality scores with damping factor alpha=0.85 and edge weights, then rank users by influence.

Output: Save the top 10 most influential users to output.csv with columns: rank, username, pagerank_score (rounded to 6 decimal places), mention_count, follower_count.

Data Loading with Pandas
Data Aggregation and Grouping
Join Operations and Merging
Data Preparation and Formatting
Element-wise Dataframe Operations
In-place vs Copy Operations
Graph Creation and Manipulation
Graph Data Preparation and Cleaning
Array and Matrix Manipulation
Indexing and Row-Level Operations
Hierarchical and Network Data Analysis
Graph Analysis and Metrics
Graph Algorithms and Clustering
Sorting, Limiting, and Ranking
Data Storage and Structuring

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['rank', 'username', 'pagerank_score', 'mention_count', 'follower_count'], thresholds={'pagerank_score': None})


Question: Identify maximal consecutive tweet sequences (within 24h intervals) from same users. Compute per sequence: length (tweet count), bot label mode (Username case-insensitive, -1 for no match), gender mode (fuzzy name matching threshold 0.8, case-insensitive, length difference ≤3, 'unknown' for no match), average engagement (retweets+likes mean, rounded to 4 decimals, missing=0). Assign user_group_id (starting from 1). Output to output.csv: user_group_id, Username, Bot_Label_Mode, Gender_Mode, Consecutive_Block_Length, Avg_Engagement, sorted by user_group_id ascending.

Data Labeling and Structured Data Handling

Change Detection and Contiguity Analysis

Numerical Operations and Type Conversion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['user_group_id', 'Username', 'Bot_Label_Mode', 'Gender_Mode', 'Consecutive_Block_Length', 'Avg_Engagement'], thresholds={'Avg_Engagement': None})


Question: A social media research team is investigating whether user engagement patterns differ significantly across subgroups defined by bot status, gender identity, and hate speech content. Using datasets containing bot classification labels with retweet metrics, gender classifications with engagement counts, and hate speech annotations with vote totals, perform statistical tests to identify significant differences. Conduct three separate analyses: (1) compare retweet engagement between human users (Bot Label = 0) and bot accounts (Bot Label = 1); (2) compare engagement counts between male and female users (gender:confidence ≥ 0.5) with confirmed profiles; (3) compare total annotator votes across three content categories (hate speech, offensive language, and neutral content). For each comparison, test normality assumptions (Shapiro-Wilk (n ≤ 5000) or Kolmogorov-Smirnov (n > 5000)),test homogeneity of variances (Levene), and select appropriate test (t-test or Mann-Whitney U test) accordingly. Apply Bonferroni correction for the three pairwise comparisons in the hate speech analysis (adjusted α = 0.05/3 ≈ 0.0167). Output the results to 'output.json' containing a JSON array in the format [{'dataset_comparison': ..., 'test_type': ..., 'p_value': ...}, ...] for all significant differences (p < 0.05 after correction), where dataset_comparison is exactly one of 'bot_human_vs_bot', 'gender_male_vs_female', 'hate_hate_vs_offensive', 'hate_hate_vs_neither', or 'hate_offensive_vs_neither', and test_type is exactly one of 't-test' or 'Mann-Whitney U'.

Probability Distributions and Statistical Modeling

Multiple Comparisons and Hypothesis Testing

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'dataset_comparison': None, 'test_type': None, 'p_value': None}, matched_keys=['dataset_comparison'])


Question:
A customer experience team at an aviation analytics firm is analyzing airline customer feedback to understand sentiment distribution and identify primary drivers of negative experiences. Using a dataset containing customer tweets with airline names, sentiment classifications, and tweet texts, perform comprehensive sentiment analysis: (1) extract all tweet records with airline identifiers and sentiment labels; (2) calculate sentiment distribution statistics for each airline, computing both absolute counts and percentages; (3) analyze negative feedback to identify primary complaint categories by matching tweet content against keywords:
- delay: delay, delayed, late, wait, waiting, hours, waited
- service: service, rude, staff, customer, attitude, unhelpful, rudest
- baggage: baggage, luggage, bag, lost, damage, suitcase, damaged, lost
- cancel: cancel, cancelled, cancellation, flight cancelled, canceled
- seat: seat, seating, overbook, overbooking, upgrade, cramped
- food: food, meal, drink, hungry, refreshment, thirsty
- other: tweets containing none of the above keywords; (4) output sentiment statistics to 'output.csv' with columns airline, sentiment (exactly one of: positive, negative, or neutral), count, and percentage rounded to exactly two decimal places; (5) output negative reason analysis to 'negative_reasons.csv' with columns airline, reasons, and count. Ensure all text analysis is case-insensitive and tweets may belong to multiple complaint categories.

Normalization and Percentile Calculations

Data Structure and Dictionary Operations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['airline', 'sentiment', 'count', 'percentage'], thresholds={'percentage': None})

compare_csv(output_file_name='negative_reasons.csv', gold_file_name='negative_reasons_gold.csv', ignore_order=True, thresholds={'count': None})


Question:
A social media analytics team needs to build a user behavior profiling system to analyze characteristics of different Twitter user types.

Requirements:

1. Create an in-memory database for efficient data processing.

2. From the bot detection dataset, categorize users by combining bot/human label (0=human, 1=bot) and verification status. For each category, compute:
   - User count
   - Average follower count, average retweet count, and average mention count (all rounded to 2 decimal places)
   - Engagement rate = (avg retweet count / max(avg follower count, 1)) × 1000, rounded to 2 decimal places

3. From the gender classifier dataset, compute user count per gender category, only including records with gender confidence score > 0.8.

4. From the hate speech dataset, compute user count per content type category, with class mapping: 0 → hate_speech, 1 → offensive, 2 → neither.

Output (output.csv):
Columns: category, user_count, avg_follower_count, avg_retweet_count, avg_mention_count, engagement_rate, data_source
- category format: '{user_type}_{verified_status}' for bot detection data (e.g., 'human_verified'), 'gender_{type}' for gender data, 'content_{type}' for hate speech data
- avg_follower_count, avg_retweet_count, avg_mention_count, engagement_rate are null for non-bot-detection rows
- data_source: 'bot_detection', 'gender_classifier', or 'hate_speech'
- Sort by data_source ascending, then category ascending

Batch Processing and Performance Optimization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['category', 'user_count', 'avg_follower_count', 'avg_retweet_count', 'avg_mention_count', 'engagement_rate', 'data_source'], thresholds={'avg_follower_count': None, 'avg_retweet_count': None, 'avg_mention_count': None, 'engagement_rate': None})


Question:
Analyze Twitter and YouTube user activity patterns and content characteristics. Time periods: 0-6/6-12/12-18/18-24 hours.

Output:

1. twitter_time_stats.csv (Twitter bot/human by time period): user_type (bot/human), time_period (early_morning/morning/afternoon/evening), metric_name (tweet_count/percentage), metric_value

2. twitter_day_stats.csv (Twitter bot/human by day type): user_type (bot/human), day_type (weekday/weekend), metric_name (tweet_count/percentage), metric_value

3. twitter_content_stats.csv (Twitter hate speech classification across three content categories): type (hate/offensive/neural), time_period (all), metric_name (tweet_count/percentage), metric_value

4. youtube_stats.csv (YouTube videos):time_period (all), metric_name (video_count/percentage), metric_value

5. youtube_vs_twitter_comparison.csv: time_period (4 periods), percentage_diff

Timestamp Conversion and Time Manipulation

Data Transformation and Column Manipulation

Custom Function Development for Time Logic

Business Day Logic and Custom Calculations

Normalization and Percentile Calculations

Incremental and Comparative Calculations

compare_csv(output_file_name='twitter_time_stats.csv', gold_file_name='twitter_time_stats_gold.csv', ignore_order=True, specified_columns=['user_type', 'time_period', 'metric_name', 'metric_value'], thresholds={'metric_value': None})

compare_csv(output_file_name='twitter_day_stats.csv', gold_file_name='twitter_day_stats_gold.csv', ignore_order=True, specified_columns=['user_type', 'day_type', 'metric_name', 'metric_value'], thresholds={'metric_value': None})

compare_csv(output_file_name='twitter_content_stats.csv', gold_file_name='twitter_content_stats_gold.csv', ignore_order=True, specified_columns=['type', 'time_period', 'metric_name', 'metric_value'], thresholds={'metric_value': None})

compare_csv(output_file_name='youtube_stats.csv', gold_file_name='youtube_stats_gold.csv', ignore_order=True, specified_columns=['time_period', 'metric_name', 'metric_value'], thresholds={'metric_value': None})

compare_csv(output_file_name='youtube_vs_twitter_comparison.csv', gold_file_name='youtube_vs_twitter_comparison_gold.csv', ignore_order=True, specified_columns=['time_period', 'percentage_diff'], thresholds={'percentage_diff': None})


Question:
Compare **bot vs human** Twitter engagement using the **bot-detection** accounts file and the **tweet activity** file. **Inner-join** on **normalized username**; dedupe bot labels to **one row per normalized username** by majority Bot Label: assign **bot** only if **1** appears more often than **0**; ties are **human**.

Per **user_type**, compute **mean likes**, **mean retweets**, **mean likes / mean retweets** (ratio to **2** decimals), and **mean tweet character length**. Per user, sum **likes + retweets**; take **top 10** bots and **top 10** humans by that sum.

Emit **output.csv** with **28** rows and columns **user_type**, **metric_type**, **metric_value**, **description**. **Block order (reference pipeline):** for **avg_likes**, **avg_retweets**, **ratio**, **avg_length** emit **human** then **bot**; then **10** `top_users` rows for **bot**; then **10** for **human**. **metric_value** uses **2** decimals for aggregates; top-user rows look like **`username(12345)`** (integer total engagement in parentheses). Descriptions are short English sentences mirroring the metric.

Categorical vs Numerical Type Identification

Data Transformation and Column Manipulation

Column-wise Transformations and Aggregation

Incremental and Comparative Calculations

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['user_type', 'metric_type', 'metric_value'])


Question:
A social media analytics team needs to analyze engagement efficiency patterns across different follower tiers on Twitter.

Analyze the engagement efficiency of bot-labeled tweet records (bot label=1) and human-labeled tweet records (bot label=0) across follower tiers. After matching the two files by normalized Username, perform the analysis on the resulting joined records. Exclude joined records with 0 followers.

Follower tiers: small (<1000), medium (1000-10000), large (>10000)

Engagement efficiency metrics (per 1000 followers):
- Retweet efficiency = (retweets / follower count) * 1000
- Like efficiency = (likes / follower count) * 1000
- Total efficiency = ((retweets + likes) / follower count) * 1000

For each user_type and follower_tier combination, calculate average efficiencies and high efficiency ratio (percentage of joined records whose total efficiency exceeds the median total efficiency of their user_type group, format: X.XX%).

Output columns: user_type, follower_tier, avg_retweet_efficiency, avg_like_efficiency, avg_total_efficiency, high_efficiency_ratio
Sort by user_type (human before bot), then follower_tier (small → medium → large). All efficiency values rounded to 2 decimal places.

Data Labeling and Structured Data Handling

Arithmetic Transformations and Normalization

Threshold-Based Categorization or Filtering

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['user_type', 'follower_tier', 'avg_retweet_efficiency', 'avg_like_efficiency', 'avg_total_efficiency', 'high_efficiency_ratio'], thresholds={'avg_retweet_efficiency': None, 'avg_like_efficiency': None, 'avg_total_efficiency': None})


Question:
Analyze engagement behavior patterns between bot and human users on Twitter. User account data contains bot detection labels (1=bot, 0=human), and tweet activity data contains engagement metrics. Match users between the two datasets by username (normalized to lowercase with whitespace stripped).

For each user type (bot vs human), compute:
- Average retweets and likes per tweet
- Average engagement rate, defined as (retweets + likes) / (follower count + 1e-8)
- Average account age in days from account creation to 2026-06-12
- Total tweet count and unique user count
- Average engagement per 1000 followers, defined as engagement rate × 1000
- Efficiency category: for each record, classify engagement per 1000 followers as "high" (>100), "medium" (>50), or "low" (≤50), then take the mode per user type

Output a CSV named output.csv with exactly 2 rows (human first, then bot), columns: user_type, avg_retweets, avg_likes, avg_engagement_rate, avg_account_age_days, total_tweets, unique_user_count, engagement_per_1000, efficiency_category. All numeric values rounded to 4 decimal places.

Numerical Operations and Type Conversion

Data Labeling and Structured Data Handling

Preprocessing and File Structure Adjustments

Arithmetic Transformations and Normalization

Time Difference and Gradient Calculation

Column-wise Transformations and Aggregation

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['user_type', 'avg_retweets', 'avg_likes', 'avg_engagement_rate', 'avg_account_age_days', 'total_tweets', 'unique_user_count', 'engagement_per_1000', 'efficiency_category'], thresholds={'avg_retweets': None, 'avg_likes': None, 'avg_engagement_rate': None, 'avg_account_age_days': None, 'total_tweets': None, 'unique_user_count': None, 'engagement_per_1000': None})


Question:
Cross-platform **engagement calibration** (Twitter, YouTube, Instagram) from the four project tables.

**Filters:** Twitter — drop rows whose tweet text appears among **hate-speech–labeled** posts in the hate-speech corpus (**class 0** there), keep **non-bot** accounts with **> 1000** followers. YouTube — **views > 100,000** and **ratings not disabled**. Instagram — parse **k/m/b** follower strings, keep **≥ 1,000,000** followers.

**Rates:** Twitter **retweets / (followers + 1)**; YouTube **(likes + comments) / (views + 1)**; Instagram **parsed avg likes / numeric followers**.

**Per-platform transforms:** Twitter **MinMax** on rates; YouTube **StandardScaler**; Instagram **log1p** then **MinMax**. **Concatenate** all transformed rates, **average rank** (`rankdata` **average**), map ranks to **[0,1]** via **(rank − 1)/(N − 1)**; split back and take the **mean** of the **Instagram** slice.

Save **output.csv** with a single column **instagram_normalized_mean** (**4** decimal places).

Incremental and Comparative Calculations

Arithmetic Transformations and Normalization

Normalization and Percentile Calculations

Mathematical and Statistical Computations

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=None, calculate_columns=['instagram_normalized_mean'], metric='mae', lower_bound=0.0, upper_bound=0.01)


Question:
A social media integrity team is evaluating a user credibility scoring model. For each user, compute a credibility score using the L-BFGS-B optimizer. Use an initial credibility score of 0.5 for every user, bounds of [0, 1], and maxiter = 1000. The optimizer attempts to minimize:

  composite_loss = mean((bot_label + toxicity_weight + anomaly_flag) × credibility_score)

Where for each user:
- bot_label: the user's bot detection label
- anomaly_flag: 1 if the user's total retweet count (aggregated across all available data sources) exceeds the 95th percentile, 0 otherwise
- toxicity_weight: determined by matching the user's tweet content against a toxicity-labeled dataset using TF-IDF cosine similarity (max_features=1000, stop_words='english', ngram_range=(1,2)); assign the toxicity class of the best match if cosine similarity ≥ 0.3, otherwise default to neutral (class 2). Weight mapping: class 0 (hate speech) = 1.0, class 1 (offensive) = 0.7, class 2 (neutral) = 0.0

After running the optimizer, calculate the final normalized composite loss (mean across all users) from the credibility scores, rounded to 6 decimal places.

Output Format (output.csv): single row with column normalized_composite_loss

Threshold-Based Categorization or Filtering

Data Labeling and Structured Data Handling

Mathematical Foundations and Algorithm Understanding

Efficient Data Structures and Algorithms

Batch Processing and Performance Optimization

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=None, calculate_columns=['normalized_composite_loss'], metric='mae', lower_bound=0.0, upper_bound=0.0001)


Question:
A social media research team is building a cross-platform user credibility model and needs to identify the most discriminative numerical features from multiple Twitter datasets through hierarchical feature selection.

Load the provided datasets and perform cross-dataset hierarchical feature analysis. For each dataset, identify relevant numerical features and classify them into two semantic types: engagement/count metrics and confidence scores. Apply z-score normalization (StandardScaler) for count features and min-max scaling to [0,1] (MinMaxScaler) for confidence features. Fill missing count features with 0 and missing confidence scores with 0.5. Convert string-type numeric columns to float before normalization.

Apply PCA (random_state=42) to each dataset separately, retaining components explaining 95% of variance. Compute feature importance using: importance_j = Σ_k(|loading_kj| × explained_variance_ratio_k) over retained components, then normalize each dataset's feature importance vector to sum to 1.

Aggregate feature importance across datasets, prefixing each feature with a short dataset name ('bot', 'gender', 'airline') in the format '{prefix}_{original_feature_name}'. Select the top-k most important features where k equals the average number of PCA components across datasets (rounded to nearest integer).

Save results to output.json with the following structure:
{
  "selected_features": ["prefix_featureA", "prefix_featureB", ...],  // top-k features (with dataset prefix) sorted alphabetically
  "n_components_per_dataset": {
    "bot_detection": <int>,
    "gender_classification": <int>,
    "airline_sentiment": <int>
  },
  "explained_variance_per_dataset": {
    "bot_detection": [<float>, ...],       // list of explained variance ratios for each retained component
    "gender_classification": [<float>, ...],
    "airline_sentiment": [<float>, ...]
  },
  "feature_importance_per_dataset": {
    "bot_detection": {"<original_feature_name>": <float>, ...},        // normalized importance scores (sum to 1) using original feature names without prefix
    "gender_classification": {"<original_feature_name>": <float>, ...},
    "airline_sentiment": {"<original_feature_name>": <float>, ...}
  },
  "overall_top_features": ["prefix_featureX", "prefix_featureY", ...]  // top-k features (with dataset prefix) ranked by descending aggregated importance
}

Domain-Specific and Contextual Imputation

Custom Functions for Missing Value Handling

Data Structure Understanding and Initialization

Feature Selection and Dimensionality Reduction

Mathematical and Statistical Computations

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'selected_features': None, 'n_components_per_dataset': {'bot_detection': None, 'gender_classification': None, 'airline_sentiment': None}, 'explained_variance_per_dataset': {'bot_detection': 0.02, 'gender_classification': 0.02, 'airline_sentiment': 0.02}, 'feature_importance_per_dataset': {'bot_detection': 0.05, 'gender_classification': 0.05, 'airline_sentiment': 0.05}, 'overall_top_features': None})


Question: A social media research team is investigating whether automated accounts show different engagement patterns on Twitter. Using the two provided datasets, standardize missing-value sentinels, load the cleaned tables into an in-memory SQLite database, and use an SQL join on case-normalized usernames to identify users appearing in both datasets. Then compute a 3×3 Pearson correlation matrix (and accompanying p-values) among: (1) bot label (0=human, 1=bot), (2) normalized engagement defined as (retweets + likes) / (follower count + 1), and (3) mention count. Restrict analysis to matched users with non-null values for all three variables. Output the results to output.json with format: {"correlation_matrix": [[...]], "p_values": [[...]], "columns": ["bot_label", "normalized_engagement", "mention_count"]}.

Custom Functions for Missing Value Handling

Arithmetic Transformations and Normalization

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'correlation_matrix': 1e-10, 'p_values': 1e-10, 'columns': None})


Question: A social media research team is investigating how user demographics influence engagement across platforms and whether automated accounts show different behavioral patterns. Using the provided datasets, conduct three statistical tests and save the results to output.csv with the following schema: feature (string), target (string), test_type (string), statistic (float), p_value (float), effect_size (float), significant (boolean). The three tests are: (1) assess whether user gender affects engagement metrics by comparing favorite counts between male and female users; (2) evaluate if video content categories impact normalized engagement (calculated as likes divided by views) using the Kruskal-Wallis H test; and (3) test whether automated accounts have different follower counts compared to human accounts using the Mann-Whitney U test. For all two-group comparisons, use the Mann-Whitney U test with Cliff's delta as the effect size; for multi-group comparisons, use the Kruskal-Wallis H test with epsilon-squared as the effect size. Use alpha=0.05 to determine statistical significance.

Statistical Testing for Feature-Target Evaluation

Normalization and Percentile Calculations

Interpreting and Communicating Statistical Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['test_type', 'significant', 'statistic', 'p_value', 'effect_size'], thresholds={'statistic': None, 'p_value': None, 'effect_size': None})


Question:
Analyze physical distinctiveness across 12 athlete groups: Olympic Basketball/Swimming/Weightlifting/Gymnastics, NBA Centers/Guards/Forwards/All, Soccer Goalkeepers/Defenders/Midfielders/Strikers. Convert height to cm, weight to kg. Calculate Cohen's d effect size, Kruskal-Wallis test, and KL divergence (10 bins) for height/weight differences. Skip groups with <10 samples. For all three metrics, treat distinctiveness as a group's average pairwise separation from the other eligible groups, with height and weight contributing equally; report the Kruskal-Wallis component on a negative-log p-value scale. Combine metrics: Total Score = Cohen's d × 0.4 + Kruskal-Wallis × 0.3 + KL × 0.3. Rank groups by total score.

For soccer players, group all players by overall_rating into 4 tiers: 'Soccer Goalkeepers' (overall_rating ≥ 85), 'Soccer Defenders' (80 ≤ overall_rating < 85), 'Soccer Midfielders' (75 ≤ overall_rating < 80), 'Soccer Strikers' (overall_rating < 75). Use the latest rating record per player.

Output: output.csv containing only the top 3 groups (Athlete_Group, Total_Distinctiveness_Score), output2.png showing top 3 groups (bar chart with title 'Top 3 Most Physiologically Distinct Athlete Groups', x-axis 'Total Distinctiveness Score', y-axis 'Athlete Group').

Preprocessing and File Structure Adjustments

Incremental and Comparative Calculations

Entropy and Log Probability Calculations

Multiple Comparisons and Hypothesis Testing

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['Total_Distinctiveness_Score'], thresholds={'Total_Distinctiveness_Score': None})


Question:
Analyze athlete physique evolution trends (1946-2016) for three groups: 'Elite_Olympic' (medalists), 'NBA_Active', 'Other'. Calculate yearly average height (cm) and weight (kg) for each group. For NBA_Active, use NBA players with a roster status in common_player_info.csv, assign each selected player to one representative year based on the 2016 reference season and season_exp, and keep that year within 1946-2016. Apply forward fill for sparse years, then backward fill to cover years before each group's first available data. Keep count as the number of original records observed for each group-year before filling, with 0 for filled-only years. Rank years 1946-2016.

Output: output1.csv (year, athlete_category, Height, Weight, count), output2.png with two panels: top shows height trends (title 'Evolution Trends of Average Athlete Height (1946-2016)', x-axis 'Year', y-axis 'Average Height (cm)', lines for each group with markers 'o', legend labels formatted as '{category} Avg Height' e.g. 'Elite_Olympic Avg Height'), bottom shows weight trends (title 'Evolution Trends of Average Athlete Weight (1946-2016)', x-axis 'Year', y-axis 'Average Weight (kg)', lines with markers 's', legend labels formatted as '{category} Avg Weight' e.g. 'Elite_Olympic Avg Weight').

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output1.csv', gold_file_name='result1.csv', ignore_order=True, specified_columns=['year', 'athlete_category', 'Height', 'Weight'], thresholds={'Height': None, 'Weight': None})


Question:
Compare physique across NBA, Soccer, and Olympic athletes. Convert to cm/kg, filter height (100-250cm) and weight (30-200kg). Calculate: (1) mean and std for each sport's height/weight, (2) ANOVA F-statistic and p-value for height/weight, (3) Cohen's d for pairwise comparisons (NBA-Soccer, NBA-Olympic, Soccer-Olympic).

Output: output.csv with columns metric, value (float, rounded to 4 decimal places). Metrics: {Sport}_H_Mean, {Sport}_H_Std, {Sport}_W_Mean, {Sport}_W_Std, ANOVA_H_F, ANOVA_H_P, ANOVA_W_F, ANOVA_W_P, D_H_{Sport1}_{Sport2}, D_W_{Sport1}_{Sport2}.

Dynamic Data Transformation and Insertion

Numerical Operations and Type Conversion

Statistical Calculations and Descriptive Statistics

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['metric'], calculate_columns=['value'], metric='mse', lower_bound=0, upper_bound=0.01)


Question:
Compare elite athletes across Olympic (team sport medalists), NBA (vertical leap Z-score > 1.0 AND sprint Z-score < -1.0), and Soccer (height and weight percentiles > 0.8). Fill missing height/weight with median within sport/gender (Olympic) or position (NBA) groups, ensure all heights in cm, weights in kg. Calculate Z-scores within each group, then compute: (1) mean and std for each group's height/weight, (2) ANOVA F-statistic and p-value, (3) Cohen's d for pairwise comparisons (Olympic-NBA, Olympic-Soccer, NBA-Soccer).

Output: output.csv with columns metric, value (float, rounded to 4 decimal places). Metrics: {Group}_Height_Mean, {Group}_Height_Std, {Group}_Weight_Mean, {Group}_Weight_Std, ANOVA_Height_F, ANOVA_Height_P, ANOVA_Weight_F, ANOVA_Weight_P, CohensD_H_{Group1}_{Group2}, CohensD_W_{Group1}_{Group2}.

Data Conversion and Post-Loading Processing

Normalization and Percentile Calculations

Mathematical and Statistical Computations

Dynamic Data Transformation and Insertion

Statistical Calculations and Descriptive Statistics

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['metric'], calculate_columns=['value'], metric='mse', lower_bound=0, upper_bound=0.05)


Question:
Cross-Source Athlete Physique Database Construction

Integrate Olympic and NBA athlete physique data into unified database. Olympics: Summer events only, drop duplicates by ID (keep first), mark missing with height_missing/weight_missing. NBA: merge player info with draft combine stats (prioritize combine data), convert units (inches×2.54→cm, pounds×0.453592→kg). Cross-source matching: normalize names (lowercase, remove spaces/punctuation), propagate NBA physique to matching Olympics basketball players. Imputation: Olympics fill by sport median, NBA fill by position median, remaining by global median. Output: athlete_id, source, sport, height, weight, height_missing (0=available, 1=missing), weight_missing.

athlete_events.csv
NBA Database/common_player_info.csv
NBA Database/draft_combine_stats.csv
NBA Database/player.csv

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['athlete_id', 'source', 'sport', 'height', 'weight', 'height_missing', 'weight_missing'], thresholds={'height': None, 'weight': None, 'height_missing': None, 'weight_missing': None})


Question:
Cross-Sport Athlete Physique Comparison with Performance Stratification

Analyze height/weight distributions across NBA, Olympics basketball, and Soccer athletes with performance stratification. Performance metrics: NBA (avg_pts from game stats), Olympics (medal_rate = medals/events), Soccer (mean rating). Stratify by terciles: q33 and q66 thresholds → high/medium/low tiers. Convert units to height (cm) and weight (kg). Visualization to athlete_physique_analysis.png: 3×3 facet grid scatter plot (rows: sport, columns: performance tier) showing height vs weight. Output output.csv: sport (NBA/Olympics/Soccer), height_mean, height_median, weight_mean, weight_median, sample_size.

Statistical Calculations and Descriptive Statistics

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['sport', 'height_mean', 'height_median', 'weight_mean', 'weight_median', 'sample_size'])


Question:
NBA Physical Attributes Impact on Career Performance

Analyze correlation between NBA players' physical attributes (height, wingspan, vertical leap, sprint speed) and career performance metrics (draft round, career span). Filter valid careers (0-25 years). Composite score = -correlation_with_draft + correlation_with_career (lower draft round is better). Normalize weights based on absolute composite scores.

Output:
1. output.csv: attribute, correlation_with_draft, p_value_draft, correlation_with_career, p_value_career, composite_score, normalized_weight
2. physical_attributes_analysis.png (2×2):
   - Panel 1 (Heatmap): Correlation matrix of Physical Attributes vs Draft Round/Career Span
   - Panel 2 (Bar Chart): Normalized Weights of Physical Attributes
   - Panel 3 (Scatter): Height (inches) vs Draft Round
   - Panel 4 (Scatter): Max Vertical Leap (inches) vs Career Span (years)

NBA Database/common_player_info.csv
NBA Database/draft_combine_stats.csv
NBA Database/draft_history.csv

Data Structure Understanding and Initialization

Numerical Operations and Type Conversion

Column-specific or Feature-wise Processing

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['correlation_with_draft', 'correlation_with_career', 'normalized_weight'], thresholds={'correlation_with_draft': None, 'correlation_with_career': None, 'normalized_weight': None})


Question:
Cross-League Athlete Height Distribution Comparison with KDE

Compare height distributions of NBA, Olympics basketball, and European soccer athletes using Kernel Density Estimation. Convert units to height (cm) and weight (kg). Filter outliers (150 < height < 250). Visualization: height_distribution_kde.png with normalized histogram (bins=50) background and Gaussian KDE curves (500 points) for three groups. Mark peak positions with vertical dashed lines and circle markers. Output output.csv: nba_height_peak_cm, nba_mean_height, nba_std_height, nba_sample_size, olympics_mean_height, olympics_std_height, olympics_sample_size, soccer_mean_height, soccer_std_height, soccer_sample_size

Preprocessing and File Structure Adjustments

Statistical Plotting and Density Estimation

Normalization and Percentile Calculations

Functional Data Handling and Iterative Plotting

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['nba_height_peak_cm', 'nba_mean_height', 'nba_sample_size'], thresholds={'nba_height_peak_cm': None, 'nba_mean_height': None, 'nba_sample_size': None})


Question:
Cross-Sport Athlete Physique Statistical Inference

Perform statistical analysis comparing NBA, Olympics basketball, and Soccer athlete physical characteristics (height, weight) with performance correlations. Convert units to height (cm) and weight (kg). Filter outliers (150 < height < 250, 50 < weight < 200). Effect size classification: Small (<0.5), Medium (0.5-0.8), Large (≥0.8) based on Cohen's d. ANOVA significance: p < 0.05 threshold. Correlations: NBA (height/weight vs draft position), Soccer (height/weight vs overall performance rating). Output:
1. output.csv: total_effect_sizes, large_effects (int), medium_effects, small_effects, significant_anova, total_correlations
2. effect_sizes.csv (NBA vs Olympics, NBA vs Soccer, Olympics vs Soccer): metric (Height/Weight), group1, group2, cohens_d, effect_size (Small/Medium/Large)
3. anova_results.csv: metric, f_statistic, p_value
4. correlations.csv: sport, metric, correlation, p_value

Preprocessing and File Structure Adjustments

Data Transformation and Column Manipulation

Numerical Operations and Type Conversion

Mathematical and Statistical Computations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['total_effect_sizes', 'large_effects', 'medium_effects', 'small_effects', 'significant_anova', 'total_correlations'], thresholds={'total_effect_sizes': None, 'large_effects': None, 'medium_effects': None, 'small_effects': None, 'significant_anova': None, 'total_correlations': None})

compare_csv(output_file_name='effect_sizes.csv', gold_file_name='effect_sizes_gold.csv', ignore_order=True, thresholds={'cohens_d': None})

compare_csv(output_file_name='anova_results.csv', gold_file_name='anova_results_gold.csv', ignore_order=True, thresholds={'f_statistic': None, 'p_value': None})

compare_csv(output_file_name='correlations.csv', gold_file_name='correlations_gold.csv', ignore_order=True, thresholds={'correlation': None, 'p_value': None})


Question:
Analyze NBA team offensive efficiency by comparing home scoring vs away points allowed.

Data preprocessing:
- Validate team abbreviations against team.csv
- Filter invalid scores
- Cross-verify scores between records and detailed scoring data (tolerance: ±1 point)
- Filter inconsistent records

Offensive efficiency calculation:
- For each team: home points scored = sum(pts_home when team is home)
- For each team, define efficiency ratio = total home points scored / total away points allowed
- If away_points_allowed = 0, efficiency ratio = 0
- Round all ratios to 4 decimal places
- Rank teams by efficiency ratio descending

Output team_offensive_efficiency.csv with schema: team_abbreviation (uppercase), pts_scored_home, home_games, pts_allowed_away, away_games, offensive_efficiency_ratio.

Output output.csv: total_teams, total_games_analyzed, avg_efficiency_ratio, median_efficiency_ratio, top_team, top_efficiency, bottom_team, bottom_efficiency

NBA Database/game.csv
NBA Database/team.csv
NBA Database/line_score.csv

Row-wise and Column-wise Logical Evaluation

Column-wise Transformations and Aggregation

Arithmetic Transformations and Normalization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['total_teams', 'avg_efficiency_ratio', 'top_team'], thresholds={'total_teams': None, 'avg_efficiency_ratio': None})

compare_csv(output_file_name='team_offensive_efficiency.csv', gold_file_name='team_offensive_efficiency_gold.csv', ignore_order=True, thresholds={'pts_scored_home': None, 'home_games': None, 'pts_allowed_away': None, 'away_games': None, 'offensive_efficiency_ratio': None})


Question:
Compare Olympic, NBA, and soccer athlete height/weight statistics.

Requirements:
- Load data and extract height/weight; convert to metric (cm, kg); remove missing values
- Calculate descriptive stats (mean, median, std, Q1/Q3) per sport
- Perform pairwise t-tests (height & weight) with Cohen's d effect sizes
- Interpret effect sizes: Negligible(<0.2)<Small(<0.5)<Medium(<0.8)<Large
- Generate box plots and mean comparison charts
- All numeric values in output should be rounded to 4 decimal places

Output files:
1. descriptive_stats.csv: sport ('Olympics', 'NBA', or 'Soccer'), height_mean, height_median, height_std, height_q25, height_q75, weight_mean, weight_median, weight_std, weight_q25, weight_q75, sample_size

2. ttest_results.csv: comparison ('Sport1 vs Sport2'), metric ('Height' or 'Weight'), t_statistic, p_value, cohens_d, significant (whether p < 0.05, 'Yes' or 'No')

3. output.csv: total_comparisons, significant_comparisons, significance_rate, large_effect_count, medium_effect_count, small_effect_count, negligible_effect_count, most_significant_comparison, largest_effect_comparison, largest_effect_size

4. physical_comparison.png: Box plots (height/weight by sport) + bar charts (mean comparison)

Statistical Calculations and Descriptive Statistics

Mathematical and Statistical Computations

Statistical Testing for Feature-Target Evaluation

Multiple Comparisons and Hypothesis Testing

Numerical Comparison and Proximity Checks

Interpreting and Communicating Statistical Results

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['total_comparisons', 'significant_comparisons', 'largest_effect_size'], thresholds={'total_comparisons': None, 'significant_comparisons': None, 'largest_effect_size': None})

compare_csv(output_file_name='descriptive_stats.csv', gold_file_name='descriptive_stats_gold.csv', ignore_order=True, specified_columns=['sport', 'height_mean', 'weight_mean', 'sample_size'], thresholds={'height_mean': None, 'weight_mean': None, 'sample_size': None})

compare_csv(output_file_name='ttest_results.csv', gold_file_name='ttest_results_gold.csv', ignore_order=True, specified_columns=['comparison', 'metric', 'p_value', 'cohens_d', 'significant'], thresholds={'p_value': None, 'cohens_d': None})


Question:
A sports analytics firm is conducting a cross-sport physique benchmarking study to understand how elite athletes' body profiles differ across Olympic basketball, Olympic football, NBA, and European club soccer during 2008–2016. Using the provided datasets, perform the following analysis and produce the specified output:

OUTPUT FILE AND SCHEMA:
Save the analysis results to 'output.csv' with exactly these columns:
- sport_category (string): 'Olympic Basketball', 'Olympic Football', 'NBA', or 'European Soccer'
- height_mean (float): mean height in cm
- height_std (float): standard deviation of height in cm
- weight_mean (float): mean weight in kg
- weight_std (float): standard deviation of weight in kg
- sample_count (int): number of athlete-event records in each group (no deduplication across Olympic years or events)
- height_anova_significance (string): 'p < 0.001' if p-value < 0.001, otherwise 'p = X.XXX' (3 decimal places)
- weight_anova_significance (string): 'p < 0.001' if p-value < 0.001, otherwise 'p = X.XXX' (3 decimal places)

REQUIRED ANALYSIS:
- Filter and consolidate data from the four sport categories for the 2008-2016 period
- Standardize physical attribute units across datasets (height in cm, weight in kg)
- Compute summary statistics (mean, standard deviation, sample count) for height and weight by sport category
- Perform one-way ANOVA tests across the four groups for both height and weight
- Format significance values according to the specified schema

FINAL OUTPUT: The output.csv file with the specified schema containing the physique benchmarking results.

Preprocessing and File Structure Adjustments

Numerical Operations and Type Conversion

Mathematical and Statistical Computations

Interpreting and Communicating Statistical Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['sport_category', 'height_mean', 'height_std', 'weight_mean', 'weight_std', 'sample_count', 'height_anova_significance', 'weight_anova_significance'], thresholds={'height_mean': 0.005, 'height_std': 0.005, 'weight_mean': 0.005, 'weight_std': 0.005, 'sample_count': None})


Question:
A sports analytics firm is evaluating whether physical attributes measured at pre-draft combines can predict NBA career longevity when leveraging cross-sport transfer learning from Olympic athlete data.

OUTPUT FILE AND SCHEMA:
Save the model evaluation results to 'output.json' with exactly this structure:
{
  "mae": (float) mean absolute error from 5-fold cross-validation,
  "mse": (float) mean squared error from 5-fold cross-validation,
  "mape": (float) mean absolute percentage error from 5-fold cross-validation,
  "r2": (float) R-squared value from 5-fold cross-validation
}

REQUIRED ANALYSIS:
- Train a StandardScaler (physique normalization encoder) on global Olympic athlete height and weight distributions
- Apply this encoder to transform NBA pre-draft combine physical measurements (height with shoes and weight); additionally include the player's is_active flag from player.csv as an untransformed feature in the Ridge regression model
- Fit a Ridge regression model with alpha=1.0 to predict NBA career length (difference between final season year and debut season year)
- Report MAE, MSE, MAPE, and R² evaluated via 5-fold cross-validation on the transformed NBA data
- Link players across datasets by matching first and last names

FINAL OUTPUT: The output.json file with the specified schema containing the four evaluation metrics.

Data Structure Handling (Dictionaries, Lists)

Data Structure Understanding and Initialization

Preprocessing and File Structure Adjustments

Arithmetic Transformations and Normalization

Column-specific or Feature-wise Processing

Data Transformation and Column Manipulation

compare_json('output.json', 'result.json', thresholds={'mae': 0.05, 'mse': 0.05, 'mape': 0.05, 'r2': 0.10})