Question: An agricultural policy research institution is assessing the degree of match between the optimal wheat cultivation conditions in high-income countries and the actual market outcomes, in order to provide reference data for trade strategies. Identify all soil-climate feature combinations labeled as "wheat", which must meet agricultural standards: pH value between 5.5 and 7.0, nitrogen (N) content greater than 0.2, and summer precipitation (PRECTOTCORR-Su) exceeding winter precipitation (PRECTOTCORR-W). From these favorable feature combinations, calculate the median values of nitrogen, phosphorus, and potassium, as well as the annual temperature difference (defined as the maximum value of T2M_MAX in all seasons minus the minimum value of T2M_MIN in all seasons). Additionally, extract the wheat yield records and combine them with the national income classification, retaining only high-income countries (achieved through fuzzy matching of country names with a similarity threshold of 80). Then, merge this data with the price data of goods labeled as "wheat" (case-insensitive) and summarize the average prices by country-year. Finally, generate a unified data frame containing country, year, wheat_yield_hg_per_ha, avg_wheat_price, median_n, median_p, median_k, and median_temp_range (where wheat_yield_hg_per_ha is wheat yield in grams per hectare, avg_wheat_price is the average wheat price in local currency, median_n/median_p/median_k are the median values of nitrogen/phosphorus/potassium, and median_temp_range is the median annual temperature difference), and save it as the "output.csv" file.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Row-wise and Column-wise Logical Evaluation

Logical Operators for Combining Conditions

Mathematical and Statistical Computations

Data Conversion and Post-Loading Processing

Data Structure and Dictionary Operations

Data Structure Handling (Dictionaries, Lists)

Column-wise Transformations and Aggregation

Preprocessing and File Structure Adjustments

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['country', 'year', 'wheat_yield_hg_per_ha', 'avg_wheat_price', 'median_n', 'median_p', 'median_k', 'median_temp_range'])


Question: An international agricultural policy research institution wants to assess whether countries whose highest-yielding crops do not match soil-climate recommendations tend to use more pesticides. Using the provided datasets for the period 1990–2013, perform the following analysis. From the crop recommendation dataset, standardize crop names by converting to lowercase, removing parenthetical content, and stripping the trailing letter 's', then determine the top 3 most frequently occurring crop types as the soil-climate suitability reference set. For each country-year pair in the yield dataset, identify the crop with the highest yield value. Classify each country-year as 'aligned' if its top-yielding crop is among the top 3 recommended crops, or 'misaligned' otherwise. For country matching, use deterministic exact keys: normalize names (lowercase, remove parenthetical content and punctuation) and reconcile each country in the yield and pesticide data to its corresponding Natural Earth NAME entry. Exclude historical or aggregate country entities. In particular, when a country is split into multiple sub-entities, map each sub-entity to its own Natural Earth NAME entry as a separate entity. Merge yield and pesticide data on the resulting country key and year, and retrieve income only from a one-to-one Natural Earth NAME match. For each country-year, compute pesticide usage intensity as the sum of all pesticide amounts for that country-year divided by the top-yielding crop's yield value. Retrieve each country's World Bank income classification from the INCOME_GRP field in the Natural Earth dataset. Drop records with missing pesticide intensity or income classification. Calculate the Pearson correlation coefficient between the binary mismatch indicator (1 for 'misaligned', 0 for 'aligned') and pesticide usage intensity across all retained observations. Group the data by income classification and alignment status, count the observations in each group, and attach the global correlation value to every row. Save the results to output.csv with columns: income_group (World Bank income classification), alignment_status ('aligned' or 'misaligned'), count (number of country-year observations per group), and correlation (the single global Pearson correlation coefficient, repeated for all rows).

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Preprocessing and File Structure Adjustments

Index-based Alignment and Set Operations

Column-wise Transformations and Aggregation

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['income_group', 'alignment_status', 'count', 'correlation'], thresholds={3: 0.02})


Question: A global agricultural policy research institution aims to explore whether the soil and climate characteristics that have statistical significance in predicting crop recommendations can also distinguish between crops that have a direct impact on food market prices and those that do not. Firstly, by using a multi-class logistic regression model (multi_class='multinomial', solver='lbfgs', max_iter=1000, random_state=42) with the following 27 features: ['Ph', 'K', 'P', 'N', 'Zn', 'S', 'QV2M-W', 'QV2M-Sp', 'QV2M-Su', 'QV2M-Au', 'T2M_MAX-W', 'T2M_MAX-Sp', 'T2M_MAX-Su', 'T2M_MAX-Au', 'T2M_MIN-W', 'T2M_MIN-Sp', 'T2M_MIN-Su', 'T2M_MIN-Au', 'PRECTOTCORR-W', 'PRECTOTCORR-Sp', 'PRECTOTCORR-Su', 'PRECTOTCORR-Au', 'WD10M', 'GWETTOP', 'CLOUD_AMT', 'WS2M_RANGE', 'PS'] and applying the backward elimination method (only retaining features with Wald p-values < 0.05), the most predictive soil and climate variables are determined. Additionally, a time series causal analysis is conducted, and it is determined which crop yield changes will causally affect their corresponding food prices. Then, for each feature selected from the backward elimination step, the Kruskal-Wallis H test (α = 0.05) is used to test whether the distribution of each selected feature has a significant difference between crops that are identified as having a causal relationship with price changes and those that do not. Save the results to output.csv containing feature, h_statistic, p_value, and is_significant (where feature is the soil or climate variable name, h_statistic is the Kruskal-Wallis H test statistic, p_value is the p-value from the test, and is_significant indicates whether the result is statistically significant at α = 0.05).

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Crop Yield Prediction Dataset/rainfall.csv

Column-specific or Feature-wise Processing

Feature Selection and Dimensionality Reduction

Statistical Testing for Feature-Target Evaluation

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['feature', 'h_statistic', 'p_value', 'is_significant'], thresholds={1: 0.02, 2: 0.02})


Question: A global agricultural policy research institution is developing a comprehensive agricultural suitability index (ASI) to guide sustainable agricultural investment for various countries and crops. Only samples with complete measurements of soil attributes (pH, phosphorus, nitrogen, potassium, zinc, sulfur) will be retained, and the records will be limited to the overlapping time window from 1990 to 2021. The crop names will be standardized (lowercase processing, removal of parentheses descriptions, and elimination of plural forms), and each standardized crop will be assigned to a topic category. Annual climate indicators (e.g., annual temperature range = maximum - minimum) will be calculated using seasonal weather variables. The filtered soil-weather data will be trained using Gaussian Naive Bayes classifier with var_smoothing = 0.3, and 20% of the data will be reserved for testing (random state = 42), and the calibrated class probabilities will be used as a component of the ASI. Finally, preserve the filtered soil-weather sample records and calculate the ASI for each individual retained record using a weighted average with the following weights: 0.4 for soil fertility score (predicted probability), 0.4 for climate suitability (standardized climate metrics), and 0.2 for pesticide score (inverse of pesticide usage), with a final score range of 0 to 1. Set Area to 'Global' and Year to 2020 for all records. Save the results to output.csv with one row per retained soil-weather sample, containing columns "Area", "Year", "Item", "crop_topic", and "ASI", where Item is the record's original crop label and crop_topic is the topic category for that record's standardized crop.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Numerical Stability and Naive Bayes Tuning

Arithmetic Transformations and Normalization

Normalization and Percentile Calculations

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['Area', 'Year', 'Item', 'crop_topic', 'ASI'])


Question: A global agricultural policy research institution is conducting an assessment of the data quality of its comprehensive agricultural dataset to ensure the reliability of the models for crop suitability and yield prediction. Through a comprehensive verification process, including: (1) checking for unreasonable values, such as pH values not within the range of [0, 14], negative nutrient levels (potassium, phosphorus, nitrogen, zinc, sulfur), and any seasonal temperature inconsistencies where T2M_MAX is less than T2M_MIN in any season; (2) verifying that crop yields and pesticide usage amounts are non-negative; (3) marking situations where the annual average temperature exceeds the biologically reasonable range ([-50°C, 60°C]). Generate a single pandas data frame containing all verification checks, with columns including "file" (the source file name), "column" (the column being checked), "check_type" (the type of validation performed, e.g., range_validity, non_negative, temp_consistency, temp_range), "anomaly_count" (the number of anomalous records found), and "description" (a brief explanation of the check criteria), and save the results to output.csv.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['file', 'column', 'anomaly_count'])


Question: An international agricultural policy research institution is studying whether countries with significant climate deviations from optimal crop-growing conditions face systemic food system pressure, evidenced by higher pesticide usage and elevated food prices. Using datasets covering crop-soil-weather recommendations, global food prices, and crop yield prediction data (including pesticides, temperature, and rainfall), perform the following analysis: From the crop recommendation dataset, compute each crop's optimal climate profile — the seasonal maximum temperature (max of T2M_MAX across seasons W, Sp, Su, Au) and total seasonal precipitation (sum of PRECTOTCORR across all seasons), aggregated by crop using the median. Only include samples with soil pH between 4.5 and 8.5, positive nutrient levels (N, P, K), and non-empty crop labels. Then combine yield, pesticide, actual temperature, and rainfall data by country and year, linking with the optimal climate profiles by crop. Integrate retail food prices (filtered to per-kilogram units); if multiple retail per-kilogram price records exist for the same country-year-crop combination, use their mean price before merging. For each country-year-crop record, calculate the absolute deviation of actual annual average temperature from the optimal seasonal peak temperature, and similarly for rainfall versus optimal total precipitation. Identify stress patterns: using the 75th percentile as the threshold for temperature and rainfall deviations, and the median as the threshold for pesticide usage and food price — a record is a stress event (1) if at least 2 of these 4 indicators exceed their respective thresholds, otherwise 0. Save the results to output.csv with columns: country, Year, crop, pesticide_value, yield_value, mp_price, temp_deviation, rainfall_deviation, stress_pattern.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Crop Yield Prediction Dataset/rainfall.csv

Preprocessing and File Structure Adjustments

Column-wise Transformations and Aggregation

Numerical Operations and Type Conversion

Incremental and Comparative Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['country', 'Year', 'crop', 'pesticide_value', 'yield_value', 'mp_price', 'temp_deviation', 'rainfall_deviation', 'stress_pattern'])


Question: An international agricultural research institution aims to assess the compatibility between soil climate suitability and the actual farming practices of major crops. For each different crop in the soil climate recommendation data, calculate the following: (1) the number of soil-climate samples recommended for the crop, (2) the annual temperature range (defined as the maximum of all seasonal maximum temperatures minus the minimum of all seasonal minimum temperatures), and (3) the annual total precipitation (sum of seasonal precipitation). For indicators (2) and (3), use median aggregation when summarizing by crop. Then, by matching standardized crop names, find the countries where each crop is actually planted and integrate the data for these countries and years. Finally, generate a DataFrame with each crop on a separate row, including: the original crop label, the three indicators based on the recommendations, the number of countries where the crop is planted, and the median values of pesticide usage, actual yield, national average temperature, and national average rainfall from all related country-year observation data. Save the results to output.csv.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Crop Yield Prediction Dataset/rainfall.csv

Data Transformation and Column Manipulation

Vectorization and Performance Optimization

Data Structure Handling (Dictionaries, Lists)

Conditional Logic and Row-wise Operations

Mathematical and Statistical Computations

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['label', 'n_samples', 'median_temp_range', 'median_precip', 'n_countries', 'median_pesticide', 'median_actual_yield', 'median_country_temp', 'median_country_rain'])


Question: An international agricultural research institution is assessing how soil characteristics and seasonal climate patterns affect crop yields, in order to provide a basis for policy recommendations. A unified dataset is being constructed, which links national-level crop yields with corresponding soil nutrient characteristics (nitrogen, phosphorus, potassium, zinc, sulfur, and pH) as well as derived annual climate indicators (annual temperature range calculated as the maximum of T2M_MAX in each season minus the minimum of T2M_MIN in each season, and total annual precipitation calculated as the sum of PRECTOTCORR in all four seasons). The crop names are standardized in each dataset. Then, using the standardized soil nutrients (pH, K, P, N, Zn, S) and derived climate features (annual temperature range and total annual precipitation) as independent variables, a linear regression prediction is made for crop yields (in hectograms per hectare), and a pandas data frame with two columns is returned: one column "features" lists each predictor variable (including an "intercept" row), and the other column "coefficients" shows the estimated regression coefficients for each variable. The results should be saved to output.csv with two columns: 'feature' and 'coefficient'.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Mathematical and Statistical Computations

Dynamic Data Transformation and Insertion

compare_csv('output.csv', 'result.csv', ignore_order=True, specified_columns=['coefficient'], thresholds={1: 0.02})


Question: A global agricultural policy research institution aims to explore the relationship between climate conditions and agricultural intensity for major crops during the period from 1990 to 2013. Using the Crop Recommendation dataset to identify the 12 core crops, construct a feature matrix from yield data, climate data (temperature and rainfall), and pesticide usage data. Calculate a standardized comprehensive climate index (combining temperature and rainfall) and a standardized agricultural intensity index (combining pesticide usage and crop yield) using MinMaxScaler normalization. Generate a static scatter plot using matplotlib where the x-axis represents the climate index, the y-axis represents the agricultural intensity index, the size of the points reflects the crop yield value, and the color indicates the country's income group. The plot should have the title 'Agricultural Trends 1990-2013: Climate vs Agricultural Intensity', x-axis label 'Normalized Climate Index (Temperature + Rainfall)', and y-axis label 'Normalized Agricultural Intensity Index (Pesticides + Yield)'. Save the visualization to output.png. Additionally, calculate and report the maximum crop yield value (unit: hectograms/hectare) across all data points, saving this result to output.csv with columns ['metric', 'value'].

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Crop Yield Prediction Dataset/rainfall.csv

Preprocessing and File Structure Adjustments

Column-wise Transformations and Aggregation

Arithmetic Transformations and Normalization

Validation and Verification of Merge Results

compare_image('output.png', 'result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])

compare_csv('output.csv', 'result.csv', ignore_order=True, specified_columns=['value'])


Question: A global agricultural policy research institution aims to assess whether the crops grown in various countries or reporting areas are suitable for their soil and climate conditions. A random forest classifier with 100 trees is constructed, with the random state set to 42 and the category weights set to "balanced". Based on soil characteristics (nitrogen, phosphorus, potassium, zinc, sulfur, and pH) and derived annual weather characteristics (annual average temperature and annual precipitation), the optimal crops are predicted. For area matching, normalize reporting area labels to deterministic keys (lowercase, with parenthetical content and punctuation removed), reconcile national weather labels to the corresponding yield-data area labels where needed, and exclude historical or aggregate entities. Then, the actual main crops of each country or reporting area are determined based on the average annual yield, and combined with representative soil profiles (average nutrient levels of each crop in the recommended dataset) and annual weather summaries at the national level (missing national weather values are filled with the global median), to generate predicted results for each country-year record from 1990 to 2013. The results are saved to output.csv with columns: 'country' (the normalized reporting area key), 'year', 'actual_top_crop', 'predicted_crop', 'prediction_probability', showing the differences between the crops recommended by the model and those actually planted, as well as the confidence score of the prediction.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/rainfall.csv

Data Structure Handling (Dictionaries, Lists)

Data Transformation and Feature Engineering

Data Transformation and Column Manipulation

Column-wise Transformations and Aggregation

Column-specific or Feature-wise Processing

Data Preprocessing and Column Management

compare_model('output.csv', 'result.csv', matched_columns=['country', 'year'], calculate_columns=['predicted_crop'], metric='accuracy', lower_bound=0.0, upper_bound=1.0)


Question: A global agricultural policy research institution aims to quantify how soil suitability, national climate conditions, pesticide usage, and economic development jointly affect the crop yields of different countries. Construct a unified country-crop-year feature set by combining median crop-level PCA soil suitability profiles from the crop recommendation data, retaining 95% explained variance, with country-year climate, pesticide, GDP, and income-group indicators; use standardized predictors for the Ridge model so coefficient magnitudes are comparable. A ridge regression model with alpha = 1.0 is used to predict crop yields (in units: hectograms per hectare), retaining 20% of the data for testing (random state set to 42), and reporting the R² score on the test set as well as the data frame of feature coefficients sorted by absolute value size. The results are saved to output.csv with columns 'metric' and 'value', where the first row contains the R² score and subsequent rows contain feature names and their corresponding coefficients.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/pesticides.csv

Crop Yield Prediction Dataset/rainfall.csv

Data Transformation and Feature Engineering

Feature Selection and Dimensionality Reduction

Dimensionality Reduction and Feature Engineering

Categorical Data Preprocessing and Simplification

Data Preprocessing and Column Management

Dynamic Data Transformation and Insertion

compare_csv('output.csv', 'result.csv', ignore_order=True, specified_columns=['value'], thresholds={1: 0.02})


Question: A global agricultural research institution aims to assess the robustness of a crop recommendation system. A random forest classifier with 100 trees (random state set to 42) is used for crop recommendations. When simulating any random components, please set the random seed to 42 to ensure the reproducibility of the results. Prepare a comprehensive evaluation report, which includes: (1) classification reports for each category, including precision, recall, F1 score, and support; (2) a standardized confusion matrix showing the misclassification patterns of crop types; (3) overall accuracy, macro F1 score, and micro F1 score. Save the classification report to output1.csv with columns for each metric (precision, recall, f1-score, support) and rows for each crop class plus overall metrics (accuracy, macro avg, micro avg). Save the confusion matrix data to output2.csv with rows and columns representing actual and predicted crop classes. Save the confusion matrix visualization to output.png with title 'Normalized Confusion Matrix - Crop Recommendation', xlabel 'Predicted', and ylabel 'Actual'.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Data Transformation and Column Manipulation

Feature Selection and Dimensionality Reduction

Data Structure Creation and Manipulation

Dynamic Data Transformation and Insertion

Normalization and Percentile Calculations

Grid-Based Predictions and Visualization

compare_image('output.png', 'result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: A global food safety research team aims to explore factors affecting short-term food price instability. Using historical food price data and agricultural production data, a multiple linear regression model is calculated, where the dependent variable is the three-year rolling standard deviation of annual food prices (in US dollars per kilogram), and the independent variables include: (1) the three-year moving average of crop yields, and (2) the pesticide use intensity defined as the total pesticide usage (tons) divided by crop yield moving average. The regression model is fitted only when all variable data are fully observed. The final output is a pandas DataFrame listing predictors with their coefficient estimates, standard errors, t-statistics, p-values, and 95% confidence intervals. Save the significant results (p-value < 0.05) to output1.csv with columns 'predictor', 'coefficient', 'std_error', 't_statistic', 'p_value', 'ci_lower', 'ci_upper'. Save all results to output2.csv with the same column schema.

Crop Yield Prediction Dataset/pesticides.csv

Data Loading with Pandas
Parsing and Reading Data Files
Statistical Calculations and Quantiles
Arithmetic and Cumulative Calculations
Data Cleaning and Transformation
Data Aggregation and Grouping
Data Preparation and Aggregation
In-place vs Copy Operations
Data Integration and Merging
Join Operations and Merging
Function Application and Vectorization
Statistical Analysis and Metrics
Regression Modeling and Interpretation
Model Training & Evaluation
Statistical Modeling and Uncertainty
Array and Matrix Manipulation
Data Export and Output Processing
Data Manipulation and Summarization

compare_csv('output1.csv', 'result1.csv', ignore_order=True, specified_columns=['predictor', 'coefficient', 'std_error', 't_statistic', 'p_value', 'ci_lower', 'ci_upper'], thresholds={1: 0.02, 2: 0.02, 3: 0.02, 4: 0.02, 5: 0.02, 6: 0.02})

compare_csv('output2.csv', 'result2.csv', ignore_order=True, specified_columns=['predictor', 'coefficient', 'std_error', 't_statistic', 'p_value', 'ci_lower', 'ci_upper'], thresholds={1: 0.02, 2: 0.02, 3: 0.02, 4: 0.02, 5: 0.02, 6: 0.02})


Question: A global agricultural policy research institution is assessing whether countries that grow crops based on their own soil and climate conditions can achieve higher yields. Using the provided crop yield, temperature, rainfall, and crop recommendation datasets, first standardize crop names to match those in the recommendation system. Then calculate the climate suitability ranges (minimum and maximum annual average temperature and total annual precipitation) for each crop based on the soil and weather data; note that the seasonal PRECTOTCORR fields represent average daily precipitation for each season, measured in mm/day. Next, identify the highest-yielding crop for each country-year combination, and check whether its growing conditions fall within the climate suitability range for that crop. Generate a histogram showing the distribution of average annual temperatures for these top-yielding crops with 30 bins, black edges, and 70% transparency. The histogram should have title 'Temperature Distribution for Top-Yield Crops', xlabel 'Average Annual Temperature (°C)', and ylabel 'Frequency'. Save the histogram as output.png. Finally, identify the temperature bin with the highest frequency and save the bin boundaries, frequency count, and total matched records to output1.csv with columns 'bin_lower_bound', 'bin_upper_bound', 'frequency', 'total_matched_records', and save the detailed matched records to output2.csv with columns 'Area', 'Year', 'Item_clean', 'Value', 'avg_temp', 'average_rain_fall_mm_per_year'.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Crop Yield Prediction Dataset/rainfall.csv

Dictionary Manipulation and Construction

Data Structure Handling (Dictionaries, Lists)

Numerical Operations and Type Conversion

compare_csv('output1.csv', 'result1.csv', ignore_order=False, specified_columns=['bin_lower_bound', 'bin_upper_bound', 'frequency', 'total_matched_records'], thresholds={'bin_lower_bound': 0.05, 'bin_upper_bound': 0.05})

compare_image('output.png', 'result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: An international agricultural research institution aims to develop a crop recommendation system based on soil and climate suitability. Using the provided crop recommendation dataset containing soil properties and seasonal weather features, first calculate derived features including the annual temperature range (difference between maximum and minimum temperatures across all seasons) and total annual precipitation (sum of precipitation across all seasons). Encode categorical features such as soil color using label encoding. Then train a multi-class random forest classifier using 5-fold stratified cross-validation to predict the best crop type. Calculate the F1 score for each crop category and output a pandas data frame containing two columns: 'crop_class' (listing the crop labels) and 'f1_score' (the F1 score of each category from cross-validation). Save the results to output.csv.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Data Structure Creation and Manipulation


Question: An international agricultural policy research institution aims to identify country-year cases where crop yields are poorly aligned with climate suitability, especially in low-income economies with high pesticide usage. Treat the four seasonal temperature and precipitation columns in the crop recommendation dataset as 4-point time series and use Fourier-based feature extraction to compute the mean, range, and dominant frequency for each crop's climate profile. For precipitation, treat PRECTOTCORR as a daily rate and compare it with rainfall.csv's average_rain_fall_mm_per_year on the same annual scale. For each country-year combination, identify the crop with the highest yield, and select records satisfying at least two of the following three conditions: (1) the highest-yield crop is not among crops whose temperature and annual-scale precipitation mean ± one standard deviation cover the country's observed annual average temperature and annual rainfall; (2) pesticide usage exceeds the 75th percentile of all combined records; (3) the country's GDP from the Natural Earth country table is below the global median. Save the results to 'output.csv' with schema: country_clean, Year, top_crop, climate_plausible_crops (comma-separated list of climate-plausible crops), pesticide_value, gdp_md.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Data Structure Creation and Manipulation

Numerical Operations and Type Conversion

Row-wise and Column-wise Logical Evaluation

Logical Operators for Combining Conditions


Question: A global agricultural policy research institution aims to assess how national economic development affects the relationship between pesticide use and crop yields under different climatic conditions. Apply an ARIMAX(1,1,1) model to each country-crop combination, using pesticide use, rainfall, and temperature as exogenous regressors to predict crop yields for the next three years. The model requires at least 5 historical data points and variation in all exogenous variables. Use the last observed values of the exogenous variables for forecasting. Only include country-crop combinations that meet the data requirements and successfully produce forecasts in the final output. Save both historical records and forecasted values to 'output.csv' with the following schema: country (string), Year (integer), crop_item (string), yield_value (float), pesticide_value (float), rainfall (float), avg_temp (float), is_forecast (boolean), GDP_MD (float), INCOME_GRP (string).

Data Transformation and Column Manipulation

Data Structure Creation and Manipulation

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['country', 'crop_item', 'Year'], calculate_columns=['yield_value'], metric='mse', lower_bound=0, upper_bound=10000000)


Question: A global food security working group is identifying those high-risk corn-producing countries that are simultaneously facing climate pressure, extensive use of chemicals, and market instability - this might indicate systemic vulnerabilities in the food supply chain. Using the dataset, find all country-year combinations that meet all of the following conditions: (1) the crop is corn, (2) the average annual temperature exceeds 20°C, (3) the pesticide usage is higher than the global median for all countries, (4) the 12-month rolling standard deviation of monthly corn prices is higher than the global median volatility of corn, (5) the year-on-year corn production change is negative, (6) the corn price of that country-year is within the top quarter of all corn prices worldwide. Save the results to "output.csv" with the following columns: ['Country', 'Year', 'Corn Production', 'Average Temperature', 'Pesticide Usage', 'Corn Price (USD)', 'Price Volatility', 'Year-on-Year Production Change'].

Mathematical and Statistical Computations

Incremental and Comparative Calculations


Question: An international agricultural policy research institution is investigating situations where countries extensively use pesticides to maintain crop yields, but these practices may conflict with local soil and climate conditions. Using the provided datasets, determine the most commonly recommended crop globally among soil samples with pH values between 5.0 and 8.5 and without missing N, P, or K values. Then, identify the crop with the highest yield value for each country and year. Mark cases where a country's highest-yielding crop differs from the globally recommended crop. Apply the following additional restrictions: pesticide usage must exceed the 75th percentile of all pesticide values, and the average seasonal temperature range for the produced crop (summer maximum temperature minus winter minimum temperature) must exceed 15°C. Derive each produced crop's seasonal temperature range from its broader temperature profile in the crop recommendation data. For each country, count the annual occurrences of such mismatch events and apply an ARIMA(1,1,1) model to predict the count over the next 3 years. Save only the historical mismatch records (not predictions) to "output1.csv" with columns: country (string), year (integer), recommended_crop (string), produced_crop (string), pesticide_value (float), temp_range (float). Save the forecast results to "output2.csv" with columns: Area (string), Year (integer), forecasted_mismatch_count (float), is_forecast (boolean).

Crop Recommendation using Soil Properties and Weather Prediction.csv

Numerical Operations and Type Conversion

Data Transformation and Column Manipulation

Custom Functions for Missing Value Handling

Numerical Comparison and Proximity Checks


Question: An international agricultural research institute aims to assess how well environmental and economic factors explain historical crop yields globally. Using the provided agricultural datasets, build a degree-2 polynomial regression model (sklearn LinearRegression with PolynomialFeatures) to predict crop yield based on average temperature, annual rainfall, and pesticide usage. Compute the R² score on the fitted data, rounded to four decimal places. Save the predictions to "output.csv" with columns: country, year, crop_item, actual_yield, predicted_yield, pesticide_value, rainfall, avg_temp. Save the R² score to "output.txt" in the format: 'Polynomial Regression R2 Score: {value}'.

Preprocessing and File Structure Adjustments

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'predicted_yield': None})


Question: An agricultural research team is analyzing the relationship between soil characteristics, weather patterns, and crop yields to determine the key factors affecting agricultural productivity. Conduct principal component analysis (PCA) on the soil nutrient features and seasonal maximum-temperature features from the crop recommendation dataset to identify the top 5 features with the highest absolute loadings on the first principal component. Merge the national yield and pesticide datasets after aggregating each dataset by country using mean values across years, and calculate the correlation between country-level average yield and country-level average pesticide use. Save the top 5 features and their PC1 loadings to "output.csv" with columns: feature (string), pc1_loading (float). Save the variance analysis results and correlation to "output.txt" containing: 'Explained variance ratio by PC1: {value}', 'Explained variance ratio by PC2: {value}', 'Cumulative explained variance (first 5 PCs): {value}', 'Yield-Pesticide Correlation: {value}'.

Crop Recommendation using Soil Properties and Weather Prediction.csv

Feature Selection and Dimensionality Reduction

Column-specific or Feature-wise Processing

Column-wise Transformations and Aggregation

Arithmetic Transformations and Normalization


Question: An international agricultural policy research institution aims to identify those countries where the actual crop yield does not match the climate suitability, and which may be inefficient (especially in low-income economies and regions with high pesticide usage). Using the dataset, each row's seasonal temperature and precipitation measurements are regarded as discrete 4-point time series, and a feature extraction method based on Fourier transformation is applied to calculate the average, range, and dominant frequency of each indicator. Then, the crop with the highest yield for each country-year combination is determined. This data is combined with other files to associate each country-year with the country's pesticide usage, annual average rainfall, and average temperature. Finally, records of the following situations are selected: (1) The crop with the highest yield is not among those whose extracted climate characteristics distribution covers the country's observed annual average temperature and total rainfall within one standard deviation range; (2) The pesticide usage exceeds the 75th percentile of all pesticide usage records. Save the priority cases to "output1.csv" with columns: country_clean (string), Year (integer), top_crop (string), climate_plausible_crops (string), pesticide_value (float). Save the Fourier transformation features to "output2.csv" containing: label (string), temp_mean (float), temp_range (float), temp_dominant_freq (float), precip_mean (float), precip_range (float), precip_dominant_freq (float).

Crop Recommendation using Soil Properties and Weather Prediction.csv

Column-specific or Feature-wise Processing

Preprocessing and File Structure Adjustments

Mathematical and Statistical Computations

Numerical Comparison and Proximity Checks

Normalization and Percentile Calculations

Validation and Verification of Merge Results


Question: An e-commerce analytics company needs to determine which platform (Amazon or Olist Brazilian marketplace) has the strongest statistically significant positive correlation between product prices and customer satisfaction in electronics categories. For Amazon, include products with main_cat "Cell Phones & Accessories" or "Electronics". For Olist, include products in English categories: telephony, computers_accessories, electronics, consoles_games. Analyze Amazon at the product level using averaged product ratings; keep Olist at the order-item/review record level, convert Olist prices to USD with a fixed 0.2 multiplier, and apply a 1.5 IQR price-outlier filter separately within each platform before calculating statistics. Calculate Pearson correlation, Spearman correlation, linear regression (predicting rating from price), and 95% confidence interval of Pearson correlation. Save to output.csv with columns: platform, pearson_r, pearson_p, spearman_r, spearman_p, lr_coef, lr_intercept, ci_lower, ci_upper, n_samples, best_platform. Platform values should be 'amazon' or 'olist'. best_platform should be the platform with statistical significance (p < 0.05 and positive correlation); if multiple qualify, select highest pearson_r; if none qualifies, use "None (No significant relationship found)". Save without index.

Data Structure Creation and Manipulation

Numerical Operations and Type Conversion

Preprocessing and File Structure Adjustments

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['platform', 'pearson_r', 'pearson_p', 'spearman_r', 'spearman_p', 'lr_coef', 'lr_intercept', 'ci_lower', 'ci_upper', 'n_samples', 'best_platform'], thresholds={1: 0.02, 2: 0.02, 3: 0.02, 4: 0.02, 5: 0.02, 6: 0.02, 7: 0.02, 8: 0.02})


Question: An e-commerce intelligence company is building a cross-platform brand performance dashboard that integrates delivery reliability data from the Brazilian market (Olist) with global pricing and product metadata from Amazon and eBay. Requirements: (1) Amazon: Filter 'Cell Phones & Accessories' category products, calculate median price per brand. (2) eBay: Filter products of type containing 'Notebook/Laptop', extract the first numerical value from price ranges (e.g., "$399.99 to $634.99" → 399.99), calculate median price per brand. (3) Olist: For delivered orders, calculate delivery delay (actual delivery date - estimated delivery date, in days) and average review score for categories: telephony, computers_accessories, electronics, consoles_games. Brand standardization: Convert to lowercase, remove special characters. Brand-to-category matching (iterate through [telephony, computers_accessories, electronics, consoles_games] in order, use first match): samsung/apple/lg/motorola/sony match if category is telephony or electronics; asus/acer/lenovo/hp/dell/toshiba/msi match if category is computers_accessories. Output one row for each unique standardized brand found in the filtered Amazon or filtered eBay data. Output CSV (output.csv) with columns: brand, amazon_median_price, ebay_median_price, avg_delivery_delay_days, avg_review_score, total_listing_count (sum of amazon and ebay listings, treat NaN as 0), olist_product_count (unique products from matched Olist category). Fill missing avg_delivery_delay_days with 0, avg_review_score with median value. Save without index.

Data Conversion and Post-Loading Processing

Preprocessing and File Structure Adjustments

Conditional Logic and Row-wise Operations

Time Difference and Gradient Calculation

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['brand', 'amazon_median_price', 'ebay_median_price', 'avg_delivery_delay_days', 'avg_review_score', 'total_listing_count', 'olist_product_count'])


Question: An e-commerce intelligence company is conducting a cross-platform analysis of customer satisfaction and pricing strategies across Amazon, eBay, and the Brazilian Olist marketplace to guide market positioning for electronics brands. The analysis focuses on Amazon products in the 'Cell Phones & Accessories' category, eBay PC laptop listings, and Olist orders for electronic product categories ('telephony', 'computers_accessories', 'electronics', 'consoles_games'). For eBay laptops, define a composite hardware score from four specifications: processor grade (i7/Ryzen 7 = 5, i5/Ryzen 5 = 4, i3/Ryzen 3 = 3, Celeron/Pentium = 2, others or missing = 1), GPU grade (NVIDIA/GeForce/RTX/GTX = 5, AMD/Radeon = 4, Intel = 3, others = 2, missing = 1), RAM in GB, and SSD capacity in GB; fill missing values with column medians, Min-Max normalize all four features, and weight them as RAM × 0.25 + SSD × 0.25 + Processor × 0.30 + GPU × 0.20. For Amazon, assign 'Budget', 'Mid', and 'Premium' brand tiers using 3-quantile price binning. For Olist, delivery delay is defined as the days from estimated delivery to actual delivery. Generate a matplotlib chart that meets publication standards, using a 3×2 GridSpec layout, including: (1) a scatter plot titled 'eBay: Price vs Hardware Score' with x-axis labeled 'Hardware Score' and y-axis labeled 'Price (USD)', as well as an embedded marginal histogram of hardware ratings; (2) a scatter plot titled 'Amazon: Price vs Rating by Brand Tier' with x-axis labeled 'Price (USD)' and y-axis labeled 'Rating'; (3) a histogram titled 'Olist: Review Score Distribution (Top 5 Categories)' with x-axis labeled 'Review Score' and y-axis labeled 'Frequency', showing the distribution of ratings in the top 5 electronic categories; (4) a scatter plot titled 'Cross-Platform: Price vs Rating' with x-axis labeled 'Price (USD)' and y-axis labeled 'Rating', overlaying Amazon product ratings and eBay seller ratings; (5) a scatter plot titled 'Olist: Freight vs Distance (colored by Delivery Delay)' with x-axis labeled '|Customer-Seller Zip Diff|' and y-axis labeled 'Freight Value (BRL)', colored by delivery delay; and (6) a scatter plot titled 'Olist: Delivery Delay vs Review Score' with x-axis labeled 'Delivery Delay (Days)' and y-axis labeled 'Review Score'. The chart must provide a shared color bar for subgraph (5), achieve precise spacing through GridSpec, and directly report the median delivery delay (in days) of ratings of 1 in Olist electronic orders from subgraph (6), retaining one decimal place. Save the visualization to output.png and the key metrics to output.csv with columns: metric_name (name of the metric), metric_value (numerical value), platform (e-commerce platform: eBay, Amazon, or Olist). The metrics must include: ebay_hardware_score_mean, ebay_price_mean, amazon_rating_mean, amazon_price_mean, olist_review_score_mean, olist_delivery_delay_median, and olist_median_delay_score_1 (median delivery delay in days for orders with review score of 1, rounded to 1 decimal place).

Conditional Logic and Row-wise Operations

Data Structure Understanding and Initialization

Data Transformation and Column Manipulation

Time Difference and Gradient Calculation

Numerical Comparison and Proximity Checks

Batch Processing and Performance Optimization

Data Transformation and Feature Engineering

Mathematical and Statistical Computations


Question: An e-commerce analytics company is conducting a benchmark test on the drivers of customer satisfaction across various global platforms to provide suppliers with recommendations on pricing strategies. The correlation between product prices and customer ratings is calculated for Amazon (in the mobile and accessories category) and Olister (in the Brazilian electronics category, including mobile phones, computer accessories, electronics, and game consoles and games). For each platform, the following analyses are performed: (1) Load and clean the data, filter by the specified category, and remove price outliers using the interquartile range method; (2) Convert Olister's prices from Brazilian reais to US dollars using a 1:0.20 exchange rate; (3) Calculate the Pearson and Spearman correlation coefficients between prices and ratings (using Amazon's "overall" and Olister's "review score"); (4) Conduct a linear regression analysis to establish a model for the relationship between prices and ratings; (5) Calculate the 95% confidence interval for the Pearson correlation coefficient; (6) Determine statistical significance at a p < 0.05 level; (7) Determine the platform with the strongest significant correlation. The statistical analysis results are saved as a file named "output.csv", which contains the following columns: platform (platform name: Amazon or Olister), Pearson correlation coefficient, Pearson p-value, Spearman correlation coefficient, Spearman p-value, linear regression coefficient, linear regression intercept, lower limit of 95% confidence interval, upper limit of 95% confidence interval, significance (whether statistically significant: True or False). The name of the platform corresponding to the strongest significant correlation is saved to the "best_platform.txt" file (e.g., "Amazon", "Olister", or "None (no significant relationship found)

Interpreting and Communicating Statistical Results

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['platform', 'pearson_r', 'pearson_p', 'spearman_r', 'spearman_p', 'linear_reg_coef', 'linear_reg_intercept', 'ci_lower', 'ci_upper', 'significant'])


Question: An e-commerce intelligence company is building a cross-platform product benchmark evaluation system to compare the hardware specifications, prices, and customer reviews of electronic products on Amazon, eBay, and the Brazilian Olist market. Create a unified dataset that only includes products related to laptops or mobile phones with available rating scores. For all sources, clean the numerical fields (price, RAM, SSD), convert the brand names to lowercase, convert the price to US dollars, and normalize the review scores to a scale of 1 to 5. Additionally, perform word tokenization on the comment text and comment messages, calculate TF-IDF scores, and identify the top 10 most frequently occurring words related to negative emotions (review score ≤ 2); use these words to label products with higher negative sentiment content. Only include products with available rating scores in the final output. Fill missing hardware specifications with the median values of each platform. Save the unified DataFrame to 'output.csv', ordered by source platform, product ID, and brand. Save the top 10 negative sentiment terms to 'negative_terms.txt' with one term per line.

Numerical Operations and Type Conversion

Categorical Data Preprocessing and Simplification

Batch Processing and Performance Optimization

Data Structure Creation and Manipulation

Dynamic Data Transformation and Insertion

Custom Functions for Missing Value Handling

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['product_id_unified', 'brand_clean', 'category_clean', 'price_usd', 'review_score_normalized', 'ram_gb', 'ssd_gb'])


Question: A global e-commerce intelligence company is conducting an assessment of customer satisfaction on various platforms to assist brands in formulating market entry strategies. Extract the grade ratings of products clearly classified as the 'mobile and accessories' category from the metadata. Separate the comment scores related to electronic device orders (categories: phones, computer accessories, electronic devices, game consoles, and laptops). Clean the price field by removing the '$' symbol and handling the range (using the lower limit), and then generate a synthetic 1 to 5 rating based on the quintiles of prices (1 = lowest 20%, 5 = highest 20%). When processing the Amazon data files, read only the first 50,000 records from each file to ensure computational efficiency. Generate a single standardized histogram covering these three distributions (Amazon, Olist, eBay agent), with the intervals centered on 1 to 5, and report the pattern of the Amazon distribution (the most frequently occurring interval). Save the histogram to output.png with title 'Cross-Platform Customer Satisfaction Distribution', xlabel 'Rating Score', and ylabel 'Normalized Frequency'. Save the mode rating results to output.csv with columns 'platform', 'mode_rating', and 'mode_frequency'.

Data Structure Creation and Manipulation

Vectorization and Performance Optimization

compare_image('output.png', 'result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: An e-commerce analysis team is building a cross-platform model to predict whether product reviews will be regarded as helpful by future customers, that is, the 'helpful vote ratio' (the number of helpful votes / total number of votes) is at least 0.5. Using Amazon reviews and their related product metadata, brand and price information are added to each review. Additionally, the 'price' column of the laptop list is cleaned and converted into a numerical value in US dollars, the 'brand' column is lowercase. When processing the Amazon data files, read only the first 10,000 records from each file to ensure computational efficiency. By combining the features of these two platforms (including brand names and price grades), a random forest classifier is trained to predict the binary usefulness label. The test accuracy of this classifier on the retained dataset is reported, rounded to four decimal places. Save the test accuracy to output.csv with columns 'metric' and 'value'.

Data Transformation and Column Manipulation

Data Labeling and Structured Data Handling

Numerical Operations and Type Conversion

Column-wise Transformations and Aggregation

Mathematical and Statistical Computations

Preprocessing and File Structure Adjustments

Arithmetic Transformations and Normalization

compare_csv('output.csv', 'result.csv', ignore_order=True, specified_columns=['value'], thresholds={1: 0.02})


Question: An e-commerce intelligence company is conducting a benchmark test on the drivers of customer satisfaction across various global platforms to provide strategic recommendations for electronic brands targeting specific markets. When processing the Amazon data files , read only the first 10,000 records from each file to ensure computational efficiency. Calculate the Pearson correlation between the price (in US dollars) of Amazon's mobile accessory products and the review ratings. Extract the numerical values of memory size (in GB) and solid-state drive capacity (in GB), and then calculate the Pearson correlation between these hardware specifications and the cleaned prices (in US dollars). For the Brazilian market, calculate the Pearson correlation between the product prices (in Brazilian reais), delivery delay days, and review ratings. Save the correlation coefficients for each platform to output.csv with columns 'platform', 'correlation_pair', and 'coefficient'.

Data Conversion and Post-Loading Processing

compare_csv('output.csv', 'result.csv', ignore_order=True, specified_columns=['coefficient'])


Question: An e-commerce analysis team is conducting a comparative analysis of customer satisfaction levels across various global platforms as a basis for cross-market product positioning. When processing the Amazon data files , read only the first 10,000 records from each file to ensure computational efficiency. They need to visualize the distribution of product evaluation scores for Amazon (in the US) and Olister (in Brazil), with the visualization stratified by price tiers and contextualized by hardware specifications from the eBay listings. They extract product evaluations and their price information from Amazon, and select items primarily categorized as "Mobile Phones and Accessories", "All Electronics", or "Computers". By connecting order IDs, product IDs, and product category names, they construct an Olister evaluation score dataset covering electronic categories (specifically "Telephones", "Computer Accessories", "Electronics", and "Game Consoles and Games"). They convert all price fields into numerical form (Amazon/Ebay in US dollars, and Olister by quartile calculation for equivalent treatment in Brazilian reais). They merge the cleaned price data from the two platforms and calculate the four global price quartiles (Q1 - Q4). Additionally, they preprocess the files to standardize memory specifications (e.g., converting "8GB" or "8GBDDR4" to "8GB"), and clean their "Price" column. For each of the four global price quartiles, they determine the most common (mode) memory size in eBay laptop listings within that quartile. Finally, they use matplotlib to generate a 2×2 overlapping histogram grid to display the rating density distribution (1 - 5 points) of Amazon and Olister within each price quartile (1 - 4). The title of each subplot should include the dominant memory size from eBay (format: 'Price Quartile QX (RAM: YGB)'). The xlabel should be 'Review Score' and the ylabel should be 'Density'. The dominant memory size shown in the subplots of the third price quartile (Q3) is reported. Save the Q3 dominant RAM size to output.csv with columns 'metric' and 'value', and save the histogram to output.png.

Numerical Operations and Type Conversion

Mathematical and Statistical Computations

Normalization and Percentile Calculations

Functional Data Handling and Iterative Plotting

compare_csv('output.csv', 'result.csv', specified_columns=['value'], ignore_order=True)

compare_image('output.png', 'result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: A global e-commerce intelligence company is conducting benchmark tests on consumer satisfaction and pricing strategies in major online markets, providing platform-specific entry-market strategies for electronic brands. When processing the Amazon data files, read all available records to ensure comprehensive analysis coverage. When processing the eBay data file, read all available records. When processing the Olist data files, read all available records. Filter review scores between 1-5, and remove price outliers (p99). A statistical visualization chart with three panels is generated: (1) shows the filled kernel density estimates (use filled KDE with fill=True) of logarithmically transformed product prices by platform (Amazon's mobile phones and accessories, eBay's laptops and netbooks, Olist's Brazilian electronics), with title 'Log-Scaled Price Distribution by Platform', xlabel 'Log Price', ylabel 'Density'; (2) displays box plots of review scores (on a 1 to 5 scale) arranged by platform, with title 'Review Score Distribution by Platform', xlabel 'Platform', ylabel 'Review Score (1-5)'; (3) plots the relationship between price and review score, with regression trends specific to each platform, with title 'Price vs Review Score with Regression', xlabel 'Price (USD)', ylabel 'Review Score'. From the generated chart, report the median review score observed in Olist's electronics orders. Save the visualization to output.png and the median review score to output.csv with columns 'platform' and 'median_review_score'.

Arithmetic Transformations and Normalization

Dynamic Data Transformation and Insertion

Statistical Plotting and Density Estimation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['platform', 'median_review_score'])

compare_image(output_file_name='output.png', gold_file_name='result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: An international electronic market research company aims to identify renowned brands that perform exceptionally globally, consistently maintaining high customer satisfaction and market influence. These brands are evaluated across two e-commerce ecosystems with brand data: Amazon (for mobile devices) and eBay (for laptops). Calculate the average product rating for each brand and the total number of Amazon review/comment records for that brand, calculate the median selling price and average seller rating for each laptop brand on eBay by using the numeric star value in the rating text, and convert the brand names into title format. Use known global brands (Apple, Samsung, Dell, Hp, Lenovo, Asus, Acer, Microsoft, Sony, Lg, Motorola, Xiaomi, Panasonic, Philips, Nintendo, Huawei) as the target brand list. For Olist (Brazilian comprehensive electronics), since brand data is not available in the dataset, determine the average evaluation score and order volume at the product category level for electronics categories (telephones, computer accessories, electronics, game consoles, and games); this category-level analysis is reported separately and is not included in the brand ranking. Calculate a comprehensive cross-platform brand performance score, defined as: (average score on Amazon × logarithm(1 + number of Amazon review/comment records)) + (average score on eBay × logarithm(1 + number of listings on eBay)). Fill missing average ratings with 0. Generate a histogram of delivery delays in days from the Olist data (actual delivery time minus estimated delivery time), filter delays to the range [-30, 30] days to remove outliers, and report the most frequent interval (in whole days); this value must be used to validate timestamp processing for verification purposes before the final ranking. Return a data frame of the top 5 brands ranked by the overall comprehensive score. Save the results to output.csv with columns: 'brand', 'amazon_composite', 'ebay_composite', 'total_composite_score'.

Batch Processing and Performance Optimization

Numerical Operations and Type Conversion

Time Difference and Gradient Calculation

Categorical Data Preprocessing and Simplification

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['brand', 'amazon_composite', 'ebay_composite', 'total_composite_score'])


Question: An e-commerce intelligence company is conducting a benchmark analysis of brand performance across eBay and Olist platforms to provide manufacturers with market entry strategy suggestions. Using the provided datasets, perform the following analyses: (1) For each eBay brand, calculate the average selling price (in USD) and determine the most common RAM size and SSD capacity. (2) For Olist orders under the categories of telephony, computer accessories, and electronics, calculate the average review score, average price (in BRL), and average shipping distance between sellers and customers (in degrees) per category. (3) Compute delivery delays for all Olist orders (actual delivery date minus estimated delivery date) and determine the most frequent delay interval (in whole days). (4) Classify the sentiment of every non-empty review comment with a fixed keyword rule: lowercase the text and count substring occurrences of the positive keywords ['good', 'great', 'excellent', 'amazing', 'love', 'perfect', 'best', 'happy', 'satisfied'] and the negative keywords ['bad', 'terrible', 'awful', 'hate', 'worst', 'disappointed', 'poor', 'horrible'], then label it 'positive' if positives exceed negatives, 'negative' if negatives exceed positives, and 'neutral' otherwise. Using review scores as proxy ground-truth labels (≥ 4 positive, ≤ 2 negative, 3 neutral), compute the macro F1 score (scikit-learn f1_score, average='macro'). Create a cross-platform summary comparing both platforms' average review scores and average prices (using electronics-category data for Olist), along with the Olist-specific sentiment F1 score and most frequent delay bin. Save results to "output1.csv" with columns: platform (string), avg_review_score (float), avg_price (float), macro_f1_sentiment (float), max_delay_bin (integer). Save Olist category statistics to "output2.csv" with columns: category (string), avg_review_score (float), avg_price_brl (float), avg_shipping_distance (float). Save eBay brand statistics to "output3.csv" with columns: brand (string), avg_price_usd (float), most_common_ram (string), most_common_ssd (string). Also save "output.json" containing: macro_f1_score (float), max_frequency_delay_bin (integer), total_olist_electronics_orders (integer), total_ebay_listings (integer).

String Manipulation and File Content Handling

Numerical Operations and Type Conversion

Custom Functions for Missing Value Handling

Numerical Comparison and Proximity Checks

Mathematical and Statistical Computations

Time Difference and Gradient Calculation

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'macro_f1_score': None})


Question: An e-commerce strategy team aims to identify electronic product brands that consistently maintain high customer satisfaction across multiple global markets. Using the Amazon (USA) and eBay platform data, find all brands that appear on at least two of these platforms using case-sensitive exact matching. For each qualifying brand, compute its satisfaction score on each platform as the median of the numerical rating values, then calculate the comprehensive satisfaction score as the average of these median scores across platforms. Also compute the total product count for each brand summed across all platforms where it appears. Additionally, perform a separate category-level satisfaction analysis on the Olist dataset, calculating the median review score per English product category. Save results to a SQLite database "output.db" with the following tables: "cross_platform_brands" containing qualifying brands with columns named exactly 'brand', 'amazon_satisfaction', 'ebay_satisfaction', 'composite_satisfaction', 'total_products', sorted by composite_satisfaction in descending order; "amazon_satisfaction" with columns 'brand' and 'amazon_satisfaction'; "ebay_satisfaction" with columns 'brand' and 'ebay_satisfaction'; "olist_satisfaction" with columns 'category' and 'olist_satisfaction'. Also save "output.json" with analysis statistics including total brands analyzed, number of brands appearing on at least two platforms, Amazon brand count, eBay brand count, and Olist category count.

Data Conversion and Post-Loading Processing

Column-wise Transformations and Aggregation

Data Structure Creation and Manipulation

compare_sqlite(output_file_name='output.db', gold_file_name='result.db', specified_schema={'cross_platform_brands': ['brand', 'amazon_satisfaction', 'ebay_satisfaction', 'composite_satisfaction', 'total_products'], 'amazon_satisfaction': ['brand', 'amazon_satisfaction'], 'ebay_satisfaction': ['brand', 'ebay_satisfaction'], 'olist_satisfaction': ['category', 'olist_satisfaction']}, ignore_order=False)

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'total_brands_analyzed': None, 'brands_in_at_least_two_platforms': None, 'amazon_brands': None, 'ebay_brands': None, 'olist_categories': None})


Question: An e-commerce platform in Brazil wants to analyze how transportation performance between states changes over time and its impact on customer satisfaction. Calculate the following metrics grouped by seller state, customer state, and purchase month (format: YYYY-MM): average delivery delay days (the difference between the actual delivery date and the estimated delivery date), average review score, total number of orders, average freight value, and average Haversine distance (in kilometers) between seller and customer locations. For Haversine distance, average the latitude and longitude per zip code prefix to eliminate duplicate coordinates. Filter to only include orders for products in the following categories (English names): "telephony", "computers_accessories", "electronics", "consoles_games". Apply an ARIMA(1,1,1) model to forecast the average monthly delivery delay for each state pair for the next 12 months, only for pairs with at least 6 monthly data points; if ARIMA fitting fails, use the historical mean and standard deviation as fallback. Save the results to a SQLite database "output.db" with the following tables: "spatiotemporal_summary" containing the historical spatiotemporal summary with columns: seller_state, customer_state, purchase_month, avg_delivery_delay_days, avg_review_score, total_orders, avg_freight_value, avg_haversine_distance_km, sorted by seller_state, customer_state, purchase_month; "arima_forecast" containing the ARIMA forecast results with columns: seller_state, customer_state, forecast_mean, forecast_std, sorted by seller_state, customer_state; "state_pair_stats" containing the state pair aggregated statistics with columns: seller_state, customer_state, avg_delivery_delay_days, avg_review_score, total_orders, avg_haversine_distance_km, sorted by seller_state, customer_state. Also save the overall analysis statistics to "output.json" containing: total_state_pairs, total_months, avg_delivery_delay_overall, avg_review_score_overall, total_orders_overall, avg_haversine_distance_overall.

Timestamp Conversion and Time Manipulation

Time Difference and Gradient Calculation

Custom Functions for Missing Value Handling

Data Transformation and Column Manipulation

Mathematical and Statistical Computations

Column-wise Transformations and Aggregation

compare_sqlite(output_file_name='output.db', gold_file_name='result.db', specified_schema={'spatiotemporal_summary': None, 'arima_forecast': None, 'state_pair_stats': None}, ignore_order=False)


Question:
Create a 2×2 matplotlib chart to evaluate the progress of the US energy transition, save as output.png. Specific requirements:
(1) Line chart: Title='U.S. Renewable Electricity Share (2001-2022)', X-axis label='Year', Y-axis label='Renewable Share (%)', showing annual percentage of renewable energy generation from 2001 to 2022 with percentage values marked on data points;
(2) Horizontal bar chart: Title='Top 10 States by Solar Carbon Offset Potential', X-axis label='Carbon Offset (metric tons)', Y-axis showing state names, displaying top 10 states by carbon_offset_metric_tons in descending order;
(3) Scatter plot: Title='Electricity Access vs CO₂ Emissions (2015+)', X-axis label='Access to Electricity (% of population)', Y-axis label='CO₂ per Capita (tonnes)', showing relationship between electricity access rate and per capita CO2 emissions for all countries from 2015 onwards, with country names annotated for those with CO2 per capita > 10 tonnes;
(4) Bar chart: Title='Steel Industry Load Type Distribution (2018)', X-axis showing Load_Type categories, Y-axis label='Count', showing count of each load type in steel industry data for 2018, with an inset text box in top-right corner displaying 'Top 10 Cities\nCarbon Offset:\n{X}M metric tons' where X is the total carbon_offset_metric_tons of top 10 cities divided by 1e6, rounded to 2 decimal places.

Cross/Merge Products and Filtering by Time Windows

Data Conversion and Post-Loading Processing

Data Mapping and Encoding in Visualizations

compare_image('output.png', 'result.png', calculate_columns=['type'] )


Question: As a national energy policy advisor, assess the correlations among solar energy potential, renewable energy deployment, electricity prices, and carbon intensity in each state of the United States to determine the priority of federal incentives in 2017. Use the provided dataset to generate a 2×3 subplot grid and save it as 'output.png', including the following six subplots with specified titles, x-labels, and y-labels: (1) PCA scatter plot with title 'U.S. States: PCA Analysis of Energy Indicators (colored by electricity price)', xlabel 'Principal Component 1', ylabel 'Principal Component 2', colorbar label 'Residential Electricity Price (cents/kWh)'; (2) Bubble chart with title 'Global: CO2 Per Capita vs Renewable Energy Share (bubble=population)', xlabel 'Renewable Energy Share (%)', ylabel 'CO2 Per Capita (tons)', xlim [0, 100], ylim [0, 30]; (3) Pie chart with title 'Steel Industry: Load Type Distribution (2018)'; (4) Bar chart with title 'Solar Potential Comparison by Geographic Granularity', xlabel showing granularity levels (State, County, City, Postal Code), ylabel 'Total Solar Potential (kW)'; (5) Horizontal bar chart with title 'Top 10 U.S. States: Fossil Fuel Generation (2017)', xlabel 'Fossil Fuel Generation (MWh)'; (6) Time series line chart with title 'Steel Industry: Energy Usage Patterns (sampled)', xlabel 'Time (sampled points)', ylabel 'Energy Usage (kWh)'. The overall figure should have suptitle 'Comprehensive Energy Analysis: Multi-dimensional Perspective'. Finally, report the name of the generated visualization file.

Data Structure Handling (Dictionaries, Lists)

Cross/Merge Products and Filtering by Time Windows

Column-specific or Feature-wise Processing

Feature Selection and Statistical Computation

Dimensionality Reduction and Visualization

Dimensionality Reduction and Feature Engineering


Question: Calculate the fossil fuel dependency index for each US state (including Washington, D.C.) in 2017. The formula is: (Total fossil fuel electricity generation MWh) / (Roof solar theoretical potential MWh) × (Industrial electricity price cents/kWh). Fossil fuels include coal, natural gas, and petroleum. Roof solar theoretical potential is total sunlight electricity. Save the results to output.csv with the following columns (in order): state_name (string, full state name), fossil_generation_MWh_2017 (float, total fossil fuel electricity generation in MWh), solar_potential_MWh_2017 (float, roof solar theoretical potential in MWh), industrial_price_cents_per_kWh_2017 (float, industrial electricity price in cents per kWh), fossil_dependency_index (float, fossil fuel dependency index calculated by the formula)

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction


Question: Estimate the accuracy of a prediction model for the carbon reduction potential from rooftop solar deployment across U.S. states. Using 2017 data, fit a RandomForestRegressor model (n_estimators=100, random_state=42) to predict the carbon offset. The model features include: 1. Residential electricity price, 2. Fossil fuel generation share, 3. National level indicators including GDP per capita and renewable energy share, 4. Total CO2 emissions. Calculate the R² score, Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) for the model predictions, and identify the 5 U.S. states with the largest absolute prediction errors. Save the model evaluation metrics to output1.csv with columns metric_name and metric_value. Save the states with largest prediction errors to output2.csv with columns state_name, actual_carbon_offset_metric_tons, predicted_carbon_offset_metric_tons, absolute_error_metric_tons.

Feature Selection and Dimensionality Reduction

Data Structure Handling (Dictionaries, Lists)

Error Function Design and Implementation

compare_csv('output1.csv', 'result1.csv', ignore_order=True, specified_columns=['metric_value'])


Question: For each record across all provided entertainment data files, generate a standardized identifier by normalizing its title — convert to lowercase, remove all non-alphabetic characters (retaining spaces), and collapse consecutive spaces into one. Prepend a source-specific prefix: M_ (movies), N_ (Netflix), S_ (Spotify), T17_ (2017 music), T18_ (2018 music), T20_ (2020 music), V_ (video games). For missing or empty titles, the identifier consists of the prefix only. Output all identifiers to output.csv with a single column harmonized_title_id.

Custom Functions for Missing Value Handling

Dynamic Data Transformation and Insertion


Question: Using entertainment media datasets (books, movies, streaming, games, and music) covering 2017–2020, perform text-based clustering and topic analysis across all sources. Apply TF-IDF vectorization (max_features=5000, stop_words='english'), K-Means clustering (n_clusters=10, random_state=42), and LDA topic modeling (n_components=10, random_state=42). For each cluster, determine the dominant LDA topic with the top 5 keywords, compute coherence_score as the mean cosine similarity between cluster members and the K-Means centroid (when a cluster exceeds 500 members, randomly sample 500 with np.random.seed(42)), and select up to 5 representative items. Output to output.csv with schema: cluster_id (int, 0–9), dominant_topic (int), top_topic_words (comma-separated), coherence_score (float, rounded to 4 decimal places), representative_items (pipe ' | ' separated, format 'source: title'). 10 rows, one per cluster.

Vectorization and Performance Optimization


Question: Analyze the multi-dimensional trends of the entertainment industry in 2017, 2018 and 2020, calculate the mean and standard deviation of the three audio features (danceability, energy, and valence) of popular songs in these three years, and conduct a one-way analysis of variance (ANOVA) for these three features; calculate the Herfindahl-Hirschman Index (HHI) of the game market; calculate the mean and median of book and movie ratings. Output the result to output.json with the following top-level keys: "music_features" (object, keys named as {feature}_{year}_mean and {feature}_{year}_std, e.g. "danceability_2017_mean", 18 float fields total), "anova" (object, keys named as {feature}_f_statistic and {feature}_p_value, 6 float fields total), "game_market_hhi" (float), "ratings" (object with keys: books_mean, books_median, movies_mean, movies_median, all float).

Data Transformation and Column Manipulation

Numerical Operations and Type Conversion

Dynamic Data Transformation and Insertion

Feature Selection and Statistical Computation

Mathematical and Statistical Computations

Dictionary Manipulation and Construction

Data Structure and Dictionary Operations

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'music_features': {'danceability_2017_mean': None, 'danceability_2017_std': None, 'danceability_2018_mean': None, 'danceability_2018_std': None, 'danceability_2020_mean': None, 'danceability_2020_std': None, 'energy_2017_mean': None, 'energy_2017_std': None, 'energy_2018_mean': None, 'energy_2018_std': None, 'energy_2020_mean': None, 'energy_2020_std': None, 'valence_2017_mean': None, 'valence_2017_std': None, 'valence_2018_mean': None, 'valence_2018_std': None, 'valence_2020_mean': None, 'valence_2020_std': None}, 'anova': {'danceability_f_statistic': None, 'danceability_p_value': None, 'energy_f_statistic': None, 'energy_p_value': None, 'valence_f_statistic': None, 'valence_p_value': None}, 'game_market_hhi': None, 'ratings': {'books_mean': None, 'books_median': None, 'movies_mean': None, 'movies_median': None}})


Question: Detect audio feature outliers in popular song charts from 2017, 2018, and 2020 using z-scores (|z| > 2) computed against the full Spotify track library's baseline statistics for danceability, energy, loudness, and valence. Also apply the same outlier detection to movie ratings, game sales, and book ratings. Output the results to output.csv with the following schema: dataset (string), outlier_count (integer). The CSV should contain one row per dataset (top2017, top2018, top50_2020, movies_metadata, vgsales, books) plus a total row.

Statistical Calculations and Descriptive Statistics

Mathematical and Statistical Computations


Question:
Build a cross-media popularity analysis system using the entertainment dataset (movies, Netflix, music, games).

Cluster Spotify music data using K-Means (select best k by silhouette score; random_state=42, n_init=10).

Calculate cross_media_popularity_score for each entity with three weighted components:
- Normalized popularity per media type, scaled to [0,1] (weight: 0.5; use 0.5 for missing values)
- Genre compatibility: fraction of other media types that share at least one genre with the entity after genre normalization (weight: 0.3)
- Temporal proximity: count of entities released within ±2 years of the entity's release year, normalized by max value (weight: 0.2)

Validate features using RandomForestRegressor (n_estimators=10, random_state=42).

Output to output.csv with columns: entity_id, title, type, release_year, cross_media_popularity_score, normalized_popularity, temporal_proximity, genre_compatibility.

Feature Selection and Dimensionality Reduction

Dynamic Data Transformation and Insertion

Numerical Comparison and Proximity Checks

Data Structure and Dictionary Operations

Statistical Testing for Feature-Target Evaluation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', specified_columns=['entity_id', 'title', 'type', 'release_year', 'cross_media_popularity_score', 'normalized_popularity', 'temporal_proximity', 'genre_compatibility'])


Question: Based on the pub_fund.sqlite database, find fund managers with total assets under management exceeding 10 billion, and their top 3 most frequently held stocks with the holding count. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: invest_advisor_name, sec_name, holding_count.

Query Construction and Logical Operators


Question: Based on the pub_fund.sqlite database, find the names of funds that rank in the top 10% of their fund type for both one-year return and three-year return. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: fund_name.

Query Construction and Logical Operators

Conditional Logic and Row-wise Operations


Question: Based on the pub_fin_data.sqlite database, find 农业银行's revenue for fiscal year 2024 and its revenue share in its industry. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: 营收, 营收占比. For numeric results, keep 营收占比 to 2 decimal places.

Conditional Logic and Row-wise Operations

SQL Set Operations and Query Combination

Lookup and Data Transformation Techniques

Mathematical and Statistical Computations


Question: Based on the pub_fin_data.sqlite database, for fiscal year 2024 annual consolidated financial statements, find the overall industry ROE and ROA for the industry where 金山办公 belongs. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: 行业总体ROE, 行业总体ROA. For numeric results, keep 行业总体ROE to 2 decimal places and 行业总体ROA to 1 decimal place.

Data Joining and Relationship Management

Conditional Logic and Row-wise Operations

Mathematical and Statistical Computations


Question: Based on the pub_fin_data.sqlite database, find the gross profit of companies established in the same year as 燕京啤酒 based on fiscal year 2024 consolidated financial statements. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: 公司名称, 毛利润. For numeric results, keep 毛利润 to 2 decimal places.

Query Construction and Logical Operators

Data Joining and Relationship Management


Question: Based on the pub_fin_data.sqlite database, find the top 10 companies with the fastest year-over-year growth in operating profit for H1 2025 and their growth rates. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: 公司名称, 营业利润同比增长率. For numeric results, keep 营业利润同比增长率 to 2 decimal places.

Incremental and Comparative Calculations


Question: Based on the pub_fin_data.sqlite database, for H1 2024 and H1 2025, which listed sector has the fastest growth in gross profit margin? List the sector and its gross profit margin growth value. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: listed_sector, 毛利率增长值. For numeric results, keep 毛利率增长值 to 2 decimal places.

Metadata and Documentation Review
Database Interaction and SQL
SQLite-Specific Operations
Data Aggregation and Grouping
Percentage and Variation Calculations
Arithmetic and Cumulative Calculations
Sorting, Limiting, and Ranking
Ranking and Top N Logic
Data Export and Output Processing
Data Serialization & File Handling


Question: Based on the pub_fin_data.sqlite database, comparing H1 2025 to H1 2024, find the names of the top three companies with the largest increase in operating costs as a proportion of total assets, and the specific increase values. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: company_name, increase_value. For numeric results, keep increase_value to 2 decimal places.

Incremental and Comparative Calculations

Conditional Logic and Row-wise Operations


Question: Based on the pub_fin_data.sqlite database, find the stock abbreviations of the top 5 companies with the largest increase in selling expense ratio from Q3 2024 to Q3 2025. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: stock_name.

Arithmetic Transformations and Normalization

Conditional Logic and Row-wise Operations

Data Joining and Relationship Management


Question: Based on the pub_fin_data.sqlite database, calculated by consolidated financial statements, for H1 2024 and H1 2025, which industry has the largest year-over-year decline in net profit margin? Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: industry_name.

Data Joining and Relationship Management


Question: Based on the pub_fin_data.sqlite database, calculated by consolidated financial statements, among all listed sectors, comparing Q3 2024 and Q3 2025, which sector has the fastest year-over-year growth in return on equity (ROE)? Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: listed_sector.

Cross/Merge Products and Filtering by Time Windows

Conditional Logic and Row-wise Operations

Mathematical and Statistical Computations


Question: Based on the pub_fin_data.sqlite database, for fiscal year 2024, find the names, stock abbreviations, and current ratios of companies in 医药制造业 with current ratios below the industry average. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: company_name, stock_name, 流动比率. For numeric results, keep 流动比率 to 2 decimal places.

Metadata and Documentation Review
Database Interaction and SQL
SQLite-Specific Operations
Query Construction and Execution
Subqueries and Complex Query Structures
Data Ingestion and Processing
Data Export and Output Processing
Formatting and Output Organization
Data Serialization & File Handling


Question: Based on the pub_fin_data.sqlite database, for the 2024 consolidated annual report, find the stock abbreviations and current ratios of companies with the highest current ratios in each industry. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: 行业名称, 公司名称, 流动比率. For numeric results, keep 流动比率 to 2 decimal places.

Data Joining and Relationship Management

Conditional Logic and Row-wise Operations


Question: Based on the pub_fin_data.sqlite database, for Q3 2025 consolidated financial statements, find the top 3 industries with year-over-year growth in net profit attributable to shareholders > 20% and the highest R&D expense to revenue ratio. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: industry_name.

Query Construction and Logical Operators

Conditional Logic and Row-wise Operations


Question: Based on the pub_fin_data.sqlite database, for fiscal year 2024, among companies listed on 主板, 创业板, and 科创板, find the company with the highest return on assets (ROA), and list its stock abbreviation, listing sector, and ROA value. Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: stock_name, listed_sector, ROA. For numeric results, keep ROA to 2 decimal places.

Data Joining and Relationship Management

Conditional Logic and Row-wise Operations


Question: Based on the pub_fin_data.sqlite database, for fiscal year 2024 consolidated financial statements, among the three listing sectors of 主板, 创业板, and 科创板, find the top 3 companies by return on assets (ROA). List the stock abbreviations, listing sectors, ROA, and their return on equity (ROE). Refer to the dataset_pub_fin_data.md document for database details. Save the final result as output.csv with the following columns: stock_name, listed_sector, ROA, ROE. For numeric results, keep ROA and ROE to 2 decimal places.

Mathematical and Statistical Computations


Question: Based on the insurance_business.sqlite database, find how much higher the claim approval rate is for policies with multiple claims compared to policies with only one claim. Refer to the dataset_insurance_business.md document for database details. Save the final result as output.csv with the following columns: 通过率差异(百分点). For numeric results, keep 通过率差异(百分点) to 2 decimal places.

Metadata and Documentation Review
Database Interaction and SQL
SQLite-Specific Operations
ORM and SQL Usage
Conditional Aggregation and Filtering
Unique Identifier and Entity Management
Unique Value Extraction
Difference and Trend Computation
Percentage and Variation Calculations
Arithmetic and Cumulative Calculations
Data Export and Output Processing
Data Storage and Structuring
Data Serialization & File Handling


Question: Based on the pub_fund.sqlite database, find the full names of equity funds where 贵州茅台 is one of the top three holdings and the one-year return is lower than the two-year annualized return. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: fund_name.

Conditional Logic and Row-wise Operations


Question: Based on the pub_fund.sqlite database, find the abbreviated names of non-FOF funds where the fund manager has fewer than 50 funds in total, the combined market value of the top three holdings accounts for less than 15% of the net asset value, and the year-to-date return is positive. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: fund_abbr_name.

Data Joining and Relationship Management

Conditional Logic and Row-wise Operations


Question: Based on the insurance_business.sqlite database, find how many days the average settlement cycle for each product type of 和泰保险 increased in 2024 compared to 2023. Refer to the dataset_insurance_business.md document for database details. Save the final result as output.csv with the following columns: 产品类型, 平均理赔时长增加天数. For numeric results, keep 平均理赔时长增加天数 to 2 decimal places.

Data Joining and Relationship Management

Time Difference and Gradient Calculation


Question: Based on the pub_fund.sqlite database, find the abbreviated fund names, the names of the largest holdings, and their proportion of net asset value for funds where the largest holding exceeds 15% of net asset value. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: fund_abbr_name, sec_name, ratio_nv. For numeric results, keep ratio_nv to 4 decimal places.

Threshold-Based Categorization or Filtering


Question: Based on the pub_fund.sqlite database, find the abbreviated names of hybrid funds where 寒武纪 is one of the top ten holdings. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: 基金简称.

Metadata and Documentation Review
Database Interaction and SQL
ORM and SQL Usage
Data Filtering and Grouping
Data Aggregation and Grouping
Data Aggregation and Window Functions
Window Functions and Partitioning
Ranking and Top N Logic
Data Filtering and Matching
Filtering and Sorting Correlation Data
Unique Value Extraction
Sorting, Limiting, and Ranking
Data Export and Output Processing
CSV Processing


Question: Based on the pub_fund.sqlite database, find the average year-to-date return and average three-year annualized return for two types of equity funds: high concentration (largest holding > 8% of net asset value) and medium-low concentration (<= 8%). Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: concentration_category, AVG(T2.rr_since_this_year), AVG(T2.annualized_rr_in_three_year). For numeric results, keep AVG(T2.rr_since_this_year) and AVG(T2.annualized_rr_in_three_year) to 15 decimal places.

Mathematical and Statistical Computations

Database Querying and Custom Aggregation


Question: Based on the pub_fund.sqlite database, find the top three concentrated industries for fund managers in the top 10% and bottom 10% by total assets under management. Refer to the dataset_pub_fund.md document for database details. Save the final result as output.csv with the following columns: 基金管理人分组, 重仓行业名称.

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['重仓行业名称'])

Conditional Logic and Row-wise Operations

Data Joining and Relationship Management


Question: Based on the multi-source medical health dataset (including lifestyle, metabolic indicators, gene expression, and disease outcome data), generate a comprehensive visual report in a 2×2 layout (saved as output.png) covering the following 4 analysis dimensions: (a) Heatmap: Use `sns.heatmap()` to show the average BMI distribution across age groups (10-year intervals) and genders. Map BMI categories to midpoint values (Normal = 22.5, Overweight = 27.5, Obese = 35.0). The x-axis represents gender, the y-axis represents age groups, using the YlOrRd color scheme with annotated values. (b) Scatter plot: Perform PCA dimensionality reduction on TCGA-PANCAN gene expression data and display the first two principal components, colored by cancer type. (c) Horizontal bar chart: Tokenize the occupation and smoking status text fields across datasets, merge word frequencies, and display the top 10 most frequent words. (d) Radar chart: Compare average risk characteristics (BMI, Glucose, Systolic BP, Age) across four disease cohorts (diabetes, stroke, heart disease, sleep disorder). Normalize each feature to 0-1 range (BMI/50, Glucose/200, Systolic BP/200, Age/100), with missing values defaulting to 0.5.

Dimensionality Reduction and Feature Engineering

Preprocessing and File Structure Adjustments

Categorical Data Preprocessing and Simplification

Arithmetic Transformations and Normalization

Normalization and Percentile Calculations

Functional Data Handling and Iterative Plotting


Question: Please build multiple prediction models for various chronic diseases based on the multi-source dataset of medical health (including diabetes, sleep disorders, stroke, heart disease, insurance costs, chronic kidney disease, and cancer gene expression data), and evaluate their performance. Load all relevant data files including structured health records, clinical datasets, and cancer gene expression data with corresponding labels. Treat sleep disorder, heart disease, insurance cost, and kidney disease as binary tasks (any disorder/disease vs none, above-median charges for insurance, and CKD vs not CKD). Use PCA to reduce the high-dimensional cancer gene expression data to 50 principal components. Train prediction models for diabetes, stroke, heart disease, sleep disorders, kidney disease, and cancer using RandomForestClassifier(n_estimators=100, random_state=42); train the insurance cost prediction model using LogisticRegression(penalty='l1', solver='liblinear', random_state=42, max_iter=1000). Divide the data into a stratified 80% training set and a 20% test set with random_state=42. Calculate accuracy, precision, recall, F1 score, and AUC-ROC. For multi-class cancer prediction, use macro averaging. Save the results to output.csv with columns task, accuracy, precision, recall, f1, auc, avg_score, and determine the best-performing task by avg_score.

Data Preprocessing and Column Management

Data Conversion and Post-Loading Processing

Feature Selection and Dimensionality Reduction

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', specified_columns=['accuracy','precision','recall','f1','auc','avg_score'], thresholds={1: 0.02, 2: 0.02, 3: 0.02, 4: 0.02, 5: 0.02, 6: 0.02})


Question:
A health research institution wants to analyze metabolic indicator differences between high-risk and low-risk patient groups across chronic disease datasets using Cohen's d effect size.

High-risk group definitions:
- Diabetes, Stroke, Insurance: Top 10% by metabolic risk score within each age decile (10 quantile-based groups)
  - Diabetes score: mean of (Glucose, BMI, BloodPressure) percentile ranks × (1 + 0.1 for each condition met: Glucose>140, BMI>30, BloodPressure>80)
  - Stroke score: mean of (avg_glucose_level, bmi) percentile ranks × (1 + 0.2 if hypertension=1 AND heart_disease=1)
  - Insurance score: bmi percentile rank × (1 + 0.1 × risk_tier), where risk_tier = min(smoker_binary + high_bmi_indicator, 2), high_bmi threshold: bmi>=30
- Heart Disease: num >= 1
- Kidney Disease: class = 'ckd'
- Sleep Health: Quality of Sleep <= 5

Biomarkers to analyze:
- Diabetes: Glucose, BMI, BloodPressure
- Stroke: avg_glucose_level, bmi
- Insurance: bmi
- Heart Disease: trestbps, chol
- Kidney Disease: bgr, bu, sc
- Sleep Health: systolic_bp (parsed from Blood Pressure field), bmi_numeric (mapped from BMI Category: Normal=22.5, Overweight=27.5, Obese=32.5)

Cohen's d interpretation: small (|d|<0.5), medium (0.5<=|d|<0.8), large (|d|>=0.8)

Output:
1. output.csv with columns: dataset, biomarker, high_risk_mean, low_risk_mean, cohens_d, interpretation
2. Return the dataset-biomarker combination with largest |Cohen's d| (alphabetical by dataset name for ties)

Normalization and Percentile Calculations

Mathematical and Statistical Computations

Feature Selection and Statistical Computation

Interpreting and Communicating Statistical Results

Incremental and Comparative Calculations

Numerical Comparison and Proximity Checks

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', specified_columns=['high_risk_mean','low_risk_mean','cohens_d'], ignore_order=True, thresholds={4: 0.02})


Question: Analyze biomarker differences between disease and non-disease groups across five chronic disease datasets (Diabetes, Stroke, Heart Disease, Kidney Disease, Sleep Health). From each dataset, extract and standardize four biomarkers where available: glucose, bmi, systolic_bp, and diastolic_bp. Stratify patients into age groups: 18-40, 40-60, and 60+. For each disease cohort, biomarker, and age group combination, calculate Cohen's d effect size (disease group minus non-disease group). Use bootstrap resampling (1000 iterations) to compute 95% confidence intervals. Exclude combinations where total samples < 10 or either group is empty. Save results to output.csv with columns: disease_cohort, biomarker, age_bin, cohen_d, ci_lower, ci_upper, n_cases, n_controls. Use disease cohort labels: Diabetes, Stroke, HeartDisease, CKD, Sleep.

Data Structure Creation and Manipulation

Data Conversion and Post-Loading Processing

Numerical Operations and Type Conversion

Data Structure Handling (Dictionaries, Lists)

Vectorization and Performance Optimization

Parameter Estimation and Bootstrap Methods

Numerical Comparison and Proximity Checks

Interpreting and Communicating Statistical Results

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['disease_cohort', 'biomarker', 'age_bin'], calculate_columns=['cohen_d', 'ci_lower', 'ci_upper'], metric='mae')