Question:
A sports science research team is conducting a comparative analysis of athlete body types across professional basketball, international Olympic competition, and European football to inform talent identification models.

OUTPUT FILES AND SCHEMAS:

1. output1.png - Kernel Density Estimation Plot
   REQUIRED SPECIFICATIONS:
   - Title: 'Cross-Sport Physique Comparison: Height vs Weight Distributions'
   - X-axis label: 'Height (cm)'
   - Y-axis label: 'Weight (kg)'
   - Legend title: 'Source' with three groups: NBA, Olympic, European Soccer
   - Plot type: 2D KDE (kernel density estimation) with filled contours
   - Transparency: alpha=0.4 for each group
   - Color palette: NBA='#1f77b4', Olympic='#ff7f0e', European Soccer='#2ca02c'

2. output2.json - NBA Height Peak Value
   REQUIRED SCHEMA:
   {
     "nba_height_peak_cm": (float) approximate height in cm where NBA density peaks
   }
   - Calculate using Gaussian KDE on NBA player height data
   - Find the height value where density is maximum
   - Precision: 1 decimal place recommended

REQUIRED ANALYSIS:
- Load athlete physique data from Olympic competitions (Basketball and Football events); duplicate athlete records across events do not need to be deduplicated
- Load NBA player physique data from professional basketball records
- Load European Soccer player physique data
- Filter all records to the 2008–2016 period for temporal alignment. For NBA players, retain those whose career overlapped with this window.
- Standardize height measurements to centimeters and weight to kilograms
- Generate a kernel density estimation (KDE) plot showing the joint distribution of height and weight for the three athlete groups
- Identify the height value where NBA player density reaches its maximum

DATA RELATIONSHIPS (for reference only):
- Olympic athlete data can be linked to country regions via NOC codes
- NBA player data includes height/weight information from common_player_info.csv
- European Soccer player data includes height/weight from the Player table in the SQLite database

FINAL OUTPUT: Both output1.png and output2.json files must be generated for successful evaluation.

Data Transformation and Feature Engineering

Arithmetic Transformations and Normalization

Data Transformation and Column Manipulation

Validation and Verification of Merge Results

Dynamic Data Transformation and Insertion

Statistical Plotting and Density Estimation

compare_json('output2.json', 'result2.json', thresholds={'nba_height_peak_cm': None})


Question:
A sports analytics firm is assessing cross-sport athlete physique trends to advise national federations on talent identification.

FINAL OUTPUT REQUIREMENT:
You MUST save the computed probability to a file named 'output.json' in the current working directory with the following exact schema:

{
  "probability": float (the empirical probability value, rounded to 4 decimal places),
  "shared_countries_count": integer (number of countries common to both Olympic medalists and NBA players),
  "nba_median_height": float (median height of NBA players in cm, rounded to 2 decimal places),
  "olympic_athletes_count": integer (number of Olympic medalist records from shared countries)
}

EVALUATION CRITERIA:
The output.json will be evaluated against result.json using compare_json() with thresholds for each field. Ensure your output exactly matches this schema.

REQUIRED ANALYSIS:

You are provided with datasets containing athlete information from multiple sources including Olympic athlete records (1980-2016) with physical attributes, competition results, and country information, as well as NBA player records with physical attributes, team affiliations (including founding year), and career information.

Your task is to compute the empirical probability that a randomly selected Olympic medalist record (1980-2016) from a country that also produced at least one NBA player on a team founded after 1970 has a height greater than the median height of those NBA players.

Key considerations:
- Olympic medalists should be filtered for years 1980-2016 with non-null Medal field
- NBA players should be filtered for teams founded after 1970
- Height data may need imputation for missing values using sport-specific or position-specific medians
- NBA heights are in feet-inches format and need conversion to centimeters (1 foot = 12 inches, 1 inch = 2.54 cm)
- Country names should be standardized across datasets for accurate matching (e.g., 'USA' vs 'United States')
- The probability should be calculated as: (number of Olympic medalist records from shared countries with height > NBA median) / (total number of Olympic medalist records from shared countries)

Logical Operators for Combining Conditions

Domain-Specific and Contextual Imputation

Data Structure Creation and Manipulation

Numerical Operations and Type Conversion

Probability Calculations and Statistical Computation

Numerical Comparison and Proximity Checks

Statistical Calculations and Descriptive Statistics

compare_json('output.json', 'result.json', thresholds={'probability': 0.05, 'shared_countries_count': 0, 'nba_median_height': 0.05, 'olympic_athletes_count': 0.05})


Question: A sports analytics firm is building a cross-sport talent identification system to detect elite athletes whose physical metrics significantly deviate from sport-specific norms. Analyze Olympic athlete records containing height, weight, sport, and NOC (National Olympic Committee) codes, along with NBA player data containing height, weight, position, and draft information. Identify all athletes who are statistical outliers in height or weight within their sport using IQR-based thresholds with an optimized multiplier (targeting 5-10% outlier ratio). The analysis should incorporate temporal context by detecting structural breaks and physique shifts in sport-wide metrics over time, as well as peak events where individual athletes exceed the 95th percentile of their sport/position. Return a pandas DataFrame with columns ['source', 'sport', 'name', 'metric', 'value', 'z_score', 'is_outlier', 'temporal_context', 'is_peak_event', 'is_physique_shift'], where each row corresponds to a detected extreme value. The output should be saved to 'output.csv'. Use source labels exactly 'Olympic' and 'NBA'; include only rows where is_outlier is True; compute z_score as a signed value rounded to two decimal places; encode temporal_context as 'structural_break_YYYY' for structural-break years and blank/NaN otherwise. NBA rows should use sport='Basketball', blank/NaN temporal_context, and is_physique_shift=False.

Rolling Statistics and Window-Based Signal Processing

Time Difference and Gradient Calculation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['source', 'sport', 'name', 'metric', 'value', 'z_score', 'is_outlier', 'temporal_context', 'is_peak_event', 'is_physique_shift'], thresholds={'value': None, 'z_score': None})


Question: A sports analytics firm wants to compare physical attributes (height and weight) across four elite athlete groups: NBA players, European soccer players, Olympic basketball athletes, and Olympic football athletes. The analysis integrates data from NBA draft combine measurements (in imperial units), a European soccer database where player heights are in cm and weights are in pounds, and Olympic athlete event records where heights are in cm and weights are in kg, with NOC region mappings. Segment all athletes into 4 physique clusters using K-Means (k=4, random_state=42) on their height (cm) and weight (kg) data, then evaluate pairwise differences between the groups by independently sampling 100 athletes from each group with random_state=42 and computing Cohen's d effect sizes and Welch's t-test p-values for both height and weight. Save results to 'output.csv' with columns: comparison_name, height_cohens_d, weight_cohens_d, height_pvalue_anova, weight_pvalue_anova. Use group labels NBA, Soccer, Olympic_Basketball, Olympic_Football, with comparison_name in format 'Group1 vs Group2' ordered as listed. Round Cohen's d to 4 decimal places. Format p-values in scientific notation (e.g., '1.23e-45') when below 0.0001, otherwise round to 6 decimal places.

Batch Processing and Performance Optimization

Dynamic Data Transformation and Insertion

Data Structure Understanding and Initialization

Feature Selection and Statistical Computation

Incremental and Comparative Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['comparison_name', 'height_cohens_d', 'weight_cohens_d', 'height_pvalue_anova', 'weight_pvalue_anova'], thresholds={'height_cohens_d': None, 'weight_cohens_d': None, 'height_pvalue_anova': None, 'weight_pvalue_anova': None})


Question: In data_5.csv, calculate the KS and LIFT values of the model on the held-out test set. The prediction_score column contains the model score. Rows with data_flag equal to 'train' or 'val' are the training set and must be used only to build 10 equal-frequency score bins after sorting prediction_score in ascending order. Rows with data_flag equal to 'test' or 'oot' are the held-out test set; calculate all output statistics on this held-out test set after assigning each row to the training-derived score bins. In this dataset, the held-out flag is 'oot'. Samples with value 1 in the '有资有意' column are positive samples, and samples with value 0 are negative samples. Use the format [A,B] to represent each training-derived bin boundary, rounded to 5 decimal places. Scores below the minimum training boundary should be assigned to bin 1; scores above the maximum training boundary should be assigned to bin 10. For each bin, calculate bin_cnt, pos_cnt, neg_cnt, pos_lift, pos_cum_rate, neg_cum_rate, and diff, where pos_lift = (bin positive rate) / (overall test-set positive rate), pos_cum_rate and neg_cum_rate are cumulative rates from low score to high score, and diff = abs(pos_cum_rate - neg_cum_rate). Add a total row with '\N' in credit_bin, 'total' in bin, test-set totals for bin_cnt/pos_cnt/neg_cnt, '\N' for pos_lift/pos_cum_rate/neg_cum_rate, and the maximum bin-level diff. Save the results to output.csv with headers: credit_bin, bin, bin_cnt, pos_cnt, neg_cnt, pos_lift, pos_cum_rate, neg_cum_rate, diff. The bin, pos_lift, pos_cum_rate, neg_cum_rate, and diff columns should all be kept to 5 decimal places. Missing values should be represented with '\N'. Sort rows by credit_bin ascending, with the total row last.

Data Loading with Pandas
Data Binning and Grid Creation
Statistical Analysis and Metrics
CSV Processing

compare_csv(output_file_name='output.csv', gold_file_name='result-strategy-2.csv', ignore_order=False, thresholds={'bin_cnt': 0, 'pos_cnt': 0, 'neg_cnt': 0, 'pos_lift': 1e-5, 'pos_cum_rate': 1e-5, 'neg_cum_rate': 1e-5, 'diff': 1e-5})


Question: In data_1.csv, calculate the conversion metrics for different score segments, including the number of outbound calls, calls connected, SMS sent, link clicks, checks, applications, and conversions. For scores above 0.99, use 0.001 as the bin width. Output the results to output.csv with headers: score_bucket, total, oncall, cnt_send, cnt_click, cnt_check, cnt_apply, cnt_upgrade. For score_bucket, use the format [A,B] for the highest bin and [A,B) for all other bins.

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output.csv', gold_file_name='result-strategy-3.csv', ignore_order=False, thresholds={'total': None, 'oncall': None, 'cnt_send': None, 'cnt_click': None, 'cnt_check': None, 'cnt_apply': None, 'cnt_upgrade': None})


Question: In data_6.csv, calculate monthly statistics for scored sample records. A scored sample record is a row where apply_date is not empty (equivalently, the score columns are populated in this dataset). Use the month derived from the dt column by integer division by 100. For each month, calculate: click_cnt = number of retained rows, register_cnt = number of rows where register_cnt > 0, apply_cnt = number of rows where apply_cnt > 0, credit_cnt = number of rows where credit_cnt > 0, credit_rate = credit_cnt / apply_cnt, credit_value = sum of credit_value, avg_credit_value = credit_value / credit_cnt, loan_cnt = number of rows where loan_cnt > 0, loan_cnt_30d = number of rows where loan_cnt > 0 and first_loan_time is 0 to 30 days after dt, loan_rate_30d = loan_cnt_30d / credit_cnt, loan_value_30d = sum of loan_value for rows counted in loan_cnt_30d, and avg_loan_value_30d = loan_value_30d / loan_cnt_30d. Values requiring decimals should be kept to 2 decimal places. Add a final overall summary row with mth equal to 'all', computed from all retained scored sample records rather than by averaging monthly rows. Save the results to output.csv with headers: mth, click_cnt, register_cnt, apply_cnt, credit_cnt, credit_rate, credit_value, avg_credit_value, loan_cnt, loan_cnt_30d, loan_rate_30d, loan_value_30d, avg_loan_value_30d. Sort month rows ascending and place the 'all' row last.

Data Loading with Pandas
Date and Time Arithmetic
Conditional Aggregation and Filtering
CSV Processing

compare_csv(output_file_name='output.csv', gold_file_name='result-strategy-5.csv', ignore_order=False, thresholds={'click_cnt': 0, 'register_cnt': 0, 'apply_cnt': 0, 'credit_cnt': 0, 'credit_rate': 0.005, 'loan_cnt': 0, 'loan_cnt_30d': 0, 'loan_rate_30d': 0.005})


Question: In data_6.csv, analyze credit approval rate and average credit amount by quality score bin for scored sample records. A scored sample record is a row where model_score_3_pct is not empty; use model_score_3_pct as the quality score. Derive the month from the dt column by taking its first six digits. Create 10 fixed-width quality_bin groups from model_score_3_pct: bin 1 is [0.0, 0.1), bin 2 is [0.1, 0.2), ..., bin 9 is [0.8, 0.9), and bin 10 is [0.9, 1.0]. Values outside [0, 1] should be clipped to the nearest endpoint before assigning the bin. For each quality_bin and each month 202512, 202601, and 202602, calculate credit_rate = count rows where credit_cnt > 0 divided by count rows where apply_cnt > 0, and avg_credit_value = sum credit_value for rows where credit_cnt > 0 divided by count rows where credit_cnt > 0. If a denominator is zero, output '\N'. Also add a quality_bin='total' row for each metric across all bins. The total column must exclude December 2025 (202512) and must be recomputed from the combined rows for months 202601 and 202602, not by averaging the displayed monthly values. Save the results to output.csv with headers: metric_type, quality_bin, 202512, 202601, 202602, total. The credit_rate rows should be kept to 5 decimal places, and avg_credit_value rows should be kept to 2 decimal places. Output all credit_rate rows for quality_bin 1 through 10 and total first, followed by all avg_credit_value rows for quality_bin 1 through 10 and total.

Data Loading with Pandas
Data Binning and Grid Creation
Conditional Aggregation and Filtering
CSV Processing


Question:
Assess whether there are systematic differences in user satisfaction among accommodation, indoor attractions, outdoor attractions, and destination experiences.
First, unify the four variables of accommodation satisfaction from short-term rental listings, indoor cultural attractions visit ratings, outdoor leisure venues visit ratings, and tourists' satisfaction with the destination from transaction records into a common scale of 1-5 points, then apply robust Z-score standardization to centralize each distribution. Calculate the mean, standard deviation, and 95% confidence interval for each of these four sources.
Conduct a one-way analysis of variance to test whether significant differences exist in the average satisfaction across different domains, and report the F-statistic, p-value, and eta-squared effect size.

Output: Two CSV files - output1.csv containing source statistics (mean, std, ci_lower, ci_upper), and output2.csv containing ANOVA results (statistic, p_value, effect_size)

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Normalization and Percentile Calculations

Arithmetic Transformations and Normalization

Robust Standard Errors and Covariance Estimation

Probability Distributions and Statistical Modeling

Mathematical and Statistical Computations

compare_csv(output_file_name='output1.csv', gold_file_name='result1.csv', ignore_order=True, specified_columns=['mean', 'std', 'ci_lower', 'ci_upper'])

compare_csv(output_file_name='output2.csv', gold_file_name='result2.csv', ignore_order=True, specified_columns=['statistic', 'p_value', 'effect_size'])


Question:
Study the ratings given by international tourists from different regions to various attractions in a Southeast Asian destination country, and interpret these ratings by combining macro-level indicators of the destination country.
First, calculate the average rating for each combination of attraction type from attraction type data and user's continent of origin from user data using transaction ratings from destination transaction data.
Then supplement each row with the destination country's GDP per capita from country macro data and its visa-free score from passport index data.

Output: Save the result to output.csv file. The DataFrame should contain columns: AttractionType, Continent, Avg_Rating, Country_GDP_per_Capita, Visa_Free_Score

Data Joining and Relationship Management

Data Transformation and Column Manipulation

Numerical Operations and Type Conversion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['AttractionType', 'Continent', 'Avg_Rating', 'Country_GDP_per_Capita', 'Visa_Free_Score'])


Question:
Construct a unified Tourism Attractiveness Score (TAS) system to conduct cross-source benchmarking and ranking of high-performing entities across short-term rentals, indoor attractions, and outdoor attractions.
First, filter high-performing entities from each data source (quality indicators ≥ 4.0 and valid demand indicators for short-term rentals, indoor attractions, and outdoor attractions; perfect rating for attraction transactions).
Then perform standardization processing on quality indicators like ratings and demand indicators like visit counts, availability days, and transaction frequency to eliminate dimensional differences, and calculate the raw TAS score for each entity as the average of standardized quality and demand indicators. Add an extra 0.1 point bonus for entries whose entity names contain UNESCO World Heritage related keywords (such as 'heritage', 'temple', 'palace', 'monument', 'cathedral', 'castle'), and finally combine entities from all sources, sort by TAS score in descending order, and extract the top 10.

Output: A list of the top 10 entities ranked by TAS in CSV format(output.csv), containing four columns: source, entity_id, entity_name, tas_score

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['entity_id', 'entity_name', 'tas_score'])


Question:
Analyze how weather conditions influence visitor satisfaction across accommodation and attraction types.
First, integrate multiple data sources including accommodation listings, indoor and outdoor attractions, and tourist site reviews.
Then, extract text features from data source category labels using TF-IDF vectorization with max_features=5 to encode categorical information.
Next, fit a Ridge regression model with alpha=1.0 to predict standardized ratings based on text features. Generate residuals and identify the histogram bin with highest frequency.
Then, estimate the effect of weather variables (temperature, rainfall) on ratings using statistical regression analysis.
Finally, apply K-Means clustering with k selected by Silhouette Score (testing k=2,3,4) on the model features to identify optimal tourism market segments.
Output(output.csv): A CSV file with columns 'metric_type', 'value'. Rows include: ridge_residual_peak_bin, ridge_r2_score, bayesian intercept and coefficient parameters, kmeans_best_k, kmeans_silhouette_score, kmeans center dimensions, and cluster sizes. The metrics are ridge_residual_peak_bin, ridge_r2_score, bayesian_intercept_mean, bayesian_intercept_std, bayesian_tmax_mean, bayesian_tmax_std, bayesian_rain_mean, bayesian_rain_std, kmeans_best_k, kmeans_silhouette_score, kmeans_center_dim0, kmeans_center_dim1, kmeans_center_dim2, kmeans_cluster_0_size, kmeans_cluster_1_size, kmeans_cluster_2_size, kmeans_cluster_3_size

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Feature Selection and Dimensionality Reduction

Vectorization and Performance Optimization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['metric_type', 'value'])


Question:
Evaluate user satisfaction differences across four tourism domains: accommodation, indoor attractions, outdoor attractions, and destination experiences through statistical analysis.
First, calculate satisfaction scores from accommodation satisfaction data in short-term rental data, cultural attraction satisfaction scores in indoor attraction data, natural attraction satisfaction scores in outdoor attraction data, and destination experience satisfaction scores in attraction transaction data.
 Then normalize ratings to a common scale, compute descriptive statistics including mean, median, standard deviation, and skewness for each group, perform six pairwise Kolmogorov-Smirnov two-sample tests to compare rating distribution differences and report p-values, and determine whether statistically significant differences exist between accommodation and attraction rating distributions at significance level α=0.05.

Output: Cross-domain satisfaction analysis results in JSON format (output1.json) with the following structure: top-level keys 'descriptive_statistics' (containing sub-keys 'Airbnb', 'Indoor Attractions', 'Outdoor Attractions', 'TripAdvisor', each with float values for 'mean', 'median', 'std', 'skewness'), 'kolmogorov_smirnov_tests' (containing sub-keys for each pairwise comparison formatted as '<Source1> vs <Source2>', e.g. 'Airbnb vs Indoor Attractions', each with 'ks_statistic' and 'p_value'), and 'accommodation_vs_attraction_test' (with 'ks_statistic' and 'p_value'); descriptive statistics in CSV format (output2.csv) with columns ['Source', 'mean', 'median', 'std', 'skewness']; Kolmogorov-Smirnov test results in CSV format (output3.csv) with columns ['Comparison', 'ks_statistic', 'p_value']; a histogram visualization (output4.png) with a 2x2 subplot layout showing rating distributions separately for each of the four tourism sources (one subplot per source), with each subplot titled by its source name, graph_title='Rating Distribution by Tourism Source', x_label='Rating (1-5)', y_label='Frequency'; and a box plot visualization (output5.png) comparing rating distributions across tourism sources with graph_title='Rating Distribution Comparison Across Tourism Sources', x_label=['Airbnb', 'Indoor', 'Outdoor', 'TripAdvisor'], y_label='Rating (1-5)'

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Normalization and Percentile Calculations

Statistical Calculations and Descriptive Statistics

Multiple Comparisons and Hypothesis Testing

Statistical Testing for Feature-Target Evaluation

Interpreting and Communicating Statistical Results

Dynamic Data Transformation and Insertion

Data Structure Creation and Manipulation

compare_json(output_file_name='output1.json', gold_file_name='result1.json', thresholds={'descriptive_statistics': {'Airbnb': {'mean': None, 'median': None, 'std': None, 'skewness': None}, 'Indoor Attractions': {'mean': None, 'median': None, 'std': None, 'skewness': None}, 'Outdoor Attractions': {'mean': None, 'median': None, 'std': None, 'skewness': None}, 'TripAdvisor': {'mean': None, 'median': None, 'std': None, 'skewness': None}}, 'kolmogorov_smirnov_tests': {'Airbnb vs Indoor Attractions': {'ks_statistic': None, 'p_value': [0, 0.001]}, 'Airbnb vs Outdoor Attractions': {'ks_statistic': None, 'p_value': [0, 0.001]}, 'Airbnb vs TripAdvisor': {'ks_statistic': None, 'p_value': [0, 0.001]}, 'Indoor Attractions vs Outdoor Attractions': {'ks_statistic': None, 'p_value': [0, 0.001]}, 'Indoor Attractions vs TripAdvisor': {'ks_statistic': None, 'p_value': [0, 0.001]}, 'Outdoor Attractions vs TripAdvisor': {'ks_statistic': None, 'p_value': [0, 0.001]}}, 'accommodation_vs_attraction_test': {'ks_statistic': None, 'p_value': [0, 0.001]}})

compare_csv(output_file_name='output2.csv', gold_file_name='result2.csv', ignore_order=True, specified_columns=['Source', 'mean', 'median', 'std', 'skewness'])

compare_csv(output_file_name='output3.csv', gold_file_name='result3.csv', ignore_order=True, specified_columns=['Comparison', 'ks_statistic', 'p_value'])


Question:
Conduct cross-market benchmarking analysis of tourist satisfaction and demand patterns across three markets: New York City accommodation, UK indoor attractions, and outdoor attractions.
First, extract pricing and service fee data from short-term rental data for outlier removal using IQR method, extract rating and visit data from indoor attraction data for IQR-based outlier filtering, and extract rating and visit data from outdoor attraction data for IQR-based outlier filtering. Then calculate mean, median, standard deviation, interquartile range (IQR), and total count of valid entries for each of the three sub-domains after cleaning.

Output output.json: JSON array containing eight fields: subdomain (NY_Accommodation, UK_Indoor_Attractions, Outdoor_Attractions), mean_value (mean of price for Airbnb or rating for attractions), median_value (median of price/rating), std_value (standard deviation), iqr_value (interquartile range), total_count (count of valid entries), has_significant_peak (whether count > 100), peak_month (July if has peak, else N/A)

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Numerical Operations and Type Conversion

Statistical Calculations and Descriptive Statistics

Mathematical and Statistical Computations

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'subdomain': None, 'mean_value': None, 'median_value': None, 'std_value': None, 'iqr_value': None, 'total_count': None, 'has_significant_peak': None, 'peak_month': None}, matched_keys=['subdomain'])


Question:
Construct a unified time-series panel dataset for cross-regional tourism demand analysis by integrating multiple tourism data sources.

The panel should combine tourism activity data from four categories across three regions:
- Indonesia attractions (TripAdvisor data): geo_category = 'Indonesia_Attractions', with attraction_type from the native type classification, aggregated monthly by attraction type
- UK indoor attractions (FISETIO dataset): geo_category = 'UK_Museums', attraction_type = 'Museum', aggregate total visits by date
- UK outdoor attractions (FISETIO dataset): geo_category = 'UK_Parks', attraction_type = 'Park', aggregate total visits by date
- NYC accommodations (Airbnb data): geo_category = 'NYC_Accommodations', attraction_type = 'Accommodation', count reviews by date as demand proxy

Enrich all rows uniformly with Indonesia's GDP per capita (from the countries dataset) and visa-free score (from the Henley passport index).

Output: Save as output.csv with columns: month_year, geo_category, attraction_type, monthly_visits, gdp_per_capita, visa_free_score.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Data Structure Understanding and Initialization

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['month_year', 'geo_category', 'attraction_type', 'monthly_visits', 'gdp_per_capita', 'visa_free_score'])


Question:
Cross-domain tourism benchmarking: one **2×2** **matplotlib** figure saved as **output1.png** from the bundled **NYC rental**, **UK indoor/outdoor monthly** panels, **Indonesian** transaction+taxonomy tables, **national GDP/population**, and **passport** scores.

**Top-left** — boxplot of **review score** by **room type**; **graph_title** `Airbnb Review Ratings by Room Type`, **x_label** `Room Type`, **y_label** `Review Rating`, tick **labels** for each room type.

**Top-right** — **dual-axis** time series of **mean monthly visits** (indoor vs outdoor) and **mean temperature**; **graph_title** `Monthly Visits and Temperature Trends`, **x_label** `Month`, left **y_label** `Average Visits`, right **y_label** `Temperature (°C)`, **labels** distinguishing indoor visits, outdoor visits, and average temperature.

**Bottom-left** — horizontal **mean rating** by **attraction type**; highlight **`Beaches`** in **coral** and show its **numeric** mean; **graph_title** `Average TripAdvisor Ratings by Attraction Type (Indonesia)`, **x_label** `Average Rating`, **y_label** `Attraction Type`.

**Bottom-right** — scatter **GDP per capita** vs **visa-free score**, point area from **national population** (use **sqrt** scaling for marker size), **viridis**-style color mapping of visa-free score; **graph_title** `GDP per Capita vs Visa-Free Score\n(Bubble size = Population)`, **x_label** `GDP per Capita ($)`, **y_label** `Visa-Free Score`, **colorbar** labeled `Visa-Free Score`.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Preprocessing and File Structure Adjustments

compare_image(output_file_name='output1.png', gold_file_name='result1.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question:
Cross-market tourism benchmark: from the bundled **NYC short-term rental**, **UK indoor/outdoor attraction**, and **Indonesian attraction-transaction** sources, build **one composite heatmap** comparing **satisfaction** (ratings read as a **1–5** scale, **cool** colors for lower and **warm** for higher) and **relative demand** (use each market’s natural volume proxies—reviews, visits, or transactions—apply **log1p**, then **min–max** demand **across the three markets** so **saturation** reflects cross-market intensity).

Layout: synthetic **x**-axis with three vertical bands labeled **`NYC (Airbnb)`**, **`UK (FISETIO)`**, **`Indonesia (TripAdvisor)`**; **y**-axis **10** graded satisfaction levels; each cell blends **hue** (satisfaction) and **saturation** (demand), with **brighter rows toward the bottom**; overlay **KDE**-derived shapes from the empirical rating structure; **colorbar** ticked **1.0–5.0** for satisfaction; annotate each band with **mean satisfaction** and **normalized demand** (e.g. μ / D style). Save as **output.png**.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Directory Structure and Validation Split Setup

Data Conversion and Post-Loading Processing

Normalization and Percentile Calculations

Mathematical and Statistical Computations

Statistical Plotting and Density Estimation


Question:
Construct a unified tourism entity dataset from multi-source tourism data, and identify tourism archetypes using NMF topic modeling and K-Means clustering.

Integrate five categories of tourism entities: New York Airbnb short-term rental listings (limited to the first 500 entries with valid names), world heritage sites, UK indoor attractions, UK outdoor attractions, and Indonesia TripAdvisor attractions. For each entity, extract descriptive text and numerical features including demand indicators and ratings. Enrich each entity with its corresponding country's per capita GDP and visa-free score.

Use TF-IDF (max_features=500, ngram_range=(1,2), min_df=2, max_df=0.8, stop_words='english') for text vectorization. Apply NMF (n_components=8, random_state=42, max_iter=500) for topic extraction. Construct a hybrid feature matrix combining topic weights with StandardScaler-normalized numerical features (demand proxy, rating, per capita GDP, visa-free score), then apply K-Means (n_clusters=6, random_state=42, n_init=10) for archetype clustering.

Entity IDs follow the format: 'airbnb_{idx}', 'whc_{idx}', 'indoor_{idx}', 'outdoor_{idx}', 'indonesia_{AttractionId}', where idx is the original DataFrame row index. Source labels: 'Airbnb', 'World_Heritage', 'UK_Indoor', 'UK_Outdoor', 'Indonesia_TripAdvisor'.

Output columns: entity_id, source, dominant_topic, cluster_label, topic_weights. Save as output.csv and output.json.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Vectorization and Performance Optimization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['entity_id', 'source', 'dominant_topic', 'cluster_label'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'entity_id': None, 'source': None, 'dominant_topic': None, 'cluster_label': None}, matched_keys=['entity_id'])


Question:
Perform a data quality assessment on multi-source global tourism datasets by detecting anomalous visitor patterns.

Construct an SQLite database from four tourism data source categories, retaining only numerical features: (1) Airbnb short-term rental accommodation data, (2) FISETIO indoor attraction visit data, (3) FISETIO outdoor attraction visit data, and (4) TripAdvisor Indonesia tourism data.

Split each data source into training (70%) and validation (30%) sets (random_state=42). On the training sets only, apply Isolation Forest (contamination=0.1, random_state=42) with StandardScaler normalization and median imputation for missing values to detect anomalies.

Generate a quality assessment report containing source, total_records, anomalous_records, and outlier_percentage (rounded to 2 decimal places). Save the report as output.csv and output.json.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Data Structure Understanding and Initialization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=[ 'total_records', 'anomalous_records', 'outlier_percentage'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'total_records': None, 'anomalous_records': None, 'outlier_percentage': None}, matched_keys=['total_records'])


Question:
This is a cross-domain destination attractiveness benchmarking task. Construct a standardized country-level feature matrix by integrating multi-source tourism data with macroeconomic indicators.

For each tourism data source, compute Z-score standardized ratings (using each source's own mean and standard deviation) and determine a visit volume proxy based on record counts:
- For the short-term rental data, retain only US listings.
- For UK indoor and outdoor attraction data, fit LinearRegression to model weather effects on visit volumes and compute residuals. Combine indoor and outdoor results for the UK using a weighted average by record count.
- For Indonesia tourism transaction data, join transactions with user and country information, and aggregate by user source country.

Before joining macroeconomic data, collapse duplicate country_name rows across sources by summing visit_volume_proxy and computing domain_rating_zscore as a count-weighted average.

Perform additive seasonal decomposition with an annual cycle on monthly-resampled time series to extract trend components. Apply K-Means clustering (k=3, random_state=42) on domain_rating_zscore and visit_volume_proxy to assign trend segments.

Join the aggregated country-level results with macroeconomic data for per-capita GDP and passport index data for visa-free scores. Fill missing gdp_per_capita with 15000 and missing visa_free_score with 100.

Save results as output.csv and output.json (records-oriented), with columns: country_name, domain_rating_zscore, visit_volume_proxy, gdp_per_capita, visa_free_score, trend_cluster.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Data Transformation and Feature Engineering

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['country_name', 'domain_rating_zscore', 'visit_volume_proxy', 'gdp_per_capita', 'visa_free_score'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'country_name': None, 'domain_rating_zscore': None, 'visit_volume_proxy': None, 'gdp_per_capita': None, 'visa_free_score': None}, matched_keys=['country_name'])


Question:
Perform a cross-domain behavioral archetype clustering analysis across multi-source tourism data (short-term rental listings, indoor attraction visits, outdoor attraction visits, and attraction transaction records).

For each data source, extract rating and visit/review volume features at the feature-row level. Apply MinMax normalization to ratings and log1p transformation to visit volumes. Compute seasonal peak intensity once per source_type by aggregating monthly normalized ratings and fitting an ARIMA(1,1,1) model: define peaks as residual values exceeding residual mean + 0.5 × residual std, and calculate peak_month_ratio as the proportion of peaks. Use 0.5 as default when fewer than 24 monthly time points are available. For the transaction data, aggregate records by attraction before feature extraction, so each attraction contributes one feature row.

Assign source_type labels: 'Airbnb', 'UK_Indoor', 'UK_Outdoor', 'Indonesia_TripAdvisor'. Apply K-Means clustering independently per source_type to the resulting feature rows with StandardScaler-standardized features. Select optimal k from [2, 10] using Silhouette Score (random_state=42, n_init=10). Assign globally unique cluster IDs starting from 0, incrementing sequentially across source types in the listed order.

Compute per-cluster statistics over the feature rows assigned to each cluster: avg_normalized_rating, median_log_visits, peak_month_ratio, behavioral_diversity_score (std of normalized ratings), region_count (the number of feature rows in the cluster). Add constant macro indicators: avg_gdp_per_capita=25000, avg_literacy_pct=85.0, avg_visa_free_score=150.

Output columns: cluster_id, source_type, avg_normalized_rating, median_log_visits, peak_month_ratio, behavioral_diversity_score, region_count, avg_gdp_per_capita, avg_literacy_pct, avg_visa_free_score. Save as output.csv and output.json.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Arithmetic Transformations and Normalization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['cluster_id', 'source_type', 'avg_normalized_rating', 'median_log_visits', 'peak_month_ratio', 'behavioral_diversity_score', 'region_count', 'avg_gdp_per_capita', 'avg_visa_free_score'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'cluster_id': None, 'source_type': None, 'avg_normalized_rating': None, 'median_log_visits': None, 'peak_month_ratio': None, 'behavioral_diversity_score': None, 'region_count': None, 'avg_gdp_per_capita': None, 'avg_visa_free_score': None}, matched_keys=['cluster_id', 'source_type'])


Question:
A **premium-destination** screening pipeline over the bundled **short-term rental**, **UK indoor/outdoor attractions**, **Indonesian transactions** (with user join where needed), **GDP**, **passport mobility**, and **UNESCO** coordinates. An entity passes only if **all** apply: **(1)** country **GDP per capita > USD 20,000** and **visa-free score ≥ 150**; **(2)** **≥ 4.5** mean score for listings and **≥ 4.0** for attractions; **(3)** demand proxy **above the category’s 75th percentile** (computed on the pre-filter pool for that category); **(4)** **indoor** sites dropped if **Pearson r(rain, visits) ≤ −0.3** on their own series; **(5)** listings **≤ USD 200** after price cleaning and within **10 km** (great-circle) of **some** heritage point; **(6)** **one consistent country-name normalization** across tables before joins.

After filtering, run **K-Means** on **rating**, **visit volume**, and **price** (missing price → **0**), **standardize** features, search **k = 2…10** (capped by sample size), pick **k** with best **silhouette**. Export **output.csv** and **output.json** containing only **entity_id**, **source_type**, **rating**, **visit_volume**, **country**.

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Data Transformation and Feature Engineering

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['entity_id', 'source_type', 'rating', 'visit_volume', 'country'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'entity_id': None, 'source_type': None, 'rating': 0.1, 'visit_volume': 0.1, 'country': None}, matched_keys=['entity_id', 'source_type'])


Question:
Construct a unified cross-regional tourism demand benchmark system to integrate real visitor transactions from Southeast Asian markets with synthetic performance indicators from major Western markets for assessing tourism resilience across diverse regions and data sources.
First, enrich each individual TripAdvisor Indonesia transaction record with visitor origin attributes and macroeconomic context by joining demographic dimensions and country-level indicators. Preserve the original transaction-level granularity: do not aggregate, deduplicate, or summarize TripAdvisor records. For these records, set country to the destination country Indonesia; set origin_country from the visitor origin country; join visa_free_score on origin_country; join gdp_per_capita and population on the destination country Indonesia; use the numeric VisitMode value for visit_mode; and leave date, visits, rolling metrics, weather correlations, and ARIMA fields blank.
Then, process temporal visitation patterns from the FISETIO Western indoor attraction data at the original row level. Sort all FISETIO indoor records globally by date, without grouping by individual attraction, before calculating rolling_3m, rolling_6m, rolling_9m, and rolling_12m as rolling cumulative sums over the globally sorted records. Quantify the global relationships between rain, temperature, sunshine, and visits, and attach those correlation values to each FISETIO indoor record.
Next, apply ARIMA(1,1,1) forecasting on the 12-month rolling cumulative visits series resampled to monthly frequency, and create exactly 12 forward-looking forecast records.
Then, process short-term rental market data from Western markets by filtering Airbnb listings where country is 'United States' and selecting key attributes including property identifiers, location information, accommodation types, pricing, and review metrics. Preserve one output row per selected Airbnb listing; do not aggregate listings.
Finally, integrate standardized entity records from all regional sources and external market data into a unified analytical schema with consistent attribute definitions, capturing data provenance, geographic identifiers, entity characteristics, quality ratings, demand volumes, temporal aggregations, environmental correlations, and predictive indicators. Use these exact provenance labels: TripAdvisor Indonesia transaction records must have source='TripAdvisor_Indonesia' and data_type='real_transactions'; FISETIO indoor attraction records must have source='FISETIO_UK_Indoor' and data_type='synthetic_indicators'; Airbnb United States listing records must have source='Airbnb_US' and data_type='synthetic_indicators'; ARIMA forecast records must have source='FISETIO_UK_Indoor' and data_type='arima_forecast'.
Output(output.csv): A unified analytical table in CSV format containing cross-regional tourism demand benchmark data with standardized columns including source,country,origin_country,data_type,date,entity_name,entity_type,rating,visit_mode,visa_free_score,gdp_per_capita,population,visits,rolling_3m,rolling_6m,rolling_9m,rolling_12m,rain_visits_corr,temp_visits_corr,sunny_visits_corr,arima_forecast

datasets/tourism/countries of the world.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

Lookup and Data Transformation Techniques

Data Structure Creation and Manipulation


Question:
Construct a unified performance benchmark system to identify top-tier visitor experiences across accommodation, indoor attractions, and outdoor destinations by analyzing satisfaction ratings from multiple data sources.
First, extract valid rating records from each source and normalize satisfaction scores to a 0-1 scale using per-source Min-Max normalization (each source's min and max computed independently): process short-term rental ratings from accommodation data, indoor attraction ratings from museum visit records, outdoor attraction ratings from park visit data, and transaction ratings from Southeast Asian attraction reviews after joining with entity dimension tables.
Then, calculate the global 90th percentile threshold across all normalized scores to establish a high-performance benchmark. Next, count rating records exceeding this threshold from each of the four source categories, including only records with valid ratings.
Finally, compute statistical summaries for interpretability by calculating mean normalized ratings and 95% bootstrap confidence intervals for each source based on resampling methods.
Output(output.csv): A CSV file with columns 'metric_type', 'Airbnb', 'UK_Indoor', 'UK_Outdoor', 'Indonesia_Attractions'. Rows include: global_90th_percentile (value stored in the 'Airbnb' column; 'UK_Indoor', 'UK_Outdoor', 'Indonesia_Attractions' columns left empty), high_performing_count, total_valid_ratings, mean, ci_lower, ci_upper.

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Arithmetic Transformations and Normalization

Normalization and Percentile Calculations

Mathematical and Statistical Computations

Data Structure Handling (Dictionaries, Lists)

Parameter Estimation and Bootstrap Methods

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['metric_type', 'Airbnb', 'UK_Indoor', 'UK_Outdoor', 'Indonesia_Attractions'])


Question:
Construct a unified predictive model for tourist satisfaction (rating) and visit volume across accommodation, indoor attractions, and outdoor sites by integrating the provided data sources into a common schema. For datasets without a price field, use review count as a proxy for price. After numeric conversion, keep only complete records with positive rating, visits, and final price/proxy price. Label the data source categories as 'Airbnb', 'UK_Indoor', and 'UK_Outdoor'.
Use the log1p-transformed price feature named 'price_log' and one-hot encoded data source features with the 'source_' prefix as the only input features. Train separate RandomForestRegressor models (n_estimators=100, random_state=42) for rating and visits prediction respectively, with a 20% test split (random_state=42). Calculate R² and MAE for both targets, and extract feature importance scores.
Output(output.csv): A CSV file with columns 'metric_type' and 'value', including: rating_r2, rating_mae, visits_r2, visits_mae, feature importance as {target}_importance_{feature_name}, and model/dataset metadata (n_estimators, random_state, test_size, total_samples, train_samples, test_samples, n_features). Round metric values to 4 decimal places where applicable.

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Numerical Operations and Type Conversion

Feature Selection and Dimensionality Reduction

Arithmetic Transformations and Normalization

Data Transformation and Column Manipulation

Categorical Data Preprocessing and Simplification

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['metric_type', 'value'])


Question:
Develop a unified tourism health index to compare tourism dynamics across accommodation, indoor attractions, and outdoor attractions from 2010 to 2019.
First, integrate multiple data sources including Airbnb listings, UK indoor and outdoor attractions, and Indonesian tourist transactions. Compute monthly metrics for each source including visit volume proxies and average ratings.
Then, calculate a health index for each source by combining visit volume and ratings normalized within each source independently using Min-Max scaling (40% each), and a fixed macroeconomic adjustment factor of 0.5 (20%), producing a composite score between 0 and 1.
Next, apply a 3-month rolling average to smooth the health index time series for each source.
Finally, use time series forecasting (Exponential Smoothing) to predict the next 12 months of health index values for each source and include both historical and forecasted values in the output.
Output(output.csv): A CSV file with columns 'year_month' (format 'YYYY-MM'), 'source' (Airbnb, FISETIO_Indoor, FISETIO_Outdoor, TripAdvisor), 'health_index' (normalized score), 'is_forecast' (True/False).

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Normalization and Percentile Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['year_month', 'source', 'health_index', 'is_forecast'])


Question:
Develop a composite Tourism Performance Index to benchmark high-performing destinations across accommodation, cultural, and natural attraction categories.
First, identify high-performing accommodation listings with availability exceeding 200 days annually and review ratings of at least 4.0.
Then, detect visitation peak months from indoor and outdoor attraction data where monthly visits exceed the mean plus one standard deviation, retaining only peaks with indoor ratings ≥4.0 or outdoor ratings ≥4.5.
Next, filter for countries with high travel accessibility by joining country reference data with passport index data and selecting nations with visa-free score ≥170.
Then, isolate 5-star rated visits from attraction transaction data and reconstruct user origin context through data joins.
Finally, aggregate all qualifying records into a unified performance index on a 0–100 scale, grouped by Country, Year-Month, and AttractionCategory ('Accommodation', 'Museum-like', 'Park-like', or 'Attraction'), using quantile-normalized performance scoring.
Output(output.csv): A CSV file with columns 'Country', 'Year_Month', 'AttractionCategory', 'Performance_Index'.

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Normalization and Percentile Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['Country', 'Year_Month', 'AttractionCategory', 'Performance_Index'])


Question:
Assess cross-market consistency in visitor satisfaction across accommodation listings, indoor attractions, and TripAdvisor attraction transactions, and in demand drivers for indoor and outdoor attractions.
First, compute the mean review rating for each of the three satisfaction domains—accommodation listings, indoor attractions, and TripAdvisor attraction transactions.
Then, calculate Pearson correlation coefficients between weather variables and monthly visits for indoor attractions (correlating maximum temperature, rainfall, and sunny hours with visit counts) and separately for outdoor attractions (correlating temperature, precipitation, and temperature difference with visit counts).
Next, perform a one-way ANOVA test to determine whether the mean ratings differ significantly across the three satisfaction domains and report both the F-statistic and p-value.
Then, compute the seasonal amplitude (standard deviation of the seasonal component) derived from an additive seasonal decomposition of total monthly visits over time, using a period of 12 months.
Finally, calculate linear regression coefficients (standardized) linking user country-level GDP per capita and visa-free travel score to attraction ratings, after enriching user data via joins to country reference data.
Output(output.csv): A CSV file with columns 'metric_type', 'value' containing all computed statistics.

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

datasets/tourism/countries of the world.csv

Statistical Calculations and Descriptive Statistics

Multiple Comparisons and Hypothesis Testing

Dictionary Manipulation and Construction

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['value'])


Question:
Benchmark normalized tourism activity across three distinct markets—accommodations, cultural attractions, and tourist sites—on a common monthly scale from 2010 to 2019.
First, calculate a composite activity metric per month for accommodation listings, grouped by the 'last review' date of each listing based on the product of total price volume (sum of price) and average review rating (mean of review rate number) per month.
Then, compute total monthly visits across all indoor and outdoor attractions.
Next, count monthly transactions as a proxy for visitation.
Finally, normalize each market's monthly activity metric to a 0–1 scale using min-max scaling within that market over the 2010–2019 period.
Output(output.csv): A CSV file with columns 'year_month' (formatted as 'YYYY-MM'), 'source' (with values 'Airbnb_NYC', 'FISETIO_UK', or 'TripAdvisor_IDN'), and 'activity_index' (the normalized score).

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/outdoors.csv

Timestamp Conversion and Time Manipulation

Arithmetic Transformations and Normalization

Normalization and Percentile Calculations

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['year_month', 'source', 'activity_index'])


Question:
Assess whether high-quality segments across accommodation, indoor attractions, and attraction transactions exhibit consistent visitor satisfaction patterns to support unified quality benchmarks in global tourism analytics.
First, identify high-quality accommodation listings in Manhattan (neighbourhood group = 'Manhattan') with verified hosts, price under $200, and annual availability greater than 100 days.
Then, select museum visits during winter months (December, January, February) with ratings above 4.0 from indoor attraction data.
Next, extract transactions where the visit mode corresponds to couples and rating is at least 4.
Compute the mean, median, population standard deviation (ddof=0), and skewness of ratings for each filtered dataset.
Then perform pairwise Kolmogorov-Smirnov two-sample tests between each pair of these three rating distributions and report all p-values.
Finally, determine whether, at a significance level of α=0.05, the null hypothesis that the rating distributions are identical can be rejected for any pair.
Output(output.csv): A CSV file with columns 'metric_type', 'n', 'mean', 'median', 'std', 'skew', 'ks_pvalue', 'significant' containing descriptive statistics and KS test results. For KS-test rows, write 'Yes' in 'significant' when the null hypothesis can be rejected at α=0.05, and 'No' otherwise.

datasets/tourism/FISETIO A FIne-grained, Structured and Enriched Tourism Dataset for Indoor and Outdoor attractions/datasets/indoors.csv

Statistical Calculations and Descriptive Statistics

Mathematical and Statistical Computations

Multiple Comparisons and Hypothesis Testing

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['n', 'mean', 'median', 'std', 'skew', 'ks_pvalue', 'significant'])


Question:
Compute urban transportation pressure index for NYC taxi zones. Perform spatial join between taxi zones and NY census tracts to extract PT_P and WFH_P. Calculate average monthly pickups per zone (Q2 2024, rounded to integer). From NCR data, find vehicle type with highest No Driver Found cancellation rate. Compute pressure index: (monthly_trips × (1 − WFH_P/100)) / (PT_P + 1), sort descending.

Output: output.json with zone, borough, monthly_pickup_count, PT_P, WFH_P, urban_mobility_stress_index, ncr_high_cancel_vehicle.

Dynamic Data Transformation and Insertion

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'urban_mobility_stress_index': 0.01}, matched_keys=['zone'])


Question:
Analyze NYC taxi travel patterns across boroughs to assess travel demand changes from January to February 2024.
Filter out trips with abnormal values (distance: 0-100 miles, fare: 0-1000$) to ensure data quality. Identify the borough with the largest absolute increase in average total fare from January to February.
First, output output1.json: JSON array containing analysis results for each borough and month with columns: borough (string), month (int), avg_trip_distance (float), avg_total_fare (float), trip_count (int), absolute_increase (float), is_max_increase (boolean).
Second, output output2.png with two subplots: (1) left subplot showing average trip distance comparison with graph_title='Average Trip Distance by Borough (Jan vs Feb 2024)', x_label='Borough', y_label='Average Trip Distance (miles)'; (2) right subplot showing average total fare comparison with graph_title='Average Total Fare by Borough (Jan vs Feb 2024)', x_label='Borough', y_label='Average Total Fare ($)'.

Data Conversion and Post-Loading Processing

Dynamic Data Transformation and Insertion

Column-wise Transformations and Aggregation

Incremental and Comparative Calculations

compare_json(output_file_name='output1.json', gold_file_name='result1.json', thresholds={'borough': None, 'month': None, 'avg_trip_distance': 0.05, 'avg_total_fare': 0.05}, matched_keys=['borough', 'month'])


Question:
Train a linear regression model to predict NYC yellow taxi fares (Q2 2024) using features: time (hour, day of week), trip (distance, passenger count), and borough dummy variables for pickup/dropoff (borough_pu_*, borough_do_*). Filter outliers using strict ranges: 0 < trip_distance < 100 miles and 0 < fare_amount < 500$. Split data (test_size=0.2, random_state=42). Output RMSE and R².

Output: output.json with {'RMSE': float, 'R2': float}.

Timestamp Conversion and Time Manipulation

Dynamic Data Transformation and Insertion

Data Transformation and Column Manipulation

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'RMSE': None, 'R2': None})


Question:
Identify NYC taxi zones with low public transit dependence (PT_P < 10%) and high car dependence (CA_P > 75%) that intersect with census tracts, then calculate their average monthly taxi pickups (Jan-May 2024), filtering zones with avg pickups > 100. Identify NCR vehicle type with highest No Driver Found cancellation rate (percentage) as benchmark.

Output: output.json with zone, borough, PT_P, avg_monthly_trips, ncr_vehicle_type, ncr_cancellation_rate.

Batch Processing and Performance Optimization

Conditional Logic and Row-wise Operations

Dynamic Data Transformation and Insertion

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'zone': None, 'borough': None, 'PT_P': None, 'avg_monthly_trips': None, 'ncr_vehicle_type': None, 'ncr_cancellation_rate': None}, matched_keys=['zone'])


Question:
Compute borough-level mobility metric comparing NYC and NCR transportation efficiency. Load NYC taxi data (Jan-Sep 2024), calculate total taxi revenue per borough. Load NCR data, compute total booking value and No Driver Found cancellation rate (percentage). Perform spatial join between taxi zones and NY census tracts to count tracts per borough. Calculate mobility metric: (taxi_revenue - ncr_booking_value) / tract_count for each borough.

Output: output.json with borough, taxi_revenue, ncr_booking_value, tract_count, mobility_metric, ncr_cancellation_rate.

Batch Processing and Performance Optimization

Cross/Merge Products and Filtering by Time Windows

Dynamic Data Transformation and Insertion

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'borough': None, 'taxi_revenue': None, 'ncr_booking_value': None, 'tract_count': None, 'mobility_metric': None, 'ncr_cancellation_rate': None}, matched_keys=['borough'])


Question:
Train a random forest regression model (n_estimators=100) to predict hourly taxi demand per NYC borough (Q2 2024). Build features: hour, day of week, weekend flag, and historical demand lag features like 1 hour ago, 2 hours ago, and 24 hours ago. Include borough dummy variables. Use temporal split (80:20, no shuffle) and random_state=42. Output RMSE and MAE.

Output: output.json with {'RMSE': float, 'MAE': float}.

Dynamic Data Transformation and Insertion

Row-Dependent and Iterative Computations

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'RMSE': None, 'MAE': None})


Question:
This is a cross-city travel pattern comparison task. Using New York City yellow taxi trip data (January to September 2024), taxi zone geographic data, and India NCR ride-hailing booking data, identify the top 10 cross-borough origin-destination pairs in NYC by total trip count (excluding same-borough trips), and match each to the most similar NCR origin-destination pair.

Semantic matching uses the following predefined keyword sets for each NYC borough:
- Manhattan: ['central', 'connaught', 'barakhamba', 'pragati', 'aiims', 'saket']
- Brooklyn: ['noida', 'faridabad', 'ghaziabad', 'indirapuram', 'vasundhara']
- Queens: ['dwarka', 'palam', 'mahipalpur', 'airport', 'igi', 'aerocity']
- Bronx: ['rohini', 'pitampura', 'shalimar', 'ashok', 'model town', 'jahangir']
- Staten Island: ['vasant', 'munirka', 'sarojini', 'lajpat', 'greater kailash']

Confidence score calculation:
1. For pickup and dropoff respectively, compute the ratio of matched keywords (case-insensitive substring match) in the corresponding NCR location name to the total keyword count for that borough. Average the two ratios.
2. Add 0.03 bonus for each hub keyword present in either NCR location name: ['airport', 'metro', 'station', 'sector', 'chowk', 'garden', 'nagar', 'colony'].
3. If the NYC pair is Manhattan→Brooklyn/Queens, add 0.1 when the NCR dropoff location contains any of ['noida', 'faridabad', 'ghaziabad', 'dwarka']. If Brooklyn/Queens→Manhattan, add 0.1 when the NCR pickup location contains any of those keywords.
4. Cap the score at 1.0 and round to 4 decimal places.

For each NYC borough pair, select the NCR pair with the highest confidence score. Save the result as output.csv with columns: pickup_borough, dropoff_borough, nyc_trip_count, ncr_pickup_location, ncr_drop_location, ncr_ride_count, analog_confidence_score (exactly 10 rows).

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Data Structure Understanding and Initialization

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['pickup_borough', 'dropoff_borough', 'nyc_trip_count', 'ncr_pickup_location', 'ncr_drop_location', 'ncr_ride_count', 'analog_confidence_score'])


Question:
This is an urban transportation service vulnerability assessment and cross-regional ride-hailing service comparison analysis task.

Part 1: Identify taxi service vulnerability zones by combining NYC taxi zone boundaries, national transportation mode survey data (filtered to New York State records only), and yellow taxi trip records from January to June 2024. For each taxi zone, calculate its average public transit commuting percentage by spatially associating it with overlapping census tracts. Determine two thresholds: the 95th percentile of maximum monthly pickup counts across zones, and the 10th percentile of average public transit commuting percentage across zones. A zone is considered vulnerable if its maximum monthly pickup count >= the 95th percentile threshold AND its average public transit commuting percentage <= the 10th percentile threshold. For each vulnerable zone, compute a Service Vulnerability Index (SVI) = (1 - pt_p/100) × (max_monthly_pickups / median of max_monthly_pickups across all zones).

Part 2: Using ride-hailing booking data, analyze cancellation patterns. For each combination of pickup location and vehicle type, calculate the cancellation rate as a percentage. Filter for combinations with at least 10 total bookings, then select the top 5 with the highest cancellation rates.

Part 3: Generate a histogram of public transit commuting percentage distribution across all taxi zones, with a red vertical dashed line marking the 10th percentile threshold.

Output files:
- output.csv: columns location_id, zone_name, borough, pt_p, max_monthly_pickups, service_vulnerability_index
- output.json: columns pickup_location, vehicle_type, cancellation_rate, cancelled_count, total_count
- output_image.png (type='bar', graph_title='Distribution of Public Transit Usage Across NYC Taxi Zones', x_label='Public Transit Commuting Percentage (PT_P)', y_label='Number of Taxi Zones')

Batch Processing and Performance Optimization

Data Conversion and Post-Loading Processing

Dynamic Data Transformation and Insertion

Normalization and Percentile Calculations

Threshold-Based Categorization or Filtering

Data Transformation and Column Manipulation

Logical Operators for Combining Conditions

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['location_id', 'zone_name', 'borough', 'pt_p', 'max_monthly_pickups', 'service_vulnerability_index'])

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'cancellation_rate': 0.05, 'cancelled_count': None, 'total_count': None}, matched_keys=['pickup_location', 'vehicle_type'])

compare_image(output_file_name='output_image.png', gold_file_name='result_image.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label'])


Question: Using commuting transportation mode survey data, New York taxi zone boundary data, and yellow taxi trip records from April to September 2024, identify high-potential zones suitable for comprehensive transportation intervention measures. Determine the public transportation commuting proportion for each taxi zone by spatially associating zones with New York State census tract-level commuting data. Aggregate taxi trip data by pickup zone to calculate average trip distance and total pickup count. A high-potential zone must simultaneously satisfy: average trip distance greater than 5 miles, public transportation commuting proportion greater than 30%, and total pickup count at or above the 80th percentile across all zones. Sort qualifying zones by pickup count in descending order and save to output.csv with columns: zone, borough, avg_trip_distance, pt_commuting_pct, pickup_count. Also generate output2.csv containing a statistical summary of all zones with columns: LocationID, zone, borough, avg_trip_distance, pt_commuting_pct, pickup_count.

Batch Processing and Performance Optimization

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['zone', 'borough', 'avg_trip_distance', 'pt_commuting_pct', 'pickup_count'])

compare_csv(output_file_name='output2.csv', gold_file_name='result2.csv', ignore_order=True, specified_columns=['LocationID', 'zone', 'borough', 'avg_trip_distance', 'pt_commuting_pct', 'pickup_count'])


Question:
This is a cross-regional travel pattern comparison analysis task. Compare travel characteristics between New York City taxi data and India National Capital Region (NCR) ride-hailing data, aggregated by geographic units.
For NYC: Using the transportation mode survey data (filtered to New York State) and taxi zone boundaries, compute the average public transit commuting proportion for each taxi zone. Then load January to April 2024 yellow taxi trip data and aggregate by borough to calculate total ride counts, average trip distances, and mean public transit commuting proportions.
For NCR: Filter ride-hailing bookings to orders with status 'Completed' or 'Incomplete'. Clean pickup locations by removing suffixes (Sector, Block, Phase, Colony, Vihar, Enclave, Extension, Area) and standardizing city name variants (e.g., Gurugram to Gurgaon). Aggregate by cleaned city, keeping only cities with at least 100 rides.
Generate a unified comparison table sorted by total rides descending. Output as output.csv with columns: geography, total_rides, avg_trip_distance, pt_commuting_pct. NYC rows include pt_commuting_pct values while NCR rows have missing values.

Dynamic Data Transformation and Insertion

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['geography', 'total_rides', 'avg_trip_distance', 'pt_commuting_pct'])


Question:
Analyze the relationship between public transportation usage and taxi demand across New York City boroughs. Use commuting mode survey data (filtered to New York State), taxi zone boundaries, yellow taxi trip records from January to September 2024, and India NCR ride-hailing booking data.
For each borough: (1) spatially associate census tracts with taxi zones to obtain public transportation usage rates, and calculate the borough-level average; (2) calculate the mean of zone-level average trip distances and total pickup count; (3) include total NCR ride-hailing bookings (total number of records) as a cross-market benchmark, with the same value for all boroughs.
Generate output.csv with columns: borough, avg_trip_distance, pickup_count, public_transit_pct, total_ncr_bookings.

Data Structure Understanding and Initialization

Column-wise Transformations and Aggregation

Batch Processing and Performance Optimization

Dynamic Data Transformation and Insertion

Mathematical and Statistical Computations

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['borough', 'avg_trip_distance', 'pickup_count', 'public_transit_pct', 'total_ncr_bookings'])


Question:
Transportation policy research: compare how **trip distance** relates to **revenue** for **NYC yellow taxis** versus **ride-hailing in India’s National Capital Region**, focusing on NYC **boroughs with low public-transit commuting**. Use the project’s commute survey layer (New York state only), taxi zone boundaries, NYC yellow trip extracts for **January–March 2024**, and the NCR booking extract.

Define low-transit boroughs by spatially linking zones to commute areas, averaging each borough’s public-transit commute share, and taking boroughs **under 10%**; if none qualify, use the borough(s) with the **lowest** average (same tie-break as a defensible default). Restrict NYC trips to those boroughs with **positive distance and fare**; restrict NCR to **completed** trips with **positive distance and paid amount**.

Build **one** figure that overlays both markets: **50 equal-width** distance bins from **0** to the **overall maximum** distance across both sides; within each bin, sum **fare** (NYC) and **booking value** (NCR) as **total revenue** per bin. Include **title**, **axis labels**, **grid**, and **legend**. Report which bin has the **largest NYC yellow-taxi revenue** and write that closed interval to **output.txt** (e.g. `17.2–18.3 miles`, using a single en-dash between bounds). Save the figure as **output.png**.

Numerical Comparison and Proximity Checks

Vectorization and Performance Optimization


Question:
Analyze how community-level public transportation commuting behavior affects taxi demand patterns in New York City. Using commuting mode survey data (New York State), taxi zone boundaries, and yellow taxi trip records (January–September 2024), compute a 7-day (168-hour) rolling average of hourly pickup volumes stratified by pickup zones' public transportation commuting proportion (PT_P) quintiles.

For each taxi zone, determine its PT_P by averaging values from spatially intersecting census tracts. Assign each trip to one of 5 PT_P quintiles (labeled 0–4) based on its pickup zone. Ensure complete hourly coverage with 0-filled gaps and use min_periods=1 for the rolling window.

Generate output.csv with columns: hour_bin, pt_quintile, rolling_avg_pickups.

Data Structure Handling (Dictionaries, Lists)

Timestamp Conversion and Time Manipulation

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['hour_bin', 'pt_quintile', 'rolling_avg_pickups'])


Question:
Conduct a comprehensive transportation pattern analysis to examine the relationship between NYC taxi usage and local commuting behaviors, while establishing cross-regional benchmarks against ride-hailing dynamics in another metropolitan region.
First, load and consolidate five months of yellow taxi trip records from January to May 2024, then aggregate pickup statistics including total trips, average base fare (fare_amount column), and average trip distance by pickup location.
Load taxi zone metadata and merge with aggregated statistics using inner join to enrich location information (only retaining zones with trip data).
Load census commuting data and filter to New York State census tracts (STATEFP = '36'), then perform spatial intersection analysis using 'intersects' predicate to associate taxi zones with commuting behavior metrics including public transit usage rate (PT_P), car-alone commute rate (CA_P), and work-from-home rate (WFH_P).
Load ride-hailing data from another metropolitan region to compute global benchmarks for average ride distance and booking cancellation rate (including 'Cancelled by Driver' and 'Cancelled by Customer' statuses). Deduplicate records by LocationID using groupby().first() to ensure one row per zone, identify top 10% high-traffic zones using 90th quantile thresholding, append cross-regional benchmark metrics as comparative references, and sort results by total pickup volume in descending order.
Output: A pandas DataFrame containing one row per NYC taxi pickup zone that has trip data and spatially overlaps with a New York census tract (STATEFP = '36'), with columns: LocationID, zone, borough, total_pickups, avg_fare, avg_trip_distance, PT_P, CA_P, WFH_P, is_top_10_pct, benchmark_avg_ride_distance, benchmark_cancellation_rate. Save the result to output.csv.

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Vectorization and Performance Optimization

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['total_pickups', 'avg_fare', 'avg_trip_distance', 'PT_P', 'CA_P', 'WFH_P', 'is_top_10_pct', 'benchmark_avg_ride_distance', 'benchmark_cancellation_rate'])


Question:
Analyze the relationship between public transit usage and taxi inflow patterns in New York City to understand complementary or competing mobility dynamics.
First, load and consolidate three months of yellow taxi trip records from January to March 2024, then calculate total weighted inflow (sum of trip distances) for each destination zone (DOLocationID). Load taxi zone geometries and merge with aggregated inflow statistics using inner join (only retaining zones with trip data).
Load census commuting data and filter to New York State census tracts (STATEFP = '36'), then perform spatial intersection analysis using 'intersects' predicate to associate taxi zones with public transit commuting percentage (PT_P).
Deduplicate records by LocationID using groupby().first() to ensure one observation per zone, remove records with missing values using dropna(), compute the Pearson correlation coefficient between weighted inflow and PT_P, and report the result rounded to four decimal places.
Output(output.csv): A CSV file containing the Pearson correlation coefficient (rounded to 4 decimal places) in a single column named 'correlation'.

Dynamic Data Transformation and Insertion

Vectorization and Performance Optimization


Question:
Analyze the relationship between commuting behavior and on-demand taxi pricing across New York City zones using hierarchical Bayesian regression.
First, load nine months of yellow taxi trip records and compute zone-month level average fare per mile (filtering for fare_amount>0 and trip_distance>0). Load taxi zone geometries and census commuting data (STATEFP='36'), then perform spatial joins using zone centroid and census tract containment to associate each zone with public transit commuting share (PT_P) and work-from-home share (WFH_P).
Apply a one-dimensional Kalman filter within each zone (F=H=1, process_variance=0.1, measurement_variance=1.0), initialized with x0 equal to the first observed average fare per mile and P0=1, and use the filtered state after each predict/update step as the denoised value. Construct a hierarchical Bayesian regression model with zone-specific random intercepts and slopes using non-centered parameterization, where the response variable is the Kalman-filtered fare per mile and predictors are the standardized PT_P and WFH_P, with likelihood precision weighted by monthly trip counts. Use PyMC priors global_intercept~Normal(0,10), beta_pt_p~Normal(0,5), beta_wfh_p~Normal(0,5), random intercept/PT slope/WFH slope scales~HalfNormal(2), and observation sigma~HalfNormal(5).
Use PyMC for MCMC inference with 2,000 post-tune samples, tune=2000, target_accept=0.9, and random_seed=42. Remove records with missing values before model fitting.
Output(output.csv): A CSV file containing the posterior mean and 94% highest-density interval (HDI) for the global parameters (global_intercept, beta_pt_p, beta_wfh_p), with columns: parameter, mean, hdi_3%, hdi_97%.

Data Conversion and Post-Loading Processing

Column-wise Transformations and Aggregation

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Advanced Modeling with State Space Framework

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['parameter', 'mean', 'hdi_3%', 'hdi_97%'], thresholds={'parameter': None, 'mean': 0.3, 'hdi_3%': 0.3, 'hdi_97%': 0.3})


Question:
Construct a unified rider archetype identification system to analyze urban mobility patterns by integrating trip transaction data with census-level commuting behavior across multiple time periods.
First, load and process multi-month trip records from urban transportation datasets, extracting temporal features including pickup times, fare amounts, trip distances, and zone identifiers, while applying data quality filters to ensure valid records.
Then, enrich trip data with geographic and demographic context by joining zone geometries and metadata, and spatially associate with census commuting data to assign mode-specific percentages including public transit usage, car availability, and work-from-home rates to each zone.
Next, construct a feature representation capturing zone trip counts across five time-of-day periods: night [0,6), morning_rush [6,9), daytime [9,17), evening_rush [17,21), late_evening [21,24) and commuting modes, apply PCA with 3 components to distill essential patterns, and apply K-Means clustering (k=5, random_state=42, n_init=10) to identify distinct rider archetypes.
Finally, post-process the identified clusters by determining dominant geographic regions and computing geographic centroids for each archetype.
Output(output.csv): A CSV file containing rider archetype analysis results with columns: cluster_id, dominant_borough, centroid_longitude, centroid_latitude.

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Feature Selection and Dimensionality Reduction

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['cluster_id', 'dominant_borough', 'centroid_longitude', 'centroid_latitude'])


Question:
Build a zone-level urban transportation demand forecasting and pattern analysis system by integrating multi-month NYC yellow taxi trip data, census commuting behavior data, and ride booking operational records.
For demand forecasting: use 2024 January–August trip records as training data and September as validation. Filter to records with positive fare and positive trip distance, with pickup zone IDs between 1 and 263 inclusive, extracting temporal features including pickup times, fare amounts, trip distances, and zone identifiers. Aggregate to zone-month level trip counts as the demand target. Enrich zones with commuting mode percentages from NTAD census data (New York State only) via nearest spatial join, filling missing values with median. Train an MLPRegressor (hidden_layer_sizes=(64, 32), max_iter=500, random_state=42, early_stopping=True, validation_fraction=0.1) with StandardScaler-normalized features: sine/cosine cyclic month encoding and commuting mode percentages (public transit, car availability, work-from-home). Evaluate on validation data using MAE, RMSE, R², and Pearson correlation at overall and per-borough levels.
For clustering analysis: apply dimensionality reduction to distill essential zone-level patterns from trip and commuting features, and perform clustering analysis to identify distinct mobility archetypes. Aggregate zone-level mean trip count, average fare, and average distance from training data, combined with public transit and work-from-home commuting percentages. Standardize, apply PCA (n_components=3), then KMeans (n_clusters=4, random_state=42, n_init=10). Report the silhouette score.
For text analysis: from the ride booking records, clean text by removing non-alphanumeric characters, apply TF-IDF vectorization (max_features=100, English stop words removed), and extract the top 5 terms associated with vehicle breakdown themes based on mean TF-IDF scores.
Output(output.json): A JSON file structured as {'overall': {'MAE': val, 'RMSE': val, 'R²': val, 'correlation': val}, 'per_borough': {borough: {'MAE': val, 'RMSE': val, 'R²': val, 'correlation': val}, ...}, 'best_silhouette_score': val}, all values rounded to 4 decimal places.
Output(output.csv): A CSV file with a single column 'top_5_vehicle_breakdown_terms' containing the top 5 terms.

Data Conversion and Post-Loading Processing

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Feature Selection and Dimensionality Reduction

Vectorization and Performance Optimization

Data Structure Handling (Dictionaries, Lists)

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'overall': {'MAE': 0.05, 'RMSE': 0.05, 'R²': 0.05, 'correlation': 0.05}, 'per_borough': None, 'best_silhouette_score': 0.1})


Question:
Construct a unified urban mobility analysis system to evaluate the relationship between local commuting behavior and on-demand transportation usage across NYC taxi zones. Use 2024 January through September yellow taxi trip data, taxi zone geographic boundaries, and NTAD census commuting data (filtered for New York State). Filter trip records to exclude non-positive fares and invalid zone IDs. Aggregate pickup counts and average fares per zone, then spatially associate each zone with its nearest census tract to obtain public transit usage and work-from-home percentages (fill missing values with median). Apply K-Means clustering (k=4, random_state=42, n_init=10) on standardized (zero-mean, unit-variance) features to classify zones. Also generate borough-level aggregate statistics, with borough average fare as the simple mean of zone-level averages and each borough's percentage share of total pickups rounded to 2 decimal places.

Output(output.csv): Zone-level cluster results with columns: LocationID, borough, cluster_id, total_pickups, avg_fare, PT_P, WFH_P.
Output(output2.csv): Borough-level statistics with columns: borough, total_pickups, avg_fare, percentage.

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Column-wise Transformations and Aggregation

Data Conversion and Post-Loading Processing

Mathematical and Statistical Computations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['LocationID', 'borough', 'cluster_id', 'total_pickups', 'avg_fare', 'PT_P', 'WFH_P'])

compare_csv(output_file_name='output2.csv', gold_file_name='result2.csv', ignore_order=True, specified_columns=['borough', 'total_pickups', 'avg_fare', 'percentage'])


Question:
Construct a unified urban mobility demand forecasting system to predict hourly transportation pickup intensity by integrating multi-month historical trip transaction records with census-level commuting patterns and geographic zone information.
First, load and split multi-month historical trip records into training (months 1-8) and testing (month 9) sets, extracting timestamps and zone identifiers while applying data quality filters to ensure valid records.
Then, aggregate transaction data at the hourly and zone level to create demand targets, and engineer temporal features including hour-of-day, day-of-week, month indicators, weekend flags, and cyclic encodings to capture periodic patterns.
Next, enrich the dataset with geographic context by loading zone geometries and spatially associating with census commuting data to assign mode-specific percentages including public transit usage, work-from-home rates, and car availability to each zone.
Then, merge temporal features with commuting characteristics to create a comprehensive feature matrix, and apply XGBoost regression (n_estimators=200, learning_rate=0.05, max_depth=6, random_state=42, objective='reg:squarederror') to model hourly demand patterns.
Finally, perform 3-fold time-series cross-validation to assess model stability, train the final model on the complete training set, evaluate predictive performance on the holdout test set using standard regression metrics, extract feature importance scores to identify key drivers of demand, and export all model metadata and evaluation results.
Output(output.json): A JSON file containing model configuration (best_hyperparameters), cross-validation performance (cross_validated_r2), test set metrics (test_r2, test_mae), and feature importance rankings (feature_importances).

Data Conversion and Post-Loading Processing

Timestamp Conversion and Time Manipulation

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Data Structure Handling (Dictionaries, Lists)

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={'cross_validated_r2': 0.1, 'test_r2': 0.1, 'test_mae': 0.1})


Question:
Construct a unified urban mobility pattern analysis system to examine the relationship between local commuting behavior and on-demand transportation usage across administrative regions by integrating multi-month trip transaction records with census-level commuting patterns and geographic zone information.
First, load and process multi-month historical trip records from urban transportation datasets, extracting timestamps and zone identifiers while applying data quality filters to ensure valid records.
Then, aggregate transaction data at the monthly and zone level to create temporal demand patterns, and enrich trip data with geographic context by joining zone geometries and administrative boundaries.
Next, spatially associate trip data with census commuting data to assign mode-specific percentages including public transit usage to each zone, and compute aggregated statistics at the administrative region level.
Then, generate comprehensive visualizations including bar charts showing total demand distribution by region with percentage annotations, and scatter plots revealing relationships between transportation usage and commuter behavior profiles.
Finally, export temporal statistics at multiple aggregation levels and visualization outputs for comprehensive mobility pattern analysis.
Output(output2.csv): A CSV file containing monthly borough statistics with columns: borough, month, total_pickups, avg_pt_p.Output(output.csv): A CSV file containing borough summary with columns: borough, total_pickups, avg_pt_p, percentage.Output(output.png): A PNG visualization with two subplots. Top subplot: A bar chart showing total yellow taxi pickups by borough (Jan-Sep 2024) with x-axis labeled 'Borough', y-axis labeled 'Total Pickups (9 months)', and percentage annotations on each bar. Bottom subplot: A scatter plot showing the relationship between average public transit commute percentage (PT_P) and total pickups by borough, with x-axis labeled 'Average Public Transit Commute Percentage (PT_P)', y-axis labeled 'Total Pickups (9 months)', and borough labels on each data point.

Timestamp Conversion and Time Manipulation

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

compare_csv(output_file_name='output2.csv', gold_file_name='result2.csv', ignore_order=True, specified_columns=['borough', 'month', 'total_pickups', 'avg_pt_p'])

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['borough', 'total_pickups', 'avg_pt_p', 'percentage'])

compare_image(output_file_name='output.png', gold_file_name='result.png', calculate_columns=['type', 'graph_title', 'x_label', 'y_label', 'xtick_labels'])


Question:
Construct a unified urban mobility typology analysis system to identify distinct transportation usage patterns across administrative regions by integrating multi-month trip transaction records with census-level commuting patterns and geographic zone information.

First, load and process nine months (January-September 2024) of yellow taxi trip records, filtering for valid zone identifiers (PULocationID and DOLocationID within 1-263 range) and positive fare amounts and trip distances.
Then, aggregate transaction data at the zone level to compute mobility intensity metrics including pickup frequency (trip count), average fare, and average trip distance.
Next, enrich zone-level data with geographic context by loading taxi zone geometries and NTAD census commuting data (filtered to New York State, STATEFP='36'), then use nearest-neighbor spatial join (sjoin_nearest) to assign public transit usage rate (PT_P) and work-from-home rate (WFH_P) to each zone, deduplicating by distance and filling missing values with median.
Then, standardize features (pickup_intensity, avg_fare, avg_trip_distance, PT_P, WFH_P) using StandardScaler, and apply K-Means clustering (k=4, random_state=42, n_init=10) to segment zones into distinct mobility-commuting profiles.
Next, compute cross-borough flow intensities by analyzing origin-destination trips, filtering out intra-borough trips, and aggregating inter-borough trip volumes by pickup borough.
Finally, aggregate cluster assignments with cross-borough flow metrics and commuting characteristics at the cluster-borough level, sort by cluster_id and borough, and export the unified typology results.

Output(output.csv): A CSV file containing mobility typology results with columns: cluster_id, borough, avg_inter_borough_flow, mean_PT_P, mean_WFH_P.

Dynamic Data Transformation and Insertion

Batch Processing and Performance Optimization

Feature Selection and Statistical Computation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['cluster_id', 'borough', 'avg_inter_borough_flow', 'mean_PT_P', 'mean_WFH_P'])


Question:
A transportation analytics firm is assessing temporal demand patterns in urban taxi systems to inform fleet allocation strategies.
The analysis begins by loading taxi zone boundary data and trip records spanning multiple months, then initializes a DuckDB database to facilitate efficient data processing.
All monthly trip datasets are consolidated into a unified view through SQL operations to enable comprehensive temporal analysis.
Trip records are enriched with borough information by joining pickup locations with zone boundaries, enabling spatial aggregation of demand patterns. Hourly pickup counts are computed for each administrative district by grouping trips based on their pickup time and location, after which a centered 7-day rolling average is applied to smooth short-term fluctuations and reveal underlying demand trends.
The final step identifies the maximum value of this smoothed hourly pickup series across all boroughs and the entire analysis period.
The query pipeline leverages SQL for data retrieval and transformation, with time-series smoothing implemented as a rolling window operation over datetime-indexed hourly aggregates.
Results should be saved to output.txt in the following format:
Line 1: "Maximum 7-day rolling average hourly pickups: <value>" (rounded to 2 decimal places)
Followed by a blank line, then "Top 10 highest values:"
Then 10 lines, each formatted as: "<borough> | <YYYY-MM-DD HH:MM> | <value>" (value rounded to 2 decimal places), sorted in descending order.

Data Structure Creation and Manipulation

Data Structure Understanding and Initialization

SQL Set Operations and Query Combination

Efficient Data Structures and Algorithms