Question:
Based on three medical health datasets (diabetes, stroke, and chronic kidney disease), analyze the differences in blood glucose levels among patient groups. Filter for patients aged 30-70 and exclude records where glucose is 0 or missing. Unify the glucose columns across datasets and label each record's source as Diabetes, Stroke, or CKD.

Perform a Kruskal-Wallis H test on the blood glucose distributions of the three groups. If the result is significant (p < 0.05), conduct Dunn's post-hoc test with Bonferroni correction. Save results to output.csv with columns: test, comparison, statistic, p_value, significant (use True/False for the significant column). In the test column, use "Kruskal-Wallis" for the overall test and "Dunn post-hoc" for pairwise comparisons. In the comparison column, use "all" for the overall test, and format pairwise comparisons as "{Group1} vs {Group2}" (e.g., "Diabetes vs Stroke", "Diabetes vs CKD", "Stroke vs CKD").

Create an overlaid density histogram of the three groups' blood glucose distributions (bins=50, alpha=0.6; Diabetes in red, Stroke in blue, CKD in green). Annotate the Kruskal-Wallis p-value on the figure. Save as output.png.

Data Conversion and Post-Loading Processing

Data Structure Understanding and Initialization

Dynamic Data Transformation and Insertion

Multiple Comparisons and Hypothesis Testing

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['test', 'comparison'], calculate_columns=['statistic', 'p_value', 'significant'], metric='mae')


Question:
Based on multi-source medical health datasets, evaluate the clinical efficacy of six disease prediction models and generate a comprehensive analysis report (output.png and output1.json).

Six prediction tasks:
1. Diabetes prediction
2. Stroke prediction
3. Heart disease prediction (binary: presence vs. absence of disease)
4. Sleep disorder prediction (binary: no disorder vs. has disorder)
5. Cancer type classification (multi-class, gene expression data preprocessed with StandardScaler and PCA reduction to 50 components, random_state=42)
6. Insurance high-cost prediction (binary: above vs. below median charges)

Handle missing values in numerical columns with median imputation.

Models:
- Tasks 1–5: RandomForestClassifier(n_estimators=100, random_state=42)
- Task 6: LogisticRegression(penalty='l1', solver='liblinear', random_state=42, max_iter=1000)
- Train/test split: 80/20 (random_state=42, stratify=y)

Evaluation metrics:
- Binary models: AUC-ROC and Brier score
- Multi-class cancer model: Macro F1 (as AUC substitute) and average per-class Brier score
- Clinical utility score = 0.7 × AUC + 0.3 × (1 - Brier)

Output:
- output1.json: mapping from model name (diabetes, stroke, heart_disease, sleep_disorder, cancer, insurance) to clinical_utility score, rounded to 4 decimal places
- output.png: bar chart comparing clinical utility scores of all six models. Use distinct bar colors (red, blue, green, purple, orange, yellow) for the six models respectively. Set the title to 'Comparison of Clinical Utility Scores of Six Disease Prediction Models', add axis labels, display numerical labels on each bar, and annotate the best-performing model information on the chart

Feature Selection and Dimensionality Reduction

compare_json(output_file_name='output1.json', gold_file_name='result1.json',thresholds={'diabetes': 0.09,'stroke': 0.09,'heart_disease': 0.09,'sleep_disorder': 0.09,'cancer': 0.09,'insurance': 0.09})


Question:
Based on a multi-source medical health dataset covering diabetes, sleep_health, stroke, heart_disease, insurance, and chronic_kidney_disease (5 CSV files and 1 ARFF file), evaluate the reliability of chronic disease prediction models and generate a comprehensive analysis report (output.json and output.csv).

Load all 6 datasets into an SQLite in-memory database. Perform feature engineering for each dataset:
- Diabetes: metabolic_risk_score = (Glucose/200) * 0.4 + (BMI/40) * 0.3 + (Age/80) * 0.3
- Sleep: Parse Blood Pressure into systolic/diastolic; sleep_quality_score = Sleep Duration * Quality of Sleep / 10; binary target: whether Sleep Disorder exists
- Stroke: Create hypertension * heart_disease interaction; metabolic_risk_score = (avg_glucose_level/200) * 0.5 + (bmi/40) * 0.5
- Heart disease: cardiac_stress_index = trestbps * oldpeak / 100; binary target from num > 0
- Insurance: Create smoker * BMI interaction; health_risk_score = (bmi/40) * 0.4 + (age/70) * 0.3 + smoker_encoded * 0.3; binary target: charges > median
- Kidney disease: kidney_function_score = (hemo/20) * 0.4 + (1 - sc/10) * 0.3 + (1 - bu/100) * 0.3

Handle missing values using median for numeric columns and mode for categorical columns. Use scikit-learn Pipeline with StandardScaler and RandomForestClassifier (n_estimators=100, random_state=42). Train-test split: 80/20 with random_state=42, stratified. Evaluate each model using ROC-AUC, Brier score, accuracy, and calibrated_auc = roc_auc * (1 - brier_score / 2), rounded to 4 decimal places.

Save output.csv with columns (dataset, roc_auc, brier_score, accuracy, calibrated_auc) and output.json with the following structure: dataset_performance is a dict keyed by dataset name, each value is a dict with keys roc_auc, brier_score, accuracy, calibrated_auc (all floats); best_dataset is a string indicating the dataset with the highest calibrated_auc; best_calibrated_auc is a float of that highest value.

Data Transformation and Feature Engineering

Arithmetic Transformations and Normalization

Numerical Operations and Type Conversion

Custom Functions for Missing Value Handling

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={"best_dataset": None, "best_calibrated_auc": None})

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['dataset'], calculate_columns=['roc_auc', 'brier_score', 'accuracy', 'calibrated_auc'], metric='mae')


Question: A national health research institute is investigating whether chronic disease comorbidities consistently precede inflammatory and coagulation laboratory abnormalities in thrombosis-risk patients. The study involves integrating comorbidity indicators from multiple chronic disease datasets to construct unified patient-level profiles. The workflow requires loading all relevant datasets into a SQLite database for efficient querying and management. Subsequently, apply a rolling window pattern detection methodology to identify temporal sequences where specific combinations of comorbidities occur within a 90-day window preceding laboratory anomaly clusters. These anomalies are defined as clusters where at least three biomarkers exceed two standard deviations from the cohort baseline. The final deliverable should report the most frequently observed alignment pattern as a structured result containing a unique pattern identifier, the specific combination of active comorbidities, the median number of days between comorbidity onset and anomaly detection, the median duration of the anomaly cluster, and the count of patients exhibiting this precise temporal alignment.

Data Conversion and Post-Loading Processing

Data Structure Creation and Manipulation

Time Difference and Gradient Calculation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', specified_columns=['days_before_anomaly', 'anomaly_duration_days', 'support_count'])


Question: The National Institute of Health is building a unified patient risk profile for predicting susceptibility to multiple system diseases. It needs to integrate metabolic, cardiovascular, lifestyle, and insurance data. Based on multiple datasets including data related to diabetes, sleep health, stroke, heart disease, and insurance, a single coordinated dataset is constructed. Each row represents a pseudo-patient and contains standardized features: (1) Parse the blood pressure data into columns of systolic and diastolic pressure values; (2) Map the smoking status to a unified category ('current-smoker', 'ex-smoker', 'non-smoker'); (3) Align BMI as a continuous variable and convert BMI Category to ordinal levels; (4) Ensure all age and gender fields are formatted consistently for cross-dataset matching. The final output must be a wide-format pandas DataFrame stored in an SQLite database, with all categorical variables factored, and missing cross-source matches retained as NaN.

Validation and Verification of Merge Results

Data Structure Handling (Dictionaries, Lists)

Categorical Data Preprocessing and Simplification

Numerical Operations and Type Conversion

Data Structure Creation and Manipulation

compare_sqlite(output_file_name='output.db', gold_file_name='result.db', specified_schema={'unified_patient_data': None}, ignore_order=False)


Question: The Multi-Institutional Oncology Research Consortium aims to develop a cancer type classification model using genomic profiles from the Cancer Genome Atlas (TCGA). Using gene expression data and cancer type labels from TCGA, train a machine learning model to classify five major types of cancer (BRCA, KIRC, LUAD, PRAD, COAD). Integrate population-level metabolic health indicators computed as aggregate statistics from non-cancer medical datasets as additional features. Use the first 500 gene expression features combined with these clinical aggregate features. Use a RandomForestClassifier (n_estimators=100, random_state=42) with 5-fold cross-validation (KFold, shuffle=True, random_state=42) to evaluate model performance. Use cross_val_predict to compute per-cancer-type accuracy and produce a bar chart of classification accuracy for each cancer type. The output should include output.csv with columns for cancer type and corresponding accuracy, and output.png containing the bar chart.

Data Conversion and Post-Loading Processing

Column-wise Transformations and Aggregation

Column-specific or Feature-wise Processing

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['cancer_type', 'accuracy'], thresholds={1: 0.02})


Question: The National Institute of Health is developing a unified cardiovascular disease risk prediction model, which requires integrating clinical characteristic data from different patient groups. Based on multiple datasets containing data related to diabetes, sleep health, stroke, heart disease, and insurance, a standardized unified dataset needs to be created. The requirements are as follows: (1) Normalize all shared numerical features (age, glucose, BMI, blood pressure components) in all datasets to the range of [0,1]; (2) Uniformly encode categorical variables such as gender, smoking status, and disease outcome; (3) Split blood pressure data into two numerical columns: systolic blood pressure and diastolic blood pressure; (4) Keep the one-hot encoded indicators of the source dataset in each row; (5) The final output should exactly contain 17 columns: age, age_bin_encoded, gender_encoded, glucose, bmi, bp_systolic, bp_diastolic, chol, htn_flag, smoker_flag, diabetes_flag, outcome_encoded, and 5 dataset source indicators. The output must include all original records, without any row filtering except for standardization.

Numerical Operations and Type Conversion

Preprocessing and File Structure Adjustments

Column-specific or Feature-wise Processing

Data Transformation and Column Manipulation

Data Structure Creation and Manipulation

Arithmetic Transformations and Normalization

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['age', 'age_bin_encoded', 'gender_encoded', 'glucose', 'bmi', 'bp_systolic', 'bp_diastolic', 'chol', 'htn_flag', 'smoker_flag', 'diabetes_flag', 'outcome_encoded'])


Question: The public health research team is developing a multi-condition risk stratification system to identify demographic hotspots where multiple chronic diseases and adverse health indicators co-occur. Based on multiple datasets containing data on diabetes, sleep health, chronic kidney disease, stroke, heart disease, insurance, and thrombosis predictions, a composite risk score is calculated for each demographic bin (age rounded to the nearest integer, BMI rounded to the nearest 0.5 unit). First, all data is loaded into the database and relevant records are extracted. A high-risk case is defined as: diabetes (Outcome=1), stroke (stroke=1), heart disease (num≥1), chronic kidney disease (class='ckd'), high insurance cost (charges above the 90th percentile), poor sleep quality (Quality of Sleep≤3 and Sleep Disorder not empty), and high-risk thrombosis (Thrombosis=1 and autoimmune diagnosis such as 'SLE', 'APS', or 'MCTD'). For thrombosis patients, the age at the time of the examination is calculated using the arithmetic of the date and time between the date of birth and the examination date. Then, for each (age ± 1, BMI ± 0.5) bin, binary indicators are assigned to all 7 conditions (if there is at least one high-risk case of this condition in the bin, it is 1, otherwise it is 0). Finally, a RandomForestClassifier with random_state=42 is fitted to verify feature importance, but only the aggregated demographic risk table is reported: a DataFrame containing columns ['age_bin', 'bmi_bin', 'diabetes_cases', 'stroke_cases', 'heart_disease_cases', 'kidney_disease_cases', 'insurance_risk_cases', 'sleep_risk_cases', 'thrombosis_cases'].

Data Conversion and Post-Loading Processing

Conditional Logic and Row-wise Operations

Time Difference and Gradient Calculation

Numerical Comparison and Proximity Checks

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['age_bin', 'bmi_bin', 'diabetes_cases', 'stroke_cases', 'heart_disease_cases', 'kidney_disease_cases', 'insurance_risk_cases', 'sleep_risk_cases', 'thrombosis_cases'])


Question: A national health research institute is developing a multi-morbidity risk stratification system to identify high-risk patients who exhibit clinical evidence of at least two of the following conditions: metabolic syndrome, cardiovascular disease, renal dysfunction, or thrombosis. Construct a unified patient-level dataset by assigning sequential patient identifiers (format: PAT-NNNNNN) across all data sources and joining them. Apply evidence-based clinical thresholds (Glucose ≥ 100 mg/dL for metabolic risk, serum creatinine ≥ 1.3 mg/dL for renal dysfunction, Thrombosis = 1 with aCL IgG ≥ 2.0 for thrombosis risk, and hypertension or heart disease for cardiovascular risk), and validate that each identified high-risk patient has supporting data from at least three independent clinical sources. The final output must be a DataFrame containing only patients who meet validated multi-system risk criteria, with columns indicating which risk categories were satisfied and a validation status flag.

Threshold-Based Categorization or Filtering

Data Transformation and Column Manipulation

Validation and Verification of Merge Results

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['patient_id', 'metabolic_risk', 'renal_risk', 'cardiovascular_risk', 'thrombosis_risk', 'sleep_risk', 'heart_risk', 'smoker_risk', 'risk_count', 'source_count', 'validation_status'])


Question: The medical data quality assessment team is developing a system for cross-dataset feature consistency analysis to evaluate the data quality and consistency among multiple medical data sources. Based on multiple datasets containing data on diabetes, sleep health, chronic kidney disease, stroke, heart disease, insurance, and thrombosis prediction, the following analyses are performed: (1) Calculate the descriptive statistics (mean, standard deviation, minimum value, maximum value, missing rate) for shared features (age, BMI, glucose) in each dataset; (2) Identify and report outliers in each dataset (using the IQR method: Q1 - 1.5*IQR and Q3 + 1.5*IQR as boundaries); (3) For datasets with overlapping features, calculate the Pearson correlation coefficient matrix between each pair of datasets (cross-dataset comparison, e.g., diabetes age vs stroke age); (4) Evaluate data integrity: Calculate the proportion of complete records (the percentage of rows without missing values) in each dataset; (5) Generate a data quality score: Calculate the comprehensive quality score for each dataset based on the missing rate (weight 0.4), outlier rate (weight 0.3), and feature consistency (weight 0.3). The output is a DataFrame containing columns ['dataset_name', 'n_records', 'n_features', 'completeness', 'outlier_rate', 'feature_consistency', 'quality_score'], sorted by quality score in descending order.

Data Conversion and Post-Loading Processing

Column-specific or Feature-wise Processing

Feature Selection and Statistical Computation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['n_records', 'n_features', 'completeness', 'outlier_rate', 'feature_consistency', 'quality_score'])


Question: Medical institutions are developing an early warning system for chronic diseases based on time series trend analysis. This system monitors the dynamic changes in patients' physiological indicators to identify high-risk patients. Based on multiple datasets including diabetes, sleep health, chronic kidney disease, stroke, heart disease, insurance, thrombosis prediction data, as well as related examinations and laboratory data, the following analyses are performed: (1) Generate a unique identifier for each patient (format: DATASET_ index); (2) For longitudinal datasets with repeated measurements (Examination and Laboratory data), sort by patient ID and timestamp, calculate the 7-day rolling average and standard deviation of each physiological indicator (glucose, BMI, blood pressure, etc.); (3) Calculate the trend slope: Use linear regression to fit the time series of each patient's recent 3 measurements, extract the slope as the trend indicator; (4) Define deterioration indicators: If the rolling average increases by more than 10% compared to the baseline (first measurement value) and the trend slope is positive, mark as deterioration (1), otherwise as stable (0); (5) For cross-sectional datasets where each patient has only one record, set trend_slope=0, rolling_mean=current_value, rolling_std=0; (6) Calculate risk score using weights: deterioration_flag * 0.4 + abs(trend_slope) * 0.3 + baseline_deviation_from_normal_range * 0.3; (7) Identify high-risk patients: Patients with a risk score exceeding the 75th percentile. The output is a DataFrame containing columns ['patient_id', 'dataset', 'baseline_value', 'current_value', 'trend_slope', 'rolling_mean', 'rolling_std', 'deterioration_flag', 'risk_score', 'high_risk_flag'], sorted by risk score in descending order.

Time Difference and Gradient Calculation

Normalization and Percentile Calculations

Incremental and Comparative Calculations

Threshold-Based Categorization or Filtering

Data Transformation and Column Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['patient_id', 'dataset', 'baseline_value', 'current_value', 'trend_slope', 'rolling_mean', 'rolling_std', 'deterioration_flag', 'risk_score', 'high_risk_flag'])


Question: The medical artificial intelligence research team is developing a multi-task learning framework to predict multiple health outcomes simultaneously. Based on multiple medical datasets including diabetes, sleep health, chronic kidney disease, stroke, heart disease, and insurance data, a multi-task machine learning model is constructed to predict the following health outcomes: (1) diabetes outcome; (2) stroke outcome; (3) heart disease outcome (binary coded as ≥1 is 1); (4) chronic kidney disease outcome (ckd is 1); (5) sleep quality outcome (<7 is marked as low quality and is 1); (6) insurance cost outcome (higher than the 75th percentile is marked as 1). For each dataset, shared features are extracted: age, BMI, glucose, and gender (encoded as 0/1). If the feature is missing, the median of that column is used to fill in. All datasets are merged into a unified DataFrame and a dataset source indicator column (one-hot encoding) is added. Then, RandomForestClassifier (n_estimators=100, random_state=42) is trained for each task separately with the shared features and dataset indicators as input and the corresponding health outcome as output. The average AUC-ROC score of each task is evaluated using 5-fold cross-validation. The output is a DataFrame containing columns ['task_name', 'auc_roc_score', 'feature_importances'], where feature_importances is a dictionary containing feature names and importance values.

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction

Column-specific or Feature-wise Processing

Data Transformation and Column Manipulation

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['task_name', 'auc_roc_score'], thresholds={1: 0.02})


Question: The public health research team is conducting a cross-cohort analysis to quantify the global impact of metabolic risk factors on various disease outcomes. Based on multiple datasets containing data on diabetes, sleep health, chronic kidney disease, stroke, heart disease, insurance, and other related medical data, the following analyses are performed: (1) Fit a Bayesian hierarchical logistic regression model to estimate the logarithmic odds ratio changes in the binary probabilities of six diseases for each standardized risk factor (age, BMI, blood pressure, glucose) when each standard deviation increases; (2) Use weak information priors and consider the specific intercepts of the data set; (3) Use RandomForestClassifier (n_estimators=50, max_depth=5, min_samples_split=10, min_samples_leaf=5, class_weight='balanced', random_state=42) and LogisticRegression (C=0.1, penalty='l2', max_iter=1000, class_weight='balanced') to validate the calibration of the stroke prediction model. The output is a DataFrame containing columns ['risk_factor', 'posterior_mean', 'ci_lower_2.5%', 'ci_upper_97.5%'], including posterior estimates of four global coefficients.

Data Structure Creation and Manipulation

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['posterior_mean', 'ci_lower_2.5%', 'ci_upper_97.5%'], thresholds={1: 0.02, 2: 0.02, 3: 0.02})


Question:
Create a 3x3 visualization dashboard using the provided medical health data and save it as 'healthcare_dashboard.png'.

**Task Requirements:**
Extract key indicators from the health data and create 9 subplots to display: disease prevalence distribution, medical cost analysis, gene expression patterns, lifestyle and health status correlation, risk factors and medical cost relationship.

**Technical Requirements:**
- Image size 18x15 inches, DPI = 150

**Output Requirements:**
- Generate healthcare_dashboard.png file
- Arrange the 9 subplots in a 3x3 grid, each subplot containing a title (bold), axis labels, and legend (if applicable)
- Subplot 1 (bar chart of diabetes prevalence by age group): Use age groups 20-29, 30-39, 40-49, 50-59, and 60+. Display percentage values (xx.x%) above the bars
- Subplot 2 (box plot of insurance charges by smoking status): Use scientific notation format on the Y-axis (1e+04)
- Subplot 3 (heat map of gene expression patterns by cancer type): Use seaborn.heatmap to draw and include a color bar
- Subplot 4 (scatter plot of sleep quality vs physical activity by BMI category): Use different colors for each BMI category
- Subplot 5 (violin plot of blood pressure distribution): Compare diabetes and heart-disease blood pressure measurements
- Subplot 6 (heat map of hypertension, heart disease, and stroke co-occurrence): Use seaborn.heatmap to draw. Include color bar, and display correlation coefficient (x.xx) in each cell
- Subplot 7 (scatter plot of age vs insurance charges): Use scientific notation format on the Y-axis (1e+04)
- Subplot 8 (scatter plot of glucose vs BMI by diabetes status): Include trend line and linear equation annotation (y=ax+b)
- Subplot 9 (scatter plot of metabolic risk vs insurance charges): Use scientific notation format on the Y-axis (1e+04)
- Add grid lines to all subplots (alpha = 0.3)

Data Loading with Pandas
Parsing and Reading Data Files
Directory and File Management
Data Cleaning and Transformation
Numerical Data Handling
Handling Missing Data
Data Preprocessing & Encoding
Encoding and Vector Representation
Data Preparation and Aggregation
Percentage and Variation Calculations
Statistical Calculations and Quantiles
Correlation Matrix Generation
Statistical Correlation Analysis
Array and Matrix Manipulation
Subplot and Layout Management
Plot Customization and Layout
Data Analysis and Visualization
Bar Chart Creation and Layout
Plot Customization (Aesthetics)
Color and Palette Usage
Multiple Series/Traces Visualization
Plot Customization and Annotation
Regression Modeling and Interpretation
Line Collection Customization
Plot Creation and Configuration


Question:
Conduct a comprehensive meta-analysis to evaluate the associations between key metabolic biomarkers (glucose levels, BMI, and blood pressure) and major chronic diseases across multiple healthcare datasets.
**Analysis Objectives:**
1. Calculate effect sizes (Cohen's d) for each biomarker-disease pair within individual datasets
2. Perform inverse-variance weighted meta-analysis to combine effect sizes across datasets; for single-study cases, compute the p-value from the z-score using a two-tailed normal distribution test
3. Apply False Discovery Rate (FDR) correction (Benjamini-Hochberg method) to account for multiple comparisons
4. Rank associations by clinical significance score: score = |meta_effect_size| × (-log10(fdr_corrected_p + 1e-10)), sorted descending
5. Classify confidence levels for each association using the following criteria:
   - high: fdr_corrected_p < 0.01 AND |meta_effect_size| > 0.5
   - medium: fdr_corrected_p < 0.05 AND |meta_effect_size| > 0.3
   - low: otherwise

Only include biomarker-disease pairs with direct continuous biomarker measurements; do not use binary indicators as substitutes.

**Input Data:**
Multiple healthcare datasets containing physiological measurements (glucose, BMI, blood pressure) and disease outcomes (diabetes, stroke, heart disease)

**Output Requirements:**
Generate a comprehensive results file 'meta_analysis_results.csv' containing the following columns:
- condition: The chronic disease condition analyzed
- biomarker: The physiological biomarker (Glucose, BMI, BloodPressure)
- meta_effect_size: The combined effect size from meta-analysis, rounded to 4 decimal places
- meta_p_value: The statistical significance of the association, rounded to 6 decimal places
- fdr_corrected_p: FDR-adjusted p-value for multiple comparisons, rounded to 6 decimal places
- confidence_level: Classification of confidence (high/medium/low) based on the criteria above
- clinical_significance_rank: Ranking from 1 (most significant) to N based on clinical significance score

**Expected Deliverables:**
- meta_analysis_results.csv: Complete meta-analysis results table
- Summary statistics showing total associations analyzed and confidence level distribution

Data Transformation and Column Manipulation

Feature Selection and Statistical Computation

Incremental and Comparative Calculations

Data Structure Creation and Manipulation

Multiple Comparisons and Hypothesis Testing

Interpreting and Communicating Statistical Results

compare_model(output_file_name='meta_analysis_results.csv', gold_file_name='result.csv', calculate_columns=['meta_effect_size', 'meta_p_value', 'fdr_corrected_p', 'clinical_significance_rank'], metric='mse', lower_bound=0, upper_bound=1)


Question:
Investigate multimorbidity patterns across diverse patient populations by analyzing the co-occurrence of chronic diseases and their relationships with lifestyle factors and healthcare expenditures. This study integrates multiple healthcare datasets to map disease prevalence across demographic strata and identify high-risk population segments.

**Analysis Workflow:**
1. Load and harmonize data from multiple chronic disease datasets (diabetes, heart disease, stroke, sleep disorders)
2. Standardize demographic variables (age, gender) and create consistent age decade groupings (30s-80s)
3. Calculate disease prevalence rates stratified by age decade and gender
4. Integrate lifestyle metrics (sleep duration, sleep quality) and healthcare cost data
5. Generate comprehensive multi-panel visualization showing disease burden and associated factors

**Visualization Requirements:**
- Upper panel: Heatmap (use `sns.heatmap()`) displaying prevalence percentages (with one decimal precision) for each condition across age-gender strata, using viridis color scale with colorbar annotation. The y-axis should be organized as grouped rows in the format “Condition–AgeDecade”
- Lower panel: Also use `sns.heatmap()` to display healthcare cost distribution (median charges) and sleep metrics (average sleep duration) by demographic groups. The y-axis row labels should follow "Metric-Sex" format
- All panels must include proper axis labels, titles (bold font), and grid lines for readability

**Specific Deliverables:**
- multimorbidity_analysis.png: Multi-panel heatmap visualization (14x10 inches, DPI 150)
- multimorbidity_analysis.csv: Complete aggregated data table with prevalence rates, costs, and sleep metrics
- stroke_prevalence_result.txt: Text file reporting the exact prevalence percentage (to one decimal place) of stroke in females aged 60-69

**Key Metric to Report:**
The precise prevalence percentage of stroke among females in their 60s, calculated from the aggregated stroke dataset and formatted as 'X.X%'

Data Transformation and Column Manipulation


Question:
Develop a hierarchical mixed-effects regression model to quantify how standardized metabolic risk factors (glucose, BMI, blood pressure) jointly influence the probability of major clinical outcomes across diverse patient populations. The model should account for baseline differences using shared metabolic coefficients and dataset-specific intercepts.

**Analysis Objectives:**
1. Integrate multiple healthcare datasets with harmonized metabolic variables
2. Apply appropriate data preprocessing: handle missing values, normalize features, align variables across sources
3. Fit a hierarchical model with shared fixed effects for metabolic variables and random intercepts per dataset
4. Estimate shared metabolic effect sizes and dataset-specific baseline risks
5. Generate predictive probabilities and model comparison metrics

**Input Data:**
Healthcare datasets containing physiological measurements (glucose, BMI, blood pressure) and binary disease outcomes from diverse patient populations

**Model Structure:**
- Shared metabolic coefficients (fixed effects) across all datasets
- Dataset-specific intercepts (random effects) to account for baseline differences

**Output Requirements:**
Generate a comprehensive results dictionary saved as 'bayesian_analysis_results.json' with the following exact structure:
1. posterior_effects: A dict mapping each metabolic variable name to its shared coefficient value (a single float). Keys must be 'glucose', 'bmi', 'bp'. Example: {"glucose": 0.07, "bmi": 0.05, "bp": 0.08}
2. dataset_intercepts: A dict mapping each dataset name to its random intercept value (a single float). Keys must be 'Diabetes', 'Stroke', 'Heart Disease'. Example: {"Diabetes": 0.19, "Stroke": -0.23, "Heart Disease": 0.04}
3. posterior_predictions: A dict mapping each dataset name to its mean predicted probability (a single float)
4. model_comparison: A dict with keys 'waic' and 'aic', each containing the corresponding model comparison metric value (float)
5. odds_ratios: A list of dicts, each containing keys 'biomarker', 'coefficient', 'odds_ratio', 'ci_lower', 'ci_upper', 'p_value'

**Additional Deliverables:**
- bayesian_summary.csv: Human-readable summary table of key findings
- bayesian_analysis_plot.png: A figure with 2 subplots side by side (1 row, 2 columns, figsize=(14, 5)):
  - Left subplot: Forest plot of odds ratios for each metabolic variable (glucose, bmi, bp). Plot the odds ratio as a point with 95% CI as horizontal error bars. Add a vertical reference line at odds_ratio=1. X-axis label: 'Odds Ratio', y-axis tick labels: variable names.
  - Right subplot: Bar chart of dataset-specific random intercepts. Each bar represents one dataset (Diabetes, Heart Disease, Stroke). Y-axis label: 'Random Intercept'.

Categorical vs Numerical Type Identification

compare_json('bayesian_analysis_results.json', 'bayesian_analysis_results_gold.json', thresholds={'posterior_effects': {'glucose': None, 'bmi': None, 'bp': None}, 'dataset_intercepts': {'Diabetes': None, 'Stroke': None, 'Heart Disease': None}, 'model_comparison': {'waic': None, 'aic': None}})


Question:
Investigate accelerating metabolic deterioration patterns across chronic disease cohorts to identify high-risk age windows for early intervention. This analysis tracks how key metabolic indicators change with age and detects periods of rapid deterioration that may warrant clinical attention.

**Analysis Workflow:**
1. Load and integrate physiological data from multiple chronic disease datasets and a general population cohort
2. Harmonize metabolic measurements (glucose, BMI, systolic blood pressure) across sources
3. Aggregate into 5-year age intervals and calculate first-order and second-order differences
4. Identify age windows with significant acceleration exceeding clinical thresholds
5. Generate visualization comparing metabolic trajectories across cohorts
- Second-order difference analysis to detect acceleration in metabolic deterioration
- Threshold criteria: absolute value of second-order difference exceeding glucose >5 mg/dL, BMI >1 kg/m², blood pressure >5 mmHg
- Compare disease cohorts against general population trends

**Output Requirements:**
Generate 'acceleration_analysis.csv' containing only accelerating age windows (is_accelerating=True) with columns:
- dataset_source: Cohort identifier (Diabetes, Stroke, Heart Disease, General Population)
- biomarker_type: Metabolic indicator (glucose, bmi, systolic_bp)
- age_bin_start, age_bin_end: 5-year age interval
- first_diff: Rate of change between consecutive age bins
- second_diff: Acceleration (change in rate of change)
- is_accelerating: Significant acceleration flag based on abs(second_diff) > threshold (True for all rows)

**Additional Deliverables:**
- metabolic_trends.png: A 3-panel (3 rows × 1 column) line chart visualization with figsize=(14, 12). Panel 1: Glucose Levels (mg/dL) showing Diabetes and Stroke cohorts. Panel 2: BMI (kg/m²) showing Diabetes, Stroke, and General Population cohorts. Panel 3: Systolic Blood Pressure (mmHg) showing Diabetes, Stroke, and Heart Disease cohorts. Each panel overlays the relevant cohort trend lines using different colors.
- Summary statistics: Total accelerating windows, breakdown by cohort and biomarker

Data Transformation and Column Manipulation

Column-wise Transformations and Aggregation

Threshold-Based Categorization or Filtering

Conditional Logic and Row-wise Operations

compare_model(output_file_name='acceleration_analysis.csv', gold_file_name='result.csv', calculate_columns=['first_diff', 'second_diff'], matched_columns=['dataset_source', 'biomarker_type', 'age_bin_start'], metric='mse', lower_bound=0, upper_bound=1)


Question: Identify the strongest clinical and lifestyle predictors of chronic disease burden across multiple patient populations through the following structured workflow: 1) Load and harmonize data from four clinical datasets covering diabetes, stroke, heart disease, and sleep disorders. For diabetes records, treat zero values in Glucose, BMI, and BloodPressure as missing before harmonization; 2) Create a unified analytical dataset by standardizing key biomarkers into common columns named 'age', 'glucose', 'bmi', and 'bp' (for the stroke dataset, use the 'hypertension' column as the 'bp' proxy), while defining a binary chronic condition indicator for each patient. After harmonization, drop records missing any of the four base biomarker columns before combining sources. Sample the stroke dataset after this filtering to at most 1000 records (random_state=42) to balance dataset sizes; 3) Engineer the following additional predictive features from the base biomarkers: 'glucose_bmi_ratio' computed as glucose/(bmi+1), 'age_glucose_interaction' computed as age*glucose/1000, and 'bp_bmi_ratio' computed as bp/(bmi+1). Also encode the data source as a numeric feature 'source_encoded' using label encoding. Fill any remaining missing values in the modeling feature matrix with column medians; 4) Apply recursive feature elimination with a random forest classifier (n_estimators=100, random_state=42, max_depth=10) to select the top 5 most predictive clinical features from the expanded feature set; 5) Perform principal component analysis (random_state=42) to reduce the dimensionality of the feature space and create 3 latent components named 'pca_component_1', 'pca_component_2', 'pca_component_3' that capture the majority of variance in the data; 6) Combine the selected features with principal components to create a hybrid feature representation; 7) Train a logistic regression model (max_iter=1000, random_state=42, C=1.0) on the standardized (using StandardScaler) hybrid features to predict chronic condition status; and 8) Extract and rank the coefficients of the final hybrid feature set by absolute magnitude, where the final feature set consists of the 5 RFE-selected features plus 3 PCA components; save the coefficient table to 'feature_coefficients.csv', save the selected feature list to 'selected_features.csv', create `feature_analysis.png` as a 1x2 figure showing the hybrid feature coefficients and PCA explained variance, and report the coefficient of the most important predictor as the final answer in 'top_coefficient.txt'.

Data Structure Handling (Dictionaries, Lists)

Data Transformation and Feature Engineering

Feature Selection and Dimensionality Reduction

compare_csv('feature_coefficients.csv', 'result_coefficients.csv', ignore_order=True, thresholds={1: 0.02})

compare_image('feature_analysis.png', 'result_plot.png', calculate_columns=['type', 'ytick_labels', 'xtick_labels'])


Question: A cancer research alliance is preparing a paper on the reliability of progression-free survival (PFI) as the survival endpoint for the TCGA cohort. Generate a 3×2 multi-panel chart that demonstrates how violation of the proportional hazards (PH) assumption affects Kaplan-Meier estimates and whether the competing risk model yields substantially different conclusions: (1) The top row shows, for 2 cancer types (excluding PanCancer) with PH test p-value < 0.05, a comparison of standard PFI Kaplan-Meier curves and competing risk PFI curves; (2) The middle row shows the same comparison for 2 cancer types (excluding PanCancer) with PH test FDR-corrected p-value ≥ 0.05; (3) The bottom-left panel is mapped to the source site through patient identifiers, comparing the institutional survival differences of breast cancer patients (using only institutions with ≥20 contributing cases), and the bottom-right panel shows the standard PFI vs competing risk PFI comparison for breast cancer patients. Each comparison panel in the top and middle rows should superimpose both curves with 95% confidence bands and include an embedded table showing PH test statistics. Also report the difference in median PFI time between the standard and competing risk models for the cancer type in the top-left panel, rounded to the nearest integer day. Save the chart as pfi_reliability_figure.png. Save the median difference result to output.txt in the format: 'Median PFI.time difference for {cancer_type}: {value} days'.

Numerical Operations and Type Conversion

Categorical vs Numerical Type Identification

Time Difference and Gradient Calculation

Incremental and Comparative Calculations


Question:
Based on all log files in the log directory, find the model configuration (hyperparameters + scene info) with the highest test_auc across all training experiments. Output to output.json with format:
{"best_log_dir": "<directory_name>", "best_scene_id": <int>, "best_test_auc": <float>, "best_params": {"num_leaves": <int>, "max_depth": <int>, "learning_rate": <float>, "num_iterations": <int>, "feature_fraction": <float>}}

Path Construction and Manipulation
File Existence and Access Verification
Command-Line and Shell Operations
File and Log Data Handling
Data Storage and Structuring
Directory Traversal and File Listing
Indexing and ID Assignment
Experiment Tracking & Version Control
Model Evaluation & Tuning
Model Evaluation & Generalization
Hyperparameter Tuning Strategies
Output and Logging
Validation and Output Formatting
Data Serialization & File Handling
Model Saving & Versioning

compare_json_normalized(output_file_name='output.json', thresholds={"best_test_auc": [0.5, 1.0]})


Question:
Analyze the relationship between hyperparameters and model performance across all trials. Group trials by learning_rate buckets (<0.02, 0.02-0.05, 0.05-0.1, >=0.1) and max_depth groups (3-4, 5-6, 7-8). Calculate the average test_auc for each combination. Output to output.json with format:
{"total_trials": <int>, "sensitivity_matrix": {"lr_<0.02": {"depth_3-4": <float>, "depth_5-6": <float>, "depth_7-8": <float>}, "lr_0.02-0.05": {"depth_3-4": <float>, "depth_5-6": <float>, "depth_7-8": <float>}, "lr_0.05-0.1": {"depth_3-4": <float>, "depth_5-6": <float>, "depth_7-8": <float>}, "lr_>=0.1": {"depth_3-4": <float>, "depth_5-6": <float>, "depth_7-8": <float>}}, "best_combination": {"learning_rate_bucket": "<string>", "max_depth_group": "<string>", "avg_test_auc": <float>}}

Data Structure Understanding and Initialization

compare_json_normalized(output_file_name='output.json', thresholds={"total_trials": [944, 944], "best_combination": {"avg_test_auc": [0.5, 1.0]}})


Question:
Analyze the return value structure differences across methods in the ModelEvaluator class. Output to output.json with format:
{"all_methods": ["<method_name1>", "<method_name2>", ...], "dataframe_methods": ["<method_name1>", ...], "dataframe_columns": {"<method_name1>": ["<column_name1>", "<column_name2>", ...], ...}, "common_columns": ["<column_name1>", ...], "dict_methods": ["<method_name1>", ...]}

Data Structure Handling (Dictionaries, Lists)

Data Structure Creation and Manipulation

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={"all_methods": None, "dataframe_methods": None, "dataframe_columns": None, "common_columns": None, "dict_methods": None})


Question: Using features.csv and label.csv, compute the correlation matrix for all features and remove highly correlated features (correlation coefficient > 0.8), keeping only one feature from each correlated group. Train an XGBoost model on the remaining features. Split data by time field (first 70% train, remaining 30% test). Output to output.json with format: {"remaining_feature_count": <int>, "train_auc": <float>, "test_auc": <float>}

Column-specific or Feature-wise Processing

Feature Selection and Dimensionality Reduction

compare_json_normalized(output_file_name='output.json', thresholds={"train_auc": [0.5, 1.0], "test_auc": [0.5, 1.0]})


Question:
Compare and analyze the design differences between two enhancement classes in the carisk.enhance module. Output to output.json with format:
{"class_names": ["<class_name1>", "<class_name2>"], "enhancer_params": ["<param_name1>", ...], "finetune_enhancer_params": ["<param_name1>", ...], "shared_params": ["<param_name1>", ...], "enhancer_only_params": ["<param_name1>", ...], "finetune_only_params": ["<param_name1>", ...], "data_required_for_enhancer": ["<data_name1>", ...], "data_required_for_finetune": ["<data_name1>", ...], "strategy_recommendation": {"has_pretrained_model": "<recommended_class>", "no_pretrained_model": "<recommended_class>"}}

Data Structure Handling (Dictionaries, Lists)

Data Structure Creation and Manipulation

compare_json(output_file_name='output.json', gold_file_name='result.json', thresholds={"class_names": None, "enhancer_params": None, "finetune_enhancer_params": None, "shared_params": None, "enhancer_only_params": None, "finetune_only_params": None, "data_required_for_enhancer": None, "data_required_for_finetune": None, "strategy_recommendation": {"has_pretrained_model": None, "no_pretrained_model": None}})


Question: Using features.csv and label.csv, apply equal-frequency binning with exactly 10 bins to all features starting with 'f', then calculate Weight of Evidence (WOE) for each bin. Train a Logistic Regression model using the WOE-encoded features. Split data by time field (first 70% train, remaining 30% test). Output to output.json with format: {"bin_count": <int>, "train_auc": <float>, "test_auc": <float>, "train_ks": <float>, "test_ks": <float>}

Column-specific or Feature-wise Processing

Vectorization and Performance Optimization

Data Structure Handling (Dictionaries, Lists)

compare_json_normalized(output_file_name='output.json', thresholds={"bin_count": [10, 10], "train_auc": [0.5, 1.0], "test_auc": [0.5, 1.0], "train_ks": [0.1, 1.0], "test_ks": [0.1, 1.0]})


Question: Using features.csv and label.csv, first train an XGBoost model with all features, then identify the top 15 most important features. Retrain a new LGBM model using exactly these 15 features. Split data by time field (first 70% train, remaining 30% test). Compare the test AUC between the full model and the simplified model. Output to output.json with format: {"full_model_test_auc": <float>, "simplified_model_test_auc": <float>, "selected_feature_count": <int>}

Column-specific or Feature-wise Processing

Feature Selection and Dimensionality Reduction

Data Preprocessing and Column Management

compare_json_normalized(output_file_name='output.json', thresholds={"full_model_test_auc": [0.5, 1.0], "simplified_model_test_auc": [0.5, 1.0], "selected_feature_count": [15, 15]})


Question: Using features.csv and label.csv, train an XGBoost model and compute SHAP values for all features. Select exactly the top 20 features with highest mean absolute SHAP values. Retrain a new model using only these 20 features and compare performance. Split data by time field (first 70% train, remaining 30% test). Output to output.json with format: {"selected_feature_count": <int>, "original_test_auc": <float>, "refined_test_auc": <float>}

Column-specific or Feature-wise Processing

Feature Selection and Statistical Computation

Feature Selection and Dimensionality Reduction

Data Preprocessing and Column Management

compare_json_normalized(output_file_name='output.json', thresholds={"selected_feature_count": [20, 20], "original_test_auc": [0.5, 1.0], "refined_test_auc": [0.5, 1.0]})


Question: Using features.csv and label.csv, first analyze the missing rate of each feature and remove features with missing rate above 30%. For remaining features, fill missing values with median. Apply SMOTE oversampling on the training set to balance the classes to approximately 50% positive ratio, then train an LGBM model. Split data by time field (first 70% train, remaining 30% test). Output to output.json with format: {"train_positive_ratio_after_smote": <float>, "test_ks": <float>}

Column-specific or Feature-wise Processing

compare_json_normalized(output_file_name='output.json', thresholds={"train_positive_ratio_after_smote": [0.5, 0.5], "test_ks": [0.0, 1.0]})


Question:
Given a risk control dataset containing model scores to be evaluated and true labels (including fields: `apply_date` in date format, `scene_code` as scene identifier, `score` as the model score to be evaluated, `label` as the true label 0 or 1). Please use Python to calculate and output a performance summary report, saved as 'output.csv'. The report should include the following two parts:

1. Overall performance by scene: Calculate the overall AUC and KS metrics for each `scene_code` (at this time, the `month` field should be filled with 'All').
2. Monthly performance by scene: Extract the month from `apply_date` (format YYYY-MM), and calculate the monthly AUC and KS metrics for each `scene_code`.

The output table schema must strictly contain the following 4 columns: `scene_code`, `month`, `auc`, `ks`. Please round the `auc` and `ks` values to 4 decimal places.

Mathematical and Statistical Computations

Batch Processing and Performance Optimization

compare_csv(output_file_name='output_1.csv', gold_file_name='result_1.csv', ignore_order=True, specified_columns=['scene_code', 'month', 'auc', 'ks'], thresholds={3: 0.02, 4: 0.02})

compare_csv(output_file_name='output_2.csv', gold_file_name='result_2.csv', ignore_order=True, specified_columns=['scene_code', 'month', 'auc', 'ks'], thresholds={3: 0.02, 4: 0.02})


Question: Given `input.csv` with `column_1`, `column_2`, and `label`, create a crossed feature `column_1_2_mix` using deterministic monotonic aggregation binning. Treat missing values in either feature as an explicit categorical value `MISSING`. Build crossed cells from `(column_1, column_2)`, compute each cell's positive sample rate, sort cells by positive sample rate ascending, and break ties by stringified `column_1` then stringified `column_2`. Greedily merge adjacent cells so each closed bin contains at least 5% of all rows; if the final remaining bin is below 5%, merge it into the previous bin. Merge adjacent bins as needed so the final bin positive rates are strictly increasing and the number of bins is between 2 and 10. Assign integer bin IDs from 0 upward in increasing positive-rate order. Save `output.csv` with `column_1`, `column_2`, `column_1_2_mix`, `label`, preserving the original input row order.

Column-wise Transformations and Aggregation

Data Structure Handling (Dictionaries, Lists)

Threshold-Based Categorization or Filtering

Custom Functions for Missing Value Handling

Data Transformation and Column Manipulation

compare_model(output_file_name='output.csv', gold_file_name='result.csv', calculate_columns=['column_1_2_mix'], metric='mae', lower_bound=0, upper_bound=1e-6)

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['column_1', 'column_2', 'column_1_2_mix', 'label'])


Question:
Given a CSV file containing two columns of data (a feature column and a label column). Please use the feature column to perform binning and feature mining with the following specific requirements:

1. Find low-risk customer segments: Look for a low-risk rule segment in the feature (such as `feature value <= some threshold` or an interval).
2. Constraints: The risk lift of this customer segment must be strictly below 0.6, and its coverage rate (i.e., number of samples in this segment / total number of samples) must be greater than 2%. If there are multiple segments meeting the conditions, please select the one with the largest coverage rate.
3. Output data: Tag the original data based on the found segment, generate a new column `column_1_pos` (if meeting this low-risk rule, group into one bin/mark as 1, otherwise mark as 0). Finally save the data as `output.csv`, the schema must strictly and only contain three columns: `column_1`, `column_1_pos`, `label`.

Mathematical and Statistical Computations

Conditional Logic and Row-wise Operations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['column_1', 'column_1_pos', 'label'])


Question:
Given a detailed dataset of customer customized model invocation records. Calculate the top 2 tenants by daily invocation count and save a wide summary table to `output.csv` with columns: `apply_date`, `tenant_top1`, `tenant_top1_cnt`, `tenant_top1_match_rate`, `tenant_top2`, `tenant_top2_cnt`, `tenant_top2_match_rate`.

Use the date part of `gmt_occur` as `apply_date` in `YYYY-MM-DD` format. For each day, `match_rate = tenant_cnt / total_invocations_that_day`, rounded to 4 decimal places. Rank tenants by invocation count descending and break ties by tenant name ascending. Sort output rows by `apply_date` ascending.

Date and Time Conversion
Data Loading with Pandas
Data Aggregation and Grouping
Time Series & Temporal Grouping
Statistical Analysis and Metrics
Percentage and Variation Calculations
Data Filtering and Matching
Ranking and Top N Logic
Sorting, Limiting, and Ranking
Formatting and Output Organization
Indexing and Row-Level Operations
Data Export and Output Processing
Data Storage and Structuring

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['apply_date', 'tenant_top1', 'tenant_top1_cnt', 'tenant_top1_match_rate', 'tenant_top2', 'tenant_top2_cnt', 'tenant_top2_match_rate'])


Question:
Given a desensitized customer customized model invocation record dataset (including: invocation time, desensitized phone number identifier, product name, tenant name, model score). To reduce server pressure, the system adopts a caching strategy for the same phone number: when a phone number makes an invocation, if no cache is available or the cache has expired, actual calculation is triggered and a new cache is generated (record this time as the cache start time); if an invocation occurs within the valid time window of that cache, it is considered a cache hit (and the start time of this cache is not updated).

Please simulate the above caching logic, traverse the records in chronological order, and calculate the cache usage rate when the cache validity period is set to 1 hour, 3 hours, and 6 hours respectively.

Please save the calculation results of the three scenarios as `output.csv`. The output CSV file should have only one valid row of data, the schema must strictly contain and only contain three columns: `cache_1h_rate`, `cache_3h_rate`, `cache_6h_rate`. All ratio values should be rounded to 4 decimal places.

Timestamp Conversion and Time Manipulation

Custom Function Development for Time Logic

Incremental and Comparative Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['cache_1h_rate', 'cache_3h_rate', 'cache_6h_rate'])


Question:
Given a risk control modeling dataset (including date column, feature columns, and label column). Please use Python and LightGBM to build a risk control model with the following specific requirements:

1. Sample split: Use data from '20250501' to '20250731' as the training set, and data from '20250801' to '20250831' as the test set.
2. Features and label: Use feature variable sets `aft_v4_f0`~`aft_v4_f44` and `pac_v2_x0`~`pac_v2_x27` for training. The target label is `label_6pd_30`.
3. Model configuration: Use LightGBM algorithm for modeling, requiring the maximum depth of decision trees (max_depth) to not exceed 5.
4. Model performance and business constraints:
   - The KS value on the test set must not be lower than 0.1.
   - The customized score must maintain trend consistency with the feature `als_m1_id_nbank_orgnum` (multi-head count): perform 10 equal-frequency binning on the model scores of the test set, and the mean of `als_m1_id_nbank_orgnum` for the bins with higher scores (i.e., higher risk groups) should also show strict monotonically increasing trend.

Finally, please use the trained model to score the test set, and save the test set scoring results to `output.csv`. The output CSV file must contain and only contain two columns: `example_id` (assuming this primary key exists in the dataset or keep the original index/ID column of the test set) and `score` (model prediction score).

Column-specific or Feature-wise Processing

compare_model(output_file_name='output.csv', gold_file_name='result.csv', matched_columns=['example_id'], calculate_columns=['score'], metric='ks', lower_bound=0.1, upper_bound=1.0)


Question: Given a dataset containing a date column `apply_date` and a model score column `score_0`. Please aggregate the data by month (format YYYY-MM) and calculate the following two metrics: 1. Monthly PSI of score_0: Use the samples from the latest month in the dataset as the base to calculate the PSI for each of the other months. 2. Adjacent month PSI of score_0: Use the samples from the previous month as the base to calculate the PSI for the current month (the first month is not calculated or set to empty). Finally, please draw a line chart based on the calculated data and save it as `output.png`. Chart requirements: the x-axis is time (month), the y-axis is PSI value, plot two lines in the same chart to represent the above two PSI metrics, and include a legend for differentiation.

Data Conversion and Post-Loading Processing

Batch Processing and Performance Optimization

Incremental and Comparative Calculations


Question:
Given a risk control dataset containing high-cardinality categorical features (fields include: high-cardinality categorical feature (province name), dataset split field, and true label). Please complete Chi-Merge binning and WOE encoding tasks with the following specific requirements:

1. Binning and encoding training: Only on the training set, perform Chi-Merge binning operation on the high-cardinality categorical feature, and calculate the WOE value for each bin. The constraints are as follows:
   - The number of bins (including the missing value bin) must not exceed 5.
   - The proportion of samples in each bin relative to the total samples in the training set must not be less than 5%.
   - The WOE encoding values of each bin after forced binning must show strict monotonic ordering (monotonically increasing or decreasing is acceptable).
2. Mapping application: Apply the WOE encoding rules obtained from the training set to the entire dataset containing all samples. Add a new column named `prov_name_woe` to the original table, saving the mapped WOE encoding values.

Finally, please save the processed complete dataset as `output.csv`. The output schema must strictly contain and only contain the following four columns: `prov_name`, `flag`, `label`, `prov_name_woe`. The values of `prov_name_woe` should be rounded to 4 decimal places.

Column-wise Transformations and Aggregation

Data Structure Handling (Dictionaries, Lists)

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['prov_name', 'flag', 'label', 'prov_name_woe'], thresholds={4: 0.02})


Question:
Input: Risk control modeling dataset `input.csv`, containing sample unique identifier `rowkey`, application date `apply_date`, scene identifier column `scene_code`, dataset split column `flag`, label column `label`, and modeling feature columns (`aft_v4_f0`~`aft_v4_f44` and `pac_v2_x0`~`pac_v2_x27`). Missing values in modeling features are filled with -1.

Output: Please process and output results according to the following flow:
1. Data preparation: Generate `weight` weight column, balance good and bad samples under each `scene_code`.
2. Model training: Use LGBM algorithm for modeling (require tree depth not to exceed 4), and use `AUC_mean` (scene average AUC) as the evaluation function `eval_metric` for model training with sample weights.
3. Sample scoring: Generate full sample scoring details excluding modeling feature information, and save the results as `output_1.csv`. The table must contain the following 6 specific column names: `rowkey`, `split_flag` (i.e., the original dataset split column flag), `score` (model prediction score), `apply_date`, `scene_code`, `label`.
4. Metric evaluation: Calculate model performance metrics under each dataset split, and save the evaluation results as `output_2.csv`. The table must contain the following 9 specific column names: `group` (dataset split identifier, e.g., 1.train), `period` (time span), `sample` (total sample size), `bad_sum` (bad sample count), `good_sum` (good sample count), `bad_rate` (bad sample proportion), `auc`, `ks`, `lift` (top 10% LIFT).

Mathematical and Statistical Computations

compare_csv(output_file_name='output_1.csv', gold_file_name='result_1.csv', ignore_order=True, thresholds={2: 0.02})

compare_csv(output_file_name='output_2.csv', gold_file_name='result_2.csv', ignore_order=True, thresholds={6: 0.02, 7: 0.02, 8: 0.02})


Question:
Given a tabular dataset containing model scores to be evaluated (including main fields: `apply_date` in date format, `scene_code` as scene identifier, and multiple score columns: `cro_general`, `cro_general_reg`, `cro_general_mle`, `cro_sceneid`). Please use Python to calculate and output stability metrics summary reports, saved in CSV format as 'output_1.csv' and 'output_2.csv'. Specific requirements are as follows:

1. Group by `scene_code` (scene) and the month extracted from `apply_date` (format YYYYMM).
2. For each score column, calculate the following stability metrics:
   - `psi`: Use the sample distribution of the latest month in each scene as the base, and calculate the PSI for each month using 2 buckets.
   - `missing_per`: Missing rate (number of missing values / total sample size, rounded to 4 decimal places).
   - `missing`: Number of missing values.
   - `unique`: Number of unique non-null values.
   - `bin_point_rate`: Distribution of values across buckets (10 buckets for output_1.csv, 2 buckets for output_2.csv).

3. Generate two output files:
   - output_1.csv: Baseline statistics aggregating all data for each scene and score column, with dt_col=197001.
   - output_2.csv: Monthly statistics for each scene, score column, and month.

The output table schema must strictly contain the following 10 columns: `scene_col`, `monitor_col`, `count`, `missing`, `missing_per`, `special_per`, `unique`, `psi`, `bin_point_rate`, `dt_col`.

Mathematical and Statistical Computations

Normalization and Percentile Calculations

Data Conversion and Post-Loading Processing

Batch Processing and Performance Optimization

Incremental and Comparative Calculations

compare_csv(output_file_name='output_1.csv', gold_file_name='result_1.csv', ignore_order=True, specified_columns=['scene_col', 'monitor_col', 'count', 'missing', 'missing_per', 'special_per', 'unique', 'psi', 'bin_point_rate', 'dt_col'], thresholds={7: 0.02})

compare_csv(output_file_name='output_2.csv', gold_file_name='result_2.csv', ignore_order=True, specified_columns=['scene_col', 'monitor_col', 'count', 'missing', 'missing_per', 'special_per', 'unique', 'psi', 'bin_point_rate', 'dt_col'], thresholds={7: 0.02})


Question:
Input: Dataset `input.csv` containing sample model prediction results and actual labels. The `'score'` column represents model prediction score (probability), `'raw_label_mob3'` column represents true good/bad label (0 or 1), and also contains `uniq_id` (primary key), `split_flag` (dataset split flag), `apply_date` (application date) and other fields.

Output: Please calculate the cross-entropy loss (i.e., logloss) for each sample to measure model prediction error. After calculation, save this error as a new column `sample_logloss`. Then, sort by `sample_logloss` in descending order, and filter out the **5%** of samples with the highest model prediction error (i.e., highest logloss). Output these filtered samples as a DataFrame and save as `output.csv`. The output table must contain the following 6 specific column names:
- `uniq_id`: Sample primary key
- `split_flag`: Dataset split flag (e.g., 1.train, 2.valid, 3.oot)
- `score`: Model prediction score
- `apply_date`: Application date
- `raw_label_mob3`: True label
- `sample_logloss`: Sample-level prediction error (logloss)

Entropy and Log Probability Calculations

Conditional Logic and Row-wise Operations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={2: 0.02, 5: 0.02})


Question:
Input: Table 1 `input_1.csv` is a table containing only primary key `id` and kv features `kv_features` (feature format like `{1:0.024591, 2:0.073246, 3:0.995855}`); Table 2 `input_2.csv` is a feature importance ranking table, containing fields `feature_name`, `gain`, `split_count`. Where `feature_name` has format `column_k` (`k` refers to the key corresponding to this feature in Table 1's `kv_features`, e.g., `column_6` corresponds to key 6). The input data guarantees that `kv_features` in Table 1 is a superset of features in Table 2.

Output: Please filter Table 1's `kv_features` based on the feature list in Table 2, only extract corresponding features that exist in Table 2, and combine the extracted results into a new kv feature field, named `filtered_kv_feature_str`. Save the final results as `output.csv`, the output table must contain the following 2 specific column names:
- `id`: Primary key id
- `filtered_kv_feature_str`: Filtered kv feature column

Column-specific or Feature-wise Processing

Data Structure Handling (Dictionaries, Lists)


Question:
Input: Table 1 `input_1.csv` is a table containing primary key `id` and KV features (feature format like `{1:0.024591, 2:0.073246}`); Table 2 `input_2.csv` is a feature mapping table, containing fields `feature_name` and `detail`. Where `feature_name` has format `column_k` (`k` refers to the key corresponding to this feature in Table 1's KV features, e.g., `column_6` corresponds to key 6 in Table 1), `detail` is the actual meaning name of the feature.

Output: Please replace all keys in Table 1's KV features with corresponding `detail` based on Table 2's mapping relationship, and save the results as `output.csv`. The output table must contain the following 2 specific column names:
- `id`: Primary key id
- `kv_features_with_names`: KV feature column after key replacement

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction


Question:
Input: A modeling dataset named `input.csv`, containing primary key `uniq_id`, date field `apply_date`, label `label_6pd_30` (assume 1 is bad sample, 0 is good sample), and feature columns `aft_v5_f0`~`aft_v5_f79` and `pac_v5_x0`~`pac_v5_x30`.

Output: Please process the data according to the following steps, and save the statistical results as `output.csv`:
1. Data split: Extract samples with `apply_date` in the period 20250501~20250731 for train and valid set split, require the train and valid sample ratio to be strictly controlled at 3:7, and use stratified sampling to ensure the proportion of `label_6pd_30`=1 in both datasets is as consistent as possible. Samples from other time periods are classified as oot set.
2. Equal-frequency binning: For the three prepared datasets (train, valid, oot), perform equal-frequency binning (5 bins) on all specified features.
3. Metric calculation and output schema: Calculate metrics for each feature in corresponding bins under each dataset, the output table must contain the following specific column names:
  - `data_type`: Dataset identifier (train, valid, oot)
  - `feature`: Feature name
  - `bins`: Bin range
  - `sample_size`: Total sample size in this bin
  - `good`: Good sample count in this bin
  - `bad`: Bad sample count in this bin
  - `bin_bad_rate`: Bad sample proportion in this bin
  - `total_bad_rate`: Overall bad sample proportion in dataset
  - `woe`: WOE value of this bin
  - `bin_iv`: IV value of this bin
  - `bin_ks`: KS value of this bin
  - `iv`: Total IV value of this feature in this dataset
  - `ksmax`: Maximum KS value of this feature in this dataset
  - `lift`: Lift value of this bin
  - `cum_lift`: Cumulative Lift value

Column-specific or Feature-wise Processing

Batch Processing and Performance Optimization

Mathematical and Statistical Computations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={8: 0.02, 9: 0.02, 10: 0.02, 11: 0.02, 12: 0.02, 13: 0.02, 14: 0.02})


Question: Please process the input sample information table `input.csv` (containing fields: rowkey, raw_bin, inc) with the following logic: First, discretize the original income (inc) field into bins according to interval boundaries `[0, 4000, 6000, 8000, 10000, 14000, 20000, 30000, np.inf]`, generate a new binning field `new_bin`; then compare the generated `new_bin` with the income binning result `raw_bin` from other sources in the original data, evaluate the consistency between the two. Finally, output and save the summary consistency metric results to the `output.csv` file. The output table must strictly contain the following 3 columns: `total` (total sample count), `consistent_count`, `consistent_rate`.

Row-wise and Column-wise Logical Evaluation


Question: Given two feature dataset files containing the same samples, `input_1.csv` and `input_2.csv`, align both files by `rowkey` ascending before comparing feature values. Compare every feature column except `rowkey`. Treat `-999` as missing (`NaN`). A sample has a deviation for a feature if exactly one side is missing, or if both values are present and `abs(val1 - val2) > 1e-6`. Exclude samples where both sides are missing from the denominator. For each feature, output `feature_name` and `diff_ratio`, rounded to 4 decimal places, to `output.csv`. Keep the original feature names, including the `_filled` suffix, and output rows in the same order as the input feature columns.

Column-specific or Feature-wise Processing

Numerical Comparison and Proximity Checks

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['feature_name', 'diff_ratio'])


Question: Read two feature scoring data tables `input_1.csv` and `input_2.csv`, both containing `id`, `uniq_id`, `score`, and `feature_json`. Join the two tables on `id`, compute `gap_score = round(abs(score_1 - score_2), 6)`, and keep only rows where `gap_score > 0.0001`. For the retained rows, parse `feature_json` and output `uniq_id_1`, `id_1`, `uniq_id_2`, `id_2`, `score_1`, `score_2`, `gap_score`; parsed feature values for `aft_v4_f0`, `aft_v4_f1`, `aft_v4_f2`, `aft_v4_f3`, `aft_v4_f4`, `aft_v4_f5`, `aft_v4_f6`, `aft_v4_f7`, `aft_v4_f8`, `aft_v4_f13`, `aft_v4_f14`, `aft_v4_f15`, `aft_v4_f16`, `aft_v4_f17`, `aft_v4_f18`, `aft_v4_f19` with suffixes `_1` and `_2`; and absolute gap columns named `gap_<feature>`. Sort output by `gap_score` descending, then `id_1` ascending, and save to `output.csv`.

Data Loading with Pandas
Parsing and Reading Data Files
Command-Line and Shell Operations
Joining and Lookup Operations
Join Operations and Merging
Data Filtering and Matching
Indexing and Selection
Difference and Trend Computation
In-place vs Copy Operations
Data Extraction from JSON
Feature Extraction and Vectorization
String Manipulation and Parsing
Function Application and Vectorization
Arithmetic and Cumulative Calculations
Formatting and Output Organization
Sorting, Limiting, and Ranking
Data Export and Output Processing
Data Serialization & File Handling

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, thresholds={'score_1': None, 'score_2': None, 'gap_score': None, 'aft_v4_f0_1': None, 'aft_v4_f1_1': None, 'aft_v4_f2_1': None, 'aft_v4_f3_1': None, 'aft_v4_f4_1': None, 'aft_v4_f5_1': None, 'aft_v4_f6_1': None, 'aft_v4_f7_1': None, 'aft_v4_f8_1': None, 'aft_v4_f13_1': None, 'aft_v4_f14_1': None, 'aft_v4_f15_1': None, 'aft_v4_f16_1': None, 'aft_v4_f17_1': None, 'aft_v4_f18_1': None, 'aft_v4_f19_1': None, 'aft_v4_f0_2': None, 'aft_v4_f1_2': None, 'aft_v4_f2_2': None, 'aft_v4_f3_2': None, 'aft_v4_f4_2': None, 'aft_v4_f5_2': None, 'aft_v4_f6_2': None, 'aft_v4_f7_2': None, 'aft_v4_f8_2': None, 'aft_v4_f13_2': None, 'aft_v4_f14_2': None, 'aft_v4_f15_2': None, 'aft_v4_f16_2': None, 'aft_v4_f17_2': None, 'aft_v4_f18_2': None, 'aft_v4_f19_2': None, 'gap_aft_v4_f0': None, 'gap_aft_v4_f1': None, 'gap_aft_v4_f2': None, 'gap_aft_v4_f3': None, 'gap_aft_v4_f4': None, 'gap_aft_v4_f5': None, 'gap_aft_v4_f6': None, 'gap_aft_v4_f7': None, 'gap_aft_v4_f8': None, 'gap_aft_v4_f13': None, 'gap_aft_v4_f14': None, 'gap_aft_v4_f15': None, 'gap_aft_v4_f16': None, 'gap_aft_v4_f17': None, 'gap_aft_v4_f18': None, 'gap_aft_v4_f19': None})


Question:
Read sample table data `input.csv` containing features, labels and weight column hy_weight. Where label=1 represents good samples (positive samples), label=0 represents bad samples (negative samples). Please group by month, biz1_encode, biz2_encode, biz3_encode fields, calculate the following metrics: 1. Sample count sample_cnt, positive sample count pos_sample_cnt, negative sample count neg_sample_cnt, overdue rate (negative sample count / sample count) overdue_rate; 2. For records with non-empty features, calculate: found sample count found_count, found positive sample count pos_found_count, found negative sample count neg_found_count, found overdue rate found_overdue_rate; 3. Calculate found rate metrics: sample count found rate found_rate, positive sample count found rate pos_found_rate, negative sample count found rate neg_found_rate.
Please save the final statistical report as output.csv.

Column-wise Transformations and Aggregation

Mathematical and Statistical Computations


Question: Read `input.csv` containing monthly labels, `hy_weight`, and features `afd_v3_subscore1` through `afd_v3_subscore9`. Use `label=1` as good samples and `label=0` as bad samples. For each feature, calculate IV, KS, and PSI by actual month only: `202405` through `202501`. Use `hy_weight` as the sample weight. Use the first month (`202405`) to create 10 equal-frequency reference bins for each feature, with missing values as an independent bin, and reuse those bins for all months. Order bins as missing first, then the reference score bins. Use `epsilon=1e-6` to smooth weighted good/bad distributions for IV and KS and weighted total distributions for PSI; the first month PSI is 0. Save `output.csv` with feature rows ordered `afd_v3_subscore1` through `afd_v3_subscore9`, and metric columns named like `202405_iv`, `202405_ks`, and `202405_psi`; do not output columns for months after `202501`.

Feature Selection and Statistical Computation

Statistical Testing for Feature-Target Evaluation

Incremental and Comparative Calculations


Question:
Read `input.csv` containing model scores, labels, restored weights, and business dimensions. The score column is `afd_v3_subscore1`, `label=1` represents positive/g samples, `label=0` represents negative/b samples, and restored weight is `hy_weight`. Generate a four-level hierarchical model quality report: Level 1 grouped by `month`, `biz1_encode`, `biz2_encode`, `biz3_encode`; Level 2 grouped by `month`, `biz1_encode`, `biz2_encode` with `biz3_encode='ALL'`; Level 3 grouped by `month`, `biz1_encode` with `biz2_encode='ALL'` and `biz3_encode='ALL'`; Level 4 grouped by `biz1_encode` only with `month='ALL'`, `biz2_encode='ALL'`, and `biz3_encode='ALL'`. Do not include `flag_col` and do not create month rollup cube rows beyond Level 4.

For each row, independently calculate sample counts, restored-weight counts using `hy_weight`, bad rates before and after restore, and KS before and after restore from the row's raw subset. Use `g_*` columns for `label=1` and `b_*` columns for `label=0`. Leave previous-version KS fields blank. Compute `Monthly_PSI` from unweighted score distributions relative to the earliest month for the same grouping; first-month PSI is 0, and Level 4 compares all-time data for the `biz1_encode` group against that group's earliest-month data. Save exactly these columns to `output.csv`: `biz1_encode`, `biz2_encode`, `biz3_encode`, `month`, `Summary_Level`, `Sample_Size`, `g_Sample_Size`, `b_Sample_Size`, `Sample_Size_Restored`, `g_Sample_Size_Restored`, `b_Sample_Size_Restored`, `Bad_Rate_Before_Restore`, `Bad_Rate_After_Restore`, `Ks_Before_Restore`, `Ks_After_Restore`, `Prev_Version_Ks_Before_Restore`, `Prev_Version_Ks_After_Restore`, `Monthly_PSI`. Sort by level order and then `month`, `biz1_encode`, `biz2_encode`, `biz3_encode`.

Mathematical and Statistical Computations

Conditional Logic and Row-wise Operations

Incremental and Comparative Calculations

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, thresholds={'Bad_Rate_Before_Restore': None, 'Bad_Rate_After_Restore': None, 'Ks_Before_Restore': None, 'Ks_After_Restore': None, 'Monthly_PSI': None})


Question:
Given a tabular dataset containing behavioral scores, including the following fields: `apply_date` (date, format YYYYMMDD), `score` (behavioral score, numeric) and `group_id` (group identifier). Please use Python to calculate and output a comprehensive score distribution statistical report. The report should include the calculation results of the following two parts:

1. Overall distribution by group: Calculate overall statistics for each group_id.
2. Monthly distribution by group: Extract month from apply_date (format YYYYMM), and calculate monthly statistics for each group_id.

For both overall and monthly distributions, the following 7 statistical metrics need to be calculated: count, max, min, mode (if there are multiple modes, take the minimum value), mean, median and standard deviation (std, use sample standard deviation, fill with 0 if sample size is insufficient to calculate standard deviation).

Output requirements:
- Save overall distribution to `output_1.csv` with columns: `group_id`, `cnt`, `max_score`, `min_score`, `mode_score`, `mean_score`, `median_score`, `std_score`
- Save monthly distribution to `output_2.csv` with columns: `group_id`, `apply_month`, `cnt`, `max_score`, `min_score`, `mode_score`, `mean_score`, `median_score`, `std_score`
- For overall distribution, the month field should be marked as 'overall' internally but not included in the final output
- Convert count column to float type

Mathematical and Statistical Computations

Statistical Calculations and Descriptive Statistics

Numerical Operations and Type Conversion

Column-wise Transformations and Aggregation

compare_csv(output_file_name='output_1.csv', gold_file_name='result_1.csv', ignore_order=True, specified_columns=['group_id', 'cnt', 'max_score', 'min_score', 'mode_score', 'mean_score', 'median_score', 'std_score'], thresholds={'max_score': None, 'min_score': None, 'mode_score': None, 'mean_score': None, 'median_score': None, 'std_score': None})

compare_csv(output_file_name='output_2.csv', gold_file_name='result_2.csv', ignore_order=True, specified_columns=['group_id', 'apply_month', 'cnt', 'max_score', 'min_score', 'mode_score', 'mean_score', 'median_score', 'std_score'], thresholds={'max_score': None, 'min_score': None, 'mode_score': None, 'mean_score': None, 'median_score': None, 'std_score': None})


Question:
Given a dataset containing `id`, `apply_date` and 100 feature columns. Use Python to perform KV (Key-Value) packing and output two CSV files:

1. `output_data.csv`: columns must be exactly `id`, `apply_date`, `kv_features`. Keep rows in the original input order. The `kv_features` value must be a Python dict string with braces, comma-space separators, and colon-space separators, for example `"{1: value1, 2: value2, ..., 100: value100}"`. Keys are natural-order integers where `1 -> feature_1`, `2 -> feature_2`, ..., `100 -> feature_100`. Format each value with Python's default float string representation after casting it to `float`.
2. `output_mapping.csv`: columns must be `key`, `feature_name`, sorted by `key` ascending, recording `feature_1` through `feature_100` in natural numeric order.

Column-specific or Feature-wise Processing

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction

compare_csv(output_file_name='output_data.csv', gold_file_name='result_data.csv', ignore_order=False, specified_columns=['id', 'apply_date', 'kv_features'])

compare_csv(output_file_name='output_mapping.csv', gold_file_name='result_mapping.csv', ignore_order=False, specified_columns=['key', 'feature_name'])


Question:
Given an account monthly overdue status transition record table (including fields: `account_id` account ID, `month` month, `prev_status` previous overdue status, `curr_status` current overdue status, where status values are M0, M1, M2, M3, etc.). Please use Python to calculate the roll rate of bills and output two CSV files:

1. Overall roll rate analysis: Calculate the overall conversion rates from M0 to M1, M1 to M2, M2 to M3. Save the results to `output_1.csv`, the schema must include `transition_type` (values are 'M0_to_M1', 'M1_to_M2', 'M2_to_M3') and `roll_rate` (conversion rate value, rounded to 4 decimal places).
2. Monthly roll rate trend analysis: Calculate the roll conversion rates for the above three stages by month to identify deterioration or improvement periods. Save the results to `output_2.csv`, the schema must include `month` (month), `M0_to_M1_rate`, `M1_to_M2_rate`, `M2_to_M3_rate` (all conversion rate values rounded to 4 decimal places).

Conditional Logic and Row-wise Operations

compare_csv(output_file_name='output_1.csv', gold_file_name='result_1.csv', ignore_order=True, specified_columns=['transition_type', 'roll_rate'], thresholds={'roll_rate': None})

compare_csv(output_file_name='output_2.csv', gold_file_name='result_2.csv', ignore_order=True, specified_columns=['month', 'M0_to_M1_rate', 'M1_to_M2_rate', 'M2_to_M3_rate'], thresholds={'M0_to_M1_rate': None, 'M1_to_M2_rate': None, 'M2_to_M3_rate': None})


Question: Given a detailed data table containing new and old version scores. Please calculate the score differences between new and old versions, and output a deviation statistics table `summary_output.csv`: separately calculate the sample proportions of two scores (new vs old versions) under different deviation thresholds. The thresholds are divided into four levels: greater than 1e-3, 1e-4, 1e-5, 1e-6. The schema must include: `score_type` (values 'score1', 'score2'), `gap_gt_1e3_ratio`, `gap_gt_1e4_ratio`, `gap_gt_1e5_ratio`, `gap_gt_1e6_ratio` (all proportion results rounded to 4 decimal places).

Threshold-Based Categorization or Filtering

compare_csv(output_file_name='summary_output.csv', gold_file_name='result_summary.csv', ignore_order=True, specified_columns=['score_type', 'gap_gt_1e3_ratio', 'gap_gt_1e4_ratio', 'gap_gt_1e5_ratio', 'gap_gt_1e6_ratio'], thresholds={'gap_gt_1e3_ratio': None, 'gap_gt_1e4_ratio': None, 'gap_gt_1e5_ratio': None, 'gap_gt_1e6_ratio': None})


Question:
Given a CSV file containing two columns of data (a feature column named 'column_1' and a label column named 'label'). Please use Python to perform monotonic binning on the feature column using a merge-based algorithm, and satisfy the following strict constraints:
1. The proportion of samples in each bin relative to the total sample size must not be less than 5%.
2. The final number of bins (excluding the missing value bin) must be controlled within the range [2, 10].
3. After binning, the positive sample rate (or default rate) of each bin must show monotonicity (monotonically increasing or monotonically decreasing).

The algorithm should work as follows:
- Start with each unique feature value as a separate bin
- Iteratively merge adjacent bins, prioritizing pairs with closest bad rates
- Ensure minimum sample constraint is satisfied during merging
- Perform final checks to ensure monotonicity and sample constraints
- Reduce bins if exceeding maximum limit by merging closest bad rate pairs

Please apply the binning mapping results to the original data, and save the final result as an `output.csv` file. The output CSV file must contain and only contain three columns: `column_1` (original feature value), `column_1_pos` (the bin encoding/bin identifier it belongs to) and `label` (original label value). Missing values in the feature column should remain as null in the output.

Mathematical and Statistical Computations

Validation and Verification of Merge Results

Data Structure Handling (Dictionaries, Lists)

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, specified_columns=['column_1', 'column_1_pos', 'label'])


Question: Train an Uplift model using data where usage='dev' and label=1 from the dataset to predict user response to coupons, evaluate on eval data, and output AUUC and Gini metrics for training and test sets. Output to output.json file in the format {"train": {"AUUC": ..., "Gini": ...}, "eval": {"AUUC": ..., "Gini": ...}}

compare_json_normalized(output_file_name='output.json', thresholds={"train": {"AUUC": [0.0, 1.0], "Gini": [0.0, 1.0]}, "eval": {"AUUC": [0.0, 1.0], "Gini": [0.0, 1.0]}})

Conditional Logic and Row-wise Operations


Question: Train a model to predict label. Feel free to choose training mode and model architecture, and find the overall best performing model. Output to output.json file in the format {"train": {"AUC": ...}, "eval": {"AUC": ...}}

compare_json_normalized(output_file_name='output.json', thresholds={"train": {"AUC": [0.5, 1.0]}, "eval": {"AUC": [0.5, 1.0]}})

Column-specific or Feature-wise Processing


Question: The dataset contains data from multiple customers. Using an Uplift model for training, compare a benefit sensitivity model trained on a single customer's own data versus a model trained on all data combined. Which approach performs better? Output to output.json file in the format {"single_customer": {"train": {"AUUC": ..., "Gini": ...}, "eval": {"AUUC": ..., "Gini": ...}}, "all_data": {"train": {"AUUC": ..., "Gini": ...}, "eval": {"AUUC": ..., "Gini": ...}}}

Column-specific or Feature-wise Processing

Data Preprocessing and Column Management


Question: Train an optimal LGB model using data_split='train' and label_pay=1 data from the dataset to predict label_upgrade=1, with kv_feature as feature columns. Evaluate on eval data and output the CVR and AUC for both training and test sets. Output to output.json file in the format {"train": {"CVR": ..., "AUC": ...}, "eval": {"CVR": ..., "AUC": ...}}

compare_json_normalized(output_file_name='output.json', thresholds={"train": {"CVR": [0.0, 1.0], "AUC": [0.5, 1.0]}, "eval": {"CVR": [0.0, 1.0], "AUC": [0.5, 1.0]}})

Model Configuration and Import
Library Usage (Scikit-Learn)
Data Loading with Pandas
Parsing and Reading Data Files
Text & String Manipulation
String Manipulation and Parsing
Data Parsing and Delimiter Handling
Data Preparation and Formatting
Indexing and Selection
Model Training & Evaluation
Model Training and Customization
Classification and Prediction Modeling
Probability Modeling and Conversion
Model Prediction and Output Handling
Model Evaluation Metrics
Performance Metrics and Optimization
Conditional Aggregation and Filtering
Model Evaluation & Validation
Data Storage and Structuring
Validation and Output Formatting
Data Serialization & File Handling


Question: Train an LGB model to predict whether label_pay is 1, with kv_feature as feature columns. The model needs to be updated daily: train using all data before day T, evaluate on day T's data, and continuously evaluate the latest 7 days. Output daily CVR and AUC for training and test sets. Output to output.json file in the format {"day_1": {"train": {"CVR": ..., "AUC": ...}, "eval": {...}}, "day_2": {...}, ...}

Column-specific or Feature-wise Processing


Question: Train a DeepFM model to predict whether label_pay is 1, with kv_feature as feature columns. The model needs to be updated daily: train using all data before day T, evaluate on day T's data, and continuously evaluate the latest 7 days. Output daily CVR and AUC for training and test sets. Evaluate using both full training and incremental training modes. Output to output.json file in the format {"full_training": {"day_1": {"train": {...}, "eval": {...}}, ...}, "incremental_training": {"day_1": {"train": {...}, "eval": {...}}, ...}}

Data Conversion and Post-Loading Processing

Online Learning and Incremental Training


Question: Train a multi-task model to predict label_pay and label_upgrade respectively, with kv_feature as feature columns. Feel free to choose training mode and model architecture, and find the overall best performing model. Output to output.json file in the format {"label_pay": {"train": {...}, "eval": {...}}, "label_upgrade": {"train": {...}, "eval": {...}}}

Data Conversion and Post-Loading Processing


Question: The dataset contains data from multiple customers. Using LGB for training, compare a model trained on a single customer's own data to predict label_pay versus a model trained on all data combined. Which approach performs better? Output to output.json file in the format {"single_customer": {"train": {"CVR": ..., "AUC": ...}, "eval": {"CVR": ..., "AUC": ...}}, "all_data": {"train": {"CVR": ..., "AUC": ...}, "eval": {"CVR": ..., "AUC": ...}}}

Data Preprocessing and Column Management

Data Labeling and Structured Data Handling

Column-specific or Feature-wise Processing

Incremental and Comparative Calculations


Question: Train a binary classification model using data_split='train' data from the dataset to predict whether label_pay is 1, and evaluate on the eval dataset. Output to output.json file in the format {"train": {"CVR": ..., "AUC": ...}, "eval": {"CVR": ..., "AUC": ...}}

compare_json_normalized(output_file_name='output.json', thresholds={"train": {"CVR": [0.0, 1.0], "AUC": [0.5, 1.0]}, "eval": {"CVR": [0.0, 1.0], "AUC": [0.5, 1.0]}})

Column-specific or Feature-wise Processing

Data Preprocessing and Column Management


Question:
Based on a multi-source real estate dataset, analyze the impact of social and economic factors on housing price growth in each state of the United States.

Task:
1. Load the Zillow housing price dataset, education expenditure data, unemployment rate data by state, and county-level census data.
2. Calculate the annual housing price growth rate using December data from 2010 to 2018.
3. Calculate three social economic indicators for 2011-2018: per capita education expenditure (total revenue / enrollment), annual average unemployment rate, and state-level average per capita income (aggregated from county-level data).
4. Use scipy.stats.pearsonr to calculate the Pearson correlation coefficient and p-value between the housing price growth rate and each of the three indicators.
5. Find the indicator with the strongest correlation (largest absolute correlation coefficient).
6. Output a CSV file containing: the strongest correlated indicator name, correlation coefficient (4 decimal places), and p-value (4 decimal places).

Save the result as correlation_output.csv.

Unemployment in America Per US State.csv

US Census Demographic Data/acs2015_county_data.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

Incremental and Comparative Calculations

compare_csv(output_file_name='correlation_output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'correlation_coefficient': None, 'p_value': None})


Question:
Please calculate the Sharpe ratio for each city based on the multi-source real estate dataset and generate an analysis report (save as top50_sharpe_ratio_cities.csv).

Using 2017 monthly housing price data from three Zillow datasets (1-bedroom, 2-bedroom, 3-bedroom), compute monthly returns for each bedroom type per city. Then calculate:
- Annualized return = mean of monthly returns × 12
- Annualized volatility = sample standard deviation (ddof=1) of monthly returns × √12
Average the annualized return and volatility across all available bedroom types for each city. Sharpe ratio = annualized return / volatility (risk-free rate = 0).

From the education dataset, calculate the average 8th-grade math score by state. From the census dataset, calculate the average per capita income by state. Merge these state-level indicators into the city data.

Select the top 50 cities with the highest Sharpe ratio. Output columns: City, State, Annualized_Return, Volatility, Sharpe_Ratio, Avg_Math_8_Score, Income_Per_Capita.

US Census Demographic Data/acs2017_county_data.csv

Zillow House Price Data/City_Zhvi_1bedroom.csv

Zillow House Price Data/City_Zhvi_2bedroom.csv

Zillow House Price Data/City_Zhvi_3bedroom.csv

Data Structure Handling (Dictionaries, Lists)

Dictionary Manipulation and Construction

Vectorization and Performance Optimization

compare_csv(output_file_name='top50_sharpe_ratio_cities.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'Annualized_Return': None, 'Volatility': None, 'Sharpe_Ratio': None, 'Avg_Math_8_Score': None, 'Income_Per_Capita': None})


Question:
Please calculate the annualized growth rate of different types of houses in each state based on the multi-source real estate dataset, and generate an analysis report (save as top5_cagr_disparity.csv). The report should include the following analysis dimensions:

Data loading: Load the house price data files of Zillow for the state levels. Determine the data columns for December 31, 2010 and December 31, 2019.

Growth rate calculation: For each data file, extract the house price data for December 31, 2010 and December 31, 2019, and calculate the growth rate (compound annual growth rate) = (end value / start value)^(1/9) - 1. Merge the results into a data frame, including three columns: state name (region name), house type (property type), and growth rate.

Statistical analysis: Calculate the growth rate disparity for each state. For each state, calculate the difference between the highest and lowest growth rates among the 8 house types (growth rate difference = maximum growth rate - minimum growth rate). Then sort by the growth rate difference in descending order and select the top 5 states.

Result output: Save the results of the top 5 states as top5_cagr_disparity.csv, including columns 'region name' and 'growth rate difference'.

Zillow House Price Data/State_Zhvi_AllHomes.csv

Zillow House Price Data/State_Zhvi_1bedroom.csv

Zillow House Price Data/State_Zhvi_2bedroom.csv

Zillow House Price Data/State_Zhvi_3bedroom.csv

Zillow House Price Data/State_Zhvi_4bedroom.csv

Zillow House Price Data/State_Zhvi_5BedroomOrMore.csv

Zillow House Price Data/State_Zhvi_Condominum.csv

Zillow House Price Data/State_Zhvi_SingleFamilyResidence.csv

Data Structure Handling (Dictionaries, Lists)

Column-wise Transformations and Aggregation

Dynamic Data Transformation and Insertion

Incremental and Comparative Calculations

compare_csv(output_file_name='top5_cagr_disparity.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'CAGR_Disparity': None})


Question:
A national housing policy research group is conducting a study on the correlation between educational investment and the dynamics of housing markets in various states of the United States, taking into account the macroeconomic and demographic background.

Task Requirements:
1. Using the dataset of median house prices in the United States, calculate the annual growth rate of median house prices for each state from 2010 to 2018.
2. Using the dataset of education in the United States, calculate the education expenditure per student (total income / enrollment number) for each state.
3. Using the dataset of unemployment rates in the United States, calculate the average annual unemployment rate for each state.
4. Using the county-level economic data from the 2015 American Community Survey and the 2017 American Community Survey, aggregate the data to the state level and obtain the average income (IncomePerCap) and poverty rate (Poverty) for 2015 and 2017.
5. Merge the above datasets by state name and year.
6. Apply the ARIMA(1,1,1) model to the time series of each student's education expenditure to predict the next 12 months.
7. Calculate the Pearson correlation coefficient between the housing price growth rate and educational investment.
8. Return the top 5 states with the strongest correlation to the target variable, including the following information: correlation coefficient, p-value, 95% confidence interval, average annual growth rate, average amount of financial aid per student.

Output Requirements: Save the results as output.csv file, including the following columns: state, correlation, p-value, lower limit of confidence interval, upper limit of confidence interval, average annual growth rate, average amount of financial aid per student.

Unemployment in America Per US State.csv

US Census Demographic Data/acs2015_county_data.csv

US Census Demographic Data/acs2017_county_data.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

Arithmetic Transformations and Normalization

Time Difference and Gradient Calculation

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=True, thresholds={'Correlation_Coefficient': None, 'P_Value': None, 'CI_Lower_95': None, 'CI_Upper_95': None, 'Avg_Housing_Growth_Rate': None, 'Avg_Revenue_Per_Student': None})


Question:
A national housing policy research group is studying the correlation between educational investment and housing market dynamics across U.S. states, taking into account macroeconomic and demographic factors.

For the period 2010-2018, analyze the relationship between each state's annual housing price growth rate and per-student education expenditure (total revenue / enrollment). Incorporate state-level unemployment rates and American Community Survey demographic data (2015 and 2017, aggregated to state level) by merging all datasets at the state-year level. Apply an ARIMA(1,1,1) model to each state's per-student education expenditure time series to forecast the next 12 periods. Compute the Pearson correlation coefficient between housing price growth rate and per-student education expenditure for each state, and identify the top 5 states with the strongest absolute correlation. For the 95% confidence interval, use the standard error of the correlation coefficient with t-distribution critical values.

Output Requirements: Save results as output.csv with columns: state, correlation, p_value, ci_lower, ci_upper, avg_annual_growth, avg_per_student_funding.

Unemployment in America Per US State.csv

Zillow House Price Data/State_Zhvi_AllHomes.csv

US Census Demographic Data/acs2015_county_data.csv

US Census Demographic Data/acs2017_county_data.csv

Incremental and Comparative Calculations

Data Conversion and Post-Loading Processing

Dynamic Data Transformation and Insertion

compare_csv(output_file_name='output.csv', gold_file_name='result.csv', ignore_order=False, specified_columns=['state', 'correlation', 'p_value', 'ci_lower', 'ci_upper', 'avg_annual_growth', 'avg_per_student_funding'], thresholds={'correlation': None, 'p_value': None, 'ci_lower': None, 'ci_upper': None, 'avg_annual_growth': None, 'avg_per_student_funding': None})