Yoel125 commited on
Commit
529205a
·
verified ·
1 Parent(s): 7cca5ca

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +27 -27
README.md CHANGED
@@ -35,57 +35,57 @@ https://huggingface.co/datasets/drukeroni/airline-satisfaction-analysis
35
  In this section, I performed data cleaning and initial analysis to understand the dataset better before building the models.
36
 
37
  #### 1. Data Cleaning
38
- I selected the 15 features that are most relevant to my research question.
39
- I checked for missing values and decided to drop rows where the target variable (Arrival Delay in Minutes) was missing to keep the data accurate.
40
- I verified that there are no duplicate rows in the dataset.
41
- I standardized all text columns by converting them to lowercase and removing extra spaces.
42
  #### 2. Descriptive Statistics & Data Structure
43
- I checked the data types and the final shape of the table after cleaning.
44
- I calculated the percentage and count for each category, like Gender, Customer Type, and satisfaction, to see how the data is distributed.
45
- I checked the service rating scales (like Inflight wifi service) to make sure all values are between 1 and 5.
46
  #### 3. Outlier Detection
47
- I used the IQR (Interquartile Range) method to find outliers in columns like Age, Flight Distance, and delay times.
48
- I calculated the percentage of outliers for each feature to understand how many extreme values exist in the data.
49
  #### Data Exploration: Answering Key Research Questions through Visualization
50
  #### Following the detection of outliers in flight distance, how extreme is their distribution and what impact might they have on the model's scaling?
51
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/oIXebKJ0KqMjckAlGjYn-.png)
52
  This boxplot displays the distribution of flight distances and identifies extreme outliers that could distort the model's data scaling.
53
  It serves as visual evidence for the capping strategy needed to ensure data quality and better performance in future modeling.
54
 
55
- ##### Following the detection of outliers in flight delays, how are departure and arrival delays distributed, and what do these extreme values indicate about the data set?
56
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/wWj_FKGb-0Et3ebK54Csh.png)
57
  Both plots show a highly right-skewed distribution with extreme outliers reaching 1,600 minutes, meaning most flights are on time while a few have massive delays.
58
  There is a strong correlation between departure and arrival delays, which requires handling outliers (like capping or log-transformation) to improve regression accuracy.
59
  While these extreme values can skew numerical predictions in regression, they are easier to handle in classification tasks where the goal is binary status prediction.
60
 
61
- ##### How are the satisfaction ratings distributed, and what does the presence of '0' values in a 1-5 scale indicate about data quality?
62
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/wgnKWFRPS--OgH--lLnEK.png)
63
  The plot shows that while most ratings are concentrated between 4 and 5, it visually confirms the presence of '0' values across various service categories.
64
 
65
- ##### What is the correlation between departure and arrival delays, and how do extreme outliers reflect unusual flight patterns?
66
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/qJy6FKjaVxlonbAhShNcf.png)
67
  This scatter plot shows a strong positive correlation between departure and arrival delays, while highlighting how extreme outliers deviate from the main cluster.
68
 
69
- ##### What is the correlation between departure and arrival delays, and how do cleaning the extreme outliers reflect the flight patterns?
70
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/udUMbiqphRopYggAj7llT.png)
71
  After cleaning the data, the scatter plot now displays a much clearer and more reliable linear relationship between the two types of delays.
72
  Removing the extreme anomalies allows us to visualize the core data patterns that will be used for our predictive modeling.
73
 
74
  # Part 3: Baseline Regression Modeling
75
- In this section, I built a baseline Linear Regression model to establish a performance benchmark for predicting flight delays.
76
  ### 1. Data Preparation & Feature Selection
77
- I defined Arrival Delay in Minutes as the target variable.
78
- I selected 8 key numerical features (like Age and Flight Distance) as predictors.
79
- I dropped missing values again just to be 100% sure the data is completely clean for the model.
80
  #### 2. Model Training
81
- I split the data into 80% training and 20% testing sets (using random_state=42 for consistency).
82
- I trained a basic Linear Regression model to learn the relationship between the features and delays.
83
  #### 3. Performance Evaluation
84
- I evaluated the model's accuracy using MAE, MSE, RMSE, and R-squared.
85
 
86
  #### Baseline Model: Actual vs Predicted Arrival Delays
87
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/_swJewXEptmPng7GTQgY_.png)
88
- The model is very accurate, explaining 90.27% of the delays with an average error of only 5.24 minutes.
89
  The scatter plot shows most points are close to the red line, proving a strong connection between predictions and reality.
90
  ### Feature Importance
91
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/YdkPz9pUWp1xl7B74ay13.png)
@@ -93,18 +93,18 @@ The chart shows that departure delay is by far the most important factor for pre
93
  Other features, such as passenger age or flight distance, show very low importance in this baseline model.
94
 
95
  # Part 4: Advanced Feature Engineering & Preprocessing
96
- In this stage, I prepared the dataset for more complex models and created new features to improve prediction power.
97
  #### 1. Creating the Classification Target
98
- I converted the regression problem into a classification task by creating a binary feature, is_delayed.
99
  A flight is labeled as "Delayed" (1) if the arrival delay is greater than 15 minutes, otherwise it is labeled (0).
100
  #### 2. Categorical Encoding
101
- I transformed all text-based features (like Gender, Class, and Customer Type) into numeric format using One-Hot Encoding.
102
- I used the drop_first=True logic to prevent multi-collinearity, ensuring the model remains statistically stable.
103
  #### 3. Feature Scaling
104
- I applied StandardScaler to numerical columns such as Age and Flight Distance.
105
  This normalizes the data so that features with larger scales do not unfairly dominate the model's learning process.
106
  #### 4. Feature Engineering with Unsupervised Learning (K-Means)
107
- I used K-Means Clustering to group passengers into 3 distinct "Service Profiles" based on their ratings of inflight services (Wi-Fi, Cleanliness, etc.).
108
  This new feature, passenger_profile, allows the model to understand complex patterns of passenger satisfaction.
109
  #### 5. Cluster Visualization (PCA)
110
  To validate the clusters, I used PCA (Principal Component Analysis) to reduce the service ratings into two dimensions.
 
35
  In this section, I performed data cleaning and initial analysis to understand the dataset better before building the models.
36
 
37
  #### 1. Data Cleaning
38
+ selected the 15 features that are most relevant to my research question.
39
+ checked for missing values and decided to drop rows where the target variable (Arrival Delay in Minutes) was missing to keep the data accurate.
40
+ verified that there are no duplicate rows in the dataset.
41
+ standardized all text columns by converting them to lowercase and removing extra spaces.
42
  #### 2. Descriptive Statistics & Data Structure
43
+ checked the data types and the final shape of the table after cleaning.
44
+ calculated the percentage and count for each category, like Gender, Customer Type, and satisfaction, to see how the data is distributed.
45
+ checked the service rating scales (like Inflight wifi service) to make sure all values are between 1 and 5.
46
  #### 3. Outlier Detection
47
+ used the IQR (Interquartile Range) method to find outliers in columns like Age, Flight Distance, and delay times.
48
+ calculated the percentage of outliers for each feature to understand how many extreme values exist in the data.
49
  #### Data Exploration: Answering Key Research Questions through Visualization
50
  #### Following the detection of outliers in flight distance, how extreme is their distribution and what impact might they have on the model's scaling?
51
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/oIXebKJ0KqMjckAlGjYn-.png)
52
  This boxplot displays the distribution of flight distances and identifies extreme outliers that could distort the model's data scaling.
53
  It serves as visual evidence for the capping strategy needed to ensure data quality and better performance in future modeling.
54
 
55
+ #### Following the detection of outliers in flight delays, how are departure and arrival delays distributed, and what do these extreme values indicate about the data set?
56
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/wWj_FKGb-0Et3ebK54Csh.png)
57
  Both plots show a highly right-skewed distribution with extreme outliers reaching 1,600 minutes, meaning most flights are on time while a few have massive delays.
58
  There is a strong correlation between departure and arrival delays, which requires handling outliers (like capping or log-transformation) to improve regression accuracy.
59
  While these extreme values can skew numerical predictions in regression, they are easier to handle in classification tasks where the goal is binary status prediction.
60
 
61
+ #### How are the satisfaction ratings distributed, and what does the presence of '0' values in a 1-5 scale indicate about data quality?
62
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/wgnKWFRPS--OgH--lLnEK.png)
63
  The plot shows that while most ratings are concentrated between 4 and 5, it visually confirms the presence of '0' values across various service categories.
64
 
65
+ #### What is the correlation between departure and arrival delays, and how do extreme outliers reflect unusual flight patterns?
66
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/qJy6FKjaVxlonbAhShNcf.png)
67
  This scatter plot shows a strong positive correlation between departure and arrival delays, while highlighting how extreme outliers deviate from the main cluster.
68
 
69
+ #### What is the correlation between departure and arrival delays, and how do cleaning the extreme outliers reflect the flight patterns?
70
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/udUMbiqphRopYggAj7llT.png)
71
  After cleaning the data, the scatter plot now displays a much clearer and more reliable linear relationship between the two types of delays.
72
  Removing the extreme anomalies allows us to visualize the core data patterns that will be used for our predictive modeling.
73
 
74
  # Part 3: Baseline Regression Modeling
75
+ In this section, we built a baseline Linear Regression model to establish a performance benchmark for predicting flight delays.
76
  ### 1. Data Preparation & Feature Selection
77
+ defined Arrival Delay in Minutes as the target variable.
78
+ selected 8 key numerical features (like Age and Flight Distance) as predictors.
79
+ dropped missing values again just to be 100% sure the data is completely clean for the model.
80
  #### 2. Model Training
81
+ splited the data into 80% training and 20% testing sets (using random_state=42 for consistency).
82
+ trained a basic Linear Regression model to learn the relationship between the features and delays.
83
  #### 3. Performance Evaluation
84
+ evaluated the model's accuracy using MAE, MSE, RMSE, and R-squared.
85
 
86
  #### Baseline Model: Actual vs Predicted Arrival Delays
87
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/_swJewXEptmPng7GTQgY_.png)
88
+ The model is accurate, explaining 90.27% of the delays with an average error of only 5.24 minutes.
89
  The scatter plot shows most points are close to the red line, proving a strong connection between predictions and reality.
90
  ### Feature Importance
91
  ![image](https://cdn-uploads.huggingface.co/production/uploads/69c79aa8f856b118f80df631/YdkPz9pUWp1xl7B74ay13.png)
 
93
  Other features, such as passenger age or flight distance, show very low importance in this baseline model.
94
 
95
  # Part 4: Advanced Feature Engineering & Preprocessing
96
+ In this stage, we prepared the dataset for more complex models and created new features to improve prediction power.
97
  #### 1. Creating the Classification Target
98
+ converted the regression problem into a classification task by creating a binary feature, is_delayed.
99
  A flight is labeled as "Delayed" (1) if the arrival delay is greater than 15 minutes, otherwise it is labeled (0).
100
  #### 2. Categorical Encoding
101
+ transformed all text-based features (like Gender, Class, and Customer Type) into numeric format using One-Hot Encoding.
102
+ used the drop_first=True logic to prevent multi-collinearity, ensuring the model remains statistically stable.
103
  #### 3. Feature Scaling
104
+ applied StandardScaler to numerical columns such as Age and Flight Distance.
105
  This normalizes the data so that features with larger scales do not unfairly dominate the model's learning process.
106
  #### 4. Feature Engineering with Unsupervised Learning (K-Means)
107
+ used K-Means Clustering to group passengers into 3 distinct "Service Profiles" based on their ratings of inflight services (Wi-Fi, Cleanliness, etc.).
108
  This new feature, passenger_profile, allows the model to understand complex patterns of passenger satisfaction.
109
  #### 5. Cluster Visualization (PCA)
110
  To validate the clusters, I used PCA (Principal Component Analysis) to reduce the service ratings into two dimensions.