Update README.md
Browse files
README.md
CHANGED
|
@@ -35,57 +35,57 @@ https://huggingface.co/datasets/drukeroni/airline-satisfaction-analysis
|
|
| 35 |
In this section, I performed data cleaning and initial analysis to understand the dataset better before building the models.
|
| 36 |
|
| 37 |
#### 1. Data Cleaning
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
#### 2. Descriptive Statistics & Data Structure
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
#### 3. Outlier Detection
|
| 47 |
-
|
| 48 |
-
|
| 49 |
#### Data Exploration: Answering Key Research Questions through Visualization
|
| 50 |
#### Following the detection of outliers in flight distance, how extreme is their distribution and what impact might they have on the model's scaling?
|
| 51 |

|
| 52 |
This boxplot displays the distribution of flight distances and identifies extreme outliers that could distort the model's data scaling.
|
| 53 |
It serves as visual evidence for the capping strategy needed to ensure data quality and better performance in future modeling.
|
| 54 |
|
| 55 |
-
####
|
| 56 |

|
| 57 |
Both plots show a highly right-skewed distribution with extreme outliers reaching 1,600 minutes, meaning most flights are on time while a few have massive delays.
|
| 58 |
There is a strong correlation between departure and arrival delays, which requires handling outliers (like capping or log-transformation) to improve regression accuracy.
|
| 59 |
While these extreme values can skew numerical predictions in regression, they are easier to handle in classification tasks where the goal is binary status prediction.
|
| 60 |
|
| 61 |
-
####
|
| 62 |

|
| 63 |
The plot shows that while most ratings are concentrated between 4 and 5, it visually confirms the presence of '0' values across various service categories.
|
| 64 |
|
| 65 |
-
####
|
| 66 |

|
| 67 |
This scatter plot shows a strong positive correlation between departure and arrival delays, while highlighting how extreme outliers deviate from the main cluster.
|
| 68 |
|
| 69 |
-
####
|
| 70 |

|
| 71 |
After cleaning the data, the scatter plot now displays a much clearer and more reliable linear relationship between the two types of delays.
|
| 72 |
Removing the extreme anomalies allows us to visualize the core data patterns that will be used for our predictive modeling.
|
| 73 |
|
| 74 |
# Part 3: Baseline Regression Modeling
|
| 75 |
-
In this section,
|
| 76 |
### 1. Data Preparation & Feature Selection
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
#### 2. Model Training
|
| 81 |
-
|
| 82 |
-
|
| 83 |
#### 3. Performance Evaluation
|
| 84 |
-
|
| 85 |
|
| 86 |
#### Baseline Model: Actual vs Predicted Arrival Delays
|
| 87 |

|
| 88 |
-
The model is
|
| 89 |
The scatter plot shows most points are close to the red line, proving a strong connection between predictions and reality.
|
| 90 |
### Feature Importance
|
| 91 |

|
|
@@ -93,18 +93,18 @@ The chart shows that departure delay is by far the most important factor for pre
|
|
| 93 |
Other features, such as passenger age or flight distance, show very low importance in this baseline model.
|
| 94 |
|
| 95 |
# Part 4: Advanced Feature Engineering & Preprocessing
|
| 96 |
-
In this stage,
|
| 97 |
#### 1. Creating the Classification Target
|
| 98 |
-
|
| 99 |
A flight is labeled as "Delayed" (1) if the arrival delay is greater than 15 minutes, otherwise it is labeled (0).
|
| 100 |
#### 2. Categorical Encoding
|
| 101 |
-
|
| 102 |
-
|
| 103 |
#### 3. Feature Scaling
|
| 104 |
-
|
| 105 |
This normalizes the data so that features with larger scales do not unfairly dominate the model's learning process.
|
| 106 |
#### 4. Feature Engineering with Unsupervised Learning (K-Means)
|
| 107 |
-
|
| 108 |
This new feature, passenger_profile, allows the model to understand complex patterns of passenger satisfaction.
|
| 109 |
#### 5. Cluster Visualization (PCA)
|
| 110 |
To validate the clusters, I used PCA (Principal Component Analysis) to reduce the service ratings into two dimensions.
|
|
|
|
| 35 |
In this section, I performed data cleaning and initial analysis to understand the dataset better before building the models.
|
| 36 |
|
| 37 |
#### 1. Data Cleaning
|
| 38 |
+
selected the 15 features that are most relevant to my research question.
|
| 39 |
+
checked for missing values and decided to drop rows where the target variable (Arrival Delay in Minutes) was missing to keep the data accurate.
|
| 40 |
+
verified that there are no duplicate rows in the dataset.
|
| 41 |
+
standardized all text columns by converting them to lowercase and removing extra spaces.
|
| 42 |
#### 2. Descriptive Statistics & Data Structure
|
| 43 |
+
checked the data types and the final shape of the table after cleaning.
|
| 44 |
+
calculated the percentage and count for each category, like Gender, Customer Type, and satisfaction, to see how the data is distributed.
|
| 45 |
+
checked the service rating scales (like Inflight wifi service) to make sure all values are between 1 and 5.
|
| 46 |
#### 3. Outlier Detection
|
| 47 |
+
used the IQR (Interquartile Range) method to find outliers in columns like Age, Flight Distance, and delay times.
|
| 48 |
+
calculated the percentage of outliers for each feature to understand how many extreme values exist in the data.
|
| 49 |
#### Data Exploration: Answering Key Research Questions through Visualization
|
| 50 |
#### Following the detection of outliers in flight distance, how extreme is their distribution and what impact might they have on the model's scaling?
|
| 51 |

|
| 52 |
This boxplot displays the distribution of flight distances and identifies extreme outliers that could distort the model's data scaling.
|
| 53 |
It serves as visual evidence for the capping strategy needed to ensure data quality and better performance in future modeling.
|
| 54 |
|
| 55 |
+
#### Following the detection of outliers in flight delays, how are departure and arrival delays distributed, and what do these extreme values indicate about the data set?
|
| 56 |

|
| 57 |
Both plots show a highly right-skewed distribution with extreme outliers reaching 1,600 minutes, meaning most flights are on time while a few have massive delays.
|
| 58 |
There is a strong correlation between departure and arrival delays, which requires handling outliers (like capping or log-transformation) to improve regression accuracy.
|
| 59 |
While these extreme values can skew numerical predictions in regression, they are easier to handle in classification tasks where the goal is binary status prediction.
|
| 60 |
|
| 61 |
+
#### How are the satisfaction ratings distributed, and what does the presence of '0' values in a 1-5 scale indicate about data quality?
|
| 62 |

|
| 63 |
The plot shows that while most ratings are concentrated between 4 and 5, it visually confirms the presence of '0' values across various service categories.
|
| 64 |
|
| 65 |
+
#### What is the correlation between departure and arrival delays, and how do extreme outliers reflect unusual flight patterns?
|
| 66 |

|
| 67 |
This scatter plot shows a strong positive correlation between departure and arrival delays, while highlighting how extreme outliers deviate from the main cluster.
|
| 68 |
|
| 69 |
+
#### What is the correlation between departure and arrival delays, and how do cleaning the extreme outliers reflect the flight patterns?
|
| 70 |

|
| 71 |
After cleaning the data, the scatter plot now displays a much clearer and more reliable linear relationship between the two types of delays.
|
| 72 |
Removing the extreme anomalies allows us to visualize the core data patterns that will be used for our predictive modeling.
|
| 73 |
|
| 74 |
# Part 3: Baseline Regression Modeling
|
| 75 |
+
In this section, we built a baseline Linear Regression model to establish a performance benchmark for predicting flight delays.
|
| 76 |
### 1. Data Preparation & Feature Selection
|
| 77 |
+
defined Arrival Delay in Minutes as the target variable.
|
| 78 |
+
selected 8 key numerical features (like Age and Flight Distance) as predictors.
|
| 79 |
+
dropped missing values again just to be 100% sure the data is completely clean for the model.
|
| 80 |
#### 2. Model Training
|
| 81 |
+
splited the data into 80% training and 20% testing sets (using random_state=42 for consistency).
|
| 82 |
+
trained a basic Linear Regression model to learn the relationship between the features and delays.
|
| 83 |
#### 3. Performance Evaluation
|
| 84 |
+
evaluated the model's accuracy using MAE, MSE, RMSE, and R-squared.
|
| 85 |
|
| 86 |
#### Baseline Model: Actual vs Predicted Arrival Delays
|
| 87 |

|
| 88 |
+
The model is accurate, explaining 90.27% of the delays with an average error of only 5.24 minutes.
|
| 89 |
The scatter plot shows most points are close to the red line, proving a strong connection between predictions and reality.
|
| 90 |
### Feature Importance
|
| 91 |

|
|
|
|
| 93 |
Other features, such as passenger age or flight distance, show very low importance in this baseline model.
|
| 94 |
|
| 95 |
# Part 4: Advanced Feature Engineering & Preprocessing
|
| 96 |
+
In this stage, we prepared the dataset for more complex models and created new features to improve prediction power.
|
| 97 |
#### 1. Creating the Classification Target
|
| 98 |
+
converted the regression problem into a classification task by creating a binary feature, is_delayed.
|
| 99 |
A flight is labeled as "Delayed" (1) if the arrival delay is greater than 15 minutes, otherwise it is labeled (0).
|
| 100 |
#### 2. Categorical Encoding
|
| 101 |
+
transformed all text-based features (like Gender, Class, and Customer Type) into numeric format using One-Hot Encoding.
|
| 102 |
+
used the drop_first=True logic to prevent multi-collinearity, ensuring the model remains statistically stable.
|
| 103 |
#### 3. Feature Scaling
|
| 104 |
+
applied StandardScaler to numerical columns such as Age and Flight Distance.
|
| 105 |
This normalizes the data so that features with larger scales do not unfairly dominate the model's learning process.
|
| 106 |
#### 4. Feature Engineering with Unsupervised Learning (K-Means)
|
| 107 |
+
used K-Means Clustering to group passengers into 3 distinct "Service Profiles" based on their ratings of inflight services (Wi-Fi, Cleanliness, etc.).
|
| 108 |
This new feature, passenger_profile, allows the model to understand complex patterns of passenger satisfaction.
|
| 109 |
#### 5. Cluster Visualization (PCA)
|
| 110 |
To validate the clusters, I used PCA (Principal Component Analysis) to reduce the service ratings into two dimensions.
|