{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60380f61",
   "metadata": {},
   "source": [
    "# NYC Taxi Fare Prediction using Artificial Neural Network\n",
    "\n",
    "This notebook demonstrates a complete machine learning workflow for predicting taxi fare amounts using the NYC Taxi Fare Prediction dataset with an Artificial Neural Network (ANN) model built using scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e5dc53",
   "metadata": {},
   "source": [
    "## 1. Objective\n",
    "\n",
    "The goal of this assignment is to:\n",
    "- **Preprocess and clean** the NYC Taxi Fare dataset, handling missing values and outliers\n",
    "- **Engineer features** from pickup datetime and geographic coordinates\n",
    "- **Train an Artificial Neural Network** (MLP Regressor) for fare amount prediction\n",
    "- **Evaluate** the model using appropriate metrics (MAE, RMSE, R²)\n",
    "- **Deploy** the model as an interactive Gradio application on Hugging Face Spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f630ea12",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Required Libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Print versions for reproducibility\n",
    "print(f\"Pandas: {pd.__version__}\")\n",
    "print(f\"NumPy: {np.__version__}\")\n",
    "print(f\"Scikit-learn: {sklearn.__version__}\")\n",
    "\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "246fdf11",
   "metadata": {},
   "source": [
    "## 2. Data Loading and Exploration\n",
    "\n",
    "The NYC Taxi Fare Prediction dataset contains historically accurate taxi trip data with fare amounts. Let's load and explore the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcd1fa7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "df = pd.read_csv('data/nyc_taxi_fare.csv')\n",
    "print(f\"Dataset shape: {df.shape}\")\n",
    "print(f\"\\nFirst few rows:\")\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b820342",
   "metadata": {},
   "source": [
    "## 3. Data Preprocessing and Cleaning\n",
    "\n",
    "We need to clean the data by:\n",
    "- Removing null values\n",
    "- Filtering out unrealistic fare amounts\n",
    "- Validating geographic coordinates (must be within NYC bounds)\n",
    "- Ensuring pickup and dropoff are different locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9ff6533",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data cleaning\n",
    "rows_before = len(df)\n",
    "\n",
    "# Convert datetime\n",
    "df['pickup_datetime'] = pd.to_datetime(df['pickup_datetime'], errors='coerce')\n",
    "\n",
    "# Remove null values\n",
    "required_cols = ['pickup_datetime', 'pickup_longitude', 'pickup_latitude', \n",
    "                 'dropoff_longitude', 'dropoff_latitude', 'passenger_count', 'fare_amount']\n",
    "df = df.dropna(subset=required_cols)\n",
    "\n",
    "# Filter fare amounts (realistic range)\n",
    "df = df[(df['fare_amount'] >= 0) & (df['fare_amount'] <= 500)]\n",
    "\n",
    "# Filter passenger count (realistic range)\n",
    "df = df[(df['passenger_count'] >= 1) & (df['passenger_count'] <= 8)]\n",
    "\n",
    "# NYC geographic bounds\n",
    "df = df[(df['pickup_latitude'] >= 40.0) & (df['pickup_latitude'] <= 42.0)]\n",
    "df = df[(df['pickup_longitude'] >= -75.5) & (df['pickup_longitude'] <= -72.5)]\n",
    "df = df[(df['dropoff_latitude'] >= 40.0) & (df['dropoff_latitude'] <= 42.0)]\n",
    "df = df[(df['dropoff_longitude'] >= -75.5) & (df['dropoff_longitude'] <= -72.5)]\n",
    "\n",
    "# Ensure pickup and dropoff are different\n",
    "df = df[(df['pickup_latitude'] != df['dropoff_latitude']) | \n",
    "        (df['pickup_longitude'] != df['dropoff_longitude'])]\n",
    "\n",
    "rows_after = len(df)\n",
    "print(f\"Rows before cleaning: {rows_before}\")\n",
    "print(f\"Rows after cleaning: {rows_after}\")\n",
    "print(f\"Rows removed: {rows_before - rows_after}\")\n",
    "print(f\"Data shape: {df.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f699c3",
   "metadata": {},
   "source": [
    "## 4. Feature Engineering\n",
    "\n",
    "We'll engineer features from the raw data:\n",
    "- **Time-based features**: hour, day of week, month, year, weekend flag\n",
    "- **Distance features**: Haversine distance between pickup and dropoff\n",
    "- **Location features**: Absolute differences in latitude and longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46902ac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Engineering\n",
    "\n",
    "# Time-based features\n",
    "df['pickup_hour'] = df['pickup_datetime'].dt.hour.fillna(0).astype(int)\n",
    "df['pickup_day_of_week'] = df['pickup_datetime'].dt.dayofweek.fillna(0).astype(int)\n",
    "df['pickup_month'] = df['pickup_datetime'].dt.month.fillna(0).astype(int)\n",
    "df['pickup_year'] = df['pickup_datetime'].dt.year.fillna(0).astype(int)\n",
    "df['is_weekend'] = (df['pickup_datetime'].dt.dayofweek >= 5).astype(int)\n",
    "\n",
    "# Haversine distance calculation\n",
    "def haversine_km(lat1, lon1, lat2, lon2):\n",
    "    earth_radius_km = 6371.0088\n",
    "    lat1_rad = np.radians(lat1)\n",
    "    lon1_rad = np.radians(lon1)\n",
    "    lat2_rad = np.radians(lat2)\n",
    "    lon2_rad = np.radians(lon2)\n",
    "    \n",
    "    delta_lat = lat2_rad - lat1_rad\n",
    "    delta_lon = lon2_rad - lon1_rad\n",
    "    a = np.sin(delta_lat/2)**2 + np.cos(lat1_rad)*np.cos(lat2_rad)*np.sin(delta_lon/2)**2\n",
    "    return 2 * earth_radius_km * np.arcsin(np.sqrt(a))\n",
    "\n",
    "df['trip_distance_km'] = haversine_km(\n",
    "    df['pickup_latitude'], df['pickup_longitude'],\n",
    "    df['dropoff_latitude'], df['dropoff_longitude']\n",
    ")\n",
    "\n",
    "# Geographic features\n",
    "df['abs_lat_diff'] = (df['pickup_latitude'] - df['dropoff_latitude']).abs()\n",
    "df['abs_lon_diff'] = (df['pickup_longitude'] - df['dropoff_longitude']).abs()\n",
    "\n",
    "print(\"Features created successfully!\")\n",
    "print(f\"Feature columns: {[col for col in df.columns if col not in ['key', 'fare_amount', 'pickup_datetime', 'pickup_longitude', 'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude', 'passenger_count']]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88e992ee",
   "metadata": {},
   "source": [
    "## 5. Model Training\n",
    "\n",
    "We'll build and train an Artificial Neural Network using scikit-learn's MLPRegressor:\n",
    "- **Architecture**: 3-layer network (128 -> 64 -> 32 neurons)\n",
    "- **Activation**: ReLU activation function\n",
    "- **Optimizer**: Adam solver\n",
    "- **Scaling**: StandardScaler for feature normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d1eca3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select features for model training\n",
    "feature_columns = ['passenger_count', 'pickup_hour', 'pickup_day_of_week', \n",
    "                   'pickup_month', 'pickup_year', 'is_weekend', \n",
    "                   'trip_distance_km', 'abs_lat_diff', 'abs_lon_diff']\n",
    "\n",
    "X = df[feature_columns].astype(float)\n",
    "y = df['fare_amount'].astype(float)\n",
    "\n",
    "# Train-test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "print(f\"Training set size: {X_train.shape[0]}\")\n",
    "print(f\"Test set size: {X_test.shape[0]}\")\n",
    "\n",
    "# Build the ANN model using Pipeline (includes scaling)\n",
    "model = Pipeline([\n",
    "    ('scaler', StandardScaler()),\n",
    "    ('ann', MLPRegressor(\n",
    "        hidden_layer_sizes=(128, 64, 32),\n",
    "        activation='relu',\n",
    "        solver='adam',\n",
    "        alpha=0.0001,\n",
    "        batch_size=1024,\n",
    "        learning_rate_init=0.001,\n",
    "        max_iter=120,\n",
    "        early_stopping=True,\n",
    "        validation_fraction=0.15,\n",
    "        n_iter_no_change=12,\n",
    "        random_state=42,\n",
    "        verbose=False\n",
    "    ))\n",
    "])\n",
    "\n",
    "# Train the model\n",
    "print(\"Training ANN model...\")\n",
    "model.fit(X_train, y_train)\n",
    "print(\"✓ Model training complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2c56bb",
   "metadata": {},
   "source": [
    "## 6. Model Evaluation\n",
    "\n",
    "Let's evaluate the trained model on the test set using standard regression metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ce1852e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions\n",
    "y_pred_train = model.predict(X_train)\n",
    "y_pred_test = model.predict(X_test)\n",
    "\n",
    "# Calculate metrics\n",
    "train_mae = mean_absolute_error(y_train, y_pred_train)\n",
    "test_mae = mean_absolute_error(y_test, y_pred_test)\n",
    "\n",
    "train_mse = mean_squared_error(y_train, y_pred_train)\n",
    "test_mse = mean_squared_error(y_test, y_pred_test)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "train_r2 = r2_score(y_train, y_pred_train)\n",
    "test_r2 = r2_score(y_test, y_pred_test)\n",
    "\n",
    "# Display results\n",
    "print(\"=\" * 50)\n",
    "print(\"MODEL PERFORMANCE METRICS\")\n",
    "print(\"=\" * 50)\n",
    "print(f\"\\nTraining Set:\")\n",
    "print(f\"  MAE:  ${train_mae:.4f}\")\n",
    "print(f\"  RMSE: ${train_rmse:.4f}\")\n",
    "print(f\"  R²:   {train_r2:.4f}\")\n",
    "\n",
    "print(f\"\\nTest Set:\")\n",
    "print(f\"  MAE:  ${test_mae:.4f}\")\n",
    "print(f\"  RMSE: ${test_rmse:.4f}\")\n",
    "print(f\"  R²:   {test_r2:.4f}\")\n",
    "print(\"=\" * 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a64b80b0",
   "metadata": {},
   "source": [
    "## 7. Visualizations\n",
    "\n",
    "Let's visualize the model's predictions versus actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e33c2f3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create visualizations\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Actual vs Predicted (Test Set)\n",
    "axes[0].scatter(y_test, y_pred_test, alpha=0.5, s=10)\n",
    "axes[0].plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2)\n",
    "axes[0].set_xlabel('Actual Fare ($)')\n",
    "axes[0].set_ylabel('Predicted Fare ($)')\n",
    "axes[0].set_title(f'Actual vs Predicted Fare (Test Set)\\nR² = {test_r2:.4f}')\n",
    "axes[0].grid(True, alpha=0.3)\n",
    "\n",
    "# Residuals\n",
    "residuals = y_test - y_pred_test\n",
    "axes[1].scatter(y_pred_test, residuals, alpha=0.5, s=10)\n",
    "axes[1].axhline(y=0, color='r', linestyle='--', lw=2)\n",
    "axes[1].set_xlabel('Predicted Fare ($)')\n",
    "axes[1].set_ylabel('Residuals ($)')\n",
    "axes[1].set_title('Residual Plot')\n",
    "axes[1].grid(True, alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "print(f\"Mean Residual: ${residuals.mean():.4f}\")\n",
    "print(f\"Std Dev Residuals: ${residuals.std():.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98a18f9",
   "metadata": {},
   "source": [
    "## 8. Make Predictions on New Data\n",
    "\n",
    "Let's demonstrate how to make predictions for new taxi trips."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02686bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example prediction for a new trip\n",
    "from datetime import datetime\n",
    "\n",
    "# Define a sample trip\n",
    "sample_trip = pd.DataFrame({\n",
    "    'pickup_datetime': [datetime(2015, 6, 15, 14, 30, 0)],\n",
    "    'pickup_longitude': [-73.985],\n",
    "    'pickup_latitude': [40.748],\n",
    "    'dropoff_longitude': [-73.971],\n",
    "    'dropoff_latitude': [40.783],\n",
    "    'passenger_count': [2]\n",
    "})\n",
    "\n",
    "# Extract features for the sample trip\n",
    "sample_trip['pickup_hour'] = sample_trip['pickup_datetime'].dt.hour\n",
    "sample_trip['pickup_day_of_week'] = sample_trip['pickup_datetime'].dt.dayofweek\n",
    "sample_trip['pickup_month'] = sample_trip['pickup_datetime'].dt.month\n",
    "sample_trip['pickup_year'] = sample_trip['pickup_datetime'].dt.year\n",
    "sample_trip['is_weekend'] = (sample_trip['pickup_datetime'].dt.dayofweek >= 5).astype(int)\n",
    "sample_trip['trip_distance_km'] = haversine_km(\n",
    "    sample_trip['pickup_latitude'], sample_trip['pickup_longitude'],\n",
    "    sample_trip['dropoff_latitude'], sample_trip['dropoff_longitude']\n",
    ")\n",
    "sample_trip['abs_lat_diff'] = (sample_trip['pickup_latitude'] - sample_trip['dropoff_latitude']).abs()\n",
    "sample_trip['abs_lon_diff'] = (sample_trip['pickup_longitude'] - sample_trip['dropoff_longitude']).abs()\n",
    "\n",
    "# Select only the feature columns\n",
    "sample_features = sample_trip[feature_columns].astype(float)\n",
    "\n",
    "# Make prediction\n",
    "predicted_fare = model.predict(sample_features)[0]\n",
    "\n",
    "print(f\"Sample Trip Details:\")\n",
    "print(f\"  Pickup: ({sample_trip['pickup_latitude'].values[0]:.6f}, {sample_trip['pickup_longitude'].values[0]:.6f})\")\n",
    "print(f\"  Dropoff: ({sample_trip['dropoff_latitude'].values[0]:.6f}, {sample_trip['dropoff_longitude'].values[0]:.6f})\")\n",
    "print(f\"  Passengers: {int(sample_trip['passenger_count'].values[0])}\")\n",
    "print(f\"  Distance: {sample_trip['trip_distance_km'].values[0]:.2f} km\")\n",
    "print(f\"\\n✓ Predicted Fare: ${predicted_fare:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39aa37a",
   "metadata": {},
   "source": [
    "## 9. Model Persistence\n",
    "\n",
    "Save the trained model for deployment in a Gradio interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebeba28e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model using joblib\n",
    "import joblib\n",
    "\n",
    "model_path = 'artifacts/taxi_fare_ann_model.joblib'\n",
    "joblib.dump(model, model_path)\n",
    "print(f\"✓ Model saved to: {model_path}\")\n",
    "\n",
    "# Save metrics\n",
    "import json\n",
    "metrics_data = {\n",
    "    'mae': float(test_mae),\n",
    "    'rmse': float(test_rmse),\n",
    "    'r2': float(test_r2)\n",
    "}\n",
    "\n",
    "metrics_path = 'artifacts/metrics.json'\n",
    "with open(metrics_path, 'w') as f:\n",
    "    json.dump(metrics_data, f, indent=2)\n",
    "print(f\"✓ Metrics saved to: {metrics_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d7bfb44",
   "metadata": {},
   "source": [
    "## 10. Deployment\n",
    "\n",
    "The trained model is deployed as an interactive Gradio web application that allows users to:\n",
    "- Input trip details (pickup/dropoff locations, datetime, passenger count)\n",
    "- Get real-time fare predictions\n",
    "- View sample predictions\n",
    "\n",
    "### Running the Gradio App\n",
    "\n",
    "```bash\n",
    "python app.py\n",
    "```\n",
    "\n",
    "The app will be available at `http://localhost:7860`\n",
    "\n",
    "### Deployment to Hugging Face Spaces\n",
    "\n",
    "To deploy on Hugging Face Spaces:\n",
    "1. Create a new Space on Hugging Face (https://huggingface.co/spaces)\n",
    "2. Choose \"Gradio\" as the SDK\n",
    "3. Connect to your GitHub repository\n",
    "4. Hugging Face will automatically run `app.py`\n",
    "\n",
    "For detailed deployment instructions, see the `README.md` file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.14.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}