pythonanalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce179b8c",
   "metadata": {},
   "source": [
    "# 🌍 Notebook 2 — Data Analysis & Visualization\n",
    "**Project:** EM Portfolio Risk Advisor\n",
    "\n",
    "**Research Question:** *How can an emerging market investment fund use news sentiment analysis and macroeconomic forecasting to identify which countries to overweight or underweight during periods of geopolitical stress?*\n",
    "\n",
    "**Team:** Amaryllis (PM) · Kuang (UX) · Tommaso (Data Analyst) · Logan (UX) · Achille (Content)\n",
    "**Course:** AI for Big Data Management — ESCP SE21\n",
    "\n",
    "---\n",
    "**Analyses performed in this notebook:**\n",
    "| # | Type | Method |\n",
    "|---|------|--------|\n",
    "| 1 | Qualitative | VADER sentiment scoring of synthetic analyst reports |\n",
    "| 2 | Quantitative | GDP growth heatmap (2000–2023) |\n",
    "| 3 | Quantitative | Geopolitical risk heatmap |\n",
    "| 4 | Quantitative | FDI & Inflation trend analysis |\n",
    "| 5 | Quantitative | Random Forest investment signal classifier |\n",
    "| 6 | Quantitative | ARIMA GDP growth forecasting (2024–2028) |\n",
    "| 7 | Mixed | Country-level investment signal + sentiment composite |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd142dab",
   "metadata": {},
   "source": [
    "## 1. Install & Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47a823f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q pandas numpy matplotlib seaborn vaderSentiment statsmodels scikit-learn plotly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0717b85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.colors as mcolors\n",
    "import seaborn as sns\n",
    "import warnings, os, json\n",
    "from pathlib import Path\n",
    "from itertools import product\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "np.random.seed(42)\n",
    "\n",
    "print('✅ Packages loaded')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aa18937",
   "metadata": {},
   "source": [
    "## 2. Output Directory Setup\n",
    "\n",
    "All figures and tables are saved to `artifacts/` so the Hugging Face app can load them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89cb932e",
   "metadata": {},
   "outputs": [],
   "source": [
    "BASE_DIR  = Path('.')\n",
    "ART_DIR   = BASE_DIR / 'artifacts'\n",
    "PY_FIG    = ART_DIR / 'py' / 'figures'\n",
    "PY_TAB    = ART_DIR / 'py' / 'tables'\n",
    "\n",
    "for p in [PY_FIG, PY_TAB]:\n",
    "    p.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "print('✅ Output folders:')\n",
    "print(' -', PY_FIG.resolve())\n",
    "print(' -', PY_TAB.resolve())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59fccaec",
   "metadata": {},
   "source": [
    "## 3. Load Datasets\n",
    "\n",
    "All five files were produced by Notebook 1 under the data contract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea5ed4b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_macro     = pd.read_csv('world_bank_macro.csv')\n",
    "df_risk      = pd.read_csv('synthetic_risk_scores.csv')\n",
    "df_sentiment = pd.read_csv('synthetic_news_sentiment.csv')\n",
    "df_master    = pd.read_csv('title_level_features.csv')\n",
    "df_monthly   = pd.read_csv('monthly_gdp_series.csv')\n",
    "\n",
    "print('world_bank_macro      :', df_macro.shape)\n",
    "print('synthetic_risk_scores :', df_risk.shape)\n",
    "print('synthetic_news_sent.  :', df_sentiment.shape)\n",
    "print('title_level_features  :', df_master.shape)\n",
    "print('monthly_gdp_series    :', df_monthly.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b70ed2",
   "metadata": {},
   "source": [
    "## 4. Data Quality Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a068f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def quality_check(df, name):\n",
    "    print(f'\\n🔍 {name}')\n",
    "    print(f'  Shape : {df.shape}')\n",
    "    nulls = df.isnull().sum()\n",
    "    nulls = nulls[nulls > 0]\n",
    "    print(f'  Nulls : {dict(nulls) if len(nulls) else \"none\"}')\n",
    "    print(f'  Dtypes: {dict(df.dtypes)}')\n",
    "    return df\n",
    "\n",
    "for df, nm in [\n",
    "    (df_macro,     'world_bank_macro'),\n",
    "    (df_risk,      'synthetic_risk_scores'),\n",
    "    (df_sentiment, 'synthetic_news_sentiment'),\n",
    "    (df_master,    'title_level_features'),\n",
    "    (df_monthly,   'monthly_gdp_series'),\n",
    "]:\n",
    "    quality_check(df, nm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d974373",
   "metadata": {},
   "source": [
    "## 5. Qualitative Analysis — VADER Sentiment Scoring\n",
    "\n",
    "We apply the **VADER** (Valence Aware Dictionary and sEntiment Reasoner) lexicon to every\n",
    "synthetic analyst report headline.  VADER returns a compound score in **[−1, +1]**:\n",
    "\n",
    "| Range | Label |\n",
    "|-------|-------|\n",
    "| ≥ 0.05 | bullish |\n",
    "| ≤ −0.05 | bearish |\n",
    "| otherwise | neutral |\n",
    "\n",
    "This mirrors how a portfolio analyst would extract sentiment from news feeds programmatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0537f27",
   "metadata": {},
   "outputs": [],
   "source": [
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "\n",
    "analyzer = SentimentIntensityAnalyzer()\n",
    "\n",
    "def vader_compound(text):\n",
    "    return analyzer.polarity_scores(str(text))['compound']\n",
    "\n",
    "def vader_label(score):\n",
    "    if score >= 0.05:  return 'bullish'\n",
    "    if score <= -0.05: return 'bearish'\n",
    "    return 'neutral'\n",
    "\n",
    "df_sentiment['vader_compound'] = df_sentiment['report_text'].apply(vader_compound)\n",
    "df_sentiment['vader_label']    = df_sentiment['vader_compound'].apply(vader_label)\n",
    "\n",
    "print('VADER label distribution:')\n",
    "print(df_sentiment['vader_label'].value_counts())\n",
    "print()\n",
    "print(df_sentiment[['country','year','sentiment_label','vader_compound','vader_label']].head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1822af7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregate VADER scores per country\n",
    "vader_by_country = (\n",
    "    df_sentiment\n",
    "    .groupby(['iso3','country'])\n",
    "    .agg(avg_vader_score=('vader_compound','mean'),\n",
    "         report_count=('vader_compound','count'))\n",
    "    .reset_index()\n",
    "    .sort_values('avg_vader_score')\n",
    ")\n",
    "vader_by_country.to_csv(PY_TAB / 'vader_by_country.csv', index=False)\n",
    "print('✅ vader_by_country.csv saved')\n",
    "print(vader_by_country)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902ea1c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregate VADER scores per country-year (for merging into master)\n",
    "vader_agg = (\n",
    "    df_sentiment\n",
    "    .groupby(['iso3','year'])['vader_compound']\n",
    "    .mean()\n",
    "    .reset_index()\n",
    "    .rename(columns={'vader_compound': 'vader_score'})\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ae2b5ee",
   "metadata": {},
   "source": [
    "## 6. Merge VADER Scores into Master Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bbfc012",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also merge risk scores into master (if not already present)\n",
    "risk_cols = ['iso3','year','geopolitical_risk_score']\n",
    "if 'geopolitical_risk_score' not in df_master.columns:\n",
    "    df_master = df_master.merge(\n",
    "        df_risk[risk_cols], on=['iso3','year'], how='left'\n",
    "    )\n",
    "\n",
    "# Merge VADER\n",
    "if 'vader_score' not in df_master.columns:\n",
    "    df_master = df_master.merge(vader_agg, on=['iso3','year'], how='left')\n",
    "\n",
    "# Clean numerics\n",
    "for col in ['gdp_growth','fdi_pct_gdp','inflation','geopolitical_risk_score','vader_score']:\n",
    "    if col in df_master.columns:\n",
    "        df_master[col] = pd.to_numeric(df_master[col], errors='coerce')\n",
    "\n",
    "print('Master dataset shape after merge:', df_master.shape)\n",
    "print(df_master[['country','year','gdp_growth','geopolitical_risk_score','vader_score']].head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f506667",
   "metadata": {},
   "source": [
    "## 7. Quantitative Analysis — GDP Growth Heatmap\n",
    "\n",
    "A heatmap gives fund managers an at-a-glance view of **which countries experienced growth\n",
    "shocks vs. booms** over the 2000–2023 period.  Red cells correspond to known crises\n",
    "(Argentina 2001–02, global GFC 2009, COVID 2020)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33ff313",
   "metadata": {},
   "outputs": [],
   "source": [
    "PALETTE = ['#28096D','#2ec4a0','#e8537a','#F2C637','#5e8fef',\n",
    "           '#c45ea8','#3dbacc','#a0522d','#6aaa3a','#d46060']\n",
    "\n",
    "# Build pivot\n",
    "pivot_gdp = df_macro.pivot_table(index='country', columns='year', values='gdp_growth')\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(18, 7))\n",
    "sns.heatmap(\n",
    "    pivot_gdp,\n",
    "    cmap='RdYlGn',\n",
    "    center=0,\n",
    "    linewidths=0.4,\n",
    "    linecolor='white',\n",
    "    annot=False,\n",
    "    fmt='.1f',\n",
    "    ax=ax,\n",
    "    cbar_kws={'label': 'GDP Growth (%)', 'shrink': 0.8}\n",
    ")\n",
    "ax.set_title('EM GDP Growth Rate (%) — 2000 to 2023', fontsize=15,\n",
    "             fontweight='bold', color='#28096D', pad=16)\n",
    "ax.set_xlabel('Year', fontsize=11, color='#28096D')\n",
    "ax.set_ylabel('Country', fontsize=11, color='#28096D')\n",
    "ax.tick_params(axis='x', labelsize=8, rotation=45)\n",
    "ax.tick_params(axis='y', labelsize=10)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'gdp_heatmap.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ gdp_heatmap.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01b2971c",
   "metadata": {},
   "source": [
    "## 8. Quantitative Analysis — Geopolitical Risk Heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9dc1711",
   "metadata": {},
   "outputs": [],
   "source": [
    "pivot_risk = df_risk.pivot_table(\n",
    "    index='country', columns='year', values='geopolitical_risk_score'\n",
    ")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(18, 7))\n",
    "sns.heatmap(\n",
    "    pivot_risk,\n",
    "    cmap='YlOrRd',\n",
    "    linewidths=0.4,\n",
    "    linecolor='white',\n",
    "    annot=False,\n",
    "    ax=ax,\n",
    "    cbar_kws={'label': 'Risk Score (0–10)', 'shrink': 0.8}\n",
    ")\n",
    "ax.set_title('Synthetic Geopolitical Risk Score — 2000 to 2023', fontsize=15,\n",
    "             fontweight='bold', color='#28096D', pad=16)\n",
    "ax.set_xlabel('Year', fontsize=11, color='#28096D')\n",
    "ax.set_ylabel('Country', fontsize=11, color='#28096D')\n",
    "ax.tick_params(axis='x', labelsize=8, rotation=45)\n",
    "ax.tick_params(axis='y', labelsize=10)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'geo_risk_heatmap.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ geo_risk_heatmap.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30066fb5",
   "metadata": {},
   "source": [
    "## 9. Qualitative Analysis — VADER Sentiment by Country\n",
    "\n",
    "We compare the average VADER compound score to the ground-truth `sentiment_label` assigned during synthetic data generation to validate alignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03c36db4",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(16, 6))\n",
    "\n",
    "# Left: VADER compound score (bar)\n",
    "colors = ['#2ec4a0' if v >= 0.05 else ('#e8537a' if v <= -0.05 else '#5e8fef')\n",
    "          for v in vader_by_country['avg_vader_score']]\n",
    "axes[0].barh(vader_by_country['country'], vader_by_country['avg_vader_score'],\n",
    "             color=colors, edgecolor='white', linewidth=0.6)\n",
    "axes[0].axvline(0, color='gray', linewidth=0.8, linestyle='--')\n",
    "axes[0].set_title('Average VADER Compound Score by Country', fontweight='bold', color='#28096D')\n",
    "axes[0].set_xlabel('Compound Score (−1 to +1)')\n",
    "axes[0].tick_params(labelsize=10)\n",
    "\n",
    "# Right: Sentiment distribution stacked bar\n",
    "sent_dist = (\n",
    "    df_sentiment.groupby(['country','vader_label'])\n",
    "    .size().unstack(fill_value=0)\n",
    ")\n",
    "for lbl in ['bullish','neutral','bearish']:\n",
    "    if lbl not in sent_dist.columns:\n",
    "        sent_dist[lbl] = 0\n",
    "sent_dist = sent_dist.reindex(columns=['bullish','neutral','bearish'])\n",
    "sent_colors = {'bullish': '#2ec4a0', 'neutral': '#5e8fef', 'bearish': '#e8537a'}\n",
    "sent_dist.plot(kind='barh', stacked=True, ax=axes[1],\n",
    "               color=[sent_colors[c] for c in sent_dist.columns],\n",
    "               edgecolor='white', linewidth=0.4)\n",
    "axes[1].set_title('Report Sentiment Distribution by Country', fontweight='bold', color='#28096D')\n",
    "axes[1].set_xlabel('Number of Reports')\n",
    "axes[1].legend(title='VADER Label', loc='lower right')\n",
    "axes[1].tick_params(labelsize=10)\n",
    "\n",
    "plt.suptitle('Qualitative Analysis: News Sentiment Across Emerging Markets',\n",
    "             fontsize=13, fontweight='bold', color='#28096D', y=1.02)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'vader_sentiment.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ vader_sentiment.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67044680",
   "metadata": {},
   "source": [
    "## 10. Quantitative Analysis — FDI & Inflation Trends"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c624d67d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(16, 6))\n",
    "\n",
    "# FDI line chart\n",
    "for i, (iso3, cname) in enumerate(df_macro.groupby(['iso3','country']).size().index):\n",
    "    sub = df_macro[df_macro['iso3'] == iso3]\n",
    "    axes[0].plot(sub['year'], sub['fdi_pct_gdp'],\n",
    "                 label=cname, color=PALETTE[i % len(PALETTE)],\n",
    "                 linewidth=1.8, alpha=0.85)\n",
    "axes[0].set_title('FDI Inflows (% of GDP)', fontweight='bold', color='#28096D')\n",
    "axes[0].set_xlabel('Year'); axes[0].set_ylabel('% of GDP')\n",
    "axes[0].legend(fontsize=7, ncol=2)\n",
    "axes[0].axhline(0, color='gray', linewidth=0.6, linestyle='--')\n",
    "\n",
    "# Inflation scatter / boxplot by country\n",
    "df_macro_clean = df_macro[df_macro['inflation'].abs() < 200]\n",
    "axes[1].boxplot(\n",
    "    [df_macro_clean[df_macro_clean['country'] == c]['inflation'].dropna()\n",
    "     for c in df_macro_clean['country'].unique()],\n",
    "    labels=df_macro_clean['country'].unique(),\n",
    "    patch_artist=True,\n",
    "    boxprops=dict(facecolor='#a48de8', alpha=0.7),\n",
    "    medianprops=dict(color='#28096D', linewidth=2)\n",
    ")\n",
    "axes[1].set_title('Inflation Distribution by Country', fontweight='bold', color='#28096D')\n",
    "axes[1].set_ylabel('Inflation (%)')\n",
    "axes[1].tick_params(axis='x', rotation=45, labelsize=9)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'fdi_inflation.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ fdi_inflation.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b4bbad",
   "metadata": {},
   "source": [
    "## 11. Quantitative Analysis — Random Forest Investment Signal Classifier\n",
    "\n",
    "We train a **Random Forest** on macro + risk + sentiment features to learn which\n",
    "country-years should be tagged *overweight*, *neutral*, or *underweight*.\n",
    "\n",
    "The investment signal target is defined by a rule:\n",
    "- **Overweight**: GDP growth above median AND geopolitical risk below median\n",
    "- **Underweight**: GDP growth in bottom tercile\n",
    "- **Neutral**: all others\n",
    "\n",
    "This rule is used to create labelled training data.  The RF then learns non-linear\n",
    "feature interactions that a simple rule would miss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbec8543",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "FEATURES = ['gdp_growth', 'fdi_pct_gdp', 'inflation',\n",
    "            'geopolitical_risk_score', 'vader_score']\n",
    "\n",
    "# Build target labels\n",
    "df_ml = df_master.dropna(subset=[c for c in FEATURES if c in df_master.columns]).copy()\n",
    "\n",
    "gdp_med  = df_ml['gdp_growth'].median()\n",
    "risk_med = df_ml['geopolitical_risk_score'].median() if 'geopolitical_risk_score' in df_ml.columns else 5\n",
    "gdp_q33  = df_ml['gdp_growth'].quantile(0.33)\n",
    "\n",
    "def label_signal(row):\n",
    "    try:\n",
    "        if row['gdp_growth'] > gdp_med and row['geopolitical_risk_score'] < risk_med:\n",
    "            return 'overweight'\n",
    "        if row['gdp_growth'] < gdp_q33:\n",
    "            return 'underweight'\n",
    "        return 'neutral'\n",
    "    except Exception:\n",
    "        return 'neutral'\n",
    "\n",
    "df_ml['investment_signal'] = df_ml.apply(label_signal, axis=1)\n",
    "print('Signal distribution:')\n",
    "print(df_ml['investment_signal'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb472edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter to available features\n",
    "avail_features = [f for f in FEATURES if f in df_ml.columns]\n",
    "X = df_ml[avail_features].fillna(df_ml[avail_features].median())\n",
    "y = df_ml['investment_signal']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,\n",
    "                                                     random_state=42, stratify=y)\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators=200, max_depth=8,\n",
    "                             class_weight='balanced', random_state=42)\n",
    "rf.fit(X_train, y_train)\n",
    "y_pred = rf.predict(X_test)\n",
    "\n",
    "print(classification_report(y_test, y_pred))\n",
    "print(f'Accuracy: {accuracy_score(y_test, y_pred):.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "726e9575",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature importance chart\n",
    "feat_imp = pd.DataFrame({\n",
    "    'feature':    avail_features,\n",
    "    'importance': rf.feature_importances_\n",
    "}).sort_values('importance')\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(9, 5))\n",
    "colors_imp = ['#28096D' if f == feat_imp.iloc[-1]['feature'] else '#a48de8'\n",
    "              for f in feat_imp['feature']]\n",
    "ax.barh(feat_imp['feature'], feat_imp['importance'],\n",
    "        color=colors_imp, edgecolor='white')\n",
    "ax.set_title('Random Forest — Feature Importances', fontweight='bold',\n",
    "             color='#28096D', fontsize=13)\n",
    "ax.set_xlabel('Importance (Gini)', fontsize=11)\n",
    "ax.tick_params(labelsize=11)\n",
    "for v, f in zip(feat_imp['importance'], feat_imp['feature']):\n",
    "    ax.text(v + 0.002, f, f'{v:.3f}', va='center', fontsize=10)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'rf_feature_importance.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ rf_feature_importance.png saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "176186e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Investment signal summary\n",
    "df_ml['rf_prediction'] = rf.predict(X)\n",
    "signal_summary = (\n",
    "    df_ml.groupby('country')\n",
    "    .apply(lambda x: (x['rf_prediction'] == 'overweight').mean() * 100)\n",
    "    .reset_index()\n",
    ")\n",
    "signal_summary.columns = ['country', 'pct_overweight']\n",
    "signal_summary = signal_summary.sort_values('pct_overweight', ascending=False)\n",
    "signal_summary.to_csv(PY_TAB / 'investment_signal_summary.csv', index=False)\n",
    "print('✅ investment_signal_summary.csv saved')\n",
    "print(signal_summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fed3baa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Investment signal chart\n",
    "fig, ax = plt.subplots(figsize=(10, 6))\n",
    "colors_sig = ['#2ec4a0' if v >= 50 else '#e8537a'\n",
    "              for v in signal_summary['pct_overweight']]\n",
    "ax.barh(signal_summary['country'], signal_summary['pct_overweight'],\n",
    "        color=colors_sig, edgecolor='white', linewidth=0.5)\n",
    "ax.axvline(50, color='gray', linewidth=1.2, linestyle='--', label='50% threshold')\n",
    "ax.set_title('Investment Signal: % of Years Classified as Overweight',\n",
    "             fontweight='bold', color='#28096D', fontsize=13)\n",
    "ax.set_xlabel('% Years Overweight', fontsize=11)\n",
    "ax.set_xlim(0, 100)\n",
    "for v, c in zip(signal_summary['pct_overweight'], signal_summary['country']):\n",
    "    ax.text(v + 1, c, f'{v:.0f}%', va='center', fontsize=10)\n",
    "ax.legend(fontsize=10)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'investment_signal.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ investment_signal.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9005249",
   "metadata": {},
   "source": [
    "## 12. Quantitative Analysis — ARIMA GDP Growth Forecasting\n",
    "\n",
    "We use **ARIMA** (Auto-Regressive Integrated Moving Average) to forecast the average\n",
    "EM GDP growth rate for 2024–2028.  ARIMA is fit on the *average* monthly series across\n",
    "all 10 countries, as this gives the fund a macro-level view of EM momentum.\n",
    "\n",
    "The `find_best_arima` helper searches across p ∈ [0,3], d ∈ [0,1], q ∈ [0,1] and\n",
    "selects the order minimising **AIC**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44eff2cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.tsa.arima.model import ARIMA\n",
    "\n",
    "def find_best_arima(series, p_range=(0, 3), d_range=(0, 1), q_range=(0, 1)):\n",
    "    best_aic, best_order, best_model = float('inf'), None, None\n",
    "    for p, d, q in product(range(p_range[0], p_range[1] + 1),\n",
    "                            range(d_range[0], d_range[1] + 1),\n",
    "                            range(q_range[0], q_range[1] + 1)):\n",
    "        try:\n",
    "            m = ARIMA(series, order=(p, d, q)).fit()\n",
    "            if m.aic < best_aic:\n",
    "                best_aic, best_order, best_model = m.aic, (p, d, q), m\n",
    "        except Exception:\n",
    "            pass\n",
    "    return best_order, best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ace7aa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare monthly series: average across countries\n",
    "df_monthly['month'] = pd.to_datetime(df_monthly['month'], errors='coerce')\n",
    "monthly_avg = (\n",
    "    df_monthly.dropna(subset=['month'])\n",
    "    .groupby('month')['gdp_growth_monthly']\n",
    "    .mean()\n",
    "    .sort_index()\n",
    ")\n",
    "\n",
    "# Fit best ARIMA\n",
    "best_order, best_fit = find_best_arima(monthly_avg)\n",
    "print(f'Best ARIMA order: {best_order}')\n",
    "print(f'AIC: {best_fit.aic:.2f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab3a37c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forecast 60 months ahead (5 years)\n",
    "n_forecast = 60\n",
    "forecast_result = best_fit.get_forecast(steps=n_forecast)\n",
    "forecast_mean   = forecast_result.predicted_mean\n",
    "forecast_ci     = forecast_result.conf_int()\n",
    "\n",
    "forecast_index = pd.date_range(\n",
    "    start=monthly_avg.index[-1] + pd.DateOffset(months=1),\n",
    "    periods=n_forecast, freq='MS'\n",
    ")\n",
    "forecast_mean.index = forecast_index\n",
    "forecast_ci.index   = forecast_index\n",
    "\n",
    "# Annual roll-up for app\n",
    "arima_annual = (\n",
    "    pd.DataFrame({'forecast': forecast_mean,\n",
    "                  'lower_ci': forecast_ci.iloc[:, 0],\n",
    "                  'upper_ci': forecast_ci.iloc[:, 1]})\n",
    "    .resample('YE').mean()\n",
    ")\n",
    "arima_annual.index = arima_annual.index.year\n",
    "arima_annual.index.name = 'year'\n",
    "arima_annual.reset_index().to_csv(PY_TAB / 'arima_gdp_forecast.csv', index=False)\n",
    "print('✅ arima_gdp_forecast.csv saved')\n",
    "print(arima_annual.round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b8d4a55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot historical + forecast\n",
    "fig, ax = plt.subplots(figsize=(14, 6))\n",
    "\n",
    "# Historical (annual average for readability)\n",
    "hist_annual = monthly_avg.resample('YE').mean()\n",
    "hist_annual.index = hist_annual.index.year\n",
    "ax.plot(hist_annual.index, hist_annual.values,\n",
    "        color='#28096D', linewidth=2.5, marker='o', markersize=5, label='Historical avg')\n",
    "\n",
    "# Forecast\n",
    "ax.plot(arima_annual.index, arima_annual['forecast'],\n",
    "        color='#e8537a', linewidth=2.5, linestyle='--', marker='s', markersize=5, label='ARIMA Forecast')\n",
    "ax.fill_between(arima_annual.index, arima_annual['lower_ci'], arima_annual['upper_ci'],\n",
    "                color='#e8537a', alpha=0.15, label='95% CI')\n",
    "\n",
    "ax.axvline(2023.5, color='gray', linewidth=1, linestyle=':', label='Forecast start')\n",
    "ax.axhline(0, color='black', linewidth=0.7, linestyle='-', alpha=0.4)\n",
    "ax.set_title('ARIMA GDP Growth Forecast — Average EM Basket (2024–2028)',\n",
    "             fontweight='bold', color='#28096D', fontsize=13)\n",
    "ax.set_xlabel('Year', fontsize=11); ax.set_ylabel('Avg GDP Growth (%)', fontsize=11)\n",
    "ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda v, _: f'{v:.1f}%'))\n",
    "ax.legend(fontsize=10)\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'arima_gdp_forecast.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ arima_gdp_forecast.png saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6856f8f5",
   "metadata": {},
   "source": [
    "## 13. Country Predictions Summary Table\n",
    "\n",
    "Final table merging the Random Forest signal, VADER score, and latest macro values into one actionable view for fund managers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07aac448",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Latest year for each country\n",
    "latest = df_ml.sort_values('year').groupby('country').last().reset_index()\n",
    "\n",
    "latest_out = latest[['country', 'iso3', 'year',\n",
    "                      'gdp_growth', 'fdi_pct_gdp', 'inflation',\n",
    "                      'geopolitical_risk_score', 'vader_score',\n",
    "                      'investment_signal', 'rf_prediction']].copy()\n",
    "\n",
    "# Readable labels\n",
    "latest_out['recommendation'] = latest_out['rf_prediction'].map({\n",
    "    'overweight':  'OVERWEIGHT  ✅',\n",
    "    'underweight': 'UNDERWEIGHT 🔻',\n",
    "    'neutral':     'NEUTRAL  ➡️'\n",
    "})\n",
    "\n",
    "latest_out.to_csv(PY_TAB / 'country_predictions_latest.csv', index=False)\n",
    "print('✅ country_predictions_latest.csv saved')\n",
    "print(latest_out[['country','gdp_growth','geopolitical_risk_score',\n",
    "                   'vader_score','recommendation']].to_string(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a01cf38",
   "metadata": {},
   "source": [
    "## 14. Dashboard Data Exports\n",
    "\n",
    "The Hugging Face app (`app.py`) reads `df_dashboard.csv` and `kpis.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1999318c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_dashboard: annual average GDP growth (used by the GDP trend chart in the app)\n",
    "df_dashboard = (\n",
    "    df_macro.groupby('year')['gdp_growth']\n",
    "    .mean()\n",
    "    .reset_index()\n",
    "    .rename(columns={'gdp_growth': 'avg_gdp_growth'})\n",
    ")\n",
    "df_dashboard.to_csv(PY_TAB / 'df_dashboard.csv', index=False)\n",
    "print('✅ df_dashboard.csv saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd440bbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# KPI summary\n",
    "top_ow = signal_summary.iloc[0]['country']\n",
    "top_uw = signal_summary.iloc[-1]['country']\n",
    "\n",
    "kpis = {\n",
    "    'Countries Analysed': int(df_macro['country'].nunique()),\n",
    "    'Years Covered':      f\"{int(df_macro['year'].min())}–{int(df_macro['year'].max())}\",\n",
    "    'Avg GDP Growth':     f\"{df_macro['gdp_growth'].mean():.2f}%\",\n",
    "    'Avg Geo Risk':       f\"{df_risk['geopolitical_risk_score'].mean():.1f}/10\",\n",
    "    'Top Overweight':     str(top_ow),\n",
    "    'Top Underweight':    str(top_uw),\n",
    "    'RF Accuracy':        f\"{accuracy_score(y_test, y_pred)*100:.1f}%\",\n",
    "    'Headlines Analysed': int(len(df_sentiment)),\n",
    "}\n",
    "\n",
    "with open(PY_TAB / 'kpis.json', 'w') as f:\n",
    "    json.dump(kpis, f, indent=2)\n",
    "\n",
    "print('✅ kpis.json saved')\n",
    "print(json.dumps(kpis, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00a6bb3e",
   "metadata": {},
   "source": [
    "## 15. Final Summary — 4-Panel Overview Chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59a6cc07",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 2, figsize=(16, 12))\n",
    "fig.suptitle('EM Portfolio Risk Advisor — Analysis Overview',\n",
    "             fontsize=16, fontweight='bold', color='#28096D')\n",
    "\n",
    "# Panel A: Average GDP growth by country\n",
    "avg_gdp = df_macro.groupby('country')['gdp_growth'].mean().sort_values()\n",
    "clrs = ['#e8537a' if v < 0 else '#2ec4a0' for v in avg_gdp]\n",
    "axes[0,0].barh(avg_gdp.index, avg_gdp.values, color=clrs, edgecolor='white')\n",
    "axes[0,0].axvline(0, color='gray', linewidth=0.8, linestyle='--')\n",
    "axes[0,0].set_title('A. Avg GDP Growth 2000–2023', fontweight='bold', color='#28096D')\n",
    "axes[0,0].set_xlabel('% per year')\n",
    "\n",
    "# Panel B: Average geo risk by country\n",
    "avg_risk = df_risk.groupby('country')['geopolitical_risk_score'].mean().sort_values()\n",
    "axes[0,1].barh(avg_risk.index, avg_risk.values,\n",
    "               color='#F2C637', edgecolor='white')\n",
    "axes[0,1].set_title('B. Avg Geopolitical Risk Score', fontweight='bold', color='#28096D')\n",
    "axes[0,1].set_xlabel('Score (0–10)')\n",
    "\n",
    "# Panel C: VADER compound score\n",
    "axes[1,0].barh(vader_by_country['country'], vader_by_country['avg_vader_score'],\n",
    "               color=['#2ec4a0' if v>=0.05 else('#e8537a' if v<=-0.05 else '#5e8fef')\n",
    "                      for v in vader_by_country['avg_vader_score']],\n",
    "               edgecolor='white')\n",
    "axes[1,0].axvline(0, color='gray', linewidth=0.8, linestyle='--')\n",
    "axes[1,0].set_title('C. Avg VADER Sentiment Score', fontweight='bold', color='#28096D')\n",
    "axes[1,0].set_xlabel('Compound Score')\n",
    "\n",
    "# Panel D: % years overweight (RF signal)\n",
    "sig_sorted = signal_summary.sort_values('pct_overweight')\n",
    "axes[1,1].barh(sig_sorted['country'], sig_sorted['pct_overweight'],\n",
    "               color=['#2ec4a0' if v>=50 else '#e8537a' for v in sig_sorted['pct_overweight']],\n",
    "               edgecolor='white')\n",
    "axes[1,1].axvline(50, color='gray', linewidth=1, linestyle='--')\n",
    "axes[1,1].set_title('D. RF Investment Signal (% yrs Overweight)', fontweight='bold', color='#28096D')\n",
    "axes[1,1].set_xlabel('% Years')\n",
    "axes[1,1].set_xlim(0, 100)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(PY_FIG / 'analysis_overview.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✅ analysis_overview.png saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a4810ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "print()\n",
    "print('=' * 55)\n",
    "print('  NOTEBOOK 2 COMPLETE — EM PORTFOLIO RISK ADVISOR')\n",
    "print('=' * 55)\n",
    "\n",
    "import os\n",
    "figures = list(PY_FIG.glob('*.png'))\n",
    "tables  = list(PY_TAB.glob('*.csv')) + list(PY_TAB.glob('*.json'))\n",
    "\n",
    "print(f'  Figures saved : {len(figures)}')\n",
    "for f in sorted(figures): print(f'    • {f.name}')\n",
    "print(f'  Tables saved  : {len(tables)}')\n",
    "for t in sorted(tables):  print(f'    • {t.name}')\n",
    "print()\n",
    "print('  Handoff to Hugging Face Space app.py ✅')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}