Upload folder using huggingface_hub

README.md

@@ -1,4 +1,4 @@
----
+---
 title: Roger Intelligence Platform
 emoji: ⚡
 colorFrom: blue
@@ -28,11 +28,19 @@ A multi-agent AI system that aggregates intelligence from 47+ data sources to pr
 - Social Media Monitor (Reddit, Twitter, Facebook, Threads, BlueSky)
 - Political Intelligence (Gazette, Parliament, District Social Media)
 - Economic Analysis (CSE Stock Market + Technical Indicators)
-- Meteorological Alerts (DMC Weather + RiverNet + **FloodWatch Integration** 🆕)
+- Meteorological Alerts (DMC Weather + RiverNet + **FloodWatch Integration**)
 - Intelligence Agent (Brand Monitoring + Threat Detection + **User-Configurable Targets**)
 - Data Retrieval Orchestrator (Web Scraping)
 - Vectorization Agent (Multilingual BERT Embeddings + Anomaly Detection)
 
+✅ **Situational Awareness Dashboard**:
+- **CEB Power Status** - Load shedding / power outage monitoring
+- **Fuel Prices** - Petrol 92/95, Diesel, Kerosene (CEYPETCO)
+- **CBSL Economic Indicators** - Inflation, policy rates, forex reserves, USD/LKR
+- **Health Alerts** - Dengue case tracking, disease outbreak monitoring
+- **Commodity Prices** - 15 essential goods (rice, sugar, gas, eggs, etc.)
+- **Water Supply Status** - NWSDB disruption alerts
+
 ✅ **ML Anomaly Detection Pipeline** (Integrated into Graph):
 - Language-specific BERT models (Sinhala, Tamil, English)
 - Real-time anomaly inference on every graph cycle
@@ -40,31 +48,31 @@ A multi-agent AI system that aggregates intelligence from 47+ data sources to pr
 - Anomaly Detection (Isolation Forest, LOF)
 - MLflow + DagsHub tracking
 
-✅ **Weather Prediction ML Pipeline** 🆕:
+✅ **Weather Prediction ML Pipeline**:
 - LSTM Neural Network (30-day sequences)
 - Predicts: Temperature, Rainfall, Flood Risk, Severity
 - 21 weather stations → 25 districts
 - Airflow DAG runs daily at 4 AM
 
-✅ **Currency Prediction ML Pipeline** 🆕:
+✅ **Currency Prediction ML Pipeline**:
 - GRU Neural Network (optimized for 8GB RAM)
 - Predicts: USD/LKR exchange rate
 - Features: Technical indicators + CSE + Gold + Oil + USD Index
 - MLflow tracking + Airflow DAG at 4 AM
 
-✅ **Stock Price Prediction ML Pipeline** 🆕:
+✅ **Stock Price Prediction ML Pipeline**:
 - Multi-Architecture: LSTM, GRU, BiLSTM, BiGRU
 - Optuna hyperparameter tuning (30 trials per stock)
 - Per-stock best model selection
 - 10 top CSE stocks (JKH, COMB, DIAL, HNB, etc.)
 
-✅ **RAG-Powered Chatbot** 🆕:
+✅ **RAG-Powered Chatbot**:
 - Chat-history aware Q&A
 - Queries all ChromaDB intelligence collections
 - Domain filtering (political, economic, weather, social)
 - Floating chat UI in dashboard
 
-✅ **Trending/Velocity Detection** 🆕:
+✅ **Trending/Velocity Detection**:
 - SQLite-based topic frequency tracking (24-hour rolling window)
 - Momentum calculation: `current_hour / avg_last_6_hours`
 - Spike alerts when topic volume > 3x baseline
@@ -74,29 +82,29 @@ A multi-agent AI system that aggregates intelligence from 47+ data sources to pr
 - Live Intelligence Feed
 - Floating AI Chatbox
 - Weather Predictions Tab
-- **Live Satellite/Weather Map** (Windy.com) 🆕
-- **National Flood Threat Score** 🆕
-- **30-Year Historical Climate Analysis** 🆕
-- **Trending Topics & Spike Alerts** 🆕
-- **Enhanced Operational Indicators** 🆕 (infrastructure_health, regulatory_activity, investment_climate)
+- **Live Satellite/Weather Map** (Windy.com)
+- **National Flood Threat Score**
+- **30-Year Historical Climate Analysis**
+- **Trending Topics & Spike Alerts**
+- **Enhanced Operational Indicators** (infrastructure_health, regulatory_activity, investment_climate)
 - Operational Risk Radar
 - ML Anomaly Detection Display
 - Market Predictions with Moving Averages
 - Risk & Opportunity Classification
 
-✅ **Weather Data Scraper for ML Training** 🆕:
+✅ **Weather Data Scraper for ML Training**:
 - Open-Meteo API (free historical data)
 - NASA FIRMS (fire/heat detection)
 - All 25 districts coverage
 - Year-wise CSV export for model training
 
-✅ **Operational Dashboard Metrics** 🆕:
+✅ **Operational Dashboard Metrics**:
 - **Logistics Friction**: Average confidence of mobility/social domain risk events
 - **Compliance Volatility**: Average confidence of political domain risks
 - **Market Instability**: Average confidence of market/economical domain risks
 - **Opportunity Index**: Average confidence of opportunity-classified events
 
-✅ **Multi-District Province-Aware Event Categorization** 🆕:
+✅ **Multi-District Province-Aware Event Categorization**:
 - Events mentioning provinces are displayed in all constituent districts
 - Supports: Western, Southern, Central, Northern, Eastern, Sabaragamuwa, Uva, North Western, North Central provinces
 - Both frontend (MapView, DistrictInfoPanel) and backend are synchronized
@@ -126,7 +134,7 @@ A multi-agent AI system that aggregates intelligence from 47+ data sources to pr
 │                    │  (Rank & Dedupe)    │                              │
 │                    └─────────┬──────────┘                              │
 │                    ┌─────────▼──────────┐                              │
-│                    │  Vectorization     │ ← NEW                         │
+│                    │  Vectorization     │                          │
 │                    │  Agent (Optional)  │                              │
 │                    └─────────┬──────────┘                              │
 │                    ┌─────────▼──────────┐                              │
@@ -168,7 +176,7 @@ graph TD
 - Loop control with configurable intervals
 - Real-time WebSocket broadcasting
 
-**Architecture Improvements (v2.1):** 🆕
+**Architecture Improvements (v2.1):**
 - **Rate Limiting**: Domain-specific rate limits prevent anti-bot detection
   - Twitter: 15 RPM, LinkedIn: 10 RPM, News: 60 RPM
   - Thread-safe semaphores for max concurrent requests
@@ -226,7 +234,7 @@ graph TD
 ---
 
 ### 4. Meteorological Agent Graph (`meteorologicalAgentGraph.py`)
-**Weather & Disaster Monitoring + FloodWatch Integration** 🆕
+**Weather & Disaster Monitoring + FloodWatch Integration**
 
 ```
 ┌─────────────────────────────────────┐
@@ -242,13 +250,13 @@ graph TD
               │
               ▼
 ┌─────────────────────────────────────┐
-│    FloodWatch Historical Data 🆕    │
+│    FloodWatch Historical Data    │
 │   (30-year climate analysis)        │
 └─────────────┬───────────────────────┘
               │
               ▼
 ┌─────────────────────────────────────┐
-│    National Threat Calculator 🆕    │
+│    National Threat Calculator    │
 │   (Aggregated flood risk 0-100)     │
 └─────────────┬───────────────────────┘
               │
@@ -265,7 +273,7 @@ graph TD
 - 🟠 Warning: Take precautions
 - 🔴 Critical: Immediate action required
 
-**FloodWatch Features (New):**
+**FloodWatch Features:**
 | Feature | Description |
 |---------|-------------|
 | **Historical Analysis** | 30-year climate data (1995-2025) |
@@ -289,11 +297,11 @@ graph TD
 ---
 
 ### 6. Intelligence Agent Graph (`intelligenceAgentGraph.py`)
-**Brand & Threat Monitoring + User-Configurable Targets** 🆕
+**Brand & Threat Monitoring + User-Configurable Targets**
 
 ```
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
-│ Brand Monitor   │    │ Threat Scanner  │    │ User Targets 🆕 │
+│ Brand Monitor   │    │ Threat Scanner  │    │ User Targets │
 │ - Company news  │    │ - Security      │    │ - Custom keys   │
 │ - Competitor    │    │ - Compliance    │    │ - User profiles │
 │ - Market share  │    │ - Geopolitical  │    │ - Products      │
@@ -307,7 +315,7 @@ graph TD
                    └─────────────────────┘
 ```
 
-**User-Configurable Monitoring** 🆕:
+**User-Configurable Monitoring**:
 Users can define custom monitoring targets via the frontend settings panel or API:
 
 | Config Type | Description | Example |
@@ -353,7 +361,7 @@ DELETE /api/intel/config/remove?target_type=profile&value=CompetitorX&platform=t
 
 ---
 
-### 8. Vectorization Agent Graph (`vectorizationAgentGraph.py`) 🆕
+### 8. Vectorization Agent Graph (`vectorizationAgentGraph.py`)
 **6-Step Multilingual NLP Pipeline with Anomaly + Trending Detection**
 
 ```
@@ -383,7 +391,7 @@ DELETE /api/intel/config/remove?target_type=profile&value=CompetitorX&platform=t
                   │
                   ▼
 ┌─────────────────────────────────────────────────┐
-│ Step 4: Trending Detection 🆕                    │
+│ Step 4: Trending Detection                    │
 │ - Entity extraction (hashtags, proper nouns)    │
 │ - Momentum: current_hour / avg_last_6_hours     │
 │ - Spike alerts when momentum > 3x               │
@@ -413,7 +421,7 @@ DELETE /api/intel/config/remove?target_type=profile&value=CompetitorX&platform=t
 
 ---
 
-### 10. Weather Prediction Pipeline (`models/weather-prediction/`) 🆕
+### 10. Weather Prediction Pipeline (`models/weather-prediction/`)
 **LSTM-Based Multi-District Weather Forecasting**
 
 ```
@@ -464,7 +472,7 @@ python main.py --mode train --station COLOMBO
 
 ---
 
-### 11. Currency Prediction Pipeline (`models/currency-volatility-prediction/`) 🆕
+### 11. Currency Prediction Pipeline (`models/currency-volatility-prediction/`)
 **GRU-Based USD/LKR Exchange Rate Forecasting**
 
 ```
@@ -599,7 +607,7 @@ mlflow.sklearn.log_model(model, "model")
 
 ---
 
-## 🌧️ Weather Data Scraper (`scripts/scrape_weather_data.py`) 🆕
+## 🌧️ Weather Data Scraper (`scripts/scrape_weather_data.py`)
 
 **Historical weather data collection for ML model training**
 
@@ -713,8 +721,8 @@ python models/anomaly-detection/download_models.py
 | `/api/weather/predictions` | GET | All district forecasts |
 | `/api/weather/predictions/{district}` | GET | Single district |
 | `/api/weather/model/status` | GET | Weather model info |
-| `/api/weather/historical` | GET | 30-year climate analysis 🆕 |
-| `/api/weather/threat` | GET | National flood threat score 🆕 |
+| `/api/weather/historical` | GET | 30-year climate analysis |
+| `/api/weather/threat` | GET | National flood threat score |
 | `/api/currency/prediction` | GET | USD/LKR next-day forecast |
 | `/api/currency/history` | GET | Historical rates |
 | `/api/currency/model/status` | GET | Currency model info |
@@ -724,6 +732,12 @@ python models/anomaly-detection/download_models.py
 | `/api/rag/chat` | POST | Chat with RAG |
 | `/api/rag/stats` | GET | RAG system stats |
 | `/api/rag/clear` | POST | Clear chat history |
+| `/api/power` | GET | CEB power/load shedding status |
+| `/api/fuel` | GET | Current fuel prices |
+| `/api/economy` | GET | CBSL economic indicators |
+| `/api/health` | GET | Health alerts & dengue data |
+| `/api/commodities` | GET | Essential goods prices |
+| `/api/water` | GET | Water supply disruptions |
 
 ### WebSocket
 - `ws://localhost:8000/ws` - Real-time updates
@@ -750,7 +764,7 @@ astro dev start
 # Open http://localhost:8080
 ```
 
-### DAG: `weather_prediction_daily` 🆕
+### DAG: `weather_prediction_daily`
 
 ```
 ingest_data → train_models → generate_predictions → publish_predictions
@@ -764,7 +778,7 @@ ingest_data → train_models → generate_predictions → publish_predictions
 - Generate 25-district predictions
 - Save to JSON for API
 
-### DAG: `currency_prediction_daily` 🆕
+### DAG: `currency_prediction_daily`
 
 ```
 ingest_data → train_model → generate_prediction → publish_prediction
@@ -800,9 +814,9 @@ Roger-Ultimate/
 │   ├── storage/                   # ChromaDB, SQLite, Neo4j stores
 │   ├── rag.py                     # RAG chatbot
 │   └── utils/
-│       └── utils.py               # Tools incl. FloodWatch 🆕
+│       └── utils.py               # Tools incl. FloodWatch
 ├── scripts/
-│   └── scrape_weather_data.py     # Weather data scraper 🆕
+│   └── scrape_weather_data.py     # Weather data scraper
 ├── models/
 │   ├── anomaly-detection/         # ML Anomaly Pipeline
 │   │   ├── src/
@@ -825,7 +839,7 @@ Roger-Ultimate/
 │       ├── artifacts/             # Trained GRU model
 │       └── main.py                # CLI entry point
 ├── datasets/
-│   └── weather/                   # Scraped weather CSVs 🆕
+│   └── weather/                   # Scraped weather CSVs
 ├── frontend/
 │   └── app/
 │       ├── components/
@@ -833,16 +847,16 @@ Roger-Ultimate/
 │       │   │   ├── AnomalyDetection.tsx
 │       │   │   ├── WeatherPredictions.tsx
 │       │   │   ├── CurrencyPrediction.tsx
-│       │   │   ├── NationalThreatCard.tsx     # Flood threat score 🆕
-│       │   │   ├── HistoricalIntel.tsx        # 30-year climate 🆕
+│       │   │   ├── NationalThreatCard.tsx     # Flood threat score
+│       │   │   ├── HistoricalIntel.tsx        # 30-year climate
 │       │   │   └── ...
 │       │   ├── map/
 │       │   │   ├── MapView.tsx
-│       │   │   └── SatelliteView.tsx          # Windy.com embed 🆕
+│       │   │   └── SatelliteView.tsx          # Windy.com embed
 │       │   ├── FloatingChatBox.tsx            # RAG chat UI
 │       │   └── ...
 │       └── pages/
-│           └── Index.tsx                       # 7 tabs incl. SATELLITE 🆕
+│           └── Index.tsx                       # 7 tabs incl. SATELLITE
 ├── main.py                        # FastAPI backend
 ├── start.sh                       # Startup script
 └── requirements.txt
@@ -1034,3 +1048,4 @@ MIT License - Built for Production
 - **Optuna** - Hyperparameter optimization
 - **MLflow** - Experiment tracking
 - Sri Lankan government for open data sources
+

main.py

@@ -495,6 +495,12 @@ def run_graph_loop():
                                         main_event_loop
                                     )
 
+        except RuntimeError as e:
+            if "cannot schedule new futures after interpreter shutdown" in str(e):
+                logger.warning("[GRAPH THREAD] Interpreter shutting down, stopping graph loop gracefully")
+                break  # Exit the loop cleanly
+            else:
+                logger.error(f"[GRAPH THREAD] RuntimeError in cycle #{cycle_count}: {e}", exc_info=True)
         except Exception as e:
             logger.error(f"[GRAPH THREAD] Error in cycle #{cycle_count}: {e}", exc_info=True)
 
@@ -508,6 +514,8 @@ def run_graph_loop():
             logger.info(f"[GRAPH THREAD] Waiting {wait_time:.1f}s before next cycle...")
             # Use Event.wait() for interruptible sleep instead of time.sleep()
             shutdown_event.wait(timeout=wait_time)
+    
+    logger.info("[GRAPH THREAD] Graph loop stopped")
 
 
 
@@ -1676,6 +1684,7 @@ def get_weather_predictor():
     try:
         import importlib.util
         from pathlib import Path
+        import json
 
         # Use importlib.util for fully isolated import (avoids package collisions)
         weather_src = Path(__file__).parent / "models" / "weather-prediction" / "src"
@@ -1685,39 +1694,52 @@ def get_weather_predictor():
             logger.error(f"[WeatherAPI] predictor.py not found at {predictor_path}")
             return None
 
-        # First, ensure entity module is loadable
-        entity_path = weather_src / "entity" / "config_entity.py"
-        if entity_path.exists():
-            entity_spec = importlib.util.spec_from_file_location(
-                "weather_config_entity",
-                str(entity_path)
-            )
-            entity_module = importlib.util.module_from_spec(entity_spec)
-            sys.modules["weather_config_entity"] = entity_module
-            entity_spec.loader.exec_module(entity_module)
-
-        # Add weather src to path temporarily for relative imports
-        # sys is already imported at top of file
+        # CRITICAL: Remove any conflicting paths (currency-volatility-prediction/src)
+        # to avoid entity.config_entity collision
+        currency_src = str(Path(__file__).parent / "models" / "currency-volatility-prediction" / "src")
+        stock_src = str(Path(__file__).parent / "models" / "stock-price-prediction" / "src")
+        anomaly_src = str(Path(__file__).parent / "models" / "anomaly-detection" / "src")
+        
+        original_path = sys.path.copy()
+        sys.path = [p for p in sys.path if currency_src not in p and stock_src not in p and anomaly_src not in p]
+        
+        # CRITICAL: Clear cached entity modules that may have been imported from wrong path
+        modules_to_clear = [k for k in sys.modules.keys() if 'entity' in k.lower() or 'config_entity' in k.lower()]
+        saved_modules = {}
+        for mod_name in modules_to_clear:
+            saved_modules[mod_name] = sys.modules.pop(mod_name, None)
+        
+        # Add weather src to path FIRST for relative imports
         weather_src_str = str(weather_src)
         if weather_src_str not in sys.path:
             sys.path.insert(0, weather_src_str)
 
-        # Now load predictor module
-        spec = importlib.util.spec_from_file_location(
-            "weather_predictor_module",
-            str(predictor_path)
-        )
-        module = importlib.util.module_from_spec(spec)
-        spec.loader.exec_module(module)
+        try:
+            # Now load predictor module
+            spec = importlib.util.spec_from_file_location(
+                "weather_predictor_module",
+                str(predictor_path)
+            )
+            module = importlib.util.module_from_spec(spec)
+            spec.loader.exec_module(module)
+
+            _weather_predictor = module.WeatherPredictor()
+            logger.info("[WeatherAPI] ✓ Weather predictor initialized via isolated import")
+        finally:
+            # Restore original path
+            sys.path = original_path
+            # Restore saved modules (to avoid breaking other parts of the system)
+            for mod_name, mod in saved_modules.items():
+                if mod is not None:
+                    sys.modules[mod_name] = mod
 
-        _weather_predictor = module.WeatherPredictor()
-        logger.info("[WeatherAPI] ✓ Weather predictor initialized via isolated import")
         return _weather_predictor
 
+
     except Exception as e:
         logger.error(f"[WeatherAPI] Failed to initialize predictor: {e}")
         import traceback
-        logger.debug(traceback.format_exc())
+        logger.error(f"[WeatherAPI] Full traceback:\n{traceback.format_exc()}")
         return None
 
 

models/currency-volatility-prediction/src/exception/exception.py

@@ -1,5 +1,5 @@
 import sys
-from src.logging import logger
+from src.log_utils import logger
 
 class NetworkSecurityException(Exception):
     def __init__(self,error_message,error_details:sys):

models/currency-volatility-prediction/src/log_utils/logger.py

@@ -0,0 +1,20 @@
+import logging
+import os
+from datetime import datetime
+
+LOG_FILE=f"{datetime.now().strftime('%m_%d_%Y_%H_%M_%S')}.log"
+
+logs_path=os.path.join(os.getcwd(), "logs", LOG_FILE)
+
+os.makedirs(logs_path, exist_ok=True)
+# Create the file only if it is not created
+
+LOG_FILE_PATH=os.path.join(logs_path, LOG_FILE)
+
+logging.basicConfig(
+    filename=LOG_FILE_PATH,
+    format="[ %(asctime)s ] %(lineno)d %(name)s - %(levelname)s - %(message)s",
+    level=logging.INFO
+)
+
+

models/weather-prediction/src/exception/exception.py

@@ -1,5 +1,5 @@
 import sys
-from src.logging import logger
+from src.log_utils import logger
 
 class NetworkSecurityException(Exception):
     def __init__(self,error_message,error_details:sys):

models/weather-prediction/src/log_utils/logger.py

@@ -0,0 +1,20 @@
+import logging
+import os
+from datetime import datetime
+
+LOG_FILE=f"{datetime.now().strftime('%m_%d_%Y_%H_%M_%S')}.log"
+
+logs_path=os.path.join(os.getcwd(), "logs", LOG_FILE)
+
+os.makedirs(logs_path, exist_ok=True)
+# Create the file only if it is not created
+
+LOG_FILE_PATH=os.path.join(logs_path, LOG_FILE)
+
+logging.basicConfig(
+    filename=LOG_FILE_PATH,
+    format="[ %(asctime)s ] %(lineno)d %(name)s - %(levelname)s - %(message)s",
+    level=logging.INFO
+)
+
+

pyproject.toml

@@ -55,6 +55,7 @@ dependencies = [
     "sentence-transformers>=5.1.2",
     "streamlit>=1.28.0",
     "tensorflow>=2.20.0",
+    "tf-keras>=2.20.1",
     "torch>=2.9.1",
     "tqdm>=4.67.1",
     "transformers>=4.57.3",

requirements.txt

@@ -138,4 +138,5 @@ python-multipart
 yfinance
 tensorflow
 streamlit
-plotly
+plotly
+tf_keras

uv.lock

@@ -3391,6 +3391,7 @@ dependencies = [
     { name = "sentence-transformers" },
     { name = "streamlit" },
     { name = "tensorflow" },
+    { name = "tf-keras" },
     { name = "torch" },
     { name = "tqdm" },
     { name = "transformers" },
@@ -3453,6 +3454,7 @@ requires-dist = [
     { name = "sentence-transformers", specifier = ">=5.1.2" },
     { name = "streamlit", specifier = ">=1.28.0" },
     { name = "tensorflow", specifier = ">=2.20.0" },
+    { name = "tf-keras", specifier = ">=2.20.1" },
     { name = "torch", specifier = ">=2.9.1" },
     { name = "tqdm", specifier = ">=4.67.1" },
     { name = "transformers", specifier = ">=4.57.3" },
@@ -6283,6 +6285,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/f9/d5/141f53d7c1eb2a80e6d3e9a390228c3222c27705cbe7f048d3623053f3ca/termcolor-3.2.0-py3-none-any.whl", hash = "sha256:a10343879eba4da819353c55cb8049b0933890c2ebf9ad5d3ecd2bb32ea96ea6", size = 7698, upload-time = "2025-10-25T19:11:41.536Z" },
 ]
 
+[[package]]
+name = "tf-keras"
+version = "2.20.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "tensorflow" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/38/6060f6c7472439bb3890b9094d69d31d9f8d5da123b16c738773e70fff91/tf_keras-2.20.1.tar.gz", hash = "sha256:884be5938fb0b2b53b1583c1ae2b660ef87215377c29b5b6a77fd221b472aeaf", size = 1254487, upload-time = "2025-09-04T21:23:41.81Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/85/6b/d9a8202bfe5c9e3b078cf550bafab962aa9d6b1a1f1180f0065399d4c9b2/tf_keras-2.20.1-py3-none-any.whl", hash = "sha256:3f0e0a34d9a4c8758f24fdc1053e6e335f16ab5534c7d34f1899b8924779760c", size = 1694335, upload-time = "2025-09-04T21:23:40.153Z" },
+]
+
 [[package]]
 name = "threadpoolctl"
 version = "3.6.0"