feat: implement key-less FX service with native CBSL web scraping and fallback cache logic

README.md

@@ -21,9 +21,9 @@ FlyRates is a high-availability Telegram bot designed for real-time currency exc
 - **Scheduled Background Tasks:** Powered by APScheduler for reliable background task execution.
 - **Database Flexibility:** Uses SQLAlchemy (supports both SQLite and PostgreSQL).
 
-## Self-Healing Multi-Tier FX Service & CBSL Web Scraper 💱
+## Key-less Web-Scraping FX Service & CBSL Web Scraper 💱
 
-To permanently resolve external API rate-limiting errors (`status 429`) and guarantee 100% uptime for rate tracking, FlyRates incorporates a custom **Self-Healing Multi-Tier FX Rates Architecture**:
+To permanently resolve external API rate-limiting errors (`status 429`) and guarantee 100% uptime for rate tracking, FlyRates incorporates a custom **100% Web-Scraping FX Rates Architecture**:
 
 1. **Official CBSL Web Scraper (Preferred Source for LKR):**
    - **How it Works:** Scrapes daily exchange rates directly from the official Central Bank of Sri Lanka website iframe endpoint (`/cbsl_custom/exrates/exrates.php`) using Python's built-in, lightweight `html.parser.HTMLParser`.
@@ -32,19 +32,17 @@ To permanently resolve external API rate-limiting errors (`status 429`) and guar
 
 2. **Persistent Disk Caching (`rate_cache.json`):**
    - Saves exchange rates to a local `rate_cache.json` file in the workspace root, preserving the cache across container redeployments and restarts.
-   - Using a **1-hour TTL**, it completely bypasses API limits, serving subsequent dashboard and bot queries instantaneously in **under 0.03 milliseconds** from memory.
+   - Using a **1-hour TTL**, it completely bypasses external dependencies, serving subsequent dashboard and bot queries instantaneously in **under 0.03 milliseconds** from memory.
 
 3. **Bridge-Rate Derivation:**
    - Cross-currency rates (e.g. `USD` to `EUR`) that do not directly involve LKR are mathematically derived on the fly using LKR as a bridge from the official CBSL rates:
      $$\text{rate}(\text{USD} \rightarrow \text{EUR}) = \frac{\text{rate}(\text{USD} \rightarrow \text{LKR})}{\text{rate}(\text{EUR} \rightarrow \text{LKR})}$$
-   - This makes the entire bot and dashboard **100% functional and key-less**!
+   - This makes the entire bot and dashboard **100% functional, key-less, and free**!
 
-4. **Self-Healing Fallback Hierarchy:**
+4. **Robust Fallback Hierarchy:**
    - **Tier 1:** Valid Cache (Memory/Disk `rate_cache.json`)
-   - **Tier 2:** Official CBSL Scraper (for LKR-based pairs)
-   - **Tier 3:** Custom ExchangeRate-API (Premium v6 key, if set in `.env`)
-   - **Tier 4:** Key-less Free Fallback API (`open.er-api.com`)
-   - **Tier 5:** Stale Cache Fallback (serves expired cached rates if all networks/APIs are down)
+   - **Tier 2:** Official CBSL Scraper (real-time scraping and dynamic bridging)
+   - **Tier 3:** Stale Cache Fallback (serves expired cached rates or mathematically derives stale bridge rates if CBSL is unreachable)
 
 5. **Asynchronous Concurrency:**
    - The `/api/stats` endpoint uses `asyncio.gather` to retrieve live exchange rates concurrently, boosting cold-cache performance by over **5x**.

main.py

@@ -58,7 +58,7 @@ async def lifespan(app: FastAPI):
     diagnostic_results.append("Starting resolution & connection diagnostics...")
     
     # Check DNS resolution
-    for host in ["api.telegram.org", "v6.exchangerate-api.com"]:
+    for host in ["api.telegram.org", "www.cbsl.gov.lk"]:
         try:
             ips = socket.getaddrinfo(host, 443)
             diagnostic_results.append(f"DNS Resolution for {host}: {ips}")

rate_cache.json

@@ -1,82 +1,82 @@
 {
     "USD_LKR": {
         "rate": 324.7184,
-        "timestamp": 1779032559.9410503
+        "timestamp": 1779042576.8926218
     },
     "LKR_USD": {
         "rate": 0.003079591424446536,
-        "timestamp": 1779032559.9410558
+        "timestamp": 1779042576.8926272
     },
     "EUR_LKR": {
         "rate": 378.3294,
-        "timestamp": 1779032559.94107
+        "timestamp": 1779042576.8926556
     },
     "LKR_EUR": {
         "rate": 0.0026431992861247365,
-        "timestamp": 1779032559.9410727
+        "timestamp": 1779042576.8926578
     },
     "GBP_LKR": {
         "rate": 434.116,
-        "timestamp": 1779032559.9410799
+        "timestamp": 1779042576.8926651
     },
     "LKR_GBP": {
         "rate": 0.0023035317749173032,
-        "timestamp": 1779032559.941081
+        "timestamp": 1779042576.8926666
     },
     "AUD_LKR": {
         "rate": 233.4401,
-        "timestamp": 1779032559.9410868
+        "timestamp": 1779042576.8926735
     },
     "LKR_AUD": {
         "rate": 0.0042837541622026375,
-        "timestamp": 1779032559.941088
+        "timestamp": 1779042576.892675
     },
     "JPY_LKR": {
         "rate": 2.0486,
-        "timestamp": 1779032559.9410925
+        "timestamp": 1779042576.892681
     },
     "LKR_JPY": {
         "rate": 0.48813824074978035,
-        "timestamp": 1779032559.9410937
+        "timestamp": 1779042576.8926826
     },
     "AED_LKR": {
         "rate": 88.4045,
-        "timestamp": 1779032559.9410985
+        "timestamp": 1779042576.8926892
     },
     "LKR_AED": {
         "rate": 0.011311641375721824,
-        "timestamp": 1779032559.9411001
+        "timestamp": 1779042576.892691
     },
     "SAR_LKR": {
         "rate": 86.5316,
-        "timestamp": 1779032559.9411047
+        "timestamp": 1779042576.8926976
     },
     "LKR_SAR": {
         "rate": 0.011556471855368443,
-        "timestamp": 1779032559.9411058
+        "timestamp": 1779042576.8926988
     },
     "INR_LKR": {
         "rate": 3.3908,
-        "timestamp": 1779032559.9411101
+        "timestamp": 1779042576.8927054
     },
     "LKR_INR": {
         "rate": 0.29491565412292087,
-        "timestamp": 1779032559.941111
+        "timestamp": 1779042576.8927069
     },
     "CNY_LKR": {
         "rate": 47.7682,
-        "timestamp": 1779032559.9411154
+        "timestamp": 1779042576.8927135
     },
     "LKR_CNY": {
         "rate": 0.020934429180919523,
-        "timestamp": 1779032559.9411163
+        "timestamp": 1779042576.892715
     },
     "QAR_LKR": {
         "rate": 89.0738,
-        "timestamp": 1779032559.9411206
+        "timestamp": 1779042576.8927214
     },
     "LKR_QAR": {
         "rate": 0.011226645770136672,
-        "timestamp": 1779032559.9411216
+        "timestamp": 1779042576.8927226
     }
 }

scratch/test_fx_service.py

@@ -74,17 +74,8 @@ async def test_fx_service():
         print("    [Simulated Scraper Failure]")
         return False
     fx_service.fetch_cbsl_rates = broken_fetch
-    
-    # We will also temporarily unset API key and break ER-API call
-    original_api_key = fx_service.api_key
-    fx_service.api_key = "" # Unset key
-    original_fetch_api = fx_service.fetch_exchangerate_api
-    async def broken_api(base):
-        print("    [Simulated API Failure]")
-        return False
-    fx_service.fetch_exchangerate_api = broken_api
 
-    print("Cache is expired. External scrapers and APIs are simulated as down.")
+    print("Cache is expired. External scraper is simulated as down.")
     start_time = time.time()
     stale_rate = await fx_service.get_rate("USD", "LKR")
     elapsed = time.time() - start_time
@@ -94,8 +85,6 @@ async def test_fx_service():
 
     # Restore original methods
     fx_service.fetch_cbsl_rates = original_fetch
-    fx_service.api_key = original_api_key
-    fx_service.fetch_exchangerate_api = original_fetch_api
 
     print("==================================================")
     print("           ALL FXSERVICE TESTS PASSED!")

services/fx_service.py

@@ -63,9 +63,6 @@ class CBSLTableParser(HTMLParser):
 
 class FXService:
     def __init__(self):
-        self.api_key = settings.fx_api_key
-        # Premium exchangerate-api URL
-        self.base_url = f"https://v6.exchangerate-api.com/v6/{self.api_key}"
         self.cache: Dict[str, Dict] = {}
         self.load_cache()
 
@@ -227,50 +224,6 @@ class FXService:
             logger.error(f"Global exception in fetch_cbsl_rates: {e}")
             return False
 
-    async def fetch_exchangerate_api(self, base_currency: str) -> bool:
-        """
-        Fetches latest exchange rates from ExchangeRate-API.
-        First tries the user's custom premium key. If rate-limited or fails, automatically
-        falls back to the free, key-less open endpoint (open.er-api.com).
-        """
-        base_currency = base_currency.upper()
-        connector = aiohttp.TCPConnector(family=socket.AF_INET)
-        
-        # 1. Custom premium key URL
-        premium_url = f"{self.base_url}/latest/{base_currency}"
-        # 2. Free open fallback URL
-        fallback_url = f"https://open.er-api.com/v6/latest/{base_currency}"
-        
-        endpoints = []
-        if self.api_key:
-            endpoints.append(("Premium Key", premium_url, "conversion_rates"))
-        endpoints.append(("Free Open Fallback", fallback_url, "rates"))
-        
-        async with aiohttp.ClientSession(connector=connector) as session:
-            for name, url, rates_key in endpoints:
-                logger.info(f"Fetching rates using {name} from {url}...")
-                try:
-                    async with session.get(url, timeout=10) as response:
-                        if response.status == 200:
-                            data = await response.json()
-                            rates = data.get(rates_key, {})
-                            if rates:
-                                logger.info(f"Successfully fetched {len(rates)} rates using {name} for base {base_currency}")
-                                for target, val in rates.items():
-                                    target = target.upper()
-                                    if target in ALLOWED_CURRENCIES:
-                                        self.update_cache_entry(base_currency, target, val)
-                                self.save_cache()
-                                return True
-                        elif response.status == 429:
-                            logger.error(f"{name} rate-limited! Status: 429 Too Many Requests.")
-                        else:
-                            logger.error(f"{name} returned error status: {response.status}")
-                except Exception as e:
-                    logger.error(f"Exception fetching {name} for base {base_currency}: {e}")
-                    
-        return False
-
     async def get_rate(self, base_currency: str, target_currency: str) -> Optional[float]:
         """
         Retrieves real-time exchange rate with support for in-memory caching,
@@ -304,29 +257,17 @@ class FXService:
                 return derived
 
         # 4. Cache MISS: Retrieve fresh rates.
-        #    If either base or target is LKR, attempt CBSL Scraper first (preferred, official, and completely free).
-        if base == "LKR" or target == "LKR":
-            logger.info(f"Cache miss for LKR conversion {base}->{target}. Launching CBSL Web Scraper...")
-            cbsl_success = await self.fetch_cbsl_rates()
-            if cbsl_success:
-                res = self.get_cached_rate(base, target)
-                if res is not None:
-                    return res
-
-        # 5. Fallback/Alternative: If CBSL failed or we need a non-LKR rate and bridge was not in cache,
-        #    fetch from ExchangeRate-API.
-        logger.info(f"Fetching from ExchangeRate-API for {base}...")
-        api_success = await self.fetch_exchangerate_api(base)
-        if api_success:
+        # Since we are fully using web scraping, any cache miss should trigger CBSL scraper.
+        # CBSL scraper fetches all rates (against LKR) and populates them into the cache.
+        logger.info(f"Cache miss for {base}->{target}. Launching CBSL Web Scraper...")
+        cbsl_success = await self.fetch_cbsl_rates()
+        if cbsl_success:
+            # Try to get rate directly from cache (if either is LKR)
             res = self.get_cached_rate(base, target)
             if res is not None:
                 return res
-
-        # 6. Still failed: Try CBSL if we haven't done it (for non-LKR pairs, we can fetch CBSL and try to bridge)
-        if base != "LKR" and target != "LKR":
-            logger.info("Attempting CBSL scrape as fallback to calculate cross rate...")
-            cbsl_success = await self.fetch_cbsl_rates()
-            if cbsl_success:
+            # Or try to derive cross rate mathematically if both are non-LKR
+            if base != "LKR" and target != "LKR":
                 base_lkr = self.get_cached_rate(base, "LKR")
                 target_lkr = self.get_cached_rate(target, "LKR")
                 if base_lkr is not None and target_lkr is not None and target_lkr > 0:
@@ -336,13 +277,24 @@ class FXService:
                     logger.info(f"Derived cross rate {base}->{target} after CBSL scrape: {derived}")
                     return derived
 
-        # 7. Total Failure: Fallback to stale/expired cached rate if it exists
+        # 5. Total Failure: Fallback to stale/expired cached rate if it exists
         logger.warning(f"All live FX sources failed for {base}->{target}. Attempting stale cache fallback...")
         stale_rate = self.get_stale_rate(base, target)
         if stale_rate is not None:
             return stale_rate
 
-        # 8. Absolute Failure: No rates available
+        # If we failed to get a direct stale rate, try to compute it from stale LKR rates
+        if base != "LKR" and target != "LKR":
+            base_lkr_stale = self.get_stale_rate(base, "LKR")
+            target_lkr_stale = self.get_stale_rate(target, "LKR")
+            if base_lkr_stale is not None and target_lkr_stale is not None and target_lkr_stale > 0:
+                derived = base_lkr_stale / target_lkr_stale
+                self.update_cache_entry(base, target, derived)
+                self.save_cache()
+                logger.warning(f"Derived stale cross rate {base}->{target} from stale LKR rates: {derived}")
+                return derived
+
+        # 6. Absolute Failure: No rates available
         logger.critical(f"No rates available in memory, disk, or APIs for {base}->{target}!")
         return None