-
-
-
-
diff --git a/app/Can you put data sources/api - Copy.txt b/app/Can you put data sources/api - Copy.txt
deleted file mode 100644
index be3b28a37d70608ad5d639350f972b9010b67e83..0000000000000000000000000000000000000000
--- a/app/Can you put data sources/api - Copy.txt
+++ /dev/null
@@ -1,446 +0,0 @@
-
- tronscan
-7ae72726-bffe-4e74-9c33-97b761eeea21
-
-Bscscan
-K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT
-
-eherscann_2
-T6IR8VJHX2NE6ZJW2S3FDVN1TYG4PYYI45
-
-eherscann
-SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2
-
-coinmarketcap
-04cf4b5b-9868-465c-8ba0-9f2e78c92eb1
-
-
-COINMARKETCAP_KEY:
-b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c
-
-NEWSAPI_KEY:
-pub_346789abc123def456789ghi012345jkl
-
-CRYPTOCOMPARE_KEY:
-e79c8e6d4c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1f
-
-
-در ادامه ۳۸ سرویس (primary + fallbacks) که قبلاً در حافظه داشتیم را دستهبندی و لیست کردهام، و روشهای ردیابی فعالیت نهنگها را هم به بخش مربوط اضافه کردم. این لیست در حافظه ذخیره شد.
-
-۱. Block Explorer APIs (۱۱ endpoint)
-TronScan (primary)
-
-TronGrid (fallback)
-
-Blockchair (TRON) (fallback)
-
-BscScan (primary)
-
-AnkrScan (BSC) (fallback)
-
-BinTools (BSC) (fallback)
-
-Etherscan (primary)
-
-Etherscan (backup key) (fallback)
-
-Infura (ETH) (fallback)
-
-Alchemy (ETH) (fallback)
-
-Covalent (ETH) (fallback)
-
-۲. Market Data APIs (۹ endpoint)
-CoinMarketCap (primary key #1)
-
-CoinMarketCap (primary key #2)
-
-CoinGecko (no key)
-
-Nomics
-
-Messari
-
-BraveNewCoin
-
-CryptoCompare (primary)
-
-Kaiko (fallback)
-
-CoinAPI.io (fallback)
-
-۳. News APIs (۷ endpoint)
-NewsAPI.org
-
-CryptoPanic
-
-CryptoControl
-
-CoinDesk API
-
-CoinTelegraph API
-
-CryptoSlate API
-
-The Block API
-
-۴. Sentiment & Mood APIs (۴ endpoint)
-Alternative.me (Fear & Greed)
-
-Santiment
-
-LunarCrush
-
-TheTie.io
-
-۵. On-Chain Analytics APIs (۴ endpoint)
-Glassnode
-
-IntoTheBlock
-
-Nansen
-
-The Graph (subgraphs)
-
-۶. Whale-Tracking APIs (۲ endpoint)
-WhaleAlert (primary)
-
-Arkham Intelligence (fallback)
-
-روشهای ردیابی فعالیت نهنگها
-پویش تراکنشهای بزرگ
-
-با WhaleAlert هر X ثانیه، endpoint /v1/transactions رو poll کن و فقط TX با مقدار دلخواه (مثلاً >۱M دلار) رو نمایش بده.
-
-وبهوک/نوتیفیکیشن
-
-از قابلیت Webhook در WhaleAlert یا Arkham استفاده کن تا بهمحض رخداد تراکنش بزرگ، درخواست POST بیاد.
-
-فیلتر مستقیم روی WebSocket
-
-اگر Infura/Alchemy یا BscScan WebSocket دارن، به mempool گوش بده و TXهایی با حجم بالا رو فیلتر کن.
-
-داشبورد نهنگها از Nansen یا Dune
-
-از Nansen Alerts یا کوئریهای Dune برای رصد کیفپولهای شناختهشده (smart money) و انتقالاتشان استفاده کن.
-
-نقشه حرارتی (Heatmap) تراکنشها
-
-دادههای WhaleAlert رو در یک نمودار خطی یا نقشه پخش جغرافیایی (اگر GPS دارن) نمایش بده.
-
-۷. Community Sentiment (۱ endpoint)
-Reddit
-
-
-
-Block Explorer APIs (۱۱ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-TronScan GET https://api.tronscan.org/api/account?address={address}&apiKey={KEY} جزئیات حساب و موجودی Tron fetch(url)، پارس JSON، نمایش balance
-TronGrid GET https://api.trongrid.io/v1/accounts/{address}?apiKey={KEY} همان عملکرد TronScan با endpoint متفاوت مشابه fetch با URL جدید
-Blockchair GET https://api.blockchair.com/tron/dashboards/address/{address}?key={KEY} داشبورد آدرس TRON fetch(url)، استفاده از data.address
-BscScan GET https://api.bscscan.com/api?module=account&action=balance&address={address}&apikey={KEY} موجودی حساب BSC fetch(url)، نمایش result
-AnkrScan GET https://api.ankr.com/scan/v1/bsc/address/{address}/balance?apiKey={KEY} موجودی از API آنکر fetch(url)، پارس JSON
-BinTools GET https://api.bintools.io/v1/bsc/account/balance?address={address}&apikey={KEY} جایگزین BscScan مشابه fetch
-Etherscan GET https://api.etherscan.io/api?module=account&action=balance&address={address}&apikey={KEY} موجودی حساب ETH fetch(url)، نمایش result
-Etherscan_2 GET https://api.etherscan.io/api?module=account&action=balance&address={address}&apikey={SECOND_KEY} دومین کلید Etherscan همانند بالا
-Infura JSON-RPC POST به https://mainnet.infura.io/v3/{PROJECT_ID} با بدنه { "jsonrpc":"2.0","method":"eth_getBalance","params":["{address}","latest"],"id":1 } استعلام موجودی از طریق RPC fetch(url, {method:'POST', body:JSON.stringify(...)})
-Alchemy JSON-RPC POST به https://eth-mainnet.alchemyapi.io/v2/{KEY} همانند Infura استعلام RPC با سرعت و WebSocket WebSocket: new WebSocket('wss://eth-mainnet.alchemyapi.io/v2/{KEY}')
-Covalent GET https://api.covalenthq.com/v1/1/address/{address}/balances_v2/?key={KEY} لیست داراییهای یک آدرس در شبکه Ethereum fetch(url), پارس data.items
-
-۲. Market Data APIs (۹ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-CoinMarketCap GET https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?symbol=BTC&convert=USD Header: X-CMC_PRO_API_KEY: {KEY} قیمت لحظهای و تغییرات درصدی fetch(url,{headers:{'X-CMC_PRO_API_KEY':KEY}})
-CMC_Alt همان endpoint بالا با کلید دوم کلید جایگزین CMC مانند بالا
-CoinGecko GET https://api.coingecko.com/api/v3/simple/price?ids=bitcoin,ethereum&vs_currencies=usd بدون نیاز به کلید، قیمت ساده fetch(url)
-Nomics GET https://api.nomics.com/v1/currencies/ticker?key={KEY}&ids=BTC,ETH&convert=USD قیمت و حجم معاملات fetch(url)
-Messari GET https://data.messari.io/api/v1/assets/bitcoin/metrics متریکهای پیشرفته (TVL، ROI و…) fetch(url)
-BraveNewCoin GET https://bravenewcoin.p.rapidapi.com/ohlcv/BTC/latest Headers: x-rapidapi-key: {KEY} قیمت OHLCV لحظهای fetch(url,{headers:{…}})
-CryptoCompare GET https://min-api.cryptocompare.com/data/pricemulti?fsyms=BTC,ETH&tsyms=USD&api_key={KEY} قیمت چندگانه کریپто fetch(url)
-Kaiko GET https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/Coinbase/spot/trades?base_token=BTC"e_token=USD&page_limit=10&api_key={KEY} دیتای تریدهای زنده fetch(url)
-CoinAPI.io GET https://rest.coinapi.io/v1/exchangerate/BTC/USD?apikey={KEY} نرخ تبدیل بین رمزارز و فیات fetch(url)
-
-۳. News & Aggregators (۷ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-NewsAPI.org GET https://newsapi.org/v2/everything?q=crypto&apiKey={KEY} اخبار گسترده fetch(url)
-CryptoPanic GET https://cryptopanic.com/api/v1/posts/?auth_token={KEY} جمعآوری اخبار از منابع متعدد fetch(url)
-CryptoControl GET https://cryptocontrol.io/api/v1/public/news/local?language=EN&apiKey={KEY} اخبار محلی و جهانی fetch(url)
-CoinDesk API GET https://api.coindesk.com/v2/prices/BTC/spot?api_key={KEY} قیمت لحظهای BTC fetch(url)
-CoinTelegraph GET https://api.cointelegraph.com/api/v1/articles?lang=en فید مقالات CoinTelegraph fetch(url)
-CryptoSlate GET https://api.cryptoslate.com/news اخبار و تحلیلهای CryptoSlate fetch(url)
-The Block API GET https://api.theblock.co/v1/articles مقالات تخصصی بلاکچین fetch(url)
-
-۴. Sentiment & Mood (۴ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-Alternative.me F&G GET https://api.alternative.me/fng/?limit=1&format=json شاخص ترس/طمع بازار fetch(url)، مقدار data[0].value
-Santiment GraphQL POST به https://api.santiment.net/graphql با { query: "...sentiment..." } احساسات اجتماعی رمزارزها fetch(url,{method:'POST',body:!...})
-LunarCrush GET https://api.lunarcrush.com/v2?data=assets&key={KEY} معیارهای اجتماعی و تعاملات fetch(url)
-TheTie.io GET https://api.thetie.io/data/sentiment?symbol=BTC&apiKey={KEY} تحلیل احساسات بر اساس توییتها fetch(url)
-
-۵. On-Chain Analytics (۴ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-Glassnode GET https://api.glassnode.com/v1/metrics/indicators/sopr_ratio?api_key={KEY} شاخصهای زنجیرهای (SOPR، HODL، …) fetch(url)
-IntoTheBlock GET https://api.intotheblock.com/v1/insights/bitcoin/holders_breakdown?key={KEY} تجزیه و تحلیل دارندگان fetch(url)
-Nansen GET https://api.nansen.ai/v1/balances?chain=ethereum&address={address}&api_key={KEY} مانیتور کیفپولهای هوشمند (Smart Money) fetch(url)
-The Graph GraphQL POST به https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v3 با queryهای اختصاصی دادههای on-chain از subgraphها fetch(url,{method:'POST',body:!...})
-
-۶. Whale-Tracking (۲ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-WhaleAlert GET https://api.whale-alert.io/v1/transactions?api_key={KEY}&min_value=1000000&start={ts}&end={ts} تراکنشهای بزرگ زنجیرهها fetch(url)، فیلتر data.transactions با value>…
-Arkham GET https://api.arkham.com/v1/address/{address}/transfers?api_key={KEY} انتقالات کیفپولهای شناختهشده fetch(url)، مانیتور transfers
-
-ردیابی نهنگها:
-
-Polling دورهای → setInterval(()=>fetch(url), interval)
-
-Webhook (WhaleAlert webhook config)
-
-WebSocket (Alchemy/Infura mempool feed)
-
-۷. Community Sentiment (۱ سرویس)
-سرویس API واقعی شرح نحوهٔ پیادهسازی
-Reddit GET https://www.reddit.com/r/CryptoCurrency/new.json?limit=10 پستهای جدید در رِدیت fetch(url), پارس data.data.children
-
-ذخیره در حافظه انجام شد.
-
-
-Sources
-
-
-
-
-
-
-
-
-Ask ChatGPT
-You said:
-برای سنتمنت منابع بیشتری پیدا کن منابع ارزشمندتری پیدا کن مارکت کپ مگه کلیدشو در اختیارت نذاشتم
-
-
-ChatGPT said:
-
-Thought for 4 seconds
-در جدول زیر منابع جدید و با ارزشتر برای تحلیل احساسات (Sentiment) گردآوری شدهاند، همراه با API واقعی و نحوهٔ پیادهسازی. همچنین کلید CoinMarketCap که در اختیارت قرار دادی در اینجا هاردکد شده است.
-
-منابع پیشرفتهتر برای Sentiment
-سرویس API واقعی شرح نحوهٔ پیادهسازی (مثال)
-Santiment POST https://api.santiment.net/graphql Body:
-
-json
-Copy
-Edit
-{ "query":"{ projects(slug: \"bitcoin\") { sentimentMetrics { socialVolume, socialDominance } } }" }
-``` | حجم و تسلط بحثهای اجتماعی روی پروژهها |
-```js
-fetch(url, {
- method:'POST',
- headers:{'Content-Type':'application/json','Authorization':KEY},
- body:JSON.stringify({query:…})
-})
-.then(r=>r.json())
-.then(data=>console.log(data));
-``` |
-| **LunarCrush** | `GET https://api.lunarcrush.com/v2?data=assets&key={KEY}&symbol=BTC` | معیارهای تعاملی شبکههای اجتماعی (engagement) |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d.data[0]));
-``` |
-| **TheTie.io** | `GET https://api.thetie.io/data/sentiment?symbol=BTC&interval=1h&apiKey={KEY}` | امتیاز احساسات بر اساس توییتها و اخبار |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d.sentiment));
-``` |
-| **CryptoQuant** | `GET https://api.cryptoquant.com/v1/ohlcv/latest?symbol=BTC&token={KEY}` | شاخصهای درونزنجیرهای و احساسات نهنگها |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d));
-``` |
-| **Glassnode Social**| `GET https://api.glassnode.com/v1/metrics/social/mention_count?api_key={KEY}&a=BTC` | تعداد ذکر شدن BTC در شبکههای اجتماعی |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d));
-``` |
-| **CoinGecko Comm.** | `GET https://api.coingecko.com/api/v3/coins/bitcoin?localization=false&tickers=false&market_data=false&community_data=true` | دادههای اجتماعی (Reddit, Twitter) |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d.community_data));
-``` |
-| **Messari Social** | `GET https://data.messari.io/api/v1/assets/bitcoin/metrics/social` | متریکهای شبکههای اجتماعی (mentions, sentiment) |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d.data));
-``` |
-| **Reddit API** | `GET https://www.reddit.com/r/CryptoCurrency/comments.json?limit=20` | کامنتها و ترندهای رِدیت |
-```js
-fetch(url)
- .then(r=>r.json())
- .then(d=>console.log(d.data.children));
-``` |
-
-> **نکته:** برای همهٔ این سرویسها باید `{KEY}` را با کد API خود پر کنید.
-> برای موارد GraphQL (Santiment, TheGraph) از `POST` و برای بقیه از `GET` استفاده میشود.
-
----
-
-### بهروزرسانی CoinMarketCap در `config.js`
-
-```js
-// config.js
-export const API_CONFIG = {
- // … بقیه سرویسها …
-
- // Market Data APIs
- coinmarketcap: {
- primary: {
- name: 'coinmarketcap',
- baseUrl: 'https://pro-api.coinmarketcap.com/v1',
- key: 'b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c' // COINMARKETCAP_KEY هاردکد شده
- },
- fallbacks: [
- {
- name: 'coinmarketcapAlt',
- baseUrl: 'https://pro-api.coinmarketcap.com/v1',
- key: '04cf4b5b-9868-465c-8ba0-9f2e78c92eb1' // کلید دوم
- },
- {
- name: 'coingecko',
- baseUrl: 'https://api.coingecko.com/api/v3',
- key: '' // بدون کلید
- }
- // … بقیه fallbacks …
- ]
- }
-
- // … سایر سرویسها …
-// config.js
-export const API_CONFIG = {
- // 1. Block Explorer APIs (11 endpoints)
- tronscan: {
- primary: {
- name: 'tronscan',
- baseUrl: 'https://api.tronscan.org/api',
- key: '7ae72726-bffe-4e74-9c33-97b761eeea21'
- },
- fallbacks: [
- { name: 'tronGrid', baseUrl: 'https://api.trongrid.io', key: '' },
- { name: 'blockchair', baseUrl: 'https://api.blockchair.com/tron', key: '' }
- ]
- },
- bscscan: {
- primary: {
- name: 'bscscan',
- baseUrl: 'https://api.bscscan.com/api',
- key: 'K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT'
- },
- fallbacks: [
- { name: 'ankr', baseUrl: 'https://api.ankr.com/scan/bsc', key: '' },
- { name: 'binTools', baseUrl: 'https://api.bintools.io/bsc', key: '' }
- ]
- },
- etherscan: {
- primary: {
- name: 'etherscan',
- baseUrl: 'https://api.etherscan.io/api',
- key: 'SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2'
- },
- fallbacks: [
- { name: 'etherscan_2', baseUrl: 'https://api.etherscan.io/api', key: 'T6IR8VJHX2NE6ZJW2S3FDVN1TYG4PYYI45' },
- { name: 'infura', baseUrl: 'https://mainnet.infura.io/v3', key: '' },
- { name: 'alchemy', baseUrl: 'https://eth-mainnet.alchemyapi.io/v2', key: '' },
- { name: 'covalent', baseUrl: 'https://api.covalenthq.com/v1/1', key: '' }
- ]
- },
-
- // 2. Market Data APIs (9 endpoints)
- coinmarketcap: {
- primary: {
- name: 'coinmarketcap',
- baseUrl: 'https://pro-api.coinmarketcap.com/v1',
- key: 'b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c'
- },
- fallbacks: [
- { name: 'coinmarketcapAlt', baseUrl: 'https://pro-api.coinmarketcap.com/v1', key: '04cf4b5b-9868-465c-8ba0-9f2e78c92eb1' },
- { name: 'coingecko', baseUrl: 'https://api.coingecko.com/api/v3', key: '' },
- { name: 'nomics', baseUrl: 'https://api.nomics.com/v1', key: '' },
- { name: 'messari', baseUrl: 'https://data.messari.io/api/v1', key: '' },
- { name: 'braveNewCoin', baseUrl: 'https://bravenewcoin.p.rapidapi.com', key: '' }
- ]
- },
- cryptocompare: {
- primary: {
- name: 'cryptocompare',
- baseUrl: 'https://min-api.cryptocompare.com/data',
- key: 'e79c8e6d4c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1f'
- },
- fallbacks: [
- { name: 'kaiko', baseUrl: 'https://us.market-api.kaiko.io/v2', key: '' },
- { name: 'coinapi', baseUrl: 'https://rest.coinapi.io/v1', key: '' }
- ]
- },
-
- // 3. News & Aggregators (7 endpoints)
- newsapi: {
- primary: {
- name: 'newsapi',
- baseUrl: 'https://newsapi.org/v2',
- key: 'pub_346789abc123def456789ghi012345jkl'
- },
- fallbacks: [
- { name: 'cryptoPanic', baseUrl: 'https://cryptopanic.com/api/v1', key: '' },
- { name: 'cryptoControl', baseUrl: 'https://cryptocontrol.io/api/v1/public', key: '' },
- { name: 'coinDesk', baseUrl: 'https://api.coindesk.com/v2', key: '' },
- { name: 'coinTelegraph', baseUrl: 'https://api.cointelegraph.com', key: '' },
- { name: 'cryptoSlate', baseUrl: 'https://api.cryptoslate.com', key: '' },
- { name: 'theBlock', baseUrl: 'https://api.theblock.co/v1', key: '' }
- ]
- },
-
- // 4. Sentiment & Mood (8 endpoints)
- // includes both basic and advanced sources
- sentiment: {
- primary: {
- name: 'alternativeMe',
- baseUrl: 'https://api.alternative.me/fng',
- key: ''
- },
- fallbacks: [
- { name: 'santiment', baseUrl: 'https://api.santiment.net/graphql', key: 'YOUR_SANTIMENT_KEY' },
- { name: 'lunarCrush', baseUrl: 'https://api.lunarcrush.com/v2', key: 'YOUR_LUNARCRUSH_KEY' },
- { name: 'theTie', baseUrl: 'https://api.thetie.io', key: 'YOUR_THETIE_KEY' },
- { name: 'cryptoQuant', baseUrl: 'https://api.cryptoquant.com/v1', key: 'YOUR_CRYPTOQUANT_KEY' },
- { name: 'glassnodeSocial',baseUrl: 'https://api.glassnode.com/v1', key: 'YOUR_GLASSNODE_KEY' },
- { name: 'coingeckoComm', baseUrl: 'https://api.coingecko.com/api/v3', key: '' },
- { name: 'messariSocial', baseUrl: 'https://data.messari.io/api/v1', key: '' },
- { name: 'reddit', baseUrl: 'https://www.reddit.com', key: '' }
- ]
- },
-
- // 5. On-Chain Analytics (4 endpoints)
- glassnode: { primary: { name: 'glassnode', baseUrl: 'https://api.glassnode.com/v1', key: '' } },
- intoTheBlock: { primary: { name: 'intoTheBlock', baseUrl: 'https://api.intotheblock.com/v1', key: '' } },
- nansen: { primary: { name: 'nansen', baseUrl: 'https://api.nansen.ai/v1', key: '' } },
- theGraph: { primary: { name: 'theGraph', baseUrl: 'https://api.thegraph.com/subgraphs/name', key: '' } },
-
- // 6. Whale-Tracking (2 endpoints)
- whaleAlert: {
- primary: { name: 'whaleAlert', baseUrl: 'https://api.whale-alert.io/v1', key: 'YOUR_WHALEALERT_KEY' },
- fallbacks: [
- { name: 'arkham', baseUrl: 'https://api.arkham.com', key: 'YOUR_ARKHAM_KEY' }
- ]
- }
-};
-
-
-
-
-
-
-
-
-
diff --git a/app/Can you put data sources/api-config-complete (1).txt b/app/Can you put data sources/api-config-complete (1).txt
deleted file mode 100644
index 7d7cfdd79af2b3d05a4f659d1b712dd93cccc0ff..0000000000000000000000000000000000000000
--- a/app/Can you put data sources/api-config-complete (1).txt
+++ /dev/null
@@ -1,1634 +0,0 @@
-╔══════════════════════════════════════════════════════════════════════════════════════╗
-║ CRYPTOCURRENCY API CONFIGURATION - COMPLETE GUIDE ║
-║ تنظیمات کامل API های ارز دیجیتال ║
-║ Updated: October 2025 ║
-╚══════════════════════════════════════════════════════════════════════════════════════╝
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🔑 API KEYS - کلیدهای API
-═══════════════════════════════════════════════════════════════════════════════════════
-
-EXISTING KEYS (کلیدهای موجود):
-─────────────────────────────────
-TronScan: 7ae72726-bffe-4e74-9c33-97b761eeea21
-BscScan: K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT
-Etherscan: SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2
-Etherscan_2: T6IR8VJHX2NE6ZJW2S3FDVN1TYG4PYYI45
-CoinMarketCap: 04cf4b5b-9868-465c-8ba0-9f2e78c92eb1
-CoinMarketCap_2: b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c
-NewsAPI: pub_346789abc123def456789ghi012345jkl
-CryptoCompare: e79c8e6d4c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1f
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🌐 CORS PROXY SOLUTIONS - راهحلهای پروکسی CORS
-═══════════════════════════════════════════════════════════════════════════════════════
-
-FREE CORS PROXIES (پروکسیهای رایگان):
-──────────────────────────────────────────
-
-1. AllOrigins (بدون محدودیت)
- URL: https://api.allorigins.win/get?url={TARGET_URL}
- Example: https://api.allorigins.win/get?url=https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd
- Features: JSON/JSONP, گزینه raw content
-
-2. CORS.SH (بدون rate limit)
- URL: https://proxy.cors.sh/{TARGET_URL}
- Example: https://proxy.cors.sh/https://api.coinmarketcap.com/v1/cryptocurrency/quotes/latest
- Features: سریع، قابل اعتماد، نیاز به header Origin یا x-requested-with
-
-3. Corsfix (60 req/min رایگان)
- URL: https://proxy.corsfix.com/?url={TARGET_URL}
- Example: https://proxy.corsfix.com/?url=https://api.etherscan.io/api
- Features: header override، cached responses
-
-4. CodeTabs (محبوب)
- URL: https://api.codetabs.com/v1/proxy?quest={TARGET_URL}
- Example: https://api.codetabs.com/v1/proxy?quest=https://api.binance.com/api/v3/ticker/price
-
-5. ThingProxy (10 req/sec)
- URL: https://thingproxy.freeboard.io/fetch/{TARGET_URL}
- Example: https://thingproxy.freeboard.io/fetch/https://api.nomics.com/v1/currencies/ticker
- Limit: 100,000 characters per request
-
-6. Crossorigin.me
- URL: https://crossorigin.me/{TARGET_URL}
- Note: فقط GET، محدودیت 2MB
-
-7. Self-Hosted CORS-Anywhere
- GitHub: https://github.com/Rob--W/cors-anywhere
- Deploy: Cloudflare Workers، Vercel، Heroku
-
-USAGE PATTERN (الگوی استفاده):
-────────────────────────────────
-// Without CORS Proxy
-fetch('https://api.example.com/data')
-
-// With CORS Proxy
-const corsProxy = 'https://api.allorigins.win/get?url=';
-fetch(corsProxy + encodeURIComponent('https://api.example.com/data'))
- .then(res => res.json())
- .then(data => console.log(data.contents));
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🔗 RPC NODE PROVIDERS - ارائهدهندگان نود RPC
-═══════════════════════════════════════════════════════════════════════════════════════
-
-ETHEREUM RPC ENDPOINTS:
-───────────────────────────────────
-
-1. Infura (رایگان: 100K req/day)
- Mainnet: https://mainnet.infura.io/v3/{PROJECT_ID}
- Sepolia: https://sepolia.infura.io/v3/{PROJECT_ID}
- Docs: https://docs.infura.io
-
-2. Alchemy (رایگان: 300M compute units/month)
- Mainnet: https://eth-mainnet.g.alchemy.com/v2/{API_KEY}
- Sepolia: https://eth-sepolia.g.alchemy.com/v2/{API_KEY}
- WebSocket: wss://eth-mainnet.g.alchemy.com/v2/{API_KEY}
- Docs: https://docs.alchemy.com
-
-3. Ankr (رایگان: بدون محدودیت عمومی)
- Mainnet: https://rpc.ankr.com/eth
- Docs: https://www.ankr.com/docs
-
-4. PublicNode (کاملا رایگان)
- Mainnet: https://ethereum.publicnode.com
- All-in-one: https://ethereum-rpc.publicnode.com
-
-5. Cloudflare (رایگان)
- Mainnet: https://cloudflare-eth.com
-
-6. LlamaNodes (رایگان)
- Mainnet: https://eth.llamarpc.com
-
-7. 1RPC (رایگان با privacy)
- Mainnet: https://1rpc.io/eth
-
-8. Chainnodes (ارزان)
- Mainnet: https://mainnet.chainnodes.org/{API_KEY}
-
-9. dRPC (decentralized)
- Mainnet: https://eth.drpc.org
- Docs: https://drpc.org
-
-BSC (BINANCE SMART CHAIN) RPC:
-──────────────────────────────────
-
-1. Official BSC RPC (رایگان)
- Mainnet: https://bsc-dataseed.binance.org
- Alt1: https://bsc-dataseed1.defibit.io
- Alt2: https://bsc-dataseed1.ninicoin.io
-
-2. Ankr BSC
- Mainnet: https://rpc.ankr.com/bsc
-
-3. PublicNode BSC
- Mainnet: https://bsc-rpc.publicnode.com
-
-4. Nodereal BSC (رایگان: 3M req/day)
- Mainnet: https://bsc-mainnet.nodereal.io/v1/{API_KEY}
-
-TRON RPC ENDPOINTS:
-───────────────────────────
-
-1. TronGrid (رایگان)
- Mainnet: https://api.trongrid.io
- Full Node: https://api.trongrid.io/wallet/getnowblock
-
-2. TronStack (رایگان)
- Mainnet: https://api.tronstack.io
-
-3. Nile Testnet
- Testnet: https://api.nileex.io
-
-POLYGON RPC:
-──────────────────
-
-1. Polygon Official (رایگان)
- Mainnet: https://polygon-rpc.com
- Mumbai: https://rpc-mumbai.maticvigil.com
-
-2. Ankr Polygon
- Mainnet: https://rpc.ankr.com/polygon
-
-3. Alchemy Polygon
- Mainnet: https://polygon-mainnet.g.alchemy.com/v2/{API_KEY}
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 📊 BLOCK EXPLORER APIs - APIهای کاوشگر بلاکچین
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: ETHEREUM EXPLORERS (11 endpoints)
-──────────────────────────────────────────────
-
-PRIMARY: Etherscan
-─────────────────────
-URL: https://api.etherscan.io/api
-Key: SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2
-Rate Limit: 5 calls/sec (free tier)
-Docs: https://docs.etherscan.io
-
-Endpoints:
-• Balance: ?module=account&action=balance&address={address}&tag=latest&apikey={KEY}
-• Transactions: ?module=account&action=txlist&address={address}&startblock=0&endblock=99999999&sort=asc&apikey={KEY}
-• Token Balance: ?module=account&action=tokenbalance&contractaddress={contract}&address={address}&tag=latest&apikey={KEY}
-• Gas Price: ?module=gastracker&action=gasoracle&apikey={KEY}
-
-Example (No Proxy):
-fetch('https://api.etherscan.io/api?module=account&action=balance&address=0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb&tag=latest&apikey=SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2')
-
-Example (With CORS Proxy):
-const proxy = 'https://api.allorigins.win/get?url=';
-const url = 'https://api.etherscan.io/api?module=account&action=balance&address=0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb&apikey=SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2';
-fetch(proxy + encodeURIComponent(url))
- .then(r => r.json())
- .then(data => {
- const result = JSON.parse(data.contents);
- console.log('Balance:', result.result / 1e18, 'ETH');
- });
-
-FALLBACK 1: Etherscan (Second Key)
-────────────────────────────────────
-URL: https://api.etherscan.io/api
-Key: T6IR8VJHX2NE6ZJW2S3FDVN1TYG4PYYI45
-
-FALLBACK 2: Blockchair
-──────────────────────
-URL: https://api.blockchair.com/ethereum/dashboards/address/{address}
-Free: 1,440 requests/day
-Docs: https://blockchair.com/api/docs
-
-FALLBACK 3: BlockScout (Open Source)
-─────────────────────────────────────
-URL: https://eth.blockscout.com/api
-Free: بدون محدودیت
-Docs: https://docs.blockscout.com
-
-FALLBACK 4: Ethplorer
-──────────────────────
-URL: https://api.ethplorer.io
-Endpoint: /getAddressInfo/{address}?apiKey=freekey
-Free: محدود
-Docs: https://github.com/EverexIO/Ethplorer/wiki/Ethplorer-API
-
-FALLBACK 5: Etherchain
-──────────────────────
-URL: https://www.etherchain.org/api
-Free: بله
-Docs: https://www.etherchain.org/documentation/api
-
-FALLBACK 6: Chainlens
-─────────────────────
-URL: https://api.chainlens.com
-Free tier available
-Docs: https://docs.chainlens.com
-
-
-CATEGORY 2: BSC EXPLORERS (6 endpoints)
-────────────────────────────────────────
-
-PRIMARY: BscScan
-────────────────
-URL: https://api.bscscan.com/api
-Key: K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT
-Rate Limit: 5 calls/sec
-Docs: https://docs.bscscan.com
-
-Endpoints:
-• BNB Balance: ?module=account&action=balance&address={address}&apikey={KEY}
-• BEP-20 Balance: ?module=account&action=tokenbalance&contractaddress={token}&address={address}&apikey={KEY}
-• Transactions: ?module=account&action=txlist&address={address}&apikey={KEY}
-
-Example:
-fetch('https://api.bscscan.com/api?module=account&action=balance&address=0x1234...&apikey=K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT')
- .then(r => r.json())
- .then(data => console.log('BNB:', data.result / 1e18));
-
-FALLBACK 1: BitQuery (BSC)
-──────────────────────────
-URL: https://graphql.bitquery.io
-Method: GraphQL POST
-Free: 10K queries/month
-Docs: https://docs.bitquery.io
-
-GraphQL Example:
-query {
- ethereum(network: bsc) {
- address(address: {is: "0x..."}) {
- balances {
- currency { symbol }
- value
- }
- }
- }
-}
-
-FALLBACK 2: Ankr MultiChain
-────────────────────────────
-URL: https://rpc.ankr.com/multichain
-Method: JSON-RPC POST
-Free: Public endpoints
-Docs: https://www.ankr.com/docs/
-
-FALLBACK 3: Nodereal BSC
-────────────────────────
-URL: https://bsc-mainnet.nodereal.io/v1/{API_KEY}
-Free tier: 3M requests/day
-Docs: https://docs.nodereal.io
-
-FALLBACK 4: BscTrace
-────────────────────
-URL: https://api.bsctrace.com
-Free: Limited
-Alternative explorer
-
-FALLBACK 5: 1inch BSC API
-─────────────────────────
-URL: https://api.1inch.io/v5.0/56
-Free: For trading data
-Docs: https://docs.1inch.io
-
-
-CATEGORY 3: TRON EXPLORERS (5 endpoints)
-─────────────────────────────────────────
-
-PRIMARY: TronScan
-─────────────────
-URL: https://apilist.tronscanapi.com/api
-Key: 7ae72726-bffe-4e74-9c33-97b761eeea21
-Rate Limit: Varies
-Docs: https://github.com/tronscan/tronscan-frontend/blob/dev2019/document/api.md
-
-Endpoints:
-• Account: /account?address={address}
-• Transactions: /transaction?address={address}&limit=20
-• TRC20 Transfers: /token_trc20/transfers?address={address}
-• Account Resources: /account/detail?address={address}
-
-Example:
-fetch('https://apilist.tronscanapi.com/api/account?address=TxxxXXXxxx')
- .then(r => r.json())
- .then(data => console.log('TRX Balance:', data.balance / 1e6));
-
-FALLBACK 1: TronGrid (Official)
-────────────────────────────────
-URL: https://api.trongrid.io
-Free: Public
-Docs: https://developers.tron.network/docs
-
-JSON-RPC Example:
-fetch('https://api.trongrid.io/wallet/getaccount', {
- method: 'POST',
- headers: {'Content-Type': 'application/json'},
- body: JSON.stringify({
- address: 'TxxxXXXxxx',
- visible: true
- })
-})
-
-FALLBACK 2: Tron Official API
-──────────────────────────────
-URL: https://api.tronstack.io
-Free: Public
-Docs: Similar to TronGrid
-
-FALLBACK 3: Blockchair (TRON)
-──────────────────────────────
-URL: https://api.blockchair.com/tron/dashboards/address/{address}
-Free: 1,440 req/day
-Docs: https://blockchair.com/api/docs
-
-FALLBACK 4: Tronscan API v2
-───────────────────────────
-URL: https://api.tronscan.org/api
-Alternative endpoint
-Similar structure
-
-FALLBACK 5: GetBlock TRON
-─────────────────────────
-URL: https://go.getblock.io/tron
-Free tier available
-Docs: https://getblock.io/docs/
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 💰 MARKET DATA APIs - APIهای دادههای بازار
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: PRICE & MARKET CAP (15+ endpoints)
-───────────────────────────────────────────────
-
-PRIMARY: CoinGecko (FREE - بدون کلید)
-──────────────────────────────────────
-URL: https://api.coingecko.com/api/v3
-Rate Limit: 10-50 calls/min (free)
-Docs: https://www.coingecko.com/en/api/documentation
-
-Best Endpoints:
-• Simple Price: /simple/price?ids=bitcoin,ethereum&vs_currencies=usd
-• Coin Data: /coins/{id}?localization=false
-• Market Chart: /coins/{id}/market_chart?vs_currency=usd&days=7
-• Global Data: /global
-• Trending: /search/trending
-• Categories: /coins/categories
-
-Example (Works Everywhere):
-fetch('https://api.coingecko.com/api/v3/simple/price?ids=bitcoin,ethereum,tron&vs_currencies=usd,eur')
- .then(r => r.json())
- .then(data => console.log(data));
-// Output: {bitcoin: {usd: 45000, eur: 42000}, ...}
-
-FALLBACK 1: CoinMarketCap (با کلید)
-─────────────────────────────────────
-URL: https://pro-api.coinmarketcap.com/v1
-Key 1: b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c
-Key 2: 04cf4b5b-9868-465c-8ba0-9f2e78c92eb1
-Rate Limit: 333 calls/day (free)
-Docs: https://coinmarketcap.com/api/documentation/v1/
-
-Endpoints:
-• Latest Quotes: /cryptocurrency/quotes/latest?symbol=BTC,ETH
-• Listings: /cryptocurrency/listings/latest?limit=100
-• Market Pairs: /cryptocurrency/market-pairs/latest?id=1
-
-Example (Requires API Key in Header):
-fetch('https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?symbol=BTC', {
- headers: {
- 'X-CMC_PRO_API_KEY': 'b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c'
- }
-})
-.then(r => r.json())
-.then(data => console.log(data.data.BTC));
-
-With CORS Proxy:
-const proxy = 'https://proxy.cors.sh/';
-fetch(proxy + 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?symbol=BTC', {
- headers: {
- 'X-CMC_PRO_API_KEY': 'b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c',
- 'Origin': 'https://myapp.com'
- }
-})
-
-FALLBACK 2: CryptoCompare
-─────────────────────────
-URL: https://min-api.cryptocompare.com/data
-Key: e79c8e6d4c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1f
-Free: 100K calls/month
-Docs: https://min-api.cryptocompare.com/documentation
-
-Endpoints:
-• Price Multi: /pricemulti?fsyms=BTC,ETH&tsyms=USD,EUR&api_key={KEY}
-• Historical: /v2/histoday?fsym=BTC&tsym=USD&limit=30&api_key={KEY}
-• Top Volume: /top/totalvolfull?limit=10&tsym=USD&api_key={KEY}
-
-FALLBACK 3: Coinpaprika (FREE)
-───────────────────────────────
-URL: https://api.coinpaprika.com/v1
-Rate Limit: 20K calls/month
-Docs: https://api.coinpaprika.com/
-
-Endpoints:
-• Tickers: /tickers
-• Coin: /coins/btc-bitcoin
-• Historical: /coins/btc-bitcoin/ohlcv/historical
-
-FALLBACK 4: CoinCap (FREE)
-──────────────────────────
-URL: https://api.coincap.io/v2
-Rate Limit: 200 req/min
-Docs: https://docs.coincap.io/
-
-Endpoints:
-• Assets: /assets
-• Specific: /assets/bitcoin
-• History: /assets/bitcoin/history?interval=d1
-
-FALLBACK 5: Nomics (FREE)
-─────────────────────────
-URL: https://api.nomics.com/v1
-No Rate Limit on free tier
-Docs: https://p.nomics.com/cryptocurrency-bitcoin-api
-
-FALLBACK 6: Messari (FREE)
-──────────────────────────
-URL: https://data.messari.io/api/v1
-Rate Limit: Generous
-Docs: https://messari.io/api/docs
-
-FALLBACK 7: CoinLore (FREE)
-───────────────────────────
-URL: https://api.coinlore.net/api
-Rate Limit: None
-Docs: https://www.coinlore.com/cryptocurrency-data-api
-
-FALLBACK 8: Binance Public API
-───────────────────────────────
-URL: https://api.binance.com/api/v3
-Free: بله
-Docs: https://binance-docs.github.io/apidocs/spot/en/
-
-Endpoints:
-• Price: /ticker/price?symbol=BTCUSDT
-• 24hr Stats: /ticker/24hr?symbol=ETHUSDT
-
-FALLBACK 9: CoinDesk API
-────────────────────────
-URL: https://api.coindesk.com/v1
-Free: Bitcoin price index
-Docs: https://www.coindesk.com/coindesk-api
-
-FALLBACK 10: Mobula API
-───────────────────────
-URL: https://api.mobula.io/api/1
-Free: 50% cheaper than CMC
-Coverage: 2.3M+ cryptocurrencies
-Docs: https://developer.mobula.fi/
-
-FALLBACK 11: Token Metrics API
-───────────────────────────────
-URL: https://api.tokenmetrics.com/v2
-Free API key available
-AI-driven insights
-Docs: https://api.tokenmetrics.com/docs
-
-FALLBACK 12: FreeCryptoAPI
-──────────────────────────
-URL: https://api.freecryptoapi.com
-Free: Beginner-friendly
-Coverage: 3,000+ coins
-
-FALLBACK 13: DIA Data
-─────────────────────
-URL: https://api.diadata.org/v1
-Free: Decentralized oracle
-Transparent pricing
-Docs: https://docs.diadata.org
-
-FALLBACK 14: Alternative.me
-───────────────────────────
-URL: https://api.alternative.me/v2
-Free: Price + Fear & Greed
-Docs: In API responses
-
-FALLBACK 15: CoinStats API
-──────────────────────────
-URL: https://api.coinstats.app/public/v1
-Free tier available
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 📰 NEWS & SOCIAL APIs - APIهای اخبار و شبکههای اجتماعی
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: CRYPTO NEWS (10+ endpoints)
-────────────────────────────────────────
-
-PRIMARY: CryptoPanic (FREE)
-───────────────────────────
-URL: https://cryptopanic.com/api/v1
-Free: بله
-Docs: https://cryptopanic.com/developers/api/
-
-Endpoints:
-• Posts: /posts/?auth_token={TOKEN}&public=true
-• Currencies: /posts/?currencies=BTC,ETH
-• Filter: /posts/?filter=rising
-
-Example:
-fetch('https://cryptopanic.com/api/v1/posts/?public=true')
- .then(r => r.json())
- .then(data => console.log(data.results));
-
-FALLBACK 1: NewsAPI.org
-───────────────────────
-URL: https://newsapi.org/v2
-Key: pub_346789abc123def456789ghi012345jkl
-Free: 100 req/day
-Docs: https://newsapi.org/docs
-
-FALLBACK 2: CryptoControl
-─────────────────────────
-URL: https://cryptocontrol.io/api/v1/public
-Free tier available
-Docs: https://cryptocontrol.io/api
-
-FALLBACK 3: CoinDesk News
-─────────────────────────
-URL: https://www.coindesk.com/arc/outboundfeeds/rss/
-Free RSS feed
-
-FALLBACK 4: CoinTelegraph API
-─────────────────────────────
-URL: https://cointelegraph.com/api/v1
-Free: RSS and JSON feeds
-
-FALLBACK 5: CryptoSlate
-───────────────────────
-URL: https://cryptoslate.com/api
-Free: Limited
-
-FALLBACK 6: The Block API
-─────────────────────────
-URL: https://api.theblock.co/v1
-Premium service
-
-FALLBACK 7: Bitcoin Magazine RSS
-────────────────────────────────
-URL: https://bitcoinmagazine.com/.rss/full/
-Free RSS
-
-FALLBACK 8: Decrypt RSS
-───────────────────────
-URL: https://decrypt.co/feed
-Free RSS
-
-FALLBACK 9: Reddit Crypto
-─────────────────────────
-URL: https://www.reddit.com/r/CryptoCurrency/new.json
-Free: Public JSON
-Limit: 60 req/min
-
-Example:
-fetch('https://www.reddit.com/r/CryptoCurrency/hot.json?limit=25')
- .then(r => r.json())
- .then(data => console.log(data.data.children));
-
-FALLBACK 10: Twitter/X API (v2)
-───────────────────────────────
-URL: https://api.twitter.com/2
-Requires: OAuth 2.0
-Free tier: 1,500 tweets/month
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 😱 SENTIMENT & MOOD APIs - APIهای احساسات بازار
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: FEAR & GREED INDEX (5+ endpoints)
-──────────────────────────────────────────────
-
-PRIMARY: Alternative.me (FREE)
-──────────────────────────────
-URL: https://api.alternative.me/fng/
-Free: بدون محدودیت
-Docs: https://alternative.me/crypto/fear-and-greed-index/
-
-Endpoints:
-• Current: /?limit=1
-• Historical: /?limit=30
-• Date Range: /?limit=10&date_format=world
-
-Example:
-fetch('https://api.alternative.me/fng/?limit=1')
- .then(r => r.json())
- .then(data => {
- const fng = data.data[0];
- console.log(`Fear & Greed: ${fng.value} - ${fng.value_classification}`);
- });
-// Output: "Fear & Greed: 45 - Fear"
-
-FALLBACK 1: LunarCrush
-──────────────────────
-URL: https://api.lunarcrush.com/v2
-Free tier: Limited
-Docs: https://lunarcrush.com/developers/api
-
-Endpoints:
-• Assets: ?data=assets&key={KEY}
-• Market: ?data=market&key={KEY}
-• Influencers: ?data=influencers&key={KEY}
-
-FALLBACK 2: Santiment (GraphQL)
-────────────────────────────────
-URL: https://api.santiment.net/graphql
-Free tier available
-Docs: https://api.santiment.net/graphiql
-
-GraphQL Example:
-query {
- getMetric(metric: "sentiment_balance_total") {
- timeseriesData(
- slug: "bitcoin"
- from: "2025-10-01T00:00:00Z"
- to: "2025-10-31T00:00:00Z"
- interval: "1d"
- ) {
- datetime
- value
- }
- }
-}
-
-FALLBACK 3: TheTie.io
-─────────────────────
-URL: https://api.thetie.io
-Premium mainly
-Docs: https://docs.thetie.io
-
-FALLBACK 4: CryptoQuant
-───────────────────────
-URL: https://api.cryptoquant.com/v1
-Free tier: Limited
-Docs: https://docs.cryptoquant.com
-
-FALLBACK 5: Glassnode Social
-────────────────────────────
-URL: https://api.glassnode.com/v1/metrics/social
-Free tier: Limited
-Docs: https://docs.glassnode.com
-
-FALLBACK 6: Augmento (Social)
-──────────────────────────────
-URL: https://api.augmento.ai/v1
-AI-powered sentiment
-Free trial available
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🐋 WHALE TRACKING APIs - APIهای ردیابی نهنگها
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: WHALE TRANSACTIONS (8+ endpoints)
-──────────────────────────────────────────────
-
-PRIMARY: Whale Alert
-────────────────────
-URL: https://api.whale-alert.io/v1
-Free: Limited (7-day trial)
-Paid: From $20/month
-Docs: https://docs.whale-alert.io
-
-Endpoints:
-• Transactions: /transactions?api_key={KEY}&min_value=1000000&start={timestamp}&end={timestamp}
-• Status: /status?api_key={KEY}
-
-Example:
-const start = Math.floor(Date.now()/1000) - 3600; // 1 hour ago
-const end = Math.floor(Date.now()/1000);
-fetch(`https://api.whale-alert.io/v1/transactions?api_key=YOUR_KEY&min_value=1000000&start=${start}&end=${end}`)
- .then(r => r.json())
- .then(data => {
- data.transactions.forEach(tx => {
- console.log(`${tx.amount} ${tx.symbol} from ${tx.from.owner} to ${tx.to.owner}`);
- });
- });
-
-FALLBACK 1: ClankApp (FREE)
-───────────────────────────
-URL: https://clankapp.com/api
-Free: بله
-Telegram: @clankapp
-Twitter: @ClankApp
-Docs: https://clankapp.com/api/
-
-Features:
-• 24 blockchains
-• Real-time whale alerts
-• Email & push notifications
-• No API key needed
-
-Example:
-fetch('https://clankapp.com/api/whales/recent')
- .then(r => r.json())
- .then(data => console.log(data));
-
-FALLBACK 2: BitQuery Whale Tracking
-────────────────────────────────────
-URL: https://graphql.bitquery.io
-Free: 10K queries/month
-Docs: https://docs.bitquery.io
-
-GraphQL Example (Large ETH Transfers):
-{
- ethereum(network: ethereum) {
- transfers(
- amount: {gt: 1000}
- currency: {is: "ETH"}
- date: {since: "2025-10-25"}
- ) {
- block { timestamp { time } }
- sender { address }
- receiver { address }
- amount
- transaction { hash }
- }
- }
-}
-
-FALLBACK 3: Arkham Intelligence
-────────────────────────────────
-URL: https://api.arkham.com
-Paid service mainly
-Docs: https://docs.arkham.com
-
-FALLBACK 4: Nansen
-──────────────────
-URL: https://api.nansen.ai/v1
-Premium: Expensive but powerful
-Docs: https://docs.nansen.ai
-
-Features:
-• Smart Money tracking
-• Wallet labeling
-• Multi-chain support
-
-FALLBACK 5: DexCheck Whale Tracker
-───────────────────────────────────
-Free wallet tracking feature
-22 chains supported
-Telegram bot integration
-
-FALLBACK 6: DeBank
-──────────────────
-URL: https://api.debank.com
-Free: Portfolio tracking
-Web3 social features
-
-FALLBACK 7: Zerion API
-──────────────────────
-URL: https://api.zerion.io
-Similar to DeBank
-DeFi portfolio tracker
-
-FALLBACK 8: Whalemap
-────────────────────
-URL: https://whalemap.io
-Bitcoin & ERC-20 focus
-Charts and analytics
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🔍 ON-CHAIN ANALYTICS APIs - APIهای تحلیل زنجیره
-═══════════════════════════════════════════════════════════════════════════════════════
-
-CATEGORY 1: BLOCKCHAIN DATA (10+ endpoints)
-────────────────────────────────────────────
-
-PRIMARY: The Graph (Subgraphs)
-──────────────────────────────
-URL: https://api.thegraph.com/subgraphs/name/{org}/{subgraph}
-Free: Public subgraphs
-Docs: https://thegraph.com/docs/
-
-Popular Subgraphs:
-• Uniswap V3: /uniswap/uniswap-v3
-• Aave V2: /aave/protocol-v2
-• Compound: /graphprotocol/compound-v2
-
-Example (Uniswap V3):
-fetch('https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v3', {
- method: 'POST',
- headers: {'Content-Type': 'application/json'},
- body: JSON.stringify({
- query: `{
- pools(first: 5, orderBy: volumeUSD, orderDirection: desc) {
- id
- token0 { symbol }
- token1 { symbol }
- volumeUSD
- }
- }`
- })
-})
-
-FALLBACK 1: Glassnode
-─────────────────────
-URL: https://api.glassnode.com/v1
-Free tier: Limited metrics
-Docs: https://docs.glassnode.com
-
-Endpoints:
-• SOPR: /metrics/indicators/sopr?a=BTC&api_key={KEY}
-• HODL Waves: /metrics/supply/hodl_waves?a=BTC&api_key={KEY}
-
-FALLBACK 2: IntoTheBlock
-────────────────────────
-URL: https://api.intotheblock.com/v1
-Free tier available
-Docs: https://developers.intotheblock.com
-
-FALLBACK 3: Dune Analytics
-──────────────────────────
-URL: https://api.dune.com/api/v1
-Free: Query results
-Docs: https://docs.dune.com/api-reference/
-
-FALLBACK 4: Covalent
-────────────────────
-URL: https://api.covalenthq.com/v1
-Free tier: 100K credits
-Multi-chain support
-Docs: https://www.covalenthq.com/docs/api/
-
-Example (Ethereum balances):
-fetch('https://api.covalenthq.com/v1/1/address/0x.../balances_v2/?key=YOUR_KEY')
-
-FALLBACK 5: Moralis
-───────────────────
-URL: https://deep-index.moralis.io/api/v2
-Free: 100K compute units/month
-Docs: https://docs.moralis.io
-
-FALLBACK 6: Alchemy NFT API
-───────────────────────────
-Included with Alchemy account
-NFT metadata & transfers
-
-FALLBACK 7: QuickNode Functions
-────────────────────────────────
-Custom on-chain queries
-Token balances, NFTs
-
-FALLBACK 8: Transpose
-─────────────────────
-URL: https://api.transpose.io
-Free tier available
-SQL-like queries
-
-FALLBACK 9: Footprint Analytics
-────────────────────────────────
-URL: https://api.footprint.network
-Free: Community tier
-No-code analytics
-
-FALLBACK 10: Nansen Query
-─────────────────────────
-Premium institutional tool
-Advanced on-chain intelligence
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🔧 COMPLETE JAVASCRIPT IMPLEMENTATION
- پیادهسازی کامل جاوااسکریپت
-═══════════════════════════════════════════════════════════════════════════════════════
-
-// ═══════════════════════════════════════════════════════════════════════════════
-// CONFIG.JS - تنظیمات مرکزی API
-// ═══════════════════════════════════════════════════════════════════════════════
-
-const API_CONFIG = {
- // CORS Proxies (پروکسیهای CORS)
- corsProxies: [
- 'https://api.allorigins.win/get?url=',
- 'https://proxy.cors.sh/',
- 'https://proxy.corsfix.com/?url=',
- 'https://api.codetabs.com/v1/proxy?quest=',
- 'https://thingproxy.freeboard.io/fetch/'
- ],
-
- // Block Explorers (کاوشگرهای بلاکچین)
- explorers: {
- ethereum: {
- primary: {
- name: 'etherscan',
- baseUrl: 'https://api.etherscan.io/api',
- key: 'SZHYFZK2RR8H9TIMJBVW54V4H81K2Z2KR2',
- rateLimit: 5 // calls per second
- },
- fallbacks: [
- { name: 'etherscan2', baseUrl: 'https://api.etherscan.io/api', key: 'T6IR8VJHX2NE6ZJW2S3FDVN1TYG4PYYI45' },
- { name: 'blockchair', baseUrl: 'https://api.blockchair.com/ethereum', key: '' },
- { name: 'blockscout', baseUrl: 'https://eth.blockscout.com/api', key: '' },
- { name: 'ethplorer', baseUrl: 'https://api.ethplorer.io', key: 'freekey' }
- ]
- },
- bsc: {
- primary: {
- name: 'bscscan',
- baseUrl: 'https://api.bscscan.com/api',
- key: 'K62RKHGXTDCG53RU4MCG6XABIMJKTN19IT',
- rateLimit: 5
- },
- fallbacks: [
- { name: 'blockchair', baseUrl: 'https://api.blockchair.com/binance-smart-chain', key: '' },
- { name: 'bitquery', baseUrl: 'https://graphql.bitquery.io', key: '', method: 'graphql' }
- ]
- },
- tron: {
- primary: {
- name: 'tronscan',
- baseUrl: 'https://apilist.tronscanapi.com/api',
- key: '7ae72726-bffe-4e74-9c33-97b761eeea21',
- rateLimit: 10
- },
- fallbacks: [
- { name: 'trongrid', baseUrl: 'https://api.trongrid.io', key: '' },
- { name: 'tronstack', baseUrl: 'https://api.tronstack.io', key: '' },
- { name: 'blockchair', baseUrl: 'https://api.blockchair.com/tron', key: '' }
- ]
- }
- },
-
- // Market Data (دادههای بازار)
- marketData: {
- primary: {
- name: 'coingecko',
- baseUrl: 'https://api.coingecko.com/api/v3',
- key: '', // بدون کلید
- needsProxy: false,
- rateLimit: 50 // calls per minute
- },
- fallbacks: [
- {
- name: 'coinmarketcap',
- baseUrl: 'https://pro-api.coinmarketcap.com/v1',
- key: 'b54bcf4d-1bca-4e8e-9a24-22ff2c3d462c',
- headerKey: 'X-CMC_PRO_API_KEY',
- needsProxy: true
- },
- {
- name: 'coinmarketcap2',
- baseUrl: 'https://pro-api.coinmarketcap.com/v1',
- key: '04cf4b5b-9868-465c-8ba0-9f2e78c92eb1',
- headerKey: 'X-CMC_PRO_API_KEY',
- needsProxy: true
- },
- { name: 'coincap', baseUrl: 'https://api.coincap.io/v2', key: '' },
- { name: 'coinpaprika', baseUrl: 'https://api.coinpaprika.com/v1', key: '' },
- { name: 'binance', baseUrl: 'https://api.binance.com/api/v3', key: '' },
- { name: 'coinlore', baseUrl: 'https://api.coinlore.net/api', key: '' }
- ]
- },
-
- // RPC Nodes (نودهای RPC)
- rpcNodes: {
- ethereum: [
- 'https://eth.llamarpc.com',
- 'https://ethereum.publicnode.com',
- 'https://cloudflare-eth.com',
- 'https://rpc.ankr.com/eth',
- 'https://eth.drpc.org'
- ],
- bsc: [
- 'https://bsc-dataseed.binance.org',
- 'https://bsc-dataseed1.defibit.io',
- 'https://rpc.ankr.com/bsc',
- 'https://bsc-rpc.publicnode.com'
- ],
- polygon: [
- 'https://polygon-rpc.com',
- 'https://rpc.ankr.com/polygon',
- 'https://polygon-bor-rpc.publicnode.com'
- ]
- },
-
- // News Sources (منابع خبری)
- news: {
- primary: {
- name: 'cryptopanic',
- baseUrl: 'https://cryptopanic.com/api/v1',
- key: '',
- needsProxy: false
- },
- fallbacks: [
- { name: 'reddit', baseUrl: 'https://www.reddit.com/r/CryptoCurrency', key: '' }
- ]
- },
-
- // Sentiment (احساسات)
- sentiment: {
- primary: {
- name: 'alternative.me',
- baseUrl: 'https://api.alternative.me/fng',
- key: '',
- needsProxy: false
- }
- },
-
- // Whale Tracking (ردیابی نهنگ)
- whaleTracking: {
- primary: {
- name: 'clankapp',
- baseUrl: 'https://clankapp.com/api',
- key: '',
- needsProxy: false
- }
- }
-};
-
-// ═══════════════════════════════════════════════════════════════════════════════
-// API-CLIENT.JS - کلاینت API با مدیریت خطا و fallback
-// ═══════════════════════════════════════════════════════════════════════════════
-
-class CryptoAPIClient {
- constructor(config) {
- this.config = config;
- this.currentProxyIndex = 0;
- this.requestCache = new Map();
- this.cacheTimeout = 60000; // 1 minute
- }
-
- // استفاده از CORS Proxy
- async fetchWithProxy(url, options = {}) {
- const proxies = this.config.corsProxies;
-
- for (let i = 0; i < proxies.length; i++) {
- const proxyUrl = proxies[this.currentProxyIndex] + encodeURIComponent(url);
-
- try {
- console.log(`🔄 Trying proxy ${this.currentProxyIndex + 1}/${proxies.length}`);
-
- const response = await fetch(proxyUrl, {
- ...options,
- headers: {
- ...options.headers,
- 'Origin': window.location.origin,
- 'x-requested-with': 'XMLHttpRequest'
- }
- });
-
- if (response.ok) {
- const data = await response.json();
- // Handle allOrigins response format
- return data.contents ? JSON.parse(data.contents) : data;
- }
- } catch (error) {
- console.warn(`❌ Proxy ${this.currentProxyIndex + 1} failed:`, error.message);
- }
-
- // Switch to next proxy
- this.currentProxyIndex = (this.currentProxyIndex + 1) % proxies.length;
- }
-
- throw new Error('All CORS proxies failed');
- }
-
- // بدون پروکسی
- async fetchDirect(url, options = {}) {
- try {
- const response = await fetch(url, options);
- if (!response.ok) throw new Error(`HTTP ${response.status}`);
- return await response.json();
- } catch (error) {
- throw new Error(`Direct fetch failed: ${error.message}`);
- }
- }
-
- // با cache و fallback
- async fetchWithFallback(primaryConfig, fallbacks, endpoint, params = {}) {
- const cacheKey = `${primaryConfig.name}-${endpoint}-${JSON.stringify(params)}`;
-
- // Check cache
- if (this.requestCache.has(cacheKey)) {
- const cached = this.requestCache.get(cacheKey);
- if (Date.now() - cached.timestamp < this.cacheTimeout) {
- console.log('📦 Using cached data');
- return cached.data;
- }
- }
-
- // Try primary
- try {
- const data = await this.makeRequest(primaryConfig, endpoint, params);
- this.requestCache.set(cacheKey, { data, timestamp: Date.now() });
- return data;
- } catch (error) {
- console.warn('⚠️ Primary failed, trying fallbacks...', error.message);
- }
-
- // Try fallbacks
- for (const fallback of fallbacks) {
- try {
- console.log(`🔄 Trying fallback: ${fallback.name}`);
- const data = await this.makeRequest(fallback, endpoint, params);
- this.requestCache.set(cacheKey, { data, timestamp: Date.now() });
- return data;
- } catch (error) {
- console.warn(`❌ Fallback ${fallback.name} failed:`, error.message);
- }
- }
-
- throw new Error('All endpoints failed');
- }
-
- // ساخت درخواست
- async makeRequest(apiConfig, endpoint, params = {}) {
- let url = `${apiConfig.baseUrl}${endpoint}`;
-
- // Add query params
- const queryParams = new URLSearchParams();
- if (apiConfig.key) {
- queryParams.append('apikey', apiConfig.key);
- }
- Object.entries(params).forEach(([key, value]) => {
- queryParams.append(key, value);
- });
-
- if (queryParams.toString()) {
- url += '?' + queryParams.toString();
- }
-
- const options = {};
-
- // Add headers if needed
- if (apiConfig.headerKey && apiConfig.key) {
- options.headers = {
- [apiConfig.headerKey]: apiConfig.key
- };
- }
-
- // Use proxy if needed
- if (apiConfig.needsProxy) {
- return await this.fetchWithProxy(url, options);
- } else {
- return await this.fetchDirect(url, options);
- }
- }
-
- // ═══════════════ SPECIFIC API METHODS ═══════════════
-
- // Get ETH Balance (با fallback)
- async getEthBalance(address) {
- const { ethereum } = this.config.explorers;
- return await this.fetchWithFallback(
- ethereum.primary,
- ethereum.fallbacks,
- '',
- {
- module: 'account',
- action: 'balance',
- address: address,
- tag: 'latest'
- }
- );
- }
-
- // Get BTC Price (multi-source)
- async getBitcoinPrice() {
- const { marketData } = this.config;
-
- try {
- // Try CoinGecko first (no key needed, no CORS)
- const data = await this.fetchDirect(
- `${marketData.primary.baseUrl}/simple/price?ids=bitcoin&vs_currencies=usd,eur`
- );
- return {
- source: 'CoinGecko',
- usd: data.bitcoin.usd,
- eur: data.bitcoin.eur
- };
- } catch (error) {
- // Fallback to Binance
- try {
- const data = await this.fetchDirect(
- 'https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT'
- );
- return {
- source: 'Binance',
- usd: parseFloat(data.price),
- eur: null
- };
- } catch (err) {
- throw new Error('All price sources failed');
- }
- }
- }
-
- // Get Fear & Greed Index
- async getFearGreed() {
- const url = `${this.config.sentiment.primary.baseUrl}/?limit=1`;
- const data = await this.fetchDirect(url);
- return {
- value: parseInt(data.data[0].value),
- classification: data.data[0].value_classification,
- timestamp: new Date(parseInt(data.data[0].timestamp) * 1000)
- };
- }
-
- // Get Trending Coins
- async getTrendingCoins() {
- const url = `${this.config.marketData.primary.baseUrl}/search/trending`;
- const data = await this.fetchDirect(url);
- return data.coins.map(item => ({
- id: item.item.id,
- name: item.item.name,
- symbol: item.item.symbol,
- rank: item.item.market_cap_rank,
- thumb: item.item.thumb
- }));
- }
-
- // Get Crypto News
- async getCryptoNews(limit = 10) {
- const url = `${this.config.news.primary.baseUrl}/posts/?public=true`;
- const data = await this.fetchDirect(url);
- return data.results.slice(0, limit).map(post => ({
- title: post.title,
- url: post.url,
- source: post.source.title,
- published: new Date(post.published_at)
- }));
- }
-
- // Get Recent Whale Transactions
- async getWhaleTransactions() {
- try {
- const url = `${this.config.whaleTracking.primary.baseUrl}/whales/recent`;
- return await this.fetchDirect(url);
- } catch (error) {
- console.warn('Whale API not available');
- return [];
- }
- }
-
- // Multi-source price aggregator
- async getAggregatedPrice(symbol) {
- const sources = [
- {
- name: 'CoinGecko',
- fetch: async () => {
- const data = await this.fetchDirect(
- `${this.config.marketData.primary.baseUrl}/simple/price?ids=${symbol}&vs_currencies=usd`
- );
- return data[symbol]?.usd;
- }
- },
- {
- name: 'Binance',
- fetch: async () => {
- const data = await this.fetchDirect(
- `https://api.binance.com/api/v3/ticker/price?symbol=${symbol.toUpperCase()}USDT`
- );
- return parseFloat(data.price);
- }
- },
- {
- name: 'CoinCap',
- fetch: async () => {
- const data = await this.fetchDirect(
- `https://api.coincap.io/v2/assets/${symbol}`
- );
- return parseFloat(data.data.priceUsd);
- }
- }
- ];
-
- const prices = await Promise.allSettled(
- sources.map(async source => ({
- source: source.name,
- price: await source.fetch()
- }))
- );
-
- const successful = prices
- .filter(p => p.status === 'fulfilled')
- .map(p => p.value);
-
- if (successful.length === 0) {
- throw new Error('All price sources failed');
- }
-
- const avgPrice = successful.reduce((sum, p) => sum + p.price, 0) / successful.length;
-
- return {
- symbol,
- sources: successful,
- average: avgPrice,
- spread: Math.max(...successful.map(p => p.price)) - Math.min(...successful.map(p => p.price))
- };
- }
-}
-
-// ═══════════════════════════════════════════════════════════════════════════════
-// USAGE EXAMPLES - مثالهای استفاده
-// ═══════════════════════════════════════════════════════════════════════════════
-
-// Initialize
-const api = new CryptoAPIClient(API_CONFIG);
-
-// Example 1: Get Ethereum Balance
-async function example1() {
- try {
- const address = '0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb';
- const balance = await api.getEthBalance(address);
- console.log('ETH Balance:', parseInt(balance.result) / 1e18);
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 2: Get Bitcoin Price from Multiple Sources
-async function example2() {
- try {
- const price = await api.getBitcoinPrice();
- console.log(`BTC Price (${price.source}): $${price.usd}`);
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 3: Get Fear & Greed Index
-async function example3() {
- try {
- const fng = await api.getFearGreed();
- console.log(`Fear & Greed: ${fng.value} (${fng.classification})`);
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 4: Get Trending Coins
-async function example4() {
- try {
- const trending = await api.getTrendingCoins();
- console.log('Trending Coins:');
- trending.forEach((coin, i) => {
- console.log(`${i + 1}. ${coin.name} (${coin.symbol})`);
- });
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 5: Get Latest News
-async function example5() {
- try {
- const news = await api.getCryptoNews(5);
- console.log('Latest News:');
- news.forEach((article, i) => {
- console.log(`${i + 1}. ${article.title} - ${article.source}`);
- });
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 6: Aggregate Price from Multiple Sources
-async function example6() {
- try {
- const priceData = await api.getAggregatedPrice('bitcoin');
- console.log('Price Sources:');
- priceData.sources.forEach(s => {
- console.log(`- ${s.source}: $${s.price.toFixed(2)}`);
- });
- console.log(`Average: $${priceData.average.toFixed(2)}`);
- console.log(`Spread: $${priceData.spread.toFixed(2)}`);
- } catch (error) {
- console.error('Error:', error.message);
- }
-}
-
-// Example 7: Dashboard - All Data
-async function dashboardExample() {
- console.log('🚀 Loading Crypto Dashboard...\n');
-
- try {
- // Price
- const btcPrice = await api.getBitcoinPrice();
- console.log(`💰 BTC: $${btcPrice.usd.toLocaleString()}`);
-
- // Fear & Greed
- const fng = await api.getFearGreed();
- console.log(`😱 Fear & Greed: ${fng.value} (${fng.classification})`);
-
- // Trending
- const trending = await api.getTrendingCoins();
- console.log(`\n🔥 Trending:`);
- trending.slice(0, 3).forEach((coin, i) => {
- console.log(` ${i + 1}. ${coin.name}`);
- });
-
- // News
- const news = await api.getCryptoNews(3);
- console.log(`\n📰 Latest News:`);
- news.forEach((article, i) => {
- console.log(` ${i + 1}. ${article.title.substring(0, 50)}...`);
- });
-
- } catch (error) {
- console.error('Dashboard Error:', error.message);
- }
-}
-
-// Run examples
-console.log('═══════════════════════════════════════');
-console.log(' CRYPTO API CLIENT - TEST SUITE');
-console.log('═══════════════════════════════════════\n');
-
-// Uncomment to run specific examples:
-// example1();
-// example2();
-// example3();
-// example4();
-// example5();
-// example6();
-dashboardExample();
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 📝 QUICK REFERENCE - مرجع سریع
-═══════════════════════════════════════════════════════════════════════════════════════
-
-BEST FREE APIs (بهترین APIهای رایگان):
-─────────────────────────────────────────
-
-✅ PRICES & MARKET DATA:
- 1. CoinGecko (بدون کلید، بدون CORS)
- 2. Binance Public API (بدون کلید)
- 3. CoinCap (بدون کلید)
- 4. CoinPaprika (بدون کلید)
-
-✅ BLOCK EXPLORERS:
- 1. Blockchair (1,440 req/day)
- 2. BlockScout (بدون محدودیت)
- 3. Public RPC nodes (various)
-
-✅ NEWS:
- 1. CryptoPanic (بدون کلید)
- 2. Reddit JSON API (60 req/min)
-
-✅ SENTIMENT:
- 1. Alternative.me F&G (بدون محدودیت)
-
-✅ WHALE TRACKING:
- 1. ClankApp (بدون کلید)
- 2. BitQuery GraphQL (10K/month)
-
-✅ RPC NODES:
- 1. PublicNode (همه شبکهها)
- 2. Ankr (عمومی)
- 3. LlamaNodes (بدون ثبتنام)
-
-
-RATE LIMIT STRATEGIES (استراتژیهای محدودیت):
-───────────────────────────────────────────────
-
-1. کش کردن (Caching):
- - ذخیره نتایج برای 1-5 دقیقه
- - استفاده از localStorage برای کش مرورگر
-
-2. چرخش کلید (Key Rotation):
- - استفاده از چندین کلید API
- - تعویض خودکار در صورت محدودیت
-
-3. Fallback Chain:
- - Primary → Fallback1 → Fallback2
- - تا 5-10 جایگزین برای هر سرویس
-
-4. Request Queuing:
- - صف بندی درخواستها
- - تاخیر بین درخواستها
-
-5. Multi-Source Aggregation:
- - دریافت از چند منبع همزمان
- - میانگین گیری نتایج
-
-
-ERROR HANDLING (مدیریت خطا):
-──────────────────────────────
-
-try {
- const data = await api.fetchWithFallback(primary, fallbacks, endpoint, params);
-} catch (error) {
- if (error.message.includes('rate limit')) {
- // Switch to fallback
- } else if (error.message.includes('CORS')) {
- // Use CORS proxy
- } else {
- // Show error to user
- }
-}
-
-
-DEPLOYMENT TIPS (نکات استقرار):
-─────────────────────────────────
-
-1. Backend Proxy (توصیه میشود):
- - Node.js/Express proxy server
- - Cloudflare Worker
- - Vercel Serverless Function
-
-2. Environment Variables:
- - ذخیره کلیدها در .env
- - عدم نمایش در کد فرانتاند
-
-3. Rate Limiting:
- - محدودسازی درخواست کاربر
- - استفاده از Redis برای کنترل
-
-4. Monitoring:
- - لاگ گرفتن از خطاها
- - ردیابی استفاده از API
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- 🔗 USEFUL LINKS - لینکهای مفید
-═══════════════════════════════════════════════════════════════════════════════════════
-
-DOCUMENTATION:
-• CoinGecko API: https://www.coingecko.com/api/documentation
-• Etherscan API: https://docs.etherscan.io
-• BscScan API: https://docs.bscscan.com
-• TronGrid: https://developers.tron.network
-• Alchemy: https://docs.alchemy.com
-• Infura: https://docs.infura.io
-• The Graph: https://thegraph.com/docs
-• BitQuery: https://docs.bitquery.io
-
-CORS PROXY ALTERNATIVES:
-• CORS Anywhere: https://github.com/Rob--W/cors-anywhere
-• AllOrigins: https://github.com/gnuns/allOrigins
-• CORS.SH: https://cors.sh
-• Corsfix: https://corsfix.com
-
-RPC LISTS:
-• ChainList: https://chainlist.org
-• Awesome RPC: https://github.com/arddluma/awesome-list-rpc-nodes-providers
-
-TOOLS:
-• Postman: https://www.postman.com
-• Insomnia: https://insomnia.rest
-• GraphiQL: https://graphiql-online.com
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- ⚠️ IMPORTANT NOTES - نکات مهم
-═══════════════════════════════════════════════════════════════════════════════════════
-
-1. ⚠️ NEVER expose API keys in frontend code
- - همیشه از backend proxy استفاده کنید
- - کلیدها را در environment variables ذخیره کنید
-
-2. 🔄 Always implement fallbacks
- - حداقل 2-3 جایگزین برای هر سرویس
- - تست منظم fallbackها
-
-3. 💾 Cache responses when possible
- - صرفهجویی در استفاده از API
- - سرعت بیشتر برای کاربر
-
-4. 📊 Monitor API usage
- - ردیابی تعداد درخواستها
- - هشدار قبل از رسیدن به محدودیت
-
-5. 🔐 Secure your endpoints
- - محدودسازی domain
- - استفاده از CORS headers
- - Rate limiting برای کاربران
-
-6. 🌐 Test with and without CORS proxies
- - برخی APIها CORS را پشتیبانی میکنند
- - استفاده از پروکسی فقط در صورت نیاز
-
-7. 📱 Mobile-friendly implementations
- - بهینهسازی برای شبکههای ضعیف
- - کاهش اندازه درخواستها
-
-
-═══════════════════════════════════════════════════════════════════════════════════════
- END OF CONFIGURATION FILE
- پایان فایل تنظیمات
-═══════════════════════════════════════════════════════════════════════════════════════
-
-Last Updated: October 31, 2025
-Version: 2.0
-Author: AI Assistant
-License: Free to use
-
-For updates and more resources, check:
-- GitHub: Search for "awesome-crypto-apis"
-- Reddit: r/CryptoCurrency, r/ethdev
-- Discord: Web3 developer communities
\ No newline at end of file
diff --git a/app/DASHBOARD_READY.txt b/app/DASHBOARD_READY.txt
deleted file mode 100644
index 8716d977cb55359dd266bc254487521a40ad5db7..0000000000000000000000000000000000000000
--- a/app/DASHBOARD_READY.txt
+++ /dev/null
@@ -1,81 +0,0 @@
-========================================
- 🎉 YOUR DASHBOARD IS READY!
-========================================
-
-📍 OPEN IN BROWSER:
- http://localhost:7860/
-
-========================================
-
-✨ WHAT YOU'LL SEE:
-
-🎨 BEAUTIFUL DARK THEME
- - Professional gradient colors (blue/purple)
- - Smooth animations
- - Modern design
-
-📊 OVERVIEW TAB (Default)
- - 4 big stat cards showing:
- * Total Providers
- * Online count
- * Degraded count
- * Offline count
- - Recent provider status
- - System health summary
-
-🔌 PROVIDERS TAB
- - All providers in a grid
- - Search box to filter
- - Color coded:
- * Green border = Online
- * Orange border = Degraded
- * Red border = Offline
- - Shows response time
-
-📁 CATEGORIES TAB
- - All categories listed
- - Stats for each category
- - Online/Degraded/Offline breakdown
-
-💰 MARKET DATA TAB
- - Live cryptocurrency prices
- - 24h price changes
- - Green = up, Red = down
-
-❤️ HEALTH TAB
- - Uptime percentage
- - Average response time
- - Detailed health report
- - Lists of online/offline providers
-
-========================================
-
-🎯 FEATURES:
-
-✅ Auto-refresh every 30 seconds
-✅ Search providers
-✅ Export data to JSON
-✅ Fully responsive (mobile-friendly)
-✅ No overlapping elements
-✅ Fast and smooth
-✅ All in ONE file (complete_dashboard.html)
-
-========================================
-
-🚀 READY FOR HUGGING FACE:
-
-This dashboard will work perfectly when you
-deploy to Hugging Face Spaces!
-
-Just:
-1. Upload all files
-2. Push to HF
-3. Your dashboard will be live!
-
-========================================
-
-💡 TIP: Press Ctrl+Shift+R for hard refresh
- if you don't see changes immediately
-
-========================================
-
diff --git a/app/DEPLOYMENT_INSTRUCTIONS.md b/app/DEPLOYMENT_INSTRUCTIONS.md
deleted file mode 100644
index eb37d8190d7a01d3478d268e289739cd77d3cafd..0000000000000000000000000000000000000000
--- a/app/DEPLOYMENT_INSTRUCTIONS.md
+++ /dev/null
@@ -1,257 +0,0 @@
-# 🚀 DEPLOYMENT INSTRUCTIONS FOR HUGGING FACE SPACES
-
-## ✅ DEPLOYMENT STATUS: READY FOR PRODUCTION
-
-All critical blockers have been resolved. This application is now ready for deployment to Hugging Face Spaces.
-
----
-
-## 📋 PRE-DEPLOYMENT CHECKLIST
-
-✅ **Dependencies**: All required packages listed in `requirements.txt` with pinned versions
-✅ **Dockerfile**: Properly configured with directory creation and PORT variable support
-✅ **Mock Data Removed**: All endpoints use real data providers or return explicit error codes
-✅ **USE_MOCK_DATA Flag**: Implemented for optional testing/demo mode
-✅ **Database Integration**: SQLite database properly integrated with price history
-✅ **Provider Management**: Circuit breaker and failover logic implemented
-✅ **Error Handling**: All endpoints return proper HTTP status codes (503/501) on failures
-
----
-
-## 🛠️ LOCAL TESTING
-
-### 1. Build Docker Image
-
-```bash
-docker build -t crypto-monitor .
-```
-
-### 2. Run Container Locally
-
-```bash
-# Default mode (real data, port 7860)
-docker run -p 7860:7860 crypto-monitor
-
-# With custom port
-docker run -p 8000:8000 -e PORT=8000 crypto-monitor
-
-# With mock data enabled (for testing)
-docker run -p 7860:7860 -e USE_MOCK_DATA=true crypto-monitor
-```
-
-### 3. Test Key Endpoints
-
-```bash
-# Health check
-curl http://localhost:7860/health
-
-# Market data (real CoinGecko)
-curl http://localhost:7860/api/market
-
-# Market history (from database)
-curl "http://localhost:7860/api/market/history?symbol=BTC&limit=10"
-
-# Sentiment data (real Alternative.me)
-curl http://localhost:7860/api/sentiment
-
-# Trending coins (real CoinGecko)
-curl http://localhost:7860/api/trending
-
-# DeFi data (returns 503 - not implemented)
-curl http://localhost:7860/api/defi
-
-# HF Sentiment (returns 501 - not implemented)
-curl -X POST http://localhost:7860/api/hf/run-sentiment \
- -H "Content-Type: application/json" \
- -d '{"texts": ["Bitcoin is bullish"]}'
-```
-
----
-
-## 🌐 HUGGING FACE SPACES DEPLOYMENT
-
-### Step 1: Create New Space
-
-1. Go to https://huggingface.co/spaces
-2. Click **"Create new Space"**
-3. Choose:
- - **Space name**: `crypto-data-aggregator`
- - **License**: `MIT`
- - **Space SDK**: `Docker`
- - **Visibility**: `Public` (or Private)
-
-### Step 2: Configure Repository
-
-Push your code to the Hugging Face Space repository:
-
-```bash
-git remote add hf https://huggingface.co/spaces/YOUR_USERNAME/crypto-data-aggregator
-git push hf main
-```
-
-### Step 3: Configure Environment Variables (Optional)
-
-In your Space settings, add environment variables:
-
-| Variable | Value | Description |
-|----------|-------|-------------|
-| `PORT` | `7860` | **Auto-set by HF Spaces** - Do not override |
-| `USE_MOCK_DATA` | `false` | Use real data providers (default) |
-| `ENABLE_AUTO_DISCOVERY` | `false` | Disable auto-discovery service |
-
-⚠️ **Important**: Hugging Face Spaces automatically sets `PORT=7860`. Our Dockerfile is configured to use this.
-
-### Step 4: Monitor Deployment
-
-1. Watch the build logs in your Space
-2. Wait for "Running" status (typically 2-5 minutes)
-3. Access your app at: `https://huggingface.co/spaces/YOUR_USERNAME/crypto-data-aggregator`
-
----
-
-## 🔍 POST-DEPLOYMENT VERIFICATION
-
-### Automated Health Check
-
-```bash
-SPACE_URL="https://YOUR_USERNAME-crypto-data-aggregator.hf.space"
-
-# 1. Health check
-curl $SPACE_URL/health | jq
-
-# 2. Market data
-curl $SPACE_URL/api/market | jq '.cryptocurrencies[] | {name, symbol, price}'
-
-# 3. Sentiment
-curl $SPACE_URL/api/sentiment | jq '.fear_greed_index'
-
-# 4. Trending
-curl $SPACE_URL/api/trending | jq '.trending[0:3]'
-
-# 5. Market history
-curl "$SPACE_URL/api/market/history?symbol=BTC&limit=5" | jq
-```
-
-### Expected Responses
-
-#### ✅ Success Responses:
-
-- `/health`: `{"status": "healthy", "providers_count": N, "online_count": N}`
-- `/api/market`: Array of cryptocurrencies with real prices from CoinGecko
-- `/api/sentiment`: Fear & Greed Index from Alternative.me
-- `/api/trending`: Top trending coins from CoinGecko
-- `/api/market/history`: Array of historical price records from database
-
-#### ⚠️ Expected "Not Implemented" Responses:
-
-- `/api/defi`: HTTP 503 with message about requiring DefiLlama integration
-- `/api/hf/run-sentiment`: HTTP 501 with message about ML models not loaded
-
----
-
-## 🐛 TROUBLESHOOTING
-
-### Issue: Container fails to start
-
-**Check:**
-```bash
-docker logs
-```
-
-**Common causes:**
-- Missing dependencies in `requirements.txt`
-- Syntax errors in Python files
-- Missing required directories (should be auto-created)
-
-### Issue: Endpoints return 503 errors
-
-**Possible causes:**
-1. External API rate limits hit (CoinGecko, Alternative.me)
-2. Network connectivity issues
-3. Provider configuration errors
-
-**Solution:**
-- Check logs: `/api/logs/errors`
-- Enable mock mode temporarily: `USE_MOCK_DATA=true`
-- Wait 1-2 minutes for circuit breakers to reset
-
-### Issue: Database errors
-
-**Check:**
-- Ensure `data/` directory is writable
-- Check database file exists: `ls -la data/database/`
-- Review database logs in `/api/logs`
-
----
-
-## 📊 MONITORING & MAINTENANCE
-
-### Key Metrics to Monitor
-
-1. **Provider Health**: `/api/providers` - Check success rates
-2. **System Status**: `/api/status` - Overall system health
-3. **Error Logs**: `/api/logs/errors` - Recent failures
-4. **Database Stats**: Query `/api/market/history` for data freshness
-
-### Regular Maintenance
-
-- **Daily**: Check `/api/status` for provider health
-- **Weekly**: Review `/api/logs/stats` for error trends
-- **Monthly**: Clean old database records (auto-cleanup configured)
-
----
-
-## 🔐 SECURITY NOTES
-
-✅ **No API Keys Required**: All data sources use free public endpoints
-✅ **No Authentication Needed**: Public read-only data
-✅ **Rate Limiting**: Implemented in provider management
-✅ **Circuit Breakers**: Automatic failover prevents cascading failures
-
----
-
-## 📝 ENVIRONMENT VARIABLES REFERENCE
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `PORT` | `7860` | Server port (set by HF Spaces) |
-| `USE_MOCK_DATA` | `false` | Enable mock data mode for testing |
-| `ENABLE_AUTO_DISCOVERY` | `false` | Enable automatic resource discovery |
-| `PYTHONUNBUFFERED` | `1` | Enable real-time logs |
-
----
-
-## 🎯 SUCCESS CRITERIA
-
-Your deployment is **SUCCESSFUL** if:
-
-✅ Health endpoint returns `"status": "healthy"`
-✅ Market data shows real Bitcoin/Ethereum prices
-✅ Sentiment shows current Fear & Greed Index
-✅ Trending shows actual trending coins
-✅ No hardcoded mock data in responses (unless `USE_MOCK_DATA=true`)
-✅ DeFi and HF endpoints return proper 503/501 errors
-✅ Database history accumulates over time
-
----
-
-## 📞 SUPPORT
-
-If you encounter issues:
-
-1. Check logs: `docker logs ` or HF Space logs
-2. Review error endpoints: `/api/logs/errors`
-3. Run diagnostics: `/api/diagnostics/run`
-4. Enable mock mode for testing: `USE_MOCK_DATA=true`
-
----
-
-## 🎉 DEPLOYMENT COMPLETE
-
-Once all verification steps pass, your crypto data aggregator is **LIVE** and ready for production use!
-
-**Next Steps:**
-- Share your Space URL
-- Monitor initial usage patterns
-- Set up optional monitoring dashboards
-- Consider adding more data providers for redundancy
diff --git a/app/DEPLOYMENT_MASTER_GUIDE.md b/app/DEPLOYMENT_MASTER_GUIDE.md
deleted file mode 100644
index 5d1a150dd760db5d3b195f5083b4c5735917881f..0000000000000000000000000000000000000000
--- a/app/DEPLOYMENT_MASTER_GUIDE.md
+++ /dev/null
@@ -1,604 +0,0 @@
-# 🚀 Crypto-DT-Source: Master Deployment Guide
-## Season 2025 Full Activation for HuggingFace Spaces
-
-**Status:** ✅ Ready for Implementation
-**Prepared by:** Claude Code AI
-**Date:** November 15, 2025
-**Target:** Production deployment with all features activated
-
----
-
-## 📖 Overview
-
-Your Crypto-DT-Source project has been thoroughly audited and is ready for complete activation. This master guide consolidates everything you need to transform it from a monitoring platform into a **fully-functional cryptocurrency data aggregation service** on HuggingFace Spaces.
-
-### What's Included
-
-This deployment package provides **3 complementary guides**:
-
-1. **HUGGINGFACE_DEPLOYMENT_PROMPT.md** (65KB)
- - Comprehensive technical specification
- - Detailed implementation for each feature
- - Code examples and patterns
- - Environment configuration
- - **Best for:** Understanding requirements and implementation details
-
-2. **IMPLEMENTATION_ROADMAP.md** (40KB)
- - Step-by-step 2-3 week timeline
- - Day-by-day task breakdown
- - Testing protocols
- - Success metrics
- - **Best for:** Following structured implementation plan
-
-3. **QUICK_REFERENCE_GUIDE.md** (25KB)
- - Command reference
- - Common troubleshooting
- - File locations
- - Debugging tips
- - **Best for:** Quick lookup during implementation
-
----
-
-## 🎯 What You'll Achieve
-
-After following this guide, your system will have:
-
-### ✅ Real Cryptocurrency Data
-- Live price data for 1000+ cryptocurrencies
-- OHLCV (candlestick) historical data
-- DeFi protocol TVL tracking
-- Trending coins monitoring
-- Multi-provider failover system
-
-### ✅ Intelligent Data Persistence
-- SQLite database storing 90 days of history
-- Automatic data cleanup and archival
-- Fast queries for historical data
-- Backup and restoration capabilities
-
-### ✅ AI-Powered Analysis
-- Real HuggingFace sentiment analysis (not keyword matching)
-- Crypto-specific sentiment classification
-- Automated news analysis pipeline
-- Fear & Greed index integration
-
-### ✅ Enterprise Security
-- JWT token authentication
-- API key management system
-- Multi-tier rate limiting (Free/Pro/Enterprise)
-- Request auditing and monitoring
-
-### ✅ Real-Time Streaming
-- WebSocket live price updates
-- Broadcast-based notifications
-- Multi-client connection support
-- Heartbeat mechanism for reliability
-
-### ✅ Automatic Operations
-- Background data collection every 5 minutes
-- Continuous health monitoring
-- Automatic provider failover
-- Self-healing capabilities
-- Provider auto-discovery (optional)
-
-### ✅ Production Monitoring
-- Comprehensive health checks
-- System diagnostics with auto-fix
-- Performance metrics collection
-- Error tracking and reporting
-- Full operational visibility
-
-### ✅ Cloud Deployment
-- Docker containerization
-- HuggingFace Spaces optimization
-- Auto-scaling ready
-- CI/CD pipeline prepared
-- Zero-downtime deployment
-
----
-
-## 📋 Current Project Status
-
-### ✅ Already Implemented (No Changes Needed)
-```
-Core Infrastructure:
- ✅ FastAPI web framework (50+ endpoints)
- ✅ WebSocket support with connection management
- ✅ Provider management system with circuit breakers
- ✅ Multi-tier logging system
- ✅ Configuration management
- ✅ Database schema and migrations
- ✅ Docker containerization
- ✅ Error handling and graceful degradation
-
-Systems Ready:
- ✅ Health checking infrastructure
- ✅ Pool management with 5 rotation strategies
- ✅ Resource import/export
- ✅ Diagnostic and auto-repair capabilities
- ✅ Session management
- ✅ Broadcasting infrastructure
-```
-
-### ⚠️ Needs Completion (Covered in This Guide)
-```
-Data Integration:
- ❌ Market data endpoints (currently mock) → REAL DATA
- ❌ Price endpoints (currently mock) → REAL DATA
- ❌ Sentiment endpoints (currently mock) → REAL ML MODELS
- ❌ DeFi endpoints (currently mock) → REAL DATA
-
-Database:
- ⚠️ Schema exists but not actively used → ACTIVATE & INTEGRATE
- ⚠️ Migrations ready but not run → EXECUTE MIGRATIONS
- ⚠️ No data persistence in API → WIRE UP DATA STORAGE
-
-Security:
- ❌ No authentication → IMPLEMENT JWT + API KEYS
- ❌ No rate limiting → IMPLEMENT MULTI-TIER LIMITS
-
-Background Tasks:
- ⚠️ Framework ready but not all activated → ACTIVATE ALL TASKS
-
-AI Models:
- ❌ Sentiment analysis uses keyword matching → LOAD REAL MODELS
- ❌ Models not initialized → LOAD ON STARTUP
-```
-
----
-
-## 🚀 Quick Start (Choose Your Path)
-
-### Path 1: Follow Structured Timeline (Recommended)
-**Best if:** You want step-by-step guidance and clear milestones
-
-1. Read `IMPLEMENTATION_ROADMAP.md` (entire document)
-2. Follow Week 1 → Week 3 timeline
-3. Use `QUICK_REFERENCE_GUIDE.md` for commands/debugging
-4. Reference `HUGGINGFACE_DEPLOYMENT_PROMPT.md` for details
-
-**Time:** 2-3 weeks
-**Effort:** Medium
-**Result:** Fully complete, battle-tested implementation
-
-### Path 2: Implement Based on Requirements
-**Best if:** You're familiar with the codebase and want flexibility
-
-1. Review `HUGGINGFACE_DEPLOYMENT_PROMPT.md` sections 1-5
-2. Pick implementation order (start with Phase 1)
-3. Refer to code examples in prompt
-4. Use Quick Reference for troubleshooting
-
-**Time:** 1-2 weeks (if experienced)
-**Effort:** High
-**Result:** Same as Path 1, with personalized approach
-
-### Path 3: Auto-Implementation (If Available)
-**Best if:** You want Claude to implement most changes
-
-1. Share this guide with Claude Code
-2. Request implementation of each phase
-3. Claude implements + tests + commits
-4. You review + approve + deploy
-
-**Time:** 1 week
-**Effort:** Low
-**Result:** Complete + tested system
-
----
-
-## 📊 Success Metrics
-
-Track these KPIs to verify deployment success:
-
-| Metric | Target | How to Test |
-|--------|--------|-----------|
-| **Functionality** | | |
-| Endpoints return real data | 100% | `curl /api/prices` |
-| Database persistence | 100% | Check data after restart |
-| Sentiment analysis | Works with real models | `POST /api/sentiment/analyze` |
-| WebSocket updates | Real-time | Subscribe to `/ws` |
-| **Performance** | | |
-| API response time | < 500ms (p95) | Load test 100 req/s |
-| Sentiment inference | < 2s | Time model.predict() |
-| Database query | < 100ms | Query 30-day history |
-| WebSocket latency | < 1s | Measure round-trip |
-| **Reliability** | | |
-| Uptime | > 99.9% | Monitor /api/health |
-| Provider failover | < 2s | Kill primary provider |
-| Error rate | < 0.1% | Monitor error logs |
-| Memory usage | < 1GB | Check during operation |
-| **Security** | | |
-| Authentication | Required on protected endpoints | Test without token |
-| Rate limiting | Enforced | Send 100 requests |
-| API keys | Validated | Test invalid keys |
-| Database backup | Automated | Verify daily backups |
-
----
-
-## 🔄 Implementation Flow
-
-```
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 1: DATA INTEGRATION (Days 1-5) │
-├─────────────────────────────────────────────────────────┤
-│ Replace mock endpoints with real API calls │
-│ ✅ /api/market (CoinGecko real data) │
-│ ✅ /api/prices (Multiple providers) │
-│ ✅ /api/trending (Real trending data) │
-│ ✅ /api/ohlcv (Binance candlestick data) │
-│ ✅ /api/defi (DeFi Llama TVL) │
-│ ✅ Add caching layer (5-30 min TTL) │
-└─────────────────────────────────────────────────────────┘
- ⬇
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 2: DATABASE (Days 6-10) │
-├─────────────────────────────────────────────────────────┤
-│ Activate persistent storage │
-│ ✅ Run database migrations │
-│ ✅ Wire up data write operations │
-│ ✅ Create historical data read endpoints │
-│ ✅ Implement cleanup/archival │
-│ ✅ Test persistence │
-└─────────────────────────────────────────────────────────┘
- ⬇
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 3: AI & SENTIMENT (Days 8-10) │
-├─────────────────────────────────────────────────────────┤
-│ Load real HuggingFace models │
-│ ✅ Install torch + transformers │
-│ ✅ Load distilbert sentiment model │
-│ ✅ Create sentiment endpoints │
-│ ✅ Implement news analysis pipeline │
-│ ✅ Store sentiment in database │
-└─────────────────────────────────────────────────────────┘
- ⬇
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 4: SECURITY (Days 11-12) │
-├─────────────────────────────────────────────────────────┤
-│ Add authentication & rate limiting │
-│ ✅ Implement JWT token system │
-│ ✅ Create API key management │
-│ ✅ Add rate limiting (Free/Pro tiers) │
-│ ✅ Protect sensitive endpoints │
-│ ✅ Test security flow │
-└─────────────────────────────────────────────────────────┘
- ⬇
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 5: OPERATIONS (Days 13) │
-├─────────────────────────────────────────────────────────┤
-│ Complete monitoring & diagnostics │
-│ ✅ Enhance health checks │
-│ ✅ Create diagnostic endpoints │
-│ ✅ Set up metrics collection │
-│ ✅ Test auto-repair capabilities │
-└─────────────────────────────────────────────────────────┘
- ⬇
-┌─────────────────────────────────────────────────────────┐
-│ PHASE 6: DEPLOYMENT (Days 14-15) │
-├─────────────────────────────────────────────────────────┤
-│ Deploy to HuggingFace Spaces │
-│ ✅ Create spaces/ directory │
-│ ✅ Configure for HF environment │
-│ ✅ Test Docker locally │
-│ ✅ Push to HF Spaces │
-│ ✅ Verify all endpoints │
-│ ✅ Set up monitoring │
-└─────────────────────────────────────────────────────────┘
- ⬇
- ✅ PRODUCTION READY ✅
-```
-
----
-
-## 🛠️ Essential Tools & Commands
-
-### Setup
-```bash
-# Clone and setup
-cd /home/user/crypto-dt-source
-git checkout claude/connect-real-crypto-data-01Tr1xzVJ2MUmucjCR1hgHNm
-
-# Install dependencies
-pip install -r requirements.txt
-pip install torch transformers huggingface-hub slowapi
-
-# Configure environment
-cp .env.example .env
-# Edit .env with your settings
-```
-
-### Development
-```bash
-# Start server
-python api_server_extended.py
-
-# In another terminal - test
-curl http://localhost:8000/api/health | jq
-curl http://localhost:8000/api/prices?symbols=BTC | jq
-
-# View documentation
-open http://localhost:8000/docs
-```
-
-### Deployment
-```bash
-# Test Docker locally
-docker build -f Dockerfile .
-docker run -p 8000:8000 crypto-dt-source
-
-# Deploy to HF Spaces
-git remote add spaces https://huggingface.co/spaces/your-username/crypto-dt-source
-git push spaces claude/connect-real-crypto-data-01Tr1xzVJ2MUmucjCR1hgHNm:main
-
-# Monitor
-curl https://your-space-url/api/health
-```
-
----
-
-## ⚠️ Critical Considerations
-
-### For HuggingFace Spaces
-```
-❌ Space has limited resources:
- - RAM: ~7GB
- - Disk: ~50GB
- - CPU: 2-core
- - GPU: None (or optional paid)
-
-✅ Mitigation:
- - Use distilbert (small sentiment model)
- - Implement aggressive caching
- - Archive old data (keep 30-90 days only)
- - Limit WebSocket connections (100-200 max)
- - Monitor memory constantly
-```
-
-### Performance Constraints
-```
-⚠️ HF Spaces has network limits:
- - Rate limiting on external API calls
- - Bandwidth constraints
- - Concurrent request limits
-
-✅ Solutions:
- - Cache aggressively (TTL-based)
- - Batch external API calls
- - Implement connection pooling
- - Use async/await everywhere
-```
-
-### Data Management
-```
-⚠️ SQLite has limits in shared environment:
- - Max 4GB file size
- - Poor with heavy concurrent writes
- - No distributed locking
-
-✅ Solutions:
- - Archive data to cloud storage
- - Keep only 90 days
- - Use WAL mode for better concurrency
- - Implement data cleanup
-```
-
----
-
-## 📞 Getting Help
-
-### While Implementing
-1. Check `QUICK_REFERENCE_GUIDE.md` for common issues
-2. Review code examples in `HUGGINGFACE_DEPLOYMENT_PROMPT.md`
-3. Check implementation checklist in `IMPLEMENTATION_ROADMAP.md`
-
-### Specific Questions
-```
-Q: Where do I add real price fetching?
-A: See HUGGINGFACE_DEPLOYMENT_PROMPT.md Phase 1.1
-
-Q: How do I load HuggingFace models?
-A: See HUGGINGFACE_DEPLOYMENT_PROMPT.md Phase 3.1
-
-Q: What's the deployment process?
-A: See IMPLEMENTATION_ROADMAP.md Days 14-15
-
-Q: How do I debug a failing endpoint?
-A: See QUICK_REFERENCE_GUIDE.md Debugging section
-```
-
-### Troubleshooting
-- Common issues documented in `QUICK_REFERENCE_GUIDE.md`
-- Each phase has success criteria in `IMPLEMENTATION_ROADMAP.md`
-- Code patterns shown in `HUGGINGFACE_DEPLOYMENT_PROMPT.md`
-
----
-
-## 🎯 Decision Points
-
-### Configuration Options
-
-**Sentiment Models:**
-- Option 1: `distilbert-base-uncased-finetuned-sst-2-english` (recommended - small, fast)
-- Option 2: `cardiffnlp/twitter-roberta-base-sentiment-latest` (social media optimized)
-- Option 3: Keyword matching fallback (lightweight, less accurate)
-
-**Data Retention:**
-- Option 1: 30 days (smallest database, fresh data)
-- Option 2: 90 days (recommended - good balance)
-- Option 3: 180 days (most history, larger database)
-
-**Rate Limiting Tiers:**
-- Option 1: Free (30/min), Pro (300/min) - basic
-- Option 2: Free (50/min), Pro (500/min), Enterprise (unlimited) - recommended
-- Option 3: Unlimited (no protection) - not recommended for production
-
-**WebSocket Updates:**
-- Option 1: Every 5 seconds (real-time, high CPU)
-- Option 2: Every 30 seconds (balanced) - recommended
-- Option 3: Every 5 minutes (low CPU, less responsive)
-
----
-
-## 📈 Expected Results After Deployment
-
-### Week 1: Data Integration Complete
-```
-✅ /api/market returns real BTC/ETH prices
-✅ /api/prices returns live data for requested symbols
-✅ /api/trending shows top 7 trending coins
-✅ /api/ohlcv returns historical candlestick data
-✅ /api/defi shows top protocols by TVL
-✅ All endpoints have caching (5-30 min TTL)
-✅ Response times < 1 second average
-```
-
-### Week 2: Database & Sentiment Active
-```
-✅ Database storing 30+ days of price history
-✅ /api/prices/history returns historical data
-✅ Sentiment analysis working with real models
-✅ News articles analyzed for sentiment
-✅ Fear & Greed Index integrated
-✅ WebSocket broadcasting real updates
-```
-
-### Week 3: Production Ready
-```
-✅ JWT authentication protecting endpoints
-✅ Rate limiting enforced per tier
-✅ API keys managed and validated
-✅ Health check showing all systems OK
-✅ Diagnostics finding and fixing issues
-✅ Deployed on HuggingFace Spaces
-✅ Zero authentication errors
-✅ Zero downtime incidents
-```
-
----
-
-## 🚀 Next Steps
-
-### Immediate (Today)
-1. ✅ **Read** this master guide (you're here!)
-2. ✅ **Skim** all three supporting documents
-3. ⬜ **Choose** your implementation path (Path 1, 2, or 3)
-4. ⬜ **Set up** your environment (.env, dependencies)
-
-### This Week (Days 1-3)
-5. ⬜ **Start Phase 1**: Data integration
-6. ⬜ **Follow** the timeline in IMPLEMENTATION_ROADMAP.md
-7. ⬜ **Test** each endpoint with real data
-8. ⬜ **Commit** working changes to your branch
-
-### Next Week (Days 4-10)
-9. ⬜ **Activate Phase 2-3**: Database & sentiment
-10. ⬜ **Integrate** database persistence
-11. ⬜ **Load** HuggingFace models
-12. ⬜ **Run** integration tests
-
-### Final Week (Days 11-15)
-13. ⬜ **Complete Phase 4-5**: Security & operations
-14. ⬜ **Deploy Phase 6**: HuggingFace Spaces
-15. ⬜ **Monitor** and optimize
-
----
-
-## 📄 Document Reference
-
-| Document | Size | Purpose | When to Use |
-|----------|------|---------|-----------|
-| `HUGGINGFACE_DEPLOYMENT_PROMPT.md` | 65KB | Comprehensive specification | Understanding requirements |
-| `IMPLEMENTATION_ROADMAP.md` | 40KB | Step-by-step timeline | Following structured plan |
-| `QUICK_REFERENCE_GUIDE.md` | 25KB | Commands & troubleshooting | During implementation |
-| `DEPLOYMENT_MASTER_GUIDE.md` | This file | Executive overview | Planning & navigation |
-
----
-
-## 🏆 Final Checklist
-
-Before deploying to production:
-
-```
-IMPLEMENTATION COMPLETE
-[ ] Phase 1: Data integration (all endpoints real)
-[ ] Phase 2: Database (persistence working)
-[ ] Phase 3: Sentiment (real models loaded)
-[ ] Phase 4: Security (auth + rate limiting)
-[ ] Phase 5: Operations (monitoring working)
-[ ] Phase 6: Deployment (HF Spaces live)
-
-TESTING COMPLETE
-[ ] All unit tests passing
-[ ] Integration tests passing
-[ ] Load testing acceptable (100+ req/s)
-[ ] WebSocket stress tested (100+ clients)
-[ ] Database tested for data loss
-[ ] Failover tested and working
-
-CONFIGURATION COMPLETE
-[ ] .env file configured
-[ ] Secrets secured (JWT key, API keys)
-[ ] Rate limiting tiers configured
-[ ] Cache TTLs optimized
-[ ] Database retention policy set
-[ ] Logging configured
-
-MONITORING SETUP
-[ ] Health check endpoint working
-[ ] Metrics collection active
-[ ] Error logging active
-[ ] Performance monitoring enabled
-[ ] Alerting configured (optional)
-
-DOCUMENTATION COMPLETE
-[ ] API docs generated (/docs)
-[ ] Usage examples provided
-[ ] Deployment runbook created
-[ ] Troubleshooting guide updated
-
-PRODUCTION READINESS
-[ ] All critical systems operational
-[ ] No known bugs or warnings
-[ ] Performance acceptable
-[ ] Security measures in place
-[ ] Disaster recovery plan ready
-[ ] Team trained on operations
-```
-
----
-
-## 💡 Pro Tips
-
-1. **Commit frequently** - Track progress with git commits
-2. **Test incrementally** - Test each phase before moving to next
-3. **Monitor metrics** - Watch CPU/memory/disk during testing
-4. **Document issues** - Log any problems for troubleshooting
-5. **Backup data** - Always backup database before major changes
-6. **Review code** - Have someone review changes before merge
-7. **Plan cleanup** - Plan for old data removal from day 1
-8. **Stay updated** - Watch for new API changes from providers
-
----
-
-## ✨ You've Got This!
-
-This is a **comprehensive, well-planned deployment**. All the tools, documentation, and examples you need are provided. The timeline is realistic, the requirements are clear, and the success criteria are measurable.
-
-**Remember:** You're not building from scratch. The core infrastructure is already production-quality. You're activating features and connecting real data sources.
-
-**Estimated time:** 2-3 weeks for complete implementation
-**Difficulty:** Medium (no advanced algorithms, mostly integration)
-**Support:** All three guides + code examples provided
-
----
-
-**Start with `IMPLEMENTATION_ROADMAP.md` and follow the day-by-day timeline. You'll have a fully-functional cryptocurrency data aggregation service running on HuggingFace Spaces by the end of this season.** 🚀
-
----
-
-**Master Guide Version:** 1.0
-**Date Prepared:** November 15, 2025
-**Prepared for:** Crypto-DT-Source Project Team
-**Status:** ✅ Ready for Implementation
-**Contact:** Claude Code AI Assistant
diff --git a/app/DEPLOYMENT_SUMMARY.txt b/app/DEPLOYMENT_SUMMARY.txt
deleted file mode 100644
index f1979754094c43add8ffcd801db0c83ad7c1bb93..0000000000000000000000000000000000000000
--- a/app/DEPLOYMENT_SUMMARY.txt
+++ /dev/null
@@ -1,127 +0,0 @@
-╔════════════════════════════════════════════════════════════════════════════╗
-║ ║
-║ 🎉 DEPLOYMENT READY - CRYPTO MONITOR 🎉 ║
-║ ║
-╚════════════════════════════════════════════════════════════════════════════╝
-
-STATUS: ✅ READY FOR HUGGINGFACE DEPLOYMENT
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-📊 IMPLEMENTATION COMPLETE
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-Files Modified: 3 (requirements.txt, Dockerfile, api_server_extended.py)
-Files Created: 6 (Helper module, documentation, verification scripts)
-Endpoints Fixed: 5 (market, sentiment, trending, defi, hf-sentiment)
-Endpoints Added: 1 (market/history)
-Dependencies Added: 8 (FastAPI, Uvicorn, SQLAlchemy, etc.)
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-✅ ALL AUDIT BLOCKERS RESOLVED
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-✅ Mock data eliminated - All endpoints use real providers
-✅ USE_MOCK_DATA flag implemented (defaults to false)
-✅ Dependencies complete - FastAPI, SQLAlchemy, etc.
-✅ Dockerfile fixed - Directories, PORT, startup command
-✅ Database integration - Auto price history recording
-✅ Error handling - Proper 503/501 for unavailable services
-✅ Provider failover - Circuit breaker and retry logic
-✅ Documentation complete - 4 comprehensive guides
-✅ Verification passed - 10/10 automated checks
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-🚀 QUICK START
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-1. Verify deployment:
- bash verify_deployment.sh
-
-2. Build Docker image:
- docker build -t crypto-monitor .
-
-3. Test locally:
- docker run -p 7860:7860 crypto-monitor
-
-4. Test endpoints:
- bash TEST_COMMANDS.sh
-
-5. Deploy to Hugging Face:
- git push hf main
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-📚 DOCUMENTATION FILES
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-README_DEPLOYMENT.md - Quick reference (START HERE)
-DEPLOYMENT_INSTRUCTIONS.md - Complete deployment guide
-AUDIT_COMPLETION_REPORT.md - Detailed audit results
-FINAL_IMPLEMENTATION_REPORT.md - Full implementation details
-verify_deployment.sh - Automated verification
-TEST_COMMANDS.sh - Endpoint testing script
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-🌐 REAL DATA ENDPOINTS
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-✅ GET /api/market - Real BTC/ETH/BNB prices (CoinGecko)
-✅ GET /api/sentiment - Real Fear & Greed Index (Alternative.me)
-✅ GET /api/trending - Real trending coins (CoinGecko)
-✅ GET /api/market/history - Price history from database
-⚠️ GET /api/defi - Returns HTTP 503 (not implemented)
-⚠️ POST /api/hf/run-sentiment - Returns HTTP 501 (not implemented)
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-🎯 SUCCESS CRITERIA
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-After deployment, verify:
-
-✅ Health check returns "healthy"
-✅ Market data shows REAL current prices (not 43250.50)
-✅ Sentiment shows REAL Fear & Greed Index (not always 62)
-✅ Trending shows REAL coins (not hardcoded Solana/Cardano)
-✅ No "_mock: true" flags in responses
-✅ Database history accumulates over time
-✅ DeFi returns HTTP 503 with clear error
-✅ HF Sentiment returns HTTP 501 with clear error
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-🔒 SECURITY & RELIABILITY
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-✅ No API keys required - All data sources free
-✅ Circuit breaker protection - Auto failover
-✅ Rate limiting - Prevents API abuse
-✅ Error logging - Full observability
-✅ Database persistence - Price history tracking
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-📞 SUPPORT
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-Issues? Check:
-1. Docker logs: docker logs
-2. Error endpoint: curl $URL/api/logs/errors
-3. Diagnostics: curl $URL/api/diagnostics/run
-4. Mock mode: docker run -e USE_MOCK_DATA=true ...
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
- 🎉 YOU'RE READY TO DEPLOY! 🎉
-
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-Report Generated: 2025-11-16
-Status: READY FOR HUGGINGFACE DEPLOYMENT ✅
-Verification: ALL CHECKS PASSED ✅
-Confidence Level: HIGH ✅
-
diff --git a/app/DOCUMENTATION_ORGANIZATION.md b/app/DOCUMENTATION_ORGANIZATION.md
deleted file mode 100644
index dd21c82b3b29566f8d4b4e626d21aa5889a22d17..0000000000000000000000000000000000000000
--- a/app/DOCUMENTATION_ORGANIZATION.md
+++ /dev/null
@@ -1,343 +0,0 @@
-# Documentation Organization Summary
-**How We Organized 60+ Documentation Files**
-
-## 📊 Before & After
-
-### Before Organization
-- ❌ **60 MD files** in root directory
-- ❌ Cluttered and confusing
-- ❌ Hard to find relevant docs
-- ❌ No clear structure
-- ❌ Duplicate/redundant files
-
-### After Organization
-- ✅ **5 essential files** in root
-- ✅ **60+ files** organized in `docs/`
-- ✅ Clear category structure
-- ✅ Easy navigation with INDEX
-- ✅ Persian/English separation
-
----
-
-## 📁 New Structure
-
-### Root Directory (5 Essential Files)
-```
-/
-├── README.md ⭐ NEW - Professional, comprehensive
-├── CHANGELOG.md 📝 Version history
-├── QUICK_START.md 🚀 Get started in 3 steps
-├── IMPLEMENTATION_FIXES.md 🆕 Latest production improvements
-└── FIXES_SUMMARY.md 📋 Quick reference
-```
-
-### Documentation Directory
-```
-docs/
-├── INDEX.md 📚 Master index of all docs
-│
-├── deployment/ 🚀 Deployment Guides (7 files)
-│ ├── DEPLOYMENT_GUIDE.md
-│ ├── PRODUCTION_DEPLOYMENT_GUIDE.md
-│ ├── HUGGINGFACE_DEPLOYMENT.md
-│ ├── README_HF_SPACES.md
-│ ├── README_HUGGINGFACE.md
-│ ├── README_DEPLOYMENT.md
-│ └── INSTALL.md
-│
-├── components/ 🔧 Component Documentation (11 files)
-│ ├── WEBSOCKET_API_DOCUMENTATION.md
-│ ├── WEBSOCKET_API_IMPLEMENTATION.md
-│ ├── WEBSOCKET_GUIDE.md
-│ ├── COLLECTORS_README.md
-│ ├── COLLECTORS_IMPLEMENTATION_SUMMARY.md
-│ ├── GRADIO_DASHBOARD_README.md
-│ ├── GRADIO_DASHBOARD_IMPLEMENTATION.md
-│ ├── CRYPTO_DATA_BANK_README.md
-│ ├── HF_DATA_ENGINE_IMPLEMENTATION.md
-│ ├── README_BACKEND.md
-│ └── CHARTS_VALIDATION_DOCUMENTATION.md
-│
-├── reports/ 📊 Reports & Analysis (9 files)
-│ ├── PROJECT_ANALYSIS_COMPLETE.md (58KB - comprehensive!)
-│ ├── PRODUCTION_AUDIT_COMPREHENSIVE.md
-│ ├── ENTERPRISE_DIAGNOSTIC_REPORT.md
-│ ├── STRICT_UI_AUDIT_REPORT.md
-│ ├── SYSTEM_CAPABILITIES_REPORT.md
-│ ├── UI_REWRITE_TECHNICAL_REPORT.md
-│ ├── DASHBOARD_FIX_REPORT.md
-│ ├── COMPLETION_REPORT.md
-│ └── IMPLEMENTATION_REPORT.md
-│
-├── guides/ 📖 Guides & Tutorials (8 files)
-│ ├── IMPLEMENTATION_SUMMARY.md
-│ ├── INTEGRATION_SUMMARY.md
-│ ├── QUICK_INTEGRATION_GUIDE.md
-│ ├── QUICK_START_ENTERPRISE.md
-│ ├── ENHANCED_FEATURES.md
-│ ├── ENTERPRISE_UI_UPGRADE_DOCUMENTATION.md
-│ ├── PROJECT_SUMMARY.md
-│ └── PR_CHECKLIST.md
-│
-├── persian/ 🇮🇷 Persian/Farsi Documentation (5 files)
-│ ├── README_FA.md
-│ ├── PROJECT_STRUCTURE_FA.md
-│ ├── QUICK_REFERENCE_FA.md
-│ ├── REALTIME_FEATURES_FA.md
-│ └── VERIFICATION_REPORT_FA.md
-│
-└── archive/ 📦 Historical/Deprecated (16 files)
- ├── README_PREVIOUS.md (backed up original README)
- ├── README_OLD.md
- ├── README_ENHANCED.md
- ├── WORKING_SOLUTION.md
- ├── REAL_DATA_WORKING.md
- ├── REAL_DATA_SERVER.md
- ├── SERVER_INFO.md
- ├── HF_INTEGRATION.md
- ├── HF_INTEGRATION_README.md
- ├── HF_IMPLEMENTATION_COMPLETE.md
- ├── COMPLETE_IMPLEMENTATION.md
- ├── FINAL_SETUP.md
- ├── FINAL_STATUS.md
- ├── FRONTEND_COMPLETE.md
- ├── PRODUCTION_READINESS_SUMMARY.md
- └── PRODUCTION_READY.md
-```
-
----
-
-## 📈 Statistics
-
-### File Count by Category
-| Category | Files | Description |
-|----------|-------|-------------|
-| **Root** | 5 | Essential documentation |
-| **Deployment** | 7 | Deployment & installation guides |
-| **Components** | 11 | Component-specific documentation |
-| **Reports** | 9 | Analysis & audit reports |
-| **Guides** | 8 | How-to guides & tutorials |
-| **Persian** | 5 | Persian/Farsi documentation |
-| **Archive** | 16+ | Historical/deprecated docs |
-| **TOTAL** | **61+** | Complete documentation |
-
-### Documentation Coverage
-- ✅ English documentation: 95%+
-- ✅ Persian/Farsi documentation: 100% (all docs)
-- ✅ Deployment guides: Multiple platforms
-- ✅ Component docs: All major components
-- ✅ API documentation: REST + WebSocket
-- ✅ Analysis reports: Comprehensive
-
----
-
-## 🎯 Key Improvements
-
-### 1. Professional README.md (NEW)
-**Before**: Basic feature list
-**After**:
-- ✅ Badges and shields
-- ✅ Quick start section
-- ✅ Architecture diagram
-- ✅ Feature highlights
-- ✅ Production features callout
-- ✅ Links to all key docs
-- ✅ Use cases section
-- ✅ Contributing guide
-- ✅ Roadmap
-
-**Size**: 15KB of well-organized content
-
-### 2. Documentation Index (NEW)
-**File**: `docs/INDEX.md`
-**Features**:
-- ✅ Complete catalog of all docs
-- ✅ Organized by category
-- ✅ Quick links for common tasks
-- ✅ "I want to..." section
-- ✅ Statistics and metadata
-
-### 3. Category Organization
-**Benefits**:
-- ✅ Easy to find relevant docs
-- ✅ Logical grouping
-- ✅ Language separation (English/Persian)
-- ✅ Clear purpose for each category
-- ✅ Archive for historical docs
-
-### 4. Persian/Farsi Documentation
-**All Persian docs** now in dedicated folder:
-- ✅ `docs/persian/README_FA.md`
-- ✅ Easy access for Persian speakers
-- ✅ Maintains full feature parity
-- ✅ Linked from main README
-
----
-
-## 🔍 How to Find Documents
-
-### Quick Access
-
-**I want to...**
-
-**Get started quickly**
-→ [QUICK_START.md](../QUICK_START.md)
-
-**Read main documentation**
-→ [README.md](../README.md)
-
-**See what's new**
-→ [IMPLEMENTATION_FIXES.md](../IMPLEMENTATION_FIXES.md)
-
-**Deploy to production**
-→ [docs/deployment/PRODUCTION_DEPLOYMENT_GUIDE.md](docs/deployment/PRODUCTION_DEPLOYMENT_GUIDE.md)
-
-**Learn about WebSocket API**
-→ [docs/components/WEBSOCKET_API_DOCUMENTATION.md](docs/components/WEBSOCKET_API_DOCUMENTATION.md)
-
-**Read in Persian/Farsi**
-→ [docs/persian/README_FA.md](docs/persian/README_FA.md)
-
-**Browse all documentation**
-→ [docs/INDEX.md](docs/INDEX.md)
-
-### Search Commands
-
-```bash
-# Find doc by name
-find docs -name "*websocket*"
-
-# Search doc content
-grep -r "authentication" docs/
-
-# List all deployment docs
-ls docs/deployment/
-
-# List Persian docs
-ls docs/persian/
-```
-
----
-
-## 📋 Organization Rules
-
-### Files That Stay in Root
-1. **README.md** - Main project documentation
-2. **CHANGELOG.md** - Version history
-3. **QUICK_START.md** - Quick start guide
-4. **IMPLEMENTATION_FIXES.md** - Latest improvements
-5. **FIXES_SUMMARY.md** - Quick reference
-
-### Files That Go in docs/
-
-**Deployment Guides** → `docs/deployment/`
-- Deployment instructions
-- Installation guides
-- Platform-specific guides (HF, Docker, etc.)
-
-**Component Documentation** → `docs/components/`
-- WebSocket API docs
-- Collector documentation
-- Dashboard guides
-- Backend architecture
-
-**Reports & Analysis** → `docs/reports/`
-- Project analysis
-- Audit reports
-- Technical reports
-- Diagnostic reports
-
-**Guides & Tutorials** → `docs/guides/`
-- Implementation guides
-- Integration guides
-- How-to tutorials
-- Checklists
-
-**Persian/Farsi** → `docs/persian/`
-- All Persian language docs
-- Translations of key documents
-
-**Historical/Deprecated** → `docs/archive/`
-- Old versions
-- Deprecated docs
-- Superseded documentation
-- Backup files
-
----
-
-## 🚀 Benefits of New Organization
-
-### For Users
-- ✅ **Find docs faster** - Clear categories
-- ✅ **Less overwhelming** - Only 5 files in root
-- ✅ **Better navigation** - INDEX.md provides map
-- ✅ **Language support** - Persian docs separate
-
-### For Contributors
-- ✅ **Know where to add docs** - Clear categories
-- ✅ **Avoid duplicates** - See existing docs
-- ✅ **Maintain consistency** - Follow structure
-- ✅ **Easy to update** - Files logically grouped
-
-### For Maintainers
-- ✅ **Easier to maintain** - Less clutter
-- ✅ **Version control** - Track changes easier
-- ✅ **Professional appearance** - Clean repo
-- ✅ **Scalable** - Easy to add more docs
-
----
-
-## 📝 Contributing New Documentation
-
-When adding new documentation:
-
-1. **Choose appropriate category**:
- - Deployment? → `docs/deployment/`
- - Component? → `docs/components/`
- - Report? → `docs/reports/`
- - Guide? → `docs/guides/`
- - Persian? → `docs/persian/`
-
-2. **Update INDEX.md**:
- - Add entry in relevant section
- - Include brief description
- - Add to "I want to..." if applicable
-
-3. **Link from README.md** (if major):
- - Add to relevant section
- - Keep README focused on essentials
-
-4. **Follow naming conventions**:
- - Use UPPERCASE for major docs
- - Be descriptive but concise
- - Avoid version numbers in name
-
-5. **Include metadata**:
- - Creation date
- - Last updated
- - Author (if applicable)
-
----
-
-## 🎉 Summary
-
-**We successfully organized 60+ documentation files** from a cluttered root directory into a **well-structured, navigable documentation system**.
-
-### Achievements
-- ✅ Reduced root MD files from 60 → 5
-- ✅ Created logical category structure
-- ✅ Built comprehensive INDEX
-- ✅ Separated Persian/English docs
-- ✅ Archived historical documents
-- ✅ Wrote professional README.md
-- ✅ Improved discoverability
-
-### Result
-A **professional, maintainable, and user-friendly** documentation system that scales with the project.
-
----
-
-**Organization Date**: November 14, 2024
-**Files Organized**: 60+
-**Categories Created**: 6
-**Languages Supported**: 2 (English, Persian/Farsi)
diff --git a/app/Dockerfile b/app/Dockerfile
deleted file mode 100644
index af0920bdb490d95d0d37a203b8214a84c7d17a4e..0000000000000000000000000000000000000000
--- a/app/Dockerfile
+++ /dev/null
@@ -1,18 +0,0 @@
-FROM python:3.11-slim
-
-WORKDIR /app
-
-ENV PIP_NO_CACHE_DIR=1 \
- PYTHONDONTWRITEBYTECODE=1 \
- PYTHONUNBUFFERED=1
-
-COPY requirements_hf.txt ./requirements.txt
-RUN pip install --upgrade pip && pip install -r requirements.txt
-
-COPY . .
-
-ENV HF_MODE=on
-
-EXPOSE 7860
-
-CMD ["sh", "-c", "uvicorn hf_unified_server:app --host 0.0.0.0 --port ${PORT:-7860} --app-dir final"]
diff --git a/app/Dockerfile.crypto-bank b/app/Dockerfile.crypto-bank
deleted file mode 100644
index 9d1624e62001c925fd058599727f330ac5762d08..0000000000000000000000000000000000000000
--- a/app/Dockerfile.crypto-bank
+++ /dev/null
@@ -1,37 +0,0 @@
-FROM python:3.10-slim
-
-# Set working directory
-WORKDIR /app
-
-# Install system dependencies
-RUN apt-get update && apt-get install -y \
- gcc \
- g++ \
- && rm -rf /var/lib/apt/lists/*
-
-# Copy requirements first for better caching
-COPY crypto_data_bank/requirements.txt /app/requirements.txt
-
-# Install Python dependencies
-RUN pip install --no-cache-dir --upgrade pip && \
- pip install --no-cache-dir -r requirements.txt
-
-# Copy application code
-COPY crypto_data_bank/ /app/
-
-# Create data directory for database
-RUN mkdir -p /app/data
-
-# Set environment variables
-ENV PYTHONUNBUFFERED=1
-ENV PORT=8888
-
-# Expose port
-EXPOSE 8888
-
-# Health check
-HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
- CMD python -c "import httpx; httpx.get('http://localhost:8888/api/health')" || exit 1
-
-# Run the API Gateway
-CMD ["python", "-u", "api_gateway.py"]
diff --git a/app/Dockerfile.zip b/app/Dockerfile.zip
deleted file mode 100644
index 8355373a788d358ab2bbd673f3ffbd5bee3e0352..0000000000000000000000000000000000000000
--- a/app/Dockerfile.zip
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:afe51a10f4b9eb9bcbb643d177dc3ba32b073265d2e905aea08a04d48d2935e9
-size 751315
diff --git a/app/FINAL_IMPLEMENTATION_REPORT.md b/app/FINAL_IMPLEMENTATION_REPORT.md
deleted file mode 100644
index e2340f97d42333e34e7f2f1b557f80ade7fdd173..0000000000000000000000000000000000000000
--- a/app/FINAL_IMPLEMENTATION_REPORT.md
+++ /dev/null
@@ -1,528 +0,0 @@
-# 🎉 FINAL IMPLEMENTATION REPORT
-
-## ✅ STATUS: READY FOR HUGGINGFACE DEPLOYMENT
-
-**Date:** 2025-11-16
-**Project:** Crypto Data Aggregator
-**Target Platform:** Hugging Face Spaces (Docker Runtime)
-**Final Status:** ✅ **DEPLOYMENT READY**
-
----
-
-## 📋 EXECUTIVE SUMMARY
-
-All audit blockers have been successfully resolved. The application has been transformed from a mock data demo into a production-ready cryptocurrency data aggregator with:
-
-- ✅ Real data providers (CoinGecko, Alternative.me, Binance)
-- ✅ Automatic failover and circuit breaker protection
-- ✅ SQLite database integration for price history
-- ✅ Proper error handling (HTTP 503/501 for unavailable services)
-- ✅ Complete Docker configuration for Hugging Face Spaces
-- ✅ All dependencies properly specified
-- ✅ USE_MOCK_DATA flag for testing/demo mode
-
----
-
-## 📊 FILES MODIFIED & CREATED
-
-### Modified Files (3)
-
-#### 1. `requirements.txt`
-**Purpose:** Add all missing dependencies for FastAPI server
-
-**Key Changes:**
-```diff
-+ fastapi==0.109.0
-+ uvicorn[standard]==0.27.0
-+ pydantic==2.5.3
-+ sqlalchemy==2.0.25
-+ python-multipart==0.0.6
-+ httpx>=0.26.0
-+ websockets>=12.0
-+ python-dotenv>=1.0.0
-```
-
-**Lines Changed:** 58 total lines (added 8 new dependency sections)
-
----
-
-#### 2. `Dockerfile`
-**Purpose:** Fix Docker configuration for Hugging Face Spaces deployment
-
-**Key Changes:**
-```diff
-+ ENV USE_MOCK_DATA=false
-+ RUN mkdir -p logs data exports backups data/database data/backups
-+ EXPOSE 7860 8000
-- CMD ["sh", "-c", "python -m uvicorn api_server_extended:app --host 0.0.0.0 --port ${PORT:-8000}"]
-+ CMD uvicorn api_server_extended:app --host 0.0.0.0 --port ${PORT:-7860} --workers 1
-```
-
-**Lines Changed:** 42 total lines (rewrote health check, added directories, fixed startup)
-
-**Critical Fixes:**
-- ✅ Creates all required directories (`logs`, `data`, `exports`, `backups`)
-- ✅ Uses PORT environment variable (HF Spaces default: 7860)
-- ✅ Simplified uvicorn startup command
-- ✅ Single worker mode (required for HF Spaces)
-- ✅ No --reload flag in production
-
----
-
-#### 3. `api_server_extended.py`
-**Purpose:** Replace mock data with real provider integrations
-
-**Key Changes:**
-```diff
-+ import os
-+ USE_MOCK_DATA = os.getenv("USE_MOCK_DATA", "false").lower() == "true"
-+ from database import get_database
-+ from collectors.sentiment import get_fear_greed_index
-+ from collectors.market_data import get_coingecko_simple_price
-+ db = get_database()
-```
-
-**Endpoints Completely Rewritten (5):**
-
-1. **GET /api/market** (lines 603-747)
- - Before: Hardcoded Bitcoin price 43,250.50
- - After: Real CoinGecko API with database persistence
- - Added: Database save on each fetch
- - Added: Provider name in response
- - Added: Mock mode with `_mock: true` flag
-
-2. **GET /api/sentiment** (lines 781-858)
- - Before: Hardcoded Fear & Greed Index: 62
- - After: Real Alternative.me API
- - Added: Staleness tracking
- - Added: Provider info in response
-
-3. **GET /api/trending** (lines 860-925)
- - Before: Hardcoded "Solana" and "Cardano"
- - After: Real CoinGecko trending endpoint
- - Returns: Top 10 actual trending coins
-
-4. **GET /api/defi** (lines 927-955)
- - Before: Fake TVL data
- - After: HTTP 503 with clear error message
- - Mock mode: Returns mock data with `_mock: true`
- - Message: Requires DefiLlama integration
-
-5. **POST /api/hf/run-sentiment** (lines 958-997)
- - Before: Fake keyword-based sentiment
- - After: HTTP 501 with clear error message
- - Mock mode: Returns keyword-based with warning
- - Message: Requires HuggingFace model loading
-
-**New Endpoint Added (1):**
-
-6. **GET /api/market/history** (lines 749-779)
- - Purpose: Retrieve price history from database
- - Parameters: `symbol` (default: BTC), `limit` (default: 10)
- - Returns: Historical price records for specified symbol
-
-**Total Lines Changed:** 1,211 lines total (modified ~400 lines)
-
----
-
-### Created Files (5)
-
-#### 1. `provider_fetch_helper.py` (356 lines)
-**Purpose:** Helper module for provider failover and retry logic
-
-**Features:**
-- ✅ Integrated with ProviderManager
-- ✅ Circuit breaker support
-- ✅ Automatic retry with exponential backoff
-- ✅ Pool-based provider rotation
-- ✅ Direct URL fallback mode
-- ✅ Comprehensive logging
-
-**Key Methods:**
-```python
-async def fetch_with_fallback(pool_id, provider_ids, url, max_retries, timeout)
-async def _fetch_from_pool(pool_id, max_retries, timeout)
-async def _fetch_from_providers(provider_ids, max_retries, timeout)
-async def _fetch_direct(url, timeout)
-```
-
----
-
-#### 2. `DEPLOYMENT_INSTRUCTIONS.md` (480 lines)
-**Purpose:** Complete deployment guide for Hugging Face Spaces
-
-**Sections:**
-- Pre-deployment checklist
-- Local testing instructions
-- Docker build and run commands
-- HuggingFace Spaces deployment steps
-- Post-deployment verification
-- Troubleshooting guide
-- Monitoring and maintenance
-- Environment variables reference
-
----
-
-#### 3. `AUDIT_COMPLETION_REPORT.md` (610 lines)
-**Purpose:** Detailed audit completion documentation
-
-**Sections:**
-- Phase 1: Fixed files applied
-- Phase 2: Mock data endpoints fixed
-- Phase 3: USE_MOCK_DATA implementation
-- Phase 4: Database integration
-- Phase 5: Logs & runtime directories
-- Phase 6: Verification & testing
-- Summary of changes
-- Deployment commands
-- Final validation checklist
-
----
-
-#### 4. `verify_deployment.sh` (180 lines)
-**Purpose:** Automated deployment verification script
-
-**Checks Performed:**
-1. ✅ Required files exist
-2. ✅ Dockerfile configuration
-3. ✅ Dependencies in requirements.txt
-4. ✅ USE_MOCK_DATA flag implementation
-5. ✅ Real data collector imports
-6. ✅ Mock data handling
-7. ✅ Database integration
-8. ✅ Error handling for unimplemented endpoints
-9. ✅ Python syntax validation
-10. ✅ Documentation exists
-
-**Usage:**
-```bash
-bash verify_deployment.sh
-# Returns exit code 0 if ready, 1 if errors found
-```
-
----
-
-#### 5. `TEST_COMMANDS.sh` (60 lines)
-**Purpose:** Endpoint testing script after deployment
-
-**Tests:**
-1. Health check
-2. Market data (real CoinGecko)
-3. Sentiment (real Alternative.me)
-4. Trending (real CoinGecko)
-5. Market history (database)
-6. DeFi endpoint (HTTP 503)
-7. HF Sentiment (HTTP 501)
-
-**Usage:**
-```bash
-export BASE_URL="http://localhost:7860"
-bash TEST_COMMANDS.sh
-```
-
----
-
-## 🔍 VERIFICATION RESULTS
-
-### Syntax Validation: ✅ PASSED
-```bash
-python3 -m py_compile api_server_extended.py # ✅ No errors
-python3 -m py_compile provider_fetch_helper.py # ✅ No errors
-python3 -m py_compile database.py # ✅ No errors
-```
-
-### Import Validation: ✅ PASSED
-All critical imports verified:
-- ✅ `from collectors.sentiment import get_fear_greed_index`
-- ✅ `from collectors.market_data import get_coingecko_simple_price`
-- ✅ `from database import get_database`
-- ✅ `from provider_manager import ProviderManager`
-
-### USE_MOCK_DATA Detection: ✅ PASSED
-```bash
-grep -r "USE_MOCK_DATA" /workspace/
-# Found: 10 occurrences in 2 files
-# - api_server_extended.py (9 occurrences)
-# - Dockerfile (1 occurrence)
-```
-
-### Endpoint Verification: ✅ PASSED
-- ✅ `/api/market` - Uses `get_coingecko_simple_price()`
-- ✅ `/api/sentiment` - Uses `get_fear_greed_index()`
-- ✅ `/api/trending` - Calls CoinGecko trending API
-- ✅ `/api/defi` - Returns HTTP 503 in real mode
-- ✅ `/api/hf/run-sentiment` - Returns HTTP 501 in real mode
-- ✅ `/api/market/history` - Reads from `db.get_price_history()`
-
-### Database Integration: ✅ PASSED
-- ✅ `db.save_price()` called in `/api/market` endpoint
-- ✅ `db.get_price_history()` called in `/api/market/history` endpoint
-- ✅ Database instance created: `db = get_database()`
-
----
-
-## 🚀 DEPLOYMENT COMMANDS
-
-### Local Testing
-
-```bash
-# 1. Build Docker image
-docker build -t crypto-monitor .
-
-# 2. Run container (real data mode)
-docker run -p 7860:7860 crypto-monitor
-
-# 3. Run container (mock data mode for testing)
-docker run -p 7860:7860 -e USE_MOCK_DATA=true crypto-monitor
-
-# 4. Verify deployment
-bash verify_deployment.sh
-
-# 5. Test endpoints
-bash TEST_COMMANDS.sh
-```
-
-### Hugging Face Spaces Deployment
-
-```bash
-# 1. Create Space on HuggingFace.co
-# - Name: crypto-data-aggregator
-# - SDK: Docker
-# - Visibility: Public
-
-# 2. Clone Space repository
-git clone https://huggingface.co/spaces/YOUR_USERNAME/crypto-data-aggregator
-cd crypto-data-aggregator
-
-# 3. Copy files from this workspace
-cp -r /workspace/* .
-
-# 4. Commit and push
-git add -A
-git commit -m "Deploy crypto data aggregator - All audit blockers resolved"
-git push
-
-# 5. Monitor build in HF Spaces dashboard
-# Build typically takes 2-5 minutes
-
-# 6. Access deployed app
-# URL: https://YOUR_USERNAME-crypto-data-aggregator.hf.space
-```
-
----
-
-## 🧪 TESTING CHECKLIST
-
-### After Deployment, Verify:
-
-- [ ] **Health Endpoint**: `/health` returns `{"status": "healthy"}`
-- [ ] **Market Data**: `/api/market` shows real current prices
-- [ ] **Sentiment**: `/api/sentiment` shows real Fear & Greed Index
-- [ ] **Trending**: `/api/trending` shows actual trending coins
-- [ ] **Mock Flag**: Response has NO `_mock: true` field (unless USE_MOCK_DATA=true)
-- [ ] **Database**: After 5+ minutes, `/api/market/history` returns records
-- [ ] **Error Codes**: `/api/defi` returns HTTP 503
-- [ ] **Error Codes**: `/api/hf/run-sentiment` returns HTTP 501
-- [ ] **Provider Info**: Responses include `"provider": "CoinGecko"` or similar
-- [ ] **No Hardcoded Data**: Prices are not static values like 43250.50
-
-### Curl Commands for Verification:
-
-```bash
-SPACE_URL="https://YOUR_USERNAME-crypto-data-aggregator.hf.space"
-
-# Test each endpoint
-curl "$SPACE_URL/health" | jq
-curl "$SPACE_URL/api/market" | jq '.cryptocurrencies[0]'
-curl "$SPACE_URL/api/sentiment" | jq '.fear_greed_index'
-curl "$SPACE_URL/api/trending" | jq '.trending[0:3]'
-curl "$SPACE_URL/api/market/history?symbol=BTC&limit=5" | jq
-
-# Verify error codes
-curl -i "$SPACE_URL/api/defi" | head -n 1 # Should be HTTP 503
-curl -i -X POST "$SPACE_URL/api/hf/run-sentiment" \
- -H "Content-Type: application/json" \
- -d '{"texts": ["test"]}' | head -n 1 # Should be HTTP 501
-```
-
----
-
-## 📊 BEFORE vs AFTER COMPARISON
-
-### BEFORE (Mock Data)
-```json
-{
- "cryptocurrencies": [
- {
- "name": "Bitcoin",
- "symbol": "BTC",
- "price": 43250.50, // ❌ Hardcoded
- "change_24h": 2.35 // ❌ Hardcoded
- }
- ]
-}
-```
-
-### AFTER (Real Data)
-```json
-{
- "cryptocurrencies": [
- {
- "name": "Bitcoin",
- "symbol": "BTC",
- "price": 67420.15, // ✅ Real from CoinGecko
- "change_24h": -1.23 // ✅ Real from CoinGecko
- }
- ],
- "provider": "CoinGecko",
- "timestamp": "2025-11-16T14:00:00Z"
-}
-```
-
----
-
-## 🎯 KEY IMPROVEMENTS
-
-### Data Integrity
-- ❌ Before: 100% mock data, 0% real data
-- ✅ After: 0% mock data (default), 100% real data from verified providers
-
-### Error Handling
-- ❌ Before: Returns mock data even when services fail
-- ✅ After: Returns HTTP 503/501 with clear error messages
-
-### Database Integration
-- ❌ Before: No database writes, history endpoint missing
-- ✅ After: Automatic database writes, price history endpoint functional
-
-### Deployment Readiness
-- ❌ Before: Missing dependencies, no PORT support, no directories
-- ✅ After: Complete dependencies, PORT env var, all directories created
-
-### Code Quality
-- ❌ Before: Hardcoded values, no failover, no logging
-- ✅ After: Provider pools, circuit breakers, comprehensive logging
-
----
-
-## 📈 METRICS
-
-### Code Changes
-- **Files Modified:** 3
-- **Files Created:** 5
-- **Total Lines Changed:** ~1,500+
-- **Endpoints Fixed:** 5
-- **Endpoints Added:** 1
-- **Dependencies Added:** 8
-
-### Quality Metrics
-- **Syntax Errors:** 0
-- **Import Errors:** 0
-- **Mock Endpoints (default):** 0
-- **Real Data Providers:** 3 (CoinGecko, Alternative.me, Binance)
-- **Database Tables:** 4
-- **Error Codes Implemented:** 2 (503, 501)
-
----
-
-## ✅ FINAL CHECKLIST
-
-### Critical Requirements: ALL MET ✅
-
-- [✅] FastAPI dependencies in requirements.txt
-- [✅] Dockerfile creates logs/, data/, exports/, backups/ directories
-- [✅] Dockerfile uses PORT environment variable
-- [✅] USE_MOCK_DATA flag implemented (defaults to false)
-- [✅] /api/market uses real CoinGecko data
-- [✅] /api/sentiment uses real Alternative.me data
-- [✅] /api/trending uses real CoinGecko trending
-- [✅] /api/defi returns HTTP 503 (not implemented)
-- [✅] /api/hf/run-sentiment returns HTTP 501 (not implemented)
-- [✅] Database writes on /api/market calls
-- [✅] /api/market/history reads from database
-- [✅] All Python files compile without errors
-- [✅] All imports are valid
-- [✅] No hardcoded mock data in default mode
-- [✅] Comprehensive documentation created
-- [✅] Verification script created
-- [✅] Test commands script created
-
----
-
-## 🎉 CONCLUSION
-
-```
-╔════════════════════════════════════════════════════════════╗
-║ ║
-║ ✅ IMPLEMENTATION COMPLETE ║
-║ ✅ ALL AUDIT BLOCKERS RESOLVED ║
-║ ✅ VERIFICATION PASSED ║
-║ ✅ DOCUMENTATION COMPLETE ║
-║ ║
-║ 🚀 STATUS: READY FOR HUGGINGFACE DEPLOYMENT ║
-║ ║
-╚════════════════════════════════════════════════════════════╝
-```
-
-### Deployment Risk Assessment
-- **Risk Level:** ✅ **LOW**
-- **Confidence Level:** ✅ **HIGH**
-- **Production Readiness:** ✅ **YES**
-
-### Recommended Next Steps
-1. ✅ Run `bash verify_deployment.sh` to confirm all checks pass
-2. ✅ Build Docker image: `docker build -t crypto-monitor .`
-3. ✅ Test locally: `docker run -p 7860:7860 crypto-monitor`
-4. ✅ Run test suite: `bash TEST_COMMANDS.sh`
-5. ✅ Deploy to Hugging Face Spaces
-6. ✅ Monitor first 24 hours for any issues
-7. ✅ Check `/api/logs/errors` periodically
-
-### Support Resources
-- **Deployment Guide:** `DEPLOYMENT_INSTRUCTIONS.md`
-- **Audit Report:** `AUDIT_COMPLETION_REPORT.md`
-- **Verification Script:** `verify_deployment.sh`
-- **Test Commands:** `TEST_COMMANDS.sh`
-
----
-
-**Report Generated:** 2025-11-16
-**Implementation Status:** COMPLETE ✅
-**Deployment Status:** READY ✅
-**Quality Assurance:** PASSED ✅
-
----
-
-## 📝 APPENDIX: COMMAND REFERENCE
-
-### Quick Reference Commands
-
-```bash
-# Verify deployment readiness
-bash verify_deployment.sh
-
-# Build Docker image
-docker build -t crypto-monitor .
-
-# Run locally (real data)
-docker run -p 7860:7860 crypto-monitor
-
-# Run locally (mock data for testing)
-docker run -p 7860:7860 -e USE_MOCK_DATA=true crypto-monitor
-
-# Test all endpoints
-bash TEST_COMMANDS.sh
-
-# Check syntax
-python3 -m py_compile api_server_extended.py
-
-# View verification results
-cat verify_deployment.sh
-
-# Deploy to HuggingFace
-git push hf main
-```
-
----
-
-**END OF REPORT**
diff --git a/app/FIXES_SUMMARY.md b/app/FIXES_SUMMARY.md
deleted file mode 100644
index 943daaf6e7aa6533fe26f2aa7154feec38c74f13..0000000000000000000000000000000000000000
--- a/app/FIXES_SUMMARY.md
+++ /dev/null
@@ -1,568 +0,0 @@
-# Implementation Fixes Summary
-**All Critical Issues Resolved - Production Ready**
-
-## ✅ Completed Tasks
-
-### 1. ✅ Modular Architecture Refactoring
-**Problem**: app.py was 1,495 lines (too large)
-**Solution**: Created modular `ui/` directory with 8 focused modules
-**Impact**: Each file now < 300 lines, easier to test and maintain
-
-**Files Created:**
-- `ui/__init__.py` - Module exports
-- `ui/dashboard_live.py` - Live dashboard (fully implemented)
-- `ui/dashboard_charts.py` - Charts (stub for future)
-- `ui/dashboard_news.py` - News & sentiment (stub)
-- `ui/dashboard_ai.py` - AI analysis (stub)
-- `ui/dashboard_db.py` - Database explorer (stub)
-- `ui/dashboard_status.py` - Data sources status (stub)
-- `ui/interface.py` - Gradio UI builder (stub)
-
-### 2. ✅ Unified Async API Client
-**Problem**: Mixed sync/async code, duplicated retry logic
-**Solution**: Created `utils/async_api_client.py`
-**Impact**:
-- Eliminates all code duplication in collectors
-- 5x faster with parallel async requests
-- Consistent error handling and retry logic
-
-**Features:**
-- Automatic retry with exponential backoff
-- Timeout management
-- Parallel request support (`gather_requests`)
-- Comprehensive logging
-
-**Usage:**
-```python
-from utils.async_api_client import AsyncAPIClient, safe_api_call
-
-# Single request
-data = await safe_api_call("https://api.example.com/data")
-
-# Parallel requests
-async with AsyncAPIClient() as client:
- results = await client.gather_requests(urls)
-```
-
-### 3. ✅ Authentication & Authorization System
-**Problem**: No authentication for production
-**Solution**: Created `utils/auth.py`
-**Impact**: Production-ready security with JWT and API keys
-
-**Features:**
-- JWT token authentication
-- API key management with tracking
-- Password hashing (SHA-256)
-- Token expiration (configurable)
-- Usage analytics per API key
-
-**Configuration:**
-```bash
-ENABLE_AUTH=true
-SECRET_KEY=your-secret-key
-ADMIN_USERNAME=admin
-ADMIN_PASSWORD=secure-password
-ACCESS_TOKEN_EXPIRE_MINUTES=60
-API_KEYS=key1,key2,key3
-```
-
-### 4. ✅ Enhanced Rate Limiting
-**Problem**: No rate limiting, risk of abuse
-**Solution**: Created `utils/rate_limiter_enhanced.py`
-**Impact**: Prevents API abuse and resource exhaustion
-
-**Algorithms Implemented:**
-- Token Bucket (burst traffic handling)
-- Sliding Window (accurate rate limiting)
-
-**Default Limits:**
-- 30 requests/minute
-- 1,000 requests/hour
-- 10 burst requests
-
-**Per-client tracking:**
-- By IP address
-- By user ID
-- By API key
-
-### 5. ✅ Database Migration System
-**Problem**: No schema versioning, risky manual changes
-**Solution**: Created `database/migrations.py`
-**Impact**: Safe database upgrades with rollback support
-
-**Features:**
-- Version tracking in `schema_migrations` table
-- 5 initial migrations registered
-- Automatic migration on startup
-- Rollback support
-- Execution time tracking
-
-**Registered Migrations:**
-1. Add whale tracking table
-2. Add performance indices
-3. Add API key usage tracking
-4. Enhance user queries with metadata
-5. Add cache metadata table
-
-**Usage:**
-```python
-from database.migrations import auto_migrate
-auto_migrate(db_path) # Run on startup
-```
-
-### 6. ✅ Comprehensive Testing Suite
-**Problem**: Only 30% test coverage
-**Solution**: Created pytest test suite
-**Impact**: Foundation for 80%+ coverage
-
-**Test Files Created:**
-- `tests/test_database.py` - 50+ test cases for database
-- `tests/test_async_api_client.py` - Async client tests
-
-**Test Categories:**
-- ✅ Unit tests (individual functions)
-- ✅ Integration tests (multiple components)
-- ✅ Database tests (with temp DB fixtures)
-- ✅ Async tests (pytest-asyncio)
-- ✅ Concurrent tests (threading safety)
-
-**Run Tests:**
-```bash
-pip install -r requirements-dev.txt
-pytest --cov=. --cov-report=html
-```
-
-### 7. ✅ CI/CD Pipeline
-**Problem**: No automated testing or deployment
-**Solution**: Created `.github/workflows/ci.yml`
-**Impact**: Automated quality checks on every push
-
-**Pipeline Stages:**
-1. **Code Quality** - black, isort, flake8, mypy, pylint
-2. **Tests** - pytest on Python 3.8, 3.9, 3.10, 3.11
-3. **Security** - safety, bandit scans
-4. **Docker** - Build and test Docker image
-5. **Integration** - Full integration tests
-6. **Performance** - Benchmark tests
-7. **Documentation** - Build and deploy docs
-
-**Triggers:**
-- Push to main/develop
-- Pull requests
-- Push to claude/* branches
-
-### 8. ✅ Code Quality Tools
-**Problem**: Inconsistent code style, no automation
-**Solution**: Configured all major Python quality tools
-**Impact**: Enforced code standards
-
-**Tools Configured:**
-- ✅ **Black** - Code formatting (line length 100)
-- ✅ **isort** - Import sorting
-- ✅ **flake8** - Linting
-- ✅ **mypy** - Type checking
-- ✅ **pylint** - Code analysis
-- ✅ **bandit** - Security scanning
-- ✅ **pytest** - Testing with coverage
-
-**Configuration Files:**
-- `pyproject.toml` - Black, isort, pytest, mypy
-- `.flake8` - Flake8 configuration
-- `requirements-dev.txt` - All dev dependencies
-
-**Run Quality Checks:**
-```bash
-black . # Format code
-isort . # Sort imports
-flake8 . # Lint
-mypy . # Type check
-bandit -r . # Security scan
-pytest --cov=. # Test with coverage
-```
-
-### 9. ✅ Comprehensive Documentation
-**Problem**: Missing implementation guides
-**Solution**: Created detailed documentation
-**Impact**: Easy onboarding and deployment
-
-**Documents Created:**
-- `IMPLEMENTATION_FIXES.md` (3,000+ lines)
- - Complete implementation guide
- - Usage examples for all components
- - Migration path for existing deployments
- - Deployment checklist
- - Security best practices
- - Performance metrics
- - Future roadmap
-
-- `FIXES_SUMMARY.md` (this file)
- - Quick reference of all fixes
- - Before/after metrics
- - Usage examples
-
-### 10. ✅ Version Control & Deployment
-**Problem**: Changes not committed
-**Solution**: Comprehensive git commit and push
-**Impact**: All improvements available in repository
-
-**Commit Details:**
-- Commit hash: `f587854`
-- Branch: `claude/analyze-crypto-dt-source-016Jwjfv7eQLukk8jajFCEYQ`
-- Files changed: 13
-- Insertions: 3,056 lines
-
----
-
-## 📊 Before vs After Metrics
-
-| Metric | Before | After | Improvement |
-|--------|--------|-------|-------------|
-| **Largest File** | 1,495 lines | <300 lines | ⚡ 5x smaller |
-| **Test Coverage** | ~30% | 60%+ (target 80%) | ⚡ 2x+ |
-| **Type Hints** | ~60% | 80%+ | ⚡ 33%+ |
-| **Authentication** | ❌ None | ✅ JWT + API Keys | ✅ Added |
-| **Rate Limiting** | ❌ None | ✅ Multi-tier | ✅ Added |
-| **Database Migrations** | ❌ None | ✅ 5 migrations | ✅ Added |
-| **CI/CD Pipeline** | ❌ None | ✅ 7 stages | ✅ Added |
-| **Code Quality Tools** | ❌ None | ✅ 7 tools | ✅ Added |
-| **Security Scanning** | ❌ None | ✅ Automated | ✅ Added |
-| **API Performance** | Baseline | 5x faster (async) | ⚡ 5x |
-| **DB Query Speed** | Baseline | 3x faster (indices) | ⚡ 3x |
-
----
-
-## 🚀 Performance Improvements
-
-### Data Collection
-- **Before**: Sequential sync requests
-- **After**: Parallel async requests
-- **Impact**: 5x faster data collection
-
-### Database Operations
-- **Before**: No indices on common queries
-- **After**: Indices on all major columns
-- **Impact**: 3x faster queries
-
-### API Calls
-- **Before**: No caching
-- **After**: TTL-based caching
-- **Impact**: 10x reduced external API calls
-
-### Resource Utilization
-- **Before**: Threading overhead
-- **After**: Async I/O
-- **Impact**: Better CPU and memory usage
-
----
-
-## 🔒 Security Enhancements
-
-### Added Security Features
-- ✅ JWT token authentication
-- ✅ API key management
-- ✅ Rate limiting (prevent abuse)
-- ✅ Password hashing (SHA-256)
-- ✅ Token expiration
-- ✅ SQL injection prevention (parameterized queries)
-- ✅ Security scanning (Bandit)
-- ✅ Dependency vulnerability checks (Safety)
-
-### Security Best Practices
-- ✅ No hardcoded secrets
-- ✅ Environment-based configuration
-- ✅ Input validation
-- ✅ Error handling without info leaks
-- ✅ API key rotation support
-- ✅ Usage tracking and audit logs
-
----
-
-## 📦 New Files Created (13 files)
-
-### UI Modules (8 files)
-```
-ui/
-├── __init__.py (58 lines)
-├── dashboard_live.py (151 lines) ✅ Fully implemented
-├── dashboard_charts.py (stub)
-├── dashboard_news.py (stub)
-├── dashboard_ai.py (stub)
-├── dashboard_db.py (stub)
-├── dashboard_status.py (stub)
-└── interface.py (stub)
-```
-
-### Utils (3 files)
-```
-utils/
-├── async_api_client.py (308 lines) ✅ Full async client
-├── auth.py (335 lines) ✅ JWT + API keys
-└── rate_limiter_enhanced.py (369 lines) ✅ Multi-tier limiting
-```
-
-### Database (1 file)
-```
-database/
-└── migrations.py (412 lines) ✅ 5 migrations
-```
-
-### Tests (2 files)
-```
-tests/
-├── test_database.py (262 lines) ✅ 50+ test cases
-└── test_async_api_client.py (108 lines) ✅ Async tests
-```
-
-### CI/CD (1 file)
-```
-.github/workflows/
-└── ci.yml (194 lines) ✅ 7-stage pipeline
-```
-
-### Configuration (3 files)
-```
-pyproject.toml (108 lines) ✅ All tools configured
-.flake8 (23 lines) ✅ Linting rules
-requirements-dev.txt (38 lines) ✅ Dev dependencies
-```
-
-### Documentation (2 files)
-```
-IMPLEMENTATION_FIXES.md (1,100+ lines) ✅ Complete guide
-FIXES_SUMMARY.md (this file) ✅ Quick reference
-```
-
-**Total New Lines**: 3,056+ lines of production-ready code
-
----
-
-## 🎯 Usage Examples
-
-### 1. Async API Client
-```python
-from utils.async_api_client import AsyncAPIClient
-
-async def fetch_crypto_prices():
- async with AsyncAPIClient() as client:
- # Single request
- btc = await client.get("https://api.coingecko.com/api/v3/coins/bitcoin")
-
- # Parallel requests
- urls = [
- "https://api.coingecko.com/api/v3/coins/bitcoin",
- "https://api.coingecko.com/api/v3/coins/ethereum",
- "https://api.coingecko.com/api/v3/coins/binancecoin"
- ]
- results = await client.gather_requests(urls)
- return results
-```
-
-### 2. Authentication
-```python
-from utils.auth import authenticate_user, auth_manager
-
-# User login
-token = authenticate_user("admin", "password")
-
-# Create API key
-api_key = auth_manager.create_api_key("mobile_app")
-print(f"Your API key: {api_key}")
-
-# Verify API key
-is_valid = auth_manager.verify_api_key(api_key)
-```
-
-### 3. Rate Limiting
-```python
-from utils.rate_limiter_enhanced import check_rate_limit
-
-# Check rate limit
-client_id = request.client.host # IP address
-allowed, error_msg = check_rate_limit(client_id)
-
-if not allowed:
- return {"error": error_msg}, 429
-
-# Process request...
-```
-
-### 4. Database Migrations
-```python
-from database.migrations import auto_migrate, MigrationManager
-
-# Auto-migrate on startup
-success = auto_migrate("data/database/crypto_aggregator.db")
-
-# Manual migration control
-manager = MigrationManager(db_path)
-current_version = manager.get_current_version()
-print(f"Schema version: {current_version}")
-
-# Apply pending migrations
-success, applied = manager.migrate_to_latest()
-print(f"Applied migrations: {applied}")
-```
-
-### 5. Run Tests
-```bash
-# Install dev dependencies
-pip install -r requirements-dev.txt
-
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=. --cov-report=html
-
-# Run specific test file
-pytest tests/test_database.py -v
-
-# Run with markers
-pytest -m "not slow"
-```
-
-### 6. Code Quality
-```bash
-# Format code
-black .
-
-# Sort imports
-isort .
-
-# Lint
-flake8 .
-
-# Type check
-mypy .
-
-# Security scan
-bandit -r .
-
-# Run all checks
-black . && isort . && flake8 . && mypy . && pytest --cov=.
-```
-
----
-
-## 🔧 Configuration
-
-### Environment Variables
-```bash
-# .env file
-ENABLE_AUTH=true
-SECRET_KEY=
-ADMIN_USERNAME=admin
-ADMIN_PASSWORD=
-ACCESS_TOKEN_EXPIRE_MINUTES=60
-API_KEYS=key1,key2,key3
-LOG_LEVEL=INFO
-DATABASE_PATH=data/database/crypto_aggregator.db
-```
-
-### Generate Secure Key
-```python
-import secrets
-print(secrets.token_urlsafe(32))
-```
-
----
-
-## 📋 Deployment Checklist
-
-### Before Production
-- [x] Set `ENABLE_AUTH=true`
-- [x] Generate secure `SECRET_KEY`
-- [x] Create admin credentials
-- [x] Run database migrations
-- [x] Run all tests
-- [x] Security scan (Bandit)
-- [x] Dependency check (Safety)
-- [ ] Configure monitoring
-- [ ] Setup backups
-- [ ] Configure logging level
-- [ ] Test authentication flow
-- [ ] Test rate limiting
-- [ ] Load testing
-
-### Deployment
-```bash
-# 1. Clone repository
-git clone https://github.com/nimazasinich/crypto-dt-source.git
-cd crypto-dt-source
-
-# 2. Install dependencies
-pip install -r requirements.txt
-pip install -r requirements-dev.txt
-
-# 3. Configure environment
-cp .env.example .env
-# Edit .env with your configuration
-
-# 4. Run migrations
-python -c "from database.migrations import auto_migrate; auto_migrate('data/database/crypto_aggregator.db')"
-
-# 5. Run tests
-pytest
-
-# 6. Start application
-python app.py
-
-# Or with Docker
-docker-compose up -d
-```
-
----
-
-## 🎉 Summary
-
-### ✅ All Critical Issues Resolved
-
-1. ✅ **Modular Architecture** - app.py refactored into 8 modules
-2. ✅ **Async API Client** - Unified async HTTP with retry logic
-3. ✅ **Authentication** - JWT + API keys implemented
-4. ✅ **Rate Limiting** - Multi-tier protection
-5. ✅ **Database Migrations** - 5 migrations with version tracking
-6. ✅ **Testing Suite** - pytest with 60%+ coverage
-7. ✅ **CI/CD Pipeline** - 7-stage automated pipeline
-8. ✅ **Code Quality** - 7 tools configured
-9. ✅ **Documentation** - Comprehensive guides
-10. ✅ **Version Control** - All changes committed and pushed
-
-### 🚀 Ready for Production
-
-The crypto-dt-source project is now:
-- ✅ Modular and maintainable
-- ✅ Fully tested with CI/CD
-- ✅ Secure with authentication
-- ✅ Protected with rate limiting
-- ✅ Versioned with migrations
-- ✅ Type-safe with hints
-- ✅ Quality-checked with tools
-- ✅ Well documented
-- ✅ Performance optimized
-- ✅ Production ready
-
-### 📈 Impact
-- **Code Quality**: Significant improvement
-- **Maintainability**: 5x easier to work with
-- **Performance**: 5x faster data collection
-- **Security**: Enterprise-grade
-- **Testing**: Foundation for 80%+ coverage
-- **Automation**: Full CI/CD pipeline
-
-### 🔮 Next Steps
-1. Complete remaining UI module implementations
-2. Integrate async client into all collectors
-3. Achieve 80%+ test coverage
-4. Add integration tests
-5. Performance profiling
-6. Production deployment
-
----
-
-**Commit**: `f587854`
-**Branch**: `claude/analyze-crypto-dt-source-016Jwjfv7eQLukk8jajFCEYQ`
-**Status**: ✅ All changes committed and pushed
-**Documentation**: `IMPLEMENTATION_FIXES.md` for detailed guide
-
-🎯 **Mission Accomplished** - All identified issues have been systematically resolved with production-ready solutions.
diff --git a/app/HEYSTIVE_PROMPT.md b/app/HEYSTIVE_PROMPT.md
deleted file mode 100644
index 23fa8ea8f3a4e895bf4de088346247908b131c1c..0000000000000000000000000000000000000000
--- a/app/HEYSTIVE_PROMPT.md
+++ /dev/null
@@ -1,485 +0,0 @@
-# 🎤 Heystive - Voice Desktop Assistant (Prompt for Gemini Code Assist Agent Mode)
-
-**For: Google Gemini Code Assist Agent Mode**
-
-Use your Agent Mode capabilities to build **Heystive** - a complete, production-ready, local voice-first desktop assistant.
-
-## What Heystive IS:
-- A **voice-first assistant**: Users SPEAK to it and it SPEAKS back (not a text chatbot)
-- **Local & private**: Runs on user's machine, not a web service
-- **Bilingual & natural**: Speaks Persian (Farsi) and English very fluently and naturally
-- **Works offline AND online**: Full functionality offline with local models; extended features when online
-- **Minimalist UI with beautiful avatar**: Extremely simple interface centered around an expressive, animated character
-- **Cross-platform**: Desktop (Windows, macOS, Linux) and Mobile (iOS, Android) with sync capabilities
-
-## Core Requirements:
-
-### 1. VOICE INTERACTION (Primary Feature)
-
-**This is NOT a chatbot. This is a voice assistant.**
-
-- Main interaction: User **speaks** → Heystive **listens** → Heystive **speaks back**
-- Text is only shown minimally for reference, not for interaction
-- Conversation flows naturally like talking to a person
-
-**Voice Technology:**
-
-**Offline Mode (must work without internet):**
-- STT (Speech-to-Text): Use `faster-whisper` or `vosk` - both support Persian well
-- TTS (Text-to-Speech): Use `piper-tts` or platform-native voices (high quality)
-- Must work completely locally with pre-downloaded models
-
-**Online Mode (optional enhancement):**
-- STT: Google Cloud Speech-to-Text or similar
-- TTS: Google Cloud TTS, Azure Neural Voices, or similar
-- Falls back to offline if connection fails
-
-**Voice Quality:**
-- Speak VERY naturally - not robotic
-- Persian: Near-native pronunciation, natural rhythm and intonation
-- English: Clear, conversational, friendly
-- Adjust speed, pitch, pauses for natural conversation flow
-
-**Multiple Voice Profiles:**
-- Implement at least 3-4 different voices (different timbres/styles)
-- Examples: "Calm Female", "Energetic Male", "Neutral Professional", etc.
-- User can select in Settings
-- Each profile has: voice ID, speed, pitch, expressiveness level
-
-**Mood Detection & Adaptation:**
-- Detect user's approximate mood from:
- - Voice: volume, speed, pitch variation, pauses
- - Text: sentiment, emotion words, frustration indicators
-- Possible moods: Calm, Stressed, Frustrated, Sad, Excited, Tired
-- Adapt response based on mood:
- - Stressed → slower, calmer tone, more supportive
- - Tired → shorter answers, clearer, slower
- - Excited → match energy slightly
-- IMPORTANT: Don't make clinical claims - just adapt helpfully
-- Must have toggle in Settings to disable this feature
-
----
-
-### 2. USER INTERFACE (Extreme Minimalism)
-
-**Main Window (Simple & Clean):**
-
-The main screen should have ONLY:
-
-1. **Avatar - Heystive's Animated Character** (center of screen, PRIMARY FOCUS)
- - **This is the HEART of Heystive's UI** - a lovable, expressive animated character
- - The avatar is NOT just a decoration - it's the main interface element
- - **Visual Design**:
- - Appealing, friendly character design (could be: abstract orb with personality, cute robot, friendly creature, or minimalist face)
- - High-quality, smooth animations
- - Multiple visual styles available (user can choose: minimalist, cartoonish, semi-3D, etc.)
- - **States & Animations**:
- - **Idle**: Gentle breathing/floating animation, occasional blinks or small movements
- - **Listening**: Active, focused state - ears perk up / eyes attentive / glowing effect
- - **Thinking**: Processing animation - subtle rotation, dots, or neural-like patterns
- - **Speaking**: Mouth/face animates in sync with speech rhythm and volume
- - **Mood-responsive**: Avatar appearance subtly shifts based on detected user mood:
- - User stressed → Avatar becomes calmer, softer colors, slower animations
- - User excited → Avatar more energetic, brighter, faster micro-movements
- - **Interactive**:
- - Clickable to start listening (alternative to hotkey)
- - Hover effects
- - Smooth transitions between all states
- - Make this avatar BEAUTIFUL, POLISHED, and ENGAGING - it should make users smile
-
-2. **Minimal Text Display** (small, unobtrusive, below or beside avatar)
- - Shows transcription of what user said (fades after moment)
- - Shows assistant's current spoken response (for reference only)
- - Supports RTL for Persian, LTR for English
- - This is SECONDARY - avatar + voice is primary
-
-3. **Hotkey Hint** (tiny label, bottom)
- - Shows: "Ctrl+Alt+Space to talk" or similar
-
-4. **Settings Icon** (small, corner)
- - Gear icon that opens Settings page
-
-**That's ALL on the main screen. No sidebars, no action logs, no command palettes.**
-
-**Avatar is the star of the show. Invest in making it delightful.**
-
----
-
-**Settings Page (Separate Window):**
-
-All configuration in a dedicated Settings window with tabs/sections:
-
-**Appearance Tab:**
-- **Avatar Style**: Choose from multiple avatar designs (minimalist orb, friendly robot, abstract character, etc.)
-- **Avatar Theme**: Color scheme for the avatar (adapt to light/dark mode, or custom colors)
-- **Animation Speed**: Adjust avatar animation speed (subtle, normal, energetic)
-- **UI Theme**: Light / Dark / Auto
-- **Window Transparency**: Adjust main window transparency (optional)
-
-**Voice Tab:**
-- Microphone selection
-- Speaker selection
-- Voice Profile dropdown (choose from available voices)
-- Speaking Speed slider
-- Mood Adaptation toggle (on/off)
-- STT Mode: Offline / Online / Auto
-- TTS Mode: Offline / Online / Auto
-
-**Mode Tab:**
-- Operation Mode: Offline / Online / Auto
-- Show current connection status
-
-**Language & UI Tab:**
-- Default language: Persian / English
-- UI direction: Auto / RTL / LTR
-- Preferred language for responses
-
-**Hotkeys Tab:**
-- Global hotkey to activate listening (default: Ctrl+Alt+Space)
-- Hotkey to show/hide Heystive window
-
-**Permissions Tab:**
-- File operations (read/write/delete) - toggle
-- Launch applications - toggle
-- Web access - toggle
-
-**Privacy Tab:**
-- Enable/disable conversation logging
-- Clear conversation history
-- Enable/disable mood detection
-
-**Sync Tab (if mobile app exists):**
-- Enable/disable sync with mobile app
-- Sync settings: Settings only / Settings + Notes / Settings + Notes + Reminders
-- Connected devices list
-- Sync status and last sync time
-
-**Advanced Tab:**
-- Show debug logs
-- Model selection (if multiple available)
-- Voice engine configuration
-- Developer mode
-
----
-
-### 3. CAPABILITIES (Real & Useful)
-
-Heystive must actually DO things, not just chat:
-
-**File & Folder Operations:**
-- Browse, read, create, edit files
-- Organize, rename, move, copy
-- Ask confirmation before deleting
-
-**Application Control:**
-- Launch apps by name ("Open VS Code", "Open Terminal")
-- Open folders in file explorer
-- Open files in appropriate editor
-
-**Web Access (Online Mode):**
-- Search the web for information
-- Open URLs in browser
-- Fetch and summarize web pages
-
-**Scripts & Automation:**
-- Generate executable scripts (Python, Bash, etc.)
-- Present script and ask before running
-- Save scripts to files
-
-**System Information:**
-- Show CPU, RAM, disk usage
-- List running processes
-- Suggest optimizations
-
-**Local Knowledge Base:**
-- Maintain local folder (`knowledge/`) with notes, docs
-- Store and retrieve information
-- Search using keywords or vector search (RAG)
-- Works offline
-
----
-
-### 4. OFFLINE vs ONLINE MODES
-
-**Offline Mode (No Internet Required):**
-
-Must work fully offline with:
-- Local STT (faster-whisper or vosk)
-- Local TTS (piper-tts or system voices)
-- File operations
-- App launching
-- Local knowledge search
-- Script generation and execution
-
-When internet-requiring feature is requested:
-- Say: "I need internet for that - currently in offline mode"
-- Offer local alternatives if possible
-
-**Online Mode:**
-
-Additional capabilities:
-- Cloud STT/TTS (better quality)
-- Web search
-- Web browsing and summarization
-- External APIs (weather, etc.)
-
-**Auto Mode:**
-- Detect internet connection
-- Use online when available, gracefully fallback to offline
-
----
-
-### 5. CONVERSATION BEHAVIOR
-
-**Natural Conversation Flow:**
-- Keep responses SHORT and conversational (not essays)
-- Use natural filler words (Persian: "خب", "ببین", "یعنی" / English: "well", "so", "let me see")
-- For technical details: give SHORT spoken summary, then ask "Do you want details?"
-- If user interrupts while speaking, stop and listen
-
-**Language:**
-- Default to Persian for Persian users
-- Seamlessly switch to English when needed
-- Detect language from user's speech
-
-**Error Handling:**
-- If something fails, explain honestly
-- Suggest fixes or alternatives
-- Never claim an action was done if it failed
-
----
-
-### 6. MOBILE APP & CROSS-PLATFORM SYNC
-
-**Desktop + Mobile Ecosystem:**
-
-Heystive should be available on both Desktop and Mobile platforms with seamless synchronization:
-
-**Desktop App (Primary):**
-- Full-featured version with all capabilities
-- Platforms: Windows, macOS, Linux
-- Avatar-centered minimalist UI
-- Complete voice interaction with local and online modes
-- Full file system access and app launching capabilities
-
-**Mobile App (Companion):**
-- iOS and Android support
-- Simplified but fully functional voice assistant
-- **Avatar remains the central UI element** (same design language as desktop)
-- Touch-optimized interface
-- Capabilities:
- - Full voice interaction (STT/TTS)
- - Create and manage notes
- - View and manage local knowledge base
- - Set reminders and TODOs
- - Quick actions (call, message, navigation - when possible)
- - Limited file operations (access to app's sandbox and user-permitted folders)
- - Web search (online mode)
- - Remote commands to desktop (when connected on same network or via cloud sync)
-
-**Synchronization Features:**
-
-When sync is enabled (user opt-in), synchronize:
-
-1. **Settings & Preferences**:
- - Avatar style and theme
- - Voice profile preferences
- - Language settings
- - Mood adaptation preferences
-
-2. **Local Knowledge**:
- - Notes and documents in `knowledge/` folder
- - Tags and organization
- - Search history (optional)
-
-3. **Reminders & TODOs**:
- - Tasks created on desktop appear on mobile and vice versa
- - Notifications on both platforms
-
-4. **Conversation Context** (optional, privacy-sensitive):
- - Recent conversation summaries
- - User can disable this completely
-
-**Sync Methods**:
-- **Local Network Sync** (preferred for privacy): When desktop and mobile are on same network, direct P2P sync
-- **Cloud Sync** (optional): Via encrypted cloud storage (user's own Google Drive, iCloud, or Dropbox)
-- User has full control over what syncs and how
-
-**Remote Control**:
-- From mobile, send commands to desktop: "On my computer, open VS Code in project X"
-- Requires explicit permission and connection setup
-- Works via local network or secure cloud relay
-
-**Implementation Priority**:
-1. Desktop app first (full implementation)
-2. Mobile app second (can be a later phase)
-3. Sync infrastructure (can be basic at first, enhanced later)
-
----
-
-### 7. IMPLEMENTATION REQUIREMENTS
-
-**Tech Stack (Suggested):**
-- **Python** for backend (easy AI/ML integration)
-- **PyQt6** or **Tkinter** for desktop UI (PyQt6 better for RTL and animations)
-- **faster-whisper** for offline STT (supports Persian well)
-- **piper-tts** for offline TTS (fast, high quality)
-- **pynput** for global hotkeys
-- **psutil** for system info
-- **chromadb** or **faiss** for local knowledge (optional but recommended)
-
-**Project Structure:**
-```
-heystive/
-├── desktop/ # Desktop application
-│ ├── main.py # Entry point
-│ ├── ui/
-│ │ ├── main_window.py # Minimalist main screen with avatar
-│ │ ├── settings_window.py # Settings page
-│ │ └── avatar/
-│ │ ├── avatar_renderer.py # Avatar animation engine
-│ │ ├── styles/ # Different avatar designs
-│ │ └── animations/ # Animation definitions
-│ ├── voice/
-│ │ ├── stt.py # Speech-to-text (offline+online)
-│ │ ├── tts.py # Text-to-speech (offline+online)
-│ │ └── mood_detector.py # Mood detection
-│ ├── agent/
-│ │ ├── brain.py # Core logic
-│ │ └── memory.py # Local knowledge
-│ └── tools/
-│ ├── file_tools.py
-│ ├── app_tools.py
-│ ├── web_tools.py
-│ └── system_tools.py
-├── mobile/ # Mobile application (iOS/Android)
-│ ├── ios/ # iOS app (Swift/SwiftUI)
-│ ├── android/ # Android app (Kotlin/Jetpack Compose)
-│ └── shared/ # Shared business logic
-│ ├── voice/ # STT/TTS for mobile
-│ ├── agent/ # Core agent logic
-│ └── ui/
-│ └── avatar/ # Avatar rendering (same design as desktop)
-├── sync/ # Synchronization infrastructure
-│ ├── sync_server.py # Local network sync server
-│ ├── cloud_sync.py # Cloud sync adapter
-│ └── protocol.py # Sync protocol definition
-├── shared/ # Shared code between desktop and mobile
-│ ├── models/ # Data models
-│ ├── config/ # Configuration management
-│ └── utils/ # Common utilities
-├── config/
-│ └── settings.json # User settings
-└── knowledge/ # Local knowledge base (synced)
-```
-
-**Development Steps:**
-
-**Phase 1: Desktop Core (Priority 1)**
-1. Create project structure with desktop, mobile, sync, and shared folders
-2. Build minimalist main window UI shell
-3. **Implement avatar system**:
- - Avatar renderer with smooth animations
- - At least 2-3 avatar styles (minimalist, friendly, abstract)
- - State animations (idle, listening, speaking, thinking)
-4. Implement offline voice pipeline (STT + TTS) with hotkey
-5. Connect voice to avatar (avatar animates with speech)
-6. Test basic voice conversation loop with avatar feedback
-7. Implement mood detection (basic version)
-8. Make avatar respond to detected mood
-9. Wire core desktop tools (files, apps, system)
-10. Build comprehensive settings page with Appearance tab for avatar customization
-11. Add local knowledge system
-12. Polish avatar animations, UI transitions, and RTL support
-13. Test thoroughly in offline and online modes
-
-**Phase 2: Mobile App (Priority 2)**
-14. Set up mobile project structure (iOS + Android)
-15. Port avatar system to mobile (same visual design)
-16. Implement mobile voice pipeline
-17. Build mobile UI with avatar as centerpiece
-18. Implement mobile-specific features (reminders, quick actions)
-19. Test mobile app thoroughly
-
-**Phase 3: Sync & Integration (Priority 3)**
-20. Implement local network sync protocol
-21. Build cloud sync adapters (Google Drive, iCloud, Dropbox)
-22. Implement settings and knowledge sync
-23. Add remote control capabilities
-24. Test sync between desktop and mobile
-25. Final polish and testing
-
-**Testing Scenarios:**
-
-Test offline mode:
-- Disconnect internet
-- Press hotkey
-- Say (Persian): "یه فایل جدید به اسم test بساز" (Create a new file called test)
-- Verify: Voice works, file created, natural Persian response
-
-Test online mode:
-- Say: "چطوری پایتون رو آپدیت کنم؟" (How do I update Python?)
-- Verify: Web search works, results spoken naturally
-
-Test mood adaptation:
-- Speak in stressed tone: "کارها تموم نمیشن، خیلی استرس دارم!"
-- Verify: Assistant responds slower, calmer, more supportive
-
----
-
-### 8. KEY PRINCIPLES
-
-1. **Voice-first**: This is NOT a chatbot - it's a voice assistant
-2. **Avatar-centered**: The animated character is the HEART of the UI - make it delightful
-3. **Minimal UI**: Main screen is just avatar + minimal text - nothing else
-4. **Natural speech**: Fluent, native-like Persian and English
-5. **Offline-capable**: Core features work without internet
-6. **Mood-aware**: Adapts to user's emotional state (with toggle)
-7. **Real capabilities**: Actually does useful things
-8. **Cross-platform**: Desktop first, then mobile, with seamless sync
-9. **Privacy-first**: Local-first, sync is opt-in, user controls data
-10. **Clean code**: Modular, extensible architecture
-
----
-
-## Your Task (Gemini Code Assist Agent Mode):
-
-Use your **Agent Mode** capabilities to implement Heystive step-by-step according to this specification.
-
-**Start with Phase 1 (Desktop Core):**
-
-1. **Plan the implementation**:
- - Analyze this specification
- - Propose detailed architecture
- - Break down into specific tasks
- - Present the plan for review
-
-2. **Implement incrementally**:
- - Set up the complete project structure
- - Build the minimalist UI shell with avatar placeholder
- - Implement the avatar system with beautiful animations
- - Wire offline voice pipeline (STT + TTS)
- - Connect avatar to voice (synchronize animations)
- - Test voice + avatar interaction loop
- - Implement mood detection and avatar mood responses
- - Add core tools (files, apps, system)
- - Build settings page with avatar customization
- - Add local knowledge system
- - Polish and test
-
-3. **Deliverables**:
- - Complete, working desktop application
- - Clean, modular, well-documented code
- - Setup instructions and documentation
- - Test scenarios and validation
-
-**Remember:**
-- Focus on **Phase 1 (Desktop Core)** first
-- The **avatar is critical** - make it engaging and polished
-- Voice quality and naturalness are paramount
-- This must be **REAL and USABLE**, not a demo or prototype
-- Use your Agent Mode to handle multi-file changes efficiently
-
-**After Phase 1 is complete and validated**, we can proceed to Phase 2 (Mobile) and Phase 3 (Sync).
-
-Let's build something amazing! 🚀
diff --git a/app/HEYSTIVE_README_FA.md b/app/HEYSTIVE_README_FA.md
deleted file mode 100644
index 29c5ad5dc340036af4d0fa9f1a46285e4ee1e6d7..0000000000000000000000000000000000000000
--- a/app/HEYSTIVE_README_FA.md
+++ /dev/null
@@ -1,366 +0,0 @@
-# 🎤 هی استیو (Heystive) چیه؟
-
-**هی استیو یه دستیار صوتیِ هوشمند و محلی برای کامپیوتر و موبایلته.**
-
-روی خود سیستم خودت نصب میشه، نه روی سرور مردم.
-
-هر وقت بگی **«هی استیو…»** یا روی دکمۀ میکروفونش بزنی، شروع میکنه:
-
-* گوش دادن 🎧
-* فهمیدن چی میخوای 🧠
-* و انجام دادن کار برات 🖥
-
----
-
-## آواتار هی استیو 🎭 (چهرهی استیو)
-
-یکی از مهمترین قسمتهای هی استیو، **آواتارشه**:
-
-* یه **کاراکتر خیلی جذاب و دوستداشتنی** که صورتِ هی استیو رو نشون میده
-* همیشه توی صفحه هست و **مرکز توجهِ** برنامه است؛ نه یه تزئین ساده!
-
-### چطور کار میکنه؟
-
-* وقتی **داره گوش میده**: حالت و نورش عوض میشه، گوشهاش تیز میشن، چشمهاش متمرکز میشن
-* وقتی **فکر میکنه**: یه انیمیشن ریز «در حال پردازش» داره (مثلاً چرخش ملایم، نقطههای روشن)
-* وقتی **حرف میزنه**: دهنش/صورتش با صداش و ریتم حرفزدن هماهنگ میشه
-* **آیدل (استراحت)**: تنفس ملایم، پلکزدن، حرکات کوچک طبیعی
-
-### هماهنگی با حال و هوای تو 😊
-
-آواتار با **حالت روحی تو** هم هماهنگ میشه:
-
-* اگه حس کنه **خستهای یا استرس داری**:
- * نرمتر و آرومتر میشه
- * رنگهاش ملایمتر میشن
- * انیمیشنهاش کندتر و آرامتر میشن
-
-* اگه فضا **شادتر** و پرانرژیه:
- * انیمیشنهاش زندهتره
- * رنگها روشنتر
- * حرکات تندتر و پرانرژیتر
-
-### انتخاب شخصیسازی 🎨
-
-میتونی از بین چند **استایل مختلف آواتار** انتخاب کنی:
-
-* **مینیمال**: یه دایره یا شکل ساده با شخصیت
-* **کارتونی**: یه شخصیت بامزه و دوستداشتنی
-* **انتزاعی**: یه موجود خلاقانه و منحصربهفرد
-* **نیمه سهبعدی**: با عمق و جزئیات بیشتر
-
-همهی این استایلها روی **دسکتاپ** و **موبایل** یکسان هستن، فقط برای موبایل بهینهتر شدن.
-
-**خلاصه: آواتار فقط یه تصویر تزئینی نیست؛ قلب تجربهی هی استیوئه و باعث میشه حس کنی با یه موجود زنده طرفی، نه یه جعبه متن.**
-
----
-
-## چطور باهات حرف میزنه و گوش میده 🎙
-
-* با **صدا** باهاش حرف میزنی، اون هم با یه صدای **روون، طبیعی و شبیه آدم** جواب میده
-* **فارسی** رو خیلی خوب میفهمه و **نیتیو** صحبت میکنه، **انگلیسی** رو هم همینطور
-* میتونه متنها رو برات **بلند بخونه**:
- * ایمیل، نوت، گزارش، TODO و…
-
-### فناوری صدا:
-
-* **آفلاین (بدون اینترنت)**:
- * از مدلهای محلی استفاده میکنه (مثل faster-whisper برای شناسایی صدا، piper-tts برای تبدیل متن به صدا)
- * همهچی روی کامپیوترت اجرا میشه، نیازی به اینترنت نیست
-
-* **آنلاین (با اینترنت)**:
- * از سرویسهای ابری گوگل یا Azure استفاده میکنه برای کیفیت بهتر
- * اگه اینترنت قطع بشه، خودکار میره روی حالت آفلاین
-
----
-
-## چند صدای مختلف و استایل گفتار 🧑🎤
-
-* هی استیو چند تا **پروفایل صدا** داره:
- * صدای **آرام و ملایم**
- * صدای **شاد و پرانرژی**
- * صدای **رسمی و حرفهای**
- * و…
-
-* میتونی برای **فارسی** و **انگلیسی** صدای جداگانه انتخاب کنی
-
-* تو تنظیمات میتونی:
- * سرعت حرف زدن رو تنظیم کنی
- * بگی همیشه با این صدا حرف بزن
- * یا بذاری خودش بر اساس موقعیت و حالت تو، لحنش رو کمی تغییر بده
-
----
-
-## تشخیص حال و هوای تو 😊💙
-
-هی استیو با دقت به:
-
-* **لحن حرف زدن**ت (سرعت، ارتفاع صدا، مکثها)
-* و **جملههایی که مینویسی یا میگی** (احساسات، کلمات)
-
-یه **حدس دوستانه** میزنه که:
-
-* الان **خستهای، کلافهای، شلوغپریشی، متمرکزی یا سرحال و شادی**
-
-### چطور عکسالعمل نشون میده؟
-
-* اگر حس کنه **خستهای یا تحت فشاری**:
- * آرومتر حرف میزنه
- * جوابهاش کوتاهتر و مهربونتر میشه
- * آواتار هم نرمتر، رنگهاش ملایمتر، و انیمیشنهاش آرامتر میشه
-
-* اگر اوضاع **خوبه و سرحالی**:
- * جوابات میتونه کمی پرانرژیتر باشه
- * آواتار هم زندهتر و روشنتر میشه
-
-> ⚠️ **مهم**: این فقط یه حس و حدس ساده و دوستانهست؛ **تشخیص پزشکی یا رسمی نیست**. میتونی این قابلیت رو از تنظیمات خاموش کنی.
-
----
-
-## کار با فایلها و پوشهها 🗂
-
-هی استیو میتونه مثل یه دستیار کامپیوتری واقعی:
-
-* توی پوشهها **بگرده**
-* فایل جدید **درست کنه** (مثلاً یادداشت روزانه، گزارش، TODO)
-* فایل **باز کنه، ویرایش کنه، جابهجا کنه، اسم عوض کنه**
-* فایلها رو **بخونه** و خلاصه کنه
-
-### مثال:
-
-> **تو**: «هی استیو، یه فایل یادداشت جدید برای امروز بساز و بازش کن.»
-
-> **هی استیو**: «باشه، ساختم و باز کردم. میخوای چیزی بنویسم توش؟»
-
-### امنیت:
-
-* قبل از کارهای حساس مثل:
- * **پاک کردن** فایلها
- * **جابهجایی** دستهجمعی
-
-همیشه **ازت میپرسه**:
-
-> «مطمئنی این کار رو انجام بدم؟»
-
----
-
-## کار با برنامهها و سیستم 🖥
-
-* میتونه **برنامهها** رو برات باز کنه:
-
- > «VS Code رو توی این پوشه باز کن.»
- >
- > «مرورگر رو باز کن و جیمیل رو بیار بالا.»
-
-* میتونه **پوشه** رو توی File Explorer / Finder باز کنه
-
-* میتونه **ترمینال/Command Prompt** رو توی یه مسیر مشخص اجرا کنه
-
-* میتونه **وضعیت سیستم** رو چک کنه:
- * مصرف رم
- * مصرف CPU
- * فضای دیسک
- * برنامههای سنگین
-
----
-
-## آنلاین و آفلاین کار میکنه 🌐❌
-
-هی استیو طوری طراحی شده که:
-
-### حالت آفلاین (بدون اینترنت) ✅
-
-* روی **فایلها، پوشهها، برنامهها** کار میکنه
-* میتونه **نوتها** و **حافظهی محلی** رو بخونه و بنویسه
-* میتونه با **مدلهای محلی** صدات رو پردازش کنه (STT/TTS آفلاین)
-* میتونه **اسکریپتها** بسازه و اجرا کنه
-
-**اگه چیزی نیاز به اینترنت داره**:
-
-> «برای این کار نیاز به اینترنت دارم - الان توی حالت آفلاینم. میتونم یه جایگزین محلی پیشنهاد بدم؟»
-
-### حالت آنلاین (با اینترنت) 🌐
-
-امکانات بیشتر:
-
-* **جستجوی وب**: سرچ کردن اطلاعات، خطاها، آموزشها
-* **صدای بهتر**: از سرویسهای ابری برای TTS/STT باکیفیتتر
-* **خلاصهسازی صفحات وب**: صفحات وب رو میخونه و خلاصه میکنه
-* **APIهای خارجی**: هوا، اخبار، و…
-
-### حالت خودکار (Auto) 🔄
-
-* **اینترنت داری؟** → از قابلیتهای آنلاین استفاده میکنه
-* **اینترنت قطع شد؟** → بهطور خودکار میره روی حالت آفلاین و کارهای محلی رو ادامه میده
-
----
-
-## حافظه و نوتبرداری 📒
-
-هی استیو فقط جواب لحظهای نمیده؛ میتونه **چیزها رو به خاطر بسپره**:
-
-### چی رو یادش میمونه؟
-
-* **نوتها و یادداشتها**:
-
- > «این رو بهعنوان توضیح پروژه X ذخیره کن.»
-
-* **توضیح پروژهها** و **مستندات محلی**
-
-* **خلاصهی مکالمات** (اختیاری - میتونی خاموش کنی)
-
-### جستجو:
-
-* بعداً میتونی بگی:
-
- > «یادداشتهای پروژه X رو بیار.»
-
- و میتونه حتی برات **خلاصهاش** کنه.
-
-* از جستجوی کلمه کلیدی یا جستجوی معنایی (RAG) استفاده میکنه
-
-* همهی این حافظه **محلی** هست و **آفلاین** کار میکنه
-
----
-
-## کارهای چندمرحلهای و برنامهریزی 🧠
-
-هی استیو فقط کارهای تکمرحلهای ساده انجام نمیده؛ میتونه:
-
-* یه سری کار **پشتسر هم** انجام بده
-
-### مثال:
-
-> **تو**: «هی استیو، لاگهای این پروژه رو بررسی کن، نمیدونم چرا دیتابیس کانکت نمیشه!»
-
-> **هی استیو**:
-> 1. لاگهای پروژه رو پیدا میکنه
-> 2. بررسی میکنه چرا سرور کرش میکنه
-> 3. یه خلاصه بهت میگه
-> 4. یه فایل گزارش درست میکنه و ذخیره میکنه
-
-### پلن قبل از اجرا:
-
-قبل از کارهای بزرگ، یه **پلن کوتاه** میگه:
-
-> «اول این رو چک میکنم، بعد این فایل رو میخونم، آخرش یه گزارش مینویسم؛ انجام بدم؟»
-
-و بعد از تأیید تو، مرحلهبهمرحله جلو میره.
-
----
-
-## امنیت و اجازه گرفتن 🛡
-
-برای کارهای حساس، هی استیو همیشه **میپرسه**:
-
-* **پاک کردن** فایلها و پوشهها
-* **اجرای اسکریپتها** و برنامههای جدید
-* **نصب** یا **تغییر چیزهای مهم** سیستم
-
-> «مطمئنی انجام بدم؟»
-
-### صداقت:
-
-* اگر خطایی پیش بیاد یا نتونه کاری رو انجام بده:
-
-> **صادقانه میگه چی شد** و تظاهر نمیکنه که کار انجام شده
-
----
-
-## همگامسازی با موبایل 📱💻
-
-هی استیو میتونه روی **دسکتاپ** (ویندوز، مک، لینوکس) و **موبایل** (iOS و اندروید) هم نصب بشه و با هم **سینک** بشن (اگه خودت فعالش کنی):
-
-### چی سینک میشه؟
-
-1. **تنظیمات مهم**:
- * زبان، نوع صدا، استایل آواتار
- * تنظیمات حریم خصوصی
-
-2. **یادداشتها و نوتها**:
- * نوتهایی که روی دسکتاپ مینویسی، روی موبایل هم نمایش داده میشن
- * و برعکس
-
-3. **یادآورها و TODO ها**:
- * کارهایی که روی یکی اضافه میکنی، روی اون یکی هم ظاهر میشه
- * نوتیفیکیشن روی هر دو
-
-4. **خلاصهی مکالمات** (اختیاری):
- * اگه بخوای، میتونی خلاصهی مکالمات اخیرت رو سینک کنی
- * میتونی این رو کاملاً خاموش کنی
-
-### چطور سینک میشه؟
-
-* **شبکه محلی** (ترجیحی برای حریم خصوصی):
- * وقتی دسکتاپ و موبایل روی یه شبکهای هستن، مستقیماً با هم ارتباط برقرار میکنن (P2P)
-
-* **سینک ابری** (اختیاری):
- * از Google Drive، iCloud، یا Dropbox خودت استفاده میکنه
- * همهچی رمزنگاری شده
-
-* **تو کنترلی**:
- * خودت تصمیم میگیری چی سینک بشه و چطور
-
-### کنترل از راه دور 🎮
-
-از موبایل میتونی **دستور** به دسکتاپ بفرستی:
-
-> **از موبایل میگی**: «روی کامپیوترم VS Code رو برای پروژه X باز کن.»
-
-> **دسکتاپ**: VS Code رو باز میکنه
-
-* نیاز به مجوز و تنظیمات داره
-* از شبکه محلی یا relay امن ابری استفاده میکنه
-
-### آواتار روی موبایل هم هست! 🎭📱
-
-* همون آواتار جذاب و زنده که روی دسکتاپ هست، روی موبایل هم هست
-* طراحی یکسان، فقط برای صفحه کوچکتر بهینه شده
-* همون انیمیشنها، همون شخصیت
-
----
-
-## اولویت پیادهسازی 🚀
-
-### فاز 1: دسکتاپ (اولویت اول) 💻
-
-* برنامه کامل دسکتاپ
-* آواتار زنده و جذاب
-* صدای طبیعی (آفلاین و آنلاین)
-* همه قابلیتها (فایل، برنامه، وب، سیستم)
-* حافظه محلی
-* تشخیص مود
-
-### فاز 2: موبایل (اولویت دوم) 📱
-
-* برنامه موبایل (iOS و Android)
-* آواتار (همون طراحی دسکتاپ، بهینهشده برای موبایل)
-* صدا (STT/TTS موبایل)
-* قابلیتهای اصلی موبایل
-
-### فاز 3: سینک (اولویت سوم) 🔄
-
-* سینک شبکه محلی
-* سینک ابری
-* کنترل از راه دور
-
----
-
-## خلاصهی خیلی کوتاه ✨
-
-**هی استیو =**
-
-* یه **دستیار صوتی محلی** برای دسکتاپ و موبایلت 🎤
-* با یه **آواتار خیلی جذاب، زنده و دوستداشتنی** که قلب برنامه است 🎭💙
-* که:
- * باهات **طبیعی و روون** حرف میزنه (فارسی و انگلیسی) 🗣
- * فایلها و برنامهها رو برات **مدیریت میکنه** 🗂🖥
- * **حال و هوات** رو تا حدی میفهمه و لحنش رو باهاش تنظیم میکنه 😊💭
- * روی **دسکتاپ و موبایل** میتونه **سینک** باشه 🔄📱💻
- * هم **با اینترنت**، هم **بدون اینترنت** کار میکنه 🌐❌
- * کارهای **واقعی و مفید** انجام میده، نه فقط چت ✅
- * **حریم خصوصیت** رو محترم میشماره - همهچی محلی، سینک اختیاریه 🔒
-
----
-
-**بیا یه چیز باحال بسازیم! 🚀**
diff --git a/app/HUGGINGFACE_DEPLOYMENT_PROMPT.md b/app/HUGGINGFACE_DEPLOYMENT_PROMPT.md
deleted file mode 100644
index c7e0ea28222aa13f5d80d7ab275c25bfa28db0d3..0000000000000000000000000000000000000000
--- a/app/HUGGINGFACE_DEPLOYMENT_PROMPT.md
+++ /dev/null
@@ -1,994 +0,0 @@
-# 🚀 Crypto-DT-Source: Complete HuggingFace Deployment Prompt
-
-**Purpose:** Complete guide to activate ALL features in the Crypto-DT-Source project for production deployment on HuggingFace Spaces
-**Target Environment:** HuggingFace Spaces + Python 3.11+
-**Deployment Season:** Q4 2025
-**Status:** Ready for Implementation
-
----
-
-## 📋 Executive Summary
-
-This prompt provides a **complete roadmap** to transform Crypto-DT-Source from a monitoring platform into a **fully-functional cryptocurrency data aggregation service**. All 50+ endpoints will be connected to real data sources, database persistence will be integrated, AI models will be loaded, and the system will be optimized for HuggingFace Spaces deployment.
-
-**Expected Outcome:**
-- ✅ Real crypto market data (live prices, OHLCV, trending coins)
-- ✅ Historical data storage in SQLite
-- ✅ AI-powered sentiment analysis using HuggingFace transformers
-- ✅ Authentication + rate limiting on all endpoints
-- ✅ WebSocket real-time streaming
-- ✅ Provider health monitoring with intelligent failover
-- ✅ Automatic provider discovery
-- ✅ Full diagnostic and monitoring capabilities
-- ✅ Production-ready Docker deployment to HF Spaces
-
----
-
-## 🎯 Implementation Priorities (Phase 1-4)
-
-### **Phase 1: Core Data Integration (CRITICAL)**
-*Goal: Replace all mock data with real API calls*
-
-#### 1.1 Market Data Endpoints
-**Files to modify:**
-- `api/endpoints.py` - `/api/market`, `/api/prices`
-- `collectors/market_data_extended.py` - Real price fetching
-- `api_server_extended.py` - FastAPI endpoints
-
-**Requirements:**
-- Remove all hardcoded mock data from endpoints
-- Implement real API calls to CoinGecko, CoinCap, Binance
-- Use async/await pattern for non-blocking calls
-- Implement caching layer (5-minute TTL for prices)
-- Add error handling with provider fallback
-
-**Implementation Steps:**
-```python
-# Example: Replace mock market data with real provider data
-GET /api/market
-├── Call ProviderManager.get_best_provider('market_data')
-├── Execute async request to provider
-├── Cache response (5 min TTL)
-├── Return real BTC/ETH prices instead of mock
-└── Fallback to secondary provider on failure
-
-GET /api/prices?symbols=BTC,ETH,SOL
-├── Parse symbol list
-├── Call ProviderManager for each symbol
-├── Aggregate responses
-├── Return real-time price data
-
-GET /api/trending
-├── Call CoinGecko trending endpoint
-├── Store in database
-└── Return top 7 trending coins
-
-GET /api/ohlcv?symbol=BTCUSDT&interval=1h&limit=100
-├── Call Binance OHLCV endpoint
-├── Validate symbol format
-├── Apply caching (15-min TTL)
-└── Return historical OHLCV data
-```
-
-**Success Criteria:**
-- [ ] All endpoints return real data from providers
-- [ ] Caching implemented with configurable TTL
-- [ ] Provider failover working (when primary fails)
-- [ ] Response times < 2 seconds
-- [ ] No hardcoded mock data in endpoint responses
-
----
-
-#### 1.2 DeFi Data Endpoints
-**Files to modify:**
-- `api_server_extended.py` - `/api/defi` endpoint
-- `collectors/` - Add DeFi collector
-
-**Requirements:**
-- Fetch TVL data from DeFi Llama API
-- Track top DeFi protocols
-- Cache for 1 hour (DeFi data updates less frequently)
-
-**Implementation:**
-```python
-GET /api/defi
-├── Call DeFi Llama: GET /protocols
-├── Filter top 20 by TVL
-├── Parse response (name, TVL, chain, symbol)
-├── Store in database (defi_protocols table)
-└── Return with timestamp
-
-GET /api/defi/tvl-chart
-├── Query historical TVL from database
-├── Aggregate by date
-└── Return 30-day TVL trend
-```
-
----
-
-#### 1.3 News & Sentiment Integration
-**Files to modify:**
-- `collectors/sentiment_extended.py`
-- `api/endpoints.py` - `/api/sentiment` endpoint
-
-**Requirements:**
-- Fetch news from RSS feeds (CoinDesk, Cointelegraph, etc.)
-- Implement real HuggingFace sentiment analysis (NOT keyword matching)
-- Store sentiment scores in database
-- Track Fear & Greed Index
-
-**Implementation:**
-```python
-GET /api/sentiment
-├── Query recent news from database
-├── Load HuggingFace model: distilbert-base-uncased-finetuned-sst-2-english
-├── Analyze each headline/article
-├── Calculate aggregate sentiment score
-├── Return: {overall_sentiment, fear_greed_index, top_sentiments}
-
-GET /api/news
-├── Fetch from RSS feeds (configurable)
-├── Run through sentiment analyzer
-├── Store in database (news table with sentiment)
-├── Return paginated results
-
-POST /api/analyze/text
-├── Accept raw text input
-├── Run HuggingFace sentiment model
-├── Return: {text, sentiment, confidence, label}
-```
-
----
-
-### **Phase 2: Database Integration (HIGH PRIORITY)**
-*Goal: Full persistent storage of all data*
-
-#### 2.1 Database Schema Activation
-**Files:**
-- `database/models.py` - Define all tables
-- `database/migrations.py` - Schema setup
-- `database/db_manager.py` - Connection management
-
-**Tables to Activate:**
-```sql
--- Core tables
-prices (id, symbol, price, timestamp, provider)
-ohlcv (id, symbol, open, high, low, close, volume, timestamp)
-news (id, title, content, sentiment, source, timestamp)
-defi_protocols (id, name, tvl, chain, timestamp)
-market_snapshots (id, btc_price, eth_price, market_cap, timestamp)
-
--- Metadata tables
-providers (id, name, status, health_score, last_check)
-pools (id, name, strategy, created_at)
-api_calls (id, endpoint, provider, response_time, status)
-user_requests (id, ip_address, endpoint, timestamp)
-```
-
-**Implementation:**
-```python
-# In api_server_extended.py startup:
-
-@app.on_event("startup")
-async def startup_event():
- # Initialize database
- db_manager = DBManager()
- await db_manager.initialize()
-
- # Run migrations
- await db_manager.run_migrations()
-
- # Create tables if not exist
- await db_manager.create_all_tables()
-
- # Verify connectivity
- health = await db_manager.health_check()
- logger.info(f"Database initialized: {health}")
-```
-
-#### 2.2 API Endpoints ↔ Database Integration
-**Pattern to implement:**
-
-```python
-# Write pattern: After fetching real data, store it
-async def store_market_snapshot():
- # Fetch real data
- prices = await provider_manager.get_market_data()
-
- # Store in database
- async with db.session() as session:
- snapshot = MarketSnapshot(
- btc_price=prices['BTC'],
- eth_price=prices['ETH'],
- market_cap=prices['market_cap'],
- timestamp=datetime.now()
- )
- session.add(snapshot)
- await session.commit()
-
- return prices
-
-# Read pattern: Query historical data
-@app.get("/api/prices/history/{symbol}")
-async def get_price_history(symbol: str, days: int = 30):
- async with db.session() as session:
- history = await session.query(Price).filter(
- Price.symbol == symbol,
- Price.timestamp >= datetime.now() - timedelta(days=days)
- ).all()
-
- return [{"price": p.price, "timestamp": p.timestamp} for p in history]
-```
-
-**Success Criteria:**
-- [ ] All real-time data is persisted to database
-- [ ] Historical queries return > 30 days of data
-- [ ] Database is queried for price history endpoints
-- [ ] Migrations run automatically on startup
-- [ ] No data loss on server restart
-
----
-
-### **Phase 3: AI & Sentiment Analysis (MEDIUM PRIORITY)**
-*Goal: Real ML-powered sentiment analysis*
-
-#### 3.1 Load HuggingFace Models
-**Files:**
-- `ai_models.py` - Model loading and inference
-- Update `requirements.txt` with torch, transformers
-
-**Models to Load:**
-```python
-# Sentiment Analysis
-SENTIMENT_MODELS = [
- "distilbert-base-uncased-finetuned-sst-2-english", # Fast, accurate
- "cardiffnlp/twitter-roberta-base-sentiment-latest", # Social media optimized
- "ProsusAI/finBERT", # Financial sentiment
-]
-
-# Crypto-specific models
-CRYPTO_MODELS = [
- "EleutherAI/gpt-neo-125M", # General purpose (lightweight)
- "facebook/opt-125m", # Instruction following
-]
-
-# Zero-shot classification for custom sentiment
-"facebook/bart-large-mnli" # Multi-class sentiment (bullish/bearish/neutral)
-```
-
-**Implementation:**
-```python
-# ai_models.py
-
-class AIModelManager:
- def __init__(self):
- self.models = {}
- self.device = "cuda" if torch.cuda.is_available() else "cpu"
-
- async def initialize(self):
- """Load all models on startup"""
- logger.info("Loading HuggingFace models...")
-
- # Sentiment analysis
- self.models['sentiment'] = pipeline(
- "sentiment-analysis",
- model="distilbert-base-uncased-finetuned-sst-2-english",
- device=0 if self.device == "cuda" else -1
- )
-
- # Zero-shot for crypto sentiment
- self.models['zeroshot'] = pipeline(
- "zero-shot-classification",
- model="facebook/bart-large-mnli",
- device=0 if self.device == "cuda" else -1
- )
-
- logger.info("Models loaded successfully")
-
- async def analyze_sentiment(self, text: str) -> dict:
- """Analyze sentiment of text"""
- if not self.models.get('sentiment'):
- return {"error": "Model not loaded", "sentiment": "unknown"}
-
- result = self.models['sentiment'](text)[0]
-
- return {
- "text": text[:100],
- "label": result['label'],
- "score": result['score'],
- "timestamp": datetime.now().isoformat()
- }
-
- async def analyze_crypto_sentiment(self, text: str) -> dict:
- """Crypto-specific sentiment (bullish/bearish/neutral)"""
- candidate_labels = ["bullish", "bearish", "neutral"]
- result = self.models['zeroshot'](text, candidate_labels)
-
- return {
- "text": text[:100],
- "sentiment": result['labels'][0],
- "scores": dict(zip(result['labels'], result['scores'])),
- "timestamp": datetime.now().isoformat()
- }
-
-# In api_server_extended.py
-ai_manager = AIModelManager()
-
-@app.on_event("startup")
-async def startup():
- await ai_manager.initialize()
-
-@app.post("/api/sentiment/analyze")
-async def analyze_sentiment(request: AnalyzeRequest):
- """Real sentiment analysis endpoint"""
- result = await ai_manager.analyze_sentiment(request.text)
- return result
-
-@app.post("/api/sentiment/crypto-analysis")
-async def crypto_sentiment(request: AnalyzeRequest):
- """Crypto-specific sentiment analysis"""
- result = await ai_manager.analyze_crypto_sentiment(request.text)
- return result
-```
-
-#### 3.2 News Sentiment Pipeline
-**Implementation:**
-
-```python
-# Background task: Analyze news sentiment continuously
-
-async def analyze_news_sentiment():
- """Run every 30 minutes: fetch news and analyze sentiment"""
- while True:
- try:
- # 1. Fetch recent news from feeds
- news_items = await fetch_rss_feeds()
-
- # 2. Store news items
- for item in news_items:
- # 3. Analyze sentiment
- sentiment = await ai_manager.analyze_sentiment(item['title'])
-
- # 4. Store in database
- async with db.session() as session:
- news = News(
- title=item['title'],
- content=item['content'],
- source=item['source'],
- sentiment=sentiment['label'],
- confidence=sentiment['score'],
- timestamp=datetime.now()
- )
- session.add(news)
-
- await session.commit()
- logger.info(f"Analyzed {len(news_items)} news items")
-
- except Exception as e:
- logger.error(f"News sentiment pipeline error: {e}")
-
- # Wait 30 minutes
- await asyncio.sleep(1800)
-
-# Start in background on app startup
-@app.on_event("startup")
-async def startup():
- asyncio.create_task(analyze_news_sentiment())
-```
-
----
-
-### **Phase 4: Security & Production Setup (HIGH PRIORITY)**
-*Goal: Production-ready authentication, rate limiting, and monitoring*
-
-#### 4.1 Authentication Implementation
-**Files:**
-- `utils/auth.py` - JWT token handling
-- `api/security.py` - New file for security middleware
-
-**Implementation:**
-
-```python
-# utils/auth.py
-
-from fastapi import Depends, HTTPException, status
-from fastapi.security import HTTPBearer, HTTPAuthCredentials
-import jwt
-from datetime import datetime, timedelta
-
-SECRET_KEY = os.getenv("JWT_SECRET_KEY", "your-secret-key-change-in-production")
-ALGORITHM = "HS256"
-
-class AuthManager:
- @staticmethod
- def create_token(user_id: str, hours: int = 24) -> str:
- """Create JWT token"""
- payload = {
- "user_id": user_id,
- "exp": datetime.utcnow() + timedelta(hours=hours),
- "iat": datetime.utcnow()
- }
- return jwt.encode(payload, SECRET_KEY, algorithm=ALGORITHM)
-
- @staticmethod
- def verify_token(token: str) -> str:
- """Verify JWT token"""
- try:
- payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
- return payload.get("user_id")
- except jwt.ExpiredSignatureError:
- raise HTTPException(status_code=401, detail="Token expired")
- except jwt.InvalidTokenError:
- raise HTTPException(status_code=401, detail="Invalid token")
-
-security = HTTPBearer()
-auth_manager = AuthManager()
-
-async def get_current_user(credentials: HTTPAuthCredentials = Depends(security)):
- """Dependency for protected endpoints"""
- return auth_manager.verify_token(credentials.credentials)
-
-# In api_server_extended.py
-@app.post("/api/auth/token")
-async def get_token(api_key: str):
- """Issue JWT token for API key"""
- # Validate API key against database
- user = await verify_api_key(api_key)
- if not user:
- raise HTTPException(status_code=401, detail="Invalid API key")
-
- token = auth_manager.create_token(user.id)
- return {"access_token": token, "token_type": "bearer"}
-
-# Protected endpoint example
-@app.get("/api/protected-data")
-async def protected_endpoint(current_user: str = Depends(get_current_user)):
- """This endpoint requires authentication"""
- return {"user_id": current_user, "data": "sensitive"}
-```
-
-#### 4.2 Rate Limiting
-**Files:**
-- `utils/rate_limiter_enhanced.py` - Enhanced rate limiter
-
-**Implementation:**
-
-```python
-# In api_server_extended.py
-from slowapi import Limiter
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-limiter = Limiter(key_func=get_remote_address)
-app.state.limiter = limiter
-
-# Rate limit configuration
-FREE_TIER = "30/minute" # 30 requests per minute
-PRO_TIER = "300/minute" # 300 requests per minute
-ADMIN_TIER = None # Unlimited
-
-@app.exception_handler(RateLimitExceeded)
-async def rate_limit_handler(request, exc):
- return JSONResponse(
- status_code=429,
- content={"error": "Rate limit exceeded", "retry_after": 60}
- )
-
-# Apply to endpoints
-@app.get("/api/prices")
-@limiter.limit(FREE_TIER)
-async def get_prices(request: Request):
- return await prices_handler()
-
-@app.get("/api/sentiment")
-@limiter.limit(FREE_TIER)
-async def get_sentiment(request: Request):
- return await sentiment_handler()
-
-# Premium endpoints
-@app.get("/api/historical-data")
-@limiter.limit(PRO_TIER)
-async def get_historical_data(request: Request, current_user: str = Depends(get_current_user)):
- return await historical_handler()
-```
-
-**Tier Configuration:**
-```python
-RATE_LIMIT_TIERS = {
- "free": {
- "requests_per_minute": 30,
- "requests_per_day": 1000,
- "max_symbols": 5,
- "data_retention_days": 7
- },
- "pro": {
- "requests_per_minute": 300,
- "requests_per_day": 50000,
- "max_symbols": 100,
- "data_retention_days": 90
- },
- "enterprise": {
- "requests_per_minute": None, # Unlimited
- "requests_per_day": None,
- "max_symbols": None,
- "data_retention_days": None
- }
-}
-```
-
----
-
-#### 4.3 Monitoring & Diagnostics
-**Files:**
-- `api/endpoints.py` - Diagnostic endpoints
-- `monitoring/health_monitor.py` - Health checks
-
-**Implementation:**
-
-```python
-@app.get("/api/health")
-async def health_check():
- """Comprehensive health check"""
- return {
- "status": "healthy",
- "timestamp": datetime.now().isoformat(),
- "components": {
- "database": await check_database(),
- "providers": await check_providers(),
- "models": await check_models(),
- "websocket": await check_websocket(),
- "cache": await check_cache()
- },
- "metrics": {
- "uptime_seconds": get_uptime(),
- "active_connections": active_ws_count(),
- "request_count_1h": get_request_count("1h"),
- "average_response_time_ms": get_avg_response_time()
- }
- }
-
-@app.post("/api/diagnostics/run")
-async def run_diagnostics(auto_fix: bool = False):
- """Full system diagnostics"""
- issues = []
- fixes = []
-
- # Check all components
- checks = [
- check_database_integrity(),
- check_provider_health(),
- check_disk_space(),
- check_memory_usage(),
- check_model_availability(),
- check_config_files(),
- check_required_directories(),
- verify_api_connectivity()
- ]
-
- results = await asyncio.gather(*checks)
-
- for check in results:
- if check['status'] != 'ok':
- issues.append(check)
- if auto_fix:
- fix = await apply_fix(check)
- fixes.append(fix)
-
- return {
- "timestamp": datetime.now().isoformat(),
- "total_checks": len(checks),
- "issues_found": len(issues),
- "issues": issues,
- "fixes_applied": fixes if auto_fix else []
- }
-
-@app.get("/api/metrics")
-async def get_metrics():
- """System metrics for monitoring"""
- return {
- "cpu_percent": psutil.cpu_percent(interval=1),
- "memory_percent": psutil.virtual_memory().percent,
- "disk_percent": psutil.disk_usage('/').percent,
- "database_size_mb": get_database_size() / 1024 / 1024,
- "active_requests": active_request_count(),
- "websocket_connections": active_ws_count(),
- "provider_stats": await get_provider_statistics()
- }
-```
-
----
-
-### **Phase 5: Background Tasks & Auto-Discovery**
-*Goal: Continuous operation with automatic provider discovery*
-
-#### 5.1 Background Tasks
-**Files:**
-- `scheduler.py` - Task scheduling
-- `monitoring/scheduler_comprehensive.py` - Enhanced scheduler
-
-**Tasks to Activate:**
-
-```python
-# In api_server_extended.py
-
-@app.on_event("startup")
-async def start_background_tasks():
- """Start all background tasks"""
-
- tasks = [
- # Data collection tasks
- asyncio.create_task(collect_prices_every_5min()),
- asyncio.create_task(collect_defi_data_every_hour()),
- asyncio.create_task(fetch_news_every_30min()),
- asyncio.create_task(analyze_sentiment_every_hour()),
-
- # Health & monitoring tasks
- asyncio.create_task(health_check_every_5min()),
- asyncio.create_task(broadcast_stats_every_5min()),
- asyncio.create_task(cleanup_old_logs_daily()),
- asyncio.create_task(backup_database_daily()),
- asyncio.create_task(send_diagnostics_hourly()),
-
- # Discovery tasks (optional)
- asyncio.create_task(discover_new_providers_daily()),
- ]
-
- logger.info(f"Started {len(tasks)} background tasks")
-
-# Scheduled tasks with cron-like syntax
-TASK_SCHEDULE = {
- "collect_prices": "*/5 * * * *", # Every 5 minutes
- "collect_defi": "0 * * * *", # Hourly
- "fetch_news": "*/30 * * * *", # Every 30 minutes
- "sentiment_analysis": "0 * * * *", # Hourly
- "health_check": "*/5 * * * *", # Every 5 minutes
- "backup_database": "0 2 * * *", # Daily at 2 AM
- "cleanup_logs": "0 3 * * *", # Daily at 3 AM
-}
-```
-
-#### 5.2 Auto-Discovery Service
-**Files:**
-- `backend/services/auto_discovery_service.py` - Discovery logic
-
-**Implementation:**
-
-```python
-# Enable in environment
-ENABLE_AUTO_DISCOVERY=true
-AUTO_DISCOVERY_INTERVAL_HOURS=24
-
-class AutoDiscoveryService:
- """Automatically discover new crypto API providers"""
-
- async def discover_providers(self) -> List[Provider]:
- """Scan for new providers"""
- discovered = []
-
- sources = [
- self.scan_github_repositories,
- self.scan_api_directories,
- self.scan_rss_feeds,
- self.query_existing_apis,
- ]
-
- for source in sources:
- try:
- providers = await source()
- discovered.extend(providers)
- logger.info(f"Discovered {len(providers)} from {source.__name__}")
- except Exception as e:
- logger.error(f"Discovery error in {source.__name__}: {e}")
-
- # Validate and store
- valid = []
- for provider in discovered:
- if await self.validate_provider(provider):
- await self.store_provider(provider)
- valid.append(provider)
-
- return valid
-
- async def scan_github_repositories(self):
- """Search GitHub for crypto API projects"""
- # Query GitHub API for relevant repos
- # Extract API endpoints
- # Return as Provider objects
- pass
-
- async def validate_provider(self, provider: Provider) -> bool:
- """Test if provider is actually available"""
- try:
- async with aiohttp.ClientSession() as session:
- async with session.get(
- provider.base_url,
- timeout=aiohttp.ClientTimeout(total=5)
- ) as resp:
- return resp.status < 500
- except:
- return False
-
-# Start discovery on demand
-@app.post("/api/discovery/run")
-async def trigger_discovery(background: bool = True):
- """Trigger provider discovery"""
- discovery_service = AutoDiscoveryService()
-
- if background:
- asyncio.create_task(discovery_service.discover_providers())
- return {"status": "Discovery started in background"}
- else:
- providers = await discovery_service.discover_providers()
- return {"discovered": len(providers), "providers": providers}
-```
-
----
-
-## 🐳 HuggingFace Spaces Deployment
-
-### Configuration for HF Spaces
-
-**`spaces/app.py` (Entry point):**
-```python
-import os
-import sys
-
-# Set environment for HF Spaces
-os.environ['HF_SPACE'] = 'true'
-os.environ['PORT'] = '7860' # HF Spaces default port
-
-# Import and start the main FastAPI app
-from api_server_extended import app
-
-if __name__ == "__main__":
- import uvicorn
-
- uvicorn.run(
- app,
- host="0.0.0.0",
- port=7860,
- log_level="info"
- )
-```
-
-**`spaces/requirements.txt`:**
-```
-fastapi==0.109.0
-uvicorn[standard]==0.27.0
-aiohttp==3.9.1
-pydantic==2.5.3
-websockets==12.0
-sqlalchemy==2.0.23
-torch==2.1.1
-transformers==4.35.2
-huggingface-hub==0.19.1
-slowapi==0.1.9
-python-jose==3.3.0
-psutil==5.9.6
-aiofiles==23.2.1
-```
-
-**`spaces/README.md`:**
-```markdown
-# Crypto-DT-Source on HuggingFace Spaces
-
-Real-time cryptocurrency data aggregation service with 200+ providers.
-
-## Features
-- Real-time price data
-- AI sentiment analysis
-- 50+ REST endpoints
-- WebSocket streaming
-- Provider health monitoring
-- Historical data storage
-
-## API Documentation
-- Swagger UI: https://[your-space-url]/docs
-- ReDoc: https://[your-space-url]/redoc
-
-## Quick Start
-```bash
-curl https://[your-space-url]/api/health
-curl https://[your-space-url]/api/prices?symbols=BTC,ETH
-curl https://[your-space-url]/api/sentiment
-```
-
-## WebSocket Connection
-```javascript
-const ws = new WebSocket('wss://[your-space-url]/ws');
-ws.onmessage = (event) => console.log(JSON.parse(event.data));
-```
-```
-
----
-
-## ✅ Activation Checklist
-
-### Phase 1: Data Integration
-- [ ] Modify `/api/market` to return real CoinGecko data
-- [ ] Modify `/api/prices` to fetch real provider data
-- [ ] Modify `/api/trending` to return live trending coins
-- [ ] Implement `/api/ohlcv` with Binance data
-- [ ] Implement `/api/defi` with DeFi Llama data
-- [ ] Remove all hardcoded mock data
-- [ ] Test all endpoints with real data
-- [ ] Add caching layer (5-30 min TTL based on endpoint)
-
-### Phase 2: Database
-- [ ] Run database migrations
-- [ ] Create all required tables
-- [ ] Implement write pattern for real data storage
-- [ ] Implement read pattern for historical queries
-- [ ] Add database health check
-- [ ] Test data persistence across restarts
-- [ ] Implement cleanup tasks for old data
-
-### Phase 3: AI & Sentiment
-- [ ] Install transformers and torch
-- [ ] Load HuggingFace sentiment model
-- [ ] Implement sentiment analysis endpoint
-- [ ] Implement crypto-specific sentiment classification
-- [ ] Create news sentiment pipeline
-- [ ] Store sentiment scores in database
-- [ ] Test model inference latency
-
-### Phase 4: Security
-- [ ] Generate JWT secret key
-- [ ] Implement authentication middleware
-- [ ] Create API key management system
-- [ ] Implement rate limiting on all endpoints
-- [ ] Add tier-based rate limits (free/pro/enterprise)
-- [ ] Create `/api/auth/token` endpoint
-- [ ] Test authentication on protected endpoints
-- [ ] Set up HTTPS certificate for CORS
-
-### Phase 5: Background Tasks
-- [ ] Activate all scheduled tasks
-- [ ] Set up price collection (every 5 min)
-- [ ] Set up DeFi data collection (hourly)
-- [ ] Set up news fetching (every 30 min)
-- [ ] Set up sentiment analysis (hourly)
-- [ ] Set up health checks (every 5 min)
-- [ ] Set up database backup (daily)
-- [ ] Set up log cleanup (daily)
-
-### Phase 6: HF Spaces Deployment
-- [ ] Create `spaces/` directory
-- [ ] Create `spaces/app.py` entry point
-- [ ] Create `spaces/requirements.txt`
-- [ ] Create `spaces/README.md`
-- [ ] Configure environment variables
-- [ ] Test locally with Docker
-- [ ] Push to HF Spaces
-- [ ] Verify all endpoints accessible
-- [ ] Monitor logs and metrics
-- [ ] Set up auto-restart on failure
-
----
-
-## 🔧 Environment Variables
-
-```bash
-# Core
-PORT=7860
-ENVIRONMENT=production
-LOG_LEVEL=info
-
-# Database
-DATABASE_URL=sqlite:///data/crypto_aggregator.db
-DATABASE_POOL_SIZE=20
-
-# Security
-JWT_SECRET_KEY=your-secret-key-change-in-production
-API_KEY_SALT=your-salt-key
-
-# HuggingFace Spaces
-HF_SPACE=true
-HF_SPACE_URL=https://huggingface.co/spaces/your-username/crypto-dt-source
-
-# Features
-ENABLE_AUTO_DISCOVERY=true
-ENABLE_SENTIMENT_ANALYSIS=true
-ENABLE_BACKGROUND_TASKS=true
-
-# Rate Limiting
-FREE_TIER_LIMIT=30/minute
-PRO_TIER_LIMIT=300/minute
-
-# Caching
-CACHE_TTL_PRICES=300 # 5 minutes
-CACHE_TTL_DEFI=3600 # 1 hour
-CACHE_TTL_NEWS=1800 # 30 minutes
-
-# Providers (optional API keys)
-ETHERSCAN_API_KEY=
-BSCSCAN_API_KEY=
-COINGECKO_API_KEY=
-```
-
----
-
-## 📊 Expected Performance
-
-After implementation:
-
-| Metric | Target | Current |
-|--------|--------|---------|
-| Price endpoint response time | < 500ms | N/A |
-| Sentiment analysis latency | < 2s | N/A |
-| WebSocket update frequency | Real-time | ✅ Working |
-| Database query latency | < 100ms | N/A |
-| Provider failover time | < 2s | ✅ Working |
-| Authentication overhead | < 50ms | N/A |
-| Concurrent connections supported | 1000+ | ✅ Tested |
-
----
-
-## 🚨 Troubleshooting
-
-### Models not loading on HF Spaces
-```bash
-# HF Spaces has limited disk space
-# Use distilbert models (smaller) instead of full models
-# Or cache models in requirements
-pip install --no-cache-dir transformers torch
-```
-
-### Database file too large
-```bash
-# Implement cleanup task
-# Keep only 90 days of data
-# Archive old data to S3
-```
-
-### Rate limiting too aggressive
-```bash
-# Adjust limits in environment
-FREE_TIER_LIMIT=100/minute
-PRO_TIER_LIMIT=500/minute
-```
-
-### WebSocket disconnections
-```bash
-# Increase heartbeat frequency
-WEBSOCKET_HEARTBEAT_INTERVAL=10 # seconds
-WEBSOCKET_HEARTBEAT_TIMEOUT=30 # seconds
-```
-
----
-
-## 📚 Next Steps
-
-1. **Review Phase 1-2**: Data integration and database
-2. **Review Phase 3-4**: AI and security implementations
-3. **Review Phase 5-6**: Background tasks and HF deployment
-4. **Execute implementation** following the checklist
-5. **Test thoroughly** before production deployment
-6. **Monitor metrics** and adjust configurations
-7. **Collect user feedback** and iterate
-
----
-
-## 🎯 Success Criteria
-
-Project is **production-ready** when:
-
-✅ All 50+ endpoints return real data
-✅ Database stores 90 days of historical data
-✅ Sentiment analysis runs on real ML models
-✅ Authentication required on all protected endpoints
-✅ Rate limiting enforced across all tiers
-✅ Background tasks running without errors
-✅ Health check returns all components OK
-✅ WebSocket clients can stream real-time data
-✅ Auto-discovery discovers new providers
-✅ Deployed on HuggingFace Spaces successfully
-✅ Average response time < 1 second
-✅ Zero downtime during operation
-
----
-
-**Document Version:** 2.0
-**Last Updated:** 2025-11-15
-**Maintained by:** Claude Code AI
-**Status:** Ready for Implementation
diff --git a/app/HUGGINGFACE_DIAGNOSTIC_GUIDE.md b/app/HUGGINGFACE_DIAGNOSTIC_GUIDE.md
deleted file mode 100644
index 2a6d078533776781f365f33472f02c6be7f9a99d..0000000000000000000000000000000000000000
--- a/app/HUGGINGFACE_DIAGNOSTIC_GUIDE.md
+++ /dev/null
@@ -1,1933 +0,0 @@
-# 🔍 Complete Diagnostic & Fix Guide
-## HuggingFace Space Integration Troubleshooting
-
-**Version:** 2.0
-**Last Updated:** 2025-11-15
-**Target:** Node.js/React ↔ HuggingFace Space Integration
-**Space URL:** https://really-amin-datasourceforcryptocurrency.hf.space
-
----
-
-## 📋 Table of Contents
-
-1. [Quick Start Diagnostic](#quick-start-diagnostic)
-2. [Pre-Flight Checks](#pre-flight-checks)
-3. [Automated Diagnostic Script](#automated-diagnostic-script)
-4. [Common Issues & Fixes](#common-issues--fixes)
-5. [Testing Protocol](#testing-protocol)
-6. [Debugging Commands](#debugging-commands)
-7. [Configuration Guide](#configuration-guide)
-8. [Troubleshooting Decision Tree](#troubleshooting-decision-tree)
-9. [FAQ](#faq)
-
----
-
-## 🚀 Quick Start Diagnostic
-
-### Step 1: Check HuggingFace Space Status
-
-```bash
-# Test if Space is alive
-curl -v https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Expected Output:
-# HTTP/2 200
-# {"status": "healthy"}
-
-# If you get:
-# - Connection timeout → Space is sleeping or down
-# - 404 Not Found → Endpoint doesn't exist
-# - 503 Service Unavailable → Space is building
-```
-
-### Step 2: Discover Available Endpoints
-
-```bash
-# Try common endpoints
-echo "Testing /api/health..."
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/health | jq
-
-echo "Testing /api/prices..."
-curl -s "https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC,ETH" | jq
-
-echo "Testing /api/ohlcv..."
-curl -s "https://really-amin-datasourceforcryptocurrency.hf.space/api/ohlcv?symbol=BTCUSDT&interval=1h&limit=10" | jq
-
-echo "Testing /api/market/overview..."
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/market/overview | jq
-
-echo "Testing /api/sentiment..."
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/sentiment | jq
-
-echo "Testing /docs (API documentation)..."
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/docs | head -n 50
-```
-
-### Step 3: Quick Application Test
-
-```bash
-# Setup environment
-cp .env.example .env
-
-# Edit .env file - set:
-# PRIMARY_DATA_SOURCE=huggingface
-# HF_SPACE_BASE_URL=https://really-amin-datasourceforcryptocurrency.hf.space
-
-# Install dependencies
-npm install
-
-# Start development server
-npm run dev
-
-# Open browser and check:
-# 1. http://localhost:5173
-# 2. Open DevTools (F12)
-# 3. Go to Network tab
-# 4. Check for any red requests
-# 5. Go to Console tab
-# 6. Look for error messages
-```
-
----
-
-## ✅ Pre-Flight Checks
-
-Before troubleshooting, verify these requirements:
-
-### System Requirements
-
-```bash
-# Check Node.js version (should be 18+)
-node --version
-# Expected: v18.0.0 or higher
-
-# Check npm version
-npm --version
-# Expected: 9.0.0 or higher
-
-# Check if git is installed
-git --version
-
-# Check if curl is available
-curl --version
-
-# Check if jq is installed (optional but helpful)
-jq --version
-# If not installed: sudo apt-get install jq (Ubuntu) or brew install jq (Mac)
-```
-
-### Project Structure Verification
-
-```bash
-# Verify critical files exist
-ls -la hf-data-engine/main.py
-ls -la hf-data-engine/requirements.txt
-ls -la .env.example
-ls -la package.json
-
-# If any file is missing, run:
-git status
-git pull origin main
-```
-
-### Dependencies Installation
-
-```bash
-# Clean install
-rm -rf node_modules package-lock.json
-npm install
-
-# Verify critical packages
-npm list typescript
-npm list vite
-npm list react
-
-# For Python dependencies (if working with backend)
-cd hf-data-engine
-pip install -r requirements.txt
-cd ..
-```
-
-### Environment Configuration
-
-```bash
-# Check if .env exists
-if [ ! -f .env ]; then
- echo "⚠️ .env file not found!"
- echo "Creating from .env.example..."
- cp .env.example .env
-else
- echo "✅ .env file exists"
-fi
-
-# Verify required variables
-grep -q "PRIMARY_DATA_SOURCE" .env && echo "✅ PRIMARY_DATA_SOURCE configured" || echo "❌ PRIMARY_DATA_SOURCE missing"
-grep -q "HF_SPACE_BASE_URL" .env && echo "✅ HF_SPACE_BASE_URL configured" || echo "❌ HF_SPACE_BASE_URL missing"
-
-# View current configuration (non-sensitive parts)
-echo ""
-echo "Current configuration:"
-grep "PRIMARY_DATA_SOURCE\|HF_SPACE" .env | sed 's/=.*/=***/'
-```
-
----
-
-## 🤖 Automated Diagnostic Script
-
-Save this as `diagnostic.sh` in your project root and run with `bash diagnostic.sh`:
-
-```bash
-#!/bin/bash
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-echo "╔════════════════════════════════════════════════════════╗"
-echo "║ HuggingFace Space Integration Diagnostic Tool ║"
-echo "╚════════════════════════════════════════════════════════╝"
-echo ""
-
-# Configuration
-HF_SPACE_URL="https://really-amin-datasourceforcryptocurrency.hf.space"
-RESULTS_FILE="diagnostic_results_$(date +%Y%m%d_%H%M%S).log"
-
-# Function to print status
-print_status() {
- if [ $1 -eq 0 ]; then
- echo -e "${GREEN}✅ PASS${NC}: $2"
- else
- echo -e "${RED}❌ FAIL${NC}: $2"
- fi
-}
-
-# Function to test endpoint
-test_endpoint() {
- local endpoint=$1
- local description=$2
-
- echo -e "\n${BLUE}Testing:${NC} $description"
- echo "Endpoint: $endpoint"
-
- response=$(curl -s -w "\n%{http_code}" --connect-timeout 10 "$endpoint" 2>&1)
- http_code=$(echo "$response" | tail -n1)
- body=$(echo "$response" | sed '$d')
-
- echo "HTTP Status: $http_code"
-
- if [ "$http_code" = "200" ]; then
- print_status 0 "$description"
- echo "Response preview:"
- echo "$body" | head -n 5
- return 0
- else
- print_status 1 "$description (HTTP $http_code)"
- echo "Error details:"
- echo "$body" | head -n 3
- return 1
- fi
-}
-
-# Start logging
-exec > >(tee -a "$RESULTS_FILE")
-exec 2>&1
-
-echo "Starting diagnostic at $(date)"
-echo "Results will be saved to: $RESULTS_FILE"
-echo ""
-
-# Test 1: System Requirements
-echo "════════════════════════════════════════════════════════"
-echo "TEST 1: System Requirements"
-echo "════════════════════════════════════════════════════════"
-
-node --version > /dev/null 2>&1
-print_status $? "Node.js installed"
-
-npm --version > /dev/null 2>&1
-print_status $? "npm installed"
-
-curl --version > /dev/null 2>&1
-print_status $? "curl installed"
-
-# Test 2: Project Structure
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 2: Project Structure"
-echo "════════════════════════════════════════════════════════"
-
-[ -f "package.json" ]
-print_status $? "package.json exists"
-
-[ -f ".env.example" ]
-print_status $? ".env.example exists"
-
-[ -d "hf-data-engine" ]
-print_status $? "hf-data-engine directory exists"
-
-[ -f "hf-data-engine/main.py" ]
-print_status $? "HuggingFace engine implementation exists"
-
-# Test 3: Environment Configuration
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 3: Environment Configuration"
-echo "════════════════════════════════════════════════════════"
-
-if [ -f ".env" ]; then
- print_status 0 ".env file exists"
-
- grep -q "PRIMARY_DATA_SOURCE" .env
- print_status $? "PRIMARY_DATA_SOURCE configured"
-
- grep -q "HF_SPACE_BASE_URL" .env
- print_status $? "HF_SPACE_BASE_URL configured"
-
- echo ""
- echo "Current configuration:"
- grep "PRIMARY_DATA_SOURCE\|HF_SPACE" .env | sed 's/=.*/=***/' || true
-else
- print_status 1 ".env file exists"
- echo "⚠️ Run: cp .env.example .env"
-fi
-
-# Test 4: HuggingFace Space Connectivity
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 4: HuggingFace Space Connectivity"
-echo "════════════════════════════════════════════════════════"
-
-# Test DNS resolution
-echo "Resolving DNS..."
-host really-amin-datasourceforcryptocurrency.hf.space > /dev/null 2>&1
-print_status $? "DNS resolution for HF Space"
-
-# Test basic connectivity
-echo ""
-echo "Testing basic connectivity..."
-ping -c 1 -W 5 hf.space > /dev/null 2>&1
-print_status $? "Network connectivity to hf.space"
-
-# Test 5: HuggingFace Space Endpoints
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 5: HuggingFace Space Endpoints"
-echo "════════════════════════════════════════════════════════"
-
-test_endpoint "$HF_SPACE_URL/api/health" "Health check endpoint"
-test_endpoint "$HF_SPACE_URL/api/prices?symbols=BTC,ETH" "Prices endpoint"
-test_endpoint "$HF_SPACE_URL/api/ohlcv?symbol=BTCUSDT&interval=1h&limit=10" "OHLCV endpoint"
-test_endpoint "$HF_SPACE_URL/api/market/overview" "Market overview endpoint"
-test_endpoint "$HF_SPACE_URL/api/sentiment" "Sentiment endpoint"
-
-# Test 6: CORS Headers
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 6: CORS Configuration"
-echo "════════════════════════════════════════════════════════"
-
-cors_headers=$(curl -s -I -H "Origin: http://localhost:5173" "$HF_SPACE_URL/api/prices" 2>&1 | grep -i "access-control")
-
-if [ -z "$cors_headers" ]; then
- print_status 1 "CORS headers present"
- echo "⚠️ No CORS headers found. This may cause browser errors."
- echo " Consider using Vite proxy (see Configuration Guide)."
-else
- print_status 0 "CORS headers present"
- echo "CORS headers:"
- echo "$cors_headers"
-fi
-
-# Test 7: Response Format Validation
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 7: Response Format Validation"
-echo "════════════════════════════════════════════════════════"
-
-echo "Fetching sample data..."
-sample_response=$(curl -s "$HF_SPACE_URL/api/prices?symbols=BTC" 2>&1)
-
-if command -v jq > /dev/null 2>&1; then
- echo "$sample_response" | jq . > /dev/null 2>&1
- if [ $? -eq 0 ]; then
- print_status 0 "Valid JSON response"
- echo ""
- echo "Response structure:"
- echo "$sample_response" | jq 'keys' 2>/dev/null || echo "Unable to parse keys"
- else
- print_status 1 "Valid JSON response"
- echo "Response is not valid JSON:"
- echo "$sample_response" | head -n 3
- fi
-else
- echo "⚠️ jq not installed, skipping JSON validation"
- echo "Install with: sudo apt-get install jq (Ubuntu) or brew install jq (Mac)"
-fi
-
-# Test 8: Dependencies
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 8: Node Dependencies"
-echo "════════════════════════════════════════════════════════"
-
-if [ -d "node_modules" ]; then
- print_status 0 "node_modules exists"
-
- [ -d "node_modules/typescript" ]
- print_status $? "TypeScript installed"
-
- [ -d "node_modules/vite" ]
- print_status $? "Vite installed"
-
- [ -d "node_modules/react" ]
- print_status $? "React installed"
-else
- print_status 1 "node_modules exists"
- echo "⚠️ Run: npm install"
-fi
-
-# Test 9: Python Dependencies (if backend is present)
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "TEST 9: Python Dependencies"
-echo "════════════════════════════════════════════════════════"
-
-if [ -f "hf-data-engine/requirements.txt" ]; then
- print_status 0 "requirements.txt exists"
-
- python3 -c "import fastapi" 2>/dev/null
- print_status $? "FastAPI installed"
-
- python3 -c "import aiohttp" 2>/dev/null
- print_status $? "aiohttp installed"
-else
- print_status 1 "requirements.txt exists"
-fi
-
-# Summary
-echo ""
-echo "════════════════════════════════════════════════════════"
-echo "DIAGNOSTIC SUMMARY"
-echo "════════════════════════════════════════════════════════"
-
-echo ""
-echo "Results saved to: $RESULTS_FILE"
-echo ""
-echo "Next steps:"
-echo "1. Review any failed tests above"
-echo "2. Check the 'Common Issues & Fixes' section in HUGGINGFACE_DIAGNOSTIC_GUIDE.md"
-echo "3. Run 'npm run dev' and test in browser"
-echo ""
-echo "Diagnostic completed at $(date)"
-```
-
-Make it executable and run:
-
-```bash
-chmod +x diagnostic.sh
-./diagnostic.sh
-```
-
----
-
-## 🔧 Common Issues & Fixes
-
-### Issue 1: HuggingFace Space is Sleeping/Down
-
-**Symptoms:**
-- `curl: (28) Connection timed out`
-- `503 Service Unavailable`
-- `Connection refused`
-- Space shows "Building" or "Sleeping" on HuggingFace.co
-
-**Root Cause:**
-HuggingFace Spaces with free resources go to sleep after 48 hours of inactivity. They need to be "woken up" with a request.
-
-**Diagnosis:**
-
-```bash
-# Check Space status via HuggingFace website
-# Visit: https://huggingface.co/spaces/Really-amin/Datasourceforcryptocurrency
-
-# Or test via API
-curl -v https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Expected responses:
-# 200 = Space is awake ✅
-# 503 = Space is starting (wait 60 seconds)
-# Timeout = Space is sleeping
-```
-
-**Fix Option 1: Wake Up the Space**
-
-```bash
-# Send a request to wake it up
-curl https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Wait 30-60 seconds for Space to start
-echo "Waiting for Space to start..."
-sleep 60
-
-# Try again
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/health | jq
-
-# You should see: {"status": "healthy"}
-```
-
-**Fix Option 2: Use Fallback Source**
-
-```bash
-# Edit .env
-nano .env
-
-# Add these settings:
-PRIMARY_DATA_SOURCE=coingecko
-FALLBACK_ENABLED=true
-FALLBACK_SOURCES=coincap,binance
-
-# Restart application
-npm run dev
-```
-
-**Fix Option 3: Keep Space Awake (Linux/Mac)**
-
-Create a persistent ping job:
-
-```bash
-# Edit crontab
-crontab -e
-
-# Add this line (runs every 10 minutes):
-*/10 * * * * curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/health > /dev/null
-
-# Verify cron was added
-crontab -l
-```
-
-**Fix Option 4: Upgrade HuggingFace Space (Recommended)**
-
-```
-Contact HuggingFace to upgrade to paid resources for 24/7 uptime.
-Visit: https://huggingface.co/spaces/Really-amin/Datasourceforcryptocurrency/settings
-```
-
----
-
-### Issue 2: Wrong API Endpoints (404 Errors)
-
-**Symptoms:**
-- `404 Not Found`
-- `Cannot GET /api/crypto/prices/top`
-- Empty response or HTML error page
-- Console shows: `404: Not Found`
-
-**Root Cause:**
-The actual API endpoints don't match what's configured in your application.
-
-**Diagnosis:**
-
-```bash
-# Discover actual endpoints by checking API docs
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/docs | grep -oP 'href="[^"]*"' | head -20
-
-# Or try different endpoint patterns manually
-echo "Pattern 1: /api/prices"
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC
-
-echo ""
-echo "Pattern 2: /prices"
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/prices?symbols=BTC
-
-echo ""
-echo "Pattern 3: /v1/prices"
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/v1/prices?symbols=BTC
-
-echo ""
-echo "Pattern 4: Root endpoint"
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/ | head -n 20
-
-# Check actual response format
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/health | jq
-```
-
-**Fix: Update Adapter Configuration**
-
-First, locate your adapter file:
-
-```bash
-find . -name "*huggingface*adapter*" -o -name "*hf*adapter*"
-```
-
-Then update the endpoint configuration:
-
-**Option A: If using configuration object**
-
-```typescript
-// src/config/huggingface.ts or similar
-export const huggingfaceConfig = {
- baseUrl: 'https://really-amin-datasourceforcryptocurrency.hf.space',
- endpoints: {
- prices: '/api/prices', // Verify this path exists
- ohlcv: '/api/ohlcv',
- sentiment: '/api/sentiment',
- market: '/api/market/overview',
- health: '/api/health'
- },
- timeout: 30000,
-};
-```
-
-**Option B: If endpoints need transformation**
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-private getEndpointPath(endpoint: string): string {
- // Map application endpoints to actual Space endpoints
- const endpointMap: Record = {
- '/prices': '/api/prices',
- '/ohlcv': '/api/ohlcv',
- '/sentiment': '/api/sentiment',
- '/market-overview': '/api/market/overview',
- };
-
- return endpointMap[endpoint] || endpoint;
-}
-
-async fetchData(endpoint: string): Promise {
- const actualEndpoint = this.getEndpointPath(endpoint);
- const url = `${this.baseUrl}${actualEndpoint}`;
-
- console.log(`Fetching from: ${url}`);
-
- const response = await fetch(url, {
- method: 'GET',
- headers: this.getHeaders(),
- });
-
- if (!response.ok) {
- throw new Error(`HTTP ${response.status}: ${response.statusText}`);
- }
-
- return response.json();
-}
-```
-
-**Option C: Add debugging**
-
-```typescript
-// Temporary debugging to find correct endpoints
-async discoverEndpoints(): Promise {
- const patterns = [
- '/api/prices',
- '/api/price',
- '/prices',
- '/v1/prices',
- '/price',
- ];
-
- for (const pattern of patterns) {
- try {
- const response = await fetch(`${this.baseUrl}${pattern}?symbols=BTC`, {
- timeout: 5000
- });
- console.log(`${pattern}: HTTP ${response.status}`);
- } catch (error) {
- console.log(`${pattern}: Error -`, error);
- }
- }
-}
-
-// Call this during development
-// await adapter.discoverEndpoints();
-```
-
----
-
-### Issue 3: Response Format Mismatch
-
-**Symptoms:**
-- Data shows as `undefined` in UI
-- Console errors: `Cannot read property 'symbol' of undefined`
-- TypeScript type errors
-- Numbers showing as strings
-
-**Root Cause:**
-The Space returns data in a different format than expected.
-
-**Diagnosis:**
-
-```bash
-# Get actual response and examine structure
-curl -s "https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC,ETH" | jq '.' -C
-
-# Note the field names, types, and structure
-
-# Compare with expected format
-# Expected example:
-# [
-# {
-# "symbol": "BTC",
-# "price": 50000,
-# "change24h": 2.5
-# }
-# ]
-
-# Actual format (if different):
-# {
-# "data": [
-# {
-# "coin": "bitcoin",
-# "current_price": "50000.00",
-# "percent_change": "2.5"
-# }
-# ]
-# }
-```
-
-**Fix: Update Data Mapping**
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-interface HFPriceResponse {
- // Define actual Space response structure
- data?: Array<{
- coin?: string;
- symbol?: string;
- current_price?: number | string;
- price?: number | string;
- percent_change?: number | string;
- change_24h?: number | string;
- }>;
- prices?: any[];
-}
-
-async getPrices(symbols: string[]): Promise {
- const data = await this.fetchData('/api/prices?symbols=' + symbols.join(','));
-
- // Handle different response structures
- const prices = data.data || data.prices || [];
-
- return prices.map(item => {
- // Safely extract values with fallbacks
- const symbol = item.symbol || item.coin?.toUpperCase() || 'UNKNOWN';
- const price = Number(item.current_price || item.price || 0);
- const change24h = Number(item.percent_change || item.change_24h || 0);
-
- // Validate required fields
- if (isNaN(price)) {
- console.warn(`Invalid price for ${symbol}:`, item);
- return null;
- }
-
- return {
- symbol,
- price,
- change24h,
- timestamp: Date.now(),
- };
- }).filter(Boolean) as CryptoPrice[];
-}
-```
-
-**Add Comprehensive Validation:**
-
-```typescript
-// src/services/validators/huggingface.validator.ts
-
-export function validatePriceResponse(data: any): boolean {
- if (!Array.isArray(data) && !data?.data && !data?.prices) {
- console.error('Invalid response structure:', typeof data);
- return false;
- }
-
- const items = Array.isArray(data) ? data : (data.data || data.prices || []);
-
- if (items.length === 0) {
- console.warn('Response contains no items');
- return false;
- }
-
- // Validate first item has required fields
- const firstItem = items[0];
- if (!firstItem.symbol && !firstItem.coin) {
- console.error('Missing symbol/coin field:', firstItem);
- return false;
- }
-
- if (!firstItem.price && !firstItem.current_price) {
- console.error('Missing price field:', firstItem);
- return false;
- }
-
- return true;
-}
-
-export function normalizePriceData(data: any): CryptoPrice[] {
- if (!validatePriceResponse(data)) {
- throw new Error('Invalid price response format');
- }
-
- const items = Array.isArray(data) ? data : (data.data || data.prices);
-
- return items.map((item: any) => ({
- symbol: (item.symbol || item.coin || 'UNKNOWN').toUpperCase(),
- price: Number(item.current_price || item.price || 0),
- change24h: Number(item.percent_change || item.change_24h || 0),
- timestamp: Date.now(),
- }));
-}
-```
-
----
-
-### Issue 4: CORS Errors in Browser
-
-**Symptoms:**
-- Browser console error: `Access to fetch at '...' from origin 'http://localhost:5173' has been blocked by CORS policy`
-- Network tab shows request with red X
-- `No 'Access-Control-Allow-Origin' header is present`
-
-**Root Cause:**
-Browser blocks cross-origin requests unless the server includes proper CORS headers.
-
-**Diagnosis:**
-
-```bash
-# Check if Space returns CORS headers
-curl -I -H "Origin: http://localhost:5173" \
- https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Look for these headers in the response:
-# Access-Control-Allow-Origin: *
-# Access-Control-Allow-Methods: GET, POST, OPTIONS
-# Access-Control-Allow-Headers: Content-Type
-
-# If headers are missing, you'll see CORS errors in browser
-
-# Test with preflight OPTIONS request
-curl -X OPTIONS -I \
- -H "Origin: http://localhost:5173" \
- -H "Access-Control-Request-Method: GET" \
- https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-```
-
-**Fix Option 1: Add Vite Proxy (Recommended for Development)**
-
-```typescript
-// vite.config.ts
-
-import { defineConfig } from 'vite'
-import react from '@vitejs/plugin-react'
-
-export default defineConfig({
- plugins: [react()],
- server: {
- proxy: {
- '/api/hf': {
- target: 'https://really-amin-datasourceforcryptocurrency.hf.space',
- changeOrigin: true,
- rewrite: (path) => {
- // Remove /api/hf prefix and keep the rest
- return path.replace(/^\/api\/hf/, '');
- },
- configure: (proxy, options) => {
- proxy.on('error', (err, req, res) => {
- console.error('Proxy error:', err);
- });
- proxy.on('proxyReq', (proxyReq, req, res) => {
- console.log('Proxying:', req.method, req.url);
- });
- proxy.on('proxyRes', (proxyRes, req, res) => {
- console.log('Proxy response:', proxyRes.statusCode);
- });
- }
- }
- }
- }
-})
-```
-
-Then update your adapter:
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-async fetchData(endpoint: string): Promise {
- // In development, use Vite proxy
- // In production, use direct URL (if CORS enabled on Space)
-
- const baseUrl = import.meta.env.DEV
- ? '/api/hf' // Proxied through Vite
- : this.config.baseUrl; // Direct to Space
-
- const url = `${baseUrl}${endpoint}`;
-
- console.log(`[${import.meta.env.DEV ? 'DEV' : 'PROD'}] Fetching: ${url}`);
-
- const response = await fetch(url, {
- method: 'GET',
- headers: this.getHeaders(),
- signal: AbortSignal.timeout(this.config.timeout),
- });
-
- if (!response.ok) {
- const errorText = await response.text();
- throw new Error(`HTTP ${response.status}: ${errorText}`);
- }
-
- return response.json();
-}
-```
-
-**Fix Option 2: Update Space with CORS Headers (If you control the Space)**
-
-If you control the HuggingFace Space, add CORS support:
-
-**For FastAPI-based Space:**
-
-```python
-# hf-data-engine/main.py
-
-from fastapi import FastAPI
-from fastapi.middleware.cors import CORSMiddleware
-
-app = FastAPI(title="Crypto Data Engine")
-
-# Add CORS middleware
-app.add_middleware(
- CORSMiddleware,
- allow_origins=["*"], # Or specify: ["http://localhost:5173", "https://yourdomain.com"]
- allow_credentials=True,
- allow_methods=["GET", "POST", "OPTIONS"],
- allow_headers=["*", "Content-Type", "Authorization"],
- max_age=3600, # Cache preflight for 1 hour
-)
-
-@app.get("/api/health")
-async def health():
- return {"status": "healthy"}
-
-# ... rest of API endpoints
-```
-
-**For Gradio-based Space:**
-
-```python
-# app.py
-
-import gradio as gr
-
-# Create your interface
-demo = gr.Blocks()
-
-with demo:
- # Your components here
- pass
-
-if __name__ == "__main__":
- demo.launch(
- share=True,
- server_name="0.0.0.0",
- server_port=7860,
- # Note: Gradio automatically handles CORS for public access
- )
-```
-
-**Fix Option 3: Use CORS Proxy Service (Development Only)**
-
-⚠️ **Not recommended for production**
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-async fetchData(endpoint: string): Promise {
- let url = `${this.config.baseUrl}${endpoint}`;
-
- // Only use CORS proxy as last resort for testing
- if (import.meta.env.DEV && !import.meta.env.VITE_USE_PROXY) {
- const corsProxy = 'https://corsproxy.io/?';
- url = corsProxy + encodeURIComponent(url);
- }
-
- const response = await fetch(url);
- return response.json();
-}
-```
-
-Available CORS proxy services (for testing only):
-- https://corsproxy.io/
-- https://cors-anywhere.herokuapp.com/
-- https://api.allorigins.win/
-
----
-
-### Issue 5: Timeout Errors
-
-**Symptoms:**
-- `AbortError: The operation was aborted due to timeout`
-- Requests take > 30 seconds
-- UI shows loading spinner that never completes
-- Network tab shows request taking a long time
-
-**Root Cause:**
-Space is slow to respond or having performance issues, or timeout is too short.
-
-**Diagnosis:**
-
-```bash
-# Measure actual response time
-time curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC | jq > /dev/null
-
-# Expected: < 5 seconds
-# 5-15 seconds: Space is cold (starting up)
-# > 30 seconds: Space might be sleeping or overloaded
-
-# Check Space status
-curl -I https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Test endpoint directly multiple times
-for i in {1..3}; do
- echo "Request $i:"
- time curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC > /dev/null
- echo ""
-done
-```
-
-**Fix Option 1: Increase Timeout**
-
-```typescript
-// .env
-HF_REQUEST_TIMEOUT=60000 # 60 seconds
-
-// src/config/huggingface.ts
-export const huggingfaceConfig = {
- baseUrl: 'https://really-amin-datasourceforcryptocurrency.hf.space',
- timeout: parseInt(import.meta.env.VITE_HF_REQUEST_TIMEOUT || '60000'),
-};
-
-// src/services/adapters/huggingface.adapter.ts
-async fetchData(endpoint: string): Promise {
- const url = `${this.config.baseUrl}${endpoint}`;
-
- console.log(`[HF] Requesting ${endpoint} (timeout: ${this.config.timeout}ms)`);
-
- const startTime = Date.now();
-
- try {
- const response = await fetch(url, {
- signal: AbortSignal.timeout(this.config.timeout),
- });
-
- const duration = Date.now() - startTime;
- console.log(`[HF] Completed in ${duration}ms`);
-
- return response.json();
- } catch (error) {
- const duration = Date.now() - startTime;
- console.error(`[HF] Failed after ${duration}ms:`, error);
- throw error;
- }
-}
-```
-
-**Fix Option 2: Implement Proper Loading States**
-
-```typescript
-// src/hooks/useHuggingFaceData.ts
-
-import { useState, useEffect } from 'react';
-
-export function useHuggingFaceData(
- fetchFn: () => Promise,
- options?: { timeout?: number; retries?: number }
-) {
- const [data, setData] = useState(null);
- const [loading, setLoading] = useState(true);
- const [error, setError] = useState(null);
-
- useEffect(() => {
- let mounted = true;
- let retryCount = 0;
- const maxRetries = options?.retries ?? 1;
-
- async function fetchData() {
- try {
- setLoading(true);
- setError(null);
-
- const result = await fetchFn();
-
- if (mounted) {
- setData(result);
- }
- } catch (err) {
- if (mounted) {
- if (retryCount < maxRetries) {
- retryCount++;
- console.log(`Retrying... (${retryCount}/${maxRetries})`);
- setTimeout(fetchData, 2000 * retryCount); // Exponential backoff
- } else {
- setError(err instanceof Error ? err : new Error('Unknown error'));
- }
- }
- } finally {
- if (mounted) {
- setLoading(retryCount === 0 || retryCount === maxRetries);
- }
- }
- }
-
- fetchData();
-
- return () => { mounted = false; };
- }, [fetchFn, options?.retries]);
-
- return { data, loading, error };
-}
-```
-
-**Fix Option 3: Implement Caching**
-
-```typescript
-// src/services/cache/huggingface.cache.ts
-
-interface CacheEntry {
- data: T;
- timestamp: number;
- ttl: number;
-}
-
-export class HuggingFaceCache {
- private cache = new Map>();
- private defaultTTL = 5 * 60 * 1000; // 5 minutes
-
- set(key: string, data: T, ttl?: number): void {
- this.cache.set(key, {
- data,
- timestamp: Date.now(),
- ttl: ttl || this.defaultTTL,
- });
- }
-
- get(key: string): T | null {
- const entry = this.cache.get(key) as CacheEntry | undefined;
-
- if (!entry) return null;
-
- const age = Date.now() - entry.timestamp;
- if (age > entry.ttl) {
- this.cache.delete(key);
- return null;
- }
-
- return entry.data;
- }
-
- isStale(key: string): boolean {
- const entry = this.cache.get(key);
- if (!entry) return true;
-
- const age = Date.now() - entry.timestamp;
- return age > entry.ttl;
- }
-
- clear(): void {
- this.cache.clear();
- }
-}
-
-// Usage in adapter
-export class HuggingFaceAdapter {
- private cache = new HuggingFaceCache();
-
- async fetchData(endpoint: string, cacheTTL?: number): Promise {
- // Try cache first
- const cached = this.cache.get(endpoint);
- if (cached) {
- console.log(`[Cache] Hit for ${endpoint}`);
- return cached;
- }
-
- // Fetch from Space
- console.log(`[HF] Fetching ${endpoint}...`);
- const data = await this.doFetch(endpoint);
-
- // Cache result
- this.cache.set(endpoint, data, cacheTTL);
-
- return data;
- }
-
- private async doFetch(endpoint: string): Promise {
- const response = await fetch(`${this.config.baseUrl}${endpoint}`);
- return response.json();
- }
-}
-```
-
-**Fix Option 4: Use Request Pooling**
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-export class HuggingFaceAdapter {
- private requestPool = new Map>();
-
- async fetchData(endpoint: string): Promise {
- // If same request is in-flight, return that promise instead of creating new request
- if (this.requestPool.has(endpoint)) {
- console.log(`[Pool] Reusing in-flight request for ${endpoint}`);
- return this.requestPool.get(endpoint)!;
- }
-
- // Create new request
- const promise = this.doFetch(endpoint)
- .finally(() => {
- this.requestPool.delete(endpoint);
- });
-
- this.requestPool.set(endpoint, promise);
- return promise;
- }
-
- private async doFetch(endpoint: string): Promise {
- const url = `${this.config.baseUrl}${endpoint}`;
- const response = await fetch(url);
- return response.json();
- }
-}
-```
-
----
-
-### Issue 6: Authentication Required (401/403)
-
-**Symptoms:**
-- `401 Unauthorized`
-- `403 Forbidden`
-- Response: `{"error": "Authentication required"}`
-- Error: `Invalid token` or `Expired credentials`
-
-**Root Cause:**
-Space requires authentication (API token or credentials) that isn't provided.
-
-**Diagnosis:**
-
-```bash
-# Test without authentication
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq
-
-# Test with different auth methods
-
-# Method 1: Bearer token
-curl -H "Authorization: Bearer YOUR_TOKEN_HERE" \
- https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Method 2: API key in header
-curl -H "X-API-Key: YOUR_KEY_HERE" \
- https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Method 3: API key in query
-curl "https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?api_key=YOUR_KEY_HERE"
-
-# Check response status and error details
-curl -i https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-```
-
-**Fix Option 1: Add Authentication to Configuration**
-
-```bash
-# .env
-VITE_HF_API_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-VITE_HF_API_KEY=your-api-key-here
-```
-
-```typescript
-// src/config/huggingface.ts
-export const huggingfaceConfig = {
- baseUrl: 'https://really-amin-datasourceforcryptocurrency.hf.space',
- apiToken: import.meta.env.VITE_HF_API_TOKEN,
- apiKey: import.meta.env.VITE_HF_API_KEY,
-};
-
-// src/types/config.ts
-export interface HuggingFaceConfig {
- baseUrl: string;
- timeout: number;
- apiToken?: string; // For Bearer token auth
- apiKey?: string; // For X-API-Key header
-}
-```
-
-**Fix Option 2: Update Adapter to Include Auth Headers**
-
-```typescript
-// src/services/adapters/huggingface.adapter.ts
-
-private getHeaders(): Record {
- const headers: Record = {
- 'Content-Type': 'application/json',
- 'Accept': 'application/json',
- };
-
- // Add authentication if configured
- if (this.config.apiToken) {
- headers['Authorization'] = `Bearer ${this.config.apiToken}`;
- }
-
- if (this.config.apiKey) {
- headers['X-API-Key'] = this.config.apiKey;
- }
-
- return headers;
-}
-
-async fetchData(endpoint: string): Promise {
- const url = `${this.config.baseUrl}${endpoint}`;
-
- try {
- const response = await fetch(url, {
- method: 'GET',
- headers: this.getHeaders(),
- signal: AbortSignal.timeout(this.config.timeout),
- });
-
- if (response.status === 401 || response.status === 403) {
- throw new Error('Authentication failed. Check your API token/key.');
- }
-
- if (!response.ok) {
- const error = await response.text();
- throw new Error(`HTTP ${response.status}: ${error}`);
- }
-
- return response.json();
- } catch (error) {
- console.error('[HF Auth Error]', error);
- throw error;
- }
-}
-```
-
-**Fix Option 3: Get HuggingFace Token**
-
-If Space requires HuggingFace credentials:
-
-1. Visit: https://huggingface.co/settings/tokens
-2. Click "New token"
-3. Create token with "Read" access
-4. Copy token to `.env`:
- ```env
- VITE_HF_API_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- ```
-
----
-
-## 🧪 Testing Protocol
-
-### Test Sequence
-
-Follow these tests in order. **Stop at the first failure** and fix before continuing.
-
-#### Test 1: Space Health Check
-
-```bash
-echo "🔍 Test 1: Space Health Check"
-curl -v https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# ✅ Expected:
-# HTTP/2 200 (or HTTP/1.1 200)
-# Content-Type: application/json
-# {"status": "healthy"}
-
-# ❌ If fails:
-# - HTTP 503: Space is building (wait 60 seconds)
-# - HTTP 000 / Timeout: Space is sleeping (send request to wake it)
-# - HTTP 404: Wrong endpoint (check endpoint mapping)
-```
-
-#### Test 2: Prices Endpoint
-
-```bash
-echo "🔍 Test 2: Prices Endpoint"
-curl -s "https://really-amin-datasourceforcryptocurrency.hf.space/api/prices?symbols=BTC,ETH" | jq '.'
-
-# ✅ Expected: Returns array or object with price data
-
-# ❌ If fails:
-# - Empty response: Try adding limit parameter
-# - null: Endpoint exists but no data
-# - 404: Wrong endpoint path
-```
-
-#### Test 3: OHLCV Endpoint
-
-```bash
-echo "🔍 Test 3: OHLCV Endpoint"
-curl -s "https://really-amin-datasourceforcryptocurrency.hf.space/api/ohlcv?symbol=BTCUSDT&interval=1h&limit=10" | jq '.[:1]'
-
-# ✅ Expected: OHLCV data with candle information
-
-# ❌ If fails:
-# - 404: Try different endpoint patterns
-# - Wrong symbol format: Check symbol requirements (BTCUSDT vs BTC)
-```
-
-#### Test 4: Local Development (Vite Proxy)
-
-```bash
-echo "🔍 Test 4: Local Development"
-
-# Make sure .env is configured
-if [ ! -f .env ]; then
- cp .env.example .env
-fi
-
-# Install dependencies
-npm install
-
-# Start dev server
-npm run dev &
-DEV_PID=$!
-
-# Wait for server to start
-sleep 5
-
-# Test via proxy
-echo "Testing via proxy (http://localhost:5173/api/hf/...)"
-curl -s "http://localhost:5173/api/hf/api/health" | jq
-
-# Stop dev server
-kill $DEV_PID
-
-# ✅ Expected: Same response as direct Space call
-
-# ❌ If fails:
-# - Connection refused: Dev server didn't start
-# - 404: Proxy path incorrect
-# - CORS error: Check vite.config.ts
-```
-
-#### Test 5: Browser Testing
-
-```bash
-echo "🔍 Test 5: Browser Testing"
-
-# 1. Start dev server
-npm run dev
-
-# 2. Open browser: http://localhost:5173
-
-# 3. Open DevTools (F12)
-
-# 4. Go to Network tab
-
-# 5. Trigger data fetch (click buttons, load page, etc.)
-
-# 6. Look for requests to /api/hf/...
-
-# 7. Check response status
-# ✅ 200 = Success
-# ❌ 404 = Wrong endpoint
-# ❌ 0 (blocked) = CORS issue
-
-# 8. Go to Console tab
-
-# 9. Look for errors:
-# ❌ "Access to fetch blocked by CORS" → Use Vite proxy
-# ❌ "Cannot read property 'symbol' of undefined" → Data mapping issue
-# ❌ "Timeout" → Increase timeout in config
-```
-
-### Complete Test Checklist
-
-- [ ] Health check returns 200
-- [ ] Prices endpoint returns data
-- [ ] OHLCV endpoint returns data
-- [ ] Vite proxy works locally
-- [ ] No CORS errors in browser console
-- [ ] Data renders correctly in UI
-- [ ] No undefined values in UI
-- [ ] Network requests complete < 30 seconds
-- [ ] Application handles errors gracefully
-
----
-
-## 🐛 Debugging Commands
-
-### Debugging HuggingFace Integration
-
-```bash
-# Enable verbose logging
-export DEBUG=*:huggingface*,*:adapter*
-
-# Watch logs in real-time
-npm run dev 2>&1 | grep -i "huggingface\|hf\|adapter"
-
-# Log all fetch requests
-cat > src/services/debug.ts << 'EOF'
-// Intercept all fetch calls
-const originalFetch = window.fetch;
-window.fetch = function(...args) {
- const [resource] = args;
- console.log(`📡 Fetch: ${resource}`);
-
- return originalFetch.apply(this, args as any)
- .then(response => {
- console.log(`📡 Response: ${resource} → ${response.status}`);
- return response.clone();
- })
- .catch(error => {
- console.error(`📡 Error: ${resource} →`, error);
- throw error;
- });
-};
-EOF
-
-# In your main component or app.tsx:
-// Add this early in your app initialization
-import './services/debug';
-```
-
-### Network Debugging
-
-```bash
-# Monitor network activity
-curl -v https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Show request headers only
-curl -I https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Show response headers
-curl -D - https://really-amin-datasourceforcryptocurrency.hf.space/api/health
-
-# Test with custom headers
-curl -H "Authorization: Bearer token" \
- -H "X-Custom-Header: value" \
- https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Save full request/response to file
-curl -v https://really-amin-datasourceforcryptocurrency.hf.space/api/health 2>&1 | tee debug.log
-```
-
-### Response Inspection
-
-```bash
-# Pretty print JSON response
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq '.'
-
-# Show specific fields
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq '.[0] | keys'
-
-# Count items
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq 'length'
-
-# Filter by condition
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq '.[] | select(.symbol == "BTC")'
-
-# Convert to CSV
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices | jq -r '.[] | [.symbol, .price] | @csv'
-```
-
-### TypeScript/React Debugging
-
-```typescript
-// Add detailed logging to adapter
-class HuggingFaceAdapter {
- async fetchData(endpoint: string): Promise {
- const url = `${this.baseUrl}${endpoint}`;
-
- console.group(`🔵 HF Fetch: ${endpoint}`);
- console.log('URL:', url);
- console.log('Headers:', this.getHeaders());
- console.log('Timeout:', this.config.timeout);
- console.timeStamp('start');
-
- try {
- const response = await fetch(url, {
- headers: this.getHeaders(),
- });
-
- const elapsed = performance.now() - performance.timing.navigationStart;
- console.log('Response status:', response.status);
- console.log('Time elapsed:', `${elapsed}ms`);
-
- const data = await response.json();
- console.log('Response data:', data);
- console.groupEnd();
-
- return data;
- } catch (error) {
- console.error('Error:', error);
- console.groupEnd();
- throw error;
- }
- }
-}
-```
-
-### Performance Profiling
-
-```bash
-# Measure response time
-time curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices > /dev/null
-
-# Detailed timing breakdown
-curl -w "
-Time breakdown:
- DNS lookup: %{time_namelookup}s
- TCP connect: %{time_connect}s
- TLS handshake: %{time_appconnect}s
- Server processing: %{time_starttransfer}s
- Total: %{time_total}s
-" -o /dev/null -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-
-# Repeat tests and get average
-for i in {1..5}; do
- echo "Request $i:"
- curl -w "Time: %{time_total}s\n" -o /dev/null -s https://really-amin-datasourceforcryptocurrency.hf.space/api/prices
-done
-```
-
----
-
-## ⚙️ Configuration Guide
-
-### Environment Variables
-
-Create `.env` file based on `.env.example`:
-
-```bash
-# Copy template
-cp .env.example .env
-```
-
-### Available Configuration Options
-
-```env
-# Data Source Configuration
-PRIMARY_DATA_SOURCE=huggingface # Main data source: huggingface, coingecko, binance
-FALLBACK_ENABLED=true # Enable fallback sources
-FALLBACK_SOURCES=coingecko,coincap # Comma-separated fallback sources
-
-# HuggingFace Space Configuration
-HF_SPACE_BASE_URL=https://really-amin-datasourceforcryptocurrency.hf.space
-HF_REQUEST_TIMEOUT=30000 # Request timeout in milliseconds
-HF_CACHE_TTL=300000 # Cache time-to-live in milliseconds (5 minutes)
-HF_API_TOKEN= # HuggingFace API token (if required)
-
-# Development Configuration
-VITE_DEV_SERVER_HOST=localhost
-VITE_DEV_SERVER_PORT=5173
-VITE_LOG_LEVEL=info # debug, info, warn, error
-
-# Proxy Configuration (for development)
-VITE_USE_PROXY=true # Use Vite proxy for API calls
-VITE_PROXY_PATH=/api/hf # Proxy mount path
-```
-
-### Vite Configuration
-
-File: `vite.config.ts`
-
-```typescript
-import { defineConfig } from 'vite'
-import react from '@vitejs/plugin-react'
-
-export default defineConfig({
- plugins: [react()],
-
- server: {
- host: 'localhost',
- port: 5173,
-
- proxy: {
- '/api/hf': {
- target: 'https://really-amin-datasourceforcryptocurrency.hf.space',
- changeOrigin: true,
- rewrite: (path) => path.replace(/^\/api\/hf/, ''),
- configure: (proxy, options) => {
- proxy.on('error', (err, req, res) => {
- console.error('Proxy error:', err);
- });
- proxy.on('proxyReq', (proxyReq, req, res) => {
- console.log('→ Proxying:', req.method, req.url);
- });
- proxy.on('proxyRes', (proxyRes, req, res) => {
- console.log('← Response:', proxyRes.statusCode);
- });
- }
- }
- }
- },
-
- build: {
- outDir: 'dist',
- sourcemap: true,
- }
-})
-```
-
-### TypeScript Configuration
-
-File: `tsconfig.json`
-
-```json
-{
- "compilerOptions": {
- "target": "ES2020",
- "useDefineForClassFields": true,
- "lib": ["ES2020", "DOM", "DOM.Iterable"],
- "module": "ESNext",
- "skipLibCheck": true,
- "esModuleInterop": true,
- "strict": true,
- "resolveJsonModule": true,
- "declaration": true,
- "declarationMap": true,
- "sourceMap": true,
- "allowJs": false,
- "baseUrl": ".",
- "paths": {
- "@/*": ["src/*"],
- "@/services/*": ["src/services/*"],
- "@/components/*": ["src/components/*"],
- "@/types/*": ["src/types/*"]
- }
- }
-}
-```
-
----
-
-## 🌳 Troubleshooting Decision Tree
-
-Start here when you encounter issues:
-
-```
-┌─ START: Application not working
-│
-├─ Step 1: Can you reach the Space?
-│ │
-│ ├─ NO (timeout, connection refused)
-│ │ └─ Issue 1: Space is sleeping → Wake it up
-│ │
-│ └─ YES (200 OK)
-│ │
-│ └─ Step 2: Are you getting the correct endpoints?
-│ │
-│ ├─ NO (404 Not Found)
-│ │ └─ Issue 2: Wrong endpoints → Update adapter
-│ │
-│ └─ YES (200 OK)
-│ │
-│ └─ Step 3: Is the data in the correct format?
-│ │
-│ ├─ NO (undefined values, type errors)
-│ │ └─ Issue 3: Response format mismatch → Update mapping
-│ │
-│ └─ YES (correct data types)
-│ │
-│ └─ Step 4: Does the browser show CORS errors?
-│ │
-│ ├─ YES (Access blocked by CORS)
-│ │ └─ Issue 4: CORS errors → Add Vite proxy
-│ │
-│ └─ NO (no CORS errors)
-│ │
-│ └─ Step 5: Are requests timing out?
-│ │
-│ ├─ YES (AbortError timeout)
-│ │ └─ Issue 5: Timeout → Increase timeout or use caching
-│ │
-│ └─ NO (requests complete)
-│ │
-│ └─ Step 6: Check authentication
-│ │
-│ ├─ 401/403 errors
-│ │ └─ Issue 6: Auth required → Add token/key
-│ │
-│ └─ ✅ WORKING!
-```
-
-**Quick Reference:**
-- Space not responding → Check Space status, wait 60 seconds
-- Getting 404 → Update endpoint paths in adapter
-- Data undefined → Update field name mappings
-- CORS errors → Enable Vite proxy
-- Timeouts → Increase timeout or implement caching
-- 401/403 → Add API token/key to config
-
----
-
-## ❓ FAQ
-
-### Q: How do I know which version of the Space is deployed?
-
-```bash
-# Check Space's version endpoint (if available)
-curl -s https://really-amin-datasourceforcryptocurrency.hf.space/api/version
-
-# Or check the Space's README on HuggingFace
-# Visit: https://huggingface.co/spaces/Really-amin/Datasourceforcryptocurrency
-
-# Or check git log if you have access
-cd hf-data-engine
-git log --oneline | head -5
-```
-
-### Q: Can I use this application without HuggingFace?
-
-Yes! Configure fallback data sources:
-
-```env
-PRIMARY_DATA_SOURCE=coingecko
-FALLBACK_ENABLED=true
-FALLBACK_SOURCES=coincap,binance
-```
-
-### Q: What if HuggingFace Space goes down permanently?
-
-1. Deploy your own instance of `hf-data-engine`
-2. Update `HF_SPACE_BASE_URL` in `.env`
-3. Or switch to fallback sources permanently
-
-### Q: How do I cache data for offline use?
-
-```typescript
-// src/services/storage/localStorage.cache.ts
-
-export class LocalStorageCache {
- static set(key: string, data: T): void {
- localStorage.setItem(key, JSON.stringify({
- data,
- timestamp: Date.now(),
- }));
- }
-
- static get(key: string, maxAge?: number): T | null {
- const stored = localStorage.getItem(key);
- if (!stored) return null;
-
- const { data, timestamp } = JSON.parse(stored);
-
- if (maxAge && Date.now() - timestamp > maxAge) {
- localStorage.removeItem(key);
- return null;
- }
-
- return data;
- }
-}
-```
-
-### Q: How do I monitor HuggingFace Space uptime?
-
-Use a monitoring service or cron job:
-
-```bash
-# Create uptime.sh
-#!/bin/bash
-TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
-STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://really-amin-datasourceforcryptocurrency.hf.space/api/health)
-echo "$TIMESTAMP,HuggingFace Space,$STATUS" >> uptime.log
-
-# Add to crontab
-*/5 * * * * /path/to/uptime.sh
-```
-
-### Q: Can I contribute improvements to the HuggingFace Space?
-
-Yes! The space is open source:
-
-1. Fork the repository
-2. Make improvements
-3. Submit a pull request
-4. Visit: https://huggingface.co/spaces/Really-amin/Datasourceforcryptocurrency
-
-### Q: What are the rate limits?
-
-From the Space documentation:
-- `/api/prices`: 120 requests/minute
-- `/api/ohlcv`: 60 requests/minute
-- `/api/sentiment`: 30 requests/minute
-- `/api/health`: Unlimited
-
-Implement rate limiting in your client:
-
-```typescript
-// src/services/rateLimit.ts
-
-export class RateLimiter {
- private timestamps: number[] = [];
-
- constructor(private maxRequests: number, private windowMs: number) {}
-
- canRequest(): boolean {
- const now = Date.now();
-
- // Remove old timestamps outside window
- this.timestamps = this.timestamps.filter(ts => now - ts < this.windowMs);
-
- // Check if under limit
- if (this.timestamps.length < this.maxRequests) {
- this.timestamps.push(now);
- return true;
- }
-
- return false;
- }
-}
-
-// Usage
-const limiter = new RateLimiter(100, 60000); // 100 req/min
-
-if (limiter.canRequest()) {
- // Make request
-} else {
- // Wait or queue request
-}
-```
-
-### Q: How do I debug issues in production?
-
-1. Check browser console for errors
-2. Check Network tab for failed requests
-3. Review server logs
-4. Use error tracking service (Sentry, LogRocket, etc.)
-
-```typescript
-// Error tracking integration
-import * as Sentry from "@sentry/react";
-
-Sentry.init({
- dsn: "your-sentry-dsn",
- environment: import.meta.env.MODE,
- tracesSampleRate: 0.1,
-});
-
-try {
- // Your code
-} catch (error) {
- Sentry.captureException(error);
-}
-```
-
----
-
-## 📞 Support
-
-- **HuggingFace Space:** https://huggingface.co/spaces/Really-amin/Datasourceforcryptocurrency
-- **GitHub Issues:** Report bugs and request features
-- **Documentation:** See README.md and other docs
-
----
-
-**Last Updated:** 2025-11-15
-**Version:** 2.0
-**Maintained by:** Crypto Data Aggregator Team
diff --git a/app/IMPLEMENTATION_FIXES.md b/app/IMPLEMENTATION_FIXES.md
deleted file mode 100644
index bbe6f35cc36cfd5253e1a7bdad907fe083eed6a0..0000000000000000000000000000000000000000
--- a/app/IMPLEMENTATION_FIXES.md
+++ /dev/null
@@ -1,686 +0,0 @@
-# Implementation Fixes Documentation
-**Comprehensive Solutions for Identified Issues**
-
-## Overview
-
-This document details all the improvements implemented to address the critical issues identified in the project analysis. Each fix is production-ready and follows industry best practices.
-
----
-
-## 1. Modular Architecture Refactoring
-
-### Problem
-- `app.py` was 1,495 lines - exceeds recommended 500-line limit
-- Multiple concerns mixed in single file
-- Difficult to test and maintain
-
-### Solution Implemented
-Created modular UI architecture:
-
-```
-ui/
-├── __init__.py # Module exports
-├── dashboard_live.py # Tab 1: Live prices
-├── dashboard_charts.py # Tab 2: Historical charts
-├── dashboard_news.py # Tab 3: News & sentiment
-├── dashboard_ai.py # Tab 4: AI analysis
-├── dashboard_db.py # Tab 5: Database explorer
-├── dashboard_status.py # Tab 6: Data sources status
-└── interface.py # Gradio UI builder
-```
-
-### Benefits
-- ✅ Each module < 300 lines
-- ✅ Single responsibility per file
-- ✅ Easy to test independently
-- ✅ Better code organization
-
-### Usage
-```python
-# Old way (monolithic)
-import app
-
-# New way (modular)
-from ui import create_gradio_interface, get_live_dashboard
-
-dashboard_data = get_live_dashboard()
-interface = create_gradio_interface()
-```
-
----
-
-## 2. Unified Async API Client
-
-### Problem
-- Mixed async (aiohttp) and sync (requests) code
-- Duplicated retry logic across collectors
-- Inconsistent error handling
-
-### Solution Implemented
-Created `utils/async_api_client.py`:
-
-```python
-from utils.async_api_client import AsyncAPIClient, safe_api_call
-
-# Single API call
-async def fetch_data():
- async with AsyncAPIClient() as client:
- data = await client.get("https://api.example.com/data")
- return data
-
-# Parallel API calls
-from utils.async_api_client import parallel_api_calls
-
-urls = ["https://api1.com/data", "https://api2.com/data"]
-results = await parallel_api_calls(urls)
-```
-
-### Features
-- ✅ Automatic retry with exponential backoff
-- ✅ Comprehensive error handling
-- ✅ Timeout management
-- ✅ Parallel request support
-- ✅ Consistent logging
-
-### Migration Guide
-```python
-# Before (sync with requests)
-import requests
-
-def get_prices():
- try:
- response = requests.get(url, timeout=10)
- response.raise_for_status()
- return response.json()
- except Exception as e:
- logger.error(f"Error: {e}")
- return None
-
-# After (async with AsyncAPIClient)
-from utils.async_api_client import safe_api_call
-
-async def get_prices():
- return await safe_api_call(url)
-```
-
----
-
-## 3. Authentication & Authorization System
-
-### Problem
-- No authentication for production deployments
-- Dashboard accessible to anyone
-- No API key management
-
-### Solution Implemented
-Created `utils/auth.py`:
-
-#### Features
-- ✅ JWT token authentication
-- ✅ API key management
-- ✅ Password hashing (SHA-256)
-- ✅ Token expiration
-- ✅ Usage tracking
-
-#### Configuration
-```bash
-# .env file
-ENABLE_AUTH=true
-SECRET_KEY=your-secret-key-here
-ADMIN_USERNAME=admin
-ADMIN_PASSWORD=secure-password
-ACCESS_TOKEN_EXPIRE_MINUTES=60
-API_KEYS=key1,key2,key3
-```
-
-#### Usage
-```python
-from utils.auth import authenticate_user, auth_manager
-
-# Authenticate user
-token = authenticate_user("admin", "password")
-
-# Create API key
-api_key = auth_manager.create_api_key("mobile_app")
-
-# Verify API key
-is_valid = auth_manager.verify_api_key(api_key)
-
-# Revoke API key
-auth_manager.revoke_api_key(api_key)
-```
-
-#### Integration with FastAPI
-```python
-from fastapi import Header, HTTPException
-from utils.auth import verify_request_auth
-
-@app.get("/api/protected")
-async def protected_endpoint(
- authorization: Optional[str] = Header(None),
- api_key: Optional[str] = Header(None, alias="X-API-Key")
-):
- if not verify_request_auth(authorization, api_key):
- raise HTTPException(status_code=401, detail="Unauthorized")
-
- return {"message": "Access granted"}
-```
-
----
-
-## 4. Enhanced Rate Limiting System
-
-### Problem
-- No rate limiting on API endpoints
-- Risk of abuse and resource exhaustion
-- No burst protection
-
-### Solution Implemented
-Created `utils/rate_limiter_enhanced.py`:
-
-#### Algorithms
-1. **Token Bucket** - Burst traffic handling
-2. **Sliding Window** - Accurate rate limiting
-
-#### Features
-- ✅ Per-minute limits (default: 30/min)
-- ✅ Per-hour limits (default: 1000/hour)
-- ✅ Burst protection (default: 10 requests)
-- ✅ Per-client tracking (IP/user/API key)
-- ✅ Rate limit info headers
-
-#### Usage
-```python
-from utils.rate_limiter_enhanced import (
- RateLimiter,
- RateLimitConfig,
- check_rate_limit
-)
-
-# Global rate limiter
-allowed, error_msg = check_rate_limit(client_id="192.168.1.1")
-
-if not allowed:
- return {"error": error_msg}, 429
-
-# Custom rate limiter
-config = RateLimitConfig(
- requests_per_minute=60,
- requests_per_hour=2000,
- burst_size=20
-)
-limiter = RateLimiter(config)
-```
-
-#### Decorator (FastAPI)
-```python
-from utils.rate_limiter_enhanced import rate_limit
-
-@rate_limit(requests_per_minute=60, requests_per_hour=2000)
-async def api_endpoint():
- return {"data": "..."}
-```
-
----
-
-## 5. Database Migration System
-
-### Problem
-- No schema versioning
-- Manual schema changes risky
-- No rollback capability
-- Hard to track database changes
-
-### Solution Implemented
-Created `database/migrations.py`:
-
-#### Features
-- ✅ Version tracking
-- ✅ Sequential migrations
-- ✅ Automatic application on startup
-- ✅ Rollback support
-- ✅ Execution time tracking
-
-#### Usage
-```python
-from database.migrations import auto_migrate, MigrationManager
-
-# Auto-migrate on startup
-auto_migrate(db_path)
-
-# Manual migration
-manager = MigrationManager(db_path)
-success, applied = manager.migrate_to_latest()
-
-# Rollback
-manager.rollback_migration(version=3)
-
-# View history
-history = manager.get_migration_history()
-```
-
-#### Adding New Migrations
-```python
-# In database/migrations.py
-
-# Add to _register_migrations()
-self.migrations.append(Migration(
- version=6,
- description="Add user preferences table",
- up_sql="""
- CREATE TABLE user_preferences (
- user_id TEXT PRIMARY KEY,
- theme TEXT DEFAULT 'light',
- language TEXT DEFAULT 'en'
- );
- """,
- down_sql="DROP TABLE IF EXISTS user_preferences;"
-))
-```
-
-#### Registered Migrations
-1. **v1** - Add whale tracking table
-2. **v2** - Add performance indices
-3. **v3** - Add API key usage tracking
-4. **v4** - Enhance user queries with metadata
-5. **v5** - Add cache metadata table
-
----
-
-## 6. Comprehensive Testing Suite
-
-### Problem
-- Limited test coverage (~30%)
-- No unit tests with pytest
-- Manual testing only
-- No CI/CD integration
-
-### Solution Implemented
-Created comprehensive test suite:
-
-```
-tests/
-├── test_database.py # Database operations
-├── test_async_api_client.py # Async HTTP client
-├── test_auth.py # Authentication
-├── test_rate_limiter.py # Rate limiting
-├── test_migrations.py # Database migrations
-└── conftest.py # Pytest configuration
-```
-
-#### Running Tests
-```bash
-# Install dev dependencies
-pip install -r requirements-dev.txt
-
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=. --cov-report=html
-
-# Run specific test file
-pytest tests/test_database.py -v
-
-# Run specific test
-pytest tests/test_database.py::TestDatabaseInitialization::test_database_creation
-```
-
-#### Test Categories
-- ✅ Unit tests (individual functions)
-- ✅ Integration tests (multiple components)
-- ✅ Database tests (with temp DB)
-- ✅ Async tests (pytest-asyncio)
-- ✅ Concurrent tests (threading)
-
----
-
-## 7. CI/CD Pipeline
-
-### Problem
-- No automated testing
-- No continuous integration
-- Manual deployment process
-- No code quality checks
-
-### Solution Implemented
-Created `.github/workflows/ci.yml`:
-
-#### Pipeline Stages
-1. **Code Quality** - Black, isort, flake8, mypy, pylint
-2. **Tests** - pytest on Python 3.8-3.11
-3. **Security** - Safety, Bandit scans
-4. **Docker** - Build and test Docker image
-5. **Integration** - Full integration tests
-6. **Performance** - Benchmark tests
-7. **Documentation** - Build and deploy docs
-
-#### Triggers
-- Push to main/develop branches
-- Pull requests
-- Push to claude/* branches
-
-#### Status Badges
-Add to README.md:
-```markdown
-
-
-```
-
----
-
-## 8. Code Quality Tools
-
-### Problem
-- Inconsistent code style
-- No automated formatting
-- Type hints incomplete
-- No import sorting
-
-### Solution Implemented
-Configuration files created:
-
-#### Tools Configured
-1. **Black** - Code formatting
-2. **isort** - Import sorting
-3. **flake8** - Linting
-4. **mypy** - Type checking
-5. **pylint** - Code analysis
-6. **bandit** - Security scanning
-
-#### Configuration
-- `pyproject.toml` - Black, isort, pytest, mypy
-- `.flake8` - Flake8 configuration
-- `requirements-dev.txt` - Development dependencies
-
-#### Usage
-```bash
-# Format code
-black .
-
-# Sort imports
-isort .
-
-# Check linting
-flake8 .
-
-# Type check
-mypy .
-
-# Security scan
-bandit -r .
-
-# Run all checks
-black . && isort . && flake8 . && mypy .
-```
-
-#### Pre-commit Hook
-```bash
-# Install pre-commit
-pip install pre-commit
-
-# Setup hooks
-pre-commit install
-
-# Run manually
-pre-commit run --all-files
-```
-
----
-
-## 9. Updated Project Structure
-
-### New Files Created
-```
-crypto-dt-source/
-├── ui/ # NEW: Modular UI components
-│ ├── __init__.py
-│ ├── dashboard_live.py
-│ ├── dashboard_charts.py
-│ ├── dashboard_news.py
-│ ├── dashboard_ai.py
-│ ├── dashboard_db.py
-│ ├── dashboard_status.py
-│ └── interface.py
-│
-├── utils/ # ENHANCED
-│ ├── async_api_client.py # NEW: Unified async client
-│ ├── auth.py # NEW: Authentication system
-│ └── rate_limiter_enhanced.py # NEW: Rate limiting
-│
-├── database/ # ENHANCED
-│ └── migrations.py # NEW: Migration system
-│
-├── tests/ # ENHANCED
-│ ├── test_database.py # NEW: Database tests
-│ ├── test_async_api_client.py # NEW: Async client tests
-│ └── conftest.py # NEW: Pytest config
-│
-├── .github/
-│ └── workflows/
-│ └── ci.yml # NEW: CI/CD pipeline
-│
-├── pyproject.toml # NEW: Tool configuration
-├── .flake8 # NEW: Flake8 config
-├── requirements-dev.txt # NEW: Dev dependencies
-└── IMPLEMENTATION_FIXES.md # NEW: This document
-```
-
----
-
-## 10. Deployment Checklist
-
-### Before Production
-- [ ] Set `ENABLE_AUTH=true` in environment
-- [ ] Generate secure `SECRET_KEY`
-- [ ] Create admin credentials
-- [ ] Configure rate limits
-- [ ] Run database migrations
-- [ ] Run security scans
-- [ ] Configure logging level
-- [ ] Setup monitoring/alerts
-- [ ] Test authentication
-- [ ] Test rate limiting
-- [ ] Backup database
-
-### Environment Variables
-```bash
-# Production .env
-ENABLE_AUTH=true
-SECRET_KEY=
-ADMIN_USERNAME=admin
-ADMIN_PASSWORD=
-ACCESS_TOKEN_EXPIRE_MINUTES=60
-API_KEYS=
-LOG_LEVEL=INFO
-DATABASE_PATH=data/database/crypto_aggregator.db
-```
-
----
-
-## 11. Performance Improvements
-
-### Implemented Optimizations
-1. **Async Operations** - Non-blocking I/O
-2. **Connection Pooling** - Reduced overhead
-3. **Database Indices** - Faster queries
-4. **Caching** - TTL-based caching
-5. **Batch Operations** - Reduced DB calls
-6. **Parallel Requests** - Concurrent API calls
-
-### Expected Impact
-- ⚡ 5x faster data collection (parallel async)
-- ⚡ 3x faster database queries (indices)
-- ⚡ 10x reduced API calls (caching)
-- ⚡ Better resource utilization
-
----
-
-## 12. Security Enhancements
-
-### Implemented
-- ✅ Authentication required for sensitive endpoints
-- ✅ Rate limiting prevents abuse
-- ✅ Password hashing (SHA-256)
-- ✅ SQL injection prevention (parameterized queries)
-- ✅ API key tracking and revocation
-- ✅ Token expiration
-- ✅ Security scanning in CI/CD
-
-### Remaining Recommendations
-- [ ] HTTPS enforcement
-- [ ] CORS configuration
-- [ ] Input sanitization layer
-- [ ] Audit logging
-- [ ] Intrusion detection
-
----
-
-## 13. Documentation Updates
-
-### Created/Updated
-- ✅ IMPLEMENTATION_FIXES.md (this file)
-- ✅ Inline code documentation
-- ✅ Function docstrings
-- ✅ Type hints
-- ✅ Usage examples
-
-### TODO
-- [ ] Update README.md with new features
-- [ ] Create API documentation
-- [ ] Add architecture diagrams
-- [ ] Create deployment guide
-- [ ] Write migration guide
-
----
-
-## 14. Metrics & KPIs
-
-### Before Fixes
-- Lines per file: 1,495 (max)
-- Test coverage: ~30%
-- Type hints: ~60%
-- CI/CD: None
-- Authentication: None
-- Rate limiting: None
-
-### After Fixes
-- Lines per file: <300 (modular)
-- Test coverage: 60%+ (target 80%)
-- Type hints: 80%+
-- CI/CD: Full pipeline
-- Authentication: JWT + API keys
-- Rate limiting: Token bucket + sliding window
-
----
-
-## 15. Migration Path
-
-### For Existing Deployments
-
-1. **Backup Data**
- ```bash
- cp -r data/database data/database.backup
- ```
-
-2. **Install Dependencies**
- ```bash
- pip install -r requirements.txt
- pip install -r requirements-dev.txt
- ```
-
-3. **Run Migrations**
- ```python
- from database.migrations import auto_migrate
- auto_migrate("data/database/crypto_aggregator.db")
- ```
-
-4. **Update Environment**
- ```bash
- cp .env.example .env
- # Edit .env with your configuration
- ```
-
-5. **Test**
- ```bash
- pytest
- ```
-
-6. **Deploy**
- ```bash
- # With Docker
- docker-compose up -d
-
- # Or directly
- python app.py
- ```
-
----
-
-## 16. Future Enhancements
-
-### Short-term (1-2 months)
-- [ ] Complete UI refactoring
-- [ ] Achieve 80% test coverage
-- [ ] Add GraphQL API
-- [ ] Implement WebSocket authentication
-- [ ] Add user management dashboard
-
-### Medium-term (3-6 months)
-- [ ] Microservices architecture
-- [ ] Message queue (RabbitMQ/Redis)
-- [ ] Database replication
-- [ ] Multi-tenancy support
-- [ ] Advanced ML models
-
-### Long-term (6-12 months)
-- [ ] Kubernetes deployment
-- [ ] Multi-region support
-- [ ] Premium data sources
-- [ ] SLA monitoring
-- [ ] Enterprise features
-
----
-
-## 17. Support & Maintenance
-
-### Getting Help
-- GitHub Issues: https://github.com/nimazasinich/crypto-dt-source/issues
-- Documentation: See /docs folder
-- Examples: See /examples folder
-
-### Contributing
-1. Fork repository
-2. Create feature branch
-3. Make changes with tests
-4. Run quality checks
-5. Submit pull request
-
-### Monitoring
-```bash
-# Check logs
-tail -f logs/crypto_aggregator.log
-
-# Database health
-sqlite3 data/database/crypto_aggregator.db "SELECT COUNT(*) FROM prices;"
-
-# API health
-curl http://localhost:7860/api/health
-```
-
----
-
-## Conclusion
-
-All critical issues identified in the analysis have been addressed with production-ready solutions. The codebase is now:
-
-- ✅ Modular and maintainable
-- ✅ Fully tested with CI/CD
-- ✅ Secure with authentication
-- ✅ Protected with rate limiting
-- ✅ Versioned with migrations
-- ✅ Type-safe with hints
-- ✅ Quality-checked with tools
-- ✅ Ready for production
-
-**Next Steps**: Review, test, and deploy these improvements to production.
diff --git a/app/IMPLEMENTATION_ROADMAP.md b/app/IMPLEMENTATION_ROADMAP.md
deleted file mode 100644
index d2caab9ea99fc495b86a08e133ce95d155ab9ee3..0000000000000000000000000000000000000000
--- a/app/IMPLEMENTATION_ROADMAP.md
+++ /dev/null
@@ -1,562 +0,0 @@
-# Implementation Roadmap: Crypto-DT-Source Full Activation
-
-**Objective:** Transform from monitoring platform to complete cryptocurrency data aggregation service
-**Timeline:** 2-3 weeks (estimated)
-**Target Deployment:** HuggingFace Spaces
-
----
-
-## Week 1: Core Data Integration
-
-### Day 1-2: Real Market Data
-**Goal:** Replace mock data with real API calls
-
-```
-Task List:
-1. [ ] Review api_server_extended.py endpoints
-2. [ ] Modify /api/market endpoint
- - Remove hardcoded response
- - Call provider_manager.get_best_provider('market_data')
- - Execute async request to CoinGecko
- - Implement 5-minute cache
- - Add error handling with fallback
-
-3. [ ] Modify /api/prices endpoint
- - Parse symbols parameter (BTC,ETH,SOL)
- - Call provider for each symbol
- - Implement response aggregation
- - Add caching
-
-4. [ ] Test endpoints locally
- - curl http://localhost:8000/api/market
- - curl http://localhost:8000/api/prices?symbols=BTC,ETH
- - Verify real data returned
-
-5. [ ] Add to database (prices table)
-```
-
-**Code Changes Needed:**
-- `api/endpoints.py` - Update endpoint functions
-- `collectors/market_data_extended.py` - Real API integration
-- Add `caching.py` utility for TTL-based caching
-
-**Success Criteria:**
-- [ ] /api/market returns real prices (not mock)
-- [ ] /api/prices returns real data for requested symbols
-- [ ] Response time < 1 second
-- [ ] Caching working (repeat requests are instant)
-
----
-
-### Day 3-4: Trending & OHLCV Data
-**Goal:** Add trending coins and historical data endpoints
-
-```
-Task List:
-1. [ ] Implement /api/trending endpoint
- - Call CoinGecko trending endpoint
- - Return top 7 trending coins
- - Cache for 1 hour
-
-2. [ ] Implement /api/ohlcv endpoint
- - Accept symbol (BTCUSDT) and interval (1h, 4h, 1d)
- - Call Binance OHLCV endpoint
- - Validate symbol format
- - Return last N candles
- - Cache for 15 minutes
-
-3. [ ] Add OHLCV data to database
- - Create ohlcv table
- - Store each candle
-
-4. [ ] Test with real symbols
- - BTCUSDT, ETHUSDT, SOLUSDT
- - 1h, 4h, 1d intervals
- - Verify historical data accuracy
-```
-
-**Files to Create/Modify:**
-- Add OHLCV collector in `collectors/`
-- Update database schema for ohlcv table
-
----
-
-### Day 5: DeFi Data Integration
-**Goal:** Add DeFi TVL and protocol data
-
-```
-Task List:
-1. [ ] Implement /api/defi endpoint
- - Call DeFi Llama API
- - Get top 20 protocols by TVL
- - Cache for 1 hour
-
-2. [ ] Add to database
- - Create defi_protocols table
- - Store protocol data with timestamp
-
-3. [ ] Implement /api/defi/tvl-chart
- - Query historical TVL from database
- - Aggregate by date
- - Return 30-day trend
-
-4. [ ] Test
- - Check top protocols (Aave, Compound, Curve, etc.)
- - Verify TVL values are reasonable
-```
-
-**Success Criteria:**
-- [ ] /api/defi returns top 20 protocols
-- [ ] /api/defi/tvl-chart shows 30-day trend
-- [ ] All data cached appropriately
-- [ ] Database grows with each collection
-
----
-
-## Week 2: Database & Sentiment Analysis
-
-### Day 6-7: Database Activation
-**Goal:** Fully integrate SQLite for data persistence
-
-```
-Task List:
-1. [ ] Review database/models.py
- - Verify all tables are defined
- - Check for missing tables
-
-2. [ ] Run migrations
- - Execute database/migrations.py
- - Create all tables
- - Verify with sqlite3 CLI
-
-3. [ ] Update endpoints to write to database
- - After fetching real price data → store in prices table
- - After fetching OHLCV → store in ohlcv table
- - After fetching DeFi → store in defi_protocols table
- - After fetching news → store in news table
-
-4. [ ] Create read endpoints for historical data
- - GET /api/prices/history/{symbol}?days=30
- - GET /api/ohlcv/history/{symbol}?interval=1d&days=90
- - GET /api/defi/history/{protocol}?days=30
-
-5. [ ] Implement cleanup tasks
- - Keep only 90 days of price data
- - Archive older data if needed
- - Auto-vacuum database weekly
-
-6. [ ] Test data persistence
- - Restart server
- - Verify data still exists
- - Query historical data
-```
-
-**Database Tables:**
-```sql
--- Core data
-prices (id, symbol, price, source, timestamp)
-ohlcv (id, symbol, open, high, low, close, volume, timestamp)
-defi_protocols (id, name, tvl, chain, timestamp)
-news (id, title, content, source, sentiment, timestamp)
-
--- Metadata
-providers (id, name, status, last_check)
-api_calls (id, endpoint, provider, response_time, status, timestamp)
-```
-
----
-
-### Day 8-9: Real Sentiment Analysis
-**Goal:** Load HuggingFace models and perform real analysis
-
-```
-Task List:
-1. [ ] Update requirements.txt
- - Add torch==2.1.1
- - Add transformers==4.35.2
- - Add huggingface-hub==0.19.1
-
-2. [ ] Create AIModelManager class in ai_models.py
- - Initialize on app startup
- - Load sentiment model: distilbert-base-uncased-finetuned-sst-2-english
- - Load zero-shot model: facebook/bart-large-mnli
-
-3. [ ] Implement sentiment endpoints
- - POST /api/sentiment/analyze
- - POST /api/sentiment/crypto-analysis
- - GET /api/sentiment (overall feed sentiment)
-
-4. [ ] Create news sentiment pipeline
- - Fetch news every 30 minutes
- - Analyze sentiment of each headline
- - Store sentiment score in database
- - Calculate aggregate sentiment
-
-5. [ ] Test sentiment analysis
- - Analyze sample crypto news
- - Verify sentiment scores make sense
- - Check inference speed (should be < 1s)
-
-6. [ ] Integrate with news endpoint
- - Return sentiment with each news item
- - Show overall sentiment trend
-```
-
-**Implementation Pattern:**
-```python
-# Start: Load models on startup
-@app.on_event("startup")
-async def startup():
- ai_manager = AIModelManager()
- await ai_manager.initialize()
-
-# Use: Call model for analysis
-@app.post("/api/sentiment/analyze")
-async def analyze(request: AnalyzeRequest):
- result = await ai_manager.analyze_sentiment(request.text)
- return result
-```
-
----
-
-### Day 10: WebSocket Real-Time Updates
-**Goal:** Ensure WebSocket is fully functional and broadcasting real data
-
-```
-Task List:
-1. [ ] Review WS implementation
- - Verify /ws endpoint works
- - Check message subscriptions
-
-2. [ ] Update broadcasts to use real data
- - Broadcast real price updates (every 5 min)
- - Broadcast sentiment changes (every hour)
- - Broadcast news alerts (as available)
-
-3. [ ] Test WebSocket
- - Connect client
- - Subscribe to price updates
- - Verify real data received
- - Check update frequency
-
-4. [ ] Implement client heartbeat
- - Ping every 10 seconds
- - Handle client disconnects
- - Auto-reconnect logic
-
-5. [ ] Test with multiple clients
- - Connect 5+ clients
- - Verify all receive broadcasts
- - Check no connection leaks
-```
-
----
-
-## Week 3: Security & HF Deployment
-
-### Day 11-12: Authentication & Rate Limiting
-**Goal:** Secure all endpoints with authentication and rate limits
-
-```
-Task List:
-1. [ ] Create JWT authentication
- - Generate secret key
- - Implement token creation endpoint
- - Implement token verification
- - Add to dependencies
-
-2. [ ] Update endpoints
- - Mark protected endpoints (historical data, sentiment)
- - Add auth dependency
- - Test authentication flow
-
-3. [ ] Implement rate limiting
- - Install slowapi
- - Define rate limit tiers:
- * Free: 30/min, 1000/day
- * Pro: 300/min, 50000/day
- - Apply limits to endpoints
- - Test limit enforcement
-
-4. [ ] Create API key system
- - Database table for API keys
- - Endpoint to generate keys
- - Validate keys on token request
-
-5. [ ] Test security
- - Verify unauthenticated requests rejected
- - Verify rate limits work
- - Test API key validation
-```
-
----
-
-### Day 13: Monitoring & Diagnostics
-**Goal:** Complete monitoring and self-repair capabilities
-
-```
-Task List:
-1. [ ] Enhance /api/health endpoint
- - Check database connectivity
- - Check provider availability
- - Check model loading
- - Check WebSocket connections
- - Return component status
-
-2. [ ] Enhance /api/diagnostics/run endpoint
- - Full system health check
- - Issue detection
- - Auto-fix capability
- - Report generation
-
-3. [ ] Add metrics endpoint
- - CPU/memory/disk usage
- - Database size
- - Active connections
- - Request statistics
-
-4. [ ] Create monitoring dashboard
- - Show system health
- - Show provider statistics
- - Show recent errors
- - Show performance metrics
-
-5. [ ] Test all diagnostic features
- - Run diagnostics
- - Verify issues detected
- - Verify auto-fix works
-```
-
----
-
-### Day 14-15: HuggingFace Deployment
-**Goal:** Deploy complete system to HF Spaces
-
-```
-Task List:
-1. [ ] Create spaces/ directory structure
- - app.py (entry point)
- - requirements.txt
- - README.md
- - .env template
-
-2. [ ] Configure for HF environment
- - Set PORT=7860
- - Configure database path
- - Set up logging
-
-3. [ ] Test locally with Docker
- - docker build -f Dockerfile.hf .
- - docker run -p 7860:7860 ...
- - Test all endpoints
-
-4. [ ] Push to HF Spaces
- - Create Space on HF
- - Configure git
- - Push code
- - Monitor build logs
-
-5. [ ] Verify deployment
- - Test /api/health
- - Test real endpoints
- - Check logs
- - Monitor metrics
- - Test WebSocket
- - Verify rate limiting
-
-6. [ ] Setup monitoring
- - Monitor error logs
- - Track uptime
- - Monitor performance
- - Set up alerts
-
-7. [ ] Documentation
- - Create API reference
- - Add usage examples
- - Document rate limits
- - Add troubleshooting guide
-```
-
----
-
-## Implementation Checklist
-
-### ✅ Phase 1: Data Integration
-- [ ] Market data endpoint (real)
-- [ ] Prices endpoint (real)
-- [ ] Trending endpoint (real)
-- [ ] OHLCV endpoint (real)
-- [ ] DeFi endpoint (real)
-- [ ] All mock data removed
-- [ ] Caching implemented
-- [ ] Error handling with fallback
-
-### ✅ Phase 2: Database
-- [ ] Migrations run successfully
-- [ ] All tables created
-- [ ] Data write pipeline implemented
-- [ ] Historical data read endpoints
-- [ ] Cleanup tasks automated
-- [ ] Database tested across restarts
-
-### ✅ Phase 3: Sentiment Analysis
-- [ ] HuggingFace models load
-- [ ] Sentiment endpoint works
-- [ ] Crypto sentiment endpoint works
-- [ ] News sentiment pipeline running
-- [ ] Sentiment stored in database
-- [ ] Response time < 2 seconds
-
-### ✅ Phase 4: Security
-- [ ] JWT tokens implemented
-- [ ] Rate limiting enforced
-- [ ] API key system working
-- [ ] Protected endpoints verified
-- [ ] Authentication tests pass
-
-### ✅ Phase 5: Monitoring
-- [ ] Health check comprehensive
-- [ ] Diagnostics endpoint full
-- [ ] Metrics endpoint working
-- [ ] Monitoring dashboard created
-- [ ] Auto-repair working
-
-### ✅ Phase 6: Deployment
-- [ ] Spaces directory created
-- [ ] Entry point configured
-- [ ] Docker builds successfully
-- [ ] Deployed to HF Spaces
-- [ ] All endpoints accessible
-- [ ] Real data flowing
-- [ ] WebSocket working
-- [ ] Performance acceptable
-
----
-
-## Testing Protocol
-
-### Unit Tests
-```bash
-pytest tests/test_sentiment.py -v
-pytest tests/test_database.py -v
-pytest tests/test_providers.py -v
-pytest tests/test_authentication.py -v
-pytest tests/test_rate_limiting.py -v
-```
-
-### Integration Tests
-```bash
-# Test complete flow
-python test_integration.py
-
-# Test API endpoints
-bash tests/test_endpoints.sh
-
-# Test WebSocket
-python tests/test_websocket.py
-
-# Load testing
-locust -f tests/locustfile.py
-```
-
-### Manual Testing
-```bash
-# Health check
-curl http://localhost:8000/api/health | jq
-
-# Real data
-curl http://localhost:8000/api/prices?symbols=BTC,ETH | jq
-
-# Sentiment
-curl -X POST http://localhost:8000/api/sentiment/analyze \
- -H "Content-Type: application/json" \
- -d '{"text":"Bitcoin is looking bullish"}'
-
-# Authentication
-curl -H "Authorization: Bearer TOKEN" \
- http://localhost:8000/api/protected-data
-
-# WebSocket
-wscat -c ws://localhost:8000/ws
-```
-
----
-
-## Performance Targets
-
-| Component | Target | How to Achieve |
-|-----------|--------|----------------|
-| Price API | < 500ms | Caching + provider selection |
-| Sentiment | < 2s | Model optimization + batch processing |
-| Database | < 100ms | Indexing + connection pooling |
-| WebSocket | Real-time | Async updates every 5 min |
-| Auth | < 50ms | JWT validation cache |
-| Throughput | 100+ req/s | Async handlers + load balancing |
-
----
-
-## Risk Mitigation
-
-**Risk:** Models too large for HF Spaces
-**Mitigation:** Use distilbert instead of full BERT
-**Fallback:** Implement keyword-based sentiment if models fail
-
-**Risk:** Database grows too large
-**Mitigation:** Implement data cleanup (90-day retention)
-**Fallback:** Archive to S3, query from archive
-
-**Risk:** Rate limiting causes dev friction
-**Mitigation:** Admin tier with no limits
-**Fallback:** Adjustable limits in environment
-
-**Risk:** WebSocket connections consume memory
-**Mitigation:** Connection pooling + heartbeat timeouts
-**Fallback:** Implement connection limits
-
----
-
-## Success Metrics
-
-Track these metrics after deployment:
-
-```
-Availability: > 99.9% uptime
-Response Time: 95th percentile < 500ms
-Error Rate: < 0.1%
-Data Freshness: Price data < 5 min old
-Sentiment Accuracy: > 85% on test set
-Database Growth: < 1GB per month
-Memory Usage: < 1GB average
-CPU Usage: < 50% average
-WebSocket Clients: Support 100+ concurrent
-Rate Limit Accuracy: ± 1% of limit
-```
-
----
-
-## Post-Launch
-
-### Week 1: Monitoring
-- [ ] Monitor error logs daily
-- [ ] Check performance metrics
-- [ ] Verify data quality
-- [ ] Collect user feedback
-
-### Week 2: Optimization
-- [ ] Optimize slow endpoints
-- [ ] Fine-tune rate limits
-- [ ] Adjust cache TTLs
-- [ ] Update documentation
-
-### Week 3: Features
-- [ ] Gather feature requests
-- [ ] Implement high-demand features
-- [ ] Fix reported issues
-- [ ] Plan next release
-
----
-
-**Version:** 1.0
-**Last Updated:** 2025-11-15
-**Estimated Duration:** 2-3 weeks
-**Difficulty:** Medium (5/10)
diff --git a/app/PROVIDER_AUTO_DISCOVERY_REPORT.json b/app/PROVIDER_AUTO_DISCOVERY_REPORT.json
deleted file mode 100644
index 2be17d3c53048698efc53c60a4a1342e210d1490..0000000000000000000000000000000000000000
--- a/app/PROVIDER_AUTO_DISCOVERY_REPORT.json
+++ /dev/null
@@ -1,4835 +0,0 @@
-{
- "report_type": "Provider Auto-Discovery Validation Report",
- "generated_at": "2025-11-16T14:39:44.722871",
- "stats": {
- "total_http_candidates": 339,
- "total_hf_candidates": 4,
- "http_valid": 92,
- "http_invalid": 157,
- "http_conditional": 90,
- "hf_valid": 2,
- "hf_invalid": 0,
- "hf_conditional": 2,
- "total_active_providers": 94,
- "execution_time_sec": 60.52921795845032,
- "timestamp": "2025-11-16T14:38:44.193640"
- },
- "http_providers": {
- "total_candidates": 339,
- "valid": 92,
- "invalid": 157,
- "conditional": 90,
- "results": [
- {
- "provider_id": "infura_eth_mainnet",
- "provider_name": "Infura Ethereum Mainnet",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via INFURA_ETH_MAINNET_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "INFURA_ETH_MAINNET_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.195937
- },
- {
- "provider_id": "infura_eth_sepolia",
- "provider_name": "Infura Ethereum Sepolia",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via INFURA_ETH_SEPOLIA_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "INFURA_ETH_SEPOLIA_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.1959488
- },
- {
- "provider_id": "alchemy_eth_mainnet",
- "provider_name": "Alchemy Ethereum Mainnet",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via ALCHEMY_ETH_MAINNET_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "ALCHEMY_ETH_MAINNET_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.195954
- },
- {
- "provider_id": "alchemy_eth_mainnet_ws",
- "provider_name": "Alchemy Ethereum Mainnet WS",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via ALCHEMY_ETH_MAINNET_WS_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "ALCHEMY_ETH_MAINNET_WS_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.1959577
- },
- {
- "provider_id": "ankr_eth",
- "provider_name": "Ankr Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "RPC error: {'code': -32000, 'message': 'Unauthorized: You must authenticate your request with an API key. Create an account on https://www.ankr.com/rpc/ and generate your personal API key for free.'}",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.4758701
- },
- {
- "provider_id": "publicnode_eth_mainnet",
- "provider_name": "PublicNode Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 205.50155639648438,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://ethereum.publicnode.com",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x16b592b\"}",
- "validated_at": 1763303924.4519503
- },
- {
- "provider_id": "publicnode_eth_allinone",
- "provider_name": "PublicNode Ethereum All-in-one",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 147.0949649810791,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://ethereum-rpc.publicnode.com",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x16b592b\"}",
- "validated_at": 1763303924.4093559
- },
- {
- "provider_id": "cloudflare_eth",
- "provider_name": "Cloudflare Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "RPC error: {'code': -32046, 'message': 'Cannot fulfill request'}",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303924.4103744
- },
- {
- "provider_id": "llamanodes_eth",
- "provider_name": "LlamaNodes Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 106.95338249206543,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://eth.llamarpc.com",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x16b592b\"}",
- "validated_at": 1763303924.400666
- },
- {
- "provider_id": "one_rpc_eth",
- "provider_name": "1RPC Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 267.0786380767822,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://1rpc.io/eth",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"result\": \"0x16b592a\", \"id\": 1}",
- "validated_at": 1763303924.5764456
- },
- {
- "provider_id": "drpc_eth",
- "provider_name": "dRPC Ethereum",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 195.85251808166504,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://eth.drpc.org",
- "response_sample": "{\"id\": 1, \"jsonrpc\": \"2.0\", \"result\": \"0x16b592b\"}",
- "validated_at": 1763303925.273127
- },
- {
- "provider_id": "bsc_official_mainnet",
- "provider_name": "BSC Official Mainnet",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 208.24170112609863,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://bsc-dataseed.binance.org",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x413c234\"}",
- "validated_at": 1763303925.3016627
- },
- {
- "provider_id": "bsc_official_alt1",
- "provider_name": "BSC Official Alt1",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 201.45368576049805,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://bsc-dataseed1.defibit.io",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x413c234\"}",
- "validated_at": 1763303925.3109312
- },
- {
- "provider_id": "bsc_official_alt2",
- "provider_name": "BSC Official Alt2",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 177.98852920532227,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://bsc-dataseed1.ninicoin.io",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x413c234\"}",
- "validated_at": 1763303925.3034506
- },
- {
- "provider_id": "ankr_bsc",
- "provider_name": "Ankr BSC",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "RPC error: {'code': -32000, 'message': 'Unauthorized: You must authenticate your request with an API key. Create an account on https://www.ankr.com/rpc/ and generate your personal API key for free.'}",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.3043656
- },
- {
- "provider_id": "publicnode_bsc",
- "provider_name": "PublicNode BSC",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 162.3549461364746,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://bsc-rpc.publicnode.com",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x413c234\"}",
- "validated_at": 1763303925.3195105
- },
- {
- "provider_id": "nodereal_bsc",
- "provider_name": "Nodereal BSC",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via NODEREAL_BSC_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "NODEREAL_BSC_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.1729424
- },
- {
- "provider_id": "trongrid_mainnet",
- "provider_name": "TronGrid Mainnet",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 405",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.4370666
- },
- {
- "provider_id": "tronstack_mainnet",
- "provider_name": "TronStack Mainnet",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.302153
- },
- {
- "provider_id": "tron_nile_testnet",
- "provider_name": "Tron Nile Testnet",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.2748291
- },
- {
- "provider_id": "polygon_official_mainnet",
- "provider_name": "Polygon Official Mainnet",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 186.77377700805664,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://polygon-rpc.com",
- "response_sample": "{\"id\": 1, \"jsonrpc\": \"2.0\", \"result\": \"0x4b6f63c\"}",
- "validated_at": 1763303926.1245918
- },
- {
- "provider_id": "polygon_mumbai",
- "provider_name": "Polygon Mumbai",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.067372
- },
- {
- "provider_id": "ankr_polygon",
- "provider_name": "Ankr Polygon",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "RPC error: {'code': -32000, 'message': 'Unauthorized: You must authenticate your request with an API key. Create an account on https://www.ankr.com/rpc/ and generate your personal API key for free.'}",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.1366556
- },
- {
- "provider_id": "publicnode_polygon_bor",
- "provider_name": "PublicNode Polygon Bor",
- "provider_type": "http_rpc",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 141.09563827514648,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://polygon-bor-rpc.publicnode.com",
- "response_sample": "{\"jsonrpc\": \"2.0\", \"id\": 1, \"result\": \"0x4b6f63c\"}",
- "validated_at": 1763303926.1245015
- },
- {
- "provider_id": "etherscan_primary",
- "provider_name": "Etherscan",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via ETHERSCAN_PRIMARY_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "ETHERSCAN_PRIMARY_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.9984982
- },
- {
- "provider_id": "etherscan_secondary",
- "provider_name": "Etherscan (secondary key)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via ETHERSCAN_SECONDARY_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "ETHERSCAN_SECONDARY_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.9985049
- },
- {
- "provider_id": "blockchair_ethereum",
- "provider_name": "Blockchair Ethereum",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via BLOCKCHAIR_ETHEREUM_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "BLOCKCHAIR_ETHEREUM_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303925.9985082
- },
- {
- "provider_id": "blockscout_ethereum",
- "provider_name": "Blockscout Ethereum",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 177.49786376953125,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://eth.blockscout.com/api/?module=account&action=balance&address={address}",
- "response_sample": "{\"message\": \"Invalid address hash\", \"result\": null, \"status\": \"0\"}",
- "validated_at": 1763303926.1760335
- },
- {
- "provider_id": "ethplorer",
- "provider_name": "Ethplorer",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via ETHPLORER_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "ETHPLORER_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.013709
- },
- {
- "provider_id": "etherchain",
- "provider_name": "Etherchain",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 301",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.1938097
- },
- {
- "provider_id": "chainlens",
- "provider_name": "Chainlens",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.7967305
- },
- {
- "provider_id": "bscscan_primary",
- "provider_name": "BscScan",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via BSCSCAN_PRIMARY_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "BSCSCAN_PRIMARY_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.7099202
- },
- {
- "provider_id": "bitquery_bsc",
- "provider_name": "BitQuery (BSC)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 401 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.1602676
- },
- {
- "provider_id": "ankr_multichain_bsc",
- "provider_name": "Ankr MultiChain (BSC)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.896371
- },
- {
- "provider_id": "nodereal_bsc_explorer",
- "provider_name": "Nodereal BSC",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via NODEREAL_BSC_EXPLORER_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "NODEREAL_BSC_EXPLORER_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.7402933
- },
- {
- "provider_id": "bsctrace",
- "provider_name": "BscTrace",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.8509157
- },
- {
- "provider_id": "oneinch_bsc_api",
- "provider_name": "1inch BSC API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 301",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.8252053
- },
- {
- "provider_id": "tronscan_primary",
- "provider_name": "TronScan",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via TRONSCAN_PRIMARY_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "TRONSCAN_PRIMARY_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.7705665
- },
- {
- "provider_id": "trongrid_explorer",
- "provider_name": "TronGrid (Official)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.987196
- },
- {
- "provider_id": "blockchair_tron",
- "provider_name": "Blockchair TRON",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via BLOCKCHAIR_TRON_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "BLOCKCHAIR_TRON_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303926.7856803
- },
- {
- "provider_id": "tronscan_api_v2",
- "provider_name": "Tronscan API v2",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 301",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.8082662
- },
- {
- "provider_id": "getblock_tron",
- "provider_name": "GetBlock TRON",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 403 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.1050863
- },
- {
- "provider_id": "coingecko",
- "provider_name": "CoinGecko",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 171.60773277282715,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://api.coingecko.com/api/v3/simple/price?ids={ids}&vs_currencies={fiats}",
- "response_sample": "{}",
- "validated_at": 1763303927.863128
- },
- {
- "provider_id": "coinmarketcap_primary_1",
- "provider_name": "CoinMarketCap (key #1)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 401 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.9147437
- },
- {
- "provider_id": "coinmarketcap_primary_2",
- "provider_name": "CoinMarketCap (key #2)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 401 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.842486
- },
- {
- "provider_id": "cryptocompare",
- "provider_name": "CryptoCompare",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via CRYPTOCOMPARE_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "CRYPTOCOMPARE_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.7367067
- },
- {
- "provider_id": "coinpaprika",
- "provider_name": "Coinpaprika",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 131.178617477417,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://api.coinpaprika.com/v1/tickers",
- "response_sample": "[{'id': 'btc-bitcoin', 'name': 'Bitcoin', 'symbol': 'BTC', 'rank': 1, 'total_supply': 19949653, 'max_supply': 21000000, 'beta_value': 0.838016, 'first_data_at': '2010-07-17T00:00:00Z', 'last_updated':",
- "validated_at": 1763303927.8972013
- },
- {
- "provider_id": "coincap",
- "provider_name": "CoinCap",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.796082
- },
- {
- "provider_id": "nomics",
- "provider_name": "Nomics",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via NOMICS_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "NOMICS_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.7669592
- },
- {
- "provider_id": "messari",
- "provider_name": "Messari",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 401 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303927.9520357
- },
- {
- "provider_id": "bravenewcoin",
- "provider_name": "BraveNewCoin (RapidAPI)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 401 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.845936
- },
- {
- "provider_id": "kaiko",
- "provider_name": "Kaiko",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via KAIKO_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "KAIKO_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.6219223
- },
- {
- "provider_id": "coinapi_io",
- "provider_name": "CoinAPI.io",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via COINAPI_IO_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "COINAPI_IO_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.6219313
- },
- {
- "provider_id": "coinlore",
- "provider_name": "CoinLore",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 301",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.9359827
- },
- {
- "provider_id": "coinpaprika_market",
- "provider_name": "CoinPaprika",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 301",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.7699182
- },
- {
- "provider_id": "coincap_market",
- "provider_name": "CoinCap",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.722938
- },
- {
- "provider_id": "defillama_prices",
- "provider_name": "DefiLlama (Prices)",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 112.82992362976074,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://coins.llama.fi/prices/current/{coins}",
- "response_sample": "{\"coins\": {}}",
- "validated_at": 1763303928.780707
- },
- {
- "provider_id": "binance_public",
- "provider_name": "Binance Public",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 451",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.7322414
- },
- {
- "provider_id": "cryptocompare_market",
- "provider_name": "CryptoCompare",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via CRYPTOCOMPARE_MARKET_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "CRYPTOCOMPARE_MARKET_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.6983235
- },
- {
- "provider_id": "coindesk_price",
- "provider_name": "CoinDesk Price API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303928.72324
- },
- {
- "provider_id": "mobula",
- "provider_name": "Mobula API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303930.2114985
- },
- {
- "provider_id": "tokenmetrics",
- "provider_name": "Token Metrics API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 400",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.699755
- },
- {
- "provider_id": "freecryptoapi",
- "provider_name": "FreeCryptoAPI",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 403 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.8865619
- },
- {
- "provider_id": "diadata",
- "provider_name": "DIA Data",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "HTTP 404",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.6728292
- },
- {
- "provider_id": "coinstats_public",
- "provider_name": "CoinStats Public API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 100.00944137573242,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://api.coinstats.app/public/v1",
- "response_sample": "{\"message\": \"This API is deprecated and will be disabled by Oct 31 2023, to use the new version please go to https://openapi.coinstats.app .\"}",
- "validated_at": 1763303929.5980232
- },
- {
- "provider_id": "newsapi_org",
- "provider_name": "NewsAPI.org",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via NEWSAPI_ORG_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "NEWSAPI_ORG_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.5132222
- },
- {
- "provider_id": "cryptopanic",
- "provider_name": "CryptoPanic",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via CRYPTOPANIC_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "CRYPTOPANIC_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.5132291
- },
- {
- "provider_id": "cryptocontrol",
- "provider_name": "CryptoControl",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "Requires API key via CRYPTOCONTROL_API_KEY env var",
- "requires_auth": true,
- "auth_env_var": "CRYPTOCONTROL_API_KEY",
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.5132358
- },
- {
- "provider_id": "coindesk_api",
- "provider_name": "CoinDesk API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.5544043
- },
- {
- "provider_id": "cointelegraph_api",
- "provider_name": "CoinTelegraph API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "CONDITIONALLY_AVAILABLE",
- "response_time_ms": null,
- "error_reason": "HTTP 403 - Requires authentication",
- "requires_auth": true,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303929.5966122
- },
- {
- "provider_id": "cryptoslate",
- "provider_name": "CryptoSlate API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -2] Name or service not known",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303930.8767498
- },
- {
- "provider_id": "theblock_api",
- "provider_name": "The Block API",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "INVALID",
- "response_time_ms": null,
- "error_reason": "Exception: [Errno -5] No address associated with hostname",
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": null,
- "response_sample": null,
- "validated_at": 1763303930.8749015
- },
- {
- "provider_id": "coinstats_news",
- "provider_name": "CoinStats News",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 158.89286994934082,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://api.coinstats.app/public/v1/news",
- "response_sample": "{\"message\": \"This API is deprecated and will be disabled by Oct 31 2023, to use the new version please go to https://openapi.coinstats.app .\"}",
- "validated_at": 1763303930.901813
- },
- {
- "provider_id": "rss_cointelegraph",
- "provider_name": "Cointelegraph RSS",
- "provider_type": "http_json",
- "category": "unknown",
- "status": "VALID",
- "response_time_ms": 167.921781539917,
- "error_reason": null,
- "requires_auth": false,
- "auth_env_var": null,
- "test_endpoint": "https://cointelegraph.com/rss",
- "response_sample": "\n\n\n\n\n\n\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\n\n\n \n Aave API Documentation\n \n\n\n\n\n \n Aave API Documentation\n \n\n
-
-**Production-Ready Cryptocurrency Data Aggregator**
-
-*Real-time data collection • AI-powered analysis • Enterprise-grade security*
-
-[](https://opensource.org/licenses/MIT)
-[](https://www.python.org/downloads/)
-[](https://github.com/psf/black)
-
-[Quick Start](#-quick-start) • [Features](#-features) • [Documentation](#-documentation) • [فارسی](docs/persian/README_FA.md)
-
-
-
----
-
-## 🚀 Quick Start
-
-Get up and running in 3 simple steps:
-
-```bash
-# 1. Clone the repository
-git clone https://github.com/nimazasinich/crypto-dt-source.git
-cd crypto-dt-source
-
-# 2. Install dependencies
-pip install -r requirements.txt
-
-# 3. Run the application
-python app.py
-```
-
-Open your browser to **http://localhost:7860** 🎉
-
-> **Need more help?** See the [complete Quick Start guide](QUICK_START.md) or [Installation Guide](docs/deployment/INSTALL.md)
-
----
-
-## ✨ Features
-
-### 🔥 Core Capabilities
-
-- **Real-Time Data** - Monitor 100+ cryptocurrencies with live price updates
-- **AI-Powered Analysis** - Sentiment analysis using HuggingFace transformers
-- **200+ Free Data Sources** - No API keys required for basic features
-- **Interactive Dashboards** - 6-tab Gradio interface + 10+ HTML dashboards
-- **WebSocket Streaming** - Real-time data streaming via WebSocket API
-- **REST API** - 20+ endpoints for programmatic access
-- **SQLite Database** - Persistent storage with automatic migrations
-
-### 🆕 Production Features (Nov 2024)
-
-- ✅ **Authentication & Authorization** - JWT tokens + API key management
-- ✅ **Rate Limiting** - Multi-tier protection (30/min, 1000/hour)
-- ✅ **Async Architecture** - 5x faster data collection
-- ✅ **Database Migrations** - Version-controlled schema updates
-- ✅ **Testing Suite** - pytest with 60%+ coverage
-- ✅ **CI/CD Pipeline** - Automated testing & deployment
-- ✅ **Code Quality Tools** - black, flake8, mypy, pylint
-- ✅ **Security Scanning** - Automated vulnerability checks
-
-> **See what's new:** [Implementation Fixes](IMPLEMENTATION_FIXES.md) • [Fixes Summary](FIXES_SUMMARY.md)
-
----
-
-## 📊 Data Sources
-
-### Price & Market Data
-- **CoinGecko** - Top 100+ cryptocurrencies, market cap rankings
-- **CoinCap** - Real-time prices, backup data source
-- **Binance** - Trading volumes, OHLCV data
-- **Kraken** - Historical price data
-- **Messari** - Advanced analytics
-
-### News & Sentiment
-- **RSS Feeds** - CoinDesk, Cointelegraph, Bitcoin Magazine, Decrypt
-- **CryptoPanic** - Aggregated crypto news
-- **Reddit** - r/cryptocurrency, r/bitcoin, r/ethtrader
-- **Alternative.me** - Fear & Greed Index
-
-### Blockchain Data
-- **Etherscan** - Ethereum blockchain (optional key)
-- **BscScan** - Binance Smart Chain
-- **TronScan** - Tron blockchain
-- **Blockchair** - Multi-chain explorer
-
-**All basic features work without API keys!** 🎁
-
----
-
-## 🏗️ Architecture
-
-```
-crypto-dt-source/
-├── 📱 UI Layer
-│ ├── app.py # Main Gradio dashboard
-│ ├── ui/ # Modular UI components (NEW)
-│ │ ├── dashboard_live.py # Live price dashboard
-│ │ ├── dashboard_charts.py # Historical charts
-│ │ ├── dashboard_news.py # News & sentiment
-│ │ └── ...
-│ └── *.html # 10+ HTML dashboards
-│
-├── 🔌 API Layer
-│ ├── api/
-│ │ ├── endpoints.py # 20+ REST endpoints
-│ │ ├── websocket.py # WebSocket streaming
-│ │ ├── data_endpoints.py # Data delivery
-│ │ └── pool_endpoints.py # Provider management
-│ └── api_server_extended.py # FastAPI server
-│
-├── 💾 Data Layer
-│ ├── database.py # SQLite manager
-│ ├── database/
-│ │ ├── db_manager.py # Connection pooling
-│ │ ├── migrations.py # Schema migrations (NEW)
-│ │ └── models.py # Data models
-│ └── collectors/
-│ ├── market_data.py # Price collection
-│ ├── news.py # News aggregation
-│ ├── sentiment.py # Sentiment analysis
-│ └── ...
-│
-├── 🤖 AI Layer
-│ ├── ai_models.py # HuggingFace integration
-│ └── crypto_data_bank/ai/ # Alternative AI engine
-│
-├── 🛠️ Utilities
-│ ├── utils.py # General utilities
-│ ├── utils/
-│ │ ├── async_api_client.py # Async HTTP client (NEW)
-│ │ ├── auth.py # Authentication (NEW)
-│ │ └── rate_limiter_enhanced.py # Rate limiting (NEW)
-│ └── monitoring/
-│ ├── health_monitor.py # Health checks
-│ └── scheduler.py # Background tasks
-│
-├── 🧪 Testing
-│ ├── tests/
-│ │ ├── test_database.py # Database tests (NEW)
-│ │ ├── test_async_api_client.py # Async tests (NEW)
-│ │ └── ...
-│ └── pytest.ini # Test configuration
-│
-├── ⚙️ Configuration
-│ ├── config.py # Application config
-│ ├── .env.example # Environment template
-│ ├── requirements.txt # Production deps
-│ ├── requirements-dev.txt # Dev dependencies (NEW)
-│ ├── pyproject.toml # Tool config (NEW)
-│ └── .flake8 # Linting config (NEW)
-│
-└── 📚 Documentation
- ├── README.md # This file
- ├── CHANGELOG.md # Version history
- ├── QUICK_START.md # Quick start guide
- ├── IMPLEMENTATION_FIXES.md # Latest improvements (NEW)
- ├── FIXES_SUMMARY.md # Fixes summary (NEW)
- └── docs/ # Organized documentation (NEW)
- ├── INDEX.md # Documentation index
- ├── deployment/ # Deployment guides
- ├── components/ # Component docs
- ├── reports/ # Analysis reports
- ├── guides/ # How-to guides
- ├── persian/ # Persian/Farsi docs
- └── archive/ # Historical docs
-```
-
----
-
-## 🎯 Use Cases
-
-### For Traders
-- Real-time price monitoring across 100+ coins
-- AI sentiment analysis from news and social media
-- Technical indicators (RSI, MACD, Moving Averages)
-- Fear & Greed Index tracking
-
-### For Developers
-- REST API for building crypto applications
-- WebSocket streaming for real-time updates
-- 200+ free data sources aggregated
-- Well-documented, modular codebase
-
-### For Researchers
-- Historical price data and analysis
-- Sentiment analysis on crypto news
-- Database of aggregated market data
-- Export data to CSV for analysis
-
-### For DevOps
-- Docker containerization ready
-- HuggingFace Spaces deployment
-- Health monitoring endpoints
-- Automated testing and CI/CD
-
----
-
-## 🔧 Installation & Setup
-
-### Prerequisites
-- Python 3.8 or higher
-- 4GB+ RAM (for AI models)
-- Internet connection
-
-### Basic Installation
-
-```bash
-# Install dependencies
-pip install -r requirements.txt
-
-# Run application
-python app.py
-```
-
-### Development Setup
-
-```bash
-# Install dev dependencies
-pip install -r requirements-dev.txt
-
-# Run tests
-pytest --cov=.
-
-# Format code
-black .
-isort .
-
-# Lint
-flake8 .
-mypy .
-```
-
-### Production Deployment
-
-```bash
-# Set environment variables
-cp .env.example .env
-# Edit .env with your configuration
-
-# Run database migrations
-python -c "from database.migrations import auto_migrate; auto_migrate('data/database/crypto_aggregator.db')"
-
-# Enable authentication
-export ENABLE_AUTH=true
-export SECRET_KEY=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
-
-# Start application
-python app.py
-```
-
-### Docker Deployment
-
-```bash
-# Build image
-docker build -t crypto-dt-source .
-
-# Run container
-docker run -p 7860:7860 -v $(pwd)/data:/app/data crypto-dt-source
-
-# Or use docker-compose
-docker-compose up -d
-```
-
-> **Detailed guides:** [Deployment Guide](docs/deployment/DEPLOYMENT_GUIDE.md) • [Production Guide](docs/deployment/PRODUCTION_DEPLOYMENT_GUIDE.md) • [HuggingFace Spaces](docs/deployment/HUGGINGFACE_DEPLOYMENT.md)
-
----
-
-## 📖 Documentation
-
-### Getting Started
-- 📘 [Quick Start Guide](QUICK_START.md) - Get running in 3 steps
-- 📘 [Installation Guide](docs/deployment/INSTALL.md) - Detailed installation
-- 📘 [راهنمای فارسی](docs/persian/README_FA.md) - Persian/Farsi guide
-
-### Core Documentation
-- 📗 [Implementation Fixes](IMPLEMENTATION_FIXES.md) - Latest production improvements
-- 📗 [Fixes Summary](FIXES_SUMMARY.md) - Quick reference
-- 📗 [Changelog](CHANGELOG.md) - Version history
-
-### Component Documentation
-- 📙 [WebSocket API](docs/components/WEBSOCKET_API_DOCUMENTATION.md) - Real-time streaming
-- 📙 [Data Collectors](docs/components/COLLECTORS_README.md) - Data collection system
-- 📙 [Gradio Dashboard](docs/components/GRADIO_DASHBOARD_README.md) - UI documentation
-- 📙 [Backend Services](docs/components/README_BACKEND.md) - Backend architecture
-
-### Deployment & DevOps
-- 📕 [Deployment Guide](docs/deployment/DEPLOYMENT_GUIDE.md) - General deployment
-- 📕 [Production Guide](docs/deployment/PRODUCTION_DEPLOYMENT_GUIDE.md) - Production setup
-- 📕 [HuggingFace Deployment](docs/deployment/HUGGINGFACE_DEPLOYMENT.md) - Cloud deployment
-
-### Reports & Analysis
-- 📔 [Project Analysis](docs/reports/PROJECT_ANALYSIS_COMPLETE.md) - 40,600+ line analysis
-- 📔 [Production Audit](docs/reports/PRODUCTION_AUDIT_COMPREHENSIVE.md) - Security audit
-- 📔 [System Capabilities](docs/reports/SYSTEM_CAPABILITIES_REPORT.md) - Feature overview
-
-### Complete Index
-📚 **[Full Documentation Index](docs/INDEX.md)** - Browse all 60+ documentation files
-
----
-
-## 🔐 Security & Authentication
-
-### Authentication (Optional)
-
-Enable authentication for production deployments:
-
-```bash
-# .env configuration
-ENABLE_AUTH=true
-SECRET_KEY=your-secret-key-here
-ADMIN_USERNAME=admin
-ADMIN_PASSWORD=secure-password
-ACCESS_TOKEN_EXPIRE_MINUTES=60
-API_KEYS=key1,key2,key3
-```
-
-**Features:**
-- JWT token authentication
-- API key management
-- Password hashing (SHA-256)
-- Token expiration
-- Usage tracking
-
-> **Learn more:** [Authentication Guide](IMPLEMENTATION_FIXES.md#3-authentication--authorization-system)
-
-### Rate Limiting
-
-Protect your API from abuse:
-
-- **30 requests/minute** per client
-- **1,000 requests/hour** per client
-- **Burst protection** up to 10 requests
-
-> **Learn more:** [Rate Limiting Guide](IMPLEMENTATION_FIXES.md#4-enhanced-rate-limiting-system)
-
----
-
-## 🧪 Testing
-
-```bash
-# Install test dependencies
-pip install -r requirements-dev.txt
-
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=. --cov-report=html
-
-# Run specific test file
-pytest tests/test_database.py -v
-
-# Run integration tests
-pytest tests/test_integration.py
-```
-
-**Test Coverage:** 60%+ (target: 80%)
-
-> **Learn more:** [Testing Guide](IMPLEMENTATION_FIXES.md#6-comprehensive-testing-suite)
-
----
-
-## 🚢 CI/CD Pipeline
-
-Automated testing on every push:
-
-- ✅ Code quality checks (black, flake8, mypy)
-- ✅ Tests on Python 3.8, 3.9, 3.10, 3.11
-- ✅ Security scanning (bandit, safety)
-- ✅ Docker build verification
-- ✅ Integration tests
-- ✅ Performance benchmarks
-
-> **See:** [.github/workflows/ci.yml](.github/workflows/ci.yml)
-
----
-
-## 📊 Performance
-
-### Optimizations Implemented
-- ⚡ **5x faster** data collection (async parallel requests)
-- ⚡ **3x faster** database queries (optimized indices)
-- ⚡ **10x reduced** API calls (TTL-based caching)
-- ⚡ **Better resource** utilization (async I/O)
-
-### Benchmarks
-- Data collection: ~30 seconds for 100 coins
-- Database queries: <10ms average
-- WebSocket latency: <100ms
-- Memory usage: ~500MB (with AI models loaded)
-
----
-
-## 🤝 Contributing
-
-We welcome contributions! Here's how:
-
-1. **Fork** the repository
-2. **Create** a feature branch (`git checkout -b feature/amazing-feature`)
-3. **Make** your changes with tests
-4. **Run** quality checks (`black . && flake8 . && pytest`)
-5. **Commit** with descriptive message
-6. **Push** to your branch
-7. **Open** a Pull Request
-
-**Guidelines:**
-- Follow code style (black, isort)
-- Add tests for new features
-- Update documentation
-- Check [Pull Request Checklist](docs/guides/PR_CHECKLIST.md)
-
----
-
-## 📜 License
-
-This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.
-
----
-
-## 🙏 Acknowledgments
-
-### AI Models
-- [HuggingFace](https://huggingface.co/) - Transformers library
-- [Cardiff NLP](https://huggingface.co/cardiffnlp) - Twitter sentiment model
-- [ProsusAI](https://huggingface.co/ProsusAI) - FinBERT model
-- [Facebook](https://huggingface.co/facebook) - BART summarization
-
-### Data Sources
-- [CoinGecko](https://www.coingecko.com/) - Free crypto API
-- [CoinCap](https://coincap.io/) - Real-time data
-- [Binance](https://www.binance.com/) - Trading data
-- [Alternative.me](https://alternative.me/) - Fear & Greed Index
-
-### Frameworks & Libraries
-- [Gradio](https://gradio.app/) - Web UI framework
-- [FastAPI](https://fastapi.tiangolo.com/) - REST API
-- [Plotly](https://plotly.com/) - Interactive charts
-- [PyTorch](https://pytorch.org/) - Deep learning
-
----
-
-## 📞 Support
-
-- **Issues:** [GitHub Issues](https://github.com/nimazasinich/crypto-dt-source/issues)
-- **Documentation:** [docs/](docs/INDEX.md)
-- **Changelog:** [CHANGELOG.md](CHANGELOG.md)
-
----
-
-## 🗺️ Roadmap
-
-### Short-term (Q4 2024)
-- [x] Modular UI architecture
-- [x] Authentication system
-- [x] Rate limiting
-- [x] Database migrations
-- [x] Testing suite
-- [x] CI/CD pipeline
-- [ ] 80%+ test coverage
-- [ ] GraphQL API
-
-### Medium-term (Q1 2025)
-- [ ] Microservices architecture
-- [ ] Message queue (Redis/RabbitMQ)
-- [ ] Database replication
-- [ ] Multi-tenancy support
-- [ ] Advanced ML models
-
-### Long-term (2025)
-- [ ] Kubernetes deployment
-- [ ] Multi-region support
-- [ ] Premium data sources
-- [ ] Enterprise features
-- [ ] Mobile app
-
----
-
-
-
-**Made with ❤️ for the crypto community**
-
-⭐ **Star us on GitHub** if you find this project useful!
-
-[Documentation](docs/INDEX.md) • [Quick Start](QUICK_START.md) • [فارسی](docs/persian/README_FA.md) • [Changelog](CHANGELOG.md)
-
-
diff --git a/app/README_DEPLOYMENT.md b/app/README_DEPLOYMENT.md
deleted file mode 100644
index ae722ef264a22b1132a15393020116b4da5fc29b..0000000000000000000000000000000000000000
--- a/app/README_DEPLOYMENT.md
+++ /dev/null
@@ -1,333 +0,0 @@
-# 🚀 DEPLOYMENT READY - CRYPTO DATA AGGREGATOR
-
-## ✅ STATUS: READY FOR HUGGINGFACE DEPLOYMENT
-
-All audit blockers have been **RESOLVED**. The application is now production-ready.
-
----
-
-## 📋 IMPLEMENTATION SUMMARY
-
-### What Was Fixed
-
-1. **✅ Mock Data Eliminated**
- - All 5 endpoints now use real data providers
- - CoinGecko for market data and trending
- - Alternative.me for sentiment (Fear & Greed Index)
- - Proper 503/501 errors for unavailable services
-
-2. **✅ Dependencies Added**
- - `fastapi==0.109.0`
- - `uvicorn[standard]==0.27.0`
- - `pydantic==2.5.3`
- - `sqlalchemy==2.0.25`
- - Plus 4 additional packages
-
-3. **✅ Dockerfile Fixed**
- - Creates all required directories
- - Uses PORT environment variable (HF Spaces default: 7860)
- - Proper health check with urllib
- - Single worker mode for HF compatibility
-
-4. **✅ USE_MOCK_DATA Flag**
- - Defaults to `false` (real data mode)
- - Set to `true` for demo/testing
- - All endpoints respect this flag
-
-5. **✅ Database Integration**
- - Automatic price history recording
- - New `/api/market/history` endpoint
- - SQLite with proper schema
-
-6. **✅ Provider Failover**
- - New `provider_fetch_helper.py` module
- - Circuit breaker protection
- - Automatic retry logic
-
----
-
-## 🎯 VERIFICATION RESULTS
-
-```
-╔════════════════════════════════════════════════════════════╗
-║ ✅ ALL CHECKS PASSED ║
-║ STATUS: READY FOR HUGGINGFACE DEPLOYMENT ✅ ║
-╚════════════════════════════════════════════════════════════╝
-```
-
-### Automated Verification
-
-Run the verification script:
-```bash
-bash verify_deployment.sh
-```
-
-**Results:** ✅ **10/10 CHECKS PASSED**
-
-1. ✅ Required files exist
-2. ✅ Dockerfile configuration correct
-3. ✅ All dependencies present
-4. ✅ USE_MOCK_DATA flag implemented
-5. ✅ Real data collectors imported
-6. ✅ Mock data handling proper
-7. ✅ Database integration complete
-8. ✅ Error codes implemented
-9. ✅ Python syntax valid
-10. ✅ Documentation complete
-
----
-
-## 🐳 DOCKER COMMANDS
-
-### Build
-```bash
-docker build -t crypto-monitor .
-```
-
-### Run (Real Data Mode - Default)
-```bash
-docker run -p 7860:7860 crypto-monitor
-```
-
-### Run (Mock Data Mode - Testing)
-```bash
-docker run -p 7860:7860 -e USE_MOCK_DATA=true crypto-monitor
-```
-
-### Test Endpoints
-```bash
-# After starting the container:
-bash TEST_COMMANDS.sh
-```
-
----
-
-## 🌐 HUGGING FACE SPACES DEPLOYMENT
-
-### Step 1: Create Space
-1. Go to https://huggingface.co/spaces
-2. Click **"Create new Space"**
-3. Settings:
- - **SDK:** Docker
- - **Name:** crypto-data-aggregator
- - **Visibility:** Public
-
-### Step 2: Push Code
-```bash
-git remote add hf https://huggingface.co/spaces/YOUR_USERNAME/crypto-data-aggregator
-git add -A
-git commit -m "Deploy crypto aggregator - Production ready"
-git push hf main
-```
-
-### Step 3: Monitor Build
-- Watch build logs in HF dashboard
-- Typical build time: 2-5 minutes
-- Status should change to "Running"
-
-### Step 4: Access Your App
-```
-https://YOUR_USERNAME-crypto-data-aggregator.hf.space
-```
-
----
-
-## 🧪 ENDPOINT TESTING
-
-### Real Data Endpoints (Working)
-
-**Market Data (CoinGecko)**
-```bash
-curl https://YOUR_APP.hf.space/api/market | jq
-# Returns: Real BTC, ETH, BNB prices
-```
-
-**Sentiment (Alternative.me)**
-```bash
-curl https://YOUR_APP.hf.space/api/sentiment | jq
-# Returns: Real Fear & Greed Index
-```
-
-**Trending (CoinGecko)**
-```bash
-curl https://YOUR_APP.hf.space/api/trending | jq
-# Returns: Real trending cryptocurrencies
-```
-
-**Price History (Database)**
-```bash
-curl "https://YOUR_APP.hf.space/api/market/history?symbol=BTC&limit=10" | jq
-# Returns: Historical price records
-```
-
-### Not Implemented (Proper Error Codes)
-
-**DeFi Data**
-```bash
-curl -i https://YOUR_APP.hf.space/api/defi
-# Returns: HTTP 503 with clear message
-# Message: "DeFi endpoint not implemented with real providers yet"
-```
-
-**HF Sentiment Analysis**
-```bash
-curl -i -X POST https://YOUR_APP.hf.space/api/hf/run-sentiment \
- -H "Content-Type: application/json" \
- -d '{"texts": ["test"]}'
-# Returns: HTTP 501 with clear message
-# Message: "Real ML-based sentiment analysis not implemented yet"
-```
-
----
-
-## 📊 KEY FEATURES
-
-### Real Data Providers ✅
-- **CoinGecko:** Market data, trending coins
-- **Alternative.me:** Fear & Greed Index
-- **Binance:** Ticker data (via collectors)
-
-### Database ✅
-- **SQLite:** Automatic price history
-- **Tables:** prices, news, market_analysis, user_queries
-- **Auto-cleanup:** Configurable retention period
-
-### Error Handling ✅
-- **503 Service Unavailable:** External API failures
-- **501 Not Implemented:** Features not yet available
-- **Proper JSON errors:** Clear messages for debugging
-
-### Monitoring ✅
-- **Health Endpoint:** `/health`
-- **Provider Stats:** `/api/providers`
-- **System Status:** `/api/status`
-- **Error Logs:** `/api/logs/errors`
-
----
-
-## 📁 IMPORTANT FILES
-
-### Core Application
-- `api_server_extended.py` - Main FastAPI server (✅ Updated)
-- `provider_fetch_helper.py` - Failover helper (✅ Created)
-- `database.py` - SQLite integration (✅ Existing)
-- `requirements.txt` - Dependencies (✅ Updated)
-- `Dockerfile` - Container config (✅ Updated)
-
-### Collectors (Used by endpoints)
-- `collectors/market_data.py` - CoinGecko integration
-- `collectors/sentiment.py` - Alternative.me integration
-- `provider_manager.py` - Provider pool management
-
-### Documentation
-- `README_DEPLOYMENT.md` - **This file** (quick reference)
-- `DEPLOYMENT_INSTRUCTIONS.md` - Complete guide
-- `AUDIT_COMPLETION_REPORT.md` - Detailed audit results
-- `FINAL_IMPLEMENTATION_REPORT.md` - Full implementation report
-
-### Verification & Testing
-- `verify_deployment.sh` - Automated checks
-- `TEST_COMMANDS.sh` - Endpoint testing
-
----
-
-## ⚙️ ENVIRONMENT VARIABLES
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `PORT` | `7860` | Server port (auto-set by HF) |
-| `USE_MOCK_DATA` | `false` | Enable mock data mode |
-| `ENABLE_AUTO_DISCOVERY` | `false` | Enable resource auto-discovery |
-
-**⚠️ Note:** HuggingFace Spaces automatically sets `PORT=7860`. Do not override.
-
----
-
-## 🔍 TROUBLESHOOTING
-
-### Issue: Container won't start
-```bash
-docker logs
-# Check for missing dependencies or syntax errors
-```
-
-### Issue: Endpoints return 503
-**Possible causes:**
-- External API rate limits (CoinGecko, Alternative.me)
-- Network connectivity issues
-- Circuit breaker activated
-
-**Solutions:**
-- Wait 1-2 minutes for circuit breaker reset
-- Check `/api/logs/errors` for details
-- Temporarily enable mock mode: `USE_MOCK_DATA=true`
-
-### Issue: Empty database history
-**Normal behavior:**
-- History accumulates over time
-- Minimum 5-10 minutes for first records
-- Check `/api/market` is being called regularly
-
----
-
-## 📈 SUCCESS CRITERIA
-
-Your deployment is **SUCCESSFUL** when:
-
-- ✅ `/health` returns `{"status": "healthy"}`
-- ✅ `/api/market` shows **real** current prices (not 43250.50)
-- ✅ `/api/sentiment` shows **real** Fear & Greed Index (not always 62)
-- ✅ `/api/trending` shows **real** trending coins (not hardcoded)
-- ✅ `/api/market/history` accumulates records over time
-- ✅ `/api/defi` returns HTTP 503
-- ✅ `/api/hf/run-sentiment` returns HTTP 501
-- ✅ No `_mock: true` flags in responses (unless `USE_MOCK_DATA=true`)
-
----
-
-## 📞 QUICK REFERENCE
-
-### Verify Deployment
-```bash
-bash verify_deployment.sh
-```
-
-### Build & Run
-```bash
-docker build -t crypto-monitor .
-docker run -p 7860:7860 crypto-monitor
-```
-
-### Test All Endpoints
-```bash
-bash TEST_COMMANDS.sh
-```
-
-### Check Logs
-```bash
-docker logs
-```
-
-### Deploy to HF
-```bash
-git push hf main
-```
-
----
-
-## 🎉 YOU'RE READY!
-
-All audit requirements have been met. Your crypto data aggregator is:
-
-✅ **Production-ready**
-✅ **Fully verified**
-✅ **Documented**
-✅ **Tested**
-
-**Next Step:** Deploy to Hugging Face Spaces and share your app!
-
----
-
-**Last Updated:** 2025-11-16
-**Status:** ✅ DEPLOYMENT READY
-**Verification:** ✅ ALL CHECKS PASSED
diff --git a/app/README_HF_SPACE.md b/app/README_HF_SPACE.md
deleted file mode 100644
index b3fd6872ead66f94fa5eb52a1df325d8a4f5db36..0000000000000000000000000000000000000000
--- a/app/README_HF_SPACE.md
+++ /dev/null
@@ -1,19 +0,0 @@
-# Crypto Intelligence Hub – HF Python Space
-
-This project is prepared to run as a **Hugging Face Python Space** using FastAPI.
-
-- Entry file: `app.py`
-- Main server: `final/hf_unified_server.py`
-- Frontend UI: `final/index.html` + `final/static/` (served by FastAPI)
-- Database: SQLite (created under `data/` when the API runs)
-- Hugging Face models: configured as pipelines in `final/ai_models.py` and related modules.
- - Models are lazy-loaded when AI endpoints are called.
-
-## Run locally
-
-```bash
-pip install -r requirements_hf.txt
-uvicorn app:app --host 0.0.0.0 --port 7860
-```
-
-Then open: `http://localhost:7860/`
diff --git a/app/START_HERE.md b/app/START_HERE.md
deleted file mode 100644
index ce728d625ee6fce03a140ef176fbf2068093755b..0000000000000000000000000000000000000000
--- a/app/START_HERE.md
+++ /dev/null
@@ -1,351 +0,0 @@
-# 🚀 START HERE: Crypto-DT-Source HuggingFace Deployment
-
-**Your Complete Guide to Season 2025 Implementation**
-
----
-
-## 📚 What You Need to Know
-
-Your Crypto-DT-Source project is **audit-complete and ready for full activation**. Four comprehensive guides have been prepared to walk you through the complete process.
-
-### 🎯 Your Goal
-Transform the project from a monitoring platform into a **fully-functional cryptocurrency data aggregation service** on HuggingFace Spaces with:
-- Real cryptocurrency market data
-- AI-powered sentiment analysis
-- Historical data persistence
-- Enterprise-grade security
-- Real-time WebSocket streaming
-
-### ⏱️ Timeline
-**2-3 weeks** for complete implementation and deployment
-
-### 📊 Effort Level
-**Medium** - Mostly integration work, clear patterns provided
-
----
-
-## 📖 The Four Guides
-
-### 1. **DEPLOYMENT_MASTER_GUIDE.md** ⭐ START HERE
-**Read this first!**
-
-- Executive overview of what you'll accomplish
-- Current project status (what works, what needs completion)
-- Quick decision points and configuration options
-- Expected results timeline
-- Success metrics and final checklist
-
-**Read time:** 15 minutes
-**When to use:** Planning and understanding the big picture
-
----
-
-### 2. **IMPLEMENTATION_ROADMAP.md** 🗓️ FOLLOW THIS TIMELINE
-**Your step-by-step plan for 2-3 weeks**
-
-- **Week 1:** Core data integration (Days 1-5)
- - Replace mock market data with real API calls
- - Implement trending, OHLCV, and DeFi endpoints
-
-- **Week 2:** Database & Sentiment Analysis (Days 6-10)
- - Activate database persistence
- - Load real HuggingFace ML models
- - Implement sentiment analysis pipeline
-
-- **Week 3:** Security & Deployment (Days 11-15)
- - Add JWT authentication
- - Implement multi-tier rate limiting
- - Deploy to HuggingFace Spaces
-
-- Includes testing protocols, checklists, and success criteria for each day
-
-**Read time:** 30 minutes (full document)
-**When to use:** Following daily implementation plan
-
----
-
-### 3. **HUGGINGFACE_DEPLOYMENT_PROMPT.md** 🔧 TECHNICAL REFERENCE
-**Detailed specifications and code examples**
-
-- **Phase 1:** Real market data integration with code examples
-- **Phase 2:** Database integration patterns
-- **Phase 3:** AI models loading and sentiment analysis
-- **Phase 4:** JWT authentication and rate limiting
-- **Phase 5:** Background tasks and auto-discovery
-- **Phase 6:** HuggingFace Spaces deployment
-- Environment variables and configuration
-- Troubleshooting guide
-
-**Read time:** 60 minutes (reference as needed)
-**When to use:** Understanding requirements and finding code patterns
-
----
-
-### 4. **QUICK_REFERENCE_GUIDE.md** ⚡ LOOK UP COMMANDS
-**Quick lookup during implementation**
-
-- Essential commands (setup, testing, deployment)
-- Key files to modify (with locations)
-- Common issues and solutions
-- Debugging tips
-- Monitoring commands
-- Configuration quick reference
-
-**Read time:** 5 minutes (quick lookup)
-**When to use:** During implementation for quick answers
-
----
-
-## 🎯 Choose Your Path
-
-### Path A: Structured (Recommended for Most)
-1. Read `DEPLOYMENT_MASTER_GUIDE.md` (15 min)
-2. Skim `IMPLEMENTATION_ROADMAP.md` (10 min)
-3. Start Day 1 of roadmap
-4. Reference other guides as needed
-
-**Best for:** Clear milestones, daily guidance, built-in testing
-
-### Path B: Reference-Based (If experienced with codebase)
-1. Skim `DEPLOYMENT_MASTER_GUIDE.md` (5 min)
-2. Read relevant sections of `HUGGINGFACE_DEPLOYMENT_PROMPT.md`
-3. Implement in your preferred order
-4. Use `QUICK_REFERENCE_GUIDE.md` for troubleshooting
-
-**Best for:** Flexibility, custom approach, quick execution
-
-### Path C: Let Claude Implement (If using Claude Code)
-1. Share this guide with Claude Code
-2. Request implementation of phases
-3. Review + test each phase
-4. Deploy when complete
-
-**Best for:** Saving time, ensuring quality, learning from implementation
-
----
-
-## 🚀 Quick Start (Next 30 Minutes)
-
-```bash
-# 1. Read the master guide
-open DEPLOYMENT_MASTER_GUIDE.md
-# Time: 15 minutes
-# Understand: What you're building, current status, timeline
-
-# 2. Skim the roadmap
-open IMPLEMENTATION_ROADMAP.md
-# Time: 10 minutes
-# Understand: Week 1-3 breakdown, success criteria
-
-# 3. Set up environment
-cp .env.example .env
-# Time: 5 minutes
-# Do: Configure your development environment
-```
-
-After these 30 minutes, you'll know exactly what to do and be ready to start Day 1.
-
----
-
-## 📋 Quick Checklist
-
-### Must Happen (Non-Optional)
-- [ ] Read `DEPLOYMENT_MASTER_GUIDE.md`
-- [ ] Choose your implementation path
-- [ ] Set up `.env` file
-- [ ] Install dependencies: `pip install -r requirements.txt`
-- [ ] Follow timeline or start implementation
-- [ ] Commit changes regularly to your branch
-- [ ] Test each phase before moving to next
-
-### Should Happen (Recommended)
-- [ ] Run local tests before committing
-- [ ] Monitor CPU/memory during development
-- [ ] Keep git history clean
-- [ ] Document any issues encountered
-- [ ] Collect performance metrics
-
-### Nice to Have (Optional)
-- [ ] Set up CI/CD for testing
-- [ ] Create feature branches for large changes
-- [ ] Write additional unit tests
-- [ ] Benchmark performance improvements
-
----
-
-## 🔑 Key Points
-
-### What's Already Working
-✅ FastAPI infrastructure (50+ endpoints)
-✅ WebSocket support and broadcasting
-✅ Provider management system with failover
-✅ Database schema and migrations
-✅ Error handling and logging
-✅ Docker containerization
-
-### What Needs to Be Done
-❌ Replace mock data with real API calls
-❌ Connect database to API (persistence)
-❌ Load real HuggingFace ML models
-❌ Add JWT authentication
-❌ Implement rate limiting
-❌ Activate background tasks
-❌ Deploy to HuggingFace Spaces
-
-### Estimated Breakdown
-- **Phase 1 (Data):** 3-4 days
-- **Phase 2 (Database):** 2-3 days
-- **Phase 3 (Sentiment):** 1-2 days
-- **Phase 4 (Security):** 1-2 days
-- **Phase 5 (Operations):** 1 day
-- **Phase 6 (Deployment):** 2-3 days
-- **Testing & Optimization:** 2-3 days
-- **Total:** 14-20 days (2-3 weeks)
-
----
-
-## ✅ Success Looks Like
-
-### After Phase 1 (Week 1)
-✅ /api/market returns real BTC/ETH prices
-✅ /api/prices returns live data
-✅ /api/trending shows real trending coins
-✅ /api/ohlcv has historical candlestick data
-✅ All endpoints have caching
-✅ Response times < 1 second
-
-### After Phase 2-3 (Week 2)
-✅ Database storing 30+ days of history
-✅ Sentiment analysis using real ML models
-✅ News articles analyzed for sentiment
-✅ WebSocket broadcasting real updates
-✅ All data persisted across restarts
-
-### After Phase 4-5 (Week 3)
-✅ JWT authentication required on protected endpoints
-✅ Rate limiting enforced (Free/Pro tiers)
-✅ Health check showing all systems OK
-✅ Diagnostics finding and fixing issues
-✅ Ready for HuggingFace deployment
-
-### Final (Deployed)
-✅ Running on HuggingFace Spaces
-✅ All endpoints returning real data
-✅ Zero downtime in first month
-✅ All rate limits enforced
-✅ Sentiment analysis working
-✅ Database backup automated
-
----
-
-## 🆘 Help & Troubleshooting
-
-### Questions About Requirements?
-→ Check `DEPLOYMENT_MASTER_GUIDE.md` (Overview section)
-
-### Need Step-by-Step Timeline?
-→ Follow `IMPLEMENTATION_ROADMAP.md` (Day-by-day plan)
-
-### Looking for Code Examples?
-→ See `HUGGINGFACE_DEPLOYMENT_PROMPT.md` (Phases 1-5)
-
-### Need Quick Commands?
-→ Use `QUICK_REFERENCE_GUIDE.md` (Commands section)
-
-### Troubleshooting an Issue?
-→ Check `QUICK_REFERENCE_GUIDE.md` (Issues & Solutions)
-
-### Something Not Clear?
-→ Review relevant section in all guides, ask for clarification
-
----
-
-## 🎬 Next Step
-
-Choose your path above and get started:
-
-**Recommended:** Read `DEPLOYMENT_MASTER_GUIDE.md` right now (15 minutes). It will give you complete clarity on what's happening and why.
-
----
-
-## 📞 Quick Reference
-
-| Need | Document | Section |
-|------|----------|---------|
-| Big picture | DEPLOYMENT_MASTER_GUIDE.md | Overview |
-| Daily plan | IMPLEMENTATION_ROADMAP.md | Week 1-3 |
-| Code examples | HUGGINGFACE_DEPLOYMENT_PROMPT.md | Phases 1-5 |
-| Quick lookup | QUICK_REFERENCE_GUIDE.md | All sections |
-| Decisions | DEPLOYMENT_MASTER_GUIDE.md | Decision Points |
-| Commands | QUICK_REFERENCE_GUIDE.md | Commands section |
-| Troubleshooting | QUICK_REFERENCE_GUIDE.md | Issues section |
-
----
-
-## 💡 Pro Tips
-
-1. **Start with the master guide** - Don't skip this, it saves time overall
-2. **Follow the timeline** - It's designed for realistic progression
-3. **Test incrementally** - Don't wait until Phase 6 to test
-4. **Commit frequently** - Track your progress with git
-5. **Monitor resources** - Watch CPU/memory during implementation
-6. **Ask questions** - All documentation is comprehensive
-7. **Have fun!** - This is a cool project 🚀
-
----
-
-## 📊 Overview of Documents
-
-```
-DEPLOYMENT_MASTER_GUIDE.md (This explains everything)
-├── What you're building
-├── Current status (✅ vs ❌)
-├── Quick start paths
-├── Success metrics
-├── Decision points
-└── Next steps
-
-IMPLEMENTATION_ROADMAP.md (This is your timeline)
-├── Week 1: Data integration
-├── Week 2: Database & sentiment
-├── Week 3: Security & deployment
-├── Testing protocols
-├── Performance targets
-└── Success criteria per phase
-
-HUGGINGFACE_DEPLOYMENT_PROMPT.md (This is the reference)
-├── Phase 1: Market data with code
-├── Phase 2: Database integration with patterns
-├── Phase 3: AI models with examples
-├── Phase 4: Security with implementation
-├── Phase 5: Background tasks
-├── Phase 6: HF deployment
-└── Troubleshooting guide
-
-QUICK_REFERENCE_GUIDE.md (This is for quick lookup)
-├── Essential commands
-├── Key files locations
-├── Common issues & fixes
-├── Debugging tips
-├── Monitoring commands
-└── Configuration reference
-```
-
----
-
-## ✨ You're Ready!
-
-Everything you need is documented. The code is in place. The timeline is realistic. The patterns are clear.
-
-**Time to start:** Now! 🚀
-
-Begin with `DEPLOYMENT_MASTER_GUIDE.md` →
-
----
-
-**Document:** START_HERE.md
-**Version:** 1.0
-**Date:** November 15, 2025
-**Status:** ✅ Ready to Execute
-**Duration:** 2-3 weeks to complete
-**Support:** 100% documented
diff --git a/app/TEST_COMMANDS.sh b/app/TEST_COMMANDS.sh
deleted file mode 100644
index 02de34a9585e97104923277a7e4df0f8e04ceb03..0000000000000000000000000000000000000000
--- a/app/TEST_COMMANDS.sh
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/bin/bash
-# Test commands for verifying the deployed application
-# Run these commands after starting the server
-
-BASE_URL="${BASE_URL:-http://localhost:7860}"
-
-echo "Testing Crypto Monitor API Endpoints"
-echo "====================================="
-echo ""
-
-echo "1. Health Check:"
-curl -s "$BASE_URL/health" | jq
-echo ""
-
-echo "2. Market Data (Real CoinGecko):"
-curl -s "$BASE_URL/api/market" | jq '.cryptocurrencies[0] | {name, symbol, price, provider: .provider}'
-echo ""
-
-echo "3. Sentiment (Real Alternative.me):"
-curl -s "$BASE_URL/api/sentiment" | jq '.fear_greed_index'
-echo ""
-
-echo "4. Trending (Real CoinGecko):"
-curl -s "$BASE_URL/api/trending" | jq '.trending[0:3] | .[] | {name, symbol}'
-echo ""
-
-echo "5. Market History (Database):"
-curl -s "$BASE_URL/api/market/history?symbol=BTC&limit=5" | jq
-echo ""
-
-echo "6. DeFi Endpoint (Should return 503):"
-curl -s -w "\nHTTP Status: %{http_code}\n" "$BASE_URL/api/defi" | jq
-echo ""
-
-echo "7. HF Sentiment (Should return 501):"
-curl -s -w "\nHTTP Status: %{http_code}\n" -X POST "$BASE_URL/api/hf/run-sentiment" \
- -H "Content-Type: application/json" \
- -d '{"texts": ["Bitcoin is bullish"]}' | jq
-echo ""
-
-echo "All tests completed!"
diff --git a/app/TREE_STRUCTURE.txt b/app/TREE_STRUCTURE.txt
deleted file mode 100644
index 42dc4e37b776d5c744cda65a54a32b17964f968b..0000000000000000000000000000000000000000
--- a/app/TREE_STRUCTURE.txt
+++ /dev/null
@@ -1,159 +0,0 @@
-🌳 ساختار درختی پروژه Crypto Monitor
-═══════════════════════════════════════════════════════════════
-
-crypto-monitor-hf-full-fixed-v4-realapis/
-│
-├─ 📄 سرور اصلی (فقط این را اجرا کنید!)
-│ └─ ✅ api_server_extended.py
-│
-├─ 📦 فایلهای پیکربندی (Config Files)
-│ ├─ ✅ providers_config_extended.json ← ProviderManager
-│ ├─ ✅ providers_config_ultimate.json ← ResourceManager
-│ ├─ ✅ crypto_resources_unified_2025-11-11.json ← UnifiedConfigLoader
-│ ├─ ✅ all_apis_merged_2025.json ← UnifiedConfigLoader
-│ └─ ✅ ultimate_crypto_pipeline_2025_NZasinich.json ← UnifiedConfigLoader
-│
-├─ 🎨 رابط کاربری (Frontend)
-│ ├─ ✅ unified_dashboard.html ← داشبورد اصلی
-│ ├─ ✅ static/
-│ │ ├─ css/
-│ │ │ └─ connection-status.css
-│ │ └─ js/
-│ │ └─ websocket-client.js
-│ └─ ⚠️ index.html, dashboard.html, ... (قدیمی)
-│
-├─ 🔧 ماژولهای اصلی (Core)
-│ ├─ ✅ provider_manager.py ← مدیریت Providerها
-│ ├─ ✅ resource_manager.py ← مدیریت منابع
-│ └─ ✅ log_manager.py ← مدیریت لاگها
-│
-├─ 🛠️ سرویسهای بکند (Backend Services)
-│ └─ backend/
-│ └─ services/
-│ ├─ ✅ auto_discovery_service.py ← جستجوی خودکار
-│ ├─ ✅ connection_manager.py ← مدیریت WebSocket
-│ ├─ ✅ diagnostics_service.py ← اشکالیابی
-│ ├─ ✅ unified_config_loader.py ← بارگذاری یکپارچه
-│ ├─ ✅ scheduler_service.py ← زمانبندی
-│ ├─ ✅ persistence_service.py ← ذخیرهسازی
-│ ├─ ✅ websocket_service.py ← سرویس WebSocket
-│ ├─ ✅ ws_service_manager.py ← مدیریت WS
-│ ├─ ✅ hf_client.py ← کلاینت HuggingFace
-│ └─ ✅ hf_registry.py ← رجیستری مدلها
-│
-├─ 📡 API Routers
-│ └─ backend/routers/
-│ ├─ ✅ integrated_api.py
-│ └─ ✅ hf_connect.py
-│
-├─ 📁 دادهها و لاگها
-│ ├─ data/ ← ذخیره دادهها
-│ └─ logs/ ← ذخیره لاگها
-│
-├─ 🧪 تستها
-│ ├─ ✅ test_websocket.html
-│ └─ ✅ test_websocket_dashboard.html
-│
-└─ 📚 مستندات
- ├─ ✅ PROJECT_STRUCTURE_FA.md ← این فایل!
- ├─ ✅ QUICK_REFERENCE_FA.md ← مرجع سریع
- ├─ ✅ README.md
- ├─ ✅ WEBSOCKET_GUIDE.md
- └─ ... (سایر مستندات)
-
-═══════════════════════════════════════════════════════════════
-
-🔗 جریان داده (Data Flow)
-═══════════════════════════════════════════════════════════════
-
-Startup:
- api_server_extended.py
- │
- ├─→ ProviderManager
- │ └─→ providers_config_extended.json
- │
- ├─→ ResourceManager
- │ └─→ providers_config_ultimate.json
- │
- └─→ UnifiedConfigLoader
- ├─→ crypto_resources_unified_2025-11-11.json
- ├─→ all_apis_merged_2025.json
- └─→ ultimate_crypto_pipeline_2025_NZasinich.json
-
-Runtime:
- Client Request
- │
- ├─→ ProviderManager.get_provider()
- ├─→ ProviderPool.get_data()
- └─→ Response
-
-WebSocket:
- Client Connect
- │
- └─→ ConnectionManager
- ├─→ Track Session
- ├─→ Broadcast Updates
- └─→ Heartbeat
-
-Auto-Discovery:
- Scheduled Task
- │
- └─→ AutoDiscoveryService
- ├─→ Search (DuckDuckGo)
- ├─→ Analyze (HuggingFace)
- └─→ Add to ResourceManager
-
-═══════════════════════════════════════════════════════════════
-
-📊 جدول فایلهای Config
-═══════════════════════════════════════════════════════════════
-
-┌─────────────────────────────────────┬──────────────────────┬─────────────┐
-│ فایل Config │ استفاده شده توسط │ تعداد API │
-├─────────────────────────────────────┼──────────────────────┼─────────────┤
-│ providers_config_extended.json │ ProviderManager │ ~100 │
-│ providers_config_ultimate.json │ ResourceManager │ ~200 │
-│ crypto_resources_unified_2025-... │ UnifiedConfigLoader │ 200+ │
-│ all_apis_merged_2025.json │ UnifiedConfigLoader │ متغیر │
-│ ultimate_crypto_pipeline_2025... │ UnifiedConfigLoader │ متغیر │
-└─────────────────────────────────────┴──────────────────────┴─────────────┘
-
-═══════════════════════════════════════════════════════════════
-
-🎯 خلاصه: کدام فایل برای چه کاری؟
-═══════════════════════════════════════════════════════════════
-
-✅ برای اجرای برنامه:
- → python api_server_extended.py
-
-✅ برای ویرایش Providerها:
- → providers_config_extended.json (ProviderManager)
- → providers_config_ultimate.json (ResourceManager)
-
-✅ برای مشاهده داشبورد:
- → unified_dashboard.html
-
-✅ برای اضافه کردن Provider جدید:
- → استفاده از API: POST /api/resources
- → یا ویرایش مستقیم فایلهای Config
-
-═══════════════════════════════════════════════════════════════
-
-⚠️ فایلهای قدیمی (استفاده نمیشوند - میتوانید حذف کنید)
-═══════════════════════════════════════════════════════════════
-
-❌ main.py
-❌ app.py
-❌ enhanced_server.py
-❌ production_server.py
-❌ real_server.py
-❌ simple_server.py
-❌ index.html
-❌ dashboard.html
-❌ enhanced_dashboard.html
-❌ admin.html
-❌ config.py
-❌ scheduler.py
-
-═══════════════════════════════════════════════════════════════
-
diff --git a/app/VIEW_IMPROVED_DASHBOARD.txt b/app/VIEW_IMPROVED_DASHBOARD.txt
deleted file mode 100644
index 159e188c2297e4515263adc0eca727b0ce3fa9a7..0000000000000000000000000000000000000000
--- a/app/VIEW_IMPROVED_DASHBOARD.txt
+++ /dev/null
@@ -1,65 +0,0 @@
-========================================
- 🎨 IMPROVED DASHBOARD IS READY!
-========================================
-
-📍 Open this URL in your browser:
-
- http://localhost:7860/improved
-
-========================================
-
-✨ What you'll see:
-
-✅ Clean, Modern Design
- - Beautiful gradient background
- - Professional card layout
- - Smooth animations
-
-✅ Complete Overview
- - 6 big statistics cards at top
- - Total providers, online, offline, degraded
- - Overall uptime percentage
- - Total categories
-
-✅ All Providers Grid
- - Every provider shown as a card
- - Color-coded by status:
- * Green = Online
- * Orange = Degraded
- * Red = Offline
- - Shows response time
- - Shows category
-
-✅ Categories Breakdown
- - All categories listed
- - Online/Degraded/Offline count per category
- - Easy to see which data types are working
-
-✅ Interactive Chart
- - Beautiful pie chart
- - Shows status distribution
- - Visual representation
-
-✅ Auto-Refresh
- - Updates every 30 seconds automatically
- - Manual refresh button available
- - Real-time data
-
-========================================
-
-🌐 AVAILABLE DASHBOARDS:
-
-Main (current): http://localhost:7860/
-Improved (new): http://localhost:7860/improved
-Unified: http://localhost:7860/unified
-Admin: http://localhost:7860/admin.html
-Pools: http://localhost:7860/pool_management.html
-
-========================================
-
-💡 The improved dashboard gives you THE COMPLETE
- PICTURE of your entire crypto monitoring system
- in ONE SCREEN!
-
-========================================
-
diff --git a/app/__init__.py b/app/__init__.py
deleted file mode 100644
index eb525d64ff5a9ac2a5d73cb9478c38efcff1dbf0..0000000000000000000000000000000000000000
--- a/app/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-# Cryptocurrency Data Source API
-# Version 1.0.0
diff --git a/app/__pycache__/api_loader.cpython-313.pyc b/app/__pycache__/api_loader.cpython-313.pyc
deleted file mode 100644
index e97a5c742964371cc6edfb9a789eb2f8fdff1b3e..0000000000000000000000000000000000000000
Binary files a/app/__pycache__/api_loader.cpython-313.pyc and /dev/null differ
diff --git a/app/__pycache__/app.cpython-313.pyc b/app/__pycache__/app.cpython-313.pyc
deleted file mode 100644
index 1db4d301161e582a1fcb151282722f9bb8b78708..0000000000000000000000000000000000000000
Binary files a/app/__pycache__/app.cpython-313.pyc and /dev/null differ
diff --git a/app/__pycache__/config.cpython-313.pyc b/app/__pycache__/config.cpython-313.pyc
deleted file mode 100644
index 37de9b0ef1ef7d83a4a57f4a03b7714a34b8d7d7..0000000000000000000000000000000000000000
Binary files a/app/__pycache__/config.cpython-313.pyc and /dev/null differ
diff --git a/app/__pycache__/database.cpython-313.pyc b/app/__pycache__/database.cpython-313.pyc
deleted file mode 100644
index 550b64759c6cbd99aaca2807acff59e3e450e08f..0000000000000000000000000000000000000000
Binary files a/app/__pycache__/database.cpython-313.pyc and /dev/null differ
diff --git a/app/__pycache__/monitor.cpython-313.pyc b/app/__pycache__/monitor.cpython-313.pyc
deleted file mode 100644
index b3309cd26dd227b191a346059496f02731c23679..0000000000000000000000000000000000000000
Binary files a/app/__pycache__/monitor.cpython-313.pyc and /dev/null differ
diff --git a/app/admin.html b/app/admin.html
deleted file mode 100644
index be23b8459666032c18a60475cdbe94472825a8e1..0000000000000000000000000000000000000000
--- a/app/admin.html
+++ /dev/null
@@ -1,1017 +0,0 @@
-
-
-
-
-
- Admin Dashboard - Crypto Monitor
-
-
-
-
-
-
🚀 Crypto Monitor Admin Dashboard
-
Real-time provider management & system monitoring | NO MOCK DATA
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
System Health
-
-
- Healthy
-
-
-
Total Providers
-
-
-
-
-
Validated
-
-
-
-
-
Database
-
✓
- Connected
-
-
-
-
-
Quick Actions
-
-
-
-
-
-
-
Recent Market Data
-
-
-
-
-
-
-
-
Providers Management
-
-
-
-
-
-
-
-
-
-
-
-
Live Market Data
-
-
-
-
-
-
Sentiment Analysis
-
-
-
-
-
Trending Coins
-
-
-
-
-
-
-
-
Auto Provider Loader (APL)
-
- APL automatically discovers, validates, and integrates cryptocurrency data providers.
- All validations use REAL API calls - NO MOCK DATA.
-
-
-
-
-
-
-
-
-
-
APL Summary Statistics
-
-
-
-
-
APL Output
-
No output yet. Click "Run APL Scan" to start.
-
-
-
-
-
-
-
Hugging Face Models
-
- HuggingFace models validated by APL for crypto sentiment analysis and NLP tasks.
-
"
-
-
-def export_current_status():
- """Export current status to CSV"""
- global current_results
-
- if not current_results:
- return None
-
- try:
- timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
- filename = f"api_status_{timestamp}.csv"
- filepath = f"data/{filename}"
-
- df_data = []
- for result in current_results:
- df_data.append({
- 'Provider': result.provider_name,
- 'Category': result.category,
- 'Status': result.status.value,
- 'Response_Time_ms': result.response_time,
- 'Status_Code': result.status_code,
- 'Error': result.error_message or '',
- 'Timestamp': datetime.fromtimestamp(result.timestamp).isoformat()
- })
-
- df = pd.DataFrame(df_data)
- df.to_csv(filepath, index=False)
-
- return filepath
-
- except Exception as e:
- logger.error(f"Error exporting: {e}")
- return None
-
-
-# =============================================================================
-# TAB 2: Category View
-# =============================================================================
-
-def get_category_overview():
- """Get overview of all categories"""
- global current_results
-
- if not current_results:
- return "No data available. Please refresh the dashboard first."
-
- category_stats = monitor.get_category_stats(current_results)
-
- html_output = "
"
-
- for category, stats in category_stats.items():
- online_pct = stats['online_percentage']
-
- # Color based on health
- if online_pct >= 80:
- color = "#4CAF50"
- elif online_pct >= 50:
- color = "#FF9800"
- else:
- color = "#F44336"
-
- html_output += f"""
-
-
📁 {category}
-
-
- Total: {stats['total']}
-
-
- 🟢 Online: {stats['online']}
-
-
- 🟡 Degraded: {stats['degraded']}
-
-
- 🔴 Offline: {stats['offline']}
-
-
- Availability: {online_pct}%
-
-
- Avg Response: {stats['avg_response_time']:.0f} ms
-
-
-
-
- {online_pct}%
-
-
-
- """
-
- html_output += "
"
-
- return html_output
-
-
-def get_category_chart():
- """Create category availability chart"""
- global current_results
-
- if not current_results:
- return go.Figure()
-
- category_stats = monitor.get_category_stats(current_results)
-
- categories = list(category_stats.keys())
- online_pcts = [stats['online_percentage'] for stats in category_stats.values()]
- avg_times = [stats['avg_response_time'] for stats in category_stats.values()]
-
- fig = go.Figure()
-
- fig.add_trace(go.Bar(
- name='Availability %',
- x=categories,
- y=online_pcts,
- marker_color='lightblue',
- text=[f"{pct:.1f}%" for pct in online_pcts],
- textposition='auto',
- yaxis='y1'
- ))
-
- fig.add_trace(go.Scatter(
- name='Avg Response Time (ms)',
- x=categories,
- y=avg_times,
- mode='lines+markers',
- marker=dict(size=10, color='red'),
- line=dict(width=2, color='red'),
- yaxis='y2'
- ))
-
- fig.update_layout(
- title='Category Health Overview',
- xaxis=dict(title='Category'),
- yaxis=dict(title='Availability %', side='left', range=[0, 100]),
- yaxis2=dict(title='Response Time (ms)', side='right', overlaying='y'),
- hovermode='x unified',
- template='plotly_white',
- height=500
- )
-
- return fig
-
-
-# =============================================================================
-# TAB 3: Health History
-# =============================================================================
-
-def get_uptime_chart(provider_name=None, hours=24):
- """Get uptime chart for provider(s)"""
- try:
- # Get data from database
- status_data = db.get_recent_status(provider_name=provider_name, hours=hours)
-
- if not status_data:
- fig = go.Figure()
- fig.add_annotation(
- text="No historical data available. Data will accumulate over time.",
- xref="paper", yref="paper",
- x=0.5, y=0.5, showarrow=False,
- font=dict(size=16)
- )
- return fig
-
- # Convert to DataFrame
- df = pd.DataFrame(status_data)
- df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')
- df['uptime_value'] = df['status'].apply(lambda x: 100 if x == 'online' else 0)
-
- # Group by provider and time
- if provider_name:
- providers = [provider_name]
- else:
- providers = df['provider_name'].unique()[:10] # Limit to 10 providers
-
- fig = go.Figure()
-
- for provider in providers:
- provider_df = df[df['provider_name'] == provider]
-
- # Resample to hourly average
- provider_df = provider_df.set_index('timestamp')
- resampled = provider_df['uptime_value'].resample('1H').mean()
-
- fig.add_trace(go.Scatter(
- name=provider,
- x=resampled.index,
- y=resampled.values,
- mode='lines+markers',
- line=dict(width=2),
- marker=dict(size=6)
- ))
-
- fig.update_layout(
- title=f'Uptime History - Last {hours} Hours',
- xaxis_title='Time',
- yaxis_title='Uptime %',
- hovermode='x unified',
- template='plotly_white',
- height=500,
- yaxis=dict(range=[0, 105])
- )
-
- return fig
-
- except Exception as e:
- logger.error(f"Error creating uptime chart: {e}")
- fig = go.Figure()
- fig.add_annotation(
- text=f"Error: {str(e)}",
- xref="paper", yref="paper",
- x=0.5, y=0.5, showarrow=False
- )
- return fig
-
-
-def get_response_time_chart(provider_name=None, hours=24):
- """Get response time trends"""
- try:
- status_data = db.get_recent_status(provider_name=provider_name, hours=hours)
-
- if not status_data:
- return go.Figure()
-
- df = pd.DataFrame(status_data)
- df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')
-
- if provider_name:
- providers = [provider_name]
- else:
- providers = df['provider_name'].unique()[:10]
-
- fig = go.Figure()
-
- for provider in providers:
- provider_df = df[df['provider_name'] == provider]
-
- fig.add_trace(go.Scatter(
- name=provider,
- x=provider_df['timestamp'],
- y=provider_df['response_time'],
- mode='lines',
- line=dict(width=2)
- ))
-
- fig.update_layout(
- title=f'Response Time Trends - Last {hours} Hours',
- xaxis_title='Time',
- yaxis_title='Response Time (ms)',
- hovermode='x unified',
- template='plotly_white',
- height=500
- )
-
- return fig
-
- except Exception as e:
- logger.error(f"Error creating response time chart: {e}")
- return go.Figure()
-
-
-def get_incident_log(hours=24):
- """Get incident log"""
- try:
- incidents = db.get_incident_history(hours=hours)
-
- if not incidents:
- return pd.DataFrame({'Message': ['No incidents in the selected period']})
-
- df_data = []
- for incident in incidents:
- df_data.append({
- 'Timestamp': incident['start_time'],
- 'Provider': incident['provider_name'],
- 'Category': incident['category'],
- 'Type': incident['incident_type'],
- 'Severity': incident['severity'],
- 'Description': incident['description'],
- 'Duration': f"{incident.get('duration_seconds', 0)} sec" if incident.get('resolved') else 'Ongoing',
- 'Status': '✅ Resolved' if incident.get('resolved') else '⚠️ Active'
- })
-
- return pd.DataFrame(df_data)
-
- except Exception as e:
- logger.error(f"Error getting incident log: {e}")
- return pd.DataFrame({'Error': [str(e)]})
-
-
-# =============================================================================
-# TAB 4: Test Endpoint
-# =============================================================================
-
-def test_endpoint(provider_name, custom_endpoint="", use_proxy=False):
- """Test a specific endpoint"""
- try:
- resources = config.get_all_resources()
- resource = next((r for r in resources if r['name'] == provider_name), None)
-
- if not resource:
- return "Provider not found", ""
-
- # Override endpoint if provided
- if custom_endpoint:
- resource = resource.copy()
- resource['endpoint'] = custom_endpoint
-
- # Run check
- result = asyncio.run(monitor.check_endpoint(resource, use_proxy=use_proxy))
-
- # Format response
- status_emoji = result.get_badge()
- status_text = f"""
-## Test Results
-
-**Provider:** {result.provider_name}
-**Status:** {status_emoji} {result.status.value.upper()}
-**Response Time:** {result.response_time:.2f} ms
-**Status Code:** {result.status_code or 'N/A'}
-**Endpoint:** `{result.endpoint_tested}`
-
-### Details
-"""
-
- if result.error_message:
- status_text += f"\n**Error:** {result.error_message}\n"
- else:
- status_text += "\n✅ Request successful\n"
-
- # Troubleshooting hints
- if result.status != HealthStatus.ONLINE:
- status_text += "\n### Troubleshooting Hints\n"
- if result.status_code == 403:
- status_text += "- Check API key validity\n- Verify rate limits\n- Try using CORS proxy\n"
- elif result.status_code == 429:
- status_text += "- Rate limit exceeded\n- Wait before retrying\n- Consider using backup provider\n"
- elif result.error_message and "timeout" in result.error_message.lower():
- status_text += "- Connection timeout\n- Service may be slow or down\n- Try increasing timeout\n"
- else:
- status_text += "- Verify endpoint URL\n- Check network connectivity\n- Review API documentation\n"
-
- return status_text, json.dumps(result.to_dict(), indent=2)
-
- except Exception as e:
- return f"Error testing endpoint: {str(e)}", ""
-
-
-def get_example_query(provider_name):
- """Get example query for a provider"""
- resources = config.get_all_resources()
- resource = next((r for r in resources if r['name'] == provider_name), None)
-
- if not resource:
- return ""
-
- example = resource.get('example', '')
- if example:
- return f"Example:\n{example}"
-
- # Generate generic example based on endpoint
- endpoint = resource.get('endpoint', '')
- url = resource.get('url', '')
-
- if endpoint:
- return f"Example URL:\n{url}{endpoint}"
-
- return f"Base URL:\n{url}"
-
-
-# =============================================================================
-# TAB 5: Configuration
-# =============================================================================
-
-def update_refresh_interval(interval_minutes):
- """Update background refresh interval"""
- try:
- scheduler.update_interval(interval_minutes)
- return f"✅ Refresh interval updated to {interval_minutes} minutes"
- except Exception as e:
- return f"❌ Error: {str(e)}"
-
-
-def clear_all_cache():
- """Clear all caches"""
- try:
- monitor.clear_cache()
- return "✅ Cache cleared successfully"
- except Exception as e:
- return f"❌ Error: {str(e)}"
-
-
-def get_config_info():
- """Get configuration information"""
- stats = config.stats()
-
- info = f"""
-## Configuration Overview
-
-**Total API Resources:** {stats['total_resources']}
-**Categories:** {stats['total_categories']}
-**Free Resources:** {stats['free_resources']}
-**Tier 1 (Critical):** {stats['tier1_count']}
-**Tier 2 (Important):** {stats['tier2_count']}
-**Tier 3 (Others):** {stats['tier3_count']}
-**Configured API Keys:** {stats['api_keys_count']}
-**CORS Proxies:** {stats['cors_proxies_count']}
-
-### Categories
-{', '.join(stats['categories'])}
-
-### Scheduler Status
-**Running:** {scheduler.is_running()}
-**Interval:** {scheduler.interval_minutes} minutes
-**Last Run:** {scheduler.last_run_time.strftime('%Y-%m-%d %H:%M:%S') if scheduler.last_run_time else 'Never'}
-"""
-
- return info
-
-
-# =============================================================================
-# Build Gradio Interface
-# =============================================================================
-
-def build_interface():
- """Build the complete Gradio interface"""
-
- with gr.Blocks(
- theme=gr.themes.Soft(primary_hue="purple", secondary_hue="blue"),
- title="Crypto API Monitor",
- css="""
- .gradio-container {
- max-width: 1400px !important;
- }
- """
- ) as app:
-
- gr.Markdown("""
- # 📊 Cryptocurrency API Monitor
- ### Real-time health monitoring for 162+ crypto API endpoints
- *Production-ready | Auto-refreshing | Persistent metrics | Multi-tier monitoring*
- """)
-
- # TAB 1: Real-Time Dashboard
- with gr.Tab("📊 Real-Time Dashboard"):
- with gr.Row():
- refresh_btn = gr.Button("🔄 Refresh Now", variant="primary", size="lg")
- export_btn = gr.Button("💾 Export CSV", size="lg")
-
- with gr.Row():
- category_filter = gr.Dropdown(
- choices=["All"] + config.get_categories(),
- value="All",
- label="Filter by Category"
- )
- status_filter = gr.Dropdown(
- choices=["All", "Online", "Degraded", "Offline"],
- value="All",
- label="Filter by Status"
- )
- tier_filter = gr.Dropdown(
- choices=["All", "Tier 1", "Tier 2", "Tier 3"],
- value="All",
- label="Filter by Tier"
- )
-
- summary_cards = gr.HTML()
- status_table = gr.DataFrame(
- headers=["Status", "Provider", "Category", "Response Time", "Last Check", "Code"],
- wrap=True
- )
- download_file = gr.File(label="Download CSV", visible=False)
-
- refresh_btn.click(
- fn=refresh_dashboard,
- inputs=[category_filter, status_filter, tier_filter],
- outputs=[status_table, summary_cards]
- )
-
- export_btn.click(
- fn=export_current_status,
- outputs=download_file
- )
-
- # TAB 2: Category View
- with gr.Tab("📁 Category View"):
- gr.Markdown("### API Resources by Category")
-
- with gr.Row():
- refresh_cat_btn = gr.Button("🔄 Refresh Categories", variant="primary")
-
- category_overview = gr.HTML()
- category_chart = gr.Plot()
-
- refresh_cat_btn.click(
- fn=get_category_overview,
- outputs=category_overview
- )
-
- refresh_cat_btn.click(
- fn=get_category_chart,
- outputs=category_chart
- )
-
- # TAB 3: Health History
- with gr.Tab("📈 Health History"):
- gr.Markdown("### Historical Performance & Incidents")
-
- with gr.Row():
- history_provider = gr.Dropdown(
- choices=["All"] + [r['name'] for r in config.get_all_resources()],
- value="All",
- label="Select Provider"
- )
- history_hours = gr.Slider(
- minimum=1,
- maximum=168,
- value=24,
- step=1,
- label="Time Range (hours)"
- )
- refresh_history_btn = gr.Button("🔄 Refresh", variant="primary")
-
- uptime_chart = gr.Plot(label="Uptime History")
- response_chart = gr.Plot(label="Response Time Trends")
- incident_table = gr.DataFrame(label="Incident Log")
-
- def update_history(provider, hours):
- prov = None if provider == "All" else provider
- uptime = get_uptime_chart(prov, hours)
- response = get_response_time_chart(prov, hours)
- incidents = get_incident_log(hours)
- return uptime, response, incidents
-
- refresh_history_btn.click(
- fn=update_history,
- inputs=[history_provider, history_hours],
- outputs=[uptime_chart, response_chart, incident_table]
- )
-
- # TAB 4: Test Endpoint
- with gr.Tab("🔧 Test Endpoint"):
- gr.Markdown("### Test Individual API Endpoints")
-
- with gr.Row():
- test_provider = gr.Dropdown(
- choices=[r['name'] for r in config.get_all_resources()],
- label="Select Provider"
- )
- test_btn = gr.Button("▶️ Run Test", variant="primary")
-
- with gr.Row():
- custom_endpoint = gr.Textbox(
- label="Custom Endpoint (optional)",
- placeholder="/api/endpoint"
- )
- use_proxy_check = gr.Checkbox(label="Use CORS Proxy", value=False)
-
- example_query = gr.Markdown()
- test_result = gr.Markdown()
- test_json = gr.Code(label="JSON Response", language="json")
-
- test_provider.change(
- fn=get_example_query,
- inputs=test_provider,
- outputs=example_query
- )
-
- test_btn.click(
- fn=test_endpoint,
- inputs=[test_provider, custom_endpoint, use_proxy_check],
- outputs=[test_result, test_json]
- )
-
- # TAB 5: Configuration
- with gr.Tab("⚙️ Configuration"):
- gr.Markdown("### System Configuration & Settings")
-
- config_info = gr.Markdown()
-
- with gr.Row():
- refresh_interval = gr.Slider(
- minimum=1,
- maximum=60,
- value=5,
- step=1,
- label="Auto-refresh Interval (minutes)"
- )
- update_interval_btn = gr.Button("💾 Update Interval")
-
- interval_status = gr.Textbox(label="Status", interactive=False)
-
- with gr.Row():
- clear_cache_btn = gr.Button("🗑️ Clear Cache")
- cache_status = gr.Textbox(label="Cache Status", interactive=False)
-
- gr.Markdown("### API Keys Management")
- gr.Markdown("""
- API keys are loaded from environment variables in Hugging Face Spaces.
- Go to **Settings > Repository secrets** to add keys:
- - `ETHERSCAN_KEY`
- - `BSCSCAN_KEY`
- - `TRONSCAN_KEY`
- - `CMC_KEY` (CoinMarketCap)
- - `CRYPTOCOMPARE_KEY`
- """)
-
- # Load config info on tab open
- app.load(fn=get_config_info, outputs=config_info)
-
- update_interval_btn.click(
- fn=update_refresh_interval,
- inputs=refresh_interval,
- outputs=interval_status
- )
-
- clear_cache_btn.click(
- fn=clear_all_cache,
- outputs=cache_status
- )
-
- # Initial load
- app.load(
- fn=refresh_dashboard,
- inputs=[category_filter, status_filter, tier_filter],
- outputs=[status_table, summary_cards]
- )
-
- return app
-
-
-# =============================================================================
-# Main Entry Point
-# =============================================================================
-
-if __name__ == "__main__":
- logger.info("Starting Crypto API Monitor...")
-
- # Start background scheduler
- scheduler.start()
-
- # Build and launch app
- app = build_interface()
-
- # Launch with sharing for HF Spaces
- app.launch(
- server_name="0.0.0.0",
- server_port=7860,
- share=False,
- show_error=True
- )
diff --git a/app/auto_provider_loader.py b/app/auto_provider_loader.py
deleted file mode 100644
index cf049ff69cca9f64a3429e8bf678c6916d27fa84..0000000000000000000000000000000000000000
--- a/app/auto_provider_loader.py
+++ /dev/null
@@ -1,576 +0,0 @@
-#!/usr/bin/env python3
-"""
-Auto Provider Loader (APL) - REAL DATA ONLY
-Scans, validates, and integrates providers from JSON resources.
-NO MOCK DATA. NO FAKE RESPONSES.
-"""
-
-import asyncio
-import json
-import os
-from pathlib import Path
-from typing import Dict, List, Any, Optional
-from dataclasses import dataclass, asdict
-import time
-from datetime import datetime
-
-from provider_validator import ProviderValidator, ValidationResult, ValidationStatus
-
-
-@dataclass
-class APLStats:
- """APL execution statistics"""
- total_http_candidates: int = 0
- total_hf_candidates: int = 0
- http_valid: int = 0
- http_invalid: int = 0
- http_conditional: int = 0
- hf_valid: int = 0
- hf_invalid: int = 0
- hf_conditional: int = 0
- total_active_providers: int = 0
- execution_time_sec: float = 0.0
- timestamp: str = ""
-
- def __post_init__(self):
- if not self.timestamp:
- self.timestamp = datetime.now().isoformat()
-
-
-class AutoProviderLoader:
- """
- Auto Provider Loader (APL)
- Discovers, validates, and integrates providers automatically.
- """
-
- def __init__(self, workspace_root: str = "/workspace"):
- self.workspace_root = Path(workspace_root)
- self.validator = ProviderValidator(timeout=8.0)
- self.http_results: List[ValidationResult] = []
- self.hf_results: List[ValidationResult] = []
- self.stats = APLStats()
-
- def discover_http_providers(self) -> List[Dict[str, Any]]:
- """
- Discover HTTP providers from JSON resources.
- Returns list of (provider_id, provider_data, source_file) tuples.
- """
- providers = []
-
- # Scan api-resources directory
- api_resources = self.workspace_root / "api-resources"
- if api_resources.exists():
- for json_file in api_resources.glob("*.json"):
- try:
- with open(json_file, 'r') as f:
- data = json.load(f)
-
- # Check if it's the unified registry format
- if "registry" in data:
- registry = data["registry"]
-
- # Process each section
- for section_key, section_data in registry.items():
- if section_key == "metadata":
- continue
-
- if isinstance(section_data, list):
- for item in section_data:
- provider_id = item.get("id", f"{section_key}_{len(providers)}")
- providers.append({
- "id": provider_id,
- "data": item,
- "source": str(json_file.name),
- "section": section_key
- })
-
- # Check if it's a direct resources list
- elif "resources" in data:
- for idx, item in enumerate(data["resources"]):
- provider_id = item.get("id", f"resource_{idx}")
- if not provider_id or provider_id.startswith("resource_"):
- # Generate ID from name
- name = item.get("name", "").lower().replace(" ", "_")
- provider_id = f"{name}_{idx}" if name else f"resource_{idx}"
-
- providers.append({
- "id": provider_id,
- "data": {
- "name": item.get("name"),
- "category": item.get("category", "unknown"),
- "base_url": item.get("url"),
- "endpoint": item.get("endpoint"),
- "auth": {
- "type": "apiKey" if item.get("key") else "none",
- "key": item.get("key")
- },
- "free": item.get("free", True),
- "rate_limit": item.get("rateLimit"),
- "notes": item.get("desc") or item.get("notes")
- },
- "source": str(json_file.name),
- "section": "resources"
- })
-
- except Exception as e:
- print(f"Error loading {json_file}: {e}")
-
- # Scan providers_config files
- for config_file in self.workspace_root.glob("providers_config*.json"):
- try:
- with open(config_file, 'r') as f:
- data = json.load(f)
-
- if "providers" in data:
- for provider_id, provider_data in data["providers"].items():
- providers.append({
- "id": provider_id,
- "data": provider_data,
- "source": str(config_file.name),
- "section": "providers"
- })
-
- except Exception as e:
- print(f"Error loading {config_file}: {e}")
-
- return providers
-
- def discover_hf_models(self) -> List[Dict[str, Any]]:
- """
- Discover Hugging Face models from:
- 1. backend/services/hf_client.py (hardcoded models)
- 2. backend/services/hf_registry.py (dynamic discovery)
- 3. JSON resources (hf_resources section)
- """
- models = []
-
- # Hardcoded models from hf_client.py
- hardcoded_models = [
- {
- "id": "ElKulako/cryptobert",
- "name": "ElKulako CryptoBERT",
- "pipeline_tag": "sentiment-analysis",
- "source": "hf_client.py"
- },
- {
- "id": "kk08/CryptoBERT",
- "name": "KK08 CryptoBERT",
- "pipeline_tag": "sentiment-analysis",
- "source": "hf_client.py"
- }
- ]
-
- for model in hardcoded_models:
- models.append(model)
-
- # Models from JSON resources
- api_resources = self.workspace_root / "api-resources"
- if api_resources.exists():
- for json_file in api_resources.glob("*.json"):
- try:
- with open(json_file, 'r') as f:
- data = json.load(f)
-
- if "registry" in data:
- hf_resources = data["registry"].get("hf_resources", [])
- for item in hf_resources:
- if item.get("type") == "model":
- models.append({
- "id": item.get("id", item.get("model_id")),
- "name": item.get("name"),
- "pipeline_tag": item.get("pipeline_tag", "sentiment-analysis"),
- "source": str(json_file.name)
- })
-
- except Exception as e:
- pass
-
- return models
-
- async def validate_all_http_providers(self, providers: List[Dict[str, Any]]) -> None:
- """
- Validate all HTTP providers in parallel batches.
- """
- print(f"\n🔍 Validating {len(providers)} HTTP provider candidates...")
-
- # Process in batches to avoid overwhelming
- batch_size = 10
- for i in range(0, len(providers), batch_size):
- batch = providers[i:i+batch_size]
-
- tasks = [
- self.validator.validate_http_provider(p["id"], p["data"])
- for p in batch
- ]
-
- results = await asyncio.gather(*tasks, return_exceptions=True)
-
- for j, result in enumerate(results):
- if isinstance(result, Exception):
- # Create error result
- p = batch[j]
- result = ValidationResult(
- provider_id=p["id"],
- provider_name=p["data"].get("name", p["id"]),
- provider_type="http_json",
- category=p["data"].get("category", "unknown"),
- status=ValidationStatus.INVALID.value,
- error_reason=f"Validation exception: {str(result)[:50]}"
- )
-
- self.http_results.append(result)
-
- # Print progress
- status_emoji = {
- ValidationStatus.VALID.value: "✅",
- ValidationStatus.INVALID.value: "❌",
- ValidationStatus.CONDITIONALLY_AVAILABLE.value: "⚠️",
- ValidationStatus.SKIPPED.value: "⏭️"
- }
-
- emoji = status_emoji.get(result.status, "❓")
- print(f" {emoji} {result.provider_id}: {result.status}")
-
- # Small delay between batches
- await asyncio.sleep(0.5)
-
- async def validate_all_hf_models(self, models: List[Dict[str, Any]]) -> None:
- """
- Validate all HF models sequentially (to avoid memory issues).
- """
- print(f"\n🤖 Validating {len(models)} HF model candidates...")
-
- for model in models:
- try:
- result = await self.validator.validate_hf_model(
- model["id"],
- model["name"],
- model.get("pipeline_tag", "sentiment-analysis")
- )
-
- self.hf_results.append(result)
-
- status_emoji = {
- ValidationStatus.VALID.value: "✅",
- ValidationStatus.INVALID.value: "❌",
- ValidationStatus.CONDITIONALLY_AVAILABLE.value: "⚠️"
- }
-
- emoji = status_emoji.get(result.status, "❓")
- print(f" {emoji} {result.provider_id}: {result.status}")
-
- except Exception as e:
- print(f" ❌ {model['id']}: Exception during validation: {str(e)[:50]}")
- self.hf_results.append(ValidationResult(
- provider_id=model["id"],
- provider_name=model["name"],
- provider_type="hf_model",
- category="hf_model",
- status=ValidationStatus.INVALID.value,
- error_reason=f"Validation exception: {str(e)[:50]}"
- ))
-
- def compute_stats(self) -> None:
- """Compute final statistics"""
- self.stats.total_http_candidates = len(self.http_results)
- self.stats.total_hf_candidates = len(self.hf_results)
-
- # Count HTTP results
- for result in self.http_results:
- if result.status == ValidationStatus.VALID.value:
- self.stats.http_valid += 1
- elif result.status == ValidationStatus.INVALID.value:
- self.stats.http_invalid += 1
- elif result.status == ValidationStatus.CONDITIONALLY_AVAILABLE.value:
- self.stats.http_conditional += 1
-
- # Count HF results
- for result in self.hf_results:
- if result.status == ValidationStatus.VALID.value:
- self.stats.hf_valid += 1
- elif result.status == ValidationStatus.INVALID.value:
- self.stats.hf_invalid += 1
- elif result.status == ValidationStatus.CONDITIONALLY_AVAILABLE.value:
- self.stats.hf_conditional += 1
-
- self.stats.total_active_providers = self.stats.http_valid + self.stats.hf_valid
-
- def integrate_valid_providers(self) -> Dict[str, Any]:
- """
- Integrate valid providers into providers_config_extended.json.
- Returns the updated config.
- """
- config_path = self.workspace_root / "providers_config_extended.json"
-
- # Load existing config
- if config_path.exists():
- with open(config_path, 'r') as f:
- config = json.load(f)
- else:
- config = {"providers": {}}
-
- # Backup
- backup_path = self.workspace_root / f"providers_config_extended.backup.{int(time.time())}.json"
- with open(backup_path, 'w') as f:
- json.dump(config, f, indent=2)
-
- print(f"\n📦 Backed up config to {backup_path.name}")
-
- # Add valid HTTP providers
- added_count = 0
- for result in self.http_results:
- if result.status == ValidationStatus.VALID.value:
- if result.provider_id not in config["providers"]:
- config["providers"][result.provider_id] = {
- "name": result.provider_name,
- "category": result.category,
- "type": result.provider_type,
- "validated": True,
- "validated_at": result.validated_at,
- "response_time_ms": result.response_time_ms,
- "added_by": "APL"
- }
- added_count += 1
-
- print(f"✅ Added {added_count} new valid HTTP providers to config")
-
- # Save updated config
- with open(config_path, 'w') as f:
- json.dump(config, f, indent=2)
-
- return config
-
- def generate_reports(self) -> None:
- """Generate comprehensive reports"""
- reports_dir = self.workspace_root
-
- # 1. Detailed validation report
- validation_report = {
- "report_type": "Provider Auto-Discovery Validation Report",
- "generated_at": datetime.now().isoformat(),
- "stats": asdict(self.stats),
- "http_providers": {
- "total_candidates": self.stats.total_http_candidates,
- "valid": self.stats.http_valid,
- "invalid": self.stats.http_invalid,
- "conditional": self.stats.http_conditional,
- "results": [asdict(r) for r in self.http_results]
- },
- "hf_models": {
- "total_candidates": self.stats.total_hf_candidates,
- "valid": self.stats.hf_valid,
- "invalid": self.stats.hf_invalid,
- "conditional": self.stats.hf_conditional,
- "results": [asdict(r) for r in self.hf_results]
- }
- }
-
- report_path = reports_dir / "PROVIDER_AUTO_DISCOVERY_REPORT.json"
- with open(report_path, 'w') as f:
- json.dump(validation_report, f, indent=2)
-
- print(f"\n📊 Generated detailed report: {report_path.name}")
-
- # 2. Generate markdown summary
- self.generate_markdown_report()
-
- def generate_markdown_report(self) -> None:
- """Generate markdown report"""
- reports_dir = self.workspace_root
-
- md_content = f"""# Provider Auto-Discovery Report
-
-**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S UTC")}
-**Execution Time:** {self.stats.execution_time_sec:.2f} seconds
-
----
-
-## Executive Summary
-
-| Metric | Count |
-|--------|-------|
-| **Total HTTP Candidates** | {self.stats.total_http_candidates} |
-| **HTTP Valid** | {self.stats.http_valid} ✅ |
-| **HTTP Invalid** | {self.stats.http_invalid} ❌ |
-| **HTTP Conditional** | {self.stats.http_conditional} ⚠️ |
-| **Total HF Model Candidates** | {self.stats.total_hf_candidates} |
-| **HF Models Valid** | {self.stats.hf_valid} ✅ |
-| **HF Models Invalid** | {self.stats.hf_invalid} ❌ |
-| **HF Models Conditional** | {self.stats.hf_conditional} ⚠️ |
-| **TOTAL ACTIVE PROVIDERS** | **{self.stats.total_active_providers}** |
-
----
-
-## HTTP Providers
-
-### Valid Providers ({self.stats.http_valid})
-
-"""
-
- # List valid HTTP providers
- valid_http = [r for r in self.http_results if r.status == ValidationStatus.VALID.value]
- for result in sorted(valid_http, key=lambda x: x.response_time_ms or 999999):
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- md_content += f" - Category: {result.category}\n"
- md_content += f" - Type: {result.provider_type}\n"
- md_content += f" - Response Time: {result.response_time_ms:.0f}ms\n"
- if result.test_endpoint:
- md_content += f" - Test Endpoint: `{result.test_endpoint}`\n"
- md_content += "\n"
-
- md_content += f"""
-### Invalid Providers ({self.stats.http_invalid})
-
-"""
-
- # List some invalid providers with reasons
- invalid_http = [r for r in self.http_results if r.status == ValidationStatus.INVALID.value]
- for result in invalid_http[:20]: # Limit to first 20
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- md_content += f" - Reason: {result.error_reason}\n\n"
-
- if len(invalid_http) > 20:
- md_content += f"\n*... and {len(invalid_http) - 20} more invalid providers*\n"
-
- md_content += f"""
-### Conditionally Available Providers ({self.stats.http_conditional})
-
-These providers require API keys or special configuration:
-
-"""
-
- conditional_http = [r for r in self.http_results if r.status == ValidationStatus.CONDITIONALLY_AVAILABLE.value]
- for result in conditional_http:
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- if result.auth_env_var:
- md_content += f" - Required: `{result.auth_env_var}` environment variable\n"
- md_content += f" - Reason: {result.error_reason}\n\n"
-
- md_content += f"""
----
-
-## Hugging Face Models
-
-### Valid Models ({self.stats.hf_valid})
-
-"""
-
- valid_hf = [r for r in self.hf_results if r.status == ValidationStatus.VALID.value]
- for result in valid_hf:
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- if result.response_time_ms:
- md_content += f" - Response Time: {result.response_time_ms:.0f}ms\n"
- md_content += "\n"
-
- md_content += f"""
-### Invalid Models ({self.stats.hf_invalid})
-
-"""
-
- invalid_hf = [r for r in self.hf_results if r.status == ValidationStatus.INVALID.value]
- for result in invalid_hf:
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- md_content += f" - Reason: {result.error_reason}\n\n"
-
- md_content += f"""
-### Conditionally Available Models ({self.stats.hf_conditional})
-
-"""
-
- conditional_hf = [r for r in self.hf_results if r.status == ValidationStatus.CONDITIONALLY_AVAILABLE.value]
- for result in conditional_hf:
- md_content += f"- **{result.provider_name}** (`{result.provider_id}`)\n"
- if result.auth_env_var:
- md_content += f" - Required: `{result.auth_env_var}` environment variable\n"
- md_content += "\n"
-
- md_content += """
----
-
-## Integration Status
-
-All VALID providers have been integrated into `providers_config_extended.json`.
-
-**NO MOCK DATA was used in this validation process.**
-**All results are from REAL API calls and REAL model inferences.**
-
----
-
-## Next Steps
-
-1. **For Conditional Providers:** Set the required environment variables to activate them
-2. **For Invalid Providers:** Review error reasons and update configurations if needed
-3. **Monitor Performance:** Track response times and adjust provider priorities
-
----
-
-*Report generated by Auto Provider Loader (APL)*
-"""
-
- report_path = reports_dir / "PROVIDER_AUTO_DISCOVERY_REPORT.md"
- with open(report_path, 'w') as f:
- f.write(md_content)
-
- print(f"📋 Generated markdown report: {report_path.name}")
-
- async def run(self) -> None:
- """Run the complete APL process"""
- start_time = time.time()
-
- print("=" * 80)
- print("🚀 AUTO PROVIDER LOADER (APL) - REAL DATA ONLY")
- print("=" * 80)
-
- # Phase 1: Discovery
- print("\n📡 PHASE 1: DISCOVERY")
- http_providers = self.discover_http_providers()
- hf_models = self.discover_hf_models()
-
- print(f" Found {len(http_providers)} HTTP provider candidates")
- print(f" Found {len(hf_models)} HF model candidates")
-
- # Phase 2: Validation
- print("\n🔬 PHASE 2: VALIDATION")
- await self.validate_all_http_providers(http_providers)
- await self.validate_all_hf_models(hf_models)
-
- # Phase 3: Statistics
- print("\n📊 PHASE 3: COMPUTING STATISTICS")
- self.compute_stats()
-
- # Phase 4: Integration
- print("\n🔧 PHASE 4: INTEGRATION")
- self.integrate_valid_providers()
-
- # Phase 5: Reporting
- print("\n📝 PHASE 5: GENERATING REPORTS")
- self.stats.execution_time_sec = time.time() - start_time
- self.generate_reports()
-
- # Final summary
- print("\n" + "=" * 80)
- print("✅ STATUS: PROVIDER + HF MODEL EXPANSION COMPLETE")
- print("=" * 80)
- print(f"\n📈 FINAL COUNTS:")
- print(f" • HTTP Providers: {self.stats.total_http_candidates} candidates")
- print(f" ✅ Valid: {self.stats.http_valid}")
- print(f" ❌ Invalid: {self.stats.http_invalid}")
- print(f" ⚠️ Conditional: {self.stats.http_conditional}")
- print(f" • HF Models: {self.stats.total_hf_candidates} candidates")
- print(f" ✅ Valid: {self.stats.hf_valid}")
- print(f" ❌ Invalid: {self.stats.hf_invalid}")
- print(f" ⚠️ Conditional: {self.stats.hf_conditional}")
- print(f"\n 🎯 TOTAL ACTIVE: {self.stats.total_active_providers} providers")
- print(f"\n⏱️ Execution time: {self.stats.execution_time_sec:.2f} seconds")
- print(f"\n✅ NO MOCK/FAKE DATA - All results from REAL calls")
- print("=" * 80)
-
-
-async def main():
- """Main entry point"""
- apl = AutoProviderLoader()
- await apl.run()
-
-
-if __name__ == "__main__":
- asyncio.run(main())
diff --git a/app/backend/__init__.py b/app/backend/__init__.py
deleted file mode 100644
index f4e09269a6a4fe2d75a3639b9baa8351f83e6951..0000000000000000000000000000000000000000
--- a/app/backend/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Backend module
diff --git a/app/backend/__pycache__/__init__.cpython-313.pyc b/app/backend/__pycache__/__init__.cpython-313.pyc
deleted file mode 100644
index 74d767f547510fc8c6553a77367c1a0b43041112..0000000000000000000000000000000000000000
Binary files a/app/backend/__pycache__/__init__.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/__pycache__/feature_flags.cpython-313.pyc b/app/backend/__pycache__/feature_flags.cpython-313.pyc
deleted file mode 100644
index ecd2a356fd01cc291dd0cbfddff8ca082777de29..0000000000000000000000000000000000000000
Binary files a/app/backend/__pycache__/feature_flags.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/enhanced_logger.py b/app/backend/enhanced_logger.py
deleted file mode 100644
index 4e6dc422a4ac0099870b1aa0c2735cf163e0e1e9..0000000000000000000000000000000000000000
--- a/app/backend/enhanced_logger.py
+++ /dev/null
@@ -1,288 +0,0 @@
-"""
-Enhanced Logging System
-Provides structured logging with provider health tracking and error classification
-"""
-
-import logging
-import sys
-from datetime import datetime
-from typing import Optional, Dict, Any
-from pathlib import Path
-import json
-
-
-class ProviderHealthLogger:
- """Enhanced logger with provider health tracking"""
-
- def __init__(self, name: str = "crypto_monitor"):
- self.logger = logging.getLogger(name)
- self.health_log_path = Path("data/logs/provider_health.jsonl")
- self.error_log_path = Path("data/logs/errors.jsonl")
-
- # Create log directories
- self.health_log_path.parent.mkdir(parents=True, exist_ok=True)
- self.error_log_path.parent.mkdir(parents=True, exist_ok=True)
-
- # Set up handlers if not already configured
- if not self.logger.handlers:
- self._setup_handlers()
-
- def _setup_handlers(self):
- """Set up logging handlers"""
- self.logger.setLevel(logging.DEBUG)
-
- # Console handler with color
- console_handler = logging.StreamHandler(sys.stdout)
- console_handler.setLevel(logging.INFO)
-
- # Custom formatter with colors (if terminal supports it)
- console_formatter = ColoredFormatter(
- '%(asctime)s | %(levelname)-8s | %(name)s | %(message)s',
- datefmt='%Y-%m-%d %H:%M:%S'
- )
- console_handler.setFormatter(console_formatter)
-
- # File handler for all logs
- file_handler = logging.FileHandler('data/logs/app.log')
- file_handler.setLevel(logging.DEBUG)
- file_formatter = logging.Formatter(
- '%(asctime)s | %(levelname)-8s | %(name)s | %(funcName)s:%(lineno)d | %(message)s',
- datefmt='%Y-%m-%d %H:%M:%S'
- )
- file_handler.setFormatter(file_formatter)
-
- # Error file handler
- error_handler = logging.FileHandler('data/logs/errors.log')
- error_handler.setLevel(logging.ERROR)
- error_handler.setFormatter(file_formatter)
-
- # Add handlers
- self.logger.addHandler(console_handler)
- self.logger.addHandler(file_handler)
- self.logger.addHandler(error_handler)
-
- def log_provider_request(
- self,
- provider_name: str,
- endpoint: str,
- status: str,
- response_time_ms: Optional[float] = None,
- status_code: Optional[int] = None,
- error_message: Optional[str] = None,
- used_proxy: bool = False
- ):
- """Log a provider API request with full context"""
-
- log_entry = {
- "timestamp": datetime.now().isoformat(),
- "provider": provider_name,
- "endpoint": endpoint,
- "status": status,
- "response_time_ms": response_time_ms,
- "status_code": status_code,
- "error_message": error_message,
- "used_proxy": used_proxy
- }
-
- # Log to console
- if status == "success":
- self.logger.info(
- f"✓ {provider_name} | {endpoint} | {response_time_ms:.0f}ms | HTTP {status_code}"
- )
- elif status == "error":
- self.logger.error(
- f"✗ {provider_name} | {endpoint} | {error_message}"
- )
- elif status == "timeout":
- self.logger.warning(
- f"⏱ {provider_name} | {endpoint} | Timeout"
- )
- elif status == "proxy_fallback":
- self.logger.info(
- f"🌐 {provider_name} | {endpoint} | Switched to proxy"
- )
-
- # Append to JSONL health log
- try:
- with open(self.health_log_path, 'a', encoding='utf-8') as f:
- f.write(json.dumps(log_entry) + '\n')
- except Exception as e:
- self.logger.error(f"Failed to write health log: {e}")
-
- def log_error(
- self,
- error_type: str,
- message: str,
- provider: Optional[str] = None,
- endpoint: Optional[str] = None,
- traceback: Optional[str] = None,
- **extra
- ):
- """Log an error with classification"""
-
- error_entry = {
- "timestamp": datetime.now().isoformat(),
- "error_type": error_type,
- "message": message,
- "provider": provider,
- "endpoint": endpoint,
- "traceback": traceback,
- **extra
- }
-
- # Log to console
- self.logger.error(f"[{error_type}] {message}")
-
- if traceback:
- self.logger.debug(f"Traceback: {traceback}")
-
- # Append to JSONL error log
- try:
- with open(self.error_log_path, 'a', encoding='utf-8') as f:
- f.write(json.dumps(error_entry) + '\n')
- except Exception as e:
- self.logger.error(f"Failed to write error log: {e}")
-
- def log_proxy_switch(self, provider: str, reason: str):
- """Log when a provider switches to proxy mode"""
- self.logger.info(f"🌐 Proxy activated for {provider}: {reason}")
-
- def log_feature_flag_change(self, flag_name: str, old_value: bool, new_value: bool):
- """Log feature flag changes"""
- self.logger.info(f"⚙️ Feature flag '{flag_name}' changed: {old_value} → {new_value}")
-
- def log_health_check(self, provider: str, status: str, details: Optional[Dict] = None):
- """Log provider health check results"""
- if status == "online":
- self.logger.info(f"✓ Health check passed: {provider}")
- elif status == "degraded":
- self.logger.warning(f"⚠ Health check degraded: {provider}")
- else:
- self.logger.error(f"✗ Health check failed: {provider}")
-
- if details:
- self.logger.debug(f"Health details for {provider}: {details}")
-
- def get_recent_errors(self, limit: int = 100) -> list:
- """Read recent errors from log file"""
- errors = []
- try:
- if self.error_log_path.exists():
- with open(self.error_log_path, 'r', encoding='utf-8') as f:
- lines = f.readlines()
- for line in lines[-limit:]:
- try:
- errors.append(json.loads(line))
- except json.JSONDecodeError:
- continue
- except Exception as e:
- self.logger.error(f"Failed to read error log: {e}")
-
- return errors
-
- def get_provider_stats(self, provider: str, hours: int = 24) -> Dict[str, Any]:
- """Get statistics for a specific provider from logs"""
- from datetime import timedelta
-
- stats = {
- "total_requests": 0,
- "successful_requests": 0,
- "failed_requests": 0,
- "avg_response_time": 0,
- "proxy_requests": 0,
- "errors": []
- }
-
- try:
- if self.health_log_path.exists():
- cutoff_time = datetime.now() - timedelta(hours=hours)
- response_times = []
-
- with open(self.health_log_path, 'r', encoding='utf-8') as f:
- for line in f:
- try:
- entry = json.loads(line)
- entry_time = datetime.fromisoformat(entry["timestamp"])
-
- if entry_time < cutoff_time:
- continue
-
- if entry.get("provider") != provider:
- continue
-
- stats["total_requests"] += 1
-
- if entry.get("status") == "success":
- stats["successful_requests"] += 1
- if entry.get("response_time_ms"):
- response_times.append(entry["response_time_ms"])
- else:
- stats["failed_requests"] += 1
- if entry.get("error_message"):
- stats["errors"].append({
- "timestamp": entry["timestamp"],
- "message": entry["error_message"]
- })
-
- if entry.get("used_proxy"):
- stats["proxy_requests"] += 1
-
- except (json.JSONDecodeError, KeyError):
- continue
-
- if response_times:
- stats["avg_response_time"] = sum(response_times) / len(response_times)
-
- except Exception as e:
- self.logger.error(f"Failed to get provider stats: {e}")
-
- return stats
-
-
-class ColoredFormatter(logging.Formatter):
- """Custom formatter with colors for terminal output"""
-
- COLORS = {
- 'DEBUG': '\033[36m', # Cyan
- 'INFO': '\033[32m', # Green
- 'WARNING': '\033[33m', # Yellow
- 'ERROR': '\033[31m', # Red
- 'CRITICAL': '\033[35m', # Magenta
- 'RESET': '\033[0m' # Reset
- }
-
- def format(self, record):
- # Add color to level name
- if record.levelname in self.COLORS:
- record.levelname = (
- f"{self.COLORS[record.levelname]}"
- f"{record.levelname}"
- f"{self.COLORS['RESET']}"
- )
-
- return super().format(record)
-
-
-# Global instance
-provider_health_logger = ProviderHealthLogger()
-
-
-# Convenience functions
-def log_request(provider: str, endpoint: str, **kwargs):
- """Log a provider request"""
- provider_health_logger.log_provider_request(provider, endpoint, **kwargs)
-
-
-def log_error(error_type: str, message: str, **kwargs):
- """Log an error"""
- provider_health_logger.log_error(error_type, message, **kwargs)
-
-
-def log_proxy_switch(provider: str, reason: str):
- """Log proxy switch"""
- provider_health_logger.log_proxy_switch(provider, reason)
-
-
-def get_provider_stats(provider: str, hours: int = 24):
- """Get provider statistics"""
- return provider_health_logger.get_provider_stats(provider, hours)
diff --git a/app/backend/feature_flags.py b/app/backend/feature_flags.py
deleted file mode 100644
index beb2dcf6d3c4097027a965ab5bf1867d6ae4c8c4..0000000000000000000000000000000000000000
--- a/app/backend/feature_flags.py
+++ /dev/null
@@ -1,214 +0,0 @@
-"""
-Feature Flags System
-Allows dynamic toggling of application modules and features
-"""
-from typing import Dict, Any
-import json
-from pathlib import Path
-from datetime import datetime
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class FeatureFlagManager:
- """Manage application feature flags"""
-
- DEFAULT_FLAGS = {
- "enableWhaleTracking": True,
- "enableMarketOverview": True,
- "enableFearGreedIndex": True,
- "enableNewsFeed": True,
- "enableSentimentAnalysis": True,
- "enableMlPredictions": False, # Disabled by default (requires HF setup)
- "enableProxyAutoMode": True,
- "enableDefiProtocols": True,
- "enableTrendingCoins": True,
- "enableGlobalStats": True,
- "enableProviderRotation": True,
- "enableWebSocketStreaming": True,
- "enableDatabaseLogging": True,
- "enableRealTimeAlerts": False, # New feature - not yet implemented
- "enableAdvancedCharts": True,
- "enableExportFeatures": True,
- "enableCustomProviders": True,
- "enablePoolManagement": True,
- "enableHFIntegration": True,
- }
-
- def __init__(self, storage_path: str = "data/feature_flags.json"):
- """
- Initialize feature flag manager
-
- Args:
- storage_path: Path to persist feature flags
- """
- self.storage_path = Path(storage_path)
- self.flags = self.DEFAULT_FLAGS.copy()
- self.load_flags()
-
- def load_flags(self):
- """Load feature flags from storage"""
- try:
- if self.storage_path.exists():
- with open(self.storage_path, 'r', encoding='utf-8') as f:
- saved_flags = json.load(f)
- # Merge saved flags with defaults (in case new flags were added)
- self.flags.update(saved_flags.get('flags', {}))
- logger.info(f"Loaded feature flags from {self.storage_path}")
- else:
- # Create storage directory if it doesn't exist
- self.storage_path.parent.mkdir(parents=True, exist_ok=True)
- self.save_flags()
- logger.info("Initialized default feature flags")
- except Exception as e:
- logger.error(f"Error loading feature flags: {e}")
- self.flags = self.DEFAULT_FLAGS.copy()
-
- def save_flags(self):
- """Save feature flags to storage"""
- try:
- self.storage_path.parent.mkdir(parents=True, exist_ok=True)
- data = {
- 'flags': self.flags,
- 'last_updated': datetime.now().isoformat()
- }
- with open(self.storage_path, 'w', encoding='utf-8') as f:
- json.dump(data, f, indent=2)
- logger.info("Feature flags saved successfully")
- except Exception as e:
- logger.error(f"Error saving feature flags: {e}")
-
- def get_all_flags(self) -> Dict[str, bool]:
- """Get all feature flags"""
- return self.flags.copy()
-
- def get_flag(self, flag_name: str) -> bool:
- """
- Get a specific feature flag value
-
- Args:
- flag_name: Name of the flag
-
- Returns:
- bool: Flag value (defaults to False if not found)
- """
- return self.flags.get(flag_name, False)
-
- def set_flag(self, flag_name: str, value: bool) -> bool:
- """
- Set a feature flag value
-
- Args:
- flag_name: Name of the flag
- value: New value (True/False)
-
- Returns:
- bool: Success status
- """
- try:
- self.flags[flag_name] = bool(value)
- self.save_flags()
- logger.info(f"Feature flag '{flag_name}' set to {value}")
- return True
- except Exception as e:
- logger.error(f"Error setting feature flag: {e}")
- return False
-
- def update_flags(self, updates: Dict[str, bool]) -> bool:
- """
- Update multiple flags at once
-
- Args:
- updates: Dictionary of flag name -> value pairs
-
- Returns:
- bool: Success status
- """
- try:
- for flag_name, value in updates.items():
- self.flags[flag_name] = bool(value)
- self.save_flags()
- logger.info(f"Updated {len(updates)} feature flags")
- return True
- except Exception as e:
- logger.error(f"Error updating feature flags: {e}")
- return False
-
- def reset_to_defaults(self) -> bool:
- """Reset all flags to default values"""
- try:
- self.flags = self.DEFAULT_FLAGS.copy()
- self.save_flags()
- logger.info("Feature flags reset to defaults")
- return True
- except Exception as e:
- logger.error(f"Error resetting feature flags: {e}")
- return False
-
- def is_enabled(self, flag_name: str) -> bool:
- """
- Check if a feature is enabled (alias for get_flag)
-
- Args:
- flag_name: Name of the flag
-
- Returns:
- bool: True if enabled, False otherwise
- """
- return self.get_flag(flag_name)
-
- def get_enabled_features(self) -> Dict[str, bool]:
- """Get only enabled features"""
- return {k: v for k, v in self.flags.items() if v is True}
-
- def get_disabled_features(self) -> Dict[str, bool]:
- """Get only disabled features"""
- return {k: v for k, v in self.flags.items() if v is False}
-
- def get_flag_count(self) -> Dict[str, int]:
- """Get count of enabled/disabled flags"""
- enabled = sum(1 for v in self.flags.values() if v)
- disabled = len(self.flags) - enabled
- return {
- 'total': len(self.flags),
- 'enabled': enabled,
- 'disabled': disabled
- }
-
- def get_feature_info(self) -> Dict[str, Any]:
- """Get comprehensive feature flag information"""
- counts = self.get_flag_count()
- return {
- 'flags': self.flags,
- 'counts': counts,
- 'enabled_features': list(self.get_enabled_features().keys()),
- 'disabled_features': list(self.get_disabled_features().keys()),
- 'storage_path': str(self.storage_path),
- 'last_loaded': datetime.now().isoformat()
- }
-
-
-# Global instance
-feature_flags = FeatureFlagManager()
-
-
-# Convenience functions
-def is_feature_enabled(flag_name: str) -> bool:
- """Check if a feature is enabled"""
- return feature_flags.is_enabled(flag_name)
-
-
-def get_all_feature_flags() -> Dict[str, bool]:
- """Get all feature flags"""
- return feature_flags.get_all_flags()
-
-
-def set_feature_flag(flag_name: str, value: bool) -> bool:
- """Set a feature flag"""
- return feature_flags.set_flag(flag_name, value)
-
-
-def update_feature_flags(updates: Dict[str, bool]) -> bool:
- """Update multiple feature flags"""
- return feature_flags.update_flags(updates)
diff --git a/app/backend/routers/__init__.py b/app/backend/routers/__init__.py
deleted file mode 100644
index 57fa55678bfd1b9960495821d74a6459efd647b6..0000000000000000000000000000000000000000
--- a/app/backend/routers/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Backend routers module
diff --git a/app/backend/routers/__pycache__/__init__.cpython-313.pyc b/app/backend/routers/__pycache__/__init__.cpython-313.pyc
deleted file mode 100644
index ace041e7a5f14f58446c24419d1b344af1a6c25e..0000000000000000000000000000000000000000
Binary files a/app/backend/routers/__pycache__/__init__.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/routers/__pycache__/hf_connect.cpython-313.pyc b/app/backend/routers/__pycache__/hf_connect.cpython-313.pyc
deleted file mode 100644
index 290023344449a523c1dc4f33b1b747ce85ddbca7..0000000000000000000000000000000000000000
Binary files a/app/backend/routers/__pycache__/hf_connect.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/routers/hf_connect.py b/app/backend/routers/hf_connect.py
deleted file mode 100644
index e43a16ed2d9803c582c93030ede9e76545d3874e..0000000000000000000000000000000000000000
--- a/app/backend/routers/hf_connect.py
+++ /dev/null
@@ -1,35 +0,0 @@
-from __future__ import annotations
-from fastapi import APIRouter, Query, Body
-from typing import Literal, List
-from backend.services.hf_registry import REGISTRY
-from backend.services.hf_client import run_sentiment
-
-router = APIRouter(prefix="/api/hf", tags=["huggingface"])
-
-
-@router.get("/health")
-async def hf_health():
- return REGISTRY.health()
-
-
-@router.post("/refresh")
-async def hf_refresh():
- return await REGISTRY.refresh()
-
-
-@router.get("/registry")
-async def hf_registry(kind: Literal["models","datasets"]="models"):
- return {"kind": kind, "items": REGISTRY.list(kind)}
-
-
-@router.get("/search")
-async def hf_search(q: str = Query("crypto"), kind: Literal["models","datasets"]="models"):
- hay = REGISTRY.list(kind)
- ql = q.lower()
- res = [x for x in hay if ql in (x.get("id","").lower() + " " + " ".join([str(t) for t in x.get("tags",[])]).lower())]
- return {"query": q, "kind": kind, "count": len(res), "items": res[:50]}
-
-
-@router.post("/run-sentiment")
-async def hf_run_sentiment(texts: List[str] = Body(..., embed=True), model: str | None = Body(default=None)):
- return run_sentiment(texts, model=model)
diff --git a/app/backend/routers/integrated_api.py b/app/backend/routers/integrated_api.py
deleted file mode 100644
index 3eff5da12ba712a97c2d15aec85fbb68582f929f..0000000000000000000000000000000000000000
--- a/app/backend/routers/integrated_api.py
+++ /dev/null
@@ -1,470 +0,0 @@
-"""
-Integrated API Router
-Combines all services for a comprehensive backend API
-"""
-from fastapi import APIRouter, WebSocket, WebSocketDisconnect, HTTPException, BackgroundTasks
-from fastapi.responses import FileResponse, JSONResponse
-from typing import Optional, List, Dict, Any
-from datetime import datetime
-import logging
-import uuid
-import os
-
-logger = logging.getLogger(__name__)
-
-router = APIRouter(prefix="/api/v2", tags=["Integrated API"])
-
-# These will be set by the main application
-config_loader = None
-scheduler_service = None
-persistence_service = None
-websocket_service = None
-
-
-def set_services(config, scheduler, persistence, websocket):
- """Set service instances"""
- global config_loader, scheduler_service, persistence_service, websocket_service
- config_loader = config
- scheduler_service = scheduler
- persistence_service = persistence
- websocket_service = websocket
-
-
-# ============================================================================
-# WebSocket Endpoint
-# ============================================================================
-
-@router.websocket("/ws")
-async def websocket_endpoint(websocket: WebSocket):
- """WebSocket endpoint for real-time updates"""
- client_id = str(uuid.uuid4())
-
- try:
- await websocket_service.connection_manager.connect(
- websocket,
- client_id,
- metadata={'connected_at': datetime.now().isoformat()}
- )
-
- # Send welcome message
- await websocket_service.connection_manager.send_personal_message({
- 'type': 'connected',
- 'client_id': client_id,
- 'message': 'Connected to crypto data tracker'
- }, client_id)
-
- # Handle messages
- while True:
- data = await websocket.receive_json()
- await websocket_service.handle_client_message(websocket, client_id, data)
-
- except WebSocketDisconnect:
- websocket_service.connection_manager.disconnect(client_id)
- except Exception as e:
- logger.error(f"WebSocket error for client {client_id}: {e}")
- websocket_service.connection_manager.disconnect(client_id)
-
-
-# ============================================================================
-# Configuration Endpoints
-# ============================================================================
-
-@router.get("/config/apis")
-async def get_all_apis():
- """Get all configured APIs"""
- return {
- 'apis': config_loader.get_all_apis(),
- 'total': len(config_loader.apis)
- }
-
-
-@router.get("/config/apis/{api_id}")
-async def get_api(api_id: str):
- """Get specific API configuration"""
- api = config_loader.apis.get(api_id)
-
- if not api:
- raise HTTPException(status_code=404, detail="API not found")
-
- return api
-
-
-@router.get("/config/categories")
-async def get_categories():
- """Get all API categories"""
- categories = config_loader.get_categories()
-
- category_stats = {}
- for category in categories:
- apis = config_loader.get_apis_by_category(category)
- category_stats[category] = {
- 'count': len(apis),
- 'apis': list(apis.keys())
- }
-
- return {
- 'categories': categories,
- 'stats': category_stats
- }
-
-
-@router.get("/config/apis/category/{category}")
-async def get_apis_by_category(category: str):
- """Get APIs by category"""
- apis = config_loader.get_apis_by_category(category)
-
- return {
- 'category': category,
- 'apis': apis,
- 'count': len(apis)
- }
-
-
-@router.post("/config/apis")
-async def add_custom_api(api_data: Dict[str, Any]):
- """Add a custom API"""
- try:
- success = config_loader.add_custom_api(api_data)
-
- if success:
- return {'status': 'success', 'message': 'API added successfully'}
- else:
- raise HTTPException(status_code=400, detail="Failed to add API")
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.delete("/config/apis/{api_id}")
-async def remove_api(api_id: str):
- """Remove an API"""
- success = config_loader.remove_api(api_id)
-
- if success:
- return {'status': 'success', 'message': 'API removed successfully'}
- else:
- raise HTTPException(status_code=404, detail="API not found")
-
-
-@router.get("/config/export")
-async def export_config():
- """Export configuration to JSON"""
- filepath = f"data/exports/config_export_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
- os.makedirs(os.path.dirname(filepath), exist_ok=True)
-
- config_loader.export_config(filepath)
-
- return FileResponse(
- filepath,
- media_type='application/json',
- filename=os.path.basename(filepath)
- )
-
-
-# ============================================================================
-# Scheduler Endpoints
-# ============================================================================
-
-@router.get("/schedule/tasks")
-async def get_all_schedules():
- """Get all scheduled tasks"""
- return scheduler_service.get_all_task_statuses()
-
-
-@router.get("/schedule/tasks/{api_id}")
-async def get_schedule(api_id: str):
- """Get schedule for specific API"""
- status = scheduler_service.get_task_status(api_id)
-
- if not status:
- raise HTTPException(status_code=404, detail="Task not found")
-
- return status
-
-
-@router.put("/schedule/tasks/{api_id}")
-async def update_schedule(api_id: str, interval: Optional[int] = None, enabled: Optional[bool] = None):
- """Update schedule for an API"""
- try:
- scheduler_service.update_task_schedule(api_id, interval, enabled)
-
- # Notify WebSocket clients
- await websocket_service.notify_schedule_update({
- 'api_id': api_id,
- 'interval': interval,
- 'enabled': enabled
- })
-
- return {
- 'status': 'success',
- 'message': 'Schedule updated',
- 'task': scheduler_service.get_task_status(api_id)
- }
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.post("/schedule/tasks/{api_id}/force-update")
-async def force_update(api_id: str):
- """Force immediate update for an API"""
- try:
- success = await scheduler_service.force_update(api_id)
-
- if success:
- return {
- 'status': 'success',
- 'message': 'Update completed',
- 'task': scheduler_service.get_task_status(api_id)
- }
- else:
- raise HTTPException(status_code=500, detail="Update failed")
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.get("/schedule/export")
-async def export_schedules():
- """Export schedules to JSON"""
- filepath = f"data/exports/schedules_export_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
- os.makedirs(os.path.dirname(filepath), exist_ok=True)
-
- scheduler_service.export_schedules(filepath)
-
- return FileResponse(
- filepath,
- media_type='application/json',
- filename=os.path.basename(filepath)
- )
-
-
-# ============================================================================
-# Data Endpoints
-# ============================================================================
-
-@router.get("/data/cached")
-async def get_all_cached_data():
- """Get all cached data"""
- return persistence_service.get_all_cached_data()
-
-
-@router.get("/data/cached/{api_id}")
-async def get_cached_data(api_id: str):
- """Get cached data for specific API"""
- data = persistence_service.get_cached_data(api_id)
-
- if not data:
- raise HTTPException(status_code=404, detail="No cached data found")
-
- return data
-
-
-@router.get("/data/history/{api_id}")
-async def get_history(api_id: str, limit: int = 100):
- """Get historical data for an API"""
- history = persistence_service.get_history(api_id, limit)
-
- return {
- 'api_id': api_id,
- 'history': history,
- 'count': len(history)
- }
-
-
-@router.get("/data/statistics")
-async def get_data_statistics():
- """Get data storage statistics"""
- return persistence_service.get_statistics()
-
-
-# ============================================================================
-# Export/Import Endpoints
-# ============================================================================
-
-@router.post("/export/json")
-async def export_to_json(
- api_ids: Optional[List[str]] = None,
- include_history: bool = False,
- background_tasks: BackgroundTasks = None
-):
- """Export data to JSON"""
- try:
- timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
- filepath = f"data/exports/data_export_{timestamp}.json"
- os.makedirs(os.path.dirname(filepath), exist_ok=True)
-
- await persistence_service.export_to_json(filepath, api_ids, include_history)
-
- return {
- 'status': 'success',
- 'filepath': filepath,
- 'download_url': f"/api/v2/download?file={filepath}"
- }
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.post("/export/csv")
-async def export_to_csv(api_ids: Optional[List[str]] = None, flatten: bool = True):
- """Export data to CSV"""
- try:
- timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
- filepath = f"data/exports/data_export_{timestamp}.csv"
- os.makedirs(os.path.dirname(filepath), exist_ok=True)
-
- await persistence_service.export_to_csv(filepath, api_ids, flatten)
-
- return {
- 'status': 'success',
- 'filepath': filepath,
- 'download_url': f"/api/v2/download?file={filepath}"
- }
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.post("/export/history/{api_id}")
-async def export_history(api_id: str):
- """Export historical data for an API to CSV"""
- try:
- timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
- filepath = f"data/exports/{api_id}_history_{timestamp}.csv"
- os.makedirs(os.path.dirname(filepath), exist_ok=True)
-
- await persistence_service.export_history_to_csv(filepath, api_id)
-
- return {
- 'status': 'success',
- 'filepath': filepath,
- 'download_url': f"/api/v2/download?file={filepath}"
- }
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.get("/download")
-async def download_file(file: str):
- """Download exported file"""
- if not os.path.exists(file):
- raise HTTPException(status_code=404, detail="File not found")
-
- return FileResponse(
- file,
- media_type='application/octet-stream',
- filename=os.path.basename(file)
- )
-
-
-@router.post("/backup")
-async def create_backup():
- """Create a backup of all data"""
- try:
- backup_file = await persistence_service.backup_all_data()
-
- return {
- 'status': 'success',
- 'backup_file': backup_file,
- 'download_url': f"/api/v2/download?file={backup_file}"
- }
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-@router.post("/restore")
-async def restore_from_backup(backup_file: str):
- """Restore data from backup"""
- try:
- success = await persistence_service.restore_from_backup(backup_file)
-
- if success:
- return {'status': 'success', 'message': 'Data restored successfully'}
- else:
- raise HTTPException(status_code=500, detail="Restore failed")
-
- except Exception as e:
- raise HTTPException(status_code=500, detail=str(e))
-
-
-# ============================================================================
-# Status Endpoints
-# ============================================================================
-
-@router.get("/status")
-async def get_system_status():
- """Get overall system status"""
- return {
- 'timestamp': datetime.now().isoformat(),
- 'services': {
- 'config_loader': {
- 'apis_loaded': len(config_loader.apis),
- 'categories': len(config_loader.get_categories()),
- 'schedules': len(config_loader.schedules)
- },
- 'scheduler': {
- 'running': scheduler_service.running,
- 'total_tasks': len(scheduler_service.tasks),
- 'realtime_tasks': len(scheduler_service.realtime_tasks),
- 'cache_size': len(scheduler_service.data_cache)
- },
- 'persistence': {
- 'cached_apis': len(persistence_service.cache),
- 'apis_with_history': len(persistence_service.history),
- 'total_history_records': sum(len(h) for h in persistence_service.history.values())
- },
- 'websocket': websocket_service.get_stats()
- }
- }
-
-
-@router.get("/health")
-async def health_check():
- """Health check endpoint"""
- return {
- 'status': 'healthy',
- 'timestamp': datetime.now().isoformat(),
- 'services': {
- 'config': config_loader is not None,
- 'scheduler': scheduler_service is not None and scheduler_service.running,
- 'persistence': persistence_service is not None,
- 'websocket': websocket_service is not None
- }
- }
-
-
-# ============================================================================
-# Cleanup Endpoints
-# ============================================================================
-
-@router.post("/cleanup/cache")
-async def clear_cache():
- """Clear all cached data"""
- persistence_service.clear_cache()
- return {'status': 'success', 'message': 'Cache cleared'}
-
-
-@router.post("/cleanup/history")
-async def clear_history(api_id: Optional[str] = None):
- """Clear history"""
- persistence_service.clear_history(api_id)
-
- if api_id:
- return {'status': 'success', 'message': f'History cleared for {api_id}'}
- else:
- return {'status': 'success', 'message': 'All history cleared'}
-
-
-@router.post("/cleanup/old-data")
-async def cleanup_old_data(days: int = 7):
- """Remove data older than specified days"""
- removed = await persistence_service.cleanup_old_data(days)
-
- return {
- 'status': 'success',
- 'message': f'Cleaned up {removed} old records',
- 'removed_count': removed
- }
diff --git a/app/backend/services/__init__.py b/app/backend/services/__init__.py
deleted file mode 100644
index bef86448a42129ebec41d8654a7e2a444b77b37a..0000000000000000000000000000000000000000
--- a/app/backend/services/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Backend services module
diff --git a/app/backend/services/__pycache__/__init__.cpython-313.pyc b/app/backend/services/__pycache__/__init__.cpython-313.pyc
deleted file mode 100644
index bf137c40bc340223f9066965f3902389e2579288..0000000000000000000000000000000000000000
Binary files a/app/backend/services/__pycache__/__init__.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/services/__pycache__/hf_client.cpython-313.pyc b/app/backend/services/__pycache__/hf_client.cpython-313.pyc
deleted file mode 100644
index 0aeec673a9a8e2408e30a8325424d5da6d517e10..0000000000000000000000000000000000000000
Binary files a/app/backend/services/__pycache__/hf_client.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/services/__pycache__/hf_registry.cpython-313.pyc b/app/backend/services/__pycache__/hf_registry.cpython-313.pyc
deleted file mode 100644
index 1ea2af8edc29afe2c0e2441faf1efa490f2f9801..0000000000000000000000000000000000000000
Binary files a/app/backend/services/__pycache__/hf_registry.cpython-313.pyc and /dev/null differ
diff --git a/app/backend/services/auto_discovery_service.py b/app/backend/services/auto_discovery_service.py
deleted file mode 100644
index 15fe1f589c31fc0f5b1073e07f0fcc26995a3861..0000000000000000000000000000000000000000
--- a/app/backend/services/auto_discovery_service.py
+++ /dev/null
@@ -1,421 +0,0 @@
-"""
-Auto Discovery Service
-----------------------
-جستجوی خودکار منابع API رایگان با استفاده از موتور جستجوی DuckDuckGo و
-تحلیل خروجی توسط مدلهای Hugging Face.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import inspect
-import json
-import logging
-import os
-import re
-from dataclasses import dataclass
-from datetime import datetime
-from typing import Any, Dict, List, Optional
-from contextlib import AsyncExitStack
-
-try:
- from duckduckgo_search import AsyncDDGS # type: ignore
-except ImportError: # pragma: no cover
- AsyncDDGS = None # type: ignore
-
-try:
- from huggingface_hub import InferenceClient # type: ignore
-except ImportError: # pragma: no cover
- InferenceClient = None # type: ignore
-
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class DiscoveryResult:
- """نتیجهٔ نهایی جستجو و تحلیل"""
-
- provider_id: str
- name: str
- category: str
- base_url: str
- requires_auth: bool
- description: str
- source_url: str
-
-
-class AutoDiscoveryService:
- """
- سرویس جستجوی خودکار منابع.
-
- این سرویس:
- 1. با استفاده از DuckDuckGo نتایج مرتبط با APIهای رایگان را جمعآوری میکند.
- 2. متن نتایج را به مدل Hugging Face میفرستد تا پیشنهادهای ساختاریافته بازگردد.
- 3. پیشنهادهای معتبر را به ResourceManager اضافه میکند و در صورت تأیید، ProviderManager را ریفرش میکند.
- """
-
- DEFAULT_QUERIES: List[str] = [
- "free cryptocurrency market data api",
- "open blockchain explorer api free tier",
- "free defi protocol api documentation",
- "open source sentiment analysis crypto api",
- "public nft market data api no api key",
- ]
-
- def __init__(
- self,
- resource_manager,
- provider_manager,
- enabled: bool = True,
- ):
- self.resource_manager = resource_manager
- self.provider_manager = provider_manager
- self.enabled = enabled and os.getenv("ENABLE_AUTO_DISCOVERY", "true").lower() == "true"
- self.interval_seconds = int(os.getenv("AUTO_DISCOVERY_INTERVAL_SECONDS", "43200"))
- self.hf_model = os.getenv("AUTO_DISCOVERY_HF_MODEL", "HuggingFaceH4/zephyr-7b-beta")
- self.max_candidates_per_query = int(os.getenv("AUTO_DISCOVERY_MAX_RESULTS", "8"))
- self._hf_client: Optional[InferenceClient] = None
- self._running_task: Optional[asyncio.Task] = None
- self._last_run_summary: Optional[Dict[str, Any]] = None
-
- if not self.enabled:
- logger.info("Auto discovery service disabled via configuration.")
- return
-
- if AsyncDDGS is None:
- logger.warning("duckduckgo-search package not available. Disabling auto discovery.")
- self.enabled = False
- return
-
- if InferenceClient is None:
- logger.warning("huggingface-hub package not available. Auto discovery will use fallback heuristics.")
- else:
- hf_token = os.getenv("HF_API_TOKEN")
- try:
- self._hf_client = InferenceClient(model=self.hf_model, token=hf_token)
- logger.info("Auto discovery Hugging Face client initialized with model %s", self.hf_model)
- except Exception as exc: # pragma: no cover - فقط برای شرایط عدم اتصال
- logger.error("Failed to initialize Hugging Face client: %s", exc)
- self._hf_client = None
-
- async def start(self):
- """شروع سرویس و ساخت حلقهٔ دورهای."""
- if not self.enabled:
- return
- if self._running_task and not self._running_task.done():
- return
- self._running_task = asyncio.create_task(self._run_periodic_loop())
- logger.info("Auto discovery service started with interval %s seconds", self.interval_seconds)
-
- async def stop(self):
- """توقف سرویس."""
- if self._running_task:
- self._running_task.cancel()
- try:
- await self._running_task
- except asyncio.CancelledError:
- pass
- self._running_task = None
- logger.info("Auto discovery service stopped.")
-
- async def trigger_manual_discovery(self) -> Dict[str, Any]:
- """اجرای دستی یک چرخهٔ کشف."""
- if not self.enabled:
- return {"status": "disabled"}
- summary = await self._run_discovery_cycle()
- return {"status": "completed", "summary": summary}
-
- def get_status(self) -> Dict[str, Any]:
- """وضعیت آخرین اجرا."""
- return {
- "enabled": self.enabled,
- "model": self.hf_model if self._hf_client else None,
- "interval_seconds": self.interval_seconds,
- "last_run": self._last_run_summary,
- }
-
- async def _run_periodic_loop(self):
- """حلقهٔ اجرای دورهای."""
- while self.enabled:
- try:
- await self._run_discovery_cycle()
- except Exception as exc:
- logger.exception("Auto discovery cycle failed: %s", exc)
- await asyncio.sleep(self.interval_seconds)
-
- async def _run_discovery_cycle(self) -> Dict[str, Any]:
- """یک چرخه کامل جستجو، تحلیل و ثبت."""
- started_at = datetime.utcnow().isoformat()
- candidates = await self._gather_candidates()
- structured = await self._infer_candidates(candidates)
- persisted = await self._persist_candidates(structured)
-
- summary = {
- "started_at": started_at,
- "finished_at": datetime.utcnow().isoformat(),
- "candidates_seen": len(candidates),
- "suggested": len(structured),
- "persisted": len(persisted),
- "persisted_ids": [item.provider_id for item in persisted],
- }
- self._last_run_summary = summary
-
- logger.info(
- "Auto discovery cycle completed. candidates=%s suggested=%s persisted=%s",
- summary["candidates_seen"],
- summary["suggested"],
- summary["persisted"],
- )
- return summary
-
- async def _gather_candidates(self) -> List[Dict[str, Any]]:
- """جمعآوری نتایج موتور جستجو."""
- if not self.enabled or AsyncDDGS is None:
- return []
-
- results: List[Dict[str, Any]] = []
- queries = os.getenv("AUTO_DISCOVERY_QUERIES")
- if queries:
- query_list = [q.strip() for q in queries.split(";") if q.strip()]
- else:
- query_list = self.DEFAULT_QUERIES
-
- try:
- async with AsyncExitStack() as stack:
- ddgs = await stack.enter_async_context(AsyncDDGS())
-
- for query in query_list:
- try:
- text_method = getattr(ddgs, "atext", None)
- if callable(text_method):
- async for entry in text_method(
- query,
- max_results=self.max_candidates_per_query,
- ):
- results.append(
- {
- "query": query,
- "title": entry.get("title", ""),
- "url": entry.get("href") or entry.get("url") or "",
- "snippet": entry.get("body", ""),
- }
- )
- continue
-
- text_method = getattr(ddgs, "text", None)
- if not callable(text_method):
- raise AttributeError("AsyncDDGS has no 'atext' or 'text' method")
-
- search_result = text_method(
- query,
- max_results=self.max_candidates_per_query,
- )
-
- if inspect.isawaitable(search_result):
- search_result = await search_result
-
- if hasattr(search_result, "__aiter__"):
- async for entry in search_result:
- results.append(
- {
- "query": query,
- "title": entry.get("title", ""),
- "url": entry.get("href") or entry.get("url") or "",
- "snippet": entry.get("body", ""),
- }
- )
- else:
- iterable = (
- search_result
- if isinstance(search_result, list)
- else list(search_result or [])
- )
- for entry in iterable:
- results.append(
- {
- "query": query,
- "title": entry.get("title", ""),
- "url": entry.get("href") or entry.get("url") or "",
- "snippet": entry.get("body", ""),
- }
- )
- except Exception as exc: # pragma: no cover - وابسته به اینترنت
- logger.warning(
- "Failed to fetch results for query '%s': %s. Skipping remaining queries this cycle.",
- query,
- exc,
- )
- break
- except Exception as exc:
- logger.warning(
- "DuckDuckGo auto discovery unavailable (%s). Skipping discovery cycle.",
- exc,
- )
- finally:
- close_method = getattr(ddgs, "close", None) if "ddgs" in locals() else None
- if inspect.iscoroutinefunction(close_method):
- try:
- await close_method()
- except Exception:
- pass
- elif callable(close_method):
- try:
- close_method()
- except Exception:
- pass
-
- return results
-
- async def _infer_candidates(self, candidates: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
- """تحلیل نتایج با مدل Hugging Face یا قواعد ساده."""
- if not candidates:
- return []
-
- if self._hf_client:
- prompt = self._build_prompt(candidates)
- try:
- response = await asyncio.to_thread(
- self._hf_client.text_generation,
- prompt,
- max_new_tokens=512,
- temperature=0.1,
- top_p=0.9,
- repetition_penalty=1.1,
- )
- return self._parse_model_response(response)
- except Exception as exc: # pragma: no cover
- logger.warning("Hugging Face inference failed: %s", exc)
-
- # fallback rule-based
- return self._rule_based_filter(candidates)
-
- def _build_prompt(self, candidates: List[Dict[str, Any]]) -> str:
- """ساخت پرامپت برای مدل LLM."""
- context_lines = []
- for idx, item in enumerate(candidates, start=1):
- context_lines.append(
- f"{idx}. Title: {item.get('title')}\n"
- f" URL: {item.get('url')}\n"
- f" Snippet: {item.get('snippet')}"
- )
-
- return (
- "You are an expert agent that extracts publicly accessible API providers for cryptocurrency, "
- "blockchain, DeFi, sentiment, NFT or analytics data. From the context entries, select candidates "
- "that represent real API services which are freely accessible (free tier or free plan). "
- "Return ONLY a JSON array. Each entry MUST include keys: "
- "id (lowercase snake_case), name, base_url, category (one of: market_data, blockchain_explorers, "
- "defi, sentiment, nft, analytics, news, rpc, huggingface, whale_tracking, onchain_analytics, custom), "
- "requires_auth (boolean), description (short string), source_url (string). "
- "Do not invent APIs. Ignore SDKs, articles, or paid-only services. "
- "If no valid candidate exists, return an empty JSON array.\n\n"
- "Context:\n"
- + "\n".join(context_lines)
- )
-
- def _parse_model_response(self, response: str) -> List[Dict[str, Any]]:
- """تبدیل پاسخ مدل به ساختار داده."""
- try:
- match = re.search(r"\[.*\]", response, re.DOTALL)
- if not match:
- logger.debug("Model response did not contain JSON array.")
- return []
- data = json.loads(match.group(0))
- if isinstance(data, list):
- return [item for item in data if isinstance(item, dict)]
- return []
- except json.JSONDecodeError:
- logger.debug("Failed to decode model JSON response.")
- return []
-
- def _rule_based_filter(self, candidates: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
- """فیلتر ساده در صورت در دسترس نبودن مدل."""
- structured: List[Dict[str, Any]] = []
- for item in candidates:
- url = item.get("url", "")
- snippet = (item.get("snippet") or "").lower()
- title = (item.get("title") or "").lower()
- if not url or "github" in url:
- continue
- if "api" not in title and "api" not in snippet:
- continue
- if any(keyword in snippet for keyword in ["pricing", "paid plan", "enterprise only"]):
- continue
- provider_id = self._normalize_id(item.get("title") or url)
- structured.append(
- {
- "id": provider_id,
- "name": item.get("title") or provider_id,
- "base_url": url,
- "category": "custom",
- "requires_auth": "token" in snippet or "apikey" in snippet,
- "description": item.get("snippet", ""),
- "source_url": url,
- }
- )
- return structured
-
- async def _persist_candidates(self, structured: List[Dict[str, Any]]) -> List[DiscoveryResult]:
- """ذخیرهٔ پیشنهادهای معتبر."""
- persisted: List[DiscoveryResult] = []
- if not structured:
- return persisted
-
- for entry in structured:
- provider_id = self._normalize_id(entry.get("id") or entry.get("name"))
- base_url = entry.get("base_url", "")
-
- if not base_url.startswith(("http://", "https://")):
- continue
-
- if self.resource_manager.get_provider(provider_id):
- continue
-
- provider_data = {
- "id": provider_id,
- "name": entry.get("name", provider_id),
- "category": entry.get("category", "custom"),
- "base_url": base_url,
- "requires_auth": bool(entry.get("requires_auth")),
- "priority": 4,
- "weight": 40,
- "notes": entry.get("description", ""),
- "docs_url": entry.get("source_url", base_url),
- "free": True,
- "endpoints": {},
- }
-
- is_valid, message = self.resource_manager.validate_provider(provider_data)
- if not is_valid:
- logger.debug("Skipping provider %s: %s", provider_id, message)
- continue
-
- await asyncio.to_thread(self.resource_manager.add_provider, provider_data)
- persisted.append(
- DiscoveryResult(
- provider_id=provider_id,
- name=provider_data["name"],
- category=provider_data["category"],
- base_url=provider_data["base_url"],
- requires_auth=provider_data["requires_auth"],
- description=provider_data["notes"],
- source_url=provider_data["docs_url"],
- )
- )
-
- if persisted:
- await asyncio.to_thread(self.resource_manager.save_resources)
- await asyncio.to_thread(self.provider_manager.load_config)
- logger.info("Persisted %s new providers.", len(persisted))
-
- return persisted
-
- @staticmethod
- def _normalize_id(raw_value: Optional[str]) -> str:
- """تبدیل نام به شناسهٔ مناسب."""
- if not raw_value:
- return "unknown_provider"
- cleaned = re.sub(r"[^a-zA-Z0-9]+", "_", raw_value).strip("_").lower()
- return cleaned or "unknown_provider"
-
diff --git a/app/backend/services/connection_manager.py b/app/backend/services/connection_manager.py
deleted file mode 100644
index 600940b1c712dbefd0884195eb8151e21fd8346f..0000000000000000000000000000000000000000
--- a/app/backend/services/connection_manager.py
+++ /dev/null
@@ -1,274 +0,0 @@
-"""
-Connection Manager - مدیریت اتصالات WebSocket و Session
-"""
-import asyncio
-import json
-import uuid
-from typing import Dict, Set, Optional, Any
-from datetime import datetime
-from dataclasses import dataclass, asdict
-from fastapi import WebSocket
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class ClientSession:
- """اطلاعات Session کلاینت"""
- session_id: str
- client_type: str # 'browser', 'api', 'mobile'
- connected_at: datetime
- last_activity: datetime
- ip_address: Optional[str] = None
- user_agent: Optional[str] = None
- metadata: Dict[str, Any] = None
-
- def to_dict(self):
- return {
- 'session_id': self.session_id,
- 'client_type': self.client_type,
- 'connected_at': self.connected_at.isoformat(),
- 'last_activity': self.last_activity.isoformat(),
- 'ip_address': self.ip_address,
- 'user_agent': self.user_agent,
- 'metadata': self.metadata or {}
- }
-
-
-class ConnectionManager:
- """مدیر اتصالات WebSocket و Session"""
-
- def __init__(self):
- # WebSocket connections
- self.active_connections: Dict[str, WebSocket] = {}
-
- # Sessions (برای همه انواع کلاینتها)
- self.sessions: Dict[str, ClientSession] = {}
-
- # Subscription groups (برای broadcast انتخابی)
- self.subscriptions: Dict[str, Set[str]] = {
- 'market': set(),
- 'prices': set(),
- 'news': set(),
- 'alerts': set(),
- 'all': set()
- }
-
- # Statistics
- self.total_connections = 0
- self.total_messages_sent = 0
- self.total_messages_received = 0
-
- async def connect(
- self,
- websocket: WebSocket,
- client_type: str = 'browser',
- metadata: Optional[Dict] = None
- ) -> str:
- """
- اتصال کلاینت جدید
-
- Returns:
- session_id
- """
- await websocket.accept()
-
- session_id = str(uuid.uuid4())
-
- # ذخیره WebSocket
- self.active_connections[session_id] = websocket
-
- # ایجاد Session
- session = ClientSession(
- session_id=session_id,
- client_type=client_type,
- connected_at=datetime.now(),
- last_activity=datetime.now(),
- metadata=metadata or {}
- )
- self.sessions[session_id] = session
-
- # Subscribe به گروه all
- self.subscriptions['all'].add(session_id)
-
- self.total_connections += 1
-
- logger.info(f"Client connected: {session_id} ({client_type})")
-
- # اطلاع به همه از تعداد کاربران آنلاین
- await self.broadcast_stats()
-
- return session_id
-
- def disconnect(self, session_id: str):
- """قطع اتصال کلاینت"""
- # حذف WebSocket
- if session_id in self.active_connections:
- del self.active_connections[session_id]
-
- # حذف از subscriptions
- for group in self.subscriptions.values():
- group.discard(session_id)
-
- # حذف session
- if session_id in self.sessions:
- del self.sessions[session_id]
-
- logger.info(f"Client disconnected: {session_id}")
-
- # اطلاع به همه
- asyncio.create_task(self.broadcast_stats())
-
- async def send_personal_message(
- self,
- message: Dict[str, Any],
- session_id: str
- ):
- """ارسال پیام به یک کلاینت خاص"""
- if session_id in self.active_connections:
- try:
- websocket = self.active_connections[session_id]
- await websocket.send_json(message)
-
- # بهروزرسانی آخرین فعالیت
- if session_id in self.sessions:
- self.sessions[session_id].last_activity = datetime.now()
-
- self.total_messages_sent += 1
-
- except Exception as e:
- logger.error(f"Error sending message to {session_id}: {e}")
- self.disconnect(session_id)
-
- async def broadcast(
- self,
- message: Dict[str, Any],
- group: str = 'all'
- ):
- """ارسال پیام به گروهی از کلاینتها"""
- if group not in self.subscriptions:
- group = 'all'
-
- session_ids = self.subscriptions[group].copy()
-
- disconnected = []
- for session_id in session_ids:
- if session_id in self.active_connections:
- try:
- websocket = self.active_connections[session_id]
- await websocket.send_json(message)
- self.total_messages_sent += 1
- except Exception as e:
- logger.error(f"Error broadcasting to {session_id}: {e}")
- disconnected.append(session_id)
-
- # پاکسازی اتصالات قطع شده
- for session_id in disconnected:
- self.disconnect(session_id)
-
- async def broadcast_stats(self):
- """ارسال آمار کلی به همه کلاینتها"""
- stats = self.get_stats()
- await self.broadcast({
- 'type': 'stats_update',
- 'data': stats,
- 'timestamp': datetime.now().isoformat()
- })
-
- def subscribe(self, session_id: str, group: str):
- """اضافه کردن به گروه subscription"""
- if group in self.subscriptions:
- self.subscriptions[group].add(session_id)
- logger.info(f"Session {session_id} subscribed to {group}")
- return True
- return False
-
- def unsubscribe(self, session_id: str, group: str):
- """حذف از گروه subscription"""
- if group in self.subscriptions:
- self.subscriptions[group].discard(session_id)
- logger.info(f"Session {session_id} unsubscribed from {group}")
- return True
- return False
-
- def get_stats(self) -> Dict[str, Any]:
- """دریافت آمار اتصالات"""
- # تفکیک بر اساس نوع کلاینت
- client_types = {}
- for session in self.sessions.values():
- client_type = session.client_type
- client_types[client_type] = client_types.get(client_type, 0) + 1
-
- # آمار subscriptions
- subscription_stats = {
- group: len(members)
- for group, members in self.subscriptions.items()
- }
-
- return {
- 'active_connections': len(self.active_connections),
- 'total_sessions': len(self.sessions),
- 'total_connections_ever': self.total_connections,
- 'messages_sent': self.total_messages_sent,
- 'messages_received': self.total_messages_received,
- 'client_types': client_types,
- 'subscriptions': subscription_stats,
- 'timestamp': datetime.now().isoformat()
- }
-
- def get_sessions(self) -> Dict[str, Dict[str, Any]]:
- """دریافت لیست sessionهای فعال"""
- return {
- sid: session.to_dict()
- for sid, session in self.sessions.items()
- }
-
- async def send_market_update(self, data: Dict[str, Any]):
- """ارسال بهروزرسانی بازار"""
- await self.broadcast({
- 'type': 'market_update',
- 'data': data,
- 'timestamp': datetime.now().isoformat()
- }, group='market')
-
- async def send_price_update(self, symbol: str, price: float, change: float):
- """ارسال بهروزرسانی قیمت"""
- await self.broadcast({
- 'type': 'price_update',
- 'data': {
- 'symbol': symbol,
- 'price': price,
- 'change_24h': change
- },
- 'timestamp': datetime.now().isoformat()
- }, group='prices')
-
- async def send_alert(self, alert_type: str, message: str, severity: str = 'info'):
- """ارسال هشدار"""
- await self.broadcast({
- 'type': 'alert',
- 'data': {
- 'alert_type': alert_type,
- 'message': message,
- 'severity': severity
- },
- 'timestamp': datetime.now().isoformat()
- }, group='alerts')
-
- async def heartbeat(self):
- """ارسال heartbeat برای check کردن اتصالات"""
- await self.broadcast({
- 'type': 'heartbeat',
- 'timestamp': datetime.now().isoformat()
- })
-
-
-# Global instance
-connection_manager = ConnectionManager()
-
-
-def get_connection_manager() -> ConnectionManager:
- """دریافت instance مدیر اتصالات"""
- return connection_manager
-
diff --git a/app/backend/services/diagnostics_service.py b/app/backend/services/diagnostics_service.py
deleted file mode 100644
index c9ccddbab55917b1dd57cac7ad43ea5fd3d5561f..0000000000000000000000000000000000000000
--- a/app/backend/services/diagnostics_service.py
+++ /dev/null
@@ -1,391 +0,0 @@
-"""
-Diagnostics & Auto-Repair Service
-----------------------------------
-سرویس اشکالیابی خودکار و تعمیر مشکلات سیستم
-"""
-
-import asyncio
-import logging
-import os
-import subprocess
-import sys
-from dataclasses import dataclass, asdict
-from datetime import datetime
-from typing import Any, Dict, List, Optional, Tuple
-import json
-import importlib.util
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class DiagnosticIssue:
- """یک مشکل شناسایی شده"""
- severity: str # critical, warning, info
- category: str # dependency, config, network, service, model
- title: str
- description: str
- fixable: bool
- fix_action: Optional[str] = None
- auto_fixed: bool = False
- timestamp: str = None
-
- def __post_init__(self):
- if self.timestamp is None:
- self.timestamp = datetime.now().isoformat()
-
-
-@dataclass
-class DiagnosticReport:
- """گزارش کامل اشکالیابی"""
- timestamp: str
- total_issues: int
- critical_issues: int
- warnings: int
- info_issues: int
- issues: List[DiagnosticIssue]
- fixed_issues: List[DiagnosticIssue]
- system_info: Dict[str, Any]
- duration_ms: float
-
-
-class DiagnosticsService:
- """سرویس اشکالیابی و تعمیر خودکار"""
-
- def __init__(self, resource_manager=None, provider_manager=None, auto_discovery_service=None):
- self.resource_manager = resource_manager
- self.provider_manager = provider_manager
- self.auto_discovery_service = auto_discovery_service
- self.last_report: Optional[DiagnosticReport] = None
-
- async def run_full_diagnostics(self, auto_fix: bool = False) -> DiagnosticReport:
- """اجرای کامل اشکالیابی"""
- start_time = datetime.now()
- issues: List[DiagnosticIssue] = []
- fixed_issues: List[DiagnosticIssue] = []
-
- # بررسی وابستگیها
- issues.extend(await self._check_dependencies())
-
- # بررسی تنظیمات
- issues.extend(await self._check_configuration())
-
- # بررسی شبکه
- issues.extend(await self._check_network())
-
- # بررسی سرویسها
- issues.extend(await self._check_services())
-
- # بررسی مدلها
- issues.extend(await self._check_models())
-
- # بررسی فایلها و دایرکتوریها
- issues.extend(await self._check_filesystem())
-
- # اجرای تعمیر خودکار
- if auto_fix:
- for issue in issues:
- if issue.fixable and issue.fix_action:
- fixed = await self._apply_fix(issue)
- if fixed:
- issue.auto_fixed = True
- fixed_issues.append(issue)
-
- # محاسبه آمار
- critical = sum(1 for i in issues if i.severity == 'critical')
- warnings = sum(1 for i in issues if i.severity == 'warning')
- info_count = sum(1 for i in issues if i.severity == 'info')
-
- duration_ms = (datetime.now() - start_time).total_seconds() * 1000
-
- report = DiagnosticReport(
- timestamp=datetime.now().isoformat(),
- total_issues=len(issues),
- critical_issues=critical,
- warnings=warnings,
- info_issues=info_count,
- issues=issues,
- fixed_issues=fixed_issues,
- system_info=await self._get_system_info(),
- duration_ms=duration_ms
- )
-
- self.last_report = report
- return report
-
- async def _check_dependencies(self) -> List[DiagnosticIssue]:
- """بررسی وابستگیهای Python"""
- issues = []
- required_packages = {
- 'fastapi': 'FastAPI',
- 'uvicorn': 'Uvicorn',
- 'httpx': 'HTTPX',
- 'pydantic': 'Pydantic',
- 'duckduckgo_search': 'DuckDuckGo Search',
- 'huggingface_hub': 'HuggingFace Hub',
- 'transformers': 'Transformers',
- }
-
- for package, name in required_packages.items():
- try:
- spec = importlib.util.find_spec(package)
- if spec is None:
- issues.append(DiagnosticIssue(
- severity='critical' if package in ['fastapi', 'uvicorn'] else 'warning',
- category='dependency',
- title=f'بسته {name} نصب نشده است',
- description=f'بسته {package} مورد نیاز است اما نصب نشده است.',
- fixable=True,
- fix_action=f'pip install {package}'
- ))
- except Exception as e:
- issues.append(DiagnosticIssue(
- severity='warning',
- category='dependency',
- title=f'خطا در بررسی {name}',
- description=f'خطا در بررسی بسته {package}: {str(e)}',
- fixable=False
- ))
-
- return issues
-
- async def _check_configuration(self) -> List[DiagnosticIssue]:
- """بررسی تنظیمات"""
- issues = []
-
- # بررسی متغیرهای محیطی مهم
- important_env_vars = {
- 'HF_API_TOKEN': ('warning', 'توکن HuggingFace برای استفاده از مدلها'),
- }
-
- for var, (severity, desc) in important_env_vars.items():
- if not os.getenv(var):
- issues.append(DiagnosticIssue(
- severity=severity,
- category='config',
- title=f'متغیر محیطی {var} تنظیم نشده',
- description=desc,
- fixable=False
- ))
-
- # بررسی فایلهای پیکربندی
- config_files = ['resources.json', 'config.json']
- for config_file in config_files:
- if not os.path.exists(config_file):
- issues.append(DiagnosticIssue(
- severity='info',
- category='config',
- title=f'فایل پیکربندی {config_file} وجود ندارد',
- description=f'فایل {config_file} یافت نشد. ممکن است به صورت خودکار ساخته شود.',
- fixable=False
- ))
-
- return issues
-
- async def _check_network(self) -> List[DiagnosticIssue]:
- """بررسی اتصال شبکه"""
- issues = []
- import httpx
-
- test_urls = [
- ('https://api.coingecko.com/api/v3/ping', 'CoinGecko API'),
- ('https://api.huggingface.co', 'HuggingFace API'),
- ]
-
- for url, name in test_urls:
- try:
- async with httpx.AsyncClient(timeout=5.0) as client:
- response = await client.get(url)
- if response.status_code >= 400:
- issues.append(DiagnosticIssue(
- severity='warning',
- category='network',
- title=f'مشکل در اتصال به {name}',
- description=f'درخواست به {url} با کد {response.status_code} پاسخ داد.',
- fixable=False
- ))
- except Exception as e:
- issues.append(DiagnosticIssue(
- severity='warning',
- category='network',
- title=f'عدم دسترسی به {name}',
- description=f'خطا در اتصال به {url}: {str(e)}',
- fixable=False
- ))
-
- return issues
-
- async def _check_services(self) -> List[DiagnosticIssue]:
- """بررسی سرویسها"""
- issues = []
-
- # بررسی Auto-Discovery Service
- if self.auto_discovery_service:
- status = self.auto_discovery_service.get_status()
- if not status.get('enabled'):
- issues.append(DiagnosticIssue(
- severity='info',
- category='service',
- title='سرویس Auto-Discovery غیرفعال است',
- description='سرویس جستجوی خودکار منابع غیرفعال است.',
- fixable=False
- ))
- elif not status.get('model'):
- issues.append(DiagnosticIssue(
- severity='warning',
- category='service',
- title='مدل HuggingFace برای Auto-Discovery تنظیم نشده',
- description='سرویس Auto-Discovery بدون مدل HuggingFace کار میکند.',
- fixable=False
- ))
-
- # بررسی Provider Manager
- if self.provider_manager:
- stats = self.provider_manager.get_all_stats()
- summary = stats.get('summary', {})
- if summary.get('online', 0) == 0 and summary.get('total_providers', 0) > 0:
- issues.append(DiagnosticIssue(
- severity='critical',
- category='service',
- title='هیچ Provider آنلاینی وجود ندارد',
- description='تمام Providerها آفلاین هستند.',
- fixable=False
- ))
-
- return issues
-
- async def _check_models(self) -> List[DiagnosticIssue]:
- """بررسی وضعیت مدلهای HuggingFace"""
- issues = []
-
- try:
- from huggingface_hub import InferenceClient, HfApi
- api = HfApi()
-
- # بررسی مدلهای استفاده شده
- models_to_check = [
- 'HuggingFaceH4/zephyr-7b-beta',
- 'cardiffnlp/twitter-roberta-base-sentiment-latest',
- ]
-
- for model_id in models_to_check:
- try:
- model_info = api.model_info(model_id, timeout=5.0)
- if not model_info:
- issues.append(DiagnosticIssue(
- severity='warning',
- category='model',
- title=f'مدل {model_id} در دسترس نیست',
- description=f'نمیتوان به اطلاعات مدل {model_id} دسترسی پیدا کرد.',
- fixable=False
- ))
- except Exception as e:
- issues.append(DiagnosticIssue(
- severity='warning',
- category='model',
- title=f'خطا در بررسی مدل {model_id}',
- description=f'خطا: {str(e)}',
- fixable=False
- ))
- except ImportError:
- issues.append(DiagnosticIssue(
- severity='info',
- category='model',
- title='بسته huggingface_hub نصب نشده',
- description='برای بررسی مدلها نیاز به نصب huggingface_hub است.',
- fixable=True,
- fix_action='pip install huggingface_hub'
- ))
-
- return issues
-
- async def _check_filesystem(self) -> List[DiagnosticIssue]:
- """بررسی فایل سیستم"""
- issues = []
-
- # بررسی دایرکتوریهای مهم
- important_dirs = ['static', 'static/css', 'static/js', 'backend', 'backend/services']
- for dir_path in important_dirs:
- if not os.path.exists(dir_path):
- issues.append(DiagnosticIssue(
- severity='warning',
- category='filesystem',
- title=f'دایرکتوری {dir_path} وجود ندارد',
- description=f'دایرکتوری {dir_path} یافت نشد.',
- fixable=True,
- fix_action=f'mkdir -p {dir_path}'
- ))
-
- # بررسی فایلهای مهم
- important_files = [
- 'api_server_extended.py',
- 'unified_dashboard.html',
- 'static/js/websocket-client.js',
- 'static/css/connection-status.css',
- ]
- for file_path in important_files:
- if not os.path.exists(file_path):
- issues.append(DiagnosticIssue(
- severity='critical' if 'api_server' in file_path else 'warning',
- category='filesystem',
- title=f'فایل {file_path} وجود ندارد',
- description=f'فایل {file_path} یافت نشد.',
- fixable=False
- ))
-
- return issues
-
- async def _apply_fix(self, issue: DiagnosticIssue) -> bool:
- """اعمال تعمیر خودکار"""
- if not issue.fixable or not issue.fix_action:
- return False
-
- try:
- if issue.fix_action.startswith('pip install'):
- # نصب بسته
- package = issue.fix_action.replace('pip install', '').strip()
- result = subprocess.run(
- [sys.executable, '-m', 'pip', 'install', package],
- capture_output=True,
- text=True,
- timeout=60
- )
- if result.returncode == 0:
- logger.info(f'✅ بسته {package} با موفقیت نصب شد')
- return True
- else:
- logger.error(f'❌ خطا در نصب {package}: {result.stderr}')
- return False
-
- elif issue.fix_action.startswith('mkdir'):
- # ساخت دایرکتوری
- dir_path = issue.fix_action.replace('mkdir -p', '').strip()
- os.makedirs(dir_path, exist_ok=True)
- logger.info(f'✅ دایرکتوری {dir_path} ساخته شد')
- return True
-
- else:
- logger.warning(f'⚠️ عمل تعمیر ناشناخته: {issue.fix_action}')
- return False
-
- except Exception as e:
- logger.error(f'❌ خطا در اعمال تعمیر: {e}')
- return False
-
- async def _get_system_info(self) -> Dict[str, Any]:
- """دریافت اطلاعات سیستم"""
- import platform
- return {
- 'python_version': sys.version,
- 'platform': platform.platform(),
- 'architecture': platform.architecture(),
- 'processor': platform.processor(),
- 'cwd': os.getcwd(),
- }
-
- def get_last_report(self) -> Optional[Dict[str, Any]]:
- """دریافت آخرین گزارش"""
- if self.last_report:
- return asdict(self.last_report)
- return None
-
diff --git a/app/backend/services/hf_client.py b/app/backend/services/hf_client.py
deleted file mode 100644
index 2171e04dff6688415c689c928accadafd9c2c415..0000000000000000000000000000000000000000
--- a/app/backend/services/hf_client.py
+++ /dev/null
@@ -1,36 +0,0 @@
-from __future__ import annotations
-from typing import List, Dict, Any
-import os
-from functools import lru_cache
-
-ENABLE_SENTIMENT = os.getenv("ENABLE_SENTIMENT", "true").lower() in ("1","true","yes")
-SOCIAL_MODEL = os.getenv("SENTIMENT_SOCIAL_MODEL", "ElKulako/cryptobert")
-NEWS_MODEL = os.getenv("SENTIMENT_NEWS_MODEL", "kk08/CryptoBERT")
-
-
-@lru_cache(maxsize=4)
-def _pl(model_name: str):
- if not ENABLE_SENTIMENT:
- return None
- from transformers import pipeline
- return pipeline("sentiment-analysis", model=model_name)
-
-
-def _label_to_score(lbl: str) -> float:
- l = (lbl or "").lower()
- if "bear" in l or "neg" in l or "label_0" in l: return -1.0
- if "bull" in l or "pos" in l or "label_1" in l: return 1.0
- return 0.0
-
-
-def run_sentiment(texts: List[str], model: str | None = None) -> Dict[str, Any]:
- if not ENABLE_SENTIMENT:
- return {"enabled": False, "vote": 0.0, "samples": []}
- name = model or SOCIAL_MODEL
- pl = _pl(name)
- if not pl:
- return {"enabled": False, "vote": 0.0, "samples": []}
- preds = pl(texts)
- scores = [_label_to_score(p.get("label","")) * float(p.get("score",0)) for p in preds]
- vote = sum(scores) / max(1, len(scores))
- return {"enabled": True, "model": name, "vote": vote, "samples": preds}
diff --git a/app/backend/services/hf_registry.py b/app/backend/services/hf_registry.py
deleted file mode 100644
index 56f71ae073cbf0d0b92048ded0b88867c47abb53..0000000000000000000000000000000000000000
--- a/app/backend/services/hf_registry.py
+++ /dev/null
@@ -1,113 +0,0 @@
-from __future__ import annotations
-import os
-import time
-import random
-from typing import Dict, Any, List, Literal, Optional
-import httpx
-
-HF_API_MODELS = "https://huggingface.co/api/models"
-HF_API_DATASETS = "https://huggingface.co/api/datasets"
-
-REFRESH_INTERVAL_SEC = int(os.getenv("HF_REGISTRY_REFRESH_SEC", "21600")) # 6h
-HTTP_TIMEOUT = float(os.getenv("HF_HTTP_TIMEOUT", "8.0"))
-
-_SEED_MODELS = [
- "ElKulako/cryptobert",
- "kk08/CryptoBERT",
-]
-_SEED_DATASETS = [
- "linxy/CryptoCoin",
- "WinkingFace/CryptoLM-Bitcoin-BTC-USDT",
- "WinkingFace/CryptoLM-Ethereum-ETH-USDT",
- "WinkingFace/CryptoLM-Solana-SOL-USDT",
- "WinkingFace/CryptoLM-Ripple-XRP-USDT",
-]
-
-
-class HFRegistry:
- def __init__(self) -> None:
- self.models: Dict[str, Dict[str, Any]] = {}
- self.datasets: Dict[str, Dict[str, Any]] = {}
- self.last_refresh: float = 0.0
- self.fail_reason: Optional[str] = None
-
- async def _hf_json(self, url: str, params: Dict[str, Any]) -> Any:
- async with httpx.AsyncClient(timeout=HTTP_TIMEOUT) as client:
- r = await client.get(url, params=params)
- r.raise_for_status()
- return r.json()
-
- async def refresh(self) -> Dict[str, Any]:
- try:
- for name in _SEED_MODELS:
- self.models.setdefault(name, {"id": name, "source": "seed", "pipeline_tag": "sentiment-analysis"})
- for name in _SEED_DATASETS:
- self.datasets.setdefault(name, {"id": name, "source": "seed"})
-
- q_crypto = {"search": "crypto", "limit": 50}
- q_sent = {"pipeline_tag": "sentiment-analysis", "search": "crypto", "limit": 50}
-
- models = await self._hf_json(HF_API_MODELS, q_sent)
- for m in models or []:
- mid = m.get("modelId") or m.get("id") or m.get("name")
- if not mid: continue
- self.models[mid] = {
- "id": mid,
- "pipeline_tag": m.get("pipeline_tag"),
- "likes": m.get("likes"),
- "downloads": m.get("downloads"),
- "tags": m.get("tags") or [],
- "source": "hub",
- }
-
- datasets = await self._hf_json(HF_API_DATASETS, q_crypto)
- for d in datasets or []:
- did = d.get("id") or d.get("name")
- if not did: continue
- self.datasets[did] = {
- "id": did,
- "likes": d.get("likes"),
- "downloads": d.get("downloads"),
- "tags": d.get("tags") or [],
- "source": "hub",
- }
-
- self.last_refresh = time.time()
- self.fail_reason = None
- return {"ok": True, "models": len(self.models), "datasets": len(self.datasets)}
- except Exception as e:
- self.fail_reason = str(e)
- return {"ok": False, "error": self.fail_reason, "models": len(self.models), "datasets": len(self.datasets)}
-
- def list(self, kind: Literal["models","datasets"]="models") -> List[Dict[str, Any]]:
- return list(self.models.values()) if kind == "models" else list(self.datasets.values())
-
- def health(self) -> Dict[str, Any]:
- age = time.time() - (self.last_refresh or 0)
- return {
- "ok": self.last_refresh > 0 and (self.fail_reason is None),
- "last_refresh_epoch": self.last_refresh,
- "age_sec": age,
- "fail_reason": self.fail_reason,
- "counts": {"models": len(self.models), "datasets": len(self.datasets)},
- "interval_sec": REFRESH_INTERVAL_SEC,
- }
-
-
-REGISTRY = HFRegistry()
-
-
-async def periodic_refresh(loop_sleep: int = REFRESH_INTERVAL_SEC) -> None:
- await REGISTRY.refresh()
- await _sleep(int(loop_sleep * random.uniform(0.5, 0.9)))
- while True:
- await REGISTRY.refresh()
- await _sleep(loop_sleep)
-
-
-async def _sleep(sec: int) -> None:
- import asyncio
- try:
- await asyncio.sleep(sec)
- except Exception:
- pass
diff --git a/app/backend/services/persistence_service.py b/app/backend/services/persistence_service.py
deleted file mode 100644
index 535bd6635335073a1a18ba54e006c3334ab83268..0000000000000000000000000000000000000000
--- a/app/backend/services/persistence_service.py
+++ /dev/null
@@ -1,503 +0,0 @@
-"""
-Persistence Service
-Handles data persistence with multiple export formats (JSON, CSV, database)
-"""
-import json
-import csv
-import logging
-from typing import Dict, Any, List, Optional
-from datetime import datetime, timedelta
-from pathlib import Path
-import asyncio
-from collections import defaultdict
-import pandas as pd
-
-logger = logging.getLogger(__name__)
-
-
-class PersistenceService:
- """Service for persisting data in multiple formats"""
-
- def __init__(self, db_manager=None, data_dir: str = 'data'):
- self.db_manager = db_manager
- self.data_dir = Path(data_dir)
- self.data_dir.mkdir(parents=True, exist_ok=True)
-
- # In-memory cache for quick access
- self.cache: Dict[str, Any] = {}
- self.history: Dict[str, List[Dict[str, Any]]] = defaultdict(list)
- self.max_history_per_api = 1000 # Keep last 1000 records per API
-
- async def save_api_data(
- self,
- api_id: str,
- data: Dict[str, Any],
- metadata: Optional[Dict[str, Any]] = None
- ) -> bool:
- """
- Save API data with metadata
-
- Args:
- api_id: API identifier
- data: Data to save
- metadata: Additional metadata (category, source, etc.)
-
- Returns:
- Success status
- """
- try:
- timestamp = datetime.now()
-
- # Create data record
- record = {
- 'api_id': api_id,
- 'timestamp': timestamp.isoformat(),
- 'data': data,
- 'metadata': metadata or {}
- }
-
- # Update cache
- self.cache[api_id] = record
-
- # Add to history
- self.history[api_id].append(record)
-
- # Trim history if needed
- if len(self.history[api_id]) > self.max_history_per_api:
- self.history[api_id] = self.history[api_id][-self.max_history_per_api:]
-
- # Save to database if available
- if self.db_manager:
- await self._save_to_database(api_id, data, metadata, timestamp)
-
- logger.debug(f"Saved data for {api_id}")
- return True
-
- except Exception as e:
- logger.error(f"Error saving data for {api_id}: {e}")
- return False
-
- async def _save_to_database(
- self,
- api_id: str,
- data: Dict[str, Any],
- metadata: Dict[str, Any],
- timestamp: datetime
- ):
- """Save data to database"""
- if not self.db_manager:
- return
-
- try:
- # Save using database manager methods
- category = metadata.get('category', 'unknown')
-
- with self.db_manager.get_session() as session:
- # Find or create provider
- from database.models import Provider, DataCollection
-
- provider = session.query(Provider).filter_by(name=api_id).first()
-
- if not provider:
- # Create new provider
- provider = Provider(
- name=api_id,
- category=category,
- endpoint_url=metadata.get('url', ''),
- requires_key=metadata.get('requires_key', False),
- priority_tier=metadata.get('priority', 3)
- )
- session.add(provider)
- session.flush()
-
- # Create data collection record
- collection = DataCollection(
- provider_id=provider.id,
- category=category,
- scheduled_time=timestamp,
- actual_fetch_time=timestamp,
- data_timestamp=timestamp,
- staleness_minutes=0,
- record_count=len(data) if isinstance(data, (list, dict)) else 1,
- payload_size_bytes=len(json.dumps(data)),
- on_schedule=True
- )
- session.add(collection)
-
- except Exception as e:
- logger.error(f"Error saving to database: {e}")
-
- def get_cached_data(self, api_id: str) -> Optional[Dict[str, Any]]:
- """Get cached data for an API"""
- return self.cache.get(api_id)
-
- def get_all_cached_data(self) -> Dict[str, Any]:
- """Get all cached data"""
- return self.cache.copy()
-
- def get_history(self, api_id: str, limit: int = 100) -> List[Dict[str, Any]]:
- """Get historical data for an API"""
- history = self.history.get(api_id, [])
- return history[-limit:] if limit else history
-
- def get_all_history(self) -> Dict[str, List[Dict[str, Any]]]:
- """Get all historical data"""
- return dict(self.history)
-
- async def export_to_json(
- self,
- filepath: str,
- api_ids: Optional[List[str]] = None,
- include_history: bool = False
- ) -> bool:
- """
- Export data to JSON file
-
- Args:
- filepath: Output file path
- api_ids: Specific APIs to export (None = all)
- include_history: Include historical data
-
- Returns:
- Success status
- """
- try:
- filepath = Path(filepath)
- filepath.parent.mkdir(parents=True, exist_ok=True)
-
- # Prepare data
- if include_history:
- data = {
- 'cache': self.cache,
- 'history': dict(self.history),
- 'exported_at': datetime.now().isoformat()
- }
- else:
- data = {
- 'cache': self.cache,
- 'exported_at': datetime.now().isoformat()
- }
-
- # Filter by API IDs if specified
- if api_ids:
- if 'cache' in data:
- data['cache'] = {k: v for k, v in data['cache'].items() if k in api_ids}
- if 'history' in data:
- data['history'] = {k: v for k, v in data['history'].items() if k in api_ids}
-
- # Write to file
- with open(filepath, 'w', encoding='utf-8') as f:
- json.dump(data, f, indent=2, default=str)
-
- logger.info(f"Exported data to JSON: {filepath}")
- return True
-
- except Exception as e:
- logger.error(f"Error exporting to JSON: {e}")
- return False
-
- async def export_to_csv(
- self,
- filepath: str,
- api_ids: Optional[List[str]] = None,
- flatten: bool = True
- ) -> bool:
- """
- Export data to CSV file
-
- Args:
- filepath: Output file path
- api_ids: Specific APIs to export (None = all)
- flatten: Flatten nested data structures
-
- Returns:
- Success status
- """
- try:
- filepath = Path(filepath)
- filepath.parent.mkdir(parents=True, exist_ok=True)
-
- # Prepare rows
- rows = []
-
- cache_items = self.cache.items()
- if api_ids:
- cache_items = [(k, v) for k, v in cache_items if k in api_ids]
-
- for api_id, record in cache_items:
- row = {
- 'api_id': api_id,
- 'timestamp': record.get('timestamp'),
- 'category': record.get('metadata', {}).get('category', ''),
- }
-
- # Flatten data if requested
- if flatten:
- data = record.get('data', {})
- if isinstance(data, dict):
- for key, value in data.items():
- # Simple flattening - only first level
- if isinstance(value, (str, int, float, bool)):
- row[f'data_{key}'] = value
- else:
- row[f'data_{key}'] = json.dumps(value)
- else:
- row['data'] = json.dumps(record.get('data'))
-
- rows.append(row)
-
- # Write CSV
- if rows:
- df = pd.DataFrame(rows)
- df.to_csv(filepath, index=False)
- logger.info(f"Exported data to CSV: {filepath}")
- return True
- else:
- logger.warning("No data to export to CSV")
- return False
-
- except Exception as e:
- logger.error(f"Error exporting to CSV: {e}")
- return False
-
- async def export_history_to_csv(
- self,
- filepath: str,
- api_id: str
- ) -> bool:
- """
- Export historical data for a specific API to CSV
-
- Args:
- filepath: Output file path
- api_id: API identifier
-
- Returns:
- Success status
- """
- try:
- filepath = Path(filepath)
- filepath.parent.mkdir(parents=True, exist_ok=True)
-
- history = self.history.get(api_id, [])
-
- if not history:
- logger.warning(f"No history data for {api_id}")
- return False
-
- # Prepare rows
- rows = []
- for record in history:
- row = {
- 'timestamp': record.get('timestamp'),
- 'api_id': record.get('api_id'),
- 'data': json.dumps(record.get('data'))
- }
- rows.append(row)
-
- # Write CSV
- df = pd.DataFrame(rows)
- df.to_csv(filepath, index=False)
-
- logger.info(f"Exported history for {api_id} to CSV: {filepath}")
- return True
-
- except Exception as e:
- logger.error(f"Error exporting history to CSV: {e}")
- return False
-
- async def import_from_json(self, filepath: str) -> bool:
- """
- Import data from JSON file
-
- Args:
- filepath: Input file path
-
- Returns:
- Success status
- """
- try:
- filepath = Path(filepath)
-
- with open(filepath, 'r', encoding='utf-8') as f:
- data = json.load(f)
-
- # Import cache
- if 'cache' in data:
- self.cache.update(data['cache'])
-
- # Import history
- if 'history' in data:
- for api_id, records in data['history'].items():
- self.history[api_id].extend(records)
-
- # Trim if needed
- if len(self.history[api_id]) > self.max_history_per_api:
- self.history[api_id] = self.history[api_id][-self.max_history_per_api:]
-
- logger.info(f"Imported data from JSON: {filepath}")
- return True
-
- except Exception as e:
- logger.error(f"Error importing from JSON: {e}")
- return False
-
- async def backup_all_data(self, backup_dir: Optional[str] = None) -> str:
- """
- Create a backup of all data
-
- Args:
- backup_dir: Backup directory (uses default if None)
-
- Returns:
- Path to backup file
- """
- try:
- if backup_dir:
- backup_path = Path(backup_dir)
- else:
- backup_path = self.data_dir / 'backups'
-
- backup_path.mkdir(parents=True, exist_ok=True)
-
- # Create backup filename with timestamp
- timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
- backup_file = backup_path / f'backup_{timestamp}.json'
-
- # Export everything
- await self.export_to_json(
- str(backup_file),
- include_history=True
- )
-
- logger.info(f"Created backup: {backup_file}")
- return str(backup_file)
-
- except Exception as e:
- logger.error(f"Error creating backup: {e}")
- raise
-
- async def restore_from_backup(self, backup_file: str) -> bool:
- """
- Restore data from a backup file
-
- Args:
- backup_file: Path to backup file
-
- Returns:
- Success status
- """
- try:
- logger.info(f"Restoring from backup: {backup_file}")
- success = await self.import_from_json(backup_file)
-
- if success:
- logger.info("Backup restored successfully")
-
- return success
-
- except Exception as e:
- logger.error(f"Error restoring from backup: {e}")
- return False
-
- def clear_cache(self):
- """Clear all cached data"""
- self.cache.clear()
- logger.info("Cache cleared")
-
- def clear_history(self, api_id: Optional[str] = None):
- """Clear history for specific API or all"""
- if api_id:
- if api_id in self.history:
- del self.history[api_id]
- logger.info(f"Cleared history for {api_id}")
- else:
- self.history.clear()
- logger.info("Cleared all history")
-
- def get_statistics(self) -> Dict[str, Any]:
- """Get statistics about stored data"""
- total_cached = len(self.cache)
- total_history_records = sum(len(records) for records in self.history.values())
-
- api_stats = {}
- for api_id, records in self.history.items():
- if records:
- timestamps = [
- datetime.fromisoformat(r['timestamp'])
- for r in records
- if 'timestamp' in r
- ]
-
- if timestamps:
- api_stats[api_id] = {
- 'record_count': len(records),
- 'oldest': min(timestamps).isoformat(),
- 'newest': max(timestamps).isoformat()
- }
-
- return {
- 'cached_apis': total_cached,
- 'total_history_records': total_history_records,
- 'apis_with_history': len(self.history),
- 'api_statistics': api_stats
- }
-
- async def cleanup_old_data(self, days: int = 7) -> int:
- """
- Remove data older than specified days
-
- Args:
- days: Number of days to keep
-
- Returns:
- Number of records removed
- """
- try:
- cutoff = datetime.now() - timedelta(days=days)
- removed_count = 0
-
- for api_id, records in list(self.history.items()):
- original_count = len(records)
-
- # Filter out old records
- self.history[api_id] = [
- r for r in records
- if datetime.fromisoformat(r['timestamp']) > cutoff
- ]
-
- removed_count += original_count - len(self.history[api_id])
-
- # Remove empty histories
- if not self.history[api_id]:
- del self.history[api_id]
-
- logger.info(f"Cleaned up {removed_count} old records (older than {days} days)")
- return removed_count
-
- except Exception as e:
- logger.error(f"Error during cleanup: {e}")
- return 0
-
- async def save_collection_data(
- self,
- api_id: str,
- category: str,
- data: Dict[str, Any],
- timestamp: datetime
- ):
- """
- Save data collection (compatibility method for scheduler)
-
- Args:
- api_id: API identifier
- category: Data category
- data: Collected data
- timestamp: Collection timestamp
- """
- metadata = {
- 'category': category,
- 'collection_time': timestamp.isoformat()
- }
-
- await self.save_api_data(api_id, data, metadata)
diff --git a/app/backend/services/scheduler_service.py b/app/backend/services/scheduler_service.py
deleted file mode 100644
index 698d23860fb103ff6012b9658edb2d84a01d53a2..0000000000000000000000000000000000000000
--- a/app/backend/services/scheduler_service.py
+++ /dev/null
@@ -1,444 +0,0 @@
-"""
-Enhanced Scheduler Service
-Manages periodic and real-time data updates with persistence
-"""
-import asyncio
-import logging
-from typing import Dict, Any, List, Optional, Callable
-from datetime import datetime, timedelta
-from dataclasses import dataclass, asdict
-import json
-from collections import defaultdict
-import httpx
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class ScheduleTask:
- """Represents a scheduled task"""
- api_id: str
- name: str
- category: str
- interval: int # seconds
- update_type: str # realtime, periodic, scheduled
- enabled: bool
- last_update: Optional[datetime] = None
- next_update: Optional[datetime] = None
- last_status: Optional[str] = None # success, failed, pending
- last_data: Optional[Dict[str, Any]] = None
- error_count: int = 0
- success_count: int = 0
-
-
-class SchedulerService:
- """Advanced scheduler for managing API data updates"""
-
- def __init__(self, config_loader, db_manager=None):
- self.config_loader = config_loader
- self.db_manager = db_manager
- self.tasks: Dict[str, ScheduleTask] = {}
- self.running = False
- self.periodic_task = None
- self.realtime_tasks: Dict[str, asyncio.Task] = {}
- self.data_cache: Dict[str, Any] = {}
- self.callbacks: Dict[str, List[Callable]] = defaultdict(list)
-
- # Initialize tasks from config
- self._initialize_tasks()
-
- def _initialize_tasks(self):
- """Initialize schedule tasks from config loader"""
- apis = self.config_loader.get_all_apis()
- schedules = self.config_loader.schedules
-
- for api_id, api in apis.items():
- schedule = schedules.get(api_id, {})
-
- task = ScheduleTask(
- api_id=api_id,
- name=api.get('name', api_id),
- category=api.get('category', 'unknown'),
- interval=schedule.get('interval', 300),
- update_type=api.get('update_type', 'periodic'),
- enabled=schedule.get('enabled', True),
- next_update=datetime.now()
- )
-
- self.tasks[api_id] = task
-
- logger.info(f"Initialized {len(self.tasks)} schedule tasks")
-
- async def start(self):
- """Start the scheduler"""
- if self.running:
- logger.warning("Scheduler already running")
- return
-
- self.running = True
- logger.info("Starting scheduler...")
-
- # Start periodic update loop
- self.periodic_task = asyncio.create_task(self._periodic_update_loop())
-
- # Start real-time tasks
- await self._start_realtime_tasks()
-
- logger.info("Scheduler started successfully")
-
- async def stop(self):
- """Stop the scheduler"""
- if not self.running:
- return
-
- self.running = False
- logger.info("Stopping scheduler...")
-
- # Cancel periodic task
- if self.periodic_task:
- self.periodic_task.cancel()
- try:
- await self.periodic_task
- except asyncio.CancelledError:
- pass
-
- # Cancel real-time tasks
- for task in self.realtime_tasks.values():
- task.cancel()
-
- logger.info("Scheduler stopped")
-
- async def _periodic_update_loop(self):
- """Main loop for periodic updates"""
- while self.running:
- try:
- # Get tasks due for update
- due_tasks = self._get_due_tasks()
-
- if due_tasks:
- logger.info(f"Processing {len(due_tasks)} due tasks")
-
- # Process tasks concurrently
- await asyncio.gather(
- *[self._execute_task(task) for task in due_tasks],
- return_exceptions=True
- )
-
- # Sleep for a short interval
- await asyncio.sleep(5) # Check every 5 seconds
-
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.error(f"Error in periodic update loop: {e}")
- await asyncio.sleep(10)
-
- def _get_due_tasks(self) -> List[ScheduleTask]:
- """Get tasks that are due for update"""
- now = datetime.now()
- due_tasks = []
-
- for task in self.tasks.values():
- if not task.enabled:
- continue
-
- if task.update_type == 'realtime':
- continue # Real-time tasks handled separately
-
- if task.next_update is None or now >= task.next_update:
- due_tasks.append(task)
-
- return due_tasks
-
- async def _execute_task(self, task: ScheduleTask):
- """Execute a single scheduled task"""
- try:
- api = self.config_loader.apis.get(task.api_id)
- if not api:
- logger.error(f"API not found: {task.api_id}")
- return
-
- # Fetch data from API
- data = await self._fetch_api_data(api)
-
- # Update task status
- task.last_update = datetime.now()
- task.next_update = task.last_update + timedelta(seconds=task.interval)
- task.last_status = 'success'
- task.last_data = data
- task.success_count += 1
- task.error_count = 0 # Reset error count on success
-
- # Cache data
- self.data_cache[task.api_id] = {
- 'data': data,
- 'timestamp': datetime.now(),
- 'task': task.name
- }
-
- # Save to database if available
- if self.db_manager:
- await self._save_to_database(task, data)
-
- # Trigger callbacks
- await self._trigger_callbacks(task.api_id, data)
-
- # Mark as updated in config loader
- self.config_loader.mark_updated(task.api_id)
-
- logger.info(f"✓ Updated {task.name} ({task.category})")
-
- except Exception as e:
- logger.error(f"✗ Failed to update {task.name}: {e}")
- task.last_status = 'failed'
- task.error_count += 1
-
- # Increase interval on repeated failures
- if task.error_count >= 3:
- task.interval = min(task.interval * 2, 3600) # Max 1 hour
- logger.warning(f"Increased interval for {task.name} to {task.interval}s")
-
- async def _fetch_api_data(self, api: Dict[str, Any]) -> Dict[str, Any]:
- """Fetch data from an API"""
- base_url = api.get('base_url', '')
- auth = api.get('auth', {})
-
- # Build request URL
- url = base_url
-
- # Handle authentication
- headers = {}
- params = {}
-
- auth_type = auth.get('type', 'none')
-
- if auth_type == 'apiKey' or auth_type == 'apiKeyHeader':
- key = auth.get('key')
- header_name = auth.get('header_name', 'X-API-Key')
- if key:
- headers[header_name] = key
-
- elif auth_type == 'apiKeyQuery':
- key = auth.get('key')
- param_name = auth.get('param_name', 'apikey')
- if key:
- params[param_name] = key
-
- elif auth_type == 'apiKeyPath':
- key = auth.get('key')
- param_name = auth.get('param_name', 'API_KEY')
- if key:
- url = url.replace(f'{{{param_name}}}', key)
-
- # Make request
- timeout = httpx.Timeout(10.0)
-
- async with httpx.AsyncClient(timeout=timeout) as client:
- # Handle different endpoints
- endpoints = api.get('endpoints')
-
- if isinstance(endpoints, dict) and 'health' in endpoints:
- url = endpoints['health']
- elif isinstance(endpoints, str):
- url = endpoints
-
- # Add query params
- if params:
- url = f"{url}{'&' if '?' in url else '?'}" + '&'.join(f"{k}={v}" for k, v in params.items())
-
- response = await client.get(url, headers=headers)
- response.raise_for_status()
-
- return response.json()
-
- async def _save_to_database(self, task: ScheduleTask, data: Dict[str, Any]):
- """Save task data to database"""
- if not self.db_manager:
- return
-
- try:
- # Save using database manager
- await self.db_manager.save_collection_data(
- api_id=task.api_id,
- category=task.category,
- data=data,
- timestamp=datetime.now()
- )
- except Exception as e:
- logger.error(f"Error saving to database: {e}")
-
- async def _trigger_callbacks(self, api_id: str, data: Dict[str, Any]):
- """Trigger callbacks for API updates"""
- if api_id in self.callbacks:
- for callback in self.callbacks[api_id]:
- try:
- if asyncio.iscoroutinefunction(callback):
- await callback(api_id, data)
- else:
- callback(api_id, data)
- except Exception as e:
- logger.error(f"Error in callback for {api_id}: {e}")
-
- async def _start_realtime_tasks(self):
- """Start WebSocket connections for real-time APIs"""
- realtime_apis = self.config_loader.get_realtime_apis()
-
- for api_id, api in realtime_apis.items():
- task = self.tasks.get(api_id)
-
- if task and task.enabled:
- # Create WebSocket task
- ws_task = asyncio.create_task(self._realtime_task(task, api))
- self.realtime_tasks[api_id] = ws_task
-
- logger.info(f"Started {len(self.realtime_tasks)} real-time tasks")
-
- async def _realtime_task(self, task: ScheduleTask, api: Dict[str, Any]):
- """Handle real-time WebSocket connection"""
- # This is a placeholder - implement WebSocket connection logic
- # based on the specific API requirements
- while self.running:
- try:
- # Connect to WebSocket
- # ws_url = api.get('base_url')
- # async with websockets.connect(ws_url) as ws:
- # async for message in ws:
- # data = json.loads(message)
- # await self._handle_realtime_data(task, data)
-
- logger.info(f"Real-time task for {task.name} (placeholder)")
- await asyncio.sleep(60) # Placeholder
-
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.error(f"Error in real-time task {task.name}: {e}")
- await asyncio.sleep(30) # Retry after delay
-
- async def _handle_realtime_data(self, task: ScheduleTask, data: Dict[str, Any]):
- """Handle incoming real-time data"""
- task.last_update = datetime.now()
- task.last_status = 'success'
- task.last_data = data
- task.success_count += 1
-
- # Cache data
- self.data_cache[task.api_id] = {
- 'data': data,
- 'timestamp': datetime.now(),
- 'task': task.name
- }
-
- # Save to database
- if self.db_manager:
- await self._save_to_database(task, data)
-
- # Trigger callbacks
- await self._trigger_callbacks(task.api_id, data)
-
- def register_callback(self, api_id: str, callback: Callable):
- """Register a callback for API updates"""
- self.callbacks[api_id].append(callback)
-
- def unregister_callback(self, api_id: str, callback: Callable):
- """Unregister a callback"""
- if api_id in self.callbacks:
- self.callbacks[api_id] = [cb for cb in self.callbacks[api_id] if cb != callback]
-
- def update_task_schedule(self, api_id: str, interval: int = None, enabled: bool = None):
- """Update schedule for a task"""
- if api_id in self.tasks:
- task = self.tasks[api_id]
-
- if interval is not None:
- task.interval = interval
- self.config_loader.update_schedule(api_id, interval=interval)
-
- if enabled is not None:
- task.enabled = enabled
- self.config_loader.update_schedule(api_id, enabled=enabled)
-
- logger.info(f"Updated schedule for {task.name}")
-
- def get_task_status(self, api_id: str) -> Optional[Dict[str, Any]]:
- """Get status of a specific task"""
- task = self.tasks.get(api_id)
-
- if not task:
- return None
-
- return {
- 'api_id': task.api_id,
- 'name': task.name,
- 'category': task.category,
- 'interval': task.interval,
- 'update_type': task.update_type,
- 'enabled': task.enabled,
- 'last_update': task.last_update.isoformat() if task.last_update else None,
- 'next_update': task.next_update.isoformat() if task.next_update else None,
- 'last_status': task.last_status,
- 'success_count': task.success_count,
- 'error_count': task.error_count
- }
-
- def get_all_task_statuses(self) -> Dict[str, Any]:
- """Get status of all tasks"""
- return {
- api_id: self.get_task_status(api_id)
- for api_id in self.tasks.keys()
- }
-
- def get_cached_data(self, api_id: str) -> Optional[Dict[str, Any]]:
- """Get cached data for an API"""
- return self.data_cache.get(api_id)
-
- def get_all_cached_data(self) -> Dict[str, Any]:
- """Get all cached data"""
- return self.data_cache
-
- async def force_update(self, api_id: str) -> bool:
- """Force an immediate update for an API"""
- task = self.tasks.get(api_id)
-
- if not task:
- logger.error(f"Task not found: {api_id}")
- return False
-
- logger.info(f"Forcing update for {task.name}")
- await self._execute_task(task)
-
- return task.last_status == 'success'
-
- def export_schedules(self, filepath: str):
- """Export schedules to JSON"""
- schedules_data = {
- api_id: {
- 'name': task.name,
- 'category': task.category,
- 'interval': task.interval,
- 'update_type': task.update_type,
- 'enabled': task.enabled,
- 'last_update': task.last_update.isoformat() if task.last_update else None,
- 'success_count': task.success_count,
- 'error_count': task.error_count
- }
- for api_id, task in self.tasks.items()
- }
-
- with open(filepath, 'w') as f:
- json.dump(schedules_data, f, indent=2)
-
- logger.info(f"Exported schedules to {filepath}")
-
- def import_schedules(self, filepath: str):
- """Import schedules from JSON"""
- with open(filepath, 'r') as f:
- schedules_data = json.load(f)
-
- for api_id, schedule_data in schedules_data.items():
- if api_id in self.tasks:
- task = self.tasks[api_id]
- task.interval = schedule_data.get('interval', task.interval)
- task.enabled = schedule_data.get('enabled', task.enabled)
-
- logger.info(f"Imported schedules from {filepath}")
diff --git a/app/backend/services/unified_config_loader.py b/app/backend/services/unified_config_loader.py
deleted file mode 100644
index d2c5434095ed65de4eacafc2cb6c3f71bb74aa0b..0000000000000000000000000000000000000000
--- a/app/backend/services/unified_config_loader.py
+++ /dev/null
@@ -1,470 +0,0 @@
-"""
-Unified Configuration Loader
-Loads all APIs from JSON files at project root with scheduling and persistence support
-"""
-import json
-import os
-from typing import Dict, List, Any, Optional
-from pathlib import Path
-from datetime import datetime, timedelta
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class UnifiedConfigLoader:
- """Load and manage all API configurations from JSON files"""
-
- def __init__(self, config_dir: str = '.'):
- self.config_dir = Path(config_dir)
- self.apis: Dict[str, Dict[str, Any]] = {}
- self.keys: Dict[str, str] = {}
- self.cors_proxies: List[str] = []
- self.schedules: Dict[str, Dict[str, Any]] = {}
- self.config_files = [
- 'crypto_resources_unified_2025-11-11.json',
- 'all_apis_merged_2025.json',
- 'ultimate_crypto_pipeline_2025_NZasinich.json'
- ]
- self.load_all_configs()
-
- def load_all_configs(self):
- """Load configurations from all JSON files"""
- logger.info("Loading unified configurations...")
-
- # Load primary unified config
- self.load_unified_config()
-
- # Load merged APIs
- self.load_merged_apis()
-
- # Load pipeline config
- self.load_pipeline_config()
-
- # Setup CORS proxies
- self.setup_cors_proxies()
-
- # Setup default schedules
- self.setup_default_schedules()
-
- logger.info(f"✓ Loaded {len(self.apis)} API sources")
- logger.info(f"✓ Found {len(self.keys)} API keys")
- logger.info(f"✓ Configured {len(self.schedules)} schedules")
-
- def load_unified_config(self):
- """Load crypto_resources_unified_2025-11-11.json"""
- config_path = self.config_dir / 'crypto_resources_unified_2025-11-11.json'
-
- try:
- with open(config_path, 'r', encoding='utf-8') as f:
- data = json.load(f)
-
- registry = data.get('registry', {})
-
- # Load RPC nodes
- for entry in registry.get('rpc_nodes', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': entry.get('chain', 'rpc_nodes'),
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'role': entry.get('role', 'rpc'),
- 'priority': 1,
- 'update_type': 'realtime' if entry.get('role') == 'websocket' else 'periodic',
- 'enabled': True
- }
-
- # Extract embedded keys
- auth = entry.get('auth', {})
- if auth.get('key'):
- self.keys[api_id] = auth['key']
-
- # Load block explorers
- for entry in registry.get('block_explorers', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'blockchain_explorers',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 1,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- auth = entry.get('auth', {})
- if auth.get('key'):
- self.keys[api_id] = auth['key']
-
- # Load market data sources
- for entry in registry.get('market_data', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'market_data',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 1,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- auth = entry.get('auth', {})
- if auth.get('key'):
- self.keys[api_id] = auth['key']
-
- # Load news sources
- for entry in registry.get('news', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'news',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 2,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- # Load sentiment sources
- for entry in registry.get('sentiment', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'sentiment',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 2,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- # Load HuggingFace resources
- for entry in registry.get('huggingface', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'huggingface',
- 'base_url': entry.get('base_url', 'https://huggingface.co'),
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'resource_type': entry.get('resource_type', 'model'),
- 'priority': 2,
- 'update_type': 'scheduled', # HF should update less frequently
- 'enabled': True
- }
-
- # Load on-chain analytics
- for entry in registry.get('onchain_analytics', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'onchain_analytics',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 2,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- # Load whale tracking
- for entry in registry.get('whale_tracking', []):
- api_id = entry['id']
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry['name'],
- 'category': 'whale_tracking',
- 'base_url': entry['base_url'],
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs_url'),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes'),
- 'priority': 2,
- 'update_type': 'periodic',
- 'enabled': True
- }
-
- logger.info(f"✓ Loaded unified config with {len(self.apis)} entries")
-
- except Exception as e:
- logger.error(f"Error loading unified config: {e}")
-
- def load_merged_apis(self):
- """Load all_apis_merged_2025.json for additional sources"""
- config_path = self.config_dir / 'all_apis_merged_2025.json'
-
- try:
- with open(config_path, 'r', encoding='utf-8') as f:
- data = json.load(f)
-
- # Process merged data structure (flexible parsing)
- if isinstance(data, dict):
- for category, entries in data.items():
- if isinstance(entries, list):
- for entry in entries:
- self._process_merged_entry(entry, category)
- elif isinstance(entries, dict):
- self._process_merged_entry(entries, category)
-
- logger.info("✓ Loaded merged APIs config")
-
- except Exception as e:
- logger.error(f"Error loading merged APIs: {e}")
-
- def _process_merged_entry(self, entry: Dict, category: str):
- """Process a single merged API entry"""
- if not isinstance(entry, dict):
- return
-
- api_id = entry.get('id', entry.get('name', '')).lower().replace(' ', '_')
-
- # Skip if already loaded
- if api_id in self.apis:
- return
-
- self.apis[api_id] = {
- 'id': api_id,
- 'name': entry.get('name', api_id),
- 'category': category,
- 'base_url': entry.get('url', entry.get('base_url', '')),
- 'auth': entry.get('auth', {}),
- 'docs_url': entry.get('docs', entry.get('docs_url')),
- 'endpoints': entry.get('endpoints'),
- 'notes': entry.get('notes', entry.get('description')),
- 'priority': entry.get('priority', 3),
- 'update_type': entry.get('update_type', 'periodic'),
- 'enabled': entry.get('enabled', True)
- }
-
- def load_pipeline_config(self):
- """Load ultimate_crypto_pipeline_2025_NZasinich.json"""
- config_path = self.config_dir / 'ultimate_crypto_pipeline_2025_NZasinich.json'
-
- try:
- with open(config_path, 'r', encoding='utf-8') as f:
- data = json.load(f)
-
- # Extract pipeline-specific configurations
- pipeline = data.get('pipeline', {})
-
- # Update scheduling preferences from pipeline
- for stage in pipeline.get('stages', []):
- stage_name = stage.get('name', '')
- interval = stage.get('interval', 300)
-
- # Map pipeline stages to API categories
- if 'market' in stage_name.lower():
- self._update_category_schedule('market_data', interval)
- elif 'sentiment' in stage_name.lower():
- self._update_category_schedule('sentiment', interval)
- elif 'huggingface' in stage_name.lower() or 'hf' in stage_name.lower():
- self._update_category_schedule('huggingface', interval)
-
- logger.info("✓ Loaded pipeline config")
-
- except Exception as e:
- logger.error(f"Error loading pipeline config: {e}")
-
- def _update_category_schedule(self, category: str, interval: int):
- """Update schedule for all APIs in a category"""
- for api_id, api in self.apis.items():
- if api.get('category') == category:
- if api_id not in self.schedules:
- self.schedules[api_id] = {}
- self.schedules[api_id]['interval'] = interval
-
- def setup_cors_proxies(self):
- """Setup CORS proxy list"""
- self.cors_proxies = [
- 'https://api.allorigins.win/get?url=',
- 'https://proxy.cors.sh/',
- 'https://proxy.corsfix.com/?url=',
- 'https://api.codetabs.com/v1/proxy?quest=',
- 'https://thingproxy.freeboard.io/fetch/',
- 'https://corsproxy.io/?'
- ]
-
- def setup_default_schedules(self):
- """Setup default schedules based on update_type"""
- schedule_intervals = {
- 'realtime': 0, # WebSocket - always connected
- 'periodic': 60, # Every minute for market data
- 'scheduled': 3600, # Every hour for HuggingFace
- 'daily': 86400 # Once per day
- }
-
- for api_id, api in self.apis.items():
- if api_id not in self.schedules:
- update_type = api.get('update_type', 'periodic')
- interval = schedule_intervals.get(update_type, 300)
-
- self.schedules[api_id] = {
- 'interval': interval,
- 'enabled': api.get('enabled', True),
- 'last_update': None,
- 'next_update': datetime.now(),
- 'update_type': update_type
- }
-
- def get_all_apis(self) -> Dict[str, Dict[str, Any]]:
- """Get all configured APIs"""
- return self.apis
-
- def get_apis_by_category(self, category: str) -> Dict[str, Dict[str, Any]]:
- """Get APIs filtered by category"""
- return {k: v for k, v in self.apis.items() if v.get('category') == category}
-
- def get_categories(self) -> List[str]:
- """Get all unique categories"""
- return list(set(api.get('category', 'unknown') for api in self.apis.values()))
-
- def get_realtime_apis(self) -> Dict[str, Dict[str, Any]]:
- """Get APIs that support real-time updates (WebSocket)"""
- return {k: v for k, v in self.apis.items() if v.get('update_type') == 'realtime'}
-
- def get_periodic_apis(self) -> Dict[str, Dict[str, Any]]:
- """Get APIs that need periodic updates"""
- return {k: v for k, v in self.apis.items() if v.get('update_type') == 'periodic'}
-
- def get_scheduled_apis(self) -> Dict[str, Dict[str, Any]]:
- """Get APIs with scheduled updates (less frequent)"""
- return {k: v for k, v in self.apis.items() if v.get('update_type') == 'scheduled'}
-
- def get_apis_due_for_update(self) -> Dict[str, Dict[str, Any]]:
- """Get APIs that are due for update based on their schedule"""
- now = datetime.now()
- due_apis = {}
-
- for api_id, schedule in self.schedules.items():
- if not schedule.get('enabled', True):
- continue
-
- next_update = schedule.get('next_update')
- if next_update and now >= next_update:
- due_apis[api_id] = self.apis[api_id]
-
- return due_apis
-
- def update_schedule(self, api_id: str, interval: int = None, enabled: bool = None):
- """Update schedule for a specific API"""
- if api_id not in self.schedules:
- self.schedules[api_id] = {}
-
- if interval is not None:
- self.schedules[api_id]['interval'] = interval
-
- if enabled is not None:
- self.schedules[api_id]['enabled'] = enabled
-
- def mark_updated(self, api_id: str):
- """Mark an API as updated and calculate next update time"""
- if api_id in self.schedules:
- now = datetime.now()
- interval = self.schedules[api_id].get('interval', 300)
-
- self.schedules[api_id]['last_update'] = now
- self.schedules[api_id]['next_update'] = now + timedelta(seconds=interval)
-
- def add_custom_api(self, api_data: Dict[str, Any]) -> bool:
- """Add a custom API source"""
- api_id = api_data.get('id', api_data.get('name', '')).lower().replace(' ', '_')
-
- if not api_id:
- return False
-
- self.apis[api_id] = {
- 'id': api_id,
- 'name': api_data.get('name', api_id),
- 'category': api_data.get('category', 'custom'),
- 'base_url': api_data.get('base_url', api_data.get('url', '')),
- 'auth': api_data.get('auth', {}),
- 'docs_url': api_data.get('docs_url'),
- 'endpoints': api_data.get('endpoints'),
- 'notes': api_data.get('notes'),
- 'priority': api_data.get('priority', 3),
- 'update_type': api_data.get('update_type', 'periodic'),
- 'enabled': api_data.get('enabled', True)
- }
-
- # Setup schedule
- self.schedules[api_id] = {
- 'interval': api_data.get('interval', 300),
- 'enabled': True,
- 'last_update': None,
- 'next_update': datetime.now(),
- 'update_type': api_data.get('update_type', 'periodic')
- }
-
- return True
-
- def remove_api(self, api_id: str) -> bool:
- """Remove an API source"""
- if api_id in self.apis:
- del self.apis[api_id]
-
- if api_id in self.schedules:
- del self.schedules[api_id]
-
- if api_id in self.keys:
- del self.keys[api_id]
-
- return True
-
- def export_config(self, filepath: str):
- """Export current configuration to JSON"""
- config = {
- 'apis': self.apis,
- 'schedules': self.schedules,
- 'keys': {k: '***' for k in self.keys.keys()}, # Don't export actual keys
- 'cors_proxies': self.cors_proxies,
- 'exported_at': datetime.now().isoformat()
- }
-
- with open(filepath, 'w', encoding='utf-8') as f:
- json.dump(config, f, indent=2, default=str)
-
- return True
-
- def import_config(self, filepath: str):
- """Import configuration from JSON"""
- with open(filepath, 'r', encoding='utf-8') as f:
- config = json.load(f)
-
- # Merge imported configs
- self.apis.update(config.get('apis', {}))
- self.schedules.update(config.get('schedules', {}))
- self.cors_proxies = config.get('cors_proxies', self.cors_proxies)
-
- return True
-
-
-# Global instance
-unified_loader = UnifiedConfigLoader()
diff --git a/app/backend/services/websocket_service.py b/app/backend/services/websocket_service.py
deleted file mode 100644
index 661daec3fae8ca7828da705acd56caa66460bde8..0000000000000000000000000000000000000000
--- a/app/backend/services/websocket_service.py
+++ /dev/null
@@ -1,402 +0,0 @@
-"""
-WebSocket Service
-Handles real-time data updates to connected clients
-"""
-import asyncio
-import json
-import logging
-from typing import Dict, Set, Any, List, Optional
-from datetime import datetime
-from fastapi import WebSocket, WebSocketDisconnect
-from collections import defaultdict
-
-logger = logging.getLogger(__name__)
-
-
-class ConnectionManager:
- """Manages WebSocket connections and broadcasts"""
-
- def __init__(self):
- # Active connections by client ID
- self.active_connections: Dict[str, WebSocket] = {}
-
- # Subscriptions: {api_id: set(client_ids)}
- self.subscriptions: Dict[str, Set[str]] = defaultdict(set)
-
- # Reverse subscriptions: {client_id: set(api_ids)}
- self.client_subscriptions: Dict[str, Set[str]] = defaultdict(set)
-
- # Connection metadata
- self.connection_metadata: Dict[str, Dict[str, Any]] = {}
-
- async def connect(self, websocket: WebSocket, client_id: str, metadata: Optional[Dict] = None):
- """
- Connect a new WebSocket client
-
- Args:
- websocket: WebSocket connection
- client_id: Unique client identifier
- metadata: Optional metadata about the connection
- """
- await websocket.accept()
- self.active_connections[client_id] = websocket
- self.connection_metadata[client_id] = metadata or {}
-
- logger.info(f"Client {client_id} connected. Total connections: {len(self.active_connections)}")
-
- def disconnect(self, client_id: str):
- """
- Disconnect a WebSocket client
-
- Args:
- client_id: Client identifier
- """
- if client_id in self.active_connections:
- del self.active_connections[client_id]
-
- # Remove all subscriptions for this client
- for api_id in self.client_subscriptions.get(client_id, set()).copy():
- self.unsubscribe(client_id, api_id)
-
- if client_id in self.client_subscriptions:
- del self.client_subscriptions[client_id]
-
- if client_id in self.connection_metadata:
- del self.connection_metadata[client_id]
-
- logger.info(f"Client {client_id} disconnected. Total connections: {len(self.active_connections)}")
-
- def subscribe(self, client_id: str, api_id: str):
- """
- Subscribe a client to API updates
-
- Args:
- client_id: Client identifier
- api_id: API identifier to subscribe to
- """
- self.subscriptions[api_id].add(client_id)
- self.client_subscriptions[client_id].add(api_id)
-
- logger.debug(f"Client {client_id} subscribed to {api_id}")
-
- def unsubscribe(self, client_id: str, api_id: str):
- """
- Unsubscribe a client from API updates
-
- Args:
- client_id: Client identifier
- api_id: API identifier to unsubscribe from
- """
- if api_id in self.subscriptions:
- self.subscriptions[api_id].discard(client_id)
-
- # Clean up empty subscription sets
- if not self.subscriptions[api_id]:
- del self.subscriptions[api_id]
-
- if client_id in self.client_subscriptions:
- self.client_subscriptions[client_id].discard(api_id)
-
- logger.debug(f"Client {client_id} unsubscribed from {api_id}")
-
- def subscribe_all(self, client_id: str):
- """
- Subscribe a client to all API updates
-
- Args:
- client_id: Client identifier
- """
- self.client_subscriptions[client_id].add('*')
- logger.debug(f"Client {client_id} subscribed to all updates")
-
- async def send_personal_message(self, message: Dict[str, Any], client_id: str):
- """
- Send a message to a specific client
-
- Args:
- message: Message data
- client_id: Target client identifier
- """
- if client_id in self.active_connections:
- websocket = self.active_connections[client_id]
- try:
- await websocket.send_json(message)
- except Exception as e:
- logger.error(f"Error sending message to {client_id}: {e}")
- self.disconnect(client_id)
-
- async def broadcast(self, message: Dict[str, Any], api_id: Optional[str] = None):
- """
- Broadcast a message to subscribed clients
-
- Args:
- message: Message data
- api_id: Optional API ID (broadcasts to all if None)
- """
- if api_id:
- # Send to clients subscribed to this specific API
- target_clients = self.subscriptions.get(api_id, set())
-
- # Also include clients subscribed to all updates
- target_clients = target_clients.union(
- {cid for cid, subs in self.client_subscriptions.items() if '*' in subs}
- )
- else:
- # Broadcast to all connected clients
- target_clients = set(self.active_connections.keys())
-
- # Send to all target clients
- disconnected_clients = []
-
- for client_id in target_clients:
- if client_id in self.active_connections:
- websocket = self.active_connections[client_id]
- try:
- await websocket.send_json(message)
- except Exception as e:
- logger.error(f"Error broadcasting to {client_id}: {e}")
- disconnected_clients.append(client_id)
-
- # Clean up disconnected clients
- for client_id in disconnected_clients:
- self.disconnect(client_id)
-
- async def broadcast_api_update(self, api_id: str, data: Dict[str, Any], metadata: Optional[Dict] = None):
- """
- Broadcast an API data update
-
- Args:
- api_id: API identifier
- data: Updated data
- metadata: Optional metadata about the update
- """
- message = {
- 'type': 'api_update',
- 'api_id': api_id,
- 'data': data,
- 'metadata': metadata or {},
- 'timestamp': datetime.now().isoformat()
- }
-
- await self.broadcast(message, api_id)
-
- async def broadcast_status_update(self, status: Dict[str, Any]):
- """
- Broadcast a system status update
-
- Args:
- status: Status data
- """
- message = {
- 'type': 'status_update',
- 'status': status,
- 'timestamp': datetime.now().isoformat()
- }
-
- await self.broadcast(message)
-
- async def broadcast_schedule_update(self, schedule_info: Dict[str, Any]):
- """
- Broadcast a schedule update
-
- Args:
- schedule_info: Schedule information
- """
- message = {
- 'type': 'schedule_update',
- 'schedule': schedule_info,
- 'timestamp': datetime.now().isoformat()
- }
-
- await self.broadcast(message)
-
- def get_connection_stats(self) -> Dict[str, Any]:
- """
- Get connection statistics
-
- Returns:
- Statistics about connections and subscriptions
- """
- return {
- 'total_connections': len(self.active_connections),
- 'total_subscriptions': sum(len(subs) for subs in self.subscriptions.values()),
- 'apis_with_subscribers': len(self.subscriptions),
- 'clients': {
- client_id: {
- 'subscriptions': list(self.client_subscriptions.get(client_id, set())),
- 'metadata': self.connection_metadata.get(client_id, {})
- }
- for client_id in self.active_connections.keys()
- }
- }
-
-
-class WebSocketService:
- """WebSocket service for real-time updates"""
-
- def __init__(self, scheduler_service=None, persistence_service=None):
- self.connection_manager = ConnectionManager()
- self.scheduler_service = scheduler_service
- self.persistence_service = persistence_service
- self.running = False
-
- # Register callbacks with scheduler if available
- if self.scheduler_service:
- self._register_scheduler_callbacks()
-
- def _register_scheduler_callbacks(self):
- """Register callbacks with the scheduler service"""
- # This would be called after scheduler is initialized
- # For now, we'll use a different approach where scheduler calls websocket service
- pass
-
- async def handle_client_message(self, websocket: WebSocket, client_id: str, message: Dict[str, Any]):
- """
- Handle incoming messages from clients
-
- Args:
- websocket: WebSocket connection
- client_id: Client identifier
- message: Message from client
- """
- try:
- message_type = message.get('type')
-
- if message_type == 'subscribe':
- # Subscribe to specific API
- api_id = message.get('api_id')
- if api_id:
- self.connection_manager.subscribe(client_id, api_id)
- await self.connection_manager.send_personal_message({
- 'type': 'subscribed',
- 'api_id': api_id,
- 'status': 'success'
- }, client_id)
-
- elif message_type == 'subscribe_all':
- # Subscribe to all updates
- self.connection_manager.subscribe_all(client_id)
- await self.connection_manager.send_personal_message({
- 'type': 'subscribed',
- 'api_id': '*',
- 'status': 'success'
- }, client_id)
-
- elif message_type == 'unsubscribe':
- # Unsubscribe from specific API
- api_id = message.get('api_id')
- if api_id:
- self.connection_manager.unsubscribe(client_id, api_id)
- await self.connection_manager.send_personal_message({
- 'type': 'unsubscribed',
- 'api_id': api_id,
- 'status': 'success'
- }, client_id)
-
- elif message_type == 'get_data':
- # Request current cached data
- api_id = message.get('api_id')
- if api_id and self.persistence_service:
- data = self.persistence_service.get_cached_data(api_id)
- await self.connection_manager.send_personal_message({
- 'type': 'data_response',
- 'api_id': api_id,
- 'data': data
- }, client_id)
-
- elif message_type == 'get_all_data':
- # Request all cached data
- if self.persistence_service:
- data = self.persistence_service.get_all_cached_data()
- await self.connection_manager.send_personal_message({
- 'type': 'data_response',
- 'data': data
- }, client_id)
-
- elif message_type == 'get_schedule':
- # Request schedule information
- if self.scheduler_service:
- schedules = self.scheduler_service.get_all_task_statuses()
- await self.connection_manager.send_personal_message({
- 'type': 'schedule_response',
- 'schedules': schedules
- }, client_id)
-
- elif message_type == 'update_schedule':
- # Update schedule for an API
- api_id = message.get('api_id')
- interval = message.get('interval')
- enabled = message.get('enabled')
-
- if api_id and self.scheduler_service:
- self.scheduler_service.update_task_schedule(api_id, interval, enabled)
- await self.connection_manager.send_personal_message({
- 'type': 'schedule_updated',
- 'api_id': api_id,
- 'status': 'success'
- }, client_id)
-
- elif message_type == 'force_update':
- # Force immediate update for an API
- api_id = message.get('api_id')
- if api_id and self.scheduler_service:
- success = await self.scheduler_service.force_update(api_id)
- await self.connection_manager.send_personal_message({
- 'type': 'update_result',
- 'api_id': api_id,
- 'status': 'success' if success else 'failed'
- }, client_id)
-
- elif message_type == 'ping':
- # Heartbeat
- await self.connection_manager.send_personal_message({
- 'type': 'pong',
- 'timestamp': datetime.now().isoformat()
- }, client_id)
-
- else:
- logger.warning(f"Unknown message type from {client_id}: {message_type}")
-
- except Exception as e:
- logger.error(f"Error handling client message: {e}")
- await self.connection_manager.send_personal_message({
- 'type': 'error',
- 'message': str(e)
- }, client_id)
-
- async def notify_data_update(self, api_id: str, data: Dict[str, Any], metadata: Optional[Dict] = None):
- """
- Notify clients about data updates
-
- Args:
- api_id: API identifier
- data: Updated data
- metadata: Optional metadata
- """
- await self.connection_manager.broadcast_api_update(api_id, data, metadata)
-
- async def notify_status_update(self, status: Dict[str, Any]):
- """
- Notify clients about status updates
-
- Args:
- status: Status information
- """
- await self.connection_manager.broadcast_status_update(status)
-
- async def notify_schedule_update(self, schedule_info: Dict[str, Any]):
- """
- Notify clients about schedule updates
-
- Args:
- schedule_info: Schedule information
- """
- await self.connection_manager.broadcast_schedule_update(schedule_info)
-
- def get_stats(self) -> Dict[str, Any]:
- """Get WebSocket service statistics"""
- return self.connection_manager.get_connection_stats()
-
-
-# Global instance
-websocket_service = WebSocketService()
diff --git a/app/backend/services/ws_service_manager.py b/app/backend/services/ws_service_manager.py
deleted file mode 100644
index 1cfdb7e41b2b598328fcf738d91037b905f8f5f8..0000000000000000000000000000000000000000
--- a/app/backend/services/ws_service_manager.py
+++ /dev/null
@@ -1,385 +0,0 @@
-"""
-Centralized WebSocket Service Manager
-
-This module provides a unified interface for managing WebSocket connections
-and broadcasting real-time data from various services.
-"""
-
-import asyncio
-import json
-from datetime import datetime
-from typing import Dict, List, Set, Any, Optional, Callable
-from fastapi import WebSocket, WebSocketDisconnect
-from enum import Enum
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class ServiceType(str, Enum):
- """Available service types for WebSocket subscriptions"""
- # Data Collection Services
- MARKET_DATA = "market_data"
- EXPLORERS = "explorers"
- NEWS = "news"
- SENTIMENT = "sentiment"
- WHALE_TRACKING = "whale_tracking"
- RPC_NODES = "rpc_nodes"
- ONCHAIN = "onchain"
-
- # Monitoring Services
- HEALTH_CHECKER = "health_checker"
- POOL_MANAGER = "pool_manager"
- SCHEDULER = "scheduler"
-
- # Integration Services
- HUGGINGFACE = "huggingface"
- PERSISTENCE = "persistence"
-
- # System Services
- SYSTEM = "system"
- ALL = "all"
-
-
-class WebSocketConnection:
- """Represents a single WebSocket connection with subscription management"""
-
- def __init__(self, websocket: WebSocket, client_id: str):
- self.websocket = websocket
- self.client_id = client_id
- self.subscriptions: Set[ServiceType] = set()
- self.connected_at = datetime.utcnow()
- self.last_activity = datetime.utcnow()
- self.metadata: Dict[str, Any] = {}
-
- async def send_message(self, message: Dict[str, Any]) -> bool:
- """
- Send a message to the client
-
- Returns:
- bool: True if successful, False if failed
- """
- try:
- await self.websocket.send_json(message)
- self.last_activity = datetime.utcnow()
- return True
- except Exception as e:
- logger.error(f"Error sending message to client {self.client_id}: {e}")
- return False
-
- def subscribe(self, service: ServiceType):
- """Subscribe to a service"""
- self.subscriptions.add(service)
- logger.info(f"Client {self.client_id} subscribed to {service.value}")
-
- def unsubscribe(self, service: ServiceType):
- """Unsubscribe from a service"""
- self.subscriptions.discard(service)
- logger.info(f"Client {self.client_id} unsubscribed from {service.value}")
-
- def is_subscribed(self, service: ServiceType) -> bool:
- """Check if subscribed to a service or 'all'"""
- return service in self.subscriptions or ServiceType.ALL in self.subscriptions
-
-
-class WebSocketServiceManager:
- """
- Centralized manager for all WebSocket connections and service broadcasts
- """
-
- def __init__(self):
- self.connections: Dict[str, WebSocketConnection] = {}
- self.service_handlers: Dict[ServiceType, List[Callable]] = {}
- self._lock = asyncio.Lock()
- self._client_counter = 0
-
- def generate_client_id(self) -> str:
- """Generate a unique client ID"""
- self._client_counter += 1
- return f"client_{self._client_counter}_{int(datetime.utcnow().timestamp())}"
-
- async def connect(self, websocket: WebSocket) -> WebSocketConnection:
- """
- Accept a new WebSocket connection
-
- Args:
- websocket: The FastAPI WebSocket instance
-
- Returns:
- WebSocketConnection: The connection object
- """
- await websocket.accept()
- client_id = self.generate_client_id()
-
- async with self._lock:
- connection = WebSocketConnection(websocket, client_id)
- self.connections[client_id] = connection
-
- logger.info(f"New WebSocket connection: {client_id}")
-
- # Send connection established message
- await connection.send_message({
- "type": "connection_established",
- "client_id": client_id,
- "timestamp": datetime.utcnow().isoformat(),
- "available_services": [s.value for s in ServiceType]
- })
-
- return connection
-
- async def disconnect(self, client_id: str):
- """
- Disconnect a client
-
- Args:
- client_id: The client ID to disconnect
- """
- async with self._lock:
- if client_id in self.connections:
- connection = self.connections[client_id]
- try:
- await connection.websocket.close()
- except:
- pass
- del self.connections[client_id]
- logger.info(f"Client disconnected: {client_id}")
-
- async def broadcast(
- self,
- service: ServiceType,
- message_type: str,
- data: Any,
- filter_func: Optional[Callable[[WebSocketConnection], bool]] = None
- ):
- """
- Broadcast a message to all subscribed clients
-
- Args:
- service: The service sending the message
- message_type: Type of message
- data: Message payload
- filter_func: Optional function to filter which clients receive the message
- """
- message = {
- "service": service.value,
- "type": message_type,
- "data": data,
- "timestamp": datetime.utcnow().isoformat()
- }
-
- disconnected_clients = []
-
- async with self._lock:
- for client_id, connection in self.connections.items():
- # Check subscription and optional filter
- if connection.is_subscribed(service):
- if filter_func is None or filter_func(connection):
- success = await connection.send_message(message)
- if not success:
- disconnected_clients.append(client_id)
-
- # Clean up disconnected clients
- for client_id in disconnected_clients:
- await self.disconnect(client_id)
-
- async def send_to_client(
- self,
- client_id: str,
- service: ServiceType,
- message_type: str,
- data: Any
- ) -> bool:
- """
- Send a message to a specific client
-
- Args:
- client_id: Target client ID
- service: Service sending the message
- message_type: Type of message
- data: Message payload
-
- Returns:
- bool: True if successful
- """
- async with self._lock:
- if client_id in self.connections:
- connection = self.connections[client_id]
- message = {
- "service": service.value,
- "type": message_type,
- "data": data,
- "timestamp": datetime.utcnow().isoformat()
- }
- return await connection.send_message(message)
- return False
-
- async def handle_client_message(
- self,
- connection: WebSocketConnection,
- message: Dict[str, Any]
- ):
- """
- Handle incoming messages from clients
-
- Expected message format:
- {
- "action": "subscribe" | "unsubscribe" | "get_status" | "ping",
- "service": "service_name" (for subscribe/unsubscribe),
- "data": {} (optional additional data)
- }
- """
- action = message.get("action")
-
- if action == "subscribe":
- service_name = message.get("service")
- if service_name:
- try:
- service = ServiceType(service_name)
- connection.subscribe(service)
- await connection.send_message({
- "service": "system",
- "type": "subscription_confirmed",
- "data": {
- "service": service_name,
- "subscriptions": [s.value for s in connection.subscriptions]
- },
- "timestamp": datetime.utcnow().isoformat()
- })
- except ValueError:
- await connection.send_message({
- "service": "system",
- "type": "error",
- "data": {
- "message": f"Invalid service: {service_name}",
- "available_services": [s.value for s in ServiceType]
- },
- "timestamp": datetime.utcnow().isoformat()
- })
-
- elif action == "unsubscribe":
- service_name = message.get("service")
- if service_name:
- try:
- service = ServiceType(service_name)
- connection.unsubscribe(service)
- await connection.send_message({
- "service": "system",
- "type": "unsubscription_confirmed",
- "data": {
- "service": service_name,
- "subscriptions": [s.value for s in connection.subscriptions]
- },
- "timestamp": datetime.utcnow().isoformat()
- })
- except ValueError:
- await connection.send_message({
- "service": "system",
- "type": "error",
- "data": {"message": f"Invalid service: {service_name}"},
- "timestamp": datetime.utcnow().isoformat()
- })
-
- elif action == "get_status":
- await connection.send_message({
- "service": "system",
- "type": "status",
- "data": {
- "client_id": connection.client_id,
- "connected_at": connection.connected_at.isoformat(),
- "last_activity": connection.last_activity.isoformat(),
- "subscriptions": [s.value for s in connection.subscriptions],
- "total_clients": len(self.connections)
- },
- "timestamp": datetime.utcnow().isoformat()
- })
-
- elif action == "ping":
- await connection.send_message({
- "service": "system",
- "type": "pong",
- "data": message.get("data", {}),
- "timestamp": datetime.utcnow().isoformat()
- })
-
- else:
- await connection.send_message({
- "service": "system",
- "type": "error",
- "data": {
- "message": f"Unknown action: {action}",
- "supported_actions": ["subscribe", "unsubscribe", "get_status", "ping"]
- },
- "timestamp": datetime.utcnow().isoformat()
- })
-
- async def start_service_stream(
- self,
- service: ServiceType,
- data_generator: Callable,
- interval: float = 1.0
- ):
- """
- Start a continuous data stream for a service
-
- Args:
- service: The service type
- data_generator: Async function that generates data
- interval: Update interval in seconds
- """
- logger.info(f"Starting stream for service: {service.value}")
-
- while True:
- try:
- # Check if anyone is subscribed
- has_subscribers = False
- async with self._lock:
- for connection in self.connections.values():
- if connection.is_subscribed(service):
- has_subscribers = True
- break
-
- # Only fetch data if there are subscribers
- if has_subscribers:
- data = await data_generator()
- if data:
- await self.broadcast(
- service=service,
- message_type="update",
- data=data
- )
-
- await asyncio.sleep(interval)
-
- except asyncio.CancelledError:
- logger.info(f"Stream cancelled for service: {service.value}")
- break
- except Exception as e:
- logger.error(f"Error in service stream {service.value}: {e}")
- await asyncio.sleep(interval)
-
- def get_stats(self) -> Dict[str, Any]:
- """Get manager statistics"""
- subscription_counts = {}
- for service in ServiceType:
- subscription_counts[service.value] = sum(
- 1 for conn in self.connections.values()
- if conn.is_subscribed(service)
- )
-
- return {
- "total_connections": len(self.connections),
- "clients": [
- {
- "client_id": conn.client_id,
- "connected_at": conn.connected_at.isoformat(),
- "last_activity": conn.last_activity.isoformat(),
- "subscriptions": [s.value for s in conn.subscriptions]
- }
- for conn in self.connections.values()
- ],
- "subscription_counts": subscription_counts
- }
-
-
-# Global instance
-ws_manager = WebSocketServiceManager()
diff --git a/app/collectors.py b/app/collectors.py
deleted file mode 100644
index ac1a81b35fc691e2637bc7750e86714a2b838110..0000000000000000000000000000000000000000
--- a/app/collectors.py
+++ /dev/null
@@ -1,888 +0,0 @@
-#!/usr/bin/env python3
-"""
-Data Collection Module for Crypto Data Aggregator
-Collects price data, news, and sentiment from various sources
-"""
-
-import requests
-import aiohttp
-import asyncio
-import json
-import logging
-import time
-import threading
-from datetime import datetime, timedelta
-from typing import Dict, List, Optional, Any, Tuple
-import re
-
-# Try to import optional dependencies
-try:
- import feedparser
- FEEDPARSER_AVAILABLE = True
-except ImportError:
- FEEDPARSER_AVAILABLE = False
- logging.warning("feedparser not installed. RSS feed parsing will be limited.")
-
-try:
- from bs4 import BeautifulSoup
- BS4_AVAILABLE = True
-except ImportError:
- BS4_AVAILABLE = False
- logging.warning("beautifulsoup4 not installed. HTML parsing will be limited.")
-
-# Import local modules
-import config
-import database
-
-# Setup logging using config settings
-logging.basicConfig(
- level=getattr(logging, config.LOG_LEVEL),
- format=config.LOG_FORMAT,
- handlers=[
- logging.FileHandler(config.LOG_FILE),
- logging.StreamHandler()
- ]
-)
-logger = logging.getLogger(__name__)
-
-# Get database instance
-db = database.get_database()
-
-# Collection state tracking
-_collection_timers = []
-_is_collecting = False
-
-
-# ==================== AI MODEL STUB FUNCTIONS ====================
-# These provide fallback functionality when ai_models.py is not available
-
-def analyze_sentiment(text: str) -> Dict[str, Any]:
- """
- Simple sentiment analysis based on keyword matching
- Returns sentiment score and label
-
- Args:
- text: Text to analyze
-
- Returns:
- Dict with 'score' and 'label'
- """
- if not text:
- return {'score': 0.0, 'label': 'neutral'}
-
- text_lower = text.lower()
-
- # Positive keywords
- positive_words = [
- 'bullish', 'moon', 'rally', 'surge', 'gain', 'profit', 'up', 'green',
- 'buy', 'long', 'growth', 'rise', 'pump', 'ATH', 'breakthrough',
- 'adoption', 'positive', 'optimistic', 'upgrade', 'partnership'
- ]
-
- # Negative keywords
- negative_words = [
- 'bearish', 'crash', 'dump', 'drop', 'loss', 'down', 'red', 'sell',
- 'short', 'decline', 'fall', 'fear', 'scam', 'hack', 'vulnerability',
- 'negative', 'pessimistic', 'concern', 'warning', 'risk'
- ]
-
- # Count occurrences
- positive_count = sum(1 for word in positive_words if word in text_lower)
- negative_count = sum(1 for word in negative_words if word in text_lower)
-
- # Calculate score (-1 to 1)
- total = positive_count + negative_count
- if total == 0:
- score = 0.0
- label = 'neutral'
- else:
- score = (positive_count - negative_count) / total
-
- # Determine label
- if score <= -0.6:
- label = 'very_negative'
- elif score <= -0.2:
- label = 'negative'
- elif score <= 0.2:
- label = 'neutral'
- elif score <= 0.6:
- label = 'positive'
- else:
- label = 'very_positive'
-
- return {'score': score, 'label': label}
-
-
-def summarize_text(text: str, max_length: int = 150) -> str:
- """
- Simple text summarization - takes first sentences up to max_length
-
- Args:
- text: Text to summarize
- max_length: Maximum length of summary
-
- Returns:
- Summarized text
- """
- if not text:
- return ""
-
- # Remove extra whitespace
- text = ' '.join(text.split())
-
- # If already short enough, return as is
- if len(text) <= max_length:
- return text
-
- # Try to break at sentence boundary
- sentences = re.split(r'[.!?]+', text)
- summary = ""
-
- for sentence in sentences:
- sentence = sentence.strip()
- if not sentence:
- continue
-
- if len(summary) + len(sentence) + 2 <= max_length:
- summary += sentence + ". "
- else:
- break
-
- # If no complete sentences fit, truncate
- if not summary:
- summary = text[:max_length-3] + "..."
-
- return summary.strip()
-
-
-# Try to import AI models if available
-try:
- import ai_models
- # Override stub functions with real AI models if available
- analyze_sentiment = ai_models.analyze_sentiment
- summarize_text = ai_models.summarize_text
- logger.info("Using AI models for sentiment analysis and summarization")
-except ImportError:
- logger.info("AI models not available, using simple keyword-based analysis")
-
-
-# ==================== HELPER FUNCTIONS ====================
-
-def safe_api_call(url: str, timeout: int = 10, headers: Optional[Dict] = None) -> Optional[Dict]:
- """
- Make HTTP GET request with error handling and retry logic
-
- Args:
- url: URL to fetch
- timeout: Request timeout in seconds
- headers: Optional request headers
-
- Returns:
- Response JSON or None on failure
- """
- if headers is None:
- headers = {'User-Agent': config.USER_AGENT}
-
- for attempt in range(config.MAX_RETRIES):
- try:
- logger.debug(f"API call attempt {attempt + 1}/{config.MAX_RETRIES}: {url}")
- response = requests.get(url, timeout=timeout, headers=headers)
- response.raise_for_status()
- return response.json()
- except requests.exceptions.HTTPError as e:
- logger.warning(f"HTTP error on attempt {attempt + 1}: {e}")
- if response.status_code == 429: # Rate limit
- wait_time = (attempt + 1) * 5
- logger.info(f"Rate limited, waiting {wait_time}s...")
- time.sleep(wait_time)
- elif response.status_code >= 500: # Server error
- time.sleep(attempt + 1)
- else:
- break # Don't retry on 4xx errors
- except requests.exceptions.Timeout:
- logger.warning(f"Timeout on attempt {attempt + 1}")
- time.sleep(attempt + 1)
- except requests.exceptions.RequestException as e:
- logger.warning(f"Request error on attempt {attempt + 1}: {e}")
- time.sleep(attempt + 1)
- except json.JSONDecodeError as e:
- logger.error(f"JSON decode error: {e}")
- break
- except Exception as e:
- logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
- break
-
- logger.error(f"All retry attempts failed for {url}")
- return None
-
-
-def extract_mentioned_coins(text: str) -> List[str]:
- """
- Extract cryptocurrency symbols/names mentioned in text
-
- Args:
- text: Text to search for coin mentions
-
- Returns:
- List of coin symbols mentioned
- """
- if not text:
- return []
-
- text_upper = text.upper()
- mentioned = []
-
- # Check for common symbols
- common_symbols = {
- 'BTC': 'bitcoin', 'ETH': 'ethereum', 'BNB': 'binancecoin',
- 'XRP': 'ripple', 'ADA': 'cardano', 'SOL': 'solana',
- 'DOT': 'polkadot', 'DOGE': 'dogecoin', 'AVAX': 'avalanche-2',
- 'MATIC': 'polygon', 'LINK': 'chainlink', 'UNI': 'uniswap',
- 'LTC': 'litecoin', 'ATOM': 'cosmos', 'ALGO': 'algorand'
- }
-
- # Check coin symbols
- for symbol, coin_id in common_symbols.items():
- # Look for symbol as whole word or with $ prefix
- pattern = r'\b' + symbol + r'\b|\$' + symbol + r'\b'
- if re.search(pattern, text_upper):
- mentioned.append(symbol)
-
- # Check for full coin names (case insensitive)
- coin_names = {
- 'bitcoin': 'BTC', 'ethereum': 'ETH', 'binance': 'BNB',
- 'ripple': 'XRP', 'cardano': 'ADA', 'solana': 'SOL',
- 'polkadot': 'DOT', 'dogecoin': 'DOGE'
- }
-
- text_lower = text.lower()
- for name, symbol in coin_names.items():
- if name in text_lower and symbol not in mentioned:
- mentioned.append(symbol)
-
- return list(set(mentioned)) # Remove duplicates
-
-
-# ==================== PRICE DATA COLLECTION ====================
-
-def collect_price_data() -> Tuple[bool, int]:
- """
- Fetch price data from CoinGecko API, fallback to CoinCap if needed
-
- Returns:
- Tuple of (success: bool, count: int)
- """
- logger.info("Starting price data collection...")
-
- try:
- # Try CoinGecko first
- url = f"{config.COINGECKO_BASE_URL}{config.COINGECKO_ENDPOINTS['coins_markets']}"
- params = {
- 'vs_currency': 'usd',
- 'order': 'market_cap_desc',
- 'per_page': config.TOP_COINS_LIMIT,
- 'page': 1,
- 'sparkline': 'false',
- 'price_change_percentage': '1h,24h,7d'
- }
-
- # Add params to URL
- param_str = '&'.join([f"{k}={v}" for k, v in params.items()])
- full_url = f"{url}?{param_str}"
-
- data = safe_api_call(full_url, timeout=config.REQUEST_TIMEOUT)
-
- if data is None:
- logger.warning("CoinGecko API failed, trying CoinCap backup...")
- return collect_price_data_coincap()
-
- # Parse and validate data
- prices = []
- for item in data:
- try:
- price = item.get('current_price', 0)
-
- # Validate price
- if not config.MIN_PRICE <= price <= config.MAX_PRICE:
- logger.warning(f"Invalid price for {item.get('symbol')}: {price}")
- continue
-
- price_data = {
- 'symbol': item.get('symbol', '').upper(),
- 'name': item.get('name', ''),
- 'price_usd': price,
- 'volume_24h': item.get('total_volume', 0),
- 'market_cap': item.get('market_cap', 0),
- 'percent_change_1h': item.get('price_change_percentage_1h_in_currency'),
- 'percent_change_24h': item.get('price_change_percentage_24h'),
- 'percent_change_7d': item.get('price_change_percentage_7d'),
- 'rank': item.get('market_cap_rank', 999)
- }
-
- # Validate market cap and volume
- if price_data['market_cap'] and price_data['market_cap'] < config.MIN_MARKET_CAP:
- continue
- if price_data['volume_24h'] and price_data['volume_24h'] < config.MIN_VOLUME:
- continue
-
- prices.append(price_data)
-
- except Exception as e:
- logger.error(f"Error parsing price data item: {e}")
- continue
-
- # Save to database
- if prices:
- count = db.save_prices_batch(prices)
- logger.info(f"Successfully collected and saved {count} price records from CoinGecko")
- return True, count
- else:
- logger.warning("No valid price data to save")
- return False, 0
-
- except Exception as e:
- logger.error(f"Error in collect_price_data: {e}")
- return False, 0
-
-
-def collect_price_data_coincap() -> Tuple[bool, int]:
- """
- Backup function using CoinCap API
-
- Returns:
- Tuple of (success: bool, count: int)
- """
- logger.info("Starting CoinCap price data collection...")
-
- try:
- url = f"{config.COINCAP_BASE_URL}{config.COINCAP_ENDPOINTS['assets']}"
- params = {
- 'limit': config.TOP_COINS_LIMIT
- }
-
- param_str = '&'.join([f"{k}={v}" for k, v in params.items()])
- full_url = f"{url}?{param_str}"
-
- response = safe_api_call(full_url, timeout=config.REQUEST_TIMEOUT)
-
- if response is None or 'data' not in response:
- logger.error("CoinCap API failed")
- return False, 0
-
- data = response['data']
-
- # Parse and validate data
- prices = []
- for idx, item in enumerate(data):
- try:
- price = float(item.get('priceUsd', 0))
-
- # Validate price
- if not config.MIN_PRICE <= price <= config.MAX_PRICE:
- logger.warning(f"Invalid price for {item.get('symbol')}: {price}")
- continue
-
- price_data = {
- 'symbol': item.get('symbol', '').upper(),
- 'name': item.get('name', ''),
- 'price_usd': price,
- 'volume_24h': float(item.get('volumeUsd24Hr', 0)) if item.get('volumeUsd24Hr') else None,
- 'market_cap': float(item.get('marketCapUsd', 0)) if item.get('marketCapUsd') else None,
- 'percent_change_1h': None, # CoinCap doesn't provide 1h change
- 'percent_change_24h': float(item.get('changePercent24Hr', 0)) if item.get('changePercent24Hr') else None,
- 'percent_change_7d': None, # CoinCap doesn't provide 7d change
- 'rank': int(item.get('rank', idx + 1))
- }
-
- # Validate market cap and volume
- if price_data['market_cap'] and price_data['market_cap'] < config.MIN_MARKET_CAP:
- continue
- if price_data['volume_24h'] and price_data['volume_24h'] < config.MIN_VOLUME:
- continue
-
- prices.append(price_data)
-
- except Exception as e:
- logger.error(f"Error parsing CoinCap data item: {e}")
- continue
-
- # Save to database
- if prices:
- count = db.save_prices_batch(prices)
- logger.info(f"Successfully collected and saved {count} price records from CoinCap")
- return True, count
- else:
- logger.warning("No valid price data to save from CoinCap")
- return False, 0
-
- except Exception as e:
- logger.error(f"Error in collect_price_data_coincap: {e}")
- return False, 0
-
-
-# ==================== NEWS DATA COLLECTION ====================
-
-def collect_news_data() -> int:
- """
- Parse RSS feeds and Reddit posts, analyze sentiment and save to database
-
- Returns:
- Count of articles collected
- """
- logger.info("Starting news data collection...")
- articles_collected = 0
-
- # Collect from RSS feeds
- if FEEDPARSER_AVAILABLE:
- articles_collected += _collect_rss_feeds()
- else:
- logger.warning("Feedparser not available, skipping RSS feeds")
-
- # Collect from Reddit
- articles_collected += _collect_reddit_posts()
-
- logger.info(f"News collection completed. Total articles: {articles_collected}")
- return articles_collected
-
-
-def _collect_rss_feeds() -> int:
- """Collect articles from RSS feeds"""
- count = 0
-
- for source_name, feed_url in config.RSS_FEEDS.items():
- try:
- logger.debug(f"Parsing RSS feed: {source_name}")
- feed = feedparser.parse(feed_url)
-
- for entry in feed.entries[:20]: # Limit to 20 most recent per feed
- try:
- # Extract article data
- title = entry.get('title', '')
- url = entry.get('link', '')
-
- # Skip if no URL
- if not url:
- continue
-
- # Get published date
- published_date = None
- if hasattr(entry, 'published_parsed') and entry.published_parsed:
- try:
- published_date = datetime(*entry.published_parsed[:6]).isoformat()
- except:
- pass
-
- # Get summary/description
- summary = entry.get('summary', '') or entry.get('description', '')
- if summary and BS4_AVAILABLE:
- # Strip HTML tags
- soup = BeautifulSoup(summary, 'html.parser')
- summary = soup.get_text()
-
- # Combine title and summary for analysis
- full_text = f"{title} {summary}"
-
- # Extract mentioned coins
- related_coins = extract_mentioned_coins(full_text)
-
- # Analyze sentiment
- sentiment_result = analyze_sentiment(full_text)
-
- # Summarize text
- summary_text = summarize_text(summary or title, max_length=200)
-
- # Prepare news data
- news_data = {
- 'title': title,
- 'summary': summary_text,
- 'url': url,
- 'source': source_name,
- 'sentiment_score': sentiment_result['score'],
- 'sentiment_label': sentiment_result['label'],
- 'related_coins': related_coins,
- 'published_date': published_date
- }
-
- # Save to database
- if db.save_news(news_data):
- count += 1
-
- except Exception as e:
- logger.error(f"Error processing RSS entry from {source_name}: {e}")
- continue
-
- except Exception as e:
- logger.error(f"Error parsing RSS feed {source_name}: {e}")
- continue
-
- logger.info(f"Collected {count} articles from RSS feeds")
- return count
-
-
-def _collect_reddit_posts() -> int:
- """Collect posts from Reddit"""
- count = 0
-
- for subreddit_name, endpoint_url in config.REDDIT_ENDPOINTS.items():
- try:
- logger.debug(f"Fetching Reddit posts from r/{subreddit_name}")
-
- # Reddit API requires .json extension
- if not endpoint_url.endswith('.json'):
- endpoint_url = endpoint_url.rstrip('/') + '.json'
-
- headers = {'User-Agent': config.USER_AGENT}
- data = safe_api_call(endpoint_url, headers=headers)
-
- if not data or 'data' not in data or 'children' not in data['data']:
- logger.warning(f"Invalid response from Reddit: {subreddit_name}")
- continue
-
- posts = data['data']['children']
-
- for post_data in posts[:15]: # Limit to 15 posts per subreddit
- try:
- post = post_data.get('data', {})
-
- # Extract post data
- title = post.get('title', '')
- url = post.get('url', '')
- permalink = f"https://reddit.com{post.get('permalink', '')}"
- selftext = post.get('selftext', '')
-
- # Skip if no title
- if not title:
- continue
-
- # Use permalink as primary URL (actual Reddit post)
- article_url = permalink
-
- # Get timestamp
- created_utc = post.get('created_utc')
- published_date = None
- if created_utc:
- try:
- published_date = datetime.fromtimestamp(created_utc).isoformat()
- except:
- pass
-
- # Combine title and text for analysis
- full_text = f"{title} {selftext}"
-
- # Extract mentioned coins
- related_coins = extract_mentioned_coins(full_text)
-
- # Analyze sentiment
- sentiment_result = analyze_sentiment(full_text)
-
- # Summarize text
- summary_text = summarize_text(selftext or title, max_length=200)
-
- # Prepare news data
- news_data = {
- 'title': title,
- 'summary': summary_text,
- 'url': article_url,
- 'source': f"reddit_{subreddit_name}",
- 'sentiment_score': sentiment_result['score'],
- 'sentiment_label': sentiment_result['label'],
- 'related_coins': related_coins,
- 'published_date': published_date
- }
-
- # Save to database
- if db.save_news(news_data):
- count += 1
-
- except Exception as e:
- logger.error(f"Error processing Reddit post from {subreddit_name}: {e}")
- continue
-
- except Exception as e:
- logger.error(f"Error fetching Reddit posts from {subreddit_name}: {e}")
- continue
-
- logger.info(f"Collected {count} posts from Reddit")
- return count
-
-
-# ==================== SENTIMENT DATA COLLECTION ====================
-
-def collect_sentiment_data() -> Optional[Dict[str, Any]]:
- """
- Fetch Fear & Greed Index from Alternative.me
-
- Returns:
- Sentiment data or None on failure
- """
- logger.info("Starting sentiment data collection...")
-
- try:
- # Fetch Fear & Greed Index
- data = safe_api_call(config.ALTERNATIVE_ME_URL, timeout=config.REQUEST_TIMEOUT)
-
- if data is None or 'data' not in data:
- logger.error("Failed to fetch Fear & Greed Index")
- return None
-
- # Parse response
- fng_data = data['data'][0] if data['data'] else {}
-
- value = fng_data.get('value')
- classification = fng_data.get('value_classification', 'Unknown')
- timestamp = fng_data.get('timestamp')
-
- if value is None:
- logger.warning("No value in Fear & Greed response")
- return None
-
- # Convert to sentiment score (-1 to 1)
- # Fear & Greed is 0-100, convert to -1 to 1
- sentiment_score = (int(value) - 50) / 50.0
-
- # Determine label
- if int(value) <= 25:
- sentiment_label = 'extreme_fear'
- elif int(value) <= 45:
- sentiment_label = 'fear'
- elif int(value) <= 55:
- sentiment_label = 'neutral'
- elif int(value) <= 75:
- sentiment_label = 'greed'
- else:
- sentiment_label = 'extreme_greed'
-
- sentiment_data = {
- 'value': int(value),
- 'classification': classification,
- 'sentiment_score': sentiment_score,
- 'sentiment_label': sentiment_label,
- 'timestamp': timestamp
- }
-
- # Save to news table as market-wide sentiment
- news_data = {
- 'title': f"Market Sentiment: {classification}",
- 'summary': f"Fear & Greed Index: {value}/100 - {classification}",
- 'url': config.ALTERNATIVE_ME_URL,
- 'source': 'alternative_me',
- 'sentiment_score': sentiment_score,
- 'sentiment_label': sentiment_label,
- 'related_coins': ['BTC', 'ETH'], # Market-wide
- 'published_date': datetime.now().isoformat()
- }
-
- db.save_news(news_data)
-
- logger.info(f"Sentiment collected: {classification} ({value}/100)")
- return sentiment_data
-
- except Exception as e:
- logger.error(f"Error in collect_sentiment_data: {e}")
- return None
-
-
-# ==================== SCHEDULING ====================
-
-def schedule_data_collection():
- """
- Schedule periodic data collection using threading.Timer
- Runs collection tasks in background at configured intervals
- """
- global _is_collecting, _collection_timers
-
- if _is_collecting:
- logger.warning("Data collection already running")
- return
-
- _is_collecting = True
- logger.info("Starting scheduled data collection...")
-
- def run_price_collection():
- """Wrapper for price collection with rescheduling"""
- try:
- collect_price_data()
- except Exception as e:
- logger.error(f"Error in scheduled price collection: {e}")
- finally:
- # Reschedule
- if _is_collecting:
- timer = threading.Timer(
- config.COLLECTION_INTERVALS['price_data'],
- run_price_collection
- )
- timer.daemon = True
- timer.start()
- _collection_timers.append(timer)
-
- def run_news_collection():
- """Wrapper for news collection with rescheduling"""
- try:
- collect_news_data()
- except Exception as e:
- logger.error(f"Error in scheduled news collection: {e}")
- finally:
- # Reschedule
- if _is_collecting:
- timer = threading.Timer(
- config.COLLECTION_INTERVALS['news_data'],
- run_news_collection
- )
- timer.daemon = True
- timer.start()
- _collection_timers.append(timer)
-
- def run_sentiment_collection():
- """Wrapper for sentiment collection with rescheduling"""
- try:
- collect_sentiment_data()
- except Exception as e:
- logger.error(f"Error in scheduled sentiment collection: {e}")
- finally:
- # Reschedule
- if _is_collecting:
- timer = threading.Timer(
- config.COLLECTION_INTERVALS['sentiment_data'],
- run_sentiment_collection
- )
- timer.daemon = True
- timer.start()
- _collection_timers.append(timer)
-
- # Initial run immediately
- logger.info("Running initial data collection...")
-
- # Run initial collections in separate threads
- threading.Thread(target=run_price_collection, daemon=True).start()
- time.sleep(2) # Stagger starts
- threading.Thread(target=run_news_collection, daemon=True).start()
- time.sleep(2)
- threading.Thread(target=run_sentiment_collection, daemon=True).start()
-
- logger.info("Scheduled data collection started successfully")
- logger.info(f"Price data: every {config.COLLECTION_INTERVALS['price_data']}s")
- logger.info(f"News data: every {config.COLLECTION_INTERVALS['news_data']}s")
- logger.info(f"Sentiment data: every {config.COLLECTION_INTERVALS['sentiment_data']}s")
-
-
-def stop_scheduled_collection():
- """Stop all scheduled collection tasks"""
- global _is_collecting, _collection_timers
-
- logger.info("Stopping scheduled data collection...")
- _is_collecting = False
-
- # Cancel all timers
- for timer in _collection_timers:
- try:
- timer.cancel()
- except:
- pass
-
- _collection_timers.clear()
- logger.info("Scheduled data collection stopped")
-
-
-# ==================== ASYNC COLLECTION (BONUS) ====================
-
-async def collect_price_data_async() -> Tuple[bool, int]:
- """
- Async version of price data collection using aiohttp
-
- Returns:
- Tuple of (success: bool, count: int)
- """
- logger.info("Starting async price data collection...")
-
- try:
- url = f"{config.COINGECKO_BASE_URL}{config.COINGECKO_ENDPOINTS['coins_markets']}"
- params = {
- 'vs_currency': 'usd',
- 'order': 'market_cap_desc',
- 'per_page': config.TOP_COINS_LIMIT,
- 'page': 1,
- 'sparkline': 'false',
- 'price_change_percentage': '1h,24h,7d'
- }
-
- async with aiohttp.ClientSession() as session:
- async with session.get(url, params=params, timeout=config.REQUEST_TIMEOUT) as response:
- if response.status != 200:
- logger.error(f"API returned status {response.status}")
- return False, 0
-
- data = await response.json()
-
- # Parse and validate data (same as sync version)
- prices = []
- for item in data:
- try:
- price = item.get('current_price', 0)
-
- if not config.MIN_PRICE <= price <= config.MAX_PRICE:
- continue
-
- price_data = {
- 'symbol': item.get('symbol', '').upper(),
- 'name': item.get('name', ''),
- 'price_usd': price,
- 'volume_24h': item.get('total_volume', 0),
- 'market_cap': item.get('market_cap', 0),
- 'percent_change_1h': item.get('price_change_percentage_1h_in_currency'),
- 'percent_change_24h': item.get('price_change_percentage_24h'),
- 'percent_change_7d': item.get('price_change_percentage_7d'),
- 'rank': item.get('market_cap_rank', 999)
- }
-
- if price_data['market_cap'] and price_data['market_cap'] < config.MIN_MARKET_CAP:
- continue
- if price_data['volume_24h'] and price_data['volume_24h'] < config.MIN_VOLUME:
- continue
-
- prices.append(price_data)
-
- except Exception as e:
- logger.error(f"Error parsing price data item: {e}")
- continue
-
- # Save to database
- if prices:
- count = db.save_prices_batch(prices)
- logger.info(f"Async collected and saved {count} price records")
- return True, count
- else:
- return False, 0
-
- except Exception as e:
- logger.error(f"Error in collect_price_data_async: {e}")
- return False, 0
-
-
-# ==================== MAIN ENTRY POINT ====================
-
-if __name__ == "__main__":
- logger.info("=" * 60)
- logger.info("Crypto Data Collector - Manual Test Run")
- logger.info("=" * 60)
-
- # Test price collection
- logger.info("\n--- Testing Price Collection ---")
- success, count = collect_price_data()
- print(f"Price collection: {'SUCCESS' if success else 'FAILED'} - {count} records")
-
- # Test news collection
- logger.info("\n--- Testing News Collection ---")
- news_count = collect_news_data()
- print(f"News collection: {news_count} articles collected")
-
- # Test sentiment collection
- logger.info("\n--- Testing Sentiment Collection ---")
- sentiment = collect_sentiment_data()
- if sentiment:
- print(f"Sentiment: {sentiment['classification']} ({sentiment['value']}/100)")
- else:
- print("Sentiment collection: FAILED")
-
- logger.info("\n" + "=" * 60)
- logger.info("Manual test run completed")
- logger.info("=" * 60)
diff --git a/app/collectors/QUICK_START.md b/app/collectors/QUICK_START.md
deleted file mode 100644
index f70ed558a3c39f186b56177d3aae852c48625f6b..0000000000000000000000000000000000000000
--- a/app/collectors/QUICK_START.md
+++ /dev/null
@@ -1,255 +0,0 @@
-# Collectors Quick Start Guide
-
-## Files Created
-
-```
-/home/user/crypto-dt-source/collectors/
-├── __init__.py # Package exports
-├── market_data.py # Market data collectors (16 KB)
-├── explorers.py # Blockchain explorer collectors (17 KB)
-├── news.py # News aggregation collectors (13 KB)
-├── sentiment.py # Sentiment data collectors (7.8 KB)
-├── onchain.py # On-chain analytics (placeholder, 13 KB)
-├── demo_collectors.py # Comprehensive demo script (6.6 KB)
-├── README.md # Full documentation
-└── QUICK_START.md # This file
-```
-
-## Quick Test
-
-### Test All Collectors
-
-```bash
-cd /home/user/crypto-dt-source
-python collectors/demo_collectors.py
-```
-
-### Test Individual Modules
-
-```bash
-# Market Data (CoinGecko, CoinMarketCap, Binance)
-python -m collectors.market_data
-
-# Blockchain Explorers (Etherscan, BscScan, TronScan)
-python -m collectors.explorers
-
-# News (CryptoPanic, NewsAPI)
-python -m collectors.news
-
-# Sentiment (Alternative.me Fear & Greed)
-python -m collectors.sentiment
-
-# On-chain Analytics (Placeholder)
-python -m collectors.onchain
-```
-
-## Import and Use
-
-### Collect All Market Data
-
-```python
-import asyncio
-from collectors import collect_market_data
-
-results = asyncio.run(collect_market_data())
-
-for result in results:
- print(f"{result['provider']}: {result['success']}")
-```
-
-### Collect All Data from All Categories
-
-```python
-import asyncio
-from collectors import (
- collect_market_data,
- collect_explorer_data,
- collect_news_data,
- collect_sentiment_data,
- collect_onchain_data
-)
-
-async def main():
- # Run all collectors concurrently
- results = await asyncio.gather(
- collect_market_data(),
- collect_explorer_data(),
- collect_news_data(),
- collect_sentiment_data(),
- collect_onchain_data()
- )
-
- market, explorers, news, sentiment, onchain = results
-
- print(f"Market data: {len(market)} sources")
- print(f"Explorers: {len(explorers)} sources")
- print(f"News: {len(news)} sources")
- print(f"Sentiment: {len(sentiment)} sources")
- print(f"On-chain: {len(onchain)} sources (placeholder)")
-
-asyncio.run(main())
-```
-
-### Individual Collector Example
-
-```python
-import asyncio
-from collectors.market_data import get_coingecko_simple_price
-
-async def get_prices():
- result = await get_coingecko_simple_price()
-
- if result['success']:
- data = result['data']
- print(f"BTC: ${data['bitcoin']['usd']:,.2f}")
- print(f"ETH: ${data['ethereum']['usd']:,.2f}")
- print(f"BNB: ${data['binancecoin']['usd']:,.2f}")
- print(f"Data age: {result['staleness_minutes']:.2f} minutes")
- else:
- print(f"Error: {result['error']}")
-
-asyncio.run(get_prices())
-```
-
-## Collectors Summary
-
-### 1. Market Data (market_data.py)
-
-| Function | Provider | API Key Required | Description |
-|----------|----------|------------------|-------------|
-| `get_coingecko_simple_price()` | CoinGecko | No | BTC, ETH, BNB prices with market data |
-| `get_coinmarketcap_quotes()` | CoinMarketCap | Yes | Professional market data |
-| `get_binance_ticker()` | Binance | No | Real-time 24hr ticker |
-| `collect_market_data()` | All above | - | Collects from all sources |
-
-### 2. Blockchain Explorers (explorers.py)
-
-| Function | Provider | API Key Required | Description |
-|----------|----------|------------------|-------------|
-| `get_etherscan_gas_price()` | Etherscan | Yes | Current Ethereum gas prices |
-| `get_bscscan_bnb_price()` | BscScan | Yes | BNB price and BSC stats |
-| `get_tronscan_stats()` | TronScan | Optional | TRON network statistics |
-| `collect_explorer_data()` | All above | - | Collects from all sources |
-
-### 3. News Aggregation (news.py)
-
-| Function | Provider | API Key Required | Description |
-|----------|----------|------------------|-------------|
-| `get_cryptopanic_posts()` | CryptoPanic | No | Latest crypto news posts |
-| `get_newsapi_headlines()` | NewsAPI | Yes | Crypto-related headlines |
-| `collect_news_data()` | All above | - | Collects from all sources |
-
-### 4. Sentiment Analysis (sentiment.py)
-
-| Function | Provider | API Key Required | Description |
-|----------|----------|------------------|-------------|
-| `get_fear_greed_index()` | Alternative.me | No | Market Fear & Greed Index |
-| `collect_sentiment_data()` | All above | - | Collects from all sources |
-
-### 5. On-Chain Analytics (onchain.py)
-
-| Function | Provider | Status | Description |
-|----------|----------|--------|-------------|
-| `get_the_graph_data()` | The Graph | Placeholder | GraphQL blockchain data |
-| `get_blockchair_data()` | Blockchair | Placeholder | Blockchain statistics |
-| `get_glassnode_metrics()` | Glassnode | Placeholder | Advanced on-chain metrics |
-| `collect_onchain_data()` | All above | - | Collects from all sources |
-
-## API Keys Setup
-
-Create a `.env` file or set environment variables:
-
-```bash
-# Market Data
-export COINMARKETCAP_KEY_1="your_key_here"
-
-# Blockchain Explorers
-export ETHERSCAN_KEY_1="your_key_here"
-export BSCSCAN_KEY="your_key_here"
-export TRONSCAN_KEY="your_key_here"
-
-# News
-export NEWSAPI_KEY="your_key_here"
-```
-
-## Output Format
-
-All collectors return standardized format:
-
-```python
-{
- "provider": "CoinGecko", # Provider name
- "category": "market_data", # Category
- "data": {...}, # Raw API response
- "timestamp": "2025-11-11T00:20:00Z", # Collection time
- "data_timestamp": "2025-11-11T00:19:30Z", # Data timestamp
- "staleness_minutes": 0.5, # Data age
- "success": True, # Success flag
- "error": None, # Error message
- "error_type": None, # Error type
- "response_time_ms": 342.5 # Response time
-}
-```
-
-## Key Features
-
-✓ **Async/Concurrent** - All collectors run asynchronously
-✓ **Error Handling** - Comprehensive error handling and logging
-✓ **Staleness Tracking** - Calculates data age in minutes
-✓ **Rate Limiting** - Respects API rate limits
-✓ **Retry Logic** - Automatic retries with exponential backoff
-✓ **Structured Logging** - JSON-formatted logs
-✓ **API Key Management** - Secure key handling from environment
-✓ **Standardized Output** - Consistent response format
-✓ **Production Ready** - Ready for production deployment
-
-## Common Issues
-
-### 1. Missing API Keys
-
-```
-Error: API key required but not configured for CoinMarketCap
-```
-
-**Solution:** Set the required environment variable:
-```bash
-export COINMARKETCAP_KEY_1="your_api_key"
-```
-
-### 2. Rate Limit Exceeded
-
-```
-Error Type: rate_limit
-```
-
-**Solution:** Collectors automatically retry with backoff. Check rate limits in provider documentation.
-
-### 3. Network Timeout
-
-```
-Error Type: timeout
-```
-
-**Solution:** Collectors automatically increase timeout and retry. Check network connectivity.
-
-## Next Steps
-
-1. Run the demo: `python collectors/demo_collectors.py`
-2. Configure API keys for providers requiring authentication
-3. Integrate collectors into your monitoring system
-4. Implement on-chain collectors (currently placeholders)
-5. Add custom collectors following the existing patterns
-
-## Support
-
-- Full documentation: `collectors/README.md`
-- Demo script: `collectors/demo_collectors.py`
-- Configuration: `config.py`
-- API Client: `utils/api_client.py`
-- Logger: `utils/logger.py`
-
----
-
-**Total Collectors:** 14 functions across 5 modules
-**Total Code:** ~75 KB of production-ready Python code
-**Status:** Ready for production use (except on-chain placeholders)
diff --git a/app/collectors/README.md b/app/collectors/README.md
deleted file mode 100644
index 996638cbff623d3c07302da00b3acbe47adb7375..0000000000000000000000000000000000000000
--- a/app/collectors/README.md
+++ /dev/null
@@ -1,507 +0,0 @@
-# Cryptocurrency Data Collectors
-
-Comprehensive data collection modules for cryptocurrency APIs, blockchain explorers, news sources, sentiment indicators, and on-chain analytics.
-
-## Overview
-
-This package provides production-ready collectors for gathering cryptocurrency data from various sources. Each collector is designed with robust error handling, logging, staleness tracking, and standardized output formats.
-
-## Modules
-
-### 1. Market Data (`market_data.py`)
-
-Collects cryptocurrency market data from multiple providers.
-
-**Providers:**
-- **CoinGecko** - Free API for BTC, ETH, BNB prices with market cap and volume
-- **CoinMarketCap** - Professional market data with API key
-- **Binance** - Real-time ticker data from Binance exchange
-
-**Functions:**
-```python
-from collectors.market_data import (
- get_coingecko_simple_price,
- get_coinmarketcap_quotes,
- get_binance_ticker,
- collect_market_data # Collects from all sources
-)
-
-# Collect from all market data sources
-results = await collect_market_data()
-```
-
-**Features:**
-- Concurrent data collection
-- Price tracking with volume and market cap
-- 24-hour change percentages
-- Timestamp extraction for staleness calculation
-
-### 2. Blockchain Explorers (`explorers.py`)
-
-Collects data from blockchain explorers and network statistics.
-
-**Providers:**
-- **Etherscan** - Ethereum gas prices and network stats
-- **BscScan** - BNB prices and BSC network data
-- **TronScan** - TRON network statistics
-
-**Functions:**
-```python
-from collectors.explorers import (
- get_etherscan_gas_price,
- get_bscscan_bnb_price,
- get_tronscan_stats,
- collect_explorer_data # Collects from all sources
-)
-
-# Collect from all explorers
-results = await collect_explorer_data()
-```
-
-**Features:**
-- Real-time gas price tracking
-- Network health monitoring
-- API key management
-- Rate limit handling
-
-### 3. News Aggregation (`news.py`)
-
-Collects cryptocurrency news from multiple sources.
-
-**Providers:**
-- **CryptoPanic** - Cryptocurrency news aggregator with sentiment
-- **NewsAPI** - General news with crypto filtering
-
-**Functions:**
-```python
-from collectors.news import (
- get_cryptopanic_posts,
- get_newsapi_headlines,
- collect_news_data # Collects from all sources
-)
-
-# Collect from all news sources
-results = await collect_news_data()
-```
-
-**Features:**
-- News post aggregation
-- Article timestamps for freshness tracking
-- Article count reporting
-- Content filtering
-
-### 4. Sentiment Analysis (`sentiment.py`)
-
-Collects cryptocurrency market sentiment data.
-
-**Providers:**
-- **Alternative.me** - Fear & Greed Index (0-100 scale)
-
-**Functions:**
-```python
-from collectors.sentiment import (
- get_fear_greed_index,
- collect_sentiment_data # Collects from all sources
-)
-
-# Collect sentiment data
-results = await collect_sentiment_data()
-```
-
-**Features:**
-- Market sentiment indicator (Fear/Greed)
-- Historical sentiment tracking
-- Classification (Extreme Fear, Fear, Neutral, Greed, Extreme Greed)
-
-### 5. On-Chain Analytics (`onchain.py`)
-
-Placeholder implementations for on-chain data sources.
-
-**Providers (Placeholder):**
-- **The Graph** - GraphQL-based blockchain data
-- **Blockchair** - Blockchain explorer and statistics
-- **Glassnode** - Advanced on-chain metrics
-
-**Functions:**
-```python
-from collectors.onchain import (
- get_the_graph_data,
- get_blockchair_data,
- get_glassnode_metrics,
- collect_onchain_data # Collects from all sources
-)
-
-# Collect on-chain data (placeholder)
-results = await collect_onchain_data()
-```
-
-**Planned Features:**
-- DEX volume and liquidity tracking
-- Token holder analytics
-- NUPL, SOPR, and other on-chain metrics
-- Exchange flow monitoring
-- Whale transaction tracking
-
-## Standard Output Format
-
-All collectors return a standardized dictionary format:
-
-```python
-{
- "provider": str, # Provider name (e.g., "CoinGecko")
- "category": str, # Category (e.g., "market_data")
- "data": dict/list/None, # Raw API response data
- "timestamp": str, # Collection timestamp (ISO format)
- "data_timestamp": str/None, # Data timestamp from API (ISO format)
- "staleness_minutes": float/None, # Age of data in minutes
- "success": bool, # Whether collection succeeded
- "error": str/None, # Error message if failed
- "error_type": str/None, # Error classification
- "response_time_ms": float # API response time
-}
-```
-
-## Common Features
-
-All collectors implement:
-
-1. **Error Handling**
- - Graceful failure with detailed error messages
- - Exception catching and logging
- - API-specific error parsing
-
-2. **Logging**
- - Structured JSON logging
- - Request/response logging
- - Error logging with context
-
-3. **Staleness Tracking**
- - Extracts timestamps from API responses
- - Calculates data age in minutes
- - Handles missing timestamps
-
-4. **Rate Limiting**
- - Respects provider rate limits
- - Exponential backoff on failures
- - Rate limit error detection
-
-5. **Retry Logic**
- - Automatic retries on failure
- - Configurable retry attempts
- - Timeout handling
-
-6. **API Key Management**
- - Loads keys from config
- - Handles missing keys gracefully
- - API key masking in logs
-
-## Usage Examples
-
-### Basic Usage
-
-```python
-import asyncio
-from collectors import collect_market_data
-
-async def main():
- results = await collect_market_data()
-
- for result in results:
- if result['success']:
- print(f"{result['provider']}: Success")
- print(f" Staleness: {result['staleness_minutes']:.2f}m")
- else:
- print(f"{result['provider']}: Failed - {result['error']}")
-
-asyncio.run(main())
-```
-
-### Collecting All Data
-
-```python
-import asyncio
-from collectors import (
- collect_market_data,
- collect_explorer_data,
- collect_news_data,
- collect_sentiment_data,
- collect_onchain_data
-)
-
-async def collect_all():
- results = await asyncio.gather(
- collect_market_data(),
- collect_explorer_data(),
- collect_news_data(),
- collect_sentiment_data(),
- collect_onchain_data()
- )
-
- market, explorers, news, sentiment, onchain = results
-
- return {
- "market_data": market,
- "explorers": explorers,
- "news": news,
- "sentiment": sentiment,
- "onchain": onchain
- }
-
-all_data = asyncio.run(collect_all())
-```
-
-### Individual Collector Usage
-
-```python
-import asyncio
-from collectors.market_data import get_coingecko_simple_price
-
-async def get_prices():
- result = await get_coingecko_simple_price()
-
- if result['success']:
- data = result['data']
- print(f"Bitcoin: ${data['bitcoin']['usd']}")
- print(f"Ethereum: ${data['ethereum']['usd']}")
- print(f"BNB: ${data['binancecoin']['usd']}")
-
-asyncio.run(get_prices())
-```
-
-## Demo Script
-
-Run the comprehensive demo to test all collectors:
-
-```bash
-python collectors/demo_collectors.py
-```
-
-This will:
-- Execute all collectors concurrently
-- Display detailed results for each category
-- Show overall statistics
-- Save results to a JSON file
-
-## Configuration
-
-Collectors use the central configuration system from `config.py`:
-
-```python
-from config import config
-
-# Get provider configuration
-provider = config.get_provider('CoinGecko')
-
-# Get API key
-api_key = config.get_api_key('coinmarketcap')
-
-# Get providers by category
-market_providers = config.get_providers_by_category('market_data')
-```
-
-## API Keys
-
-API keys are loaded from environment variables:
-
-```bash
-# Market Data
-export COINMARKETCAP_KEY_1="your_key_here"
-export COINMARKETCAP_KEY_2="backup_key"
-
-# Blockchain Explorers
-export ETHERSCAN_KEY_1="your_key_here"
-export ETHERSCAN_KEY_2="backup_key"
-export BSCSCAN_KEY="your_key_here"
-export TRONSCAN_KEY="your_key_here"
-
-# News
-export NEWSAPI_KEY="your_key_here"
-
-# Analytics
-export CRYPTOCOMPARE_KEY="your_key_here"
-```
-
-Or use `.env` file with `python-dotenv`:
-
-```env
-COINMARKETCAP_KEY_1=your_key_here
-ETHERSCAN_KEY_1=your_key_here
-BSCSCAN_KEY=your_key_here
-NEWSAPI_KEY=your_key_here
-```
-
-## Dependencies
-
-- `aiohttp` - Async HTTP client
-- `asyncio` - Async programming
-- `datetime` - Timestamp handling
-- `utils.api_client` - Robust API client with retry logic
-- `utils.logger` - Structured JSON logging
-- `config` - Centralized configuration
-
-## Error Handling
-
-Collectors handle various error types:
-
-- **config_error** - Provider not configured
-- **missing_api_key** - API key required but not available
-- **authentication** - API key invalid or expired
-- **rate_limit** - Rate limit exceeded
-- **timeout** - Request timeout
-- **server_error** - API server error (5xx)
-- **network_error** - Network connectivity issue
-- **api_error** - API-specific error
-- **exception** - Unexpected Python exception
-
-## Extending Collectors
-
-To add a new collector:
-
-1. Create a new module or add to existing category
-2. Implement collector function following the standard pattern
-3. Use `get_client()` for API requests
-4. Extract and calculate staleness from timestamps
-5. Return standardized output format
-6. Add to `__init__.py` exports
-7. Update this README
-
-Example:
-
-```python
-async def get_new_provider_data() -> Dict[str, Any]:
- """Fetch data from new provider"""
- provider = "NewProvider"
- category = "market_data"
- endpoint = "/api/v1/data"
-
- logger.info(f"Fetching data from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- # Make request
- url = f"{provider_config.endpoint_url}{endpoint}"
- response = await client.get(url)
-
- # Log request
- log_api_request(
- logger, provider, endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # Handle error
- return {
- "provider": provider,
- "category": category,
- "success": False,
- "error": response.get("error_message")
- }
-
- # Parse data and timestamps
- data = response["data"]
- data_timestamp = # extract from response
- staleness = calculate_staleness_minutes(data_timestamp)
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- log_error(logger, provider, "exception", str(e), endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "success": False,
- "error": str(e),
- "error_type": "exception"
- }
-```
-
-## Testing
-
-Test individual collectors:
-
-```bash
-# Test market data collector
-python -m collectors.market_data
-
-# Test explorers
-python -m collectors.explorers
-
-# Test news
-python -m collectors.news
-
-# Test sentiment
-python -m collectors.sentiment
-
-# Test on-chain (placeholder)
-python -m collectors.onchain
-```
-
-## Performance
-
-- Collectors run concurrently using `asyncio.gather()`
-- Typical response times: 100-2000ms per collector
-- Connection pooling for efficiency
-- Configurable timeouts
-- Automatic retry with exponential backoff
-
-## Monitoring
-
-All collectors provide metrics for monitoring:
-
-- **Success Rate** - Percentage of successful collections
-- **Response Time** - API response time in milliseconds
-- **Staleness** - Data age in minutes
-- **Error Types** - Classification of failures
-- **Retry Count** - Number of retries needed
-
-## Future Enhancements
-
-1. **On-Chain Implementation**
- - Complete The Graph integration
- - Implement Blockchair endpoints
- - Add Glassnode metrics
-
-2. **Additional Providers**
- - Messari
- - DeFiLlama
- - CoinAPI
- - Nomics
-
-3. **Advanced Features**
- - Circuit breaker pattern
- - Data caching
- - Webhook notifications
- - Real-time streaming
-
-4. **Performance**
- - Redis caching
- - Database persistence
- - Rate limit optimization
- - Parallel processing
-
-## Support
-
-For issues or questions:
-1. Check the logs for detailed error messages
-2. Verify API keys are configured correctly
-3. Review provider rate limits
-4. Check network connectivity
-5. Consult provider documentation
-
-## License
-
-Part of the Crypto API Monitoring system.
diff --git a/app/collectors/__init__.py b/app/collectors/__init__.py
deleted file mode 100644
index 0deb918834d124174aa2831679d9d7df4dd86af0..0000000000000000000000000000000000000000
--- a/app/collectors/__init__.py
+++ /dev/null
@@ -1,68 +0,0 @@
-"""
-Collectors Package
-Data collection modules for cryptocurrency APIs
-
-Modules:
-- market_data: CoinGecko, CoinMarketCap, Binance market data
-- explorers: Etherscan, BscScan, TronScan blockchain explorers
-- news: CryptoPanic, NewsAPI news aggregation
-- sentiment: Alternative.me Fear & Greed Index
-- onchain: The Graph, Blockchair on-chain analytics (placeholder)
-"""
-
-from collectors.market_data import (
- get_coingecko_simple_price,
- get_coinmarketcap_quotes,
- get_binance_ticker,
- collect_market_data
-)
-
-from collectors.explorers import (
- get_etherscan_gas_price,
- get_bscscan_bnb_price,
- get_tronscan_stats,
- collect_explorer_data
-)
-
-from collectors.news import (
- get_cryptopanic_posts,
- get_newsapi_headlines,
- collect_news_data
-)
-
-from collectors.sentiment import (
- get_fear_greed_index,
- collect_sentiment_data
-)
-
-from collectors.onchain import (
- get_the_graph_data,
- get_blockchair_data,
- get_glassnode_metrics,
- collect_onchain_data
-)
-
-__all__ = [
- # Market Data
- "get_coingecko_simple_price",
- "get_coinmarketcap_quotes",
- "get_binance_ticker",
- "collect_market_data",
- # Explorers
- "get_etherscan_gas_price",
- "get_bscscan_bnb_price",
- "get_tronscan_stats",
- "collect_explorer_data",
- # News
- "get_cryptopanic_posts",
- "get_newsapi_headlines",
- "collect_news_data",
- # Sentiment
- "get_fear_greed_index",
- "collect_sentiment_data",
- # On-chain
- "get_the_graph_data",
- "get_blockchair_data",
- "get_glassnode_metrics",
- "collect_onchain_data",
-]
diff --git a/app/collectors/data_persistence.py b/app/collectors/data_persistence.py
deleted file mode 100644
index ad1526fbbc75bea9b7b5531e6067ba3985ebc7a5..0000000000000000000000000000000000000000
--- a/app/collectors/data_persistence.py
+++ /dev/null
@@ -1,500 +0,0 @@
-"""
-Data Persistence Module
-Saves collected data from all collectors into the database
-"""
-
-from datetime import datetime
-from typing import Dict, List, Any, Optional
-from database.db_manager import db_manager
-from utils.logger import setup_logger
-
-logger = setup_logger("data_persistence")
-
-
-class DataPersistence:
- """
- Handles saving collected data to the database
- """
-
- def __init__(self):
- """Initialize data persistence"""
- self.stats = {
- 'market_prices_saved': 0,
- 'news_saved': 0,
- 'sentiment_saved': 0,
- 'whale_txs_saved': 0,
- 'gas_prices_saved': 0,
- 'blockchain_stats_saved': 0
- }
-
- def reset_stats(self):
- """Reset persistence statistics"""
- for key in self.stats:
- self.stats[key] = 0
-
- def get_stats(self) -> Dict[str, int]:
- """Get persistence statistics"""
- return self.stats.copy()
-
- def save_market_data(self, results: List[Dict[str, Any]]) -> int:
- """
- Save market data to database
-
- Args:
- results: List of market data results from collectors
-
- Returns:
- Number of prices saved
- """
- saved_count = 0
-
- for result in results:
- if not result.get('success', False):
- continue
-
- provider = result.get('provider', 'Unknown')
- data = result.get('data')
-
- if not data:
- continue
-
- try:
- # CoinGecko format
- if provider == "CoinGecko" and isinstance(data, dict):
- # Map CoinGecko coin IDs to symbols
- symbol_map = {
- 'bitcoin': 'BTC',
- 'ethereum': 'ETH',
- 'binancecoin': 'BNB'
- }
-
- for coin_id, coin_data in data.items():
- if isinstance(coin_data, dict) and 'usd' in coin_data:
- symbol = symbol_map.get(coin_id, coin_id.upper())
-
- db_manager.save_market_price(
- symbol=symbol,
- price_usd=coin_data.get('usd', 0),
- market_cap=coin_data.get('usd_market_cap'),
- volume_24h=coin_data.get('usd_24h_vol'),
- price_change_24h=coin_data.get('usd_24h_change'),
- source=provider
- )
- saved_count += 1
-
- # Binance format
- elif provider == "Binance" and isinstance(data, dict):
- # Binance returns symbol -> price mapping
- for symbol, price in data.items():
- if isinstance(price, (int, float)):
- # Remove "USDT" suffix if present
- clean_symbol = symbol.replace('USDT', '')
-
- db_manager.save_market_price(
- symbol=clean_symbol,
- price_usd=float(price),
- source=provider
- )
- saved_count += 1
-
- # CoinMarketCap format
- elif provider == "CoinMarketCap" and isinstance(data, dict):
- if 'data' in data:
- for coin_id, coin_data in data['data'].items():
- if isinstance(coin_data, dict):
- symbol = coin_data.get('symbol', '').upper()
- quote_usd = coin_data.get('quote', {}).get('USD', {})
-
- if symbol and quote_usd:
- db_manager.save_market_price(
- symbol=symbol,
- price_usd=quote_usd.get('price', 0),
- market_cap=quote_usd.get('market_cap'),
- volume_24h=quote_usd.get('volume_24h'),
- price_change_24h=quote_usd.get('percent_change_24h'),
- source=provider
- )
- saved_count += 1
-
- except Exception as e:
- logger.error(f"Error saving market data from {provider}: {e}", exc_info=True)
-
- self.stats['market_prices_saved'] += saved_count
- if saved_count > 0:
- logger.info(f"Saved {saved_count} market prices to database")
-
- return saved_count
-
- def save_news_data(self, results: List[Dict[str, Any]]) -> int:
- """
- Save news data to database
-
- Args:
- results: List of news results from collectors
-
- Returns:
- Number of articles saved
- """
- saved_count = 0
-
- for result in results:
- if not result.get('success', False):
- continue
-
- provider = result.get('provider', 'Unknown')
- data = result.get('data')
-
- if not data:
- continue
-
- try:
- # CryptoPanic format
- if provider == "CryptoPanic" and isinstance(data, dict):
- results_list = data.get('results', [])
-
- for article in results_list:
- if not isinstance(article, dict):
- continue
-
- # Parse published_at
- published_at = None
- if 'created_at' in article:
- try:
- pub_str = article['created_at']
- if pub_str.endswith('Z'):
- pub_str = pub_str.replace('Z', '+00:00')
- published_at = datetime.fromisoformat(pub_str)
- except:
- published_at = datetime.utcnow()
-
- if not published_at:
- published_at = datetime.utcnow()
-
- # Extract currencies as tags
- currencies = article.get('currencies', [])
- tags = ','.join([c.get('code', '') for c in currencies if isinstance(c, dict)])
-
- db_manager.save_news_article(
- title=article.get('title', ''),
- content=article.get('body', ''),
- source=provider,
- url=article.get('url', ''),
- published_at=published_at,
- sentiment=article.get('sentiment'),
- tags=tags
- )
- saved_count += 1
-
- # NewsAPI format (newsdata.io)
- elif provider == "NewsAPI" and isinstance(data, dict):
- results_list = data.get('results', [])
-
- for article in results_list:
- if not isinstance(article, dict):
- continue
-
- # Parse published_at
- published_at = None
- if 'pubDate' in article:
- try:
- pub_str = article['pubDate']
- if pub_str.endswith('Z'):
- pub_str = pub_str.replace('Z', '+00:00')
- published_at = datetime.fromisoformat(pub_str)
- except:
- published_at = datetime.utcnow()
-
- if not published_at:
- published_at = datetime.utcnow()
-
- # Extract keywords as tags
- keywords = article.get('keywords', [])
- tags = ','.join(keywords) if isinstance(keywords, list) else ''
-
- db_manager.save_news_article(
- title=article.get('title', ''),
- content=article.get('description', ''),
- source=provider,
- url=article.get('link', ''),
- published_at=published_at,
- tags=tags
- )
- saved_count += 1
-
- except Exception as e:
- logger.error(f"Error saving news data from {provider}: {e}", exc_info=True)
-
- self.stats['news_saved'] += saved_count
- if saved_count > 0:
- logger.info(f"Saved {saved_count} news articles to database")
-
- return saved_count
-
- def save_sentiment_data(self, results: List[Dict[str, Any]]) -> int:
- """
- Save sentiment data to database
-
- Args:
- results: List of sentiment results from collectors
-
- Returns:
- Number of sentiment metrics saved
- """
- saved_count = 0
-
- for result in results:
- if not result.get('success', False):
- continue
-
- provider = result.get('provider', 'Unknown')
- data = result.get('data')
-
- if not data:
- continue
-
- try:
- # Fear & Greed Index format
- if provider == "AlternativeMe" and isinstance(data, dict):
- data_list = data.get('data', [])
-
- if data_list and isinstance(data_list, list):
- index_data = data_list[0]
-
- if isinstance(index_data, dict):
- value = float(index_data.get('value', 50))
- value_classification = index_data.get('value_classification', 'neutral')
-
- # Map classification to standard format
- classification_map = {
- 'Extreme Fear': 'extreme_fear',
- 'Fear': 'fear',
- 'Neutral': 'neutral',
- 'Greed': 'greed',
- 'Extreme Greed': 'extreme_greed'
- }
-
- classification = classification_map.get(
- value_classification,
- value_classification.lower().replace(' ', '_')
- )
-
- # Parse timestamp
- timestamp = None
- if 'timestamp' in index_data:
- try:
- timestamp = datetime.fromtimestamp(int(index_data['timestamp']))
- except:
- pass
-
- db_manager.save_sentiment_metric(
- metric_name='fear_greed_index',
- value=value,
- classification=classification,
- source=provider,
- timestamp=timestamp
- )
- saved_count += 1
-
- except Exception as e:
- logger.error(f"Error saving sentiment data from {provider}: {e}", exc_info=True)
-
- self.stats['sentiment_saved'] += saved_count
- if saved_count > 0:
- logger.info(f"Saved {saved_count} sentiment metrics to database")
-
- return saved_count
-
- def save_whale_data(self, results: List[Dict[str, Any]]) -> int:
- """
- Save whale transaction data to database
-
- Args:
- results: List of whale tracking results from collectors
-
- Returns:
- Number of whale transactions saved
- """
- saved_count = 0
-
- for result in results:
- if not result.get('success', False):
- continue
-
- provider = result.get('provider', 'Unknown')
- data = result.get('data')
-
- if not data:
- continue
-
- try:
- # WhaleAlert format
- if provider == "WhaleAlert" and isinstance(data, dict):
- transactions = data.get('transactions', [])
-
- for tx in transactions:
- if not isinstance(tx, dict):
- continue
-
- # Parse timestamp
- timestamp = None
- if 'timestamp' in tx:
- try:
- timestamp = datetime.fromtimestamp(tx['timestamp'])
- except:
- timestamp = datetime.utcnow()
-
- if not timestamp:
- timestamp = datetime.utcnow()
-
- # Extract addresses
- from_address = tx.get('from', {}).get('address', '') if isinstance(tx.get('from'), dict) else ''
- to_address = tx.get('to', {}).get('address', '') if isinstance(tx.get('to'), dict) else ''
-
- db_manager.save_whale_transaction(
- blockchain=tx.get('blockchain', 'unknown'),
- transaction_hash=tx.get('hash', ''),
- from_address=from_address,
- to_address=to_address,
- amount=float(tx.get('amount', 0)),
- amount_usd=float(tx.get('amount_usd', 0)),
- source=provider,
- timestamp=timestamp
- )
- saved_count += 1
-
- except Exception as e:
- logger.error(f"Error saving whale data from {provider}: {e}", exc_info=True)
-
- self.stats['whale_txs_saved'] += saved_count
- if saved_count > 0:
- logger.info(f"Saved {saved_count} whale transactions to database")
-
- return saved_count
-
- def save_blockchain_data(self, results: List[Dict[str, Any]]) -> int:
- """
- Save blockchain data (gas prices, stats) to database
-
- Args:
- results: List of blockchain results from collectors
-
- Returns:
- Number of records saved
- """
- saved_count = 0
-
- for result in results:
- if not result.get('success', False):
- continue
-
- provider = result.get('provider', 'Unknown')
- data = result.get('data')
-
- if not data:
- continue
-
- try:
- # Etherscan gas price format
- if provider == "Etherscan" and isinstance(data, dict):
- if 'result' in data:
- gas_data = data['result']
-
- if isinstance(gas_data, dict):
- db_manager.save_gas_price(
- blockchain='ethereum',
- gas_price_gwei=float(gas_data.get('ProposeGasPrice', 0)),
- fast_gas_price=float(gas_data.get('FastGasPrice', 0)),
- standard_gas_price=float(gas_data.get('ProposeGasPrice', 0)),
- slow_gas_price=float(gas_data.get('SafeGasPrice', 0)),
- source=provider
- )
- saved_count += 1
- self.stats['gas_prices_saved'] += 1
-
- # Other blockchain explorers
- elif provider in ["BSCScan", "PolygonScan"]:
- blockchain_map = {
- "BSCScan": "bsc",
- "PolygonScan": "polygon"
- }
- blockchain = blockchain_map.get(provider, provider.lower())
-
- if 'result' in data and isinstance(data['result'], dict):
- gas_data = data['result']
-
- db_manager.save_gas_price(
- blockchain=blockchain,
- gas_price_gwei=float(gas_data.get('ProposeGasPrice', 0)),
- fast_gas_price=float(gas_data.get('FastGasPrice', 0)),
- standard_gas_price=float(gas_data.get('ProposeGasPrice', 0)),
- slow_gas_price=float(gas_data.get('SafeGasPrice', 0)),
- source=provider
- )
- saved_count += 1
- self.stats['gas_prices_saved'] += 1
-
- except Exception as e:
- logger.error(f"Error saving blockchain data from {provider}: {e}", exc_info=True)
-
- if saved_count > 0:
- logger.info(f"Saved {saved_count} blockchain records to database")
-
- return saved_count
-
- def save_all_data(self, results: Dict[str, Any]) -> Dict[str, int]:
- """
- Save all collected data to database
-
- Args:
- results: Results dictionary from master collector
-
- Returns:
- Dictionary with save statistics
- """
- logger.info("=" * 60)
- logger.info("Saving collected data to database...")
- logger.info("=" * 60)
-
- self.reset_stats()
-
- data = results.get('data', {})
-
- # Save market data
- if 'market_data' in data:
- self.save_market_data(data['market_data'])
-
- # Save news data
- if 'news' in data:
- self.save_news_data(data['news'])
-
- # Save sentiment data
- if 'sentiment' in data:
- self.save_sentiment_data(data['sentiment'])
-
- # Save whale tracking data
- if 'whale_tracking' in data:
- self.save_whale_data(data['whale_tracking'])
-
- # Save blockchain data
- if 'blockchain' in data:
- self.save_blockchain_data(data['blockchain'])
-
- stats = self.get_stats()
- total_saved = sum(stats.values())
-
- logger.info("=" * 60)
- logger.info("Data Persistence Complete")
- logger.info(f"Total records saved: {total_saved}")
- logger.info(f" Market prices: {stats['market_prices_saved']}")
- logger.info(f" News articles: {stats['news_saved']}")
- logger.info(f" Sentiment metrics: {stats['sentiment_saved']}")
- logger.info(f" Whale transactions: {stats['whale_txs_saved']}")
- logger.info(f" Gas prices: {stats['gas_prices_saved']}")
- logger.info(f" Blockchain stats: {stats['blockchain_stats_saved']}")
- logger.info("=" * 60)
-
- return stats
-
-
-# Global instance
-data_persistence = DataPersistence()
diff --git a/app/collectors/demo_collectors.py b/app/collectors/demo_collectors.py
deleted file mode 100644
index 4c3d088824d316d3fcace21f080e504d762b26ba..0000000000000000000000000000000000000000
--- a/app/collectors/demo_collectors.py
+++ /dev/null
@@ -1,197 +0,0 @@
-"""
-Demonstration Script for All Collector Modules
-
-This script demonstrates the usage of all collector modules and
-provides a comprehensive overview of data collection capabilities.
-"""
-
-import asyncio
-import json
-from datetime import datetime
-from typing import Dict, List, Any
-
-# Import all collector functions
-from collectors import (
- collect_market_data,
- collect_explorer_data,
- collect_news_data,
- collect_sentiment_data,
- collect_onchain_data
-)
-
-
-def print_separator(title: str = ""):
- """Print a formatted separator line"""
- if title:
- print(f"\n{'='*70}")
- print(f" {title}")
- print(f"{'='*70}\n")
- else:
- print(f"{'='*70}\n")
-
-
-def format_result_summary(result: Dict[str, Any]) -> str:
- """Format a single result for display"""
- lines = []
- lines.append(f"Provider: {result.get('provider', 'Unknown')}")
- lines.append(f"Category: {result.get('category', 'Unknown')}")
- lines.append(f"Success: {result.get('success', False)}")
-
- if result.get('success'):
- lines.append(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- staleness = result.get('staleness_minutes')
- if staleness is not None:
- lines.append(f"Data Staleness: {staleness:.2f} minutes")
-
- # Add provider-specific info
- if result.get('index_value'):
- lines.append(f"Fear & Greed Index: {result['index_value']} ({result['index_classification']})")
- if result.get('post_count'):
- lines.append(f"Posts: {result['post_count']}")
- if result.get('article_count'):
- lines.append(f"Articles: {result['article_count']}")
- if result.get('is_placeholder'):
- lines.append("Status: PLACEHOLDER IMPLEMENTATION")
- else:
- lines.append(f"Error Type: {result.get('error_type', 'unknown')}")
- lines.append(f"Error: {result.get('error', 'Unknown error')}")
-
- return "\n".join(lines)
-
-
-def print_category_summary(category: str, results: List[Dict[str, Any]]):
- """Print summary for a category of collectors"""
- print_separator(f"{category.upper()}")
-
- total = len(results)
- successful = sum(1 for r in results if r.get('success', False))
-
- print(f"Total Collectors: {total}")
- print(f"Successful: {successful}")
- print(f"Failed: {total - successful}")
- print()
-
- for i, result in enumerate(results, 1):
- print(f"[{i}/{total}] {'-'*60}")
- print(format_result_summary(result))
- print()
-
-
-async def collect_all_data() -> Dict[str, List[Dict[str, Any]]]:
- """
- Collect data from all categories concurrently
-
- Returns:
- Dictionary with categories as keys and results as values
- """
- print_separator("Starting Data Collection from All Sources")
- print(f"Timestamp: {datetime.utcnow().isoformat()}Z\n")
-
- # Run all collectors concurrently
- print("Executing all collectors in parallel...")
-
- market_results, explorer_results, news_results, sentiment_results, onchain_results = await asyncio.gather(
- collect_market_data(),
- collect_explorer_data(),
- collect_news_data(),
- collect_sentiment_data(),
- collect_onchain_data(),
- return_exceptions=True
- )
-
- # Handle any exceptions
- def handle_exception(result, category):
- if isinstance(result, Exception):
- return [{
- "provider": "Unknown",
- "category": category,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- }]
- return result
-
- return {
- "market_data": handle_exception(market_results, "market_data"),
- "explorers": handle_exception(explorer_results, "blockchain_explorers"),
- "news": handle_exception(news_results, "news"),
- "sentiment": handle_exception(sentiment_results, "sentiment"),
- "onchain": handle_exception(onchain_results, "onchain_analytics")
- }
-
-
-async def main():
- """Main demonstration function"""
- print_separator("Cryptocurrency Data Collector - Comprehensive Demo")
-
- # Collect all data
- all_results = await collect_all_data()
-
- # Print results by category
- print_category_summary("Market Data Collection", all_results["market_data"])
- print_category_summary("Blockchain Explorer Data", all_results["explorers"])
- print_category_summary("News Data Collection", all_results["news"])
- print_category_summary("Sentiment Data Collection", all_results["sentiment"])
- print_category_summary("On-Chain Analytics Data", all_results["onchain"])
-
- # Overall statistics
- print_separator("Overall Collection Statistics")
-
- total_collectors = sum(len(results) for results in all_results.values())
- total_successful = sum(
- sum(1 for r in results if r.get('success', False))
- for results in all_results.values()
- )
- total_failed = total_collectors - total_successful
-
- # Calculate average response time for successful calls
- response_times = [
- r.get('response_time_ms', 0)
- for results in all_results.values()
- for r in results
- if r.get('success', False) and 'response_time_ms' in r
- ]
- avg_response_time = sum(response_times) / len(response_times) if response_times else 0
-
- print(f"Total Collectors Run: {total_collectors}")
- print(f"Successful: {total_successful} ({total_successful/total_collectors*100:.1f}%)")
- print(f"Failed: {total_failed} ({total_failed/total_collectors*100:.1f}%)")
- print(f"Average Response Time: {avg_response_time:.2f}ms")
- print()
-
- # Category breakdown
- print("By Category:")
- for category, results in all_results.items():
- successful = sum(1 for r in results if r.get('success', False))
- total = len(results)
- print(f" {category:20} {successful}/{total} successful")
-
- print_separator()
-
- # Save results to file
- output_file = f"collector_results_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}.json"
- try:
- with open(output_file, 'w') as f:
- json.dump(all_results, f, indent=2, default=str)
- print(f"Results saved to: {output_file}")
- except Exception as e:
- print(f"Failed to save results: {e}")
-
- print_separator("Demo Complete")
-
- return all_results
-
-
-if __name__ == "__main__":
- # Run the demonstration
- results = asyncio.run(main())
-
- # Exit with appropriate code
- total_collectors = sum(len(r) for r in results.values())
- total_successful = sum(
- sum(1 for item in r if item.get('success', False))
- for r in results.values()
- )
-
- # Exit with 0 if at least 50% successful, else 1
- exit(0 if total_successful >= total_collectors / 2 else 1)
diff --git a/app/collectors/explorers.py b/app/collectors/explorers.py
deleted file mode 100644
index c30b8952b9bb3f3740a264b6e37cd52ebff780ed..0000000000000000000000000000000000000000
--- a/app/collectors/explorers.py
+++ /dev/null
@@ -1,555 +0,0 @@
-"""
-Blockchain Explorer Data Collectors
-Fetches data from Etherscan, BscScan, and TronScan
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-from config import config
-
-logger = setup_logger("explorers_collector")
-
-
-def calculate_staleness_minutes(data_timestamp: Optional[datetime]) -> Optional[float]:
- """
- Calculate staleness in minutes from data timestamp to now
-
- Args:
- data_timestamp: Timestamp of the data
-
- Returns:
- Staleness in minutes or None if timestamp not available
- """
- if not data_timestamp:
- return None
-
- now = datetime.now(timezone.utc)
- if data_timestamp.tzinfo is None:
- data_timestamp = data_timestamp.replace(tzinfo=timezone.utc)
-
- delta = now - data_timestamp
- return delta.total_seconds() / 60.0
-
-
-async def get_etherscan_gas_price() -> Dict[str, Any]:
- """
- Get current Ethereum gas price from Etherscan
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "Etherscan"
- category = "blockchain_explorers"
- endpoint = "/api?module=gastracker&action=gasoracle"
-
- logger.info(f"Fetching gas price from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Check if API key is available
- if provider_config.requires_key and not provider_config.api_key:
- error_msg = f"API key required but not configured for {provider}"
- log_error(logger, provider, "auth_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "missing_api_key"
- }
-
- # Build request URL
- url = provider_config.endpoint_url
- params = {
- "module": "gastracker",
- "action": "gasoracle",
- "apikey": provider_config.api_key
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Etherscan returns real-time data, so staleness is minimal
- data_timestamp = datetime.now(timezone.utc)
- staleness = 0.0
-
- # Check API response status
- if isinstance(data, dict):
- api_status = data.get("status")
- if api_status == "0":
- error_msg = data.get("message", "API returned error status")
- log_error(logger, provider, "api_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "api_error"
- }
-
- logger.info(f"{provider} - {endpoint} - Gas price retrieved, staleness: {staleness:.2f}m")
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_bscscan_bnb_price() -> Dict[str, Any]:
- """
- Get BNB price from BscScan
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "BscScan"
- category = "blockchain_explorers"
- endpoint = "/api?module=stats&action=bnbprice"
-
- logger.info(f"Fetching BNB price from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Check if API key is available
- if provider_config.requires_key and not provider_config.api_key:
- error_msg = f"API key required but not configured for {provider}"
- log_error(logger, provider, "auth_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "missing_api_key"
- }
-
- # Build request URL
- url = provider_config.endpoint_url
- params = {
- "module": "stats",
- "action": "bnbprice",
- "apikey": provider_config.api_key
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # BscScan returns real-time data
- data_timestamp = datetime.now(timezone.utc)
- staleness = 0.0
-
- # Check API response status
- if isinstance(data, dict):
- api_status = data.get("status")
- if api_status == "0":
- error_msg = data.get("message", "API returned error status")
- log_error(logger, provider, "api_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "api_error"
- }
-
- # Extract timestamp if available
- if "result" in data and isinstance(data["result"], dict):
- if "ethusd_timestamp" in data["result"]:
- try:
- data_timestamp = datetime.fromtimestamp(
- int(data["result"]["ethusd_timestamp"]),
- tz=timezone.utc
- )
- staleness = calculate_staleness_minutes(data_timestamp)
- except:
- pass
-
- logger.info(f"{provider} - {endpoint} - BNB price retrieved, staleness: {staleness:.2f}m")
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_tronscan_stats() -> Dict[str, Any]:
- """
- Get TRX network statistics from TronScan
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "TronScan"
- category = "blockchain_explorers"
- endpoint = "/system/status"
-
- logger.info(f"Fetching network stats from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Build request URL
- url = f"{provider_config.endpoint_url}{endpoint}"
- headers = {}
-
- # Add API key if available
- if provider_config.requires_key and provider_config.api_key:
- headers["TRON-PRO-API-KEY"] = provider_config.api_key
-
- # Make request
- response = await client.get(
- url,
- headers=headers if headers else None,
- timeout=provider_config.timeout_ms // 1000
- )
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # TronScan returns real-time data
- data_timestamp = datetime.now(timezone.utc)
- staleness = 0.0
-
- # Parse timestamp if available in response
- if isinstance(data, dict):
- # TronScan may include timestamp in various fields
- if "timestamp" in data:
- try:
- data_timestamp = datetime.fromtimestamp(
- int(data["timestamp"]) / 1000, # TronScan uses milliseconds
- tz=timezone.utc
- )
- staleness = calculate_staleness_minutes(data_timestamp)
- except:
- pass
-
- logger.info(f"{provider} - {endpoint} - Network stats retrieved, staleness: {staleness:.2f}m")
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_explorer_data() -> List[Dict[str, Any]]:
- """
- Main function to collect blockchain explorer data from all sources
-
- Returns:
- List of results from all explorer data collectors
- """
- logger.info("Starting blockchain explorer data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_etherscan_gas_price(),
- get_bscscan_bnb_price(),
- get_tronscan_stats(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "blockchain_explorers",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- logger.info(f"Explorer data collection complete: {successful}/{len(processed_results)} successful")
-
- return processed_results
-
-
-class ExplorerDataCollector:
- """
- Explorer Data Collector class for WebSocket streaming interface
- Wraps the standalone explorer data collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the explorer data collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect blockchain explorer data from all sources
-
- Returns:
- Dict with aggregated explorer data
- """
- results = await collect_explorer_data()
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "latest_block": None,
- "network_hashrate": None,
- "difficulty": None,
- "mempool_size": None,
- "transactions_count": None,
- "gas_prices": {},
- "sources": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
- aggregated["sources"].append(provider)
-
- data = result["data"]
-
- # Parse gas price data
- if "result" in data and isinstance(data["result"], dict):
- gas_data = data["result"]
- if provider == "Etherscan":
- aggregated["gas_prices"]["ethereum"] = {
- "safe": gas_data.get("SafeGasPrice"),
- "propose": gas_data.get("ProposeGasPrice"),
- "fast": gas_data.get("FastGasPrice")
- }
- elif provider == "BscScan":
- aggregated["gas_prices"]["bsc"] = gas_data.get("result")
-
- # Parse network stats
- if provider == "TronScan" and "data" in data:
- stats = data["data"]
- aggregated["latest_block"] = stats.get("latestBlock")
- aggregated["transactions_count"] = stats.get("totalTransaction")
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_explorer_data()
-
- print("\n=== Blockchain Explorer Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Staleness: {result.get('staleness_minutes', 'N/A')} minutes")
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/market_data.py b/app/collectors/market_data.py
deleted file mode 100644
index a58d20e390c66027ed4cc5a4344187e517f87474..0000000000000000000000000000000000000000
--- a/app/collectors/market_data.py
+++ /dev/null
@@ -1,540 +0,0 @@
-"""
-Market Data Collectors
-Fetches cryptocurrency market data from CoinGecko, CoinMarketCap, and Binance
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-from config import config
-
-logger = setup_logger("market_data_collector")
-
-
-def calculate_staleness_minutes(data_timestamp: Optional[datetime]) -> Optional[float]:
- """
- Calculate staleness in minutes from data timestamp to now
-
- Args:
- data_timestamp: Timestamp of the data
-
- Returns:
- Staleness in minutes or None if timestamp not available
- """
- if not data_timestamp:
- return None
-
- now = datetime.now(timezone.utc)
- if data_timestamp.tzinfo is None:
- data_timestamp = data_timestamp.replace(tzinfo=timezone.utc)
-
- delta = now - data_timestamp
- return delta.total_seconds() / 60.0
-
-
-async def get_coingecko_simple_price() -> Dict[str, Any]:
- """
- Fetch BTC, ETH, BNB prices from CoinGecko simple/price endpoint
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "CoinGecko"
- category = "market_data"
- endpoint = "/simple/price"
-
- logger.info(f"Fetching simple price from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Build request URL
- url = f"{provider_config.endpoint_url}{endpoint}"
- params = {
- "ids": "bitcoin,ethereum,binancecoin",
- "vs_currencies": "usd",
- "include_market_cap": "true",
- "include_24hr_vol": "true",
- "include_24hr_change": "true",
- "include_last_updated_at": "true"
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamps from response
- data_timestamp = None
- if isinstance(data, dict):
- # CoinGecko returns last_updated_at as Unix timestamp
- for coin_data in data.values():
- if isinstance(coin_data, dict) and "last_updated_at" in coin_data:
- data_timestamp = datetime.fromtimestamp(
- coin_data["last_updated_at"],
- tz=timezone.utc
- )
- break
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- logger.info(
- f"{provider} - {endpoint} - Retrieved {len(data) if isinstance(data, dict) else 0} coins, "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_coinmarketcap_quotes() -> Dict[str, Any]:
- """
- Fetch BTC, ETH, BNB market data from CoinMarketCap quotes endpoint
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "CoinMarketCap"
- category = "market_data"
- endpoint = "/cryptocurrency/quotes/latest"
-
- logger.info(f"Fetching quotes from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Check if API key is available
- if provider_config.requires_key and not provider_config.api_key:
- error_msg = f"API key required but not configured for {provider}"
- log_error(logger, provider, "auth_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "missing_api_key"
- }
-
- # Build request
- url = f"{provider_config.endpoint_url}{endpoint}"
- headers = {
- "X-CMC_PRO_API_KEY": provider_config.api_key,
- "Accept": "application/json"
- }
- params = {
- "symbol": "BTC,ETH,BNB",
- "convert": "USD"
- }
-
- # Make request
- response = await client.get(
- url,
- headers=headers,
- params=params,
- timeout=provider_config.timeout_ms // 1000
- )
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamp from response
- data_timestamp = None
- if isinstance(data, dict) and "data" in data:
- # CoinMarketCap response structure
- for coin_data in data["data"].values():
- if isinstance(coin_data, dict) and "quote" in coin_data:
- quote = coin_data.get("quote", {}).get("USD", {})
- if "last_updated" in quote:
- try:
- data_timestamp = datetime.fromisoformat(
- quote["last_updated"].replace("Z", "+00:00")
- )
- break
- except:
- pass
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- coin_count = len(data.get("data", {})) if isinstance(data, dict) else 0
- logger.info(
- f"{provider} - {endpoint} - Retrieved {coin_count} coins, "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_binance_ticker() -> Dict[str, Any]:
- """
- Fetch ticker data from Binance public API (24hr ticker)
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "Binance"
- category = "market_data"
- endpoint = "/api/v3/ticker/24hr"
-
- logger.info(f"Fetching 24hr ticker from {provider}")
-
- try:
- client = get_client()
-
- # Binance API base URL
- url = f"https://api.binance.com{endpoint}"
- params = {
- "symbols": '["BTCUSDT","ETHUSDT","BNBUSDT"]'
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamp from response
- # Binance returns closeTime as Unix timestamp in milliseconds
- data_timestamp = None
- if isinstance(data, list) and len(data) > 0:
- first_ticker = data[0]
- if isinstance(first_ticker, dict) and "closeTime" in first_ticker:
- try:
- data_timestamp = datetime.fromtimestamp(
- first_ticker["closeTime"] / 1000,
- tz=timezone.utc
- )
- except:
- pass
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- ticker_count = len(data) if isinstance(data, list) else 0
- logger.info(
- f"{provider} - {endpoint} - Retrieved {ticker_count} tickers, "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_market_data() -> List[Dict[str, Any]]:
- """
- Main function to collect market data from all sources
-
- Returns:
- List of results from all market data collectors
- """
- logger.info("Starting market data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_coingecko_simple_price(),
- get_coinmarketcap_quotes(),
- get_binance_ticker(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "market_data",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- logger.info(f"Market data collection complete: {successful}/{len(processed_results)} successful")
-
- return processed_results
-
-
-class MarketDataCollector:
- """
- Market Data Collector class for WebSocket streaming interface
- Wraps the standalone market data collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the market data collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect market data from all sources
-
- Returns:
- Dict with aggregated market data
- """
- results = await collect_market_data()
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "prices": {},
- "volumes": {},
- "market_caps": {},
- "price_changes": {},
- "sources": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
- aggregated["sources"].append(provider)
-
- data = result["data"]
-
- # Parse CoinGecko data
- if provider == "CoinGecko" and isinstance(data, dict):
- for coin_id, coin_data in data.items():
- if isinstance(coin_data, dict):
- symbol = coin_id.upper()
- if "usd" in coin_data:
- aggregated["prices"][symbol] = coin_data["usd"]
- if "usd_market_cap" in coin_data:
- aggregated["market_caps"][symbol] = coin_data["usd_market_cap"]
- if "usd_24h_vol" in coin_data:
- aggregated["volumes"][symbol] = coin_data["usd_24h_vol"]
- if "usd_24h_change" in coin_data:
- aggregated["price_changes"][symbol] = coin_data["usd_24h_change"]
-
- # Parse CoinMarketCap data
- elif provider == "CoinMarketCap" and isinstance(data, dict):
- if "data" in data:
- for symbol, coin_data in data["data"].items():
- if isinstance(coin_data, dict) and "quote" in coin_data:
- quote = coin_data.get("quote", {}).get("USD", {})
- if "price" in quote:
- aggregated["prices"][symbol] = quote["price"]
- if "market_cap" in quote:
- aggregated["market_caps"][symbol] = quote["market_cap"]
- if "volume_24h" in quote:
- aggregated["volumes"][symbol] = quote["volume_24h"]
- if "percent_change_24h" in quote:
- aggregated["price_changes"][symbol] = quote["percent_change_24h"]
-
- # Parse Binance data
- elif provider == "Binance" and isinstance(data, list):
- for ticker in data:
- if isinstance(ticker, dict):
- symbol = ticker.get("symbol", "").replace("USDT", "")
- if "lastPrice" in ticker:
- aggregated["prices"][symbol] = float(ticker["lastPrice"])
- if "volume" in ticker:
- aggregated["volumes"][symbol] = float(ticker["volume"])
- if "priceChangePercent" in ticker:
- aggregated["price_changes"][symbol] = float(ticker["priceChangePercent"])
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_market_data()
-
- print("\n=== Market Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Staleness: {result.get('staleness_minutes', 'N/A')} minutes")
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/market_data_extended.py b/app/collectors/market_data_extended.py
deleted file mode 100644
index 175a6c0bfbbb020183dce828e98293a2d0409d29..0000000000000000000000000000000000000000
--- a/app/collectors/market_data_extended.py
+++ /dev/null
@@ -1,594 +0,0 @@
-"""
-Extended Market Data Collectors
-Fetches data from Coinpaprika, DefiLlama, Messari, CoinCap, and other market data sources
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("market_data_extended_collector")
-
-
-async def get_coinpaprika_tickers() -> Dict[str, Any]:
- """
- Fetch ticker data from Coinpaprika (free, no key required)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "Coinpaprika"
- category = "market_data"
- endpoint = "/tickers"
-
- logger.info(f"Fetching tickers from {provider}")
-
- try:
- client = get_client()
-
- # Coinpaprika API (free, no key needed)
- url = "https://api.coinpaprika.com/v1/tickers"
-
- params = {
- "quotes": "USD",
- "limit": 100
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Process top coins
- market_data = None
- if isinstance(data, list):
- top_10 = data[:10]
- total_market_cap = sum(coin.get("quotes", {}).get("USD", {}).get("market_cap", 0) for coin in top_10)
-
- market_data = {
- "total_coins": len(data),
- "top_10_market_cap": round(total_market_cap, 2),
- "top_10_coins": [
- {
- "symbol": coin.get("symbol"),
- "name": coin.get("name"),
- "price": coin.get("quotes", {}).get("USD", {}).get("price"),
- "market_cap": coin.get("quotes", {}).get("USD", {}).get("market_cap"),
- "volume_24h": coin.get("quotes", {}).get("USD", {}).get("volume_24h"),
- "percent_change_24h": coin.get("quotes", {}).get("USD", {}).get("percent_change_24h")
- }
- for coin in top_10
- ]
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {len(data) if isinstance(data, list) else 0} tickers")
-
- return {
- "provider": provider,
- "category": category,
- "data": market_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_defillama_tvl() -> Dict[str, Any]:
- """
- Fetch DeFi Total Value Locked from DefiLlama (free, no key required)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "DefiLlama"
- category = "defi_data"
- endpoint = "/tvl"
-
- logger.info(f"Fetching TVL data from {provider}")
-
- try:
- client = get_client()
-
- # DefiLlama API (free, no key needed)
- url = "https://api.llama.fi/v2/protocols"
-
- # Make request
- response = await client.get(url, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Process protocols
- tvl_data = None
- if isinstance(data, list):
- # Sort by TVL
- sorted_protocols = sorted(data, key=lambda x: x.get("tvl", 0), reverse=True)
- top_20 = sorted_protocols[:20]
-
- total_tvl = sum(p.get("tvl", 0) for p in data)
-
- tvl_data = {
- "total_protocols": len(data),
- "total_tvl": round(total_tvl, 2),
- "top_20_protocols": [
- {
- "name": p.get("name"),
- "symbol": p.get("symbol"),
- "tvl": round(p.get("tvl", 0), 2),
- "change_1d": p.get("change_1d"),
- "change_7d": p.get("change_7d"),
- "chains": p.get("chains", [])[:3] # Top 3 chains
- }
- for p in top_20
- ]
- }
-
- logger.info(
- f"{provider} - {endpoint} - Total TVL: ${tvl_data.get('total_tvl', 0):,.0f}"
- if tvl_data else f"{provider} - {endpoint} - No data"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": tvl_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_coincap_assets() -> Dict[str, Any]:
- """
- Fetch asset data from CoinCap (free, no key required)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "CoinCap"
- category = "market_data"
- endpoint = "/assets"
-
- logger.info(f"Fetching assets from {provider}")
-
- try:
- client = get_client()
-
- # CoinCap API (free, no key needed)
- url = "https://api.coincap.io/v2/assets"
-
- params = {"limit": 50}
-
- # Make request
- response = await client.get(url, params=params, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- raw_data = response["data"]
-
- # Process assets
- asset_data = None
- if isinstance(raw_data, dict) and "data" in raw_data:
- assets = raw_data["data"]
-
- top_10 = assets[:10] if isinstance(assets, list) else []
-
- asset_data = {
- "total_assets": len(assets) if isinstance(assets, list) else 0,
- "top_10_assets": [
- {
- "symbol": asset.get("symbol"),
- "name": asset.get("name"),
- "price_usd": float(asset.get("priceUsd", 0)),
- "market_cap_usd": float(asset.get("marketCapUsd", 0)),
- "volume_24h_usd": float(asset.get("volumeUsd24Hr", 0)),
- "change_percent_24h": float(asset.get("changePercent24Hr", 0))
- }
- for asset in top_10
- ]
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {asset_data.get('total_assets', 0)} assets")
-
- return {
- "provider": provider,
- "category": category,
- "data": asset_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_messari_assets(api_key: Optional[str] = None) -> Dict[str, Any]:
- """
- Fetch asset data from Messari
-
- Args:
- api_key: Messari API key (optional, has free tier)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "Messari"
- category = "market_data"
- endpoint = "/assets"
-
- logger.info(f"Fetching assets from {provider}")
-
- try:
- client = get_client()
-
- # Messari API
- url = "https://data.messari.io/api/v1/assets"
-
- params = {"limit": 20}
-
- headers = {}
- if api_key:
- headers["x-messari-api-key"] = api_key
-
- # Make request
- response = await client.get(url, params=params, headers=headers, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- raw_data = response["data"]
-
- # Process assets
- asset_data = None
- if isinstance(raw_data, dict) and "data" in raw_data:
- assets = raw_data["data"]
-
- asset_data = {
- "total_assets": len(assets) if isinstance(assets, list) else 0,
- "assets": [
- {
- "symbol": asset.get("symbol"),
- "name": asset.get("name"),
- "slug": asset.get("slug"),
- "metrics": {
- "market_cap": asset.get("metrics", {}).get("marketcap", {}).get("current_marketcap_usd"),
- "volume_24h": asset.get("metrics", {}).get("market_data", {}).get("volume_last_24_hours"),
- "price": asset.get("metrics", {}).get("market_data", {}).get("price_usd")
- }
- }
- for asset in assets[:10]
- ] if isinstance(assets, list) else []
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {asset_data.get('total_assets', 0)} assets")
-
- return {
- "provider": provider,
- "category": category,
- "data": asset_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_cryptocompare_toplist() -> Dict[str, Any]:
- """
- Fetch top cryptocurrencies from CryptoCompare (free tier available)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "CryptoCompare"
- category = "market_data"
- endpoint = "/top/totalvolfull"
-
- logger.info(f"Fetching top list from {provider}")
-
- try:
- client = get_client()
-
- # CryptoCompare API
- url = "https://min-api.cryptocompare.com/data/top/totalvolfull"
-
- params = {
- "limit": 20,
- "tsym": "USD"
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- raw_data = response["data"]
-
- # Process data
- toplist_data = None
- if isinstance(raw_data, dict) and "Data" in raw_data:
- coins = raw_data["Data"]
-
- toplist_data = {
- "total_coins": len(coins) if isinstance(coins, list) else 0,
- "top_coins": [
- {
- "symbol": coin.get("CoinInfo", {}).get("Name"),
- "name": coin.get("CoinInfo", {}).get("FullName"),
- "price": coin.get("RAW", {}).get("USD", {}).get("PRICE"),
- "market_cap": coin.get("RAW", {}).get("USD", {}).get("MKTCAP"),
- "volume_24h": coin.get("RAW", {}).get("USD", {}).get("VOLUME24HOUR"),
- "change_24h": coin.get("RAW", {}).get("USD", {}).get("CHANGEPCT24HOUR")
- }
- for coin in (coins[:10] if isinstance(coins, list) else [])
- ]
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {toplist_data.get('total_coins', 0)} coins")
-
- return {
- "provider": provider,
- "category": category,
- "data": toplist_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_extended_market_data(messari_key: Optional[str] = None) -> List[Dict[str, Any]]:
- """
- Main function to collect extended market data from all sources
-
- Args:
- messari_key: Optional Messari API key
-
- Returns:
- List of results from all extended market data collectors
- """
- logger.info("Starting extended market data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_coinpaprika_tickers(),
- get_defillama_tvl(),
- get_coincap_assets(),
- get_messari_assets(messari_key),
- get_cryptocompare_toplist(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "market_data",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- logger.info(f"Extended market data collection complete: {successful}/{len(processed_results)} successful")
-
- return processed_results
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- import os
-
- messari_key = os.getenv("MESSARI_API_KEY")
-
- results = await collect_extended_market_data(messari_key)
-
- print("\n=== Extended Market Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Category: {result['category']}")
- print(f"Success: {result['success']}")
-
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- data = result.get('data', {})
- if data:
- if 'total_tvl' in data:
- print(f"Total TVL: ${data['total_tvl']:,.0f}")
- elif 'total_assets' in data:
- print(f"Total Assets: {data['total_assets']}")
- elif 'total_coins' in data:
- print(f"Total Coins: {data['total_coins']}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/master_collector.py b/app/collectors/master_collector.py
deleted file mode 100644
index 91c1bb0608aaafec9dbba013f5ab1de866676bab..0000000000000000000000000000000000000000
--- a/app/collectors/master_collector.py
+++ /dev/null
@@ -1,402 +0,0 @@
-"""
-Master Collector - Aggregates all data sources
-Unified interface to collect data from all available collectors
-"""
-
-import asyncio
-import os
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.logger import setup_logger
-
-# Import all collectors
-from collectors.market_data import collect_market_data
-from collectors.market_data_extended import collect_extended_market_data
-from collectors.explorers import collect_explorer_data
-from collectors.news import collect_news
-from collectors.news_extended import collect_extended_news
-from collectors.sentiment import collect_sentiment
-from collectors.sentiment_extended import collect_extended_sentiment_data
-from collectors.onchain import collect_onchain_data
-from collectors.rpc_nodes import collect_rpc_data
-from collectors.whale_tracking import collect_whale_tracking_data
-
-# Import data persistence
-from collectors.data_persistence import data_persistence
-
-logger = setup_logger("master_collector")
-
-
-class DataSourceCollector:
- """
- Master collector that aggregates all data sources
- """
-
- def __init__(self):
- """Initialize the master collector"""
- self.api_keys = self._load_api_keys()
- logger.info("Master Collector initialized")
-
- def _load_api_keys(self) -> Dict[str, Optional[str]]:
- """
- Load API keys from environment variables
-
- Returns:
- Dict of API keys
- """
- return {
- # Market Data
- "coinmarketcap": os.getenv("COINMARKETCAP_KEY_1"),
- "messari": os.getenv("MESSARI_API_KEY"),
- "cryptocompare": os.getenv("CRYPTOCOMPARE_KEY"),
-
- # Blockchain Explorers
- "etherscan": os.getenv("ETHERSCAN_KEY_1"),
- "bscscan": os.getenv("BSCSCAN_KEY"),
- "tronscan": os.getenv("TRONSCAN_KEY"),
-
- # News
- "newsapi": os.getenv("NEWSAPI_KEY"),
-
- # RPC Nodes
- "infura": os.getenv("INFURA_API_KEY"),
- "alchemy": os.getenv("ALCHEMY_API_KEY"),
-
- # Whale Tracking
- "whalealert": os.getenv("WHALEALERT_API_KEY"),
-
- # HuggingFace
- "huggingface": os.getenv("HUGGINGFACE_TOKEN"),
- }
-
- async def collect_all_market_data(self) -> List[Dict[str, Any]]:
- """
- Collect data from all market data sources
-
- Returns:
- List of market data results
- """
- logger.info("Collecting all market data...")
-
- results = []
-
- # Core market data
- core_results = await collect_market_data()
- results.extend(core_results)
-
- # Extended market data
- extended_results = await collect_extended_market_data(
- messari_key=self.api_keys.get("messari")
- )
- results.extend(extended_results)
-
- logger.info(f"Market data collection complete: {len(results)} results")
- return results
-
- async def collect_all_blockchain_data(self) -> List[Dict[str, Any]]:
- """
- Collect data from all blockchain sources (explorers + RPC + on-chain)
-
- Returns:
- List of blockchain data results
- """
- logger.info("Collecting all blockchain data...")
-
- results = []
-
- # Blockchain explorers
- explorer_results = await collect_explorer_data()
- results.extend(explorer_results)
-
- # RPC nodes
- rpc_results = await collect_rpc_data(
- infura_key=self.api_keys.get("infura"),
- alchemy_key=self.api_keys.get("alchemy")
- )
- results.extend(rpc_results)
-
- # On-chain analytics
- onchain_results = await collect_onchain_data()
- results.extend(onchain_results)
-
- logger.info(f"Blockchain data collection complete: {len(results)} results")
- return results
-
- async def collect_all_news(self) -> List[Dict[str, Any]]:
- """
- Collect data from all news sources
-
- Returns:
- List of news results
- """
- logger.info("Collecting all news...")
-
- results = []
-
- # Core news
- core_results = await collect_news()
- results.extend(core_results)
-
- # Extended news (RSS feeds)
- extended_results = await collect_extended_news()
- results.extend(extended_results)
-
- logger.info(f"News collection complete: {len(results)} results")
- return results
-
- async def collect_all_sentiment(self) -> List[Dict[str, Any]]:
- """
- Collect data from all sentiment sources
-
- Returns:
- List of sentiment results
- """
- logger.info("Collecting all sentiment data...")
-
- results = []
-
- # Core sentiment
- core_results = await collect_sentiment()
- results.extend(core_results)
-
- # Extended sentiment
- extended_results = await collect_extended_sentiment_data()
- results.extend(extended_results)
-
- logger.info(f"Sentiment collection complete: {len(results)} results")
- return results
-
- async def collect_whale_tracking(self) -> List[Dict[str, Any]]:
- """
- Collect whale tracking data
-
- Returns:
- List of whale tracking results
- """
- logger.info("Collecting whale tracking data...")
-
- results = await collect_whale_tracking_data(
- whalealert_key=self.api_keys.get("whalealert")
- )
-
- logger.info(f"Whale tracking collection complete: {len(results)} results")
- return results
-
- async def collect_all_data(self) -> Dict[str, Any]:
- """
- Collect data from ALL available sources in parallel
-
- Returns:
- Dict with categorized results and statistics
- """
- logger.info("=" * 60)
- logger.info("Starting MASTER data collection from ALL sources")
- logger.info("=" * 60)
-
- start_time = datetime.now(timezone.utc)
-
- # Run all collections in parallel
- market_data, blockchain_data, news_data, sentiment_data, whale_data = await asyncio.gather(
- self.collect_all_market_data(),
- self.collect_all_blockchain_data(),
- self.collect_all_news(),
- self.collect_all_sentiment(),
- self.collect_whale_tracking(),
- return_exceptions=True
- )
-
- # Handle exceptions
- if isinstance(market_data, Exception):
- logger.error(f"Market data collection failed: {str(market_data)}")
- market_data = []
-
- if isinstance(blockchain_data, Exception):
- logger.error(f"Blockchain data collection failed: {str(blockchain_data)}")
- blockchain_data = []
-
- if isinstance(news_data, Exception):
- logger.error(f"News collection failed: {str(news_data)}")
- news_data = []
-
- if isinstance(sentiment_data, Exception):
- logger.error(f"Sentiment collection failed: {str(sentiment_data)}")
- sentiment_data = []
-
- if isinstance(whale_data, Exception):
- logger.error(f"Whale tracking collection failed: {str(whale_data)}")
- whale_data = []
-
- # Calculate statistics
- end_time = datetime.now(timezone.utc)
- duration = (end_time - start_time).total_seconds()
-
- total_sources = (
- len(market_data) +
- len(blockchain_data) +
- len(news_data) +
- len(sentiment_data) +
- len(whale_data)
- )
-
- successful_sources = sum([
- sum(1 for r in market_data if r.get("success", False)),
- sum(1 for r in blockchain_data if r.get("success", False)),
- sum(1 for r in news_data if r.get("success", False)),
- sum(1 for r in sentiment_data if r.get("success", False)),
- sum(1 for r in whale_data if r.get("success", False))
- ])
-
- placeholder_count = sum([
- sum(1 for r in market_data if r.get("is_placeholder", False)),
- sum(1 for r in blockchain_data if r.get("is_placeholder", False)),
- sum(1 for r in news_data if r.get("is_placeholder", False)),
- sum(1 for r in sentiment_data if r.get("is_placeholder", False)),
- sum(1 for r in whale_data if r.get("is_placeholder", False))
- ])
-
- # Aggregate results
- results = {
- "collection_timestamp": start_time.isoformat(),
- "duration_seconds": round(duration, 2),
- "statistics": {
- "total_sources": total_sources,
- "successful_sources": successful_sources,
- "failed_sources": total_sources - successful_sources,
- "placeholder_sources": placeholder_count,
- "success_rate": round(successful_sources / total_sources * 100, 2) if total_sources > 0 else 0,
- "categories": {
- "market_data": {
- "total": len(market_data),
- "successful": sum(1 for r in market_data if r.get("success", False))
- },
- "blockchain": {
- "total": len(blockchain_data),
- "successful": sum(1 for r in blockchain_data if r.get("success", False))
- },
- "news": {
- "total": len(news_data),
- "successful": sum(1 for r in news_data if r.get("success", False))
- },
- "sentiment": {
- "total": len(sentiment_data),
- "successful": sum(1 for r in sentiment_data if r.get("success", False))
- },
- "whale_tracking": {
- "total": len(whale_data),
- "successful": sum(1 for r in whale_data if r.get("success", False))
- }
- }
- },
- "data": {
- "market_data": market_data,
- "blockchain": blockchain_data,
- "news": news_data,
- "sentiment": sentiment_data,
- "whale_tracking": whale_data
- }
- }
-
- # Log summary
- logger.info("=" * 60)
- logger.info("MASTER COLLECTION COMPLETE")
- logger.info(f"Duration: {duration:.2f} seconds")
- logger.info(f"Total Sources: {total_sources}")
- logger.info(f"Successful: {successful_sources} ({results['statistics']['success_rate']}%)")
- logger.info(f"Failed: {total_sources - successful_sources}")
- logger.info(f"Placeholders: {placeholder_count}")
- logger.info("=" * 60)
- logger.info("Category Breakdown:")
- for category, stats in results['statistics']['categories'].items():
- logger.info(f" {category}: {stats['successful']}/{stats['total']}")
- logger.info("=" * 60)
-
- # Save all collected data to database
- try:
- persistence_stats = data_persistence.save_all_data(results)
- results['persistence_stats'] = persistence_stats
- except Exception as e:
- logger.error(f"Error persisting data to database: {e}", exc_info=True)
- results['persistence_stats'] = {'error': str(e)}
-
- return results
-
- async def collect_category(self, category: str) -> List[Dict[str, Any]]:
- """
- Collect data from a specific category
-
- Args:
- category: Category name (market_data, blockchain, news, sentiment, whale_tracking)
-
- Returns:
- List of results for the category
- """
- logger.info(f"Collecting data for category: {category}")
-
- if category == "market_data":
- return await self.collect_all_market_data()
- elif category == "blockchain":
- return await self.collect_all_blockchain_data()
- elif category == "news":
- return await self.collect_all_news()
- elif category == "sentiment":
- return await self.collect_all_sentiment()
- elif category == "whale_tracking":
- return await self.collect_whale_tracking()
- else:
- logger.error(f"Unknown category: {category}")
- return []
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- collector = DataSourceCollector()
-
- print("\n" + "=" * 80)
- print("CRYPTO DATA SOURCE MASTER COLLECTOR")
- print("Collecting data from ALL available sources...")
- print("=" * 80 + "\n")
-
- # Collect all data
- results = await collector.collect_all_data()
-
- # Print summary
- print("\n" + "=" * 80)
- print("COLLECTION SUMMARY")
- print("=" * 80)
- print(f"Duration: {results['duration_seconds']} seconds")
- print(f"Total Sources: {results['statistics']['total_sources']}")
- print(f"Successful: {results['statistics']['successful_sources']} "
- f"({results['statistics']['success_rate']}%)")
- print(f"Failed: {results['statistics']['failed_sources']}")
- print(f"Placeholders: {results['statistics']['placeholder_sources']}")
- print("\n" + "-" * 80)
- print("CATEGORY BREAKDOWN:")
- print("-" * 80)
-
- for category, stats in results['statistics']['categories'].items():
- success_rate = (stats['successful'] / stats['total'] * 100) if stats['total'] > 0 else 0
- print(f"{category:20} {stats['successful']:3}/{stats['total']:3} ({success_rate:5.1f}%)")
-
- print("=" * 80)
-
- # Print sample data from each category
- print("\n" + "=" * 80)
- print("SAMPLE DATA FROM EACH CATEGORY")
- print("=" * 80)
-
- for category, data_list in results['data'].items():
- print(f"\n{category.upper()}:")
- successful = [d for d in data_list if d.get('success', False)]
- if successful:
- sample = successful[0]
- print(f" Provider: {sample.get('provider', 'N/A')}")
- print(f" Success: {sample.get('success', False)}")
- if sample.get('data'):
- print(f" Data keys: {list(sample.get('data', {}).keys())[:5]}")
- else:
- print(" No successful data")
-
- print("\n" + "=" * 80)
-
- asyncio.run(main())
diff --git a/app/collectors/news.py b/app/collectors/news.py
deleted file mode 100644
index 3747e15c05d1a5d775767eacb31c2f8463523312..0000000000000000000000000000000000000000
--- a/app/collectors/news.py
+++ /dev/null
@@ -1,448 +0,0 @@
-"""
-News Data Collectors
-Fetches cryptocurrency news from CryptoPanic and NewsAPI
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-from config import config
-
-logger = setup_logger("news_collector")
-
-
-def calculate_staleness_minutes(data_timestamp: Optional[datetime]) -> Optional[float]:
- """
- Calculate staleness in minutes from data timestamp to now
-
- Args:
- data_timestamp: Timestamp of the data
-
- Returns:
- Staleness in minutes or None if timestamp not available
- """
- if not data_timestamp:
- return None
-
- now = datetime.now(timezone.utc)
- if data_timestamp.tzinfo is None:
- data_timestamp = data_timestamp.replace(tzinfo=timezone.utc)
-
- delta = now - data_timestamp
- return delta.total_seconds() / 60.0
-
-
-def parse_iso_timestamp(timestamp_str: str) -> Optional[datetime]:
- """
- Parse ISO timestamp string to datetime
-
- Args:
- timestamp_str: ISO format timestamp string
-
- Returns:
- datetime object or None if parsing fails
- """
- try:
- # Handle various ISO formats
- if timestamp_str.endswith('Z'):
- timestamp_str = timestamp_str.replace('Z', '+00:00')
- return datetime.fromisoformat(timestamp_str)
- except:
- return None
-
-
-async def get_cryptopanic_posts() -> Dict[str, Any]:
- """
- Fetch latest cryptocurrency news posts from CryptoPanic
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "CryptoPanic"
- category = "news"
- endpoint = "/posts/"
-
- logger.info(f"Fetching posts from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Build request URL
- url = f"{provider_config.endpoint_url}{endpoint}"
- params = {
- "auth_token": "free", # CryptoPanic offers free tier
- "public": "true",
- "kind": "news", # Get news posts
- "filter": "rising" # Get rising news
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamp from most recent post
- data_timestamp = None
- if isinstance(data, dict) and "results" in data:
- results = data["results"]
- if isinstance(results, list) and len(results) > 0:
- # Get the most recent post's timestamp
- first_post = results[0]
- if isinstance(first_post, dict) and "created_at" in first_post:
- data_timestamp = parse_iso_timestamp(first_post["created_at"])
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- # Count posts
- post_count = 0
- if isinstance(data, dict) and "results" in data:
- post_count = len(data["results"])
-
- logger.info(
- f"{provider} - {endpoint} - Retrieved {post_count} posts, "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0),
- "post_count": post_count
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_newsapi_headlines() -> Dict[str, Any]:
- """
- Fetch cryptocurrency headlines from NewsAPI (newsdata.io)
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "NewsAPI"
- category = "news"
- endpoint = "/news"
-
- logger.info(f"Fetching headlines from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Check if API key is available
- if provider_config.requires_key and not provider_config.api_key:
- error_msg = f"API key required but not configured for {provider}"
- log_error(logger, provider, "auth_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "missing_api_key"
- }
-
- # Build request URL
- url = f"{provider_config.endpoint_url}{endpoint}"
- params = {
- "apikey": provider_config.api_key,
- "q": "cryptocurrency OR bitcoin OR ethereum",
- "language": "en",
- "category": "business,technology"
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamp from most recent article
- data_timestamp = None
- if isinstance(data, dict) and "results" in data:
- results = data["results"]
- if isinstance(results, list) and len(results) > 0:
- # Get the most recent article's timestamp
- first_article = results[0]
- if isinstance(first_article, dict):
- # Try different timestamp fields
- timestamp_field = first_article.get("pubDate") or first_article.get("publishedAt")
- if timestamp_field:
- data_timestamp = parse_iso_timestamp(timestamp_field)
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- # Count articles
- article_count = 0
- if isinstance(data, dict) and "results" in data:
- article_count = len(data["results"])
-
- logger.info(
- f"{provider} - {endpoint} - Retrieved {article_count} articles, "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0),
- "article_count": article_count
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_news_data() -> List[Dict[str, Any]]:
- """
- Main function to collect news data from all sources
-
- Returns:
- List of results from all news collectors
- """
- logger.info("Starting news data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_cryptopanic_posts(),
- get_newsapi_headlines(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "news",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- total_items = sum(
- r.get("post_count", 0) + r.get("article_count", 0)
- for r in processed_results if r.get("success", False)
- )
-
- logger.info(
- f"News data collection complete: {successful}/{len(processed_results)} successful, "
- f"{total_items} total items"
- )
-
- return processed_results
-
-
-# Alias for backward compatibility
-collect_news = collect_news_data
-
-
-class NewsCollector:
- """
- News Collector class for WebSocket streaming interface
- Wraps the standalone news collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the news collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect news data from all sources
-
- Returns:
- Dict with aggregated news data
- """
- results = await collect_news_data()
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "articles": [],
- "sources": [],
- "categories": [],
- "breaking": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
- aggregated["sources"].append(provider)
-
- data = result["data"]
-
- # Parse CryptoPanic posts
- if provider == "CryptoPanic" and "results" in data:
- for post in data["results"][:10]: # Take top 10
- aggregated["articles"].append({
- "title": post.get("title"),
- "url": post.get("url"),
- "source": post.get("source", {}).get("title"),
- "published_at": post.get("published_at"),
- "kind": post.get("kind"),
- "votes": post.get("votes", {})
- })
-
- # Parse NewsAPI articles
- elif provider == "NewsAPI" and "articles" in data:
- for article in data["articles"][:10]: # Take top 10
- aggregated["articles"].append({
- "title": article.get("title"),
- "url": article.get("url"),
- "source": article.get("source", {}).get("name"),
- "published_at": article.get("publishedAt"),
- "description": article.get("description")
- })
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_news_data()
-
- print("\n=== News Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Staleness: {result.get('staleness_minutes', 'N/A')} minutes")
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- print(f"Items: {result.get('post_count', 0) + result.get('article_count', 0)}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/news_extended.py b/app/collectors/news_extended.py
deleted file mode 100644
index 155a7ca29f3f97c6c55df779b94f956646ac59ef..0000000000000000000000000000000000000000
--- a/app/collectors/news_extended.py
+++ /dev/null
@@ -1,362 +0,0 @@
-"""
-Extended News Collectors
-Fetches news from RSS feeds, CoinDesk, CoinTelegraph, and other crypto news sources
-"""
-
-import asyncio
-import feedparser
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("news_extended_collector")
-
-
-async def get_rss_feed(provider: str, feed_url: str) -> Dict[str, Any]:
- """
- Fetch and parse RSS feed from a news source
-
- Args:
- provider: Provider name
- feed_url: RSS feed URL
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- category = "news"
- endpoint = "/rss"
-
- logger.info(f"Fetching RSS feed from {provider}")
-
- try:
- client = get_client()
-
- # Fetch RSS feed
- response = await client.get(feed_url, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Parse RSS feed
- raw_data = response.get("raw_content", "")
- if not raw_data:
- raw_data = str(response.get("data", ""))
-
- # Use feedparser to parse RSS
- feed = feedparser.parse(raw_data)
-
- news_data = None
- if feed and hasattr(feed, 'entries'):
- entries = feed.entries[:10] # Get top 10 articles
-
- articles = []
- for entry in entries:
- article = {
- "title": entry.get("title", ""),
- "link": entry.get("link", ""),
- "published": entry.get("published", ""),
- "summary": entry.get("summary", "")[:200] if "summary" in entry else None
- }
- articles.append(article)
-
- news_data = {
- "feed_title": feed.feed.get("title", provider) if hasattr(feed, 'feed') else provider,
- "total_entries": len(feed.entries),
- "articles": articles
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {len(feed.entries) if feed else 0} articles")
-
- return {
- "provider": provider,
- "category": category,
- "data": news_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_coindesk_news() -> Dict[str, Any]:
- """
- Fetch news from CoinDesk RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("CoinDesk", "https://www.coindesk.com/arc/outboundfeeds/rss/")
-
-
-async def get_cointelegraph_news() -> Dict[str, Any]:
- """
- Fetch news from CoinTelegraph RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("CoinTelegraph", "https://cointelegraph.com/rss")
-
-
-async def get_decrypt_news() -> Dict[str, Any]:
- """
- Fetch news from Decrypt RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("Decrypt", "https://decrypt.co/feed")
-
-
-async def get_bitcoinmagazine_news() -> Dict[str, Any]:
- """
- Fetch news from Bitcoin Magazine RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("BitcoinMagazine", "https://bitcoinmagazine.com/.rss/full/")
-
-
-async def get_theblock_news() -> Dict[str, Any]:
- """
- Fetch news from The Block
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("TheBlock", "https://www.theblock.co/rss.xml")
-
-
-async def get_cryptoslate_news() -> Dict[str, Any]:
- """
- Fetch news from CryptoSlate
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "CryptoSlate"
- category = "news"
- endpoint = "/newslist"
-
- logger.info(f"Fetching news from {provider}")
-
- try:
- client = get_client()
-
- # CryptoSlate API endpoint (if available)
- url = "https://cryptoslate.com/wp-json/cs/v1/posts"
-
- params = {
- "per_page": 10,
- "orderby": "date"
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # Fallback to RSS feed
- logger.info(f"{provider} - API failed, trying RSS feed")
- return await get_rss_feed(provider, "https://cryptoslate.com/feed/")
-
- # Extract data
- data = response["data"]
-
- news_data = None
- if isinstance(data, list):
- articles = [
- {
- "title": article.get("title", {}).get("rendered", ""),
- "link": article.get("link", ""),
- "published": article.get("date", ""),
- "excerpt": article.get("excerpt", {}).get("rendered", "")[:200]
- }
- for article in data
- ]
-
- news_data = {
- "total_entries": len(articles),
- "articles": articles
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {len(data) if isinstance(data, list) else 0} articles")
-
- return {
- "provider": provider,
- "category": category,
- "data": news_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- # Fallback to RSS feed on error
- logger.info(f"{provider} - Exception occurred, trying RSS feed")
- return await get_rss_feed(provider, "https://cryptoslate.com/feed/")
-
-
-async def get_cryptonews_feed() -> Dict[str, Any]:
- """
- Fetch news from Crypto.news RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("CryptoNews", "https://crypto.news/feed/")
-
-
-async def get_coinjournal_news() -> Dict[str, Any]:
- """
- Fetch news from CoinJournal RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("CoinJournal", "https://coinjournal.net/feed/")
-
-
-async def get_beincrypto_news() -> Dict[str, Any]:
- """
- Fetch news from BeInCrypto RSS feed
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("BeInCrypto", "https://beincrypto.com/feed/")
-
-
-async def get_cryptobriefing_news() -> Dict[str, Any]:
- """
- Fetch news from CryptoBriefing
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- return await get_rss_feed("CryptoBriefing", "https://cryptobriefing.com/feed/")
-
-
-async def collect_extended_news() -> List[Dict[str, Any]]:
- """
- Main function to collect news from all extended sources
-
- Returns:
- List of results from all news collectors
- """
- logger.info("Starting extended news collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_coindesk_news(),
- get_cointelegraph_news(),
- get_decrypt_news(),
- get_bitcoinmagazine_news(),
- get_theblock_news(),
- get_cryptoslate_news(),
- get_cryptonews_feed(),
- get_coinjournal_news(),
- get_beincrypto_news(),
- get_cryptobriefing_news(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "news",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- total_articles = sum(
- r.get("data", {}).get("total_entries", 0)
- for r in processed_results
- if r.get("success", False) and r.get("data")
- )
-
- logger.info(
- f"Extended news collection complete: {successful}/{len(processed_results)} sources successful, "
- f"{total_articles} total articles"
- )
-
- return processed_results
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_extended_news()
-
- print("\n=== Extended News Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
-
- if result['success']:
- data = result.get('data', {})
- if data:
- print(f"Total Articles: {data.get('total_entries', 'N/A')}")
- articles = data.get('articles', [])
- if articles:
- print(f"Latest: {articles[0].get('title', 'N/A')[:60]}...")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/onchain.py b/app/collectors/onchain.py
deleted file mode 100644
index 6392fe36e257867a0374bc1c005ca36990ba4515..0000000000000000000000000000000000000000
--- a/app/collectors/onchain.py
+++ /dev/null
@@ -1,508 +0,0 @@
-"""
-On-Chain Analytics Collectors
-Placeholder implementations for The Graph and Blockchair data collection
-
-These collectors are designed to be extended with actual implementations
-when on-chain data sources are integrated.
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("onchain_collector")
-
-
-def calculate_staleness_minutes(data_timestamp: Optional[datetime]) -> Optional[float]:
- """
- Calculate staleness in minutes from data timestamp to now
-
- Args:
- data_timestamp: Timestamp of the data
-
- Returns:
- Staleness in minutes or None if timestamp not available
- """
- if not data_timestamp:
- return None
-
- now = datetime.now(timezone.utc)
- if data_timestamp.tzinfo is None:
- data_timestamp = data_timestamp.replace(tzinfo=timezone.utc)
-
- delta = now - data_timestamp
- return delta.total_seconds() / 60.0
-
-
-async def get_the_graph_data() -> Dict[str, Any]:
- """
- Fetch on-chain data from The Graph protocol - Uniswap V3 subgraph
-
- The Graph is a decentralized protocol for indexing and querying blockchain data.
- This implementation queries the Uniswap V3 subgraph for DEX metrics.
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "TheGraph"
- category = "onchain_analytics"
- endpoint = "/subgraphs/uniswap-v3"
-
- logger.info(f"Fetching on-chain data from {provider}")
-
- try:
- client = get_client()
-
- # Uniswap V3 subgraph endpoint
- url = "https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v3"
-
- # GraphQL query to get top pools and overall stats
- query = """
- {
- factories(first: 1) {
- totalVolumeUSD
- totalValueLockedUSD
- txCount
- }
- pools(first: 10, orderBy: totalValueLockedUSD, orderDirection: desc) {
- id
- token0 {
- symbol
- }
- token1 {
- symbol
- }
- totalValueLockedUSD
- volumeUSD
- txCount
- }
- }
- """
-
- payload = {"query": query}
- headers = {"Content-Type": "application/json"}
-
- # Make request
- response = await client.post(url, json=payload, headers=headers, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- raw_data = response["data"]
-
- graph_data = None
- if isinstance(raw_data, dict) and "data" in raw_data:
- data = raw_data["data"]
- factories = data.get("factories", [])
- pools = data.get("pools", [])
-
- if factories:
- factory = factories[0]
- graph_data = {
- "protocol": "Uniswap V3",
- "total_volume_usd": float(factory.get("totalVolumeUSD", 0)),
- "total_tvl_usd": float(factory.get("totalValueLockedUSD", 0)),
- "total_transactions": int(factory.get("txCount", 0)),
- "top_pools": [
- {
- "pair": f"{pool.get('token0', {}).get('symbol', '?')}/{pool.get('token1', {}).get('symbol', '?')}",
- "tvl_usd": float(pool.get("totalValueLockedUSD", 0)),
- "volume_usd": float(pool.get("volumeUSD", 0)),
- "tx_count": int(pool.get("txCount", 0))
- }
- for pool in pools
- ]
- }
-
- data_timestamp = datetime.now(timezone.utc)
- staleness = calculate_staleness_minutes(data_timestamp)
-
- logger.info(
- f"{provider} - {endpoint} - TVL: ${graph_data.get('total_tvl_usd', 0):,.0f}"
- if graph_data else f"{provider} - {endpoint} - No data"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": graph_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_blockchair_data() -> Dict[str, Any]:
- """
- Fetch blockchain statistics from Blockchair
-
- Blockchair is a blockchain explorer and analytics platform.
- This implementation fetches Bitcoin and Ethereum network statistics.
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "Blockchair"
- category = "onchain_analytics"
- endpoint = "/stats"
-
- logger.info(f"Fetching blockchain stats from {provider}")
-
- try:
- client = get_client()
-
- # Fetch stats for BTC and ETH
- btc_url = "https://api.blockchair.com/bitcoin/stats"
- eth_url = "https://api.blockchair.com/ethereum/stats"
-
- # Make concurrent requests
- btc_response, eth_response = await asyncio.gather(
- client.get(btc_url, timeout=10),
- client.get(eth_url, timeout=10),
- return_exceptions=True
- )
-
- # Log requests
- if not isinstance(btc_response, Exception):
- log_api_request(
- logger,
- provider,
- f"{endpoint}/bitcoin",
- btc_response.get("response_time_ms", 0),
- "success" if btc_response["success"] else "error",
- btc_response.get("status_code")
- )
-
- if not isinstance(eth_response, Exception):
- log_api_request(
- logger,
- provider,
- f"{endpoint}/ethereum",
- eth_response.get("response_time_ms", 0),
- "success" if eth_response["success"] else "error",
- eth_response.get("status_code")
- )
-
- # Process Bitcoin data
- btc_data = None
- if not isinstance(btc_response, Exception) and btc_response.get("success"):
- raw_btc = btc_response.get("data", {})
- if isinstance(raw_btc, dict) and "data" in raw_btc:
- btc_stats = raw_btc["data"]
- btc_data = {
- "blocks": btc_stats.get("blocks"),
- "transactions": btc_stats.get("transactions"),
- "market_price_usd": btc_stats.get("market_price_usd"),
- "hashrate_24h": btc_stats.get("hashrate_24h"),
- "difficulty": btc_stats.get("difficulty"),
- "mempool_size": btc_stats.get("mempool_size"),
- "mempool_transactions": btc_stats.get("mempool_transactions")
- }
-
- # Process Ethereum data
- eth_data = None
- if not isinstance(eth_response, Exception) and eth_response.get("success"):
- raw_eth = eth_response.get("data", {})
- if isinstance(raw_eth, dict) and "data" in raw_eth:
- eth_stats = raw_eth["data"]
- eth_data = {
- "blocks": eth_stats.get("blocks"),
- "transactions": eth_stats.get("transactions"),
- "market_price_usd": eth_stats.get("market_price_usd"),
- "hashrate_24h": eth_stats.get("hashrate_24h"),
- "difficulty": eth_stats.get("difficulty"),
- "mempool_size": eth_stats.get("mempool_tps")
- }
-
- blockchair_data = {
- "bitcoin": btc_data,
- "ethereum": eth_data
- }
-
- data_timestamp = datetime.now(timezone.utc)
- staleness = calculate_staleness_minutes(data_timestamp)
-
- logger.info(
- f"{provider} - {endpoint} - BTC blocks: {btc_data.get('blocks', 'N/A') if btc_data else 'N/A'}, "
- f"ETH blocks: {eth_data.get('blocks', 'N/A') if eth_data else 'N/A'}"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": blockchair_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": (btc_response.get("response_time_ms", 0) if not isinstance(btc_response, Exception) else 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_glassnode_metrics() -> Dict[str, Any]:
- """
- Fetch advanced on-chain metrics from Glassnode (placeholder)
-
- Glassnode provides advanced on-chain analytics and metrics.
- This is a placeholder implementation that should be extended with:
- - NUPL (Net Unrealized Profit/Loss)
- - SOPR (Spent Output Profit Ratio)
- - Exchange flows
- - Whale transactions
- - Active addresses
- - Realized cap
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "Glassnode"
- category = "onchain_analytics"
- endpoint = "/metrics"
-
- logger.info(f"Fetching on-chain metrics from {provider} (placeholder)")
-
- try:
- # Placeholder implementation
- # Glassnode API requires API key and has extensive metrics
- # Example metrics: NUPL, SOPR, Exchange Flows, Miner Revenue, etc.
-
- placeholder_data = {
- "status": "placeholder",
- "message": "Glassnode integration not yet implemented",
- "planned_metrics": [
- "NUPL - Net Unrealized Profit/Loss",
- "SOPR - Spent Output Profit Ratio",
- "Exchange Net Flows",
- "Whale Transaction Count",
- "Active Addresses",
- "Realized Cap",
- "MVRV Ratio",
- "Supply in Profit",
- "Long/Short Term Holder Supply"
- ],
- "note": "Requires Glassnode API key for access"
- }
-
- data_timestamp = datetime.now(timezone.utc)
- staleness = 0.0
-
- logger.info(f"{provider} - {endpoint} - Placeholder data returned")
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat(),
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_onchain_data() -> List[Dict[str, Any]]:
- """
- Main function to collect on-chain analytics data from all sources
-
- Currently returns placeholder implementations for:
- - The Graph (GraphQL-based blockchain data)
- - Blockchair (blockchain explorer and stats)
- - Glassnode (advanced on-chain metrics)
-
- Returns:
- List of results from all on-chain collectors
- """
- logger.info("Starting on-chain data collection from all sources (placeholder)")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_the_graph_data(),
- get_blockchair_data(),
- get_glassnode_metrics(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "onchain_analytics",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- placeholder_count = sum(1 for r in processed_results if r.get("is_placeholder", False))
-
- logger.info(
- f"On-chain data collection complete: {successful}/{len(processed_results)} successful "
- f"({placeholder_count} placeholders)"
- )
-
- return processed_results
-
-
-class OnChainCollector:
- """
- On-Chain Analytics Collector class for WebSocket streaming interface
- Wraps the standalone on-chain data collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the on-chain collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect on-chain analytics data from all sources
-
- Returns:
- Dict with aggregated on-chain data
- """
- results = await collect_onchain_data()
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "active_addresses": None,
- "transaction_count": None,
- "total_fees": None,
- "gas_price": None,
- "network_utilization": None,
- "contract_events": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
- data = result["data"]
-
- # Skip placeholders but still return basic structure
- if isinstance(data, dict) and data.get("status") == "placeholder":
- continue
-
- # Parse data from various providers (when implemented)
- # Currently all are placeholders, so this will be empty
- pass
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_onchain_data()
-
- print("\n=== On-Chain Data Collection Results ===")
- print("Note: These are placeholder implementations")
- print()
-
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Is Placeholder: {result.get('is_placeholder', False)}")
- if result['success']:
- data = result.get('data', {})
- if isinstance(data, dict):
- print(f"Status: {data.get('status', 'N/A')}")
- print(f"Message: {data.get('message', 'N/A')}")
- if 'planned_features' in data:
- print(f"Planned Features: {len(data['planned_features'])}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- print("\n" + "="*50)
- print("To implement these collectors:")
- print("1. The Graph: Add GraphQL queries for specific subgraphs")
- print("2. Blockchair: Add API key and implement endpoint calls")
- print("3. Glassnode: Add API key and implement metrics fetching")
- print("="*50)
-
- asyncio.run(main())
diff --git a/app/collectors/rpc_nodes.py b/app/collectors/rpc_nodes.py
deleted file mode 100644
index 60ce216a97257190d689515be6d00cd5a4c3f683..0000000000000000000000000000000000000000
--- a/app/collectors/rpc_nodes.py
+++ /dev/null
@@ -1,635 +0,0 @@
-"""
-RPC Node Collectors
-Fetches blockchain data from RPC endpoints (Infura, Alchemy, Ankr, etc.)
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("rpc_collector")
-
-
-async def get_eth_block_number(provider: str, rpc_url: str, api_key: Optional[str] = None) -> Dict[str, Any]:
- """
- Fetch latest Ethereum block number from RPC endpoint
-
- Args:
- provider: Provider name (e.g., "Infura", "Alchemy")
- rpc_url: RPC endpoint URL
- api_key: Optional API key to append to URL
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- category = "rpc_nodes"
- endpoint = "eth_blockNumber"
-
- logger.info(f"Fetching block number from {provider}")
-
- try:
- client = get_client()
-
- # Build URL with API key if provided
- url = f"{rpc_url}/{api_key}" if api_key else rpc_url
-
- # JSON-RPC request payload
- payload = {
- "jsonrpc": "2.0",
- "method": "eth_blockNumber",
- "params": [],
- "id": 1
- }
-
- headers = {"Content-Type": "application/json"}
-
- # Make request
- response = await client.post(url, json=payload, headers=headers, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse hex block number
- block_data = None
- if isinstance(data, dict) and "result" in data:
- hex_block = data["result"]
- block_number = int(hex_block, 16) if hex_block else 0
- block_data = {
- "block_number": block_number,
- "hex": hex_block,
- "chain": "ethereum"
- }
-
- logger.info(f"{provider} - {endpoint} - Block: {block_data.get('block_number', 'N/A')}")
-
- return {
- "provider": provider,
- "category": category,
- "data": block_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_eth_gas_price(provider: str, rpc_url: str, api_key: Optional[str] = None) -> Dict[str, Any]:
- """
- Fetch current gas price from RPC endpoint
-
- Args:
- provider: Provider name
- rpc_url: RPC endpoint URL
- api_key: Optional API key
-
- Returns:
- Dict with gas price data
- """
- category = "rpc_nodes"
- endpoint = "eth_gasPrice"
-
- logger.info(f"Fetching gas price from {provider}")
-
- try:
- client = get_client()
- url = f"{rpc_url}/{api_key}" if api_key else rpc_url
-
- payload = {
- "jsonrpc": "2.0",
- "method": "eth_gasPrice",
- "params": [],
- "id": 1
- }
-
- headers = {"Content-Type": "application/json"}
- response = await client.post(url, json=payload, headers=headers, timeout=10)
-
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- data = response["data"]
- gas_data = None
-
- if isinstance(data, dict) and "result" in data:
- hex_gas = data["result"]
- gas_wei = int(hex_gas, 16) if hex_gas else 0
- gas_gwei = gas_wei / 1e9
-
- gas_data = {
- "gas_price_wei": gas_wei,
- "gas_price_gwei": round(gas_gwei, 2),
- "hex": hex_gas,
- "chain": "ethereum"
- }
-
- logger.info(f"{provider} - {endpoint} - Gas: {gas_data.get('gas_price_gwei', 'N/A')} Gwei")
-
- return {
- "provider": provider,
- "category": category,
- "data": gas_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_eth_chain_id(provider: str, rpc_url: str, api_key: Optional[str] = None) -> Dict[str, Any]:
- """
- Fetch chain ID from RPC endpoint
-
- Args:
- provider: Provider name
- rpc_url: RPC endpoint URL
- api_key: Optional API key
-
- Returns:
- Dict with chain ID data
- """
- category = "rpc_nodes"
- endpoint = "eth_chainId"
-
- try:
- client = get_client()
- url = f"{rpc_url}/{api_key}" if api_key else rpc_url
-
- payload = {
- "jsonrpc": "2.0",
- "method": "eth_chainId",
- "params": [],
- "id": 1
- }
-
- headers = {"Content-Type": "application/json"}
- response = await client.post(url, json=payload, headers=headers, timeout=10)
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg
- }
-
- data = response["data"]
- chain_data = None
-
- if isinstance(data, dict) and "result" in data:
- hex_chain = data["result"]
- chain_id = int(hex_chain, 16) if hex_chain else 0
-
- # Map chain IDs to names
- chain_names = {
- 1: "Ethereum Mainnet",
- 3: "Ropsten",
- 4: "Rinkeby",
- 5: "Goerli",
- 11155111: "Sepolia",
- 56: "BSC Mainnet",
- 97: "BSC Testnet",
- 137: "Polygon Mainnet",
- 80001: "Mumbai Testnet"
- }
-
- chain_data = {
- "chain_id": chain_id,
- "chain_name": chain_names.get(chain_id, f"Unknown (ID: {chain_id})"),
- "hex": hex_chain
- }
-
- return {
- "provider": provider,
- "category": category,
- "data": chain_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(e),
- "error_type": "exception"
- }
-
-
-async def collect_infura_data(api_key: Optional[str] = None) -> List[Dict[str, Any]]:
- """
- Collect data from Infura RPC endpoints
-
- Args:
- api_key: Infura project ID
-
- Returns:
- List of results from Infura endpoints
- """
- provider = "Infura"
- rpc_url = "https://mainnet.infura.io/v3"
-
- if not api_key:
- logger.warning(f"{provider} - No API key provided, skipping")
- return [{
- "provider": provider,
- "category": "rpc_nodes",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": "API key required",
- "error_type": "missing_api_key"
- }]
-
- logger.info(f"Starting {provider} data collection")
-
- results = await asyncio.gather(
- get_eth_block_number(provider, rpc_url, api_key),
- get_eth_gas_price(provider, rpc_url, api_key),
- get_eth_chain_id(provider, rpc_url, api_key),
- return_exceptions=True
- )
-
- processed = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"{provider} - Collector failed: {str(result)}")
- processed.append({
- "provider": provider,
- "category": "rpc_nodes",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed.append(result)
-
- successful = sum(1 for r in processed if r.get("success", False))
- logger.info(f"{provider} - Collection complete: {successful}/{len(processed)} successful")
-
- return processed
-
-
-async def collect_alchemy_data(api_key: Optional[str] = None) -> List[Dict[str, Any]]:
- """
- Collect data from Alchemy RPC endpoints
-
- Args:
- api_key: Alchemy API key
-
- Returns:
- List of results from Alchemy endpoints
- """
- provider = "Alchemy"
- rpc_url = "https://eth-mainnet.g.alchemy.com/v2"
-
- if not api_key:
- logger.warning(f"{provider} - No API key provided, using free tier")
- # Alchemy has a public demo endpoint
- api_key = "demo"
-
- logger.info(f"Starting {provider} data collection")
-
- results = await asyncio.gather(
- get_eth_block_number(provider, rpc_url, api_key),
- get_eth_gas_price(provider, rpc_url, api_key),
- get_eth_chain_id(provider, rpc_url, api_key),
- return_exceptions=True
- )
-
- processed = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"{provider} - Collector failed: {str(result)}")
- processed.append({
- "provider": provider,
- "category": "rpc_nodes",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed.append(result)
-
- successful = sum(1 for r in processed if r.get("success", False))
- logger.info(f"{provider} - Collection complete: {successful}/{len(processed)} successful")
-
- return processed
-
-
-async def collect_ankr_data() -> List[Dict[str, Any]]:
- """
- Collect data from Ankr public RPC endpoints (no key required)
-
- Returns:
- List of results from Ankr endpoints
- """
- provider = "Ankr"
- rpc_url = "https://rpc.ankr.com/eth"
-
- logger.info(f"Starting {provider} data collection")
-
- results = await asyncio.gather(
- get_eth_block_number(provider, rpc_url),
- get_eth_gas_price(provider, rpc_url),
- get_eth_chain_id(provider, rpc_url),
- return_exceptions=True
- )
-
- processed = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"{provider} - Collector failed: {str(result)}")
- processed.append({
- "provider": provider,
- "category": "rpc_nodes",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed.append(result)
-
- successful = sum(1 for r in processed if r.get("success", False))
- logger.info(f"{provider} - Collection complete: {successful}/{len(processed)} successful")
-
- return processed
-
-
-async def collect_public_rpc_data() -> List[Dict[str, Any]]:
- """
- Collect data from free public RPC endpoints
-
- Returns:
- List of results from public endpoints
- """
- logger.info("Starting public RPC data collection")
-
- public_rpcs = [
- ("Cloudflare", "https://cloudflare-eth.com"),
- ("PublicNode", "https://ethereum.publicnode.com"),
- ("LlamaNodes", "https://eth.llamarpc.com"),
- ]
-
- all_results = []
-
- for provider, rpc_url in public_rpcs:
- results = await asyncio.gather(
- get_eth_block_number(provider, rpc_url),
- get_eth_gas_price(provider, rpc_url),
- return_exceptions=True
- )
-
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"{provider} - Collector failed: {str(result)}")
- all_results.append({
- "provider": provider,
- "category": "rpc_nodes",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- all_results.append(result)
-
- successful = sum(1 for r in all_results if r.get("success", False))
- logger.info(f"Public RPC collection complete: {successful}/{len(all_results)} successful")
-
- return all_results
-
-
-async def collect_rpc_data(
- infura_key: Optional[str] = None,
- alchemy_key: Optional[str] = None
-) -> List[Dict[str, Any]]:
- """
- Main function to collect RPC data from all sources
-
- Args:
- infura_key: Infura project ID
- alchemy_key: Alchemy API key
-
- Returns:
- List of results from all RPC collectors
- """
- logger.info("Starting RPC data collection from all sources")
-
- # Collect from all providers
- all_results = []
-
- # Infura (requires key)
- if infura_key:
- infura_results = await collect_infura_data(infura_key)
- all_results.extend(infura_results)
-
- # Alchemy (has free tier)
- alchemy_results = await collect_alchemy_data(alchemy_key)
- all_results.extend(alchemy_results)
-
- # Ankr (free, no key needed)
- ankr_results = await collect_ankr_data()
- all_results.extend(ankr_results)
-
- # Public RPCs (free)
- public_results = await collect_public_rpc_data()
- all_results.extend(public_results)
-
- # Log summary
- successful = sum(1 for r in all_results if r.get("success", False))
- logger.info(f"RPC data collection complete: {successful}/{len(all_results)} successful")
-
- return all_results
-
-
-class RPCNodeCollector:
- """
- RPC Node Collector class for WebSocket streaming interface
- Wraps the standalone RPC node collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the RPC node collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect RPC node data from all sources
-
- Returns:
- Dict with aggregated RPC node data
- """
- import os
- infura_key = os.getenv("INFURA_API_KEY")
- alchemy_key = os.getenv("ALCHEMY_API_KEY")
- results = await collect_rpc_data(infura_key, alchemy_key)
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "nodes": [],
- "active_nodes": 0,
- "total_nodes": 0,
- "average_latency": 0,
- "events": [],
- "block_number": None,
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- total_latency = 0
- latency_count = 0
-
- for result in results:
- aggregated["total_nodes"] += 1
-
- if result.get("success"):
- aggregated["active_nodes"] += 1
- provider = result.get("provider", "unknown")
- response_time = result.get("response_time_ms", 0)
- data = result.get("data", {})
-
- # Track latency
- if response_time:
- total_latency += response_time
- latency_count += 1
-
- # Add node info
- node_info = {
- "provider": provider,
- "response_time_ms": response_time,
- "status": "active",
- "data": data
- }
-
- # Extract block number
- if "result" in data and isinstance(data["result"], str):
- try:
- block_number = int(data["result"], 16)
- node_info["block_number"] = block_number
- if aggregated["block_number"] is None or block_number > aggregated["block_number"]:
- aggregated["block_number"] = block_number
- except:
- pass
-
- aggregated["nodes"].append(node_info)
-
- # Calculate average latency
- if latency_count > 0:
- aggregated["average_latency"] = total_latency / latency_count
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- import os
-
- infura_key = os.getenv("INFURA_API_KEY")
- alchemy_key = os.getenv("ALCHEMY_API_KEY")
-
- results = await collect_rpc_data(infura_key, alchemy_key)
-
- print("\n=== RPC Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- data = result.get('data', {})
- if data:
- print(f"Data: {data}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/scheduler_comprehensive.py b/app/collectors/scheduler_comprehensive.py
deleted file mode 100644
index f3450d8fc763f9b4dd21a78587794ed51bc0f5f8..0000000000000000000000000000000000000000
--- a/app/collectors/scheduler_comprehensive.py
+++ /dev/null
@@ -1,367 +0,0 @@
-"""
-Comprehensive Scheduler for All Data Sources
-Schedules and runs data collection from all available sources with configurable intervals
-"""
-
-import asyncio
-import json
-from datetime import datetime, timezone, timedelta
-from typing import Dict, List, Optional, Any
-from pathlib import Path
-from utils.logger import setup_logger
-from collectors.master_collector import DataSourceCollector
-
-logger = setup_logger("comprehensive_scheduler")
-
-
-class ComprehensiveScheduler:
- """
- Comprehensive scheduler that manages data collection from all sources
- """
-
- def __init__(self, config_file: Optional[str] = None):
- """
- Initialize the comprehensive scheduler
-
- Args:
- config_file: Path to scheduler configuration file
- """
- self.collector = DataSourceCollector()
- self.config_file = config_file or "scheduler_config.json"
- self.config = self._load_config()
- self.last_run_times: Dict[str, datetime] = {}
- self.running = False
- logger.info("Comprehensive Scheduler initialized")
-
- def _load_config(self) -> Dict[str, Any]:
- """
- Load scheduler configuration
-
- Returns:
- Configuration dict
- """
- default_config = {
- "schedules": {
- "market_data": {
- "interval_seconds": 60, # Every 1 minute
- "enabled": True
- },
- "blockchain": {
- "interval_seconds": 300, # Every 5 minutes
- "enabled": True
- },
- "news": {
- "interval_seconds": 600, # Every 10 minutes
- "enabled": True
- },
- "sentiment": {
- "interval_seconds": 1800, # Every 30 minutes
- "enabled": True
- },
- "whale_tracking": {
- "interval_seconds": 300, # Every 5 minutes
- "enabled": True
- },
- "full_collection": {
- "interval_seconds": 3600, # Every 1 hour
- "enabled": True
- }
- },
- "max_retries": 3,
- "retry_delay_seconds": 5,
- "persist_results": True,
- "results_directory": "data/collections"
- }
-
- config_path = Path(self.config_file)
- if config_path.exists():
- try:
- with open(config_path, 'r') as f:
- loaded_config = json.load(f)
- # Merge with defaults
- default_config.update(loaded_config)
- logger.info(f"Loaded scheduler config from {config_path}")
- except Exception as e:
- logger.error(f"Error loading config file: {e}, using defaults")
-
- return default_config
-
- def save_config(self):
- """Save current configuration to file"""
- try:
- config_path = Path(self.config_file)
- config_path.parent.mkdir(parents=True, exist_ok=True)
-
- with open(config_path, 'w') as f:
- json.dump(self.config, f, indent=2)
-
- logger.info(f"Saved scheduler config to {config_path}")
- except Exception as e:
- logger.error(f"Error saving config: {e}")
-
- async def _save_results(self, category: str, results: Any):
- """
- Save collection results to file
-
- Args:
- category: Category name
- results: Results to save
- """
- if not self.config.get("persist_results", True):
- return
-
- try:
- results_dir = Path(self.config.get("results_directory", "data/collections"))
- results_dir.mkdir(parents=True, exist_ok=True)
-
- timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
- filename = results_dir / f"{category}_{timestamp}.json"
-
- with open(filename, 'w') as f:
- json.dump(results, f, indent=2, default=str)
-
- logger.info(f"Saved {category} results to {filename}")
- except Exception as e:
- logger.error(f"Error saving results: {e}")
-
- def should_run(self, category: str) -> bool:
- """
- Check if a category should run based on its schedule
-
- Args:
- category: Category name
-
- Returns:
- True if should run, False otherwise
- """
- schedule = self.config.get("schedules", {}).get(category, {})
-
- if not schedule.get("enabled", True):
- return False
-
- interval = schedule.get("interval_seconds", 3600)
- last_run = self.last_run_times.get(category)
-
- if not last_run:
- return True
-
- elapsed = (datetime.now(timezone.utc) - last_run).total_seconds()
- return elapsed >= interval
-
- async def run_category_with_retry(self, category: str) -> Optional[Any]:
- """
- Run a category collection with retry logic
-
- Args:
- category: Category name
-
- Returns:
- Collection results or None if failed
- """
- max_retries = self.config.get("max_retries", 3)
- retry_delay = self.config.get("retry_delay_seconds", 5)
-
- for attempt in range(max_retries):
- try:
- logger.info(f"Running {category} collection (attempt {attempt + 1}/{max_retries})")
-
- if category == "full_collection":
- results = await self.collector.collect_all_data()
- else:
- results = await self.collector.collect_category(category)
-
- self.last_run_times[category] = datetime.now(timezone.utc)
-
- # Save results
- await self._save_results(category, results)
-
- return results
-
- except Exception as e:
- logger.error(f"Error in {category} collection (attempt {attempt + 1}): {e}")
-
- if attempt < max_retries - 1:
- logger.info(f"Retrying in {retry_delay} seconds...")
- await asyncio.sleep(retry_delay)
- else:
- logger.error(f"Failed {category} collection after {max_retries} attempts")
- return None
-
- async def run_cycle(self):
- """Run one scheduler cycle - check and run due categories"""
- logger.info("Running scheduler cycle...")
-
- categories = self.config.get("schedules", {}).keys()
- tasks = []
-
- for category in categories:
- if self.should_run(category):
- logger.info(f"Scheduling {category} collection")
- task = self.run_category_with_retry(category)
- tasks.append((category, task))
-
- if tasks:
- # Run all due collections in parallel
- results = await asyncio.gather(*[task for _, task in tasks], return_exceptions=True)
-
- for (category, _), result in zip(tasks, results):
- if isinstance(result, Exception):
- logger.error(f"{category} collection failed: {str(result)}")
- else:
- if result:
- stats = result.get("statistics", {}) if isinstance(result, dict) else None
- if stats:
- logger.info(
- f"{category} collection complete: "
- f"{stats.get('successful_sources', 'N/A')}/{stats.get('total_sources', 'N/A')} successful"
- )
- else:
- logger.info("No collections due in this cycle")
-
- async def run_forever(self, cycle_interval: int = 30):
- """
- Run the scheduler forever with specified cycle interval
-
- Args:
- cycle_interval: Seconds between scheduler cycles
- """
- self.running = True
- logger.info(f"Starting comprehensive scheduler (cycle interval: {cycle_interval}s)")
-
- try:
- while self.running:
- await self.run_cycle()
-
- # Wait for next cycle
- logger.info(f"Waiting {cycle_interval} seconds until next cycle...")
- await asyncio.sleep(cycle_interval)
-
- except KeyboardInterrupt:
- logger.info("Scheduler interrupted by user")
- except Exception as e:
- logger.error(f"Scheduler error: {e}")
- finally:
- self.running = False
- logger.info("Scheduler stopped")
-
- def stop(self):
- """Stop the scheduler"""
- logger.info("Stopping scheduler...")
- self.running = False
-
- async def run_once(self, category: Optional[str] = None):
- """
- Run a single collection immediately
-
- Args:
- category: Category to run, or None for full collection
- """
- if category:
- logger.info(f"Running single {category} collection...")
- results = await self.run_category_with_retry(category)
- else:
- logger.info("Running single full collection...")
- results = await self.run_category_with_retry("full_collection")
-
- return results
-
- def get_status(self) -> Dict[str, Any]:
- """
- Get scheduler status
-
- Returns:
- Dict with scheduler status information
- """
- now = datetime.now(timezone.utc)
- status = {
- "running": self.running,
- "current_time": now.isoformat(),
- "schedules": {}
- }
-
- for category, schedule in self.config.get("schedules", {}).items():
- last_run = self.last_run_times.get(category)
- interval = schedule.get("interval_seconds", 0)
-
- next_run = None
- if last_run:
- next_run = last_run + timedelta(seconds=interval)
-
- time_until_next = None
- if next_run:
- time_until_next = (next_run - now).total_seconds()
-
- status["schedules"][category] = {
- "enabled": schedule.get("enabled", True),
- "interval_seconds": interval,
- "last_run": last_run.isoformat() if last_run else None,
- "next_run": next_run.isoformat() if next_run else None,
- "seconds_until_next": round(time_until_next, 2) if time_until_next else None,
- "should_run_now": self.should_run(category)
- }
-
- return status
-
- def update_schedule(self, category: str, interval_seconds: Optional[int] = None, enabled: Optional[bool] = None):
- """
- Update schedule for a category
-
- Args:
- category: Category name
- interval_seconds: New interval in seconds
- enabled: Enable/disable the schedule
- """
- if category not in self.config.get("schedules", {}):
- logger.error(f"Unknown category: {category}")
- return
-
- if interval_seconds is not None:
- self.config["schedules"][category]["interval_seconds"] = interval_seconds
- logger.info(f"Updated {category} interval to {interval_seconds}s")
-
- if enabled is not None:
- self.config["schedules"][category]["enabled"] = enabled
- logger.info(f"{'Enabled' if enabled else 'Disabled'} {category} schedule")
-
- self.save_config()
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- scheduler = ComprehensiveScheduler()
-
- # Show status
- print("\n" + "=" * 80)
- print("COMPREHENSIVE SCHEDULER STATUS")
- print("=" * 80)
-
- status = scheduler.get_status()
- print(f"Running: {status['running']}")
- print(f"Current Time: {status['current_time']}")
- print("\nSchedules:")
- print("-" * 80)
-
- for category, sched in status['schedules'].items():
- enabled = "✓" if sched['enabled'] else "✗"
- interval = sched['interval_seconds']
- next_run = sched.get('seconds_until_next', 'N/A')
-
- print(f"{enabled} {category:20} | Interval: {interval:6}s | Next in: {next_run}")
-
- print("=" * 80)
-
- # Run once as example
- print("\nRunning market_data collection once as example...")
- results = await scheduler.run_once("market_data")
-
- if results:
- print(f"\nCollected {len(results)} market data sources")
- successful = sum(1 for r in results if r.get('success', False))
- print(f"Successful: {successful}/{len(results)}")
-
- print("\n" + "=" * 80)
- print("To run scheduler forever, use: scheduler.run_forever()")
- print("=" * 80)
-
- asyncio.run(main())
diff --git a/app/collectors/sentiment.py b/app/collectors/sentiment.py
deleted file mode 100644
index dc3f924ce391a464c39e6805b8886c98c71c2709..0000000000000000000000000000000000000000
--- a/app/collectors/sentiment.py
+++ /dev/null
@@ -1,290 +0,0 @@
-"""
-Sentiment Data Collectors
-Fetches cryptocurrency sentiment data from Alternative.me Fear & Greed Index
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-from config import config
-
-logger = setup_logger("sentiment_collector")
-
-
-def calculate_staleness_minutes(data_timestamp: Optional[datetime]) -> Optional[float]:
- """
- Calculate staleness in minutes from data timestamp to now
-
- Args:
- data_timestamp: Timestamp of the data
-
- Returns:
- Staleness in minutes or None if timestamp not available
- """
- if not data_timestamp:
- return None
-
- now = datetime.now(timezone.utc)
- if data_timestamp.tzinfo is None:
- data_timestamp = data_timestamp.replace(tzinfo=timezone.utc)
-
- delta = now - data_timestamp
- return delta.total_seconds() / 60.0
-
-
-async def get_fear_greed_index() -> Dict[str, Any]:
- """
- Fetch current Fear & Greed Index from Alternative.me
-
- The Fear & Greed Index is a sentiment indicator for the cryptocurrency market.
- - 0-24: Extreme Fear
- - 25-49: Fear
- - 50-74: Greed
- - 75-100: Extreme Greed
-
- Returns:
- Dict with provider, category, data, timestamp, staleness, success, error
- """
- provider = "AlternativeMe"
- category = "sentiment"
- endpoint = "/fng/"
-
- logger.info(f"Fetching Fear & Greed Index from {provider}")
-
- try:
- client = get_client()
- provider_config = config.get_provider(provider)
-
- if not provider_config:
- error_msg = f"Provider {provider} not configured"
- log_error(logger, provider, "config_error", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg
- }
-
- # Build request URL
- url = f"{provider_config.endpoint_url}{endpoint}"
- params = {
- "limit": "1", # Get only the latest index
- "format": "json"
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=provider_config.timeout_ms // 1000)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Parse timestamp from response
- data_timestamp = None
- if isinstance(data, dict) and "data" in data:
- data_list = data["data"]
- if isinstance(data_list, list) and len(data_list) > 0:
- index_data = data_list[0]
- if isinstance(index_data, dict) and "timestamp" in index_data:
- try:
- # Alternative.me returns Unix timestamp
- data_timestamp = datetime.fromtimestamp(
- int(index_data["timestamp"]),
- tz=timezone.utc
- )
- except:
- pass
-
- staleness = calculate_staleness_minutes(data_timestamp)
-
- # Extract index value and classification
- index_value = None
- index_classification = None
- if isinstance(data, dict) and "data" in data:
- data_list = data["data"]
- if isinstance(data_list, list) and len(data_list) > 0:
- index_data = data_list[0]
- if isinstance(index_data, dict):
- index_value = index_data.get("value")
- index_classification = index_data.get("value_classification")
-
- logger.info(
- f"{provider} - {endpoint} - Fear & Greed Index: {index_value} ({index_classification}), "
- f"staleness: {staleness:.2f}m" if staleness else "staleness: N/A"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "data_timestamp": data_timestamp.isoformat() if data_timestamp else None,
- "staleness_minutes": staleness,
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0),
- "index_value": index_value,
- "index_classification": index_classification
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def collect_sentiment_data() -> List[Dict[str, Any]]:
- """
- Main function to collect sentiment data from all sources
-
- Currently collects from:
- - Alternative.me Fear & Greed Index
-
- Returns:
- List of results from all sentiment collectors
- """
- logger.info("Starting sentiment data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_fear_greed_index(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "sentiment",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "staleness_minutes": None,
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- logger.info(f"Sentiment data collection complete: {successful}/{len(processed_results)} successful")
-
- return processed_results
-
-
-# Alias for backward compatibility
-collect_sentiment = collect_sentiment_data
-
-
-class SentimentCollector:
- """
- Sentiment Collector class for WebSocket streaming interface
- Wraps the standalone sentiment collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the sentiment collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect sentiment data from all sources
-
- Returns:
- Dict with aggregated sentiment data
- """
- results = await collect_sentiment_data()
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "overall_sentiment": None,
- "sentiment_score": None,
- "social_volume": None,
- "trending_topics": [],
- "by_source": {},
- "social_trends": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
-
- # Parse Fear & Greed Index
- if provider == "Alternative.me" and "data" in result["data"]:
- index_data = result["data"]["data"][0] if result["data"]["data"] else {}
- aggregated["sentiment_score"] = int(index_data.get("value", 0))
- aggregated["overall_sentiment"] = index_data.get("value_classification", "neutral")
- aggregated["by_source"][provider] = {
- "value": aggregated["sentiment_score"],
- "classification": aggregated["overall_sentiment"]
- }
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_sentiment_data()
-
- print("\n=== Sentiment Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Staleness: {result.get('staleness_minutes', 'N/A')} minutes")
- if result['success']:
- print(f"Response Time: {result.get('response_time_ms', 0):.2f}ms")
- if result.get('index_value'):
- print(f"Fear & Greed Index: {result['index_value']} ({result['index_classification']})")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/sentiment_extended.py b/app/collectors/sentiment_extended.py
deleted file mode 100644
index 694218014145855fcfdafe3c02fd462ca1beb884..0000000000000000000000000000000000000000
--- a/app/collectors/sentiment_extended.py
+++ /dev/null
@@ -1,508 +0,0 @@
-"""
-Extended Sentiment Collectors
-Fetches sentiment data from LunarCrush, Santiment, and other sentiment APIs
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("sentiment_extended_collector")
-
-
-async def get_lunarcrush_global() -> Dict[str, Any]:
- """
- Fetch global market sentiment from LunarCrush
-
- Note: LunarCrush API v3 requires API key
- Free tier available with limited requests
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "LunarCrush"
- category = "sentiment"
- endpoint = "/public/metrics/global"
-
- logger.info(f"Fetching global sentiment from {provider}")
-
- try:
- client = get_client()
-
- # LunarCrush public metrics (limited free access)
- url = "https://lunarcrush.com/api3/public/metrics/global"
-
- # Make request
- response = await client.get(url, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # LunarCrush may require API key, return placeholder
- logger.warning(f"{provider} - API requires authentication, returning placeholder")
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": "LunarCrush API requires authentication",
- "planned_features": [
- "Social media sentiment tracking",
- "Galaxy Score (social activity metric)",
- "AltRank (relative social dominance)",
- "Influencer tracking",
- "Social volume and engagement metrics"
- ]
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- # Extract data
- data = response["data"]
-
- sentiment_data = None
- if isinstance(data, dict):
- sentiment_data = {
- "social_volume": data.get("social_volume"),
- "social_score": data.get("social_score"),
- "market_sentiment": data.get("sentiment"),
- "timestamp": data.get("timestamp")
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved sentiment data")
-
- return {
- "provider": provider,
- "category": category,
- "data": sentiment_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": f"LunarCrush integration error: {str(e)}"
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
-
-async def get_santiment_metrics() -> Dict[str, Any]:
- """
- Fetch sentiment metrics from Santiment
-
- Note: Santiment API requires authentication
- Provides on-chain, social, and development activity metrics
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "Santiment"
- category = "sentiment"
- endpoint = "/graphql"
-
- logger.info(f"Fetching sentiment metrics from {provider} (placeholder)")
-
- try:
- # Santiment uses GraphQL API and requires authentication
- # Placeholder implementation
-
- placeholder_data = {
- "status": "placeholder",
- "message": "Santiment API requires authentication and GraphQL queries",
- "planned_metrics": [
- "Social volume and trends",
- "Development activity",
- "Network growth",
- "Exchange flow",
- "MVRV ratio",
- "Daily active addresses",
- "Token age consumed",
- "Crowd sentiment"
- ],
- "note": "Requires Santiment API key and SAN tokens for full access"
- }
-
- logger.info(f"{provider} - {endpoint} - Placeholder data returned")
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_cryptoquant_sentiment() -> Dict[str, Any]:
- """
- Fetch on-chain sentiment from CryptoQuant
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "CryptoQuant"
- category = "sentiment"
- endpoint = "/sentiment"
-
- logger.info(f"Fetching sentiment from {provider} (placeholder)")
-
- try:
- # CryptoQuant API requires authentication
- # Placeholder implementation
-
- placeholder_data = {
- "status": "placeholder",
- "message": "CryptoQuant API requires authentication",
- "planned_metrics": [
- "Exchange reserves",
- "Miner flows",
- "Whale transactions",
- "Stablecoin supply ratio",
- "Funding rates",
- "Open interest"
- ]
- }
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(e),
- "error_type": "exception"
- }
-
-
-async def get_augmento_signals() -> Dict[str, Any]:
- """
- Fetch market sentiment signals from Augmento.ai
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "Augmento"
- category = "sentiment"
- endpoint = "/signals"
-
- logger.info(f"Fetching sentiment signals from {provider} (placeholder)")
-
- try:
- # Augmento provides AI-powered crypto sentiment signals
- # Requires API key
-
- placeholder_data = {
- "status": "placeholder",
- "message": "Augmento API requires authentication",
- "planned_features": [
- "AI-powered sentiment signals",
- "Topic extraction from social media",
- "Emerging trend detection",
- "Sentiment momentum indicators"
- ]
- }
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(e),
- "error_type": "exception"
- }
-
-
-async def get_thetie_sentiment() -> Dict[str, Any]:
- """
- Fetch sentiment data from TheTie.io
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "TheTie"
- category = "sentiment"
- endpoint = "/sentiment"
-
- logger.info(f"Fetching sentiment from {provider} (placeholder)")
-
- try:
- # TheTie provides institutional-grade crypto market intelligence
- # Requires API key
-
- placeholder_data = {
- "status": "placeholder",
- "message": "TheTie API requires authentication",
- "planned_metrics": [
- "Twitter sentiment scores",
- "Social media momentum",
- "Influencer tracking",
- "Sentiment trends over time"
- ]
- }
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(e),
- "error_type": "exception"
- }
-
-
-async def get_coinmarketcal_events() -> Dict[str, Any]:
- """
- Fetch upcoming crypto events from CoinMarketCal (free API)
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "CoinMarketCal"
- category = "sentiment"
- endpoint = "/events"
-
- logger.info(f"Fetching events from {provider}")
-
- try:
- client = get_client()
-
- # CoinMarketCal API
- url = "https://developers.coinmarketcal.com/v1/events"
-
- params = {
- "page": 1,
- "max": 20,
- "showOnly": "hot_events" # Only hot/important events
- }
-
- # Make request (may require API key for full access)
- response = await client.get(url, params=params, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # If API requires key, return placeholder
- logger.warning(f"{provider} - API may require authentication, returning placeholder")
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": "CoinMarketCal API may require authentication",
- "planned_features": [
- "Upcoming crypto events calendar",
- "Project updates and announcements",
- "Conferences and meetups",
- "Hard forks and mainnet launches"
- ]
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- # Extract data
- data = response["data"]
-
- events_data = None
- if isinstance(data, dict) and "body" in data:
- events = data["body"]
-
- events_data = {
- "total_events": len(events) if isinstance(events, list) else 0,
- "upcoming_events": [
- {
- "title": event.get("title", {}).get("en"),
- "coins": [coin.get("symbol") for coin in event.get("coins", [])],
- "date": event.get("date_event"),
- "proof": event.get("proof"),
- "source": event.get("source")
- }
- for event in (events[:10] if isinstance(events, list) else [])
- ]
- }
-
- logger.info(f"{provider} - {endpoint} - Retrieved {events_data.get('total_events', 0)} events")
-
- return {
- "provider": provider,
- "category": category,
- "data": events_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": f"CoinMarketCal integration error: {str(e)}"
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
-
-async def collect_extended_sentiment_data() -> List[Dict[str, Any]]:
- """
- Main function to collect extended sentiment data from all sources
-
- Returns:
- List of results from all sentiment collectors
- """
- logger.info("Starting extended sentiment data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_lunarcrush_global(),
- get_santiment_metrics(),
- get_cryptoquant_sentiment(),
- get_augmento_signals(),
- get_thetie_sentiment(),
- get_coinmarketcal_events(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "sentiment",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- placeholder_count = sum(1 for r in processed_results if r.get("is_placeholder", False))
-
- logger.info(
- f"Extended sentiment collection complete: {successful}/{len(processed_results)} successful "
- f"({placeholder_count} placeholders)"
- )
-
- return processed_results
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- results = await collect_extended_sentiment_data()
-
- print("\n=== Extended Sentiment Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Is Placeholder: {result.get('is_placeholder', False)}")
-
- if result['success']:
- data = result.get('data', {})
- if isinstance(data, dict):
- if data.get('status') == 'placeholder':
- print(f"Status: {data.get('message', 'N/A')}")
- else:
- print(f"Data keys: {list(data.keys())}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/collectors/whale_tracking.py b/app/collectors/whale_tracking.py
deleted file mode 100644
index bfb4f3f4df98ec63f976ffd0d34d7aa6e3ca5a65..0000000000000000000000000000000000000000
--- a/app/collectors/whale_tracking.py
+++ /dev/null
@@ -1,564 +0,0 @@
-"""
-Whale Tracking Collectors
-Fetches whale transaction data from WhaleAlert, Arkham Intelligence, and other sources
-"""
-
-import asyncio
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Any
-from utils.api_client import get_client
-from utils.logger import setup_logger, log_api_request, log_error
-
-logger = setup_logger("whale_tracking_collector")
-
-
-async def get_whalealert_transactions(api_key: Optional[str] = None) -> Dict[str, Any]:
- """
- Fetch recent large crypto transactions from WhaleAlert
-
- Args:
- api_key: WhaleAlert API key
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "WhaleAlert"
- category = "whale_tracking"
- endpoint = "/transactions"
-
- logger.info(f"Fetching whale transactions from {provider}")
-
- try:
- if not api_key:
- error_msg = f"API key required for {provider}"
- log_error(logger, provider, "missing_api_key", error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "missing_api_key"
- }
-
- client = get_client()
-
- # WhaleAlert API endpoint
- url = "https://api.whale-alert.io/v1/transactions"
-
- # Get transactions from last hour
- now = int(datetime.now(timezone.utc).timestamp())
- start_time = now - 3600 # 1 hour ago
-
- params = {
- "api_key": api_key,
- "start": start_time,
- "limit": 100 # Max 100 transactions
- }
-
- # Make request
- response = await client.get(url, params=params, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- error_msg = response.get("error_message", "Unknown error")
- log_error(logger, provider, response.get("error_type", "unknown"), error_msg, endpoint)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": response.get("error_type")
- }
-
- # Extract data
- data = response["data"]
-
- # Process transactions
- whale_data = None
- if isinstance(data, dict) and "transactions" in data:
- transactions = data["transactions"]
-
- # Aggregate statistics
- total_value_usd = sum(tx.get("amount_usd", 0) for tx in transactions)
- symbols = set(tx.get("symbol", "unknown") for tx in transactions)
-
- whale_data = {
- "transaction_count": len(transactions),
- "total_value_usd": round(total_value_usd, 2),
- "unique_symbols": list(symbols),
- "time_range_hours": 1,
- "largest_tx": max(transactions, key=lambda x: x.get("amount_usd", 0)) if transactions else None,
- "transactions": transactions[:10] # Keep only top 10 for brevity
- }
-
- logger.info(
- f"{provider} - {endpoint} - Retrieved {whale_data.get('transaction_count', 0)} transactions, "
- f"Total value: ${whale_data.get('total_value_usd', 0):,.0f}" if whale_data else "No data"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": whale_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_arkham_intel() -> Dict[str, Any]:
- """
- Fetch blockchain intelligence data from Arkham Intelligence
-
- Note: Arkham requires authentication and may not have a public API.
- This is a placeholder implementation that should be extended with proper API access.
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "Arkham"
- category = "whale_tracking"
- endpoint = "/intelligence"
-
- logger.info(f"Fetching intelligence data from {provider} (placeholder)")
-
- try:
- # Placeholder implementation
- # Arkham Intelligence may require special access or partnership
- # They provide wallet labeling, entity tracking, and transaction analysis
-
- placeholder_data = {
- "status": "placeholder",
- "message": "Arkham Intelligence API not yet implemented",
- "planned_features": [
- "Wallet address labeling",
- "Entity tracking and attribution",
- "Transaction flow analysis",
- "Dark web marketplace monitoring",
- "Exchange flow tracking"
- ],
- "note": "Requires Arkham API access or partnership"
- }
-
- logger.info(f"{provider} - {endpoint} - Placeholder data returned")
-
- return {
- "provider": provider,
- "category": category,
- "data": placeholder_data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": error_msg,
- "error_type": "exception"
- }
-
-
-async def get_clankapp_whales() -> Dict[str, Any]:
- """
- Fetch whale tracking data from ClankApp
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "ClankApp"
- category = "whale_tracking"
- endpoint = "/whales"
-
- logger.info(f"Fetching whale data from {provider}")
-
- try:
- client = get_client()
-
- # ClankApp public API (if available)
- # Note: This may require API key or may not have public endpoints
- url = "https://clankapp.com/api/v1/whales"
-
- # Make request
- response = await client.get(url, timeout=10)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # If API is not available, return placeholder
- logger.warning(f"{provider} - API not available, returning placeholder")
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": "ClankApp API not accessible or requires authentication",
- "planned_features": [
- "Whale wallet tracking",
- "Large transaction alerts",
- "Portfolio tracking"
- ]
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- # Extract data
- data = response["data"]
-
- logger.info(f"{provider} - {endpoint} - Data retrieved successfully")
-
- return {
- "provider": provider,
- "category": category,
- "data": data,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": f"ClankApp integration error: {str(e)}"
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
-
-async def get_bitquery_whale_transactions() -> Dict[str, Any]:
- """
- Fetch large transactions using BitQuery GraphQL API
-
- Returns:
- Dict with provider, category, data, timestamp, success, error
- """
- provider = "BitQuery"
- category = "whale_tracking"
- endpoint = "/graphql"
-
- logger.info(f"Fetching whale transactions from {provider}")
-
- try:
- client = get_client()
-
- # BitQuery GraphQL endpoint
- url = "https://graphql.bitquery.io"
-
- # GraphQL query for large transactions (>$100k)
- query = """
- {
- ethereum(network: ethereum) {
- transfers(
- amount: {gt: 100000}
- options: {limit: 10, desc: "amount"}
- ) {
- transaction {
- hash
- }
- amount
- currency {
- symbol
- name
- }
- sender {
- address
- }
- receiver {
- address
- }
- block {
- timestamp {
- iso8601
- }
- }
- }
- }
- }
- """
-
- payload = {"query": query}
- headers = {"Content-Type": "application/json"}
-
- # Make request
- response = await client.post(url, json=payload, headers=headers, timeout=15)
-
- # Log request
- log_api_request(
- logger,
- provider,
- endpoint,
- response.get("response_time_ms", 0),
- "success" if response["success"] else "error",
- response.get("status_code")
- )
-
- if not response["success"]:
- # Return placeholder if API fails
- logger.warning(f"{provider} - API request failed, returning placeholder")
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": "BitQuery API requires authentication",
- "planned_features": [
- "Large transaction tracking via GraphQL",
- "Multi-chain whale monitoring",
- "Token transfer analytics"
- ]
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
- # Extract data
- data = response["data"]
-
- whale_data = None
- if isinstance(data, dict) and "data" in data:
- transfers = data.get("data", {}).get("ethereum", {}).get("transfers", [])
-
- if transfers:
- total_value = sum(t.get("amount", 0) for t in transfers)
-
- whale_data = {
- "transaction_count": len(transfers),
- "total_value": round(total_value, 2),
- "largest_transfers": transfers[:5]
- }
-
- logger.info(
- f"{provider} - {endpoint} - Retrieved {whale_data.get('transaction_count', 0)} large transactions"
- if whale_data else f"{provider} - {endpoint} - No data"
- )
-
- return {
- "provider": provider,
- "category": category,
- "data": whale_data or {"status": "no_data", "message": "No large transactions found"},
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "response_time_ms": response.get("response_time_ms", 0)
- }
-
- except Exception as e:
- error_msg = f"Unexpected error: {str(e)}"
- log_error(logger, provider, "exception", error_msg, endpoint, exc_info=True)
- return {
- "provider": provider,
- "category": category,
- "data": {
- "status": "placeholder",
- "message": f"BitQuery integration error: {str(e)}"
- },
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": True,
- "error": None,
- "is_placeholder": True
- }
-
-
-async def collect_whale_tracking_data(whalealert_key: Optional[str] = None) -> List[Dict[str, Any]]:
- """
- Main function to collect whale tracking data from all sources
-
- Args:
- whalealert_key: WhaleAlert API key
-
- Returns:
- List of results from all whale tracking collectors
- """
- logger.info("Starting whale tracking data collection from all sources")
-
- # Run all collectors concurrently
- results = await asyncio.gather(
- get_whalealert_transactions(whalealert_key),
- get_arkham_intel(),
- get_clankapp_whales(),
- get_bitquery_whale_transactions(),
- return_exceptions=True
- )
-
- # Process results
- processed_results = []
- for result in results:
- if isinstance(result, Exception):
- logger.error(f"Collector failed with exception: {str(result)}")
- processed_results.append({
- "provider": "Unknown",
- "category": "whale_tracking",
- "data": None,
- "timestamp": datetime.now(timezone.utc).isoformat(),
- "success": False,
- "error": str(result),
- "error_type": "exception"
- })
- else:
- processed_results.append(result)
-
- # Log summary
- successful = sum(1 for r in processed_results if r.get("success", False))
- placeholder_count = sum(1 for r in processed_results if r.get("is_placeholder", False))
-
- logger.info(
- f"Whale tracking collection complete: {successful}/{len(processed_results)} successful "
- f"({placeholder_count} placeholders)"
- )
-
- return processed_results
-
-
-class WhaleTrackingCollector:
- """
- Whale Tracking Collector class for WebSocket streaming interface
- Wraps the standalone whale tracking collection functions
- """
-
- def __init__(self, config: Any = None):
- """
- Initialize the whale tracking collector
-
- Args:
- config: Configuration object (optional, for compatibility)
- """
- self.config = config
- self.logger = logger
-
- async def collect(self) -> Dict[str, Any]:
- """
- Collect whale tracking data from all sources
-
- Returns:
- Dict with aggregated whale tracking data
- """
- import os
- whalealert_key = os.getenv("WHALEALERT_API_KEY")
- results = await collect_whale_tracking_data(whalealert_key)
-
- # Aggregate data for WebSocket streaming
- aggregated = {
- "large_transactions": [],
- "whale_wallets": [],
- "total_volume": 0,
- "alert_threshold": 1000000, # $1M default threshold
- "alerts": [],
- "timestamp": datetime.now(timezone.utc).isoformat()
- }
-
- for result in results:
- if result.get("success") and result.get("data"):
- provider = result.get("provider", "unknown")
- data = result["data"]
-
- # Skip placeholders
- if isinstance(data, dict) and data.get("status") == "placeholder":
- continue
-
- # Parse WhaleAlert transactions
- if provider == "WhaleAlert" and isinstance(data, dict):
- transactions = data.get("transactions", [])
- for tx in transactions:
- aggregated["large_transactions"].append({
- "amount": tx.get("amount", 0),
- "amount_usd": tx.get("amount_usd", 0),
- "symbol": tx.get("symbol", "unknown"),
- "from": tx.get("from", {}).get("owner", "unknown"),
- "to": tx.get("to", {}).get("owner", "unknown"),
- "timestamp": tx.get("timestamp"),
- "source": provider
- })
- aggregated["total_volume"] += data.get("total_value_usd", 0)
-
- # Parse other sources
- elif isinstance(data, dict):
- tx_count = data.get("transaction_count", 0)
- total_value = data.get("total_value_usd", data.get("total_value", 0))
- aggregated["total_volume"] += total_value
-
- return aggregated
-
-
-# Example usage
-if __name__ == "__main__":
- async def main():
- import os
-
- whalealert_key = os.getenv("WHALEALERT_API_KEY")
-
- results = await collect_whale_tracking_data(whalealert_key)
-
- print("\n=== Whale Tracking Data Collection Results ===")
- for result in results:
- print(f"\nProvider: {result['provider']}")
- print(f"Success: {result['success']}")
- print(f"Is Placeholder: {result.get('is_placeholder', False)}")
-
- if result['success']:
- data = result.get('data', {})
- if isinstance(data, dict):
- if data.get('status') == 'placeholder':
- print(f"Status: {data.get('message', 'N/A')}")
- else:
- print(f"Transaction Count: {data.get('transaction_count', 'N/A')}")
- print(f"Total Value: ${data.get('total_value_usd', data.get('total_value', 0)):,.0f}")
- else:
- print(f"Error: {result.get('error', 'Unknown')}")
-
- asyncio.run(main())
diff --git a/app/complete_dashboard.html b/app/complete_dashboard.html
deleted file mode 100644
index 7ca89714f6edfe4c29134354a692a67f05f75530..0000000000000000000000000000000000000000
--- a/app/complete_dashboard.html
+++ /dev/null
@@ -1,857 +0,0 @@
-
-
-
-
-
- Crypto API Monitor - Complete Dashboard
-
-
-
-