api/collectors/QUICK_START.md

# Collectors Quick Start Guide

## Files Created

```
/home/user/crypto-dt-source/collectors/
├── __init__.py              # Package exports
├── market_data.py           # Market data collectors (16 KB)
├── explorers.py             # Blockchain explorer collectors (17 KB)
├── news.py                  # News aggregation collectors (13 KB)
├── sentiment.py             # Sentiment data collectors (7.8 KB)
├── onchain.py               # On-chain analytics (placeholder, 13 KB)
├── demo_collectors.py       # Comprehensive demo script (6.6 KB)
├── README.md                # Full documentation
└── QUICK_START.md          # This file
```

## Quick Test

### Test All Collectors

```bash
cd /home/user/crypto-dt-source
python collectors/demo_collectors.py
```

### Test Individual Modules

```bash
# Market Data (CoinGecko, CoinMarketCap, Binance)
python -m collectors.market_data

# Blockchain Explorers (Etherscan, BscScan, TronScan)
python -m collectors.explorers

# News (CryptoPanic, NewsAPI)
python -m collectors.news

# Sentiment (Alternative.me Fear & Greed)
python -m collectors.sentiment

# On-chain Analytics (Placeholder)
python -m collectors.onchain
```

## Import and Use

### Collect All Market Data

```python
import asyncio
from collectors import collect_market_data

results = asyncio.run(collect_market_data())

for result in results:
    print(f"{result['provider']}: {result['success']}")
```

### Collect All Data from All Categories

```python
import asyncio
from collectors import (
    collect_market_data,
    collect_explorer_data,
    collect_news_data,
    collect_sentiment_data,
    collect_onchain_data
)

async def main():
    # Run all collectors concurrently
    results = await asyncio.gather(
        collect_market_data(),
        collect_explorer_data(),
        collect_news_data(),
        collect_sentiment_data(),
        collect_onchain_data()
    )

    market, explorers, news, sentiment, onchain = results

    print(f"Market data: {len(market)} sources")
    print(f"Explorers: {len(explorers)} sources")
    print(f"News: {len(news)} sources")
    print(f"Sentiment: {len(sentiment)} sources")
    print(f"On-chain: {len(onchain)} sources (placeholder)")

asyncio.run(main())
```

### Individual Collector Example

```python
import asyncio
from collectors.market_data import get_coingecko_simple_price

async def get_prices():
    result = await get_coingecko_simple_price()

    if result['success']:
        data = result['data']
        print(f"BTC: ${data['bitcoin']['usd']:,.2f}")
        print(f"ETH: ${data['ethereum']['usd']:,.2f}")
        print(f"BNB: ${data['binancecoin']['usd']:,.2f}")
        print(f"Data age: {result['staleness_minutes']:.2f} minutes")
    else:
        print(f"Error: {result['error']}")

asyncio.run(get_prices())
```

## Collectors Summary

### 1. Market Data (market_data.py)

| Function | Provider | API Key Required | Description |
|----------|----------|------------------|-------------|
| `get_coingecko_simple_price()` | CoinGecko | No | BTC, ETH, BNB prices with market data |
| `get_coinmarketcap_quotes()` | CoinMarketCap | Yes | Professional market data |
| `get_binance_ticker()` | Binance | No | Real-time 24hr ticker |
| `collect_market_data()` | All above | - | Collects from all sources |

### 2. Blockchain Explorers (explorers.py)

| Function | Provider | API Key Required | Description |
|----------|----------|------------------|-------------|
| `get_etherscan_gas_price()` | Etherscan | Yes | Current Ethereum gas prices |
| `get_bscscan_bnb_price()` | BscScan | Yes | BNB price and BSC stats |
| `get_tronscan_stats()` | TronScan | Optional | TRON network statistics |
| `collect_explorer_data()` | All above | - | Collects from all sources |

### 3. News Aggregation (news.py)

| Function | Provider | API Key Required | Description |
|----------|----------|------------------|-------------|
| `get_cryptopanic_posts()` | CryptoPanic | No | Latest crypto news posts |
| `get_newsapi_headlines()` | NewsAPI | Yes | Crypto-related headlines |
| `collect_news_data()` | All above | - | Collects from all sources |

### 4. Sentiment Analysis (sentiment.py)

| Function | Provider | API Key Required | Description |
|----------|----------|------------------|-------------|
| `get_fear_greed_index()` | Alternative.me | No | Market Fear & Greed Index |
| `collect_sentiment_data()` | All above | - | Collects from all sources |

### 5. On-Chain Analytics (onchain.py)

| Function | Provider | Status | Description |
|----------|----------|--------|-------------|
| `get_the_graph_data()` | The Graph | Placeholder | GraphQL blockchain data |
| `get_blockchair_data()` | Blockchair | Placeholder | Blockchain statistics |
| `get_glassnode_metrics()` | Glassnode | Placeholder | Advanced on-chain metrics |
| `collect_onchain_data()` | All above | - | Collects from all sources |

## API Keys Setup

Create a `.env` file or set environment variables:

```bash
# Market Data
export COINMARKETCAP_KEY_1="your_key_here"

# Blockchain Explorers
export ETHERSCAN_KEY_1="your_key_here"
export BSCSCAN_KEY="your_key_here"
export TRONSCAN_KEY="your_key_here"

# News
export NEWSAPI_KEY="your_key_here"
```

## Output Format

All collectors return standardized format:

```python
{
    "provider": "CoinGecko",                    # Provider name
    "category": "market_data",                  # Category
    "data": {...},                              # Raw API response
    "timestamp": "2025-11-11T00:20:00Z",       # Collection time
    "data_timestamp": "2025-11-11T00:19:30Z",  # Data timestamp
    "staleness_minutes": 0.5,                   # Data age
    "success": True,                            # Success flag
    "error": None,                              # Error message
    "error_type": None,                         # Error type
    "response_time_ms": 342.5                   # Response time
}
```

## Key Features

✓ **Async/Concurrent** - All collectors run asynchronously
✓ **Error Handling** - Comprehensive error handling and logging
✓ **Staleness Tracking** - Calculates data age in minutes
✓ **Rate Limiting** - Respects API rate limits
✓ **Retry Logic** - Automatic retries with exponential backoff
✓ **Structured Logging** - JSON-formatted logs
✓ **API Key Management** - Secure key handling from environment
✓ **Standardized Output** - Consistent response format
✓ **Production Ready** - Ready for production deployment

## Common Issues

### 1. Missing API Keys

```
Error: API key required but not configured for CoinMarketCap
```

**Solution:** Set the required environment variable:
```bash
export COINMARKETCAP_KEY_1="your_api_key"
```

### 2. Rate Limit Exceeded

```
Error Type: rate_limit
```

**Solution:** Collectors automatically retry with backoff. Check rate limits in provider documentation.

### 3. Network Timeout

```
Error Type: timeout
```

**Solution:** Collectors automatically increase timeout and retry. Check network connectivity.

## Next Steps

1. Run the demo: `python collectors/demo_collectors.py`
2. Configure API keys for providers requiring authentication
3. Integrate collectors into your monitoring system
4. Implement on-chain collectors (currently placeholders)
5. Add custom collectors following the existing patterns

## Support

- Full documentation: `collectors/README.md`
- Demo script: `collectors/demo_collectors.py`
- Configuration: `config.py`
- API Client: `utils/api_client.py`
- Logger: `utils/logger.py`

---

**Total Collectors:** 14 functions across 5 modules
**Total Code:** ~75 KB of production-ready Python code
**Status:** Ready for production use (except on-chain placeholders)