Upload 374 files

DEPLOYMENT_GUIDE.md

@@ -1,600 +1,269 @@
-# Deployment Guide - Crypto Resource Aggregator
+# 🚀 Deployment Guide - HuggingFace Space
 
-## Quick Deployment to Hugging Face Spaces
+## مشکل Dependency Conflict و حل آن
 
-### Method 1: Web Interface (Recommended for Beginners)
-
-1. **Create a Hugging Face Account**
-   - Go to https://huggingface.co/join
-   - Sign up for a free account
-
-2. **Create a New Space**
-   - Go to https://huggingface.co/new-space
-   - Choose a name (e.g., `crypto-resource-aggregator`)
-   - Select SDK: **Docker**
-   - Choose visibility: **Public** or **Private**
-   - Click "Create Space"
-
-3. **Upload Files**
-   Upload the following files to your Space:
-   - `app.py` - Main application file
-   - `requirements.txt` - Python dependencies
-   - `all_apis_merged_2025.json` - Resource configuration
-   - `README.md` - Documentation
-   - `Dockerfile` - Docker configuration
-
-4. **Wait for Build**
-   - The Space will automatically build and deploy
-   - This may take 2-5 minutes
-   - You'll see the build logs in real-time
-
-5. **Access Your API**
-   - Once deployed, your API will be available at:
-     `https://[your-username]-[space-name].hf.space`
-   - Example: `https://username-crypto-resource-aggregator.hf.space`
-
-### Method 2: Git CLI (Recommended for Advanced Users)
-
-```bash
-# Clone your Space repository
-git clone https://huggingface.co/spaces/[your-username]/[space-name]
-cd [space-name]
-
-# Copy all files to the repository
-cp app.py requirements.txt all_apis_merged_2025.json README.md Dockerfile .
-
-# Commit and push
-git add .
-git commit -m "Initial deployment of Crypto Resource Aggregator"
-git push
+### مشکل قبلی:
 ```
-
----
-
-## Alternative Deployment Options
-
-### Option 1: Heroku
-
-```bash
-# Install Heroku CLI
-# https://devcenter.heroku.com/articles/heroku-cli
-
-# Create a new app
-heroku create crypto-resource-aggregator
-
-# Create Procfile
-echo "web: python app.py" > Procfile
-
-# Deploy
-git add .
-git commit -m "Deploy to Heroku"
-git push heroku main
-
-# Open your app
-heroku open
+ERROR: Cannot install gradio and websockets>=12.0 because these package versions have conflicting dependencies.
 ```
 
-### Option 2: Railway
-
-```bash
-# Install Railway CLI
-npm i -g @railway/cli
-
-# Login
-railway login
-
-# Initialize project
-railway init
-
-# Deploy
-railway up
-
-# Get deployment URL
-railway domain
+### حل شده:
+```python
+websockets>=10.4,<12.0  # Compatible with gradio-client
 ```
 
-### Option 3: Render
-
-1. Go to https://render.com
-2. Click "New +" → "Web Service"
-3. Connect your GitHub repository
-4. Configure:
-   - **Build Command**: `pip install -r requirements.txt`
-   - **Start Command**: `python app.py`
-   - **Environment**: Python 3
-5. Click "Create Web Service"
-
-### Option 4: Docker (Self-Hosted)
+## نصب و راه‌اندازی
 
+### روش 1: Docker (توصیه می‌شود)
 ```bash
-# Build the Docker image
-docker build -t crypto-aggregator .
-
-# Run the container
-docker run -d -p 7860:7860 --name crypto-aggregator crypto-aggregator
-
-# Check logs
-docker logs crypto-aggregator
-
-# Stop the container
-docker stop crypto-aggregator
-
-# Remove the container
-docker rm crypto-aggregator
-```
-
-### Option 5: Docker Compose (Self-Hosted)
-
-Create `docker-compose.yml`:
+# Build
+docker build -t crypto-hf-hub .
 
-```yaml
-version: '3.8'
-
-services:
-  aggregator:
-    build: .
-    ports:
-      - "7860:7860"
-    restart: unless-stopped
-    volumes:
-      - ./history.db:/app/history.db
-    environment:
-      - ENVIRONMENT=production
-```
+# Run
+docker run -d \
+  -p 7860:7860 \
+  -e HF_TOKEN=your_token_here \
+  --name crypto-hub \
+  crypto-hf-hub
 
-Run:
-```bash
-docker-compose up -d
+# Logs
+docker logs -f crypto-hub
 ```
 
-### Option 6: AWS EC2
-
+### روش 2: مستقیم (Local Development)
 ```bash
-# Connect to your EC2 instance
-ssh -i your-key.pem ubuntu@your-instance-ip
-
-# Install Python and dependencies
-sudo apt update
-sudo apt install python3-pip python3-venv -y
-
-# Create virtual environment
+# ایجاد virtual environment
 python3 -m venv venv
-source venv/bin/activate
-
-# Upload files (from local machine)
-scp -i your-key.pem app.py requirements.txt all_apis_merged_2025.json ubuntu@your-instance-ip:~/
+source venv/bin/activate  # Windows: venv\Scripts\activate
 
-# Install dependencies
+# نصب dependencies
+pip install --upgrade pip
 pip install -r requirements.txt
 
-# Run with nohup
-nohup python app.py > output.log 2>&1 &
-
-# Or use systemd service (recommended)
-sudo nano /etc/systemd/system/crypto-aggregator.service
-```
-
-Create systemd service file:
-```ini
-[Unit]
-Description=Crypto Resource Aggregator
-After=network.target
-
-[Service]
-User=ubuntu
-WorkingDirectory=/home/ubuntu/crypto-aggregator
-ExecStart=/home/ubuntu/venv/bin/python app.py
-Restart=always
-
-[Install]
-WantedBy=multi-user.target
-```
+# تنظیم environment variables
+export HF_TOKEN=your_huggingface_token
+export PORT=7860
 
-Enable and start:
-```bash
-sudo systemctl enable crypto-aggregator
-sudo systemctl start crypto-aggregator
-sudo systemctl status crypto-aggregator
+# اجرا
+uvicorn hf_unified_server:app --host 0.0.0.0 --port 7860 --reload
 ```
 
-### Option 7: Google Cloud Run
-
+### روش 3: HuggingFace Space
 ```bash
-# Install gcloud CLI
-# https://cloud.google.com/sdk/docs/install
+# 1. ساخت Space جدید در HuggingFace
+#    Space SDK: Docker
+#    Hardware: CPU basic (free)
 
-# Authenticate
-gcloud auth login
+# 2. کلون کردن repository
+git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+cd YOUR_SPACE_NAME
 
-# Set project
-gcloud config set project YOUR_PROJECT_ID
+# 3. کپی کردن فایل‌های پروژه
+cp -r crypto-hf-integrated/* .
 
-# Build and deploy
-gcloud run deploy crypto-aggregator \
-  --source . \
-  --platform managed \
-  --region us-central1 \
-  --allow-unauthenticated
-
-# Get URL
-gcloud run services describe crypto-aggregator --region us-central1 --format 'value(status.url)'
+# 4. Commit و Push
+git add .
+git commit -m "Initial deployment with HF integration"
+git push
 ```
 
-### Option 8: DigitalOcean App Platform
-
-1. Go to https://cloud.digitalocean.com/apps
-2. Click "Create App"
-3. Connect your GitHub repository
-4. Configure:
-   - **Run Command**: `python app.py`
-   - **Environment**: Python 3.11
-   - **HTTP Port**: 7860
-5. Click "Deploy"
+## تنظیمات HuggingFace Space
 
----
+### Dockerfile
+```dockerfile
+FROM python:3.10-slim
 
-## Environment Variables (Optional)
+WORKDIR /app
 
-You can configure the following environment variables:
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
 
-```bash
-# Port (default: 7860)
-export PORT=8000
+# Copy requirements
+COPY requirements.txt .
 
-# Log level (default: INFO)
-export LOG_LEVEL=DEBUG
-
-# Database path (default: history.db)
-export DATABASE_PATH=/path/to/history.db
-```
+# Install Python dependencies
+RUN pip install --no-cache-dir -r requirements.txt
 
----
+# Copy application
+COPY . .
 
-## Post-Deployment Testing
+# Create directories
+RUN mkdir -p data/database logs
 
-### 1. Test Health Endpoint
+# Expose port
+EXPOSE 7860
 
-```bash
-curl https://your-deployment-url.com/health
-```
+# Set environment
+ENV PORT=7860
+ENV PYTHONUNBUFFERED=1
 
-Expected response:
-```json
-{
-  "status": "healthy",
-  "timestamp": "2025-11-10T...",
-  "resources_loaded": true,
-  "database_connected": true
-}
+# Start server
+CMD ["uvicorn", "hf_unified_server:app", "--host", "0.0.0.0", "--port", "7860"]
 ```
 
-### 2. Test Resource Listing
+### Space Settings (در UI)
+```yaml
+# Secrets
+HF_TOKEN: your_token_here
 
-```bash
-curl https://your-deployment-url.com/resources
+# Hardware
+- CPU basic (Free) - برای شروع
+- CPU upgrade (Paid) - برای production با traffic بیشتر
 ```
 
-### 3. Test Query Endpoint
+## تست Endpoints
 
+### Health Check
 ```bash
-curl -X POST https://your-deployment-url.com/query \
-  -H "Content-Type: application/json" \
-  -d '{
-    "resource_type": "market_data",
-    "resource_name": "coingecko",
-    "endpoint": "/simple/price",
-    "params": {
-      "ids": "bitcoin",
-      "vs_currencies": "usd"
-    }
-  }'
+curl http://localhost:7860/api/health
 ```
 
-### 4. Test Status Monitoring
-
+### Top Coins
 ```bash
-curl https://your-deployment-url.com/status
+curl http://localhost:7860/api/coins/top?limit=10
 ```
 
-### 5. Run Full Test Suite
-
-From your local machine:
-
+### Market Stats
 ```bash
-# Update BASE_URL in test_aggregator.py
-# Change: BASE_URL = "http://localhost:7860"
-# To: BASE_URL = "https://your-deployment-url.com"
-
-# Run tests
-python test_aggregator.py
+curl http://localhost:7860/api/market/stats
 ```
 
----
-
-## Performance Optimization
-
-### 1. Enable Caching
-
-Add Redis for caching (optional):
-
-```python
-import redis
-import json
-
-# Connect to Redis
-redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)
-
-# Cache resource data
-def get_cached_data(key, ttl=300):
-    cached = redis_client.get(key)
-    if cached:
-        return json.loads(cached)
-    return None
-
-def set_cached_data(key, data, ttl=300):
-    redis_client.setex(key, ttl, json.dumps(data))
-```
-
-### 2. Use Connection Pooling
-
-Already implemented with `aiohttp.ClientSession`
-
-### 3. Add Rate Limiting
-
-Install:
+### Sentiment Analysis
 ```bash
-pip install slowapi
-```
-
-Add to `app.py`:
-```python
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-from slowapi.errors import RateLimitExceeded
-
-limiter = Limiter(key_func=get_remote_address)
-app.state.limiter = limiter
-app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
-
-@app.post("/query")
-@limiter.limit("60/minute")
-async def query_resource(request: Request, query: ResourceQuery):
-    # ... existing code
+curl -X POST http://localhost:7860/api/sentiment/analyze \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Bitcoin is breaking new ATH!"}'
 ```
 
-### 4. Add Monitoring
-
-Use Sentry for error tracking:
-
+### Models List
 ```bash
-pip install sentry-sdk
+curl http://localhost:7860/api/models/list
 ```
 
-```python
-import sentry_sdk
-from sentry_sdk.integrations.fastapi import FastApiIntegration
-
-sentry_sdk.init(
-    dsn="your-sentry-dsn",
-    integrations=[FastApiIntegration()],
-    traces_sample_rate=1.0,
-)
+### Datasets List
+```bash
+curl http://localhost:7860/api/datasets/list
 ```
 
----
+### WebSocket Test (در browser console)
+```javascript
+const ws = new WebSocket('ws://localhost:7860/ws');
 
-## Security Best Practices
-
-### 1. API Key Management
-
-Store API keys in environment variables:
-
-```python
-import os
-
-API_KEYS = {
-    'etherscan': os.getenv('ETHERSCAN_API_KEY', 'default-key'),
-    'coinmarketcap': os.getenv('CMC_API_KEY', 'default-key'),
-}
+ws.onopen = () => console.log('Connected');
+ws.onmessage = (event) => console.log('Update:', JSON.parse(event.data));
+ws.onerror = (error) => console.error('Error:', error);
 ```
 
-### 2. Enable HTTPS
-
-Most platforms (Hugging Face, Heroku, etc.) provide HTTPS by default.
-
-For self-hosted, use Let's Encrypt:
+## مشکلات رایج و راه حل
 
+### 1. Out of Memory
 ```bash
-# Install Certbot
-sudo apt install certbot python3-certbot-nginx
+# کاهش مدل‌های load شده
+# در ai_models.py فقط crypto_sent_0 و financial_sent_0 load میشه
 
-# Get certificate
-sudo certbot --nginx -d your-domain.com
+# یا استفاده از CPU basic به جای GPU
 ```
 
-### 3. Add Authentication (Optional)
-
+### 2. Slow Model Loading
 ```bash
-pip install python-jose[cryptography] passlib[bcrypt]
-```
-
-```python
-from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
-from fastapi import Security
-
-security = HTTPBearer()
-
-@app.post("/query")
-async def query_resource(
-    query: ResourceQuery,
-    credentials: HTTPAuthorizationCredentials = Security(security)
-):
-    # Verify token
-    if credentials.credentials != "your-secret-token":
-        raise HTTPException(status_code=401, detail="Invalid token")
-    # ... existing code
+# استفاده از model caching در HuggingFace
+export TRANSFORMERS_CACHE=/app/.cache/huggingface
 ```
 
----
-
-## Monitoring & Maintenance
-
-### 1. Monitor Logs
-
-Hugging Face Spaces:
-- View logs in the Space settings → "Logs" tab
-
-Docker:
+### 3. Rate Limiting
 ```bash
-docker logs -f crypto-aggregator
+# استفاده از HF_TOKEN برای rate limit بالاتر
+export HF_TOKEN=hf_your_token
 ```
 
-Systemd:
+### 4. WebSocket Connection Failed
 ```bash
-journalctl -u crypto-aggregator -f
+# چک کردن CORS settings
+# مطمئن شوید که در hf_unified_server.py:
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
 ```
 
-### 2. Database Maintenance
-
-Backup database regularly:
-
-```bash
-# Local backup
-cp history.db history_backup_$(date +%Y%m%d).db
+## بهینه‌سازی برای Production
 
-# Remote backup
-scp user@server:/path/to/history.db ./backups/
-```
-
-Clean old records:
+### 1. Caching
+```python
+# در hf_unified_server.py
+from functools import lru_cache
 
-```sql
--- Remove records older than 30 days
-DELETE FROM query_history WHERE timestamp < datetime('now', '-30 days');
-DELETE FROM resource_status WHERE last_check < datetime('now', '-30 days');
+@lru_cache(maxsize=128)
+def get_cached_coins(limit: int):
+    # Cache results for 1 minute
+    pass
 ```
 
-### 3. Update Resources
-
-To add new resources, update `all_apis_merged_2025.json` and redeploy.
-
-### 4. Health Checks
+### 2. Rate Limiting
+```python
+from slowapi import Limiter
+from slowapi.util import get_remote_address
 
-Set up automated health checks:
+limiter = Limiter(key_func=get_remote_address)
+app.state.limiter = limiter
 
-```bash
-# Cron job (every 5 minutes)
-*/5 * * * * curl https://your-deployment-url.com/health || echo "API is down!"
+@app.get("/api/sentiment/analyze")
+@limiter.limit("10/minute")
+async def analyze_sentiment(...):
+    pass
 ```
 
-Use UptimeRobot or similar service for monitoring.
-
----
-
-## Troubleshooting
-
-### Issue: Server won't start
-
-**Solution:**
-```bash
-# Check if port 7860 is in use
-lsof -i :7860
-
-# Kill existing process
-kill -9 $(lsof -t -i:7860)
-
-# Or use a different port
-PORT=8000 python app.py
+### 3. Model Optimization
+```python
+# Use smaller models for faster inference
+# quantization برای کاهش حجم
+from transformers import AutoModelForSequenceClassification
+model = AutoModelForSequenceClassification.from_pretrained(
+    "model_id",
+    torch_dtype=torch.float16  # Half precision
+)
 ```
 
-### Issue: Database locked
+## Monitoring
 
-**Solution:**
+### Logs
 ```bash
-# Stop all instances
-pkill -f app.py
-
-# Remove lock (if exists)
-rm history.db-journal
+# Docker logs
+docker logs -f crypto-hub
 
-# Restart
-python app.py
+# HuggingFace Space logs
+# قابل مشاهده در dashboard Space
 ```
 
-### Issue: High memory usage
-
-**Solution:**
-- Add connection limits
-- Implement request queuing
-- Scale horizontally with multiple instances
-
-### Issue: API rate limits
-
-**Solution:**
-- Implement caching
-- Add multiple API keys for rotation
-- Use fallback resources
-
----
-
-## Scaling
-
-### Horizontal Scaling
-
-Use a load balancer with multiple instances:
+### Metrics
+```bash
+# Health endpoint برای monitoring
+curl http://your-space.hf.space/api/health
 
-```yaml
-# docker-compose-scaled.yml
-version: '3.8'
-
-services:
-  aggregator:
-    build: .
-    deploy:
-      replicas: 3
-    environment:
-      - WORKER_ID=${HOSTNAME}
-
-  nginx:
-    image: nginx:alpine
-    ports:
-      - "80:80"
-    volumes:
-      - ./nginx.conf:/etc/nginx/nginx.conf
-    depends_on:
-      - aggregator
+# Response:
+{
+  "status": "healthy",
+  "uptime": 3600,
+  "models_loaded": 2,
+  "datasets_available": 14
+}
 ```
 
-### Vertical Scaling
-
-Increase resources on your hosting platform:
-- Hugging Face: Upgrade to paid tier
-- AWS: Use larger EC2 instance
-- Docker: Adjust container resources
-
----
-
 ## Support
 
-For issues or questions:
-1. Check `/health` endpoint
-2. Review application logs
-3. Test individual resources with `/status`
-4. Verify database with SQLite browser
-
----
-
-## Next Steps
-
-After deployment:
-
-1. **Integrate with your main app** using the provided client examples
-2. **Set up monitoring** with health checks and alerts
-3. **Configure backups** for the history database
-4. **Add custom resources** by updating the JSON file
-5. **Implement caching** for frequently accessed data
-6. **Enable authentication** if needed for security
+- **Documentation**: README_HF_INTEGRATION.md
+- **Issues**: Check logs first
+- **Performance**: Use HF Space with GPU for heavy models
 
----
+## نکات مهم
 
-**Congratulations! Your Crypto Resource Aggregator is now deployed and ready to use!** 🚀
+1. ⚠️ **HF_TOKEN**: حتماً در secrets تنظیم کنید
+2. 🔥 **Models**: اولین بار کند است (downloading)
+3. 💾 **Cache**: مدل‌ها cache می‌شوند
+4. 🚀 **WebSocket**: برای real-time updates
+5. 📊 **Dashboard**: در `/` قابل دسترسی

Dockerfile.optimized

@@ -0,0 +1,51 @@
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    git \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+
+# Upgrade pip
+RUN pip install --no-cache-dir --upgrade pip
+
+# Install dependencies
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy application code
+COPY . .
+
+# Create necessary directories
+RUN mkdir -p \
+    data/database \
+    data/backups \
+    logs \
+    static/css \
+    static/js \
+    .cache/huggingface
+
+# Set permissions
+RUN chmod -R 755 /app
+
+# Environment variables
+ENV PORT=7860 \
+    PYTHONUNBUFFERED=1 \
+    TRANSFORMERS_CACHE=/app/.cache/huggingface \
+    HF_HOME=/app/.cache/huggingface \
+    PYTHONDONTWRITEBYTECODE=1
+
+# Expose port
+EXPOSE 7860
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+    CMD curl -f http://localhost:7860/api/health || exit 1
+
+# Run application
+CMD ["uvicorn", "hf_unified_server:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]

requirements.txt

@@ -1,41 +1,57 @@
-# Unified dependencies for API server + Gradio dashboards (Hugging Face Spaces).
-#
-# This single file is what Hugging Face Spaces will install. It includes:
-# - Core API server stack
-# - Database/SQLAlchemy layer
-# - Gradio dashboard + Plotly charts
-# - Optional Transformers-based AI features
-
-# Core API Server Requirements
-fastapi==0.109.0
-uvicorn
-pydantic==2.5.3
-sqlalchemy==2.0.25
-httpx==0.26.0
-websockets>=12.0
-python-dotenv
-python-multipart
-requests
-aiohttp>=3.8.0
-
-# Data Processing
-pandas>=2.1.0
-
-# Gradio Dashboard & UI (Required for app.py)
-gradio==4.12.0
-plotly==5.18.0
-psutil==5.9.6
-
-# AI/ML Libraries (optional but recommended for AI features)
-transformers>=4.36.0
-torch>=2.0.0
-sentencepiece>=0.1.99
-
-# RSS Feed Parsing (Optional)
-feedparser>=6.0.10
-
-# HTML Parsing (Optional)
-beautifulsoup4>=4.12.0
-
-# HuggingFace Hub (For model validation)
-huggingface-hub>=0.19.0
+# Unified dependencies for Crypto Intelligence Hub (HuggingFace Space)
+# Optimized for HF deployment with minimal conflicts
+
+# ===== Core API Stack =====
+fastapi==0.115.0
+uvicorn[standard]==0.30.0
+pydantic==2.9.0
+pydantic-settings==2.5.0
+sqlalchemy==2.0.35
+httpx==0.27.2
+websockets>=10.4,<12.0  # Compatible with gradio-client
+python-dotenv==1.0.1
+python-multipart==0.0.9
+requests==2.32.3
+aiohttp==3.10.5
+
+# ===== Data Processing =====
+pandas==2.2.3
+numpy>=1.26.0,<2.0.0
+
+# ===== Gradio Dashboard =====
+gradio==4.44.0
+plotly==5.24.1
+psutil==6.0.0
+
+# ===== HuggingFace & AI/ML =====
+# Core transformers with minimal deps
+transformers>=4.45.0
+tokenizers>=0.20.0
+huggingface-hub>=0.25.0
+safetensors>=0.4.0
+
+# Datasets library
+datasets>=3.0.0
+
+# PyTorch - CPU only for space efficiency
+torch>=2.4.0,<2.5.0
+torchaudio>=2.4.0,<2.5.0
+
+# Optional: Sentence transformers for embeddings
+sentence-transformers>=3.1.0
+
+# Tokenization
+sentencepiece==0.2.0
+
+# ===== Content Parsing =====
+feedparser==6.0.11
+beautifulsoup4==4.12.3
+lxml==5.3.0
+
+# ===== Utilities =====
+python-dateutil>=2.9.0
+pytz>=2024.1
+tenacity>=9.0.0
+
+# ===== Optional: Acceleration =====
+# accelerate>=0.34.0  # Uncomment if using multi-GPU