README.md

metadata

title: Topic AI Scraper
emoji: 🔍
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false

Topic AI Scraper

FastAPI application that scrapes websites and classifies their topics using AI.

Features

• Upload URLs for batch processing
• Asynchronous web scraping
• AI-powered topic classification
• Progress tracking with Redis
• Task management (stop, delete, download results)

API Endpoints

• POST /upload_urls - Submit URLs for processing
• GET /tasks - List processing tasks
• GET /progress/{task_id} - Check task progress
• GET /download/{task_id} - Download results
• POST /stop/{task_id} - Stop a running task
• POST /delete/{task_id} - Delete a task
• GET /logs - View application logs
• GET / - Homepage with GPU status

Usage

curl -X POST "https://YOUR-SPACE-URL/upload_urls" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com", "https://example2.com"]}'

Environment Variables

Set BACKEND_API_URL to your Modal API endpoint in the Space settings.