README.md

metadata

license: mit
title: Async Web Scraper
sdk: docker
emoji: 🦀
colorFrom: pink
colorTo: pink

AI-Powered Async Web Scraper

A professional, high-concurrency web scraping pipeline that extracts clean text from URLs, processes it through an LLM (Groq), and stores the structured results in both JSONL and PostgreSQL.

Quick Start (Docker)

The easiest way to run the entire stack is using Docker Compose:

1. Configure Environment: Create a .env file in the Backend directory with your Groq API Key:

   GROQ_API_KEY=your_key_here
   DB_PASSWORD=your_db_password
   

2. Launch:

   docker-compose up -d
   

3. Access:

   • Frontend: http://localhost
   • Backend API: http://localhost:8000

Architecture

• Frontend: A modern dashboard built with Vite and TypeScript for real-time job monitoring and result downloads.
• Backend: A high-performance FastAPI service managing an asynchronous scraping pipeline.
• Database: PostgreSQL for persistent storage of parsed text and structured AI extractions.
• Scraper: Uses curl_cffi for advanced request impersonation to bypass bot detection.
• Parser: Leverages trafilatura for high-quality main-content extraction.
• Extraction: Integrates with Groq for lightning-fast LLM-based data structuring.

Project Structure

• Backend/: FastAPI application, database models, and scraper logic.
• Frontend/: Vite/TS dashboard for job status tracking and results visualization.
• docker-compose.yml: Orchestration for the frontend, backend, and persistence layers.

---

For detailed setup and development instructions, see the individual component's documentation:

• Backend Documentation
• Frontend Documentation