---
license: mit
title: Async Web Scraper
sdk: docker
emoji: 🦀
colorFrom: pink
colorTo: pink
---
# AI-Powered Async Web Scraper

A professional, high-concurrency web scraping pipeline that extracts clean text from URLs, processes it through an LLM (Groq), and stores the structured results in both JSONL and PostgreSQL.

##  Quick Start (Docker)

The easiest way to run the entire stack is using Docker Compose:

1.  **Configure Environment**:
    Create a `.env` file in the `Backend` directory with your Groq API Key:
    ```bash
    GROQ_API_KEY=your_key_here
    DB_PASSWORD=your_db_password
    ```

2.  **Launch**:
    ```bash
    docker-compose up -d
    ```

3.  **Access**:
    - **Frontend**: [http://localhost](http://localhost)
    - **Backend API**: [http://localhost:8000](http://localhost:8000)

## Architecture

- **Frontend**: A modern dashboard built with **Vite** and **TypeScript** for real-time job monitoring and result downloads.
- **Backend**: A high-performance **FastAPI** service managing an asynchronous scraping pipeline.
- **Database**: **PostgreSQL** for persistent storage of parsed text and structured AI extractions.
- **Scraper**: Uses `curl_cffi` for advanced request impersonation to bypass bot detection.
- **Parser**: Leverages `trafilatura` for high-quality main-content extraction.
- **Extraction**: Integrates with **Groq** for lightning-fast LLM-based data structuring.


##  Project Structure

- `Backend/`: FastAPI application, database models, and scraper logic.
- `Frontend/`: Vite/TS dashboard for job status tracking and results visualization.
- `docker-compose.yml`: Orchestration for the frontend, backend, and persistence layers.

---
For detailed setup and development instructions, see the individual component's documentation:
- [Backend Documentation](./Backend/README.md)
- [Frontend Documentation](./Frontend/README.md)