Spaces:

orakzai-io
/

Async_Web_scraper

Sleeping

orakzai-io

Initial Version

34d0dc9 7 months ago

2.32 kB

Backend - Async Web Scraper API

A high-performance FastAPI service coordinating an asynchronous database-driven web scraping and AI extraction pipeline.

The backend is organized into a modular pipeline, with each file and class having a dedicated responsibility.

main.py: The application entry point.
- Handles FastAPI initialization and lifespan management.
- Configures Loguru for structured logging and sets up CORS middleware.
- Runs the Uvicorn server.
api.py: The interface layer.
- Defines the APIRouter and endpoints (/scrape, /results, /download).
- Manages in-memory job state and initiates BackgroundTasks for asynchronous execution.
- Coordinates the run() loop for processing a batch of URLs.

scraper_manager.py (ScraperManager): The orchestrator.
- Coordinates the full 6-stage scraping pipeline (Fetch -> Parse -> DB Store -> LLM -> JSONL -> DB Store).
- Interfaces between all the specialized modules.
browser_client.py (BrowserClient): The fetching layer.
- Uses curl_cffi to fetch HTML while impersonating modern browser fingerprints (TLS/HTTP2) to bypass bot protection.
parser.py (Parser): The content engine.
- Leverages trafilatura to extract clean, structured text from raw HTML.
- Includes token-counting logic using tiktoken.
llm_extractor.py (LLMExtractor): The AI intelligence.
- Interfaces with Groq to transform raw text into structured JSON based on user-defined prompts.
- Implements tenacity retry logic for robust AI interactions.
datastore.py (DataStore): The database manager.
- Handles all SQLAlchemy operations, including saving parsed content and structured extractions to PostgreSQL.
models.py & database.py: The data foundation.
- Defines the relational schema (ParsedData, LlmExtraction) and sets up the database engine/session management.
JSONStorage.py (JSONStorage): File persistence.
- Asynchronously appends structured results to a local .jsonl file using aiofiles.