Spaces:
Sleeping
Sleeping
| # Backend - Async Web Scraper API | |
| A high-performance **FastAPI** service coordinating an asynchronous database-driven web scraping and AI extraction pipeline. | |
| ## Project Structure & Core Classes | |
| The backend is organized into a modular pipeline, with each file and class having a dedicated responsibility. | |
| ### Root Files | |
| - **`main.py`**: The application entry point. | |
| - Handles **FastAPI** initialization and lifespan management. | |
| - Configures **Loguru** for structured logging and sets up **CORS** middleware. | |
| - Runs the **Uvicorn** server. | |
| - **`api.py`**: The interface layer. | |
| - Defines the **APIRouter** and endpoints (`/scrape`, `/results`, `/download`). | |
| - Manages in-memory job state and initiates **BackgroundTasks** for asynchronous execution. | |
| - Coordinates the `run()` loop for processing a batch of URLs. | |
| ### App Module (`app/`) | |
| - **`scraper_manager.py` (`ScraperManager`)**: The orchestrator. | |
| - Coordinates the full 6-stage scraping pipeline (Fetch -> Parse -> DB Store -> LLM -> JSONL -> DB Store). | |
| - Interfaces between all the specialized modules. | |
| - **`browser_client.py` (`BrowserClient`)**: The fetching layer. | |
| - Uses **`curl_cffi`** to fetch HTML while impersonating modern browser fingerprints (TLS/HTTP2) to bypass bot protection. | |
| - **`parser.py` (`Parser`)**: The content engine. | |
| - Leverages **`trafilatura`** to extract clean, structured text from raw HTML. | |
| - Includes token-counting logic using **`tiktoken`**. | |
| - **`llm_extractor.py` (`LLMExtractor`)**: The AI intelligence. | |
| - Interfaces with **Groq** to transform raw text into structured JSON based on user-defined prompts. | |
| - Implements **`tenacity`** retry logic for robust AI interactions. | |
| - **`datastore.py` (`DataStore`)**: The database manager. | |
| - Handles all **SQLAlchemy** operations, including saving parsed content and structured extractions to PostgreSQL. | |
| - **`models.py` & `database.py`**: The data foundation. | |
| - Defines the relational schema (`ParsedData`, `LlmExtraction`) and sets up the database engine/session management. | |
| - **`JSONStorage.py` (`JSONStorage`)**: File persistence. | |
| - Asynchronously appends structured results to a local `.jsonl` file using **`aiofiles`**. | |
| --- | |
| [Return to Root Project README](../README.md) | |