PROJECT_SCAN.md

# CyberScraper-2077 Project Scan & Integration Analysis

## 1. Project Overview
CyberScraper-2077 is an advanced, AI-powered web scraping tool with a futuristic cyberpunk theme. It leverages Large Language Models (LLMs) like OpenAI, Gemini, and Ollama to extract structured data from websites. It features a Streamlit-based UI, supports Tor for .onion sites, and currently uses a custom `PlaywrightScraper` (based on `patchright`) for stealthy scraping.

## 2. Architecture Analysis

### Core Components
*   **Frontend (`main.py`, `app/`):** Built with Streamlit. Handles user interaction, chat interface, and displays scraped data. Manages session state and API keys.
*   **Orchestrator (`src/web_extractor.py`):** The central logic (`WebExtractor` class). It coordinates:
    *   **Fetching:** Calls `PlaywrightScraper` or `TorScraper` to get raw HTML.
    *   **Preprocessing:** Cleans HTML using BeautifulSoup.
    *   **Extraction:** Uses LangChain and LLMs to parse content and answer user queries.
    *   **Agentic Loop:** Has an experimental agentic loop for iterative investigation using `browser_tools`.
*   **Scraping Engine (`src/scrapers/`):**
    *   **`PlaywrightScraper`:** A custom implementation using `patchright` (undetected Playwright). Features:
        *   **Stealth:** Uses persistent contexts and `patchright`'s built-in stealth.
        *   **Concurrency:** `scrape_multiple_pages` handles basic pagination.
        *   **CAPTCHA:** Manual `handle_captcha` method.
    *   **`TorScraper`:** Uses `requests` with SOCKS proxy for .onion sites.
*   **Utilities:**
    *   `src/models.py` / `src/ollama_models.py`: Model management.
    *   `src/utils/`: Error handling, Google Sheets integration.

### Data Flow
1.  User inputs URL/Query in Streamlit.
2.  `StreamlitWebScraperChat` initializes `WebExtractor`.
3.  `WebExtractor` determines if it's a new URL or chat.
4.  If new URL:
    *   `PlaywrightScraper.fetch_content` is called.
    *   Content is preprocessed and cached.
    *   LLM is called with context to answer query or extract data.

## 3. Scrapling Integration Analysis
The `Scrapling` folder contains a new, sophisticated scraping library. Integrating it will enhance CyberScraper-2077 by providing:

*   **Adaptive Parsing:** Robustness against layout changes.
*   **Advanced Fetching:** `StealthyFetcher` and `DynamicFetcher` (wrapping Playwright) for better evasion.
*   **Spider Framework:** A Scrapy-like `Spider` class for efficient, concurrent, and persistent crawling of multiple pages.

### Key Integration Points
*   **Replacement/Augmentation of `PlaywrightScraper`:**
    *   `Scrapling`'s `StealthyFetcher` can replace the manual `patchright` setup in `PlaywrightScraper`.
    *   `Scrapling`'s `Spider` can replace the manual `scrape_multiple_pages` loop, offering better concurrency and state management.
*   **Dependencies:**
    *   `Scrapling` requires `playwright` (implied by imports in `Scrapling/scrapling/engines/_browsers/_controllers.py`).
    *   Current repo uses `patchright`. Integration must ensure `Scrapling` works with the installed environment. `Scrapling`'s `pyproject.toml` lists both `playwright` and `patchright` in optional dependencies, but the code imports `playwright`. I will need to ensure `playwright` is installed or alias it if `patchright` is to be used exclusively (though `patchright` module name is different).
*   **Async Compatibility:**
    *   `Scrapling` supports async fetchers and spiders, compatible with `WebExtractor`'s async nature.

## 4. Proposed Integration Plan

1.  **Dependency Management:**
    *   Install `Scrapling` in editable mode (`pip install -e Scrapling[fetchers]`).
    *   Ensure compatible versions of `playwright`/`patchright`.
2.  **Adapter Implementation (`src/scrapers/scrapling_adapter.py`):**
    *   Create a `ScraplingAdapter` class implementing the interface expected by `WebExtractor` (matching `PlaywrightScraper`'s `fetch_content` signature).
    *   Implement `fetch_content` to use:
        *   `StealthyFetcher` (or `DynamicFetcher`) for single URLs.
        *   A dynamic `Spider` subclass for multi-page/crawl tasks.
3.  **WebExtractor Update:**
    *   Modify `src/web_extractor.py` to instantiate `ScraplingAdapter` instead of `PlaywrightScraper`.
    *   Map existing config (headless, proxy) to `Scrapling` configuration.
4.  **Verification:**
    *   Test single page extraction.
    *   Test multi-page crawling.

## 5. Potential Issues & Risks
*   **Dependency Conflict:** `Scrapling` might pull a different version of `playwright` or `patchright`.
*   **Browser Management:** `Scrapling` manages its own browser instances. I need to ensure it plays nicely with Streamlit's lifecycle (though `WebExtractor` creates a new scraper instance).