Spaces:

lenson78
/

Scrapling

Paused

App Files Files Community

Karim shoair commited on Sep 14, 2025

Commit

028ed17

1 Parent(s): 309c3e5

docs: Update all pages related to last changes

Browse files

Files changed (4) hide show

README.md +28 -24
docs/fetching/dynamic.md +4 -1
docs/fetching/stealthy.md +2 -1
docs/index.md +24 -20

README.md CHANGED Viewed

@@ -92,7 +92,7 @@ Built for the modern Web, Scrapling has its own rapid parsing engine and its fet
 - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
 - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
 - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
-- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage.
 ### High-Performance & battle-tested Architecture
 - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -134,7 +134,7 @@ quotes = page.css('.quote .text::text')
 # Advanced stealth mode (Keep the browser open until you finish)
 with StealthySession(headless=True, solve_cloudflare=True) as session:
-    page = session.fetch('https://nopecha.com/demo/cloudflare')
     data = page.css('#padded_content a')
 # Or use one-off request style, it opens the browser for this request, then closes it after finishing
@@ -143,7 +143,7 @@ data = page.css('#padded_content a')
 # Full browser automation (Keep the browser open until you finish)
 with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:
-    page = session.fetch('https://quotes.toscrape.com/')
     data = page.xpath('//span[@class="text"]/text()')  # XPath selector if you prefer it
 # Or use one-off request style, it opens the browser for this request, then closes it after finishing
@@ -187,7 +187,7 @@ from scrapling.parser import Selector
 page = Selector("<html>...</html>")
 ```
-And it works exactly the same way!
 ### Async Session Management Examples
 ```python
@@ -271,29 +271,33 @@ Scrapling requires Python 3.10 or higher:
 pip install scrapling
 ```
-#### Fetchers Setup
-If you are going to use any of the fetchers or their classes, then install browser dependencies with
-```bash
-scrapling install
-```
-This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
 ### Optional Dependencies
-- Install the MCP server feature:
-```bash
-pip install "scrapling[ai]"
-```
-- Install shell features (Web Scraping shell and the `extract` command):
-```bash
-pip install "scrapling[shell]"
-```
-- Install everything:
-```bash
-pip install "scrapling[all]"
-```
 ## Contributing

 - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
 - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
 - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
+- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
 ### High-Performance & battle-tested Architecture
 - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
 # Advanced stealth mode (Keep the browser open until you finish)
 with StealthySession(headless=True, solve_cloudflare=True) as session:
+    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
     data = page.css('#padded_content a')
 # Or use one-off request style, it opens the browser for this request, then closes it after finishing
 # Full browser automation (Keep the browser open until you finish)
 with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:
+    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
     data = page.xpath('//span[@class="text"]/text()')  # XPath selector if you prefer it
 # Or use one-off request style, it opens the browser for this request, then closes it after finishing
 page = Selector("<html>...</html>")
 ```
+And it works precisely the same way!
 ### Async Session Management Examples
 ```python
 pip install scrapling
 ```
+Starting with v0.3.2, this installation only includes the parser engine and its dependencies, without any fetchers.
 ### Optional Dependencies
+1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with
+    ```bash
+    pip install "scrapling[fetchers]"
+    scrapling install
+    ```
+    This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
+2. Extra features:
+   - Install the MCP server feature:
+       ```bash
+       pip install "scrapling[ai]"
+       ```
+   - Install shell features (Web Scraping shell and the `extract` command):
+       ```bash
+       pip install "scrapling[shell]"
+       ```
+   - Install everything:
+       ```bash
+       pip install "scrapling[all]"
+       ```
+   Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
 ## Contributing

docs/fetching/dynamic.md CHANGED Viewed

@@ -62,7 +62,7 @@ DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
 Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
 ## Full list of arguments
-Scrapling provides many options with this fetcher. To make it as simple as possible, we will list the options here and give examples of using most of them.
 |      Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                                | Optional |
 |:-------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
@@ -90,6 +90,9 @@ Scrapling provides many options with this fetcher. To make it as simple as possi
 |       cdp_url       | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.                                                                                                                                                                                                                                                                                                                 |    ✔️    |
 |   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                                                                                                                                                                                                                                   |    ✔️    |
 ## Examples
 It's easier to understand with examples, so let's take a look.

 Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
 ## Full list of arguments
+Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of using most of them.
 |      Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                                | Optional |
 |:-------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
 |       cdp_url       | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.                                                                                                                                                                                                                                                                                                                 |    ✔️    |
 |   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                                                                                                                                                                                                                                   |    ✔️    |
+In the session classes, all these arguments can be set for the session globally. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, and `selector_config`.
 ## Examples
 It's easier to understand with examples, so let's take a look.

docs/fetching/stealthy.md CHANGED Viewed

@@ -15,7 +15,7 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
 > Note: The async version of the `fetch` method is the `async_fetch` method, of course.
 ## Full list of arguments
-Before jumping to [examples](#examples), here's the full list of arguments
 |      Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                                | Optional |
@@ -47,6 +47,7 @@ Before jumping to [examples](#examples), here's the full list of arguments
 |   additional_args   | Additional arguments to be passed to Camoufox as additional settings, and they take higher priority than Scrapling's settings.                                                                                                                                                                                                                                                                                             |    ✔️    |
 |   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                                                                                                                                                                                                                                   |    ✔️    |
 ## Examples
 It's easier to understand with examples, so we will now review most of the arguments individually with examples.

 > Note: The async version of the `fetch` method is the `async_fetch` method, of course.
 ## Full list of arguments
+Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
 |      Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                                | Optional |
 |   additional_args   | Additional arguments to be passed to Camoufox as additional settings, and they take higher priority than Scrapling's settings.                                                                                                                                                                                                                                                                                             |    ✔️    |
 |   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                                                                                                                                                                                                                                   |    ✔️    |
+In the session classes, all these arguments can be set for the session globally. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, and `selector_config`.
 ## Examples
 It's easier to understand with examples, so we will now review most of the arguments individually with examples.

docs/index.md CHANGED Viewed

@@ -114,29 +114,33 @@ Scrapling requires Python 3.10 or higher:
 pip install scrapling
 ```
-#### Fetchers Setup
-If you are going to use any of the fetchers or their session classes, then install browser dependencies with
-```bash
-scrapling install
-```
-This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
 ### Optional Dependencies
-- Install the MCP server feature:
-```bash
-pip install "scrapling[ai]"
-```
-- Install shell features (Web Scraping shell and the `extract` command):
-```bash
-pip install "scrapling[shell]"
-```
-- Install everything:
-```bash
-pip install "scrapling[all]"
-```
 ## How the documentation is organized
 Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).

 pip install scrapling
 ```
+Starting with v0.3.2, this installation only includes the parser engine and its dependencies, without any fetchers.
 ### Optional Dependencies
+1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with
+    ```bash
+    pip install "scrapling[fetchers]"
+    scrapling install
+    ```
+    This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
+2. Extra features:
+   - Install the MCP server feature:
+       ```bash
+       pip install "scrapling[ai]"
+       ```
+   - Install shell features (Web Scraping shell and the `extract` command):
+       ```bash
+       pip install "scrapling[shell]"
+       ```
+   - Install everything:
+       ```bash
+       pip install "scrapling[all]"
+       ```
+   Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
 ## How the documentation is organized
 Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).