File size: 19,349 Bytes
0f8d78f be5aa87 0f8d78f 706c444 3dc1188 0f8d78f be5aa87 0f8d78f e357f95 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f be5aa87 4329d45 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f 4329d45 0f8d78f be5aa87 0f8d78f be5aa87 0f8d78f e357f95 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 | # Scrapling Extract Command Guide
**Web Scraping through the terminal without requiring any programming!**
The `scrapling extract` command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
!!! success "Prerequisites"
1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md), [Dynamic websites](../fetching/dynamic.md), or [Dynamic websites with hard protections](../fetching/stealthy.md).
## What is the Extract Command group?
The extract command is a set of simple terminal tools that:
- **Downloads web pages** and saves their content to files.
- **Converts HTML to readable formats** like Markdown, keeps it as HTML, or just extracts the text content of the page.
- **Supports custom CSS selectors** to extract specific parts of the page.
- **Handles HTTP requests and fetching through browsers**
- **Highly customizable** with custom headers, cookies, proxies, and the rest of the options. Almost all the options available through the code are also accessible through the command line.
## Quick Start
- **Basic Website Download**
Download a website's text content as clean, readable text:
```bash
scrapling extract get "https://example.com" page_content.txt
```
This makes an HTTP GET request and saves the webpage's text content to `page_content.txt`.
- **Save as Different Formats**
Choose your output format by changing the file extension:
```bash
# Convert the HTML content to Markdown, then save it to the file (great for documentation)
scrapling extract get "https://blog.example.com" article.md
# Save the HTML content as it is to the file
scrapling extract get "https://example.com" page.html
# Save a clean version of the text content of the webpage to the file
scrapling extract get "https://example.com" content.txt
# Or use the Docker image with something like this:
docker run -v $(pwd)/output:/output scrapling extract get "https://blog.example.com" /output/article.md
```
- **Extract Specific Content**
All commands can use CSS selectors to extract specific parts of the page through `--css-selector` or `-s` as you will see in the examples below.
## Available Commands
You can display the available commands through `scrapling extract --help` to get the following list:
```bash
Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...
Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.
Options:
--help Show this message and exit.
Commands:
get Perform a GET request and save the content to a file.
post Perform a POST request and save the content to a file.
put Perform a PUT request and save the content to a file.
delete Perform a DELETE request and save the content to a file.
fetch Use DynamicFetcher to fetch content with browser...
stealthy-fetch Use StealthyFetcher to fetch content with advanced...
```
We will go through each command in detail below.
### HTTP Requests
1. **GET Request**
The most common command for downloading website content:
```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Basic download
scrapling extract get "https://news.site.com" news.md
# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60
# Extract only specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
# Send a request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
# Add user agent
scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```
Get the available options for the command with `scrapling extract get --help` as follows:
```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
Perform a GET request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--help Show this message and exit.
```
Note that the options will work in the same way for all other request commands, so no need to repeat them.
2. **Post Request**
```bash
scrapling extract post [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Submit form data
scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"
# Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract post --help` as follows:
```bash
Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
Perform a POST request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-d, --data TEXT Form data to include in the request body (as string, ex: "param1=value1¶m2=value2")
-j, --json TEXT JSON data to include in the request body (as string)
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--help Show this message and exit.
```
3. **Put Request**
```bash
scrapling extract put [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Send data
scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"
# Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract put --help` as follows:
```bash
Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
Perform a PUT request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-d, --data TEXT Form data to include in the request body
-j, --json TEXT JSON data to include in the request body (as string)
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--help Show this message and exit.
```
4. **Delete Request**
```bash
scrapling extract delete [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Send data
scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
```
Get the available options for the command with `scrapling extract delete --help` as follows:
```bash
Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
Perform a DELETE request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--help Show this message and exit.
```
### Browsers fetching
1. **fetch - Handle Dynamic Content**
For websites that load content with dynamic content or have slight protection
```bash
scrapling extract fetch [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Wait for JavaScript to load content and finish network activity
scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
# Wait for specific content to appear
scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
# Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
```
Get the available options for the command with `scrapling extract fetch --help` as follows:
```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
Use DynamicFetcher to fetch content with browser automation.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--network-idle / --no-network-idle Wait for network idle (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--locale TEXT Specify user locale. Defaults to the system default locale.
--real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
```
2. **stealthy-fetch - Bypass Protection**
For websites with anti-bot protection or Cloudflare protection
```bash
scrapling extract stealthy-fetch [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Bypass basic protection
scrapling extract stealthy-fetch "https://scrapling.requestcatcher.com" content.md
# Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use a proxy for anonymity.
scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
```
Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
```bash
Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
Use StealthyFetcher to fetch content with advanced stealth features.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True)
--network-idle / --no-network-idle Wait for network idle (default: False)
--real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
```
## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
- Use **`get`** with simple websites, blogs, or news articles
- Use **`fetch`** with modern web apps, or sites with dynamic content
- Use **`stealthy-fetch`** with protected sites, Cloudflare, or anti-bot systems
## Legal and Ethical Considerations
⚠️ **Important Guidelines:**
- **Check robots.txt**: Visit `https://website.com/robots.txt` to see scraping rules
- **Respect rate limits**: Don't overwhelm servers with requests
- **Terms of Service**: Read and comply with website terms
- **Copyright**: Respect intellectual property rights
- **Privacy**: Be mindful of personal data protection laws
- **Commercial use**: Ensure you have permission for business purposes
---
*Happy scraping! Remember to always respect website policies and comply with all applicable laws and regulations.* |