Krixter commited on
Commit
62c35b6
Β·
verified Β·
1 Parent(s): 396b242

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +32 -7
README.md CHANGED
@@ -1,13 +1,38 @@
1
  ---
2
- title: EXT
3
- emoji: πŸ”₯
4
- colorFrom: purple
5
- colorTo: pink
6
  sdk: gradio
7
- sdk_version: 6.18.0
8
- python_version: '3.13'
9
  app_file: app.py
10
  pinned: false
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: Novel Scraper
3
+ emoji: πŸ“š
4
+ colorFrom: indigo
5
+ colorTo: purple
6
  sdk: gradio
7
+ sdk_version: 4.44.1
 
8
  app_file: app.py
9
  pinned: false
10
  ---
11
 
12
+ # Novel Scraper
13
+
14
+ Scrapes **every novel's title + description** from:
15
+ - **novelbin.com** β€” HTML scraping with BeautifulSoup
16
+ - **webnovel.com** β€” Public JSON API (no JS rendering needed)
17
+
18
+ ## Output files (in `output/` folder)
19
+ | File | Contents |
20
+ |------|----------|
21
+ | `novelbin_novels.csv` | novelbin data as CSV |
22
+ | `novelbin_novels.jsonl` | novelbin data as JSONL |
23
+ | `webnovel_novels.csv` | webnovel data as CSV |
24
+ | `webnovel_novels.jsonl` | webnovel data as JSONL |
25
+ | `scraper.log` | Full log of the run |
26
+ | `progress.json` | Resumable checkpoint |
27
+
28
+ ## Features
29
+ - βœ… **Resumable** β€” restart the Space and it picks up where it left off
30
+ - βœ… **Rate-limited** β€” polite delays to avoid getting blocked
31
+ - βœ… **Retry logic** β€” 5 retries with exponential backoff per failed request
32
+ - βœ… **Dual output** β€” CSV for spreadsheets, JSONL for pipelines
33
+ - βœ… **Live log** β€” watch progress in real-time in the UI
34
+
35
+ ## Usage
36
+ 1. Click **β–Ά Start Scraper**
37
+ 2. Click **πŸ”„ Refresh Status** any time to see the log
38
+ 3. When done, download files from the **Download Output Files** section