Scrapling / scrapling /spiders /engine.py

Commit History

feat(proxy control): Force a proxy at request level at any given point
32daccc

Karim shoair commited on

feat(spiders/requests): Use a reliable method for fingerprint generation
e227110

Karim shoair commited on

feat(spiders): Make graceful shutdown always enabled
5ec9f75

Karim shoair commited on

fix(spiders): handle errors with manual pause during stream mode
47633d8

Karim shoair commited on

fix(spiders): fix a bug with session stats counter
98a7a0d

Karim shoair commited on

fix(spiders): Fix for when the pausing system has large queue
59117ec

Karim shoair commited on

feat(spiders): Add pause/resume system for crawls
1721cbb

Karim shoair commited on

feat(spiders): Change items hook to make it for processing items + add a stat for this
01127e9

Karim shoair commited on

fix(spiders engine): Improve logging
168160d

Karim shoair commited on

feat: add hook to run after collecting an item
48cc7a2

Karim shoair commited on

feat(spiders): Add a stream mode to collect items in realtime
7145885

Karim shoair commited on

feat(spiders): add simple export system for the results
c5ebc9d

Karim shoair commited on

fix: make `retry_blocked_request` take the response as well
e251ff4

Karim shoair commited on

feat(spiders): Add logic to detect blocked requests and retry them
337bce0

Karim shoair commited on

fix(browsers): Close pages that give error
8209e61

Karim shoair commited on

feat(spiders system): a prototype of the new spiders system
f8f7469

Karim shoair commited on