Scrapling / scrapling /engines /_browsers /_controllers.py

Commit History

fix: update code and docstrings to remove the old google referer logic
a28879b

Karim shoair commited on

style: Fix all mypy errors and add type hints to untyped function bodies
31c2447

Karim shoair commited on

feat(browsers): Add option to block requests to specific domains
47dd985

Karim shoair commited on

feat(proxy control): Force a proxy at request level at any given point
32daccc

Karim shoair commited on

feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
ed96cdc

Karim shoair commited on

fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
8ff23b3

Karim shoair commited on

feat(browsers): Add option to retry tabs that gives errors
024cbba

Karim shoair commited on

fix(browsers): Close pages that give error
8209e61

Karim shoair commited on

fix(browsers): solve a bug with setting referer on the request level
c7e573a

Karim shoair commited on

refactor: internal API changes to be easily used as indicators for spiders
1803348

Karim shoair commited on

docs: change docstring style for some methods
daaad4e

Karim shoair commited on

build: update deps
7e1bee4

Karim shoair commited on

docs: shorten docstrings for some functions
1812d2b

Karim shoair commited on

refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
ee2299e

Karim shoair commited on

docs: add docs for `timezone_id` argument
17e25c2

Karim shoair commited on

refactor(fetchers): rename internal api
a17a010

Karim shoair commited on

refactor(browser fetchers): Make all the type hints dynamic + Faster validation
c908f33

Karim shoair commited on

fix: added `raw_response` to separate collected response from rendered response
2784aa7

Karim shoair commited on

refactor(validation): All browser based fetchers are ~8-15% faster now in most cases
8bf21c0

Karim shoair commited on

feat(DynamicSession): New option to add extra browser flags
b6969b2

Karim shoair commited on

refactor(fetchers): Less duplicated code and better handling for unstable websites
6f2d7b6

Karim shoair commited on

fix: Addressing the issue of collecting response after `page_action` in #100
d765ef3

Karim shoair commited on

feat(Dynamic fetcher): Add `additional_args` option to customize browser as in StealthyFetcher
0de8025

Karim shoair commited on

feat(fetchers): Add argument to control the user data directory
c61a805

Karim shoair commited on

fix(fetchers): More accurate collection of the final response
0de3dd1

Karim shoair commited on

refactor: Making all the codebase acceptable by PyRight
e5ecf76

Karim shoair commited on

style: Removing dead code/docstrings
f58c872

Karim shoair commited on

perf: Improving all browser-based fetches by ~15%
da46469

Karim shoair commited on

feat(DynamicFetcher): Replace `rebrowser` with `patchright`
1fe694e

Karim shoair commited on

perf(session): improve page management and speed up execution
81b00b6

Abdullah commited on

fix: Stop removing browser default page
7666ce3

Karim shoair commited on

refactor: Make all fetchers as an optional dependency group
a85d2c8

Karim shoair commited on

feat(browser fetchers): A new option to control page JS
66fd35f

Karim shoair commited on

feat: Validate all fetch-level parameters
3f819b6

Karim shoair commited on

style: Using keywords for validation
cdaab9d

Karim shoair commited on

docs: Update all docstring according to the new changes
60d0c55

Karim shoair commited on

refactor/feat(browser fetchers): make it possible to have a configuration per page in sessions
c181b7d

Karim shoair commited on

refactor/fix(browser fetchers): Solving a logic bug in page rotation
98f7c8f

Karim shoair commited on

perf(DynamicSession): Close persistent context default page
4e6a590

Karim shoair commited on

feat(DynamicFetcher): add the ability to add `init_script` to context (#56)
03de577

Karim shoair commited on

fix(dynamicSession): issue with headless
b9b8563

Karim shoair commited on

style: add flags for tests coverage
9b40891

Karim shoair commited on

refactor(dynamic Fetcher): small optimization
70ac9c0

Karim shoair commited on

fix(dynamicFetcher): Fix a bug with `real_chrome` argument
fa8564d

Karim shoair commited on

style: A lot of type hints correction
916182a

Karim shoair commited on

refactor: huge change, many features/class got a better naming
8e67a4c

Karim shoair commited on

refactor(DynamicSession): Optimization + Removing `max_pages` from sync version
e39bf62

Karim shoair commited on

feat(DynamicFetcher): Use persistent context by default for better stealth
78b9e69

Karim shoair commited on

fix(controllers): Fix issue with async adding stealth scripts
fd88ce6

Karim shoair commited on

fix(controllers): Fix issue with adding stealth scripts
9048b09

Karim shoair commited on