dev.altai / src /crawler.py

Commit History

Enable Verbose Logging: Track all WAF interactions, stealth steps, and navigation events for debugging
059751d

prince1604 commited on

Update Stealth: Use playwright-stealth library and Frame-based checkbox solver for improved WAF evasion
dd23225

prince1604 commited on

Update Stealth Strategy: Hardcode Chrome UA, mock plugins/permissions/languages, and randomize mouse movements
d0c6b9b

prince1604 commited on

Update Crawler: Add stealth args and challenge detection for Cloudflare
503011f

prince1604 commited on

Enhance API stability: Add KeepAlive, increase timeout, and optimize crawler threads
f2e524e

prince1604 commited on

Fix URL normalization: strip trailing slash from start_url
7e72c45

prince1604 commited on

Fix Amazon crawling: Add Start URL Retry logic and more block keywords
94af2f2

prince1604 commited on

Fix crawl evasion: Add WAF block detection to Tier 1 fetch
d92f150

prince1604 commited on

Fix syntax errors: Restore init, fix indentation, remove invalid arg
747f2d2

prince1604 commited on

Ultra-fast crawler update: Caching, SSL disable, and Thread tuning
6d07192

prince1604 commited on

Optimize crawler speed: BFS + Reduced delays
d3e6b5e

prince1604 commited on

Fix syntax error in crawler
e824b70

prince1604 commited on

Optimize memory usage to prevent crashes
dc99297

prince1604 commited on

Fix crawling for eminentcoders.com
53a61cf

prince1604 commited on

Initial commit for deployment
b458f3d

prince1604 commited on