data/github/web_wizard/README.md

metadata

project_name: web_wizard
github_url: https://github.com/neural-arun/web_wizard
language: HTML
stars: 0
topics:
  - None
updated_at: 2026-04-08T06:10:58.000Z

web_wizard

GitHub Repository: https://github.com/neural-arun/web_wizard
Primary Language: HTML | Stars: 0 | Forks: 0
Description: No description provided.

---

Playwright Web Automation & Scraping Curriculum

Welcome to my learning repository for advanced web automation, scraping, and testing using Playwright for Python. This repository tracks my progress as I work through a comprehensive, progressive curriculum, from absolute basics to distributed, AI-integrated crawler systems.

🚀 The Tech Stack

This project focuses on modern, resilient Python automation:

• Core Automation: Playwright for Python (playwright, playwright-python)
• Testing & CI: pytest, pytest-playwright, GitHub Actions, Docker
• Data Processing: pandas
• Data Storage: PostgreSQL (via sqlalchemy, psycopg2), Redis
• Message Queues & Orchestration: Celery, RabbitMQ
• AI & Vector Integration: ChromaDB / Pinecone, Langchain (for Agent tool building)

🛠️ Environment Setup

To run the scripts and projects in this repository, you'll need to set up the Python virtual environment and install the required browser binaries.

1. Clone the repository:

   git clone https://github.com/neural-arun/web_wizard.git
   cd web_wizard
   

2. Create and activate the virtual environment (Windows):

   python -m venv venv
   .\venv\Scripts\Activate.ps1
   

3. Install Python dependencies:

   pip install -r requirements.txt
   

4. Install Playwright Browsers (Chromium, Firefox, WebKit):

   playwright install
   

📚 Curriculum Map

The learning journey spans from foundational concepts to production-ready, distributed architectures.

Part 1: Foundations & Core API

• Module A: Foundations (Python async, HTTP, DOM, DevTools)
• Module B: Playwright Core API (Locators, navigation, tracing, emulation)
• Module C: Network & Data Extraction (Interception, JSON parsing, API mocking)
• Module D: Automation Reliability & Debugging (Flaky tests, fallbacks, Playwright Inspector)

Part 2: Advanced Scraping & Scale

• Module E: Stealth & Anti-bot (Context isolation, proxies, rate limits)
• Module F: Performance & Concurrency (Asyncio pooling, memory optimization)
• Module G: Integration with Data Stack (Postgres, Pandas, Export formats)
• Module H: Testing, CI, Containerization (Docker, Pytest fixtures)

Part 3: Production & AI Integration

• Module I: Advanced Targets (Infinite scroll, SPAs, multi-step auth)
• Module J: Anti-fraud & Resilience (Ban recovery, circuit breakers)
• Module K: Production Orchestration (Queueing with Celery/RabbitMQ, Autoscaling)
• Module L: Playwright as an Agent Tool (Building sandbox tools for LLM integration)

🏗️ Hands-on Projects

1. Project 1: Single-page scraper (CSV Export)
2. Project 2: Login-protected scraper
3. Project 3: Infinite-scroll scraper & deduplication
4. Project 4: XHR-intercept scraper
5. Project 5: Multi-user crawler writing to Postgres
6. Project 6: Playwright-agent connector for LLM/RAG workflows
7. Capstone: Distributed, Dockerized crawler with queue-based orchestration and vector DB pipeline.

See the full detailed syllabus in curriculum.md.