Hugging Face's logo

Spaces:
paras9909
/
opticparse-python
Paused

App Files Community
opticparse-python
File size: 4,108 Bytes
bcf46c3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
---
title: "I Built a Free API That Scrapes Any Website Using Plain English — No CSS Selectors"
published: true
description: "How I built Opticparse: a vision-based web scraper that uses AI to extract data from any webpage using natural language, powered by Playwright + Vision AI (Groq LLaMA Vision + Gemini)."
tags: webdev, python, ai, opensource
cover_image: 
---

I've wasted days of my life maintaining CSS selectors.

You know the drill — you write the perfect scraper, it works great for a week, then the site does a frontend redesign, your selectors break, and you spend another afternoon hunting through the DOM again.

So I built **Opticparse** — a completely different approach.

## How It Works

Instead of selectors, Opticparse:
1. Opens a real Chromium browser (via Playwright)
2. Navigates to your URL and waits for JavaScript to load
3. **Screenshots the page**
4. Sends the screenshot to a vision AI model
5. Returns structured JSON based on your natural language query

```bash
curl -X POST https://opticparse.onrender.com/api/vision-scrape \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_url": "https://news.ycombinator.com",
    "extraction_query": "Extract all story titles and upvote counts as a JSON array"
  }'
```

```json
[
  {"title": "Show HN: I built a vision web scraper", "upvotes": 342},
  {"title": "The future of AI agents", "upvotes": 187}
]
```

No selectors. No XPath. No DOM inspection. The AI figures out where everything is from the screenshot.

## The Technical Stack

```
Request
  │
  ▼
FastAPI (Python)
  │
  ▼
Playwright Chromium (headless, stealth mode)
  ├── navigator.webdriver = undefined (anti-bot)
  ├── Route interception (blocks media/fonts → 60% less bandwidth)
  └── Screenshots full page as PNG
  │
  ▼
AI Provider Rotation (tries in order until success):
  1. Groq — llama-3.2-11b-vision (< 1s, free)
  2. GitHub Models — Vision AI (Groq LLaMA Vision + Gemini) (free 150 req/day)
  3. OpenRouter — Vision AI (Groq LLaMA Vision + Gemini) (free credits)
  │
  ▼
5-minute response cache (MD5 hash of url + query)
  │
  ▼
Cleaned JSON response
```

## The Stealth Mode

Cloudflare and other WAFs detect headless browsers by checking `navigator.webdriver`. I added a simple init script to neutralize this:

```python
await context.add_init_script(
    "Object.defineProperty(navigator, 'webdriver', {get: () => undefined});"
)
```

Combined with a real Chrome user agent and `Accept-Language` headers, this bypasses most basic bot detection.

## The AI Provider Rotation

The smartest part: if one AI provider rate-limits, the next one kicks in automatically. Zero downtime, effectively unlimited free capacity.

```python
AI_PROVIDERS = [
    {"name": "Groq", "model": "llama-3.2-11b-vision-preview"},  # Fastest
    {"name": "GitHub Models", "model": "Vision AI (Groq LLaMA Vision + Gemini)"},               # Fallback
    {"name": "OpenRouter", "model": "openai/Vision AI (Groq LLaMA Vision + Gemini)"},           # Fallback
]

for provider in AI_PROVIDERS:
    try:
        result = await call_provider(provider, screenshot, query)
        return result  # Success — stop here
    except RateLimitError:
        continue  # Try next provider
```

## The 5-Minute Cache

Same URL + same query = instant response from cache. Especially useful for dashboard-style apps that poll the same pages repeatedly.

```python
def get_cache(url, query):
    key = hashlib.md5(f"{url}|{query}".encode()).hexdigest()
    entry = _cache.get(key)
    if entry and time.time() - entry["ts"] < 300:
        return entry["data"]
    return None
```

## Try It Free

Available on **opticparse.com** with a free tier — no credit card: [Opticparse on opticparse.com](https://opticparse.com.com/parastejpal987cmyk/api/opticparse-ai-vision-web-scraper)

Full source code: [GitHub](https://github.com/parastejpal987-cmyk/opticparse) (MIT license)

---

*What websites have you tried to scrape that kept breaking? Let me know in the comments — I'll test it against Opticparse.*