README.md

---
title: README
emoji: 🕸️
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
---

# Crawlora

Crawlora is a hosted web-data API: structured, normalized JSON from the public web —
search/SERP, e-commerce, social, finance, app stores, media, and reviews — instead of
HTML to parse. We also publish the open datasets behind our own data-journalism studies,
CC BY 4.0, hosted here for anyone to load with `datasets`, `pandas`, or DuckDB.

- **Website:** [crawlora.net](https://crawlora.net?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile)
- **API docs:** [crawlora.net/docs](https://crawlora.net/docs?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile)
- **Playground:** [crawlora.net/playground](https://crawlora.net/playground?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile)
- **Pricing:** [crawlora.net/pricing](https://crawlora.net/pricing?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile) — free tier, 2,000 credits/mo, no card
- **All open datasets & studies:** [crawlora.net/datasets](https://crawlora.net/datasets?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile)
- **Source code:** [github.com/Crawlora-org](https://github.com/Crawlora-org)

## Featured open datasets

| Dataset | Study |
| --- | --- |
| [dead-web-index](https://huggingface.co/datasets/crawlora-net/dead-web-index) | [Dead-Web Index](https://crawlora.net/dead-web-index?utm_source=huggingface&utm_medium=referral&utm_campaign=dead-web-index) — is a domain alive, redirecting, blocked, or dead? |
| [anti-bot-adoption-index](https://huggingface.co/datasets/crawlora-net/anti-bot-adoption-index) | [Anti-Bot Adoption Index](https://crawlora.net/anti-bot-index?utm_source=huggingface&utm_medium=referral&utm_campaign=anti-bot-index) — which WAF/anti-bot vendor protects the top web |
| [ai-crawler-blocking-index](https://huggingface.co/datasets/crawlora-net/ai-crawler-blocking-index) | [AI-Crawler Blocking Index](https://crawlora.net/ai-crawler-index?utm_source=huggingface&utm_medium=referral&utm_campaign=ai-crawler-index) — who blocks GPTBot, ClaudeBot, CCBot |
| [steam-catalog](https://huggingface.co/datasets/crawlora-net/steam-catalog) | [State of Steam 2026](https://crawlora.net/blog/state-of-steam-2026?utm_source=huggingface&utm_medium=referral&utm_campaign=state-of-steam-2026) — top 10,000 Steam games |
| [song-length-index](https://huggingface.co/datasets/crawlora-net/song-length-index) | [Are Songs Really Getting Shorter?](https://crawlora.net/blog/song-length-2026?utm_source=huggingface&utm_medium=referral&utm_campaign=song-length) — 6,618 Hot 100 hits, 1959–2025 |

[Browse all 11+ datasets on this org](https://huggingface.co/crawlora-net) or the full catalog at
[crawlora.net/datasets](https://crawlora.net/datasets?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile).

## For developers

The same public API backs SDKs (Go, TypeScript, Python, Ruby, Java, PHP), a hosted
[MCP server](https://crawlora.net/mcp?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface-org-profile)
with 778 tools, Agent Skills, and n8n/Zapier/Make integrations — see the
[GitHub org](https://github.com/Crawlora-org) for all of them.