README.md

metadata

title: CC Domain Backlink Graph Builder
emoji: 🕸️
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 4.44.0
python_version: '3.12'
app_file: app.py
pinned: false

Common Crawl domain backlink-graph builder

A one-time (per quarterly release) build job that turns the Common Crawl domain hyperlink graph into partitioned Parquet on a Hugging Face dataset repo, so you can query backlinks ("who links to domain X") remotely with DuckDB — no 24 GB local download.

How it works

build.py streams CC's gzipped graph CSVs straight from the CDN (data.commoncrawl.org) via DuckDB + httpfs, in a single pass:

• vertices → vertices_by_domain.parquet (sorted by domain) + vertices_by_id.parquet (sorted by id)
• ranks → ranks_by_id.parquet (harmonic-centrality position per node)
• edges → edges/part=<n>/ partitioned by to_id // part_size (default 1,000,000 → ~122 partitions)

Because edges are bucketed by to_id, a backlink query reads only the one partition whose range contains the target's id — a few MB over HTTP Range, not the whole 22.9 GB edge set.

Deploy as a Space

1. Create a new Gradio Space and add these files (app.py, build.py, requirements.txt, this README.md).
2. Add two Space secrets:
   • HF_TOKEN — a write token for your target dataset repo
   • REPO — e.g. yourname/cc-domain-graph
3. Open the Space, click Build & push. Progress streams in the box.

Notes:

• The Space needs ~30 GB ephemeral disk for the Parquet output (the source is streamed, not stored). If the build runs out of disk, upgrade the Space hardware for the one-time run, or run build.py on Colab/Kaggle instead.
• The build takes roughly 1–2 hours (dominated by streaming + partitioning the 22.9 GB edge file).

Or run it anywhere

pip install -r requirements.txt
HF_TOKEN=hf_xxx python3 build.py --repo yourname/cc-domain-graph
# or build locally without pushing:
python3 build.py --repo x --no-push --out ./graph_out

Refresh

Common Crawl publishes a new graph quarterly. Bump RELEASE in build.py to the newest cc-main-YYYY-... and re-run to keep link rot bounded to months.