README.md

---
title: CC Domain Backlink Graph Builder
emoji: 🕸️
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 4.44.0
python_version: "3.12"
app_file: app.py
pinned: false
---

# Common Crawl domain backlink-graph builder

A one-time (per quarterly release) build job that turns the Common Crawl
**domain hyperlink graph** into partitioned Parquet on a Hugging Face dataset
repo, so you can query *backlinks* ("who links to domain X") remotely with
DuckDB — no 24 GB local download.

## How it works

`build.py` streams CC's gzipped graph CSVs straight from the CDN
(`data.commoncrawl.org`) via DuckDB + httpfs, in a single pass:

- **vertices** → `vertices_by_domain.parquet` (sorted by domain) +
  `vertices_by_id.parquet` (sorted by id)
- **ranks** → `ranks_by_id.parquet` (harmonic-centrality position per node)
- **edges** → `edges/part=<n>/` partitioned by `to_id // part_size`
  (default 1,000,000 → ~122 partitions)

Because edges are bucketed by `to_id`, a backlink query reads only the one
partition whose range contains the target's id — a few MB over HTTP Range,
not the whole 22.9 GB edge set.

## Deploy as a Space

1. Create a new **Gradio Space** and add these files (`app.py`, `build.py`,
   `requirements.txt`, this `README.md`).
2. Add two **Space secrets**:
   - `HF_TOKEN` — a write token for your target dataset repo
   - `REPO` — e.g. `yourname/cc-domain-graph`
3. Open the Space, click **Build & push**. Progress streams in the box.

Notes:
- The Space needs ~30 GB ephemeral disk for the Parquet output (the *source*
  is streamed, not stored). If the build runs out of disk, upgrade the Space
  hardware for the one-time run, or run `build.py` on Colab/Kaggle instead.
- The build takes roughly 1–2 hours (dominated by streaming + partitioning the
  22.9 GB edge file).

## Or run it anywhere

```bash
pip install -r requirements.txt
HF_TOKEN=hf_xxx python3 build.py --repo yourname/cc-domain-graph
# or build locally without pushing:
python3 build.py --repo x --no-push --out ./graph_out
```

## Refresh

Common Crawl publishes a new graph quarterly. Bump `RELEASE` in `build.py`
to the newest `cc-main-YYYY-...` and re-run to keep link rot bounded to months.