README.md

---
title: cc-repackage
emoji: 🐠
colorFrom: indigo
colorTo: red
sdk: gradio
sdk_version: 6.19.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
hf_oauth: true
hf_oauth_scopes:
  # write-repos = write access to the user's existing buckets (to store the range-jobs
  # CSV + output WARC). We do NOT request manage-repos: the target bucket must already
  # exist, so no creation/deletion/settings access is needed.
  - write-repos
  - jobs
hf_oauth_expiration_minutes: 480
---

# 🐠 Common Crawl WARC Repackager

A simple Hugging Face Space to **repackage Common Crawl WARC archives**: pick the
hostnames/domains you care about and the crawls to scan, and get back a new WARC
file in your own HF bucket containing only those records.

All heavy lifting runs on [**HF Jobs**](https://huggingface.co/docs/hub/jobs) using
[`cdx_toolkit`](https://github.com/commoncrawl/cdx_toolkit)'s `repackage` command.
Data is read directly from the
[`commoncrawl` HF bucket](https://huggingface.co/buckets/commoncrawl/commoncrawl)
(columnar index + WARC archives) — **no AWS credentials needed**.

## How it works

1. **Filter target** — enter registered domains (subdomains included) and/or exact hostnames.
2. **Crawls** — select one or more crawls available on the bucket (≤ 10 keeps the index scan cheap).
3. **Output** — give an existing HF **bucket** (`namespace/bucket`) + a path. Datasets are not supported as a target.
4. **Estimate** — a cheap `--no-fetch` job runs a DuckDB query against the columnar
   parquet index, materializes a reproducible `ranges.csv`, and reports the record
   count + total size so you can review the **estimated runtime and cost** (`cpu-upgrade` ≈ $0.03/hr).
5. **Launch** — a second job reads that `ranges.csv` and repackages the matching WARC
   ranges into `<name>.warc.gz` in your bucket.

> HF Jobs require a **Pro / Team / Enterprise** plan.

## Architecture

Two HF Jobs (`python:3.12`, `cpu-upgrade`). Neither mounts the large `commoncrawl`
bucket — both read it directly over its CDN (the fastest backend in the cdxt
benchmarks). Only your (small) target bucket is mounted read-write for the
range-jobs CSV + output WARC.

- **estimate** — enumerates each crawl's columnar-index parquet files via the bucket
  API and reads them over their CDN resolve URLs with DuckDB
  (`https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/…`), writing a
  range-jobs CSV + an `ESTIMATE n_records=… total_bytes=…` summary. (DuckDB can't read
  `hf://buckets` and can't glob over HTTPS, hence the explicit file list — see
  `src/index_query.py`.) The index SQL is vendored, so this job installs only DuckDB +
  `huggingface_hub` — no `cdx_toolkit`.
- **fetch** — `cdxt repackage --target-source csv --csv-path …` with
  `--warc-download-prefix hf://buckets/commoncrawl/commoncrawl --hf-reader cdn`. This
  is the one place `cdx_toolkit` is used, installed inside the job's container.

The filter target is one or more registered domains and/or exact hostnames, optionally
narrowed to records whose **primary detected language** (the columnar index's
`content_languages`, ISO-639-3) is one of a selected set.

You can cancel a running estimate or repackage job at any time with the **Cancel job** button.

## Local development & verification

Run the whole flow locally with **podman** instead of HF Jobs (no HF/AWS, no cost):

```bash
uv pip install -r requirements.txt
./scripts/verify_local.sh        # builds fixtures + a job image, runs estimate+fetch
```

Or drive the UI manually:

```bash
export CC_REPACKAGE_EXECUTOR=podman
export CC_PODMAN_CC_DIR=tests/fixtures/cc        # toy index + WARCs mounted at /cc
export CC_PODMAN_OUT_DIR=/tmp/cc-repackage-out   # mounted at /out
export CC_PODMAN_IMAGE=cc-repackage-job:dev      # prebuilt image with cdxt (fetch job)
export CC_PODMAN_SKIP_PIP=1
python app.py
```

Config lives in `src/config.py`; the executor is selected by `CC_REPACKAGE_EXECUTOR`
(`hfjobs` | `podman`).