README.md

metadata

title: cc-repackage
emoji: 🐠
colorFrom: indigo
colorTo: red
sdk: gradio
sdk_version: 6.19.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
hf_oauth: true
hf_oauth_scopes:
  - write-repos
  - jobs
hf_oauth_expiration_minutes: 480

🐠 Common Crawl WARC Repackager

A simple Hugging Face Space to repackage Common Crawl WARC archives: pick the hostnames/domains you care about and the crawls to scan, and get back a new WARC file in your own HF bucket containing only those records.

All heavy lifting runs on HF Jobs using cdx_toolkit's repackage command. Data is read directly from the commoncrawl HF bucket (columnar index + WARC archives) — no AWS credentials needed.

How it works

1. Filter target — enter registered domains (subdomains included) and/or exact hostnames.
2. Crawls — select one or more crawls available on the bucket (≤ 10 keeps the index scan cheap).
3. Output — give an existing HF bucket (namespace/bucket) + a path. Datasets are not supported as a target.
4. Estimate — a cheap --no-fetch job runs a DuckDB query against the columnar parquet index, materializes a reproducible ranges.csv, and reports the record count + total size so you can review the estimated runtime and cost (cpu-upgrade ≈ $0.03/hr).
5. Launch — a second job reads that ranges.csv and repackages the matching WARC ranges into <name>.warc.gz in your bucket.

HF Jobs require a Pro / Team / Enterprise plan.

Architecture

Two HF Jobs (python:3.12, cpu-upgrade). Neither mounts the large commoncrawl bucket — both read it directly over its CDN (the fastest backend in the cdxt benchmarks). Only your (small) target bucket is mounted read-write for the range-jobs CSV + output WARC.

• estimate — enumerates each crawl's columnar-index parquet files via the bucket API and reads them over their CDN resolve URLs with DuckDB (https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/…), writing a range-jobs CSV + an ESTIMATE n_records=… total_bytes=… summary. (DuckDB can't read hf://buckets and can't glob over HTTPS, hence the explicit file list — see src/index_query.py.) The index SQL is vendored, so this job installs only DuckDB + huggingface_hub — no cdx_toolkit.
• fetch — cdxt repackage --target-source csv --csv-path … with --warc-download-prefix hf://buckets/commoncrawl/commoncrawl --hf-reader cdn. This is the one place cdx_toolkit is used, installed inside the job's container.

The filter target is one or more registered domains and/or exact hostnames, optionally narrowed to records whose primary detected language (the columnar index's content_languages, ISO-639-3) is one of a selected set.

You can cancel a running estimate or repackage job at any time with the Cancel job button.

Local development & verification

Run the whole flow locally with podman instead of HF Jobs (no HF/AWS, no cost):

uv pip install -r requirements.txt
./scripts/verify_local.sh        # builds fixtures + a job image, runs estimate+fetch

Or drive the UI manually:

export CC_REPACKAGE_EXECUTOR=podman
export CC_PODMAN_CC_DIR=tests/fixtures/cc        # toy index + WARCs mounted at /cc
export CC_PODMAN_OUT_DIR=/tmp/cc-repackage-out   # mounted at /out
export CC_PODMAN_IMAGE=cc-repackage-job:dev      # prebuilt image with cdxt (fetch job)
export CC_PODMAN_SKIP_PIP=1
python app.py

Config lives in src/config.py; the executor is selected by CC_REPACKAGE_EXECUTOR (hfjobs | podman).