Building a Bloom Filter for 1.4 Billion Web Captures
Looking at Hacker News, I came across and article on LWN "An update on the scraper situation" By Jonathan Corbet July 10, 2026 where he says "Maybe it would work better to set up a static central registry of the whole internet's contents that could be scraped by such companies as much as they want without killing small web sites. It could still take a lot of time before we start to see something like this happen though."
A comment https://lwn.net/Articles/1082335/ says:
The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/ The idea is that you contribute resources to their project, they have one scraper that downloads web content in a polite way (respecting rate limits and robots.txt), and then anyone who wants to can make use of the scraped content without having to re-scrape it.
There are some problems with the approach, but I'm generally pretty happy when I see Common Crawl go by in the server logs because I know that's a bunch of unrelated projects that don't need to send us multiple requests.
Then I commented on Hacker News, and the responses encouraged me to explore further. https://news.ycombinator.com/item?id=48864828
After poking around at common crawl, I noticed there was no index to the news dataset. For the regular crawl, Common Crawl provides a "CDX" index for every month. I decided I wanted to try to build a tool that could take a URL for a news article and then get the HTML back from the archive.
I had an idea, from being a long time reader of hackernews, that bloom filters might be a resource. I started trying to do some googling around about bloom filters, but the ai in google Search ended up talking me into building this cdxj index https://huggingface.co/datasets/brian-learns/cdx-cc-news
Then I created a bloom filter with RocksDB https://huggingface.co/buckets/brian-learns/cc-news-cdx-server-storage
Then I created a FastAPI "lookup" endpoint https://huggingface.co/spaces/brian-learns/cc-news-cdx-server
Then I built this command line tool to pull articles from the archive https://github.com/brian-learns/ccnget