I checked what make loaddata actually pulls before pulling it. Your bucket reconciles to the byte against the LOG it ships.

hf buckets ls on cc-news-cdx-server-storage gives 1,123 objects, 1,111 of them .sst, 75,532,239,021 bytes. Take out the 12 non-sst files and the SSTs come to 75,516,444,342.

Your own LOG, same bucket. At open on 08-01 the picker prints files[3 0 0 0 61 159 885], so 1,108 live files, and the delete scheduler prints total_size 75,066,314,049. Three flushes land after that: 5851 at 187,953,067, 5854 at 194,745,281, 5856 at 67,431,945.

1,108 + 3 = 1,111. And 75,066,314,049 + 450,130,293 = 75,516,444,342.

Same number both ways. So the sync is not shipping dead SSTs, which is the thing I went looking for and did not find.

What it does ship is 15,794,679 bytes the DB no longer owns. CURRENT reads MANIFEST-005845, so MANIFEST-000005 is a dead 1.7 MB, OPTIONS-000007 with it, LOG.old at 13.6 MB, and 005839.log, which your LOG records deleting at 16:46:03.895773 that same morning. It is still in the bucket, dated 07-23 12:55:59.

0.02%, so not worth a fix on its own. It is worth knowing why it is there. hf sync defaults to --no-delete, so the bucket is additive, and L6 is 885 files and 55.77 GiB. The first compaction that retires part of that leaves the retired files behind, and the puller downloads them.

The other thing the LOG says is about the state it froze in. The DB closed 41 seconds into a compaction that never landed. Job 3 logs compaction_finished, 4 output files, 206,687,332 bytes to L3, and the next lines delete 5848, 5849, 5852 and 5857. lsm_state still reads [6, 0, 0, 0, 61, 159, 885] with L3 empty, and the inputs 5818 and 5770 are still in the bucket. So what ships is pre-compaction: six sorted runs in L0, max score 1.50.

Your compose handles that. :ro mount plus ROCKS_SHADOW means a puller never writes into it.

The README's Local Setup path is the one I would check. Does uv run cdx-rocks "$ROCKS_READONLY" "$ROCKS_SHADOW" open read-only too, or does the first local run re-do that compaction on someone else's disk?

The --delete fix lands on the bucket, but the August refresh breaks twice before it gets there. I ran both.

First is make loaddata. rocksdb_index is a directory prerequisite, so make calls it satisfied the moment it exists. Second run, a month later, new data in the bucket:

make: Nothing to be done for 'loaddata'.

Exit 0. No sync, no output, and the server comes back up on the July index.

Second is the guard. You said setup_shadow runs if the shadow directory is empty. The code checks is_dir():

if not Path(ROCKS_SHADOW).is_dir():
    setup_shadow(Path(ROCKS_READONLY), Path(ROCKS_SHADOW))

Your setup_shadow is idempotent, it unlinks and relinks every entry. The guard is what never lets it run twice. And ROCKS_SHADOW is /app/rocksdb, which lives in the container layer and not a volume, so it survives restart: unless-stopped.

I ran your setup_shadow and your read-only open against a real rocksdict DB. July, 500 keys, sync, boot. Then indexed 500 more, synced again with the --no-delete default, restarted:

boot 1   setup_shadow ran: True    9 entries   OPEN OK, 500 keys
boot 2   setup_shadow ran: False   9 entries   OPEN FAILED
         IO error: No such file or directory: MANIFEST-000013
boot 3   setup_shadow ran: True   14 entries   OPEN OK, 1000 keys

boot 3 is that same August data with the shadow wiped first.

Why it is fatal and not merely stale: CURRENT keeps its filename, so the additive sync overwrites it through the symlink, and it now names MANIFEST-000013. Every other August file arrived under a new number with no link pointing at it. So --no-delete hides the staleness on every file except the one the open reads first.

A CURRENT content check instead of is_dir() refreshes it. Or put the shadow somewhere the layer cannot keep, since rebuilding it is only symlinks.

Does the ccnget side pin an index version, or does it trust whatever /lookup happens to have mounted?

Then the failure ccnget cannot see is not a 404. It is a 200.

The catalog is a separate artifact from the index. ID_TO_PATH is built positionally at boot, enumerate(text_stream, start=1), and the RocksDB value stores only that position: struct.unpack("!HQI", value). So a capture resolves through whatever catalog the process happened to load.

I ran your app/main.py against a real rocksdict with two captures, ids 2 and 5, and booted it twice with different catalogs. Fresh shadow both times, so this is not the symlink thing:

CONTROL  catalog 5 paths (matches index)
  ts=20260601120000  offset=111111  warc_path=crawl-data/.../00002.warc.gz
  ts=20260731193544  offset=999999  warc_path=crawl-data/.../00005.warc.gz

SKEW     catalog 3 paths (one refresh behind)
  ts=20260601120000  offset=111111  warc_path=crawl-data/.../00002.warc.gz
  ts=20260731193544  offset=999999  warc_path=PATH_NOT_FOUND

Both are HTTP 200. Both validate against LookupResponse. The 404 only fires on if not captures, and captures is non-empty, so PATH_NOT_FOUND ships to the client as a warc_path with a real offset and length beside it.

What produces the skew is the same Makefile line as before, on its other leg:

loaddata: rocksdb_index all_warc_paths.txt.zst

Two existence-only targets. So the natural fix for the staleness, delete the index and re-sync, refreshes exactly one of them:

$ rm -rf rocksdb_index && make -n loaddata
uvx hf sync hf://buckets/.../cc-news-cdx-server-storage rocksdb_index

New index, July catalog. Deleting both gets both.

The version token you say ccnget lacks is already computed. Your lifespan logs Catalog loaded (51,101 paths) and then never returns it. Put that count in the response and a client can pin it.

Worth knowing what that number is doing, though. I pulled all_warc_paths.txt.zst: 51,101 paths, and !H addresses 65,535. CC-NEWS added 459/month over the last twelve and 366/month over the last six, so the id space runs out in 2.6 to 3.3 years.

Does the builder pack ids from the same catalog file the server loads, or from a listing it takes at build time?

Your catalog is append-only, and I checked instead of taking your word. It bounds my last message.

all_warc_paths.txt.zst at rev fabe94d ("corrected sort order", 07-19) is 50,958 lines. At c603b13 (08-01) it is 51,101. The first 50,958 lines are byte-identical, md5 0d9c6c5063fa5b72c6c2069bec6067b0 both times. The July refresh appended 143 and moved nothing.

So a stale catalog cannot remap an old id onto a wrong path. It can only run off its own end. PATH_NOT_FOUND is the only wrong answer available, never a real path pointing at the wrong file. That is a stronger property than I gave you credit for.

I checked the join too, since both builders key name_to_id on os.path.basename while the catalog stores full paths. All 51,101 basenames are distinct, so the dict loses nothing. Clean.

The 65,535 ceiling is worse than needing a new format, though. It is not a crash.

struct.error subclasses Exception, and the pack sits inside except Exception: continue in both rocksdb_from_cdxj.py and rocksdb_monthly.py. I ran your loop body on 5 real cdxj lines from cc-news_2016_08, changing only the id:

id=51,101   5 records written   "Update complete: Added 5 records."
id=65,535   5 records written   "Update complete: Added 5 records."
id=65,536   0 records written   "No new records added (file empty or no matches)."
            5 swallowed: 'H' format requires 0 <= number <= 65535

The counter only counts successes. So the first month past the ceiling reports as an empty input file, not an overflow. Every record of every WARC above catalog line 65,535 disappears quietly, and the operator reads a sentence that means "nothing to do here."

On which field is oversized, I measured, and it is not the one you named.

8,141,383 records, the oldest month and the newest:

Neither month's offset ever reaches 2^30. The 2016 max stops 10,776 bytes short of it. Both length maxima stop within about 500 bytes of 1 MiB. Ten years apart, so those are producer-side caps, not my sample.

length wastes 12 bits. offset wastes 34 and nobody has flagged it. The field that breaks is the only one too small. !III is 12 bytes against your 14 and retires the ceiling.

Timing comes out of your own catalog. 51,101 of 65,535 is 78.0%, so 14,434 left. The last six months run 353 to 394, mean 366, so about 40 months. But your rate is not flat. 2025-12 alone was 616, and your all-time high is 745 in 2023-03. At that rate it is 19 months.

Cheap guard in the meantime: refuse to boot if the catalog exceeds 65,535 lines, and log len(name_to_id) against the enumerate count. Does anything downstream compare records-written to cdxj lines-read? If not, the month it happens just looks like a quiet month.

Your value is 14 bytes on disk, not 16. The ! turns padding off.

struct.calcsize("!HQI") is 14. The ! prefix selects standard sizes with no alignment, so there is nothing to pad. VALUE_SIZE at app/main.py:37 already evaluates to 14 today. The comment two lines above it says 16.

Your own two files disagree, and the builder is the one that is right:

rocksdb_from_cdxj.py:112   # Pack into our universal 14-byte footprint:
app/main.py:31             # Each value is a 16-byte big-endian struct:
app/main.py:35             # Total: 2 + 8 + 4 = 14 bytes (struct alignment pads to 16)

Nothing is broken by it. VALUE_SIZE appears exactly once in the whole repo, at its own definition, and the unpack at :143 reads the format rather than the size. So it is a stale comment, not a data bug. It is just the comment the proposal was reasoning from.

Which makes !IQI a real +2 bytes, not a free move into space you had already paid for.

You are still right that it does not change the db, so I built it instead of arguing the arithmetic. cc-news_2016_08, 299,609 records after your inline dedup, your options: raw_mode, zstd, universal compaction, 512MB write buffer. Compacted each one, then summed SST bytes.

!IQI adds 599,218 raw bytes and 56,378 of them reach disk. 9.4% pass-through. The two bytes are the high half of a 16-bit id widened to 32, so they are zeros, and zstd charges almost nothing for them.

So the size question answers itself, and it answers itself the same way for every option. Mean key in that month is 105.5 bytes against a 14-byte value, so the value is 11.7% of the raw kv bytes. Nothing you can do inside three integer fields moves this file.

That leaves headroom as the only axis that decides it. And on headroom the winner is also the smaller one.

The offset field is why. Max offset in 2016_08 is 1,073,731,048, which stops 10,776 bytes short of 2^30. 2026_07 lands in the same place. Ten years apart, both stopping just under 1 GiB, is a producer-side cap on WARC size and not an artifact of my sample. uint32 leaves 4x over a ceiling that has held for the life of the dataset. uint64 is buying range against something nobody has been able to cross.

So !III retires the 65,535 id ceiling, keeps 4x on offset, and gives back 1.05%. !IQI retires the same ceiling and costs 0.45%. Both are fine. Only one of them is smaller than what you ship now.

One thing makes either change cheaper than I expected. I grepped ccnget for struct and got zero hits. The client reads offset and length out of your JSON already unpacked, so the 14 bytes never leave the server and nobody downstream is parsing them.

Which puts the whole cost in the rebuild. What was the wall clock on the last full cdxj-to-rocksdb pass?

Your port worry has an easy answer, and the index-version worry has a date on it.

Port first. The Space is already on 7860 and that is the HF default, so nothing collides. The public URL has no port at all: https://brian-learns-cc-news-cdx-server.hf.space/lookup answers and / redirects to /docs. The port only exists inside the container and on a local docker run. Moving off it means adding app_port: to the README frontmatter, which is one more pair of things to keep in sync. I would leave it.

Before the rest I pulled your index end to end. Asked the Space for bbc.com/news, took the row it gave back:

crawl-data/CC-NEWS/2018/04/CC-NEWS-20180408051913-00192.warc.gz  offset 677849923  length 56405

Range-fetched exactly those bytes from data.commoncrawl.org and gunzipped:

WARC-Target-URI: http://www.bbc.com/news
WARC-Date: 2018-04-08T09:16:22Z

Matching the 20180408091622 you returned. The id-to-path mapping is correct.

Now the id field, because it is the one with a deadline. all_warc_paths.txt.zst is 51,101 lines, 78.0% of 65,535. CC-NEWS adds 365.5 files a month over the trailing six, 459 over the trailing twelve and twenty-four. So uint16 runs out in 31 to 40 months, somewhere in 2029.

Which sharpens the argument I made badly last time. !IQI and !III both retire the field that has a date. The extra bytes !IQI spends go to the offset, and offset has no date, since the max in both 2016_08 and 2026_07 stops just under 2^30 and uint32 already carries 4x. Same fix on the field that matters, either way, so !IQI is a defensible call.

One thing about the catalog itself that I did not expect. It is not append-only. All 51,101 lines are exactly sorted lexicographically and the embedded timestamps are monotonic, so it is regenerated by sorting. A backfilled WARC in an old month is an insert, not an append. That 2018 file is id 2339, so an insert there shifts 48,762 ids, 95.4% of them. And nothing binds the two artifacts: the catalog is COPYd into the image at build time, the rocksdb mounts at /data, and they version independently.

I checked how loudly that fails rather than guessing. Refetched the same offset and length under id 2340, the next path over:

gzip: Not a gzipped file (b')/')

So drift fails at decompression, not with a plausible wrong article. That is the good outcome. It is still undiagnosable from the client, which is the same shape as the thing you said you did not want ccnget to carry.

Which is where the two indexes come in, and it is bigger than versions.

30 of 75 is 40%. By your own cdxj bytes, 115.28 GB across 120 months, a recency-ordered 40% starts at 2022-10. 46 months kept, 74 dropped, and 50.5% of your WARC ids sit before that line.

I measured what the client sees using your own timestamp seek. Per URL, exact=true limit=1, then exact=true at=20221001 limit=1.

Of 17 news front pages, 6 have an exact-SURT capture. Five of those six have nothing at or after 2022-10-01:

www.bbc.com/news                first 20180408091622   after cutoff: none
www.reuters.com/world           first 20210722030958   after cutoff: none
www.cnn.com/politics            first 20170101074548   after cutoff: none
www.thehindu.com/news           first 20210203142541   after cutoff: none
www.lemonde.fr/international    first 20210814122104   after cutoff: none
timesofindia.../world           first 20260213115325   survives

n is 6, so take the five-of-six as a direction and not a rate. The mechanism is the solid part.

Your newest commit points the same way. "make not 404 on no results" means a miss is now a 200 with an empty body, which is the right call for a client, and it also means the two kinds of miss look more alike than ever. openapi.json has one path. LookupResponse is seven fields: query_url, surt_prefix, exact_match, at_timestamp, total_results, limit, results. None of them names the index. I ran a hit and a certain miss against the live Space just now, and the miss is total_results 0, results empty, with nothing else in the object to tell a subset miss from a real one.

So the client cannot stay dumb by being told nothing. It stays dumb if the server says what it covers. Earliest and latest timestamp, or a coverage label, in the response or on an /about route. Then ccnget never learns about index versions, it just repeats what it was told.

Would you put coverage in LookupResponse itself, or keep /lookup lean and add a route?

You are right about the ordering, and I ran the test that should have come before my claim.

Pulled the catalog and checked it three ways.

lines                        51,101
lexical inversions                0
timestamp inversions              0
ts month outside its own dir      0

Ten years, 120 months, no counterexample. Then I diffed it against Common Crawl's own published warc.paths for three months:

2018/04   cc 205   catalog 205   only-in-cc 0   only-in-catalog 0
2023/06   cc 686   catalog 686   only-in-cc 0   only-in-catalog 0
2026/07   cc 353   catalog 353   only-in-cc 0   only-in-catalog 0

Your catalog is exactly their set. So the mid-insert has zero instances, and my "an insert shifts 95.4% of ids" was a hypothetical with no data under it. Withdrawn.

One detail I did not expect, and it is the reason your ordering claim holds rather than a lucky coincidence. The sequence number is not monotonic. 2026/07 opens at -08467 and closes at -00313. Lexical order only tracks time because the 14-digit timestamp sits before the sequence field in the filename. That is the entire guarantee your ids rest on, and it is a naming convention on Common Crawl's side. Worth one line in the card, since it is the assumption a future reader would not think to check.

On /extent. The number is right and I cannot reach it.

GET https://brian-learns-cc-news-cdx-server.hf.space/extent
{"detail":"Not Found"}

openapi.json paths: ["/lookup"]
Space sha 34959cc, 2026-08-10T00:36:49Z

Your file_extent 51101 matches my line count exactly, so the code is right. The Space just has no commit since 2026-08-10 00:36, so the one published index still cannot answer /extent. It is in ccnget and not in the thing a stranger hits.

The two misses, said better, because I said it badly. A client gets total_results 0 for two different reasons. The URL was never in CC-NEWS at all. Or it was, and this index does not cover the window it was crawled in. Both come back 200 with an empty list and nothing distinguishes them. robots.txt blocking is the first kind and it is honest data. /extent is exactly the fix for the second kind: file_oldest and file_newest hand the client a window to check its own expectation against. So deploy it and most of the ambiguity goes away without ccnget carrying any diagnosis logic.

And yes on zstd, frames concatenate, so appending a new month is a cat.

The mini-format directory with the catalog at root and the rocksdb in a subdir sounds right to me. Would you put file_extent in a small plain manifest at the root as well, so a client can check the two artifacts match before the server starts, rather than after?

You are right on the two kinds of not-found, and I want to withdraw the half of it that was wrong.

No index can tell robots-blocked from never-crawled. Wayback does not, index.commoncrawl.org does not, and cdx-rocks should not try. That is diagnosis and it does not belong in a lookup.

The other half is not diagnosis, it is coverage, and CC does publish that.

collinfo.json, 126 collections, every one carrying from and to:
CC-MAIN-2026-30    from 2026-07-10T07:05:34  to 2026-07-23T01:13:28
CC-MAIN-2026-25    from 2026-06-05T21:48:11  to 2026-06-18T19:32:05
CC-MAIN-2008-2009  from 2008-05-09T05:37:12  to 2009-01-09T20:58:29

That is /extent at collection granularity, shipped by the index you are matching. The client never learns why a URL is missing. It learns whether it was asking inside the window.

Then the thing I did not expect. I grepped all 126 for CC-NEWS. Zero hits, every id is CC-MAIN. CC's own CDX server does not index CC-NEWS at all.

So yours is not a mirror of an index that already publishes its window. It is the only one. There is no upstream to cross-check your coverage against, which makes the window more worth publishing rather than redundant.

On redundant, I pulled the catalog out of the Space rather than asking you:

all_warc_paths.txt.zst   463,043 bytes, 51,101 lines, 120 months
oldest 20160826124520    newest 20260731214950
LC_ALL=C sorted                    : True
lexical order == timestamp order   : True

Two things. Your ordering guarantee holds on the real file, all 51,101 of them, not just the three months I diffed. And the extent is reachable, so you are right that it is redundant. The price is what I would push on: 463 KB and 51,101 filenames parsed to learn two timestamps.

Your own wiki puts the answer 84 bytes away. cdx-rocks.json already carries catalog, db and struct_format. A fourth key beside them makes the database self-describing without a scan, and it travels with the tar. Someone who downloads the directory and never touches your Space still knows what they have. That is the part an endpoint cannot do.

On the 40-hit hang, I did not try to reproduce it. No point wedging it for other people. But your app.py has a number in it that is close enough to 40 to rule out first:

block_opts.set_cache_index_and_filter_blocks(True)            # 161
block_opts.set_pin_l0_filter_and_index_blocks_in_cache(True)  # 163
...
# CRITICAL FIX 1: Caps open network file descriptors.
opts.set_max_open_files(32)                                   # 172

32 open table handles, and index and filter blocks living in an evictable cache. Consecutive lookups on distinct URLs hit distinct SSTs, so past 32 the table cache evicts on every request and each reopen is a fresh open on the FUSE mount, index and filter re-read with it. That is a config cliff at a fixed number rather than a bucket that gets tired, and the two look identical from outside.

You already print the number that decides it. Startup logs Files in directory: {file_count} for /data. If that is well above 32, the cap is binding by construction, and max_open_files(-1) keeps the readers open with their index and filter resident outside the block cache. If it is near 32, I am wrong and it really is the mount.

What does that line print?

Your local hf-mount repro is the part that pins it. Hanging on the first query that touches the db files is the index, not the file count, and the index is built into each file, so no runtime option reaches it. That is why 1 and -1 feel the same.

I censused the regenerated monthly bucket instead of extrapolating from the demo. Every SST's own footer, 1,142 of 1,142 parsed.

1,142 SSTs        77,423,870,080 B
index total          645,278,497 B    0.833% of file bytes
per file          min 124,586   mean 565,042   max 1,736,140

It is worse resident than on disk. I sampled 120 of the 1,142 for both the on-disk index handle and rocksdb.index.size:

disk 68,210,888 B    raw 219,380,913 B    3.22x
                     min 2.60  median 3.16  max 4.34

So July is about 2.08 GB of index once decompressed, not 645 MB.

filter.size is 0 on all 120 and index.type is 0 on all 120. OPTIONS-000007 in the regenerated bucket agrees: index_type=kBinarySearch, block_size=4096, filter_policy=nullptr. The new build command did not move either one, and the cleanest way to say that is that it did not move anything: monthly OPTIONS-000007 and both demo generations, OPTIONS-000011 and OPTIONS-000015, are the same 7,573 bytes and the same sha256 38139b195a3d. With no filter there is nothing to skip a file with, so every candidate file pays its index on every lookup.

Here is what max_open_files does to that, from strace on the demo, not from guessing:

32     open  5,453,442 B   then 641,186 B per lookup, forever
                           34 of 34 reads over 50 KB
-1     open 31,493,209 B   then   2,302 B per lookup
                           same key 20x: 650 B each, zero reads over 50 KB

Before blaming max_open_files, I checked the knob that should make it not matter. Your OPTIONS ships cache_index_and_filter_blocks=0 and pin_l0_filter_and_index_blocks_in_cache=0, so index blocks are held per open table reader rather than charged to the block cache. My strace rig above had both flipped ON, which is the friendlier setting, so those numbers are a best case for you rather than a worst one.

The flags turn out not to be a lever at all. Resident after open on your demo, same 51 files, varying only the cache settings:

max_open_files   cifb/pin   block cache      RSS after open
   -1            off        64 MB            106,983,424 B
   -1            on         64 MB            106,106,880 B
   -1            off        4 GB             107,376,640 B
   32            off        64 MB             18,907,136 B
   32            on         64 MB             18,890,752 B

Flipping cache_index_and_filter_blocks moves it under 1 MB at every cache size I tried, and a 64 MB cache does not bound a ~100 MB index, so those blocks are not living in the cache whatever the flag says. I would read that as the flag not reaching rocksdict rather than as a fact about RocksDB, but either way it is not a knob you can pull today.

What the same table shows is the real trade. At 32 the open is cheap and the lookups are not: 18,907,136 B at open, 39,010,304 B after six prefix scans, about 3.35 MB of index pulled per lookup and re-pulled after eviction. At -1 you pay it all up front and nothing after. Those two numbers are the same event seen twice, and they agree: 31,493,209 B read off disk at open becomes about 107 MB resident, which is the 3.22x decompression ratio I measured on July, arriving on the demo. Over hf-mount that is the difference between a long first query and a permanently slow every query.

Crossover is 49 lookups. Under that, 32 is cheaper. Past it, -1 has already paid less and keeps winning by 279x. On July both ends are bad in opposite directions: -1 reads ~645 MB before it serves anything, and 32 slots against 1,142 files is a cache that cannot hold a working set whatever keys arrive. Over hf-mount, that opening cost is network bytes.

Now the part I did not expect, and I think it is the real story.

bytes per key    123.8
bytes per value   16.0      exactly your !IQI
key share of raw  88.6%
entries per 4 KB data block   37.8
raw index bytes per entry     54.5

Your values are 16 bytes and your keys are 124. The DB is 88.6% key by raw bytes. At block_size=4096 that is one index entry per 37.8 records, each costing 54.5 bytes to describe 4 KB. The index is fat because the keys are fat, not because the DB is large.

Which makes block_size the cheapest lever you have, and it needs no format change. 32 KB divides the entry count by 8, and for a prefix scan a 32 KB block is likely to serve several hits rather than one.

The sharper version is the index type. I rebuilt 4,000,000 consecutive records out of your demo DB twice, same records, same zstd, same 10 output SSTs, only the index type differs:

                        per lookup (same key, 20x)     on disk
binary_search (yours)          186,996 B            224,730,731 B
two_level_index_search             174 B            225,016,263 B

+0.13% on disk. rocksdict exposes it already: BlockBasedIndexType.two_level_index_search() with set_metadata_block_size(4096).

One honest limit on that pair. My binary_search rebuild re-reads its index every lookup, where your demo reads each index once at open and stops. Read those two rows against each other, not as a prediction of your absolute numbers.

Which makes the demo the experiment. 51 files is 31 MB of index, not 645 MB.

You ran that experiment before I could ask for it, so I measured your Space instead of asking.

/cdx-index/lookup   bbc.com/news, same key 5x   0.933 0.912 0.941 0.922 0.967 s
                    nytimes.com                 0.921
                    aljazeera.com/news          0.919
                    a domain certainly absent   0.938
/openapi.json       static, never opens the db  0.968  1.034
/docs               static, never opens the db  1.001

A lookup costs what fetching a static JSON file costs. The index work is under the transport floor, and a certain miss is indistinguishable from a hit. So at 51 SSTs and 3.69 GB it is size, not build, and the 31 MB against 645 MB split is the whole story.

Then I read your two manifests against each other, and this is the part I actually want to raise.

demo     "struct_format": "!HQI"   14 B   file_extent  2,831   2023/05 .. 2023/08
monthly  "struct_format": "!IQI"   16 B   file_extent 51,101   2016/08 .. 2026/07

!HQI is DEFAULT_STRUCT_FORMAT, build.py:35. The file id is an H, so it stops at 65,535. I ran your own schema.safe_pack at the edge rather than reasoning about it:

warc_id 65,535   OK, 14 B
warc_id 65,536   struct.error: 'H' format requires 0 <= number <= 65535

struct.error subclasses Exception, and build.py:137 catches Exception, logs it, and continues. Over ids 65,530 to 65,544 that is 6 kept and 9 dropped, with the build returning normally.

Three things make that quiet rather than loud.

extent.json takes file_extent from len(name_to_id), build.py:164, which is the catalog and not the database. So the artifact keeps advertising files whose records never landed.

The summary at build.py:187 prints records, duplicates dropped, and WARC files. Duplicates are the only drop it names, so the line reads clean.

And load_catalog numbers with enumerate(..., start=1) over a catalog that is strictly oldest first. I checked rather than assumed: zero out-of-order month transitions across all 51,101 lines, id 1 is 2016/08, id 51,101 is 2026/07. So the highest ids are the newest files, and an overflow lands on the freshest data first. The index would quietly stop getting newer while file_newest kept moving.

You are at 51,101 of 65,536. That is 78.0%, and 14,435 files of headroom.

last 12 complete months   5,653 files   471.1/mo   30.6 months
last  6 complete months   2,193 files   365.5/mo   39.5 months

The rate is falling, 616 in 2025-12 down to 353 in 2026-07, so the slower number is the fairer one. Either way it is 2029, not never.

The monthly bucket only escapes this because you passed an explicit !IQI. The default is what someone else gets.

Would you rather lift the default to !IQI and pay the two bytes, or keep !HQI and have build_index refuse up front when len(name_to_id) will not fit the format it was handed?