Waves, queues, and the three free gates
A reproduction campaign is a batch-processing problem wearing a research costume. Papers arrive in the hundreds, each one passes through the same stages, failures are common and must not poison the rest, and the whole thing runs unattended overnight on one machine. This post covers the middle of my pipeline: how papers move, and the three free checks everything must pass before a dollar or a publish slot gets spent.
Waves and lanes
Papers enter as wave manifests, a dozen papers per wave, selected by the miner described in the companion post. A supervisor loops over the fleet around the clock with one source of truth: a single state ledger that records every paper's stage, every dollar spent, and every dollar reserved before a paid call is attempted. Reservations matter because parallel workers share one budget, and the ledger is the only thing standing between a 20 dollar ceiling and an accidental 400 dollar weekend. Every state change is a small atomic write, so a crash resumes cleanly instead of rerunning or double-spending.
Before any compute is spent on a paper, an intake audit files it into one of four lanes, in effect a first sort by where each paper sits on the expected-points-versus-cost frontier, so the dominated candidates never reach a worker:
- perfect-score: every claim looks testable, the release is pinned to an exact commit, expected capture is high. Full automation.
- normal-throughput: workable but imperfect. Automation with wider tolerances.
- release-audit-only: the release is withdrawn, empty, or vaporware. Record the audit, spend nothing further.
- blocked-quarantined: something disqualifying, from gated data to a claim count mismatch. Parked with a receipt explaining why.
The receipts are immutable, and the runner fails closed: a paper whose manifest row went stale since intake simply does not run. On top of the lanes, a Thompson bandit keeps a Beta posterior over each (release type, compute class) pair, with reward equal to the points fraction actually earned. Selection starts from priors and gets smarter as verdicts land, which is how the campaign discovered empirically that code-bearing papers outperform theory papers on capture rate.
The three free gates
Everything that wants to ship must pass three local checks. They cost nothing to run, and they catch nearly every failure that would otherwise cost money or reputation.
Gate 1: determinism. The paper's validator runs twice, and the two stdouts must match byte for byte. This sounds trivial and is not: it has caught unseeded randomness, wall-clock timestamps in output, dictionary-ordering drift, and floating-point instability that would have made published numbers unreproducible. A logbook whose own validator cannot reproduce itself has no business claiming to reproduce a paper.
Gate 2: the judge-window simulation. The official judge ingests at most 120,000 characters per logbook, with specific rules about page order and fence handling. I reimplemented that ingestion locally and require every page to fit and every must-see number to be visible inside the window. My first-ever submission scored 0 of 4 because a 1.37-million-character page pushed the actual evidence out of the judge's view. The grader defines reality; simulate the grader.
Gate 3: number integrity. A scanner walks every number on every authored logbook page and requires it to appear in a raw result file produced by a run. No raw support, no ship. This is the anti-fabrication gate, and it also catches the subtler crime of quoting 17 digits of a float that only reproduces to 12. The rule forced a habit: write results files first, then quote them verbatim.
Where the money gate sits
Only after all three free gates pass does a bundle reach the paid stage, and even then a local fine-tuned screener predicts the official score first, as covered in the distilled-judge post. The paid GLM-5.2 pre-judge runs once per unique content digest, cached forever. The ordering is the entire cost model: free gates before free screening, free screening before the single paid call, the paid call before a publish slot. Each stage is strictly cheaper than the one it protects. That ordering is what keeps the pipeline on the cost-efficiency frontier: you never climb to a more expensive resource until the cheaper one has rejected everything it can.
What transfers
- One ledger, atomic writes, reservations before spending. Crash-safety is a budget feature, not just an uptime feature.
- Classify before you compute. The intake lanes killed vaporware papers in seconds that would have wasted GPU-hours.
- Determinism is a shippability requirement. Run everything twice and diff.
- If an external system grades you, reimplement its ingestion and test against it locally. Assumptions about what a grader sees are the most expensive assumptions in the pipeline.
- Order your gates by cost. Everything free runs before anything metered, and everything metered runs before anything irreversible.