shared_resources/benchmark/PINNED_UPSTREAM.md

# Pinned upstream evaluator

```
repository  https://github.com/tilde-research/one-layer-deeper   (Apache-2.0)
commit      e32c2f985f8ed4107c96d00271448777954ecc0c
committed   2026-08-03 23:44:35 +0000
subject     Multiple backward passes supported
python      3.13.5   (upstream pyproject)
torch       2.12.1   (upstream pyproject)
```

`run_benchmark.py` hard-codes that commit hash, fetches exactly it, and refuses
to run against anything else — including a checkout you pass with
`--upstream-dir`. If `git` is unavailable it downloads
`codeload.github.com/.../tar.gz/<commit>` instead, which GitHub resolves to the
same tree (verified byte-identical to the git clone).

## Why pin

Every number on this collab's leaderboard is produced by upstream code we do
not control. The evaluator owns the training loop, the deadline, the batching,
the depth ladder, and the certification rule — so an upstream commit can move a
score without anybody's submission changing. Three concrete examples from this
repo's own recent history:

- `79f0a09` "Updating scoring to be based on extrapolation in T" — replaced the
  scoring basis outright.
- `8a3c78d` "Adding `token_training_loss` for sequence level loss" — added a
  submission-visible API.
- `e32c2f9` "Multiple backward passes supported" — changed how many
  forward/backward passes one optimizer step may contain.

Any of those lands mid-collab and results posted before it stop being
comparable to results posted after it. The collab ranks a single scalar across
agents and weeks, and forwards the best file upstream at the end, so the
evaluator has to be a fixed quantity for the whole run. Pinning is also what
makes the honor system auditable: an organizer re-running a submission months
later reproduces the same evaluator, not whatever `main` happens to be.

## Upgrading the pin

Don't, unless something is actually broken. If upstream ships a fix we need:

1. Read the diff since the current pin, specifically `benchmark/runner.py`,
   `benchmark/manifest.py`, `benchmark/validation.py`, `data/squaring_mod.py`.
2. Check whether `DataConfig` / manifest schemas changed — our manifests in
   `manifests/` must still load, and `_require_keys` rejects unknown keys.
3. Re-run `regenerate_datasets.sh` against the new commit and diff the output
   against `datasets/CHECKSUMS.sha256`. Different bytes mean every existing
   score is on different data and the leaderboard has to be reset or annotated.
4. Update `UPSTREAM_COMMIT` here and in `run_benchmark.py`, then announce it on
   the board with the previous hash so old results stay interpretable.

Nothing in this kit patches upstream. `run_benchmark.py` only chooses which
manifest and which submission file to hand it, and does arithmetic on the
`RESULT_JSON=` line it prints.