Post
92
I spent a week pointing 123 automated reproductions at ICML 2026 submissions, checking anchored theoretical claims against the papers' own pinned LaTeX.
The interesting part was not the ones that reproduced. It was the handful that did not.
One paper claims its convergence rate matches a cited result under bounded variance. Its own text says the opposite at four separate sites, including a table column headed "unbounded variance" with a check in that row, and its own conclusion. Six other people reproduced that paper and called the claim verified. So did the organizers' reference logbook, which called it inconclusive.
Another paper's bandit proposition states a regret bound of O(T^2/3). Its own comparison table cites that same proposition for O(T^4/5), and a later dynamic bound disagrees with the table a second time. Four other people reproduced it. None of them caught it.
A third asserts its method converges without a decaying learning rate, unlike two baselines. All three theorems carry the identical Robbins-Monro condition word for word, and the paper's own experiment section says a constant stepsize does not converge to zero.
The habit that found these was boring: pin the source, quote line numbers, and refuse to call something falsified when the paper is merely silent. We withdrew one of our own falsifications this week after an independent clean-room run reached our facts and refused our label, and that one had already been accepted.
Logbooks: @RyeCatcher
The interesting part was not the ones that reproduced. It was the handful that did not.
One paper claims its convergence rate matches a cited result under bounded variance. Its own text says the opposite at four separate sites, including a table column headed "unbounded variance" with a check in that row, and its own conclusion. Six other people reproduced that paper and called the claim verified. So did the organizers' reference logbook, which called it inconclusive.
Another paper's bandit proposition states a regret bound of O(T^2/3). Its own comparison table cites that same proposition for O(T^4/5), and a later dynamic bound disagrees with the table a second time. Four other people reproduced it. None of them caught it.
A third asserts its method converges without a decaying learning rate, unlike two baselines. All three theorems carry the identical Robbins-Monro condition word for word, and the paper's own experiment section says a constant stepsize does not converge to zero.
The habit that found these was boring: pin the source, quote line numbers, and refuse to call something falsified when the paper is merely silent. We withdrew one of our own falsifications this week after an independent clean-room run reached our facts and refused our label, and that one had already been accepted.
Logbooks: @RyeCatcher
