Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
RDTvlokipΒ 
posted an update Aug 12
Post
1,759
I published a second article about a reader taking apart four of my numbers.

He came back a fifth time, and went to the experiment that had never run.

He found the one derived threshold in it was built on a sample maximum. He was right, and there was worse. That maximum cannot converge, because the codes I wanted to declare unreachable are themselves inside the null distribution. Its supremum is exactly the value I was excluding.

So I ran the whole thing. Seven questions in one day, in a world of 27 referents small enough that the optimum, the null distribution and the gradient are computed rather than estimated.

The certificate carrying the project does not survive two agents. A symmetry argument replaces it, with a corollary I did not expect: a free per-object embedding table cancels in advance anything the message structure could contribute. No training trick recovers it. You can check that by reading an architecture, without running it.

Eight of my hypotheses died that day. Not one was an arithmetic error.

Then I did the literature review, last, and found the argument published in 2021.

The real lesson:

Shrinking a world until everything is exact removes one class of mistake and leaves untouched the class that was doing the damage. Twenty minutes of searching would have saved a day of deriving. What caught my errors was never the exactness. It was an outside reader, a second route to the same number, and predictions written down before measuring.

Code, every number including the ones I would have cut, and sixteen dated refutations πŸ‘‡

πŸ”— https://huggingface.co/blog/RDTvlokip/i-made-my-world-small-enough-to-compute-everything
πŸ’» https://github.com/RDTvlokip/RDTRL
πŸ“¦ https://doi.org/10.5281/zenodo.21726216

The distinction between exact computation and model validity is an important warning for agentic work. A practical guardrail is to preregister assumptions and expected failure modes, then run a small independent reproduction before scaling; it turns outside review into a testable part of the workflow.

Β·

Thank you. I have direct evidence on both halves of your guardrail, and they came out unevenly, which I think is the useful part.

Preregistration: I did it, and it failed in a way I would not have predicted. The design document recorded a dated commitment before any data existed, naming the statistic and the expected outcome, and it was presented as what made the test sharply falsifiable. When the experiment finally ran, the commitment turned out to be underspecified on three counts. It never named the parametrization, which is the variable that decides the answer, so the same commitment was confirmed on two parametrizations and refuted on a third. Its interpretation clause said an excess would prove a flaw in a specific piece of reasoning, and that inference does not follow. And its first half predicted near-perfect bijections, of which there was one in twenty.

So preregistration protected me from fitting after the fact, which is what it is for, and protected me from nothing else. It does not catch an omitted variable, and it does not catch writing down the wrong interpretation in advance. If I keep one rule from this, it is that a recorded prediction has to name three things and not one: on which population, under which parametrization, and what each outcome would prove. I only ever wrote the first.

Independent reproduction: this is the half that worked, and it worked six times out of six. Every correction to this project has come from one outside reader who reruns the code before commenting. Not from review of the write-up, but from re-derivation from the artifact. That is a real constraint on me rather than on him: it only works if I publish enough that re-derivation is possible, so the guardrail is upstream, in what gets released.

It also has a failure mode worth naming, because it happened. He and I once agreed, and we were both wrong, because his argument was correct under an assumption we both carried without stating it. Two people can share a frame, particularly when the second is reading the first's documents. What broke it was not a third reader but running a step of the programme that had never been run.

Which is the version of your guardrail I would defend most: anything that does not inherit your specification. An outside reader is the most available instance and the highest yield. A second independent route to the same quantity is the cheapest, and computing a threshold from a Hessian rather than from a bisection caught that I had been measuring my optimiser rather than my objective. And when all of those agree and are wrong together, the only thing structurally guaranteed not to share your frame is the measurement you have not made yet.

On the exact-computation point you opened with: making the world small enough to compute everything exactly removed the estimation-error class entirely and left the specification-error class untouched. Eight of my hypotheses died in one day and not one was an arithmetic error. Exactness earned its keep only by making the second route cheap, which is a real benefit and not the one I thought I was buying.