Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
RDTvlokip 
posted an update 3 days ago
Post
63
I published an article about training a network to write from reward alone. Code on GitHub with it.

Then someone read the code.

Dipankar Sarkar commented four times in a day. Each time he had run something first. He rebuilt my statistics in numpy because he had no torch installed.

He found a bound I had missed. A policy that never learned the determiner to noun dependency has a product support, so at full validity it cannot exceed the largest fully valid product in the sublanguage it entered. That is 12 on one side and 24 on the other, computable before any training. Over 70 seeds it is never crossed, and the most common outcome is the bound itself. I had published one of those numbers as an interesting coincidence.

Then four of my published numbers came apart. Three were a single seed. The fourth was twenty seeds, and I had produced it while fixing the other three.

And the test I built to validate his bound tested nothing. I had swapped two conditions so cleanly that the two grammars were isomorphic. Seventy seeds would have returned the mirror image by construction.

The real lesson:

A relabelling can permute, but it cannot change a ratio. A perfectly symmetric control is often a perfectly empty one. None of my errors were in the reasoning. They were in the plumbing, and nothing in my own process caught a single one.

Code, figures, and the notebook with eight dated refutations 👇

🔗 https://huggingface.co/blog/RDTvlokip/i-published-my-rl-experiments
💻 https://github.com/RDTvlokip/RDTRL
📦 https://doi.org/10.5281/zenodo.21726216

Thank you for writing this up. Test 3 is the interesting part, so I went there instead.

The one number in your null is the one you called derived.

I ran grammaire3.py at seed 0 before saying anything. Your table reproduces exactly: mean 0.1273, sd 0.0332, min 0.0305, max 0.3305, quantiles 0.0661 / 0.1246 / 0.2171 / 0.2537.

Then I ran the same statistic at 10,000,000 draws instead of 20,000. Vectorised, checked against your own concentration() on 3,000 codes first, max absolute difference 5.6e-17.

Every row holds except one.

20,000 10,000,000
mean 0.1273 0.1269
sd 0.0332 0.0330
q99 0.2171 0.2161
q99.9 0.2537 0.2525
max 0.3305 0.3979

500x the draws moves q99.9 by 0.0012 and the max by 0.067. The body of the null is settled at 20,000. The max is the only row still moving, and the ~0.35 line is the one thing built on it. 14 of the 10M draws clear 0.35.

That does not hurt you and I would rather say so than dress it up. At 100 seeds the chance any run clears 0.35 under the null is 1.4e-4. The threshold is safe for the experiment you planned.

The reason to drop it anyway is that §6.2 is already better without it.

Your recorded commitment is about a distribution, not a run. 100 emergent concentrations against a null of sd 0.033 resolves a mean shift of 0.013 at p<0.001 with 80% power. 50 seeds resolves 0.018. The threshold needs one run sitting 0.223 above the null mean.

So the instrument you specified is 17x more sensitive at 100 seeds than the line written next to it. And a weak outside pressure is far likelier to lift every run by 0.02 than to throw one outlier past 0.35. That pressure is invisible to the threshold and a 6-sigma result to the test you already have.

Quote a quantile instead. q99.9 is stable at 20,000. q99.999 needs the long run and sits at 0.3195.

Separately, on the statistic itself, and this one I am less sure about.

concentration takes max over attributes independently per column, so one attribute can win two positions. I checked how often that happens: 74.6% of 500,000 uniform draws, and 46% of the top 200 by concentration. Scoring those same top 200 with a Hungarian matching instead of a per-column max gives 0.2640 against 0.2810. Mean inflation 0.0170, largest single 0.0915.

For your §6.2 comparison this cancels. Same transform on both sides, emergent and null, so the test is unaffected and I am not asking you to change it.

Where it does not cancel is §6.1, where you read the scalar as a position in the space of perfect codes rather than a binary. Under that reading a 0.28 code is being credited 0.017 of structure it does not have, in the direction that makes a holistic code look composed.

Concentration 1.0 still forces an injective argmax, so the top of the scale is safe. A code where two positions each fully determine one attribute would collapse that attribute's nine referents onto three messages, and a bijection cannot.

So the inflation is real in the middle and absent at both ends.

Is the max standing in for a matching you have not needed yet, or is there a reason to want the double count?

·

You went to the one part of the project that has never run. That is the right place to go, and it is the first time in five rounds that a criticism has arrived before the data rather than after. That matters at the end of this reply.

Your numbers

All of them. Seed 0 at 20,000 reproduces to the digit. My own 10M draw is an independent stream, so it is a check on the estimand and not on your RNG:

you, 10M me, 10M
mean 0.1269 0.12688
sd 0.0330 0.03304
q99 0.2161 0.21613
q99.9 0.2525 0.25254
q99.999 0.3195 0.31963
max 0.3979 0.39788
draws ≥ 0.35 14 14

Two independent samples agreeing on a maximum to four digits made me check whether the statistic was discrete enough to explain it. It is discrete — 2,951 distinct values in 2M draws, since the whole matrix is a function of nine integer counts — but the top rungs are singletons, so discreteness explains nothing. It is a coincidence, and the next section is what happens when you stop relying on one.

Your vectorised-vs-scalar check: I did the same and got max absolute difference 5.551e-17 on 3,000 codes, which I assume is your 5.6e-17. For a bijection both margins of the joint are exactly uniform, so I(A_i;M_j) = Σ (c/27)·log2(c/3) with c integer — that identity is what makes the whole thing a lookup table, and validating against the scalar path is really validating the identity.

The threshold: you are right, and the reason is worse than "still moving"

Your framing is that the max is the only row still in motion. Mine is that it cannot stop.

The 1,296 compositional codes are bijections. They are in the null, with probability 1296/27! ≈ 1.19e-25, and they score exactly 1. So the supremum of the null distribution is exactly 1 — the very value the threshold was built to declare unreachable. A sample maximum here is not estimating a threshold. It is estimating 1, infinitely slowly. No sample size fixes that, because there is nothing to converge to except the thing being tested.

Empirically, twelve independent blocks of 10,000,000 (variabilite_du_maximum.py, 384 s):

row mean over 12 blocks range across blocks
mean 0.1269 0.0000
sd 0.0330 0.0000
q99 0.2159 0.0003
q99.9 0.2527 0.0006
q99.99 0.2863 0.0019
max 0.3950 0.0509

The block maxima run from 0.3775 to 0.4283. That range is 1.54 standard deviations of the null itself. A threshold read off that row inherits it: it measures how long I was willing to sample, not the distribution.

So: threshold withdrawn, quantile quoted instead, exactly as you proposed. q99.9 = 0.2525.

The part you found without pointing at it

§6.1 should never have carried a pass/fail line at all, and my own document says so three paragraphs earlier. §5 reads:

On abandonne délibérément le critère pass/fail. […] la mesure de concentration positionnelle définie en §6.1 sera statistiquement indiscernable de celle d'une permutation tirée uniformément au hasard.

That is a commitment about a distribution. Then §6.1 reintroduces a pass/fail and congratulates itself for it being derived rather than arbitrary. The defect was never the arbitrariness. It was the pass/fail, which §5 had already discarded for the reasons you then re-derived from the power side. I wrote both paragraphs. It took an outside reader for me to read them next to each other.

Your power numbers reproduce exactly. One-sided, p < 0.001, 80% power, δ = 3.93σ/√n:

seeds max-per-column matched
50 0.0184 0.0175
100 0.0130 0.0124

against 0.223 on a single run for the threshold. Ratio 17. That calculation is now in §6.2 where it belongs.

The matching: your diagnosis is right, your localisation is not

74.6% double-count rate on uniform draws — I get 74.6%. On my top 200, matching gives mean inflation 0.0179 against your 0.0170. Same phenomenon.

Where I disagree is "real in the middle and absent at both ends". I could not test that against the null alone, because the null has no structured codes in it. So I built two populations where the structure is known independently of the MI matrix.

Codes with k of 3 positions cleanly encoding an attribute, the rest scrambled conditionally (still bijections), 4,000 each:

k expected max-per-column matched double count inflation
0 0.0000 0.1268 0.1170 74.0% 0.0098
1 0.3333 0.4111 0.4059 48.5% 0.0052
2 0.6667 0.7045 0.7045 0.8% 0.0000
3 1.0000 1.0000 1.0000 0.0% 0.0000

A ladder from a compositional code outward by transpositions, scored against d = minimum Hamming distance to any of the 1,296 (so relabellings are free):

 d      0    2    4    6    8    9   10   12   14   16   18   20   21   23
 infl  .000 .000 .000 .000 .000 .000 .0001 .0004 .0022 .0051 .0081 .0099 .0103 .0090
 dbl%   0    0    0    0    0    0   1.1   3.5  17.4 35.5 57.1 68.8 73.2 77.7

Inflation is exactly zero out to nine transpositions, then climbs and meets the null's own 0.0101 around d = 21.

So the inflation does not live in the middle of the scale. It lives in the region with no positional structure at all, and it is identically zero everywhere there is structure to read. Which means the specific §6.1 worry — a 0.28 code credited 0.017 it does not have — is real only for codes that reached 0.28 by accident, and those have nothing to read either way.

I want to flag how I nearly got this backwards. My first sweep binned the null by concentration level and found inflation rising with concentration: 0.0014 below 0.05, 0.0228 above 0.30. That is a true measurement and it points the opposite way from the ladder, which at the same concentration ≈ 0.27 gives 0.0022. Both are correct. Inflation tracks structure, not level, and those are not the same axis — a uniform draw at 0.27 got there by accident and usually double-counts, a near-compositional code at 0.27 did not. One population would have convinced me of the wrong thing.

Your actual question

Is the max standing in for a matching you have not needed yet, or is there a reason to want the double count?

Standing in for a matching. There was no reason.

There is one reason to keep reporting it, which I did not know when I wrote it and had to go check: the unconstrained per-position form is the field standard. posdis (Chaabouni et al. 2020) takes the argmax independently per position with no one-to-one constraint. So dropping it costs comparability with published numbers.

Both are now in grammaire3.py. concentration() unchanged, so the 20,000-draw table still reproduces bit for bit. concentration_appariee() added — exact Hungarian, and at 3×3 that is six permutations, so enumerated rather than solved. §6.1 now reads the matched one as its position.

One argument for your fix that you did not make

You wrote that the change cancels in §6.2 because it is the same transform on both sides. That holds under H0 — if the emergent codes are ordinary bijections, both populations shift by the same 0.0101 and nothing changes.

It does not hold under H1. If a pressure creates positional structure, the double count disappears with it, as the tables above show. The 0.0101 then comes off the null and not off the emergent population. The matched statistic lowers the reference without moving the signal: neutral where it must be, favourable where I want power. Small — sd also drops 0.0330 → 0.0315 — but it points the right way, and it is a better argument for your change than the one you offered.

Bounding the damage

Local hill-climbing over transpositions, so these are lower bounds, not maxima:

  • largest max-minus-matched gap found: 0.1443, on a code scoring 0.2473 — i.e. inside the body of the null, where nothing is concluded anyway;
  • highest concentration reachable while double-counting: 0.6314, and that code still scores 0.5560 matched. Even pushed adversarially, the max does not turn an unstructured code into a structured one;
  • highest non-compositional concentration found: 0.9294. Your argument that concentration 1.0 forces an injective argmax is right — two positions each fully determining one attribute would collapse nine referents onto three messages, and a bijection cannot — and the top of the scale turns out to be isolated as well as safe: 1.0000, then 0.9294.

All three of those are hill-climbing over permutations, which matters more than I realised when I wrote them. See below.

A follow-up that makes your fix necessary rather than optional

After answering you I ran §6.7, the two-agent check that had never been done. One of its results changes the standing of your second question.

Exact gradient ascent on the joint objective, no sampling anywhere, does not reach a bijection from a uniform start. One start in forty. It settles on codes where 1 to 4 referents collide — E[R] lands on 23/27, 24/27, 25/27, 26/27. Started on a perfect code it stays at exactly 1.

So the emergent codes will not be bijections, and your safety argument for the top of the scale is conditional on bijectivity. Drop that condition and it fails outright. Take m₁ = a₁, m₂ = a₁, m₃ = a₂ — duplicate the first attribute across two positions, discard the third:

distinct messages used : 9 of 27          bijective : no

I(A_i ; M_j) in bits          concentration, max      : 1.000000
[[1.585 1.585 0.   ]          concentration, matched  : 0.666667
 [0.    0.    1.585]
 [0.    0.    0.   ]]

The published statistic hands a perfect 1.0000 — the value reserved for compositional codes — to a code that throws away one attribute in three and uses a third of the message space. The matched version returns 0.667, which is the right answer: two attributes read out of three.

So the double count is not a middle-of-the-scale artefact confined to structureless codes, which is what the measurement on bijections suggested to both of us. Once bijectivity goes, it reaches the top. Your fix is not a 0.48-point improvement in rank concordance. It is the only one of the two statistics that stays interpretable in the regime the experiment will actually be in.

I would not have found this by measuring harder on the null. It came from running the step that had never been run, and it is the second time in this exchange that the useful thing was upstream of the number under discussion.

Why this round was cheap and the next one would not be

No Test 3 training has run. There is no measured emergent concentration anywhere in the repository. So changing the instrument today cannot have been motivated by a result, and anyone can check that from the git history. The identical change made after a first run would be unverifiable from outside, and should be refused. That is the whole reason §5 recorded its commitment with a date, and it is the first time the mechanism has actually been worth something.

Thank you for going to the unpublished part. It is where the mistakes are cheapest to fix and where nobody has any reason to look.