The check grows. All three of your points were the same failure the check was
written to catch, one level up, and it is rewritten rather than patched.
Coverage is now the default. It walks to any object holding a numeric oracle
at any depth instead of matching d["arms"], and takes the bounded set as every
numeric sibling not exempt for a stated reason instead of an allowlist. Your
count was 266 of 1,373. It now judges 2,501 cells across 23 files.
Per file, old against new on the shapes you listed:
verifier/exec_guided.json 0 -> 148
verifier/results.json 0 -> 666
hivemind_census.json 0 -> 60
he_k32_k_scaling.json 0 -> 72
exec_guided is the one that makes your second point concrete. It was on the
allowlist as a file name, its columns are exec and exec_verifier, and
neither was on the list, so the file printed a line and judged nothing. All
five dead names are gone; there is no list of judged columns any more, only a
list of exempt ones, and each carries its reason: subset rate, count, model
size, difference between two columns, ranking score, or outside [0, 1].
The closing line now names its own scope:
scanned 746, carried an oracle 22, judged 2501 cells, skipped 930
no column scored over all problems exceeds its oracle in any file scanned above
--verbose prints the 930 by reason. Largest are 936 floor, which is now
judged rather than skipped and is bounded by construction, 188
consensus_on_covered, 82 n, 81 gain_over_floor, 78 on_resolved_only, 42
headroom_captured.
Your subset-rate counts reproduce exactly on the current files: verifier
chat_consensus 6, chat_consensus_mbpp 1, consensus 12, consensus_mbpp 1;
verifier_1024 chat_consensus 3, consensus 7, consensus_mbpp 1. Zero violations
on any all-rows column, which is also what you found by hand across all 660
cells the old shape could not reach.
Your third point paid for itself immediately. artifacts/nla/q4/ one_forward_pass_probe.json has been unparseable since 24 August: Python
implicit string concatenation inside a .json file, from a paste. The old
check caught the exception, returned None, printed nothing, and closed with
"no all-rows selector exceeds its oracle in any current file", exit 0. Three
weeks of runs called the repo clean without reading it. It now prints
UNREADABLE and the file is fixed; all 1,035 artifacts under artifacts/ parse.
paper_hivemind 881 said "one pass over every results file" when the pass was
two shapes wide. Rather than narrow the sentence, the paragraph now states what
the first version did, 266 cells and a silent return, and what the current one
does with the scanned/judged/skipped counts above.