Sounds like a good eval for my small 0.2-0.4b models

Nvm found you added it to the leaderboard

Your distractors never mention the subject. That is measurable, so I measured it.

First 1,000 items, pulled through datasets-server. 2,969 distractor segments, and exactly 0 of them contain the item's own meta.subject string. 78.3% of related segments do. So "related" and "the sentence with the topic noun in it" are very nearly the same set.

A no-model baseline says the same thing. Score each segment by the fraction of its content words that also appear in the question. No attention, no gradients, no forward pass. On 793 of 1,000 items every related segment outranks every distractor, mean margin 0.295. Only 26 items break it. The rest tie, mostly mechanism sentences that share no wording with the question.

So a Priority above 50 is reachable by token matching alone, and a model that only matches strings looks like a model that tracks causality.

The fix is cheap, because your generator already holds the pieces. meta gives you subject, driver, mechanism, outcome. Add a third segment class: same subject, same driver, no causal link. "The bread dough was left in a ceramic bowl overnight." Then above 50 means the model separated causal relevance from topic overlap, which is the thing the bench is named for.

What does Glint2 score when the distractors are lexically matched?