Five machine raters, one definition, and answers ranging from 0 to 78
A few months ago I published a check on the objective-projection dataset's applied_rules field. Every scene in that corpus carries six machine-generated flags — two prohibitions (naming an emotion outright, using a simile) and four positive techniques (materialized metaphor, micro-focus, temporal anchor, atmosphere contradiction). Those flags had been used to describe the corpus and select examples. Whether a human would agree with them, nobody had checked.
I have now checked it three times. The paper reporting all three studies is here: 10.5281/zenodo.21740239.
The three studies
Study 1 (120 Turkish scenes). I labelled them blind myself, then compared against the rule-based detector. I am the author of the rules, so this is the weakest of the three designs, and I say so in the paper.
Study 2 (100 different scenes, zero overlap with Study 1). An independent volunteer with no stake in the methodology labelled them blind. Her labels were locked before any machine ran. Three machine labellers were scored: the detector, Gemini 2.5 Flash, and Grok.
Study 2b (same 100 scenes, same locked labels, same prompt blocks). Claude Fable 5 (High) and ChatGPT 5.5, in fresh sessions with no project context.
The number that matters
One rule — materialized metaphor — asks whether an abstract inner state has been rendered as a concrete physical detail instead of being named. It is the rule closest to what the whole methodology is about.
Here is what five machine labellers said about the same hundred scenes:
| Grok | Gemini 2.5 | ChatGPT 5.5 | Detector | Claude Fable 5 | Human | |
|---|---|---|---|---|---|---|
| Scenes marked present | 0 | 1 | 40 | 72 | 78 | 9 |
Cohen's κ was at or indistinguishable from chance for five of the six labellers: 0.004, 0.015, 0.000, 0.019, 0.027. Gemini's 0.185 is the one exception, and it rests on a single positive judgement.
Two systems given the same written definition, applied to the same hundred scenes, produced 0 and 78. That is not a ranking of accuracy. It is evidence they were not attempting the same task.
The same divergence shows up between the models themselves. On micro-focus, Gemini marked 9 scenes and Grok marked 82 — against a human count of 96.
A correction to what I wrote last time
In my previous post I said the machines missed atmosphere contradiction essentially entirely. That was true of the Study 2 raters. It is not true generally.
Against the human's 44 positives, Claude scored κ 0.269 and ChatGPT 0.184. Both are far from strong agreement, but both are clearly above chance — and this is the one rule whose class distribution can support that judgement at all. I was wrong to generalise from three raters, and the newer models partially recovered a feature I had written off.
I also mis-stated Grok's atmosphere numbers: it marked 6 scenes, 3 of them correctly, not 3 total.
The trap in the headline numbers
ChatGPT posted the highest overall raw agreement of any system in any of the three studies: 84.5%. Its materialized-metaphor κ was 0.019.
Both facts are true because five of the six rules have severely lopsided human distributions — positives of 0, 1, 9, 96, 99 and 44 out of 100. When one rater says "present" to 99 scenes, agreement statistics have no variance to work with. Any leaderboard that ranks these systems by raw agreement is ranking them by their willingness to say "absent."
This is a defect of my evaluation set, not of the systems evaluated. Four of the six rules were, in effect, not tested at all. Only atmosphere contradiction (44/56) sits in a usable distribution — which is precisely why it is the only rule where I am willing to say two models did better than chance.
What I cannot tell you
Two readings survive this data and I cannot choose between them.
(a) The feature is genuinely inferential. Deciding whether a physical detail is carrying an unstated inner state means reconstructing what the text withholds. If that is a reader-side act rather than a text-side property, no detector should find it and no current model should either.
(b) The definition is not operational. In a corpus built entirely from the instruction "render states physically," nearly every scene qualifies under a permissive reading and few under a strict one. The human's 9-out-of-100 is itself a strict reading. My own criterion drifted mid-pass in Study 1 — direct evidence that the rule is unstable even within one rater.
The two make one distinguishing prediction: under (a), two independent humans should agree with each other much better than any machine agrees with either. Under (b), two humans should also disagree.
I have one human rater per study. So I do not decide. Both readings converge on the same practical conclusion — do not treat that annotation layer as ground truth for inferential features — but they disagree about where the fault lies, and that difference matters more to me than the practical one.
Worth adding: if two humans do disagree, that would not automatically condemn the rule. There is a serious argument in the annotation literature that human label variation on interpretive tasks is signal about the task rather than noise to be minimised. That would make "unstable definition versus legitimately variable judgement" the next question rather than the closing one.
Two things I had to reconstruct
Honesty about the machinery, since it affects what you can verify.
The Gemini and Grok per-scene label files were lost. Their per-rule κ and agreement values survived. It turns out that is enough: with a known human positive count and a known agreement count, the entire confusion table is a function of one unknown, and κ pins it down. Eleven of the twelve rule-by-rater cells came back with a unique solution; the twelfth was settled by a sentence in the original write-up. Those counts are derived, not raw, and the paper says so. Scene-level quantities are gone for good.
The detector's Study 2 labels were likewise reconstructed — from the applied_rules field already published in the corpus, mapped through the published ID mapping. It reproduces the original numbers exactly. It is still a reconstruction.
Both derivation scripts are published so you can check the arithmetic rather than trust it.
What would settle it
In order of value:
- A second independent human rater on the same 100 scenes, blind to all of this. No further model run substitutes for it.
- A balanced evaluation set for the four working rules — sample so each feature is present in roughly half the scenes, or accept that the feature is untested.
- An operational rewrite of the materialized-metaphor rule with worked boundary cases, then re-labelling.
Everything needed for step 1 is open: the 100 scene texts, the six definitions, the locked human labels, every model's labels, the prompt blocks, the ID mapping and the scoring scripts. They are in the evaluation/ directory of the dataset.
- Paper:https://leventbulut.com/llm-annotation-reliability-benchmark/- Dataset: https://huggingface.co/datasets/leventbulut/objective-projection (DOI
10.57967/hf/8960, Zenodo10.5281/zenodo.19511369) - Background: https://leventbulut.com
If you label those hundred scenes and disagree with my volunteer, that disagreement is more useful to this work than any additional model I could run.
Levent Bulut — Independent Researcher. ORCID 0009-0007-7500-2261. Data and code CC BY-NC-ND 4.0.