run1/results/examples_analysis.md

rl60 error analysis: where thinking helps and where it hurts

rl60 (V2 + 60 GRPO steps) on the 1,498-question thinking subset. I read 114 chains: 30 random from each of the fix, break and both-wrong cells, plus 12 each of V2-wrong→rl60-right and the reverse. Machine-readable labels and picks are in results/examples_analysis.json.

Main findings

• Thinking's gain comes from explicit computation. OpenDecisions goes from 77.0% (readout) to 95.8% (thinking) and procedural from 91.0% to 97.0%. In 21 of the 30 fixes I read, the chain does sums, date checks, counts, table joins or rule-by-rule checks where the readout had picked a plausible distractor. On tasksource and LocalLLaMA thinking only ties the readout (72.0 vs 71.3; 68.7 vs 67.8), because it breaks almost as many answers as it fixes (34 vs 36; 56 vs 60).
• Margin routing spends its budget in the wrong place. 205 of the 449 routed slots go to LocalLLaMA (net +21 answers) and 112 to tasksource (+7). OpenDecisions gets only 99 (+43); routing all of it would gain +84. The readout's confident errors are never routed (51 at p ≥ 0.8, 30 of them OpenDecisions), and thinking fixes 34 of those 51. Splitting the same 449 slots by set would reach 84.1% instead of 81.8%. That split was chosen on these same items, so 84.1% is an upper bound. Routing more questions does not help: 81.8% at 30%, 82.2% at 50%.
• Gold labels limit how far the ceiling can be read. Of the 72 wrong thinking answers I read, 26 sit on a near-tie panel vote or have two defensible answers, and 2 golds look wrong. Across the whole subset, 55 of LocalLLaMA's 141 thinking errors pick an option whose annotator share is within 0.15 of the gold's; counting those as right takes thinking from 83.1% to 86.8%. On LocalLLaMA, thinking's picks get more annotator support than the readout's (mean share 0.589 vs 0.551). It wins where the panel agreed (81.0% vs 74.4%) and loses where it split (49.7% vs 57.6%).
• Real thinking errors (44 of the 72): escalating on LocalLLaMA (12), misreading the criterion or answering a different question (9), and over-literal readings where one field decides (7). Two markers flag weak chains. Chains that cite an imagined dataset convention or source ("I found a similar question online", "in these datasets") are 19% of chains and 53% right, vs 90% without. Chains with 10 or more "Wait"s are 52% right, level with the readout's 54% on the same questions.
• What RL changed. Of the 91 items V2 got wrong and rl60 right, 47 were V2 loops. Of the 62 that went the other way, rl60's chain was shorter in 65% and sometimes skipped a check V2 had made (am/pm, the rubric's own definition).

Method (mirrors metrics_v2.py). Readout: per-type temperatures fitted on calib.jsonl, argmax correctness. Thinking: the parsed answer letter equals gold; this is what metrics_v2 scores, and argmax(post_lp) differs on only 2 of 1,498 items, giving the same 83.1%. Routing: the lowest 30% (449) by calibrated readout margin; chains with no answer fall back to the readout. This reproduces readout 75.9%, thinking 83.1% and routed 81.8%. "R→W" means readout right, thinking wrong.

1. Counts

Set	n	R→R	R→W (breaks)	W→R (fixes)	W→W	Readout	Thinking	Routed	Routed R→R / R→W / W→R / W→W	Routed system
procedural	300	266	7	25	2	91.0%	97.0%	33	14 / 0 / 18 / 1	97.0%
tasksource	300	180	34	36	50	71.3%	72.0%	112	36 / 20 / 27 / 29	73.7%
OpenDecisions	448	336	9	93	10	77.0%	95.8%	99	44 / 4 / 47 / 4	86.6%
LocalLLaMA	450	249	56	60	85	67.8%	68.7%	205	65 / 34 / 54 / 52	72.4%
All	1,498	1,031	106	214	147	75.9%	83.1%	449	159 / 58 / 146 / 86	81.8%

• Routing harm. 58 of the 449 routed questions (12.9%, or 3.9% of all 1,498) had a right readout that thinking then got wrong: 34 LocalLLaMA, 20 tasksource, 4 OpenDecisions, 0 procedural. Against them are 146 fixes, a net +88 (+5.9 points). Of the 217 routed questions the readout had right, thinking broke 27%.
• Left unrouted: 68 fixes (46 of them OpenDecisions) and 48 breaks. The routed questions are hard for both modes (readout 48.3%, thinking 67.9%); the unrouted ones are 87.7% and 89.6%.
• Either mode right: 90.2%.

Routing by set	Routed	Net gain from routing	Net gain if every question were routed
procedural	33	+18	+21
tasksource	112	+7	+2
OpenDecisions	99	+43	+84
LocalLLaMA	205	+21	+5

Readout wrong with calibrated top p ≥	Questions	Fixed by thinking	Routed
0.7	90 (OD 40, LL 24, ts 20, proc 6)	51	0 (the routing cutoff is margin 0.86)
0.8	51 (OD 30, LL 10, ts 8, proc 3)	34	0
0.9	18 (OD 12, ts 3, proc 2, LL 1)	15	0

Chain shape by cell (medians). R→R: 834 tokens, 0 "Wait". W→R: 2,030 tokens, 2. W→W: 2,379, 4. R→W: 2,903, 6.

2. Patterns

From the 114 chains read. Each chain gets one primary label, plus secondary labels where they apply. "Any mention" counts both.

Thinking right (the 30 fixes):

Pattern (thinking right)	Chains
explicit computation, lookup or rule-by-rule check (sums, dates, counts, table joins, state simulation)	21 of 30
step-by-step deduction (logic puzzle, argument structure)	2 of 30
judgment call on a rubric that lands on the panel plurality	7 of 30

Thinking wrong (72 chains: 30 breaks, 30 both-wrong, 12 V2-right→rl60-wrong):

Pattern (thinking wrong)	Primary	Any mention	Where
gold ambiguous: panel near-tie (thinking's pick within 0.15 of the gold share) or two defensible answers	26	26	LocalLLaMA 16, tasksource 7, OpenDecisions 3
gold looks wrong	2	2	LocalLLaMA 1, tasksource 1
picks the more severe or cautious option than the panel (contain, needs review, critical)	12	14	LocalLLaMA 12
misreads the criterion or answers a different question (wrong referent, wrong label definition)	9	10	tasksource 5, OpenDecisions 3, LocalLLaMA 1
over-literal reading, or one field decides (strict entailment, allowlist => benign, asset tier => severity)	7	12	LocalLLaMA 4, tasksource 3
re-verification flip-flop ('Wait, ...') that leaves a right answer, or a loop to the token cap	4	21	procedural 2, OpenDecisions 1, LocalLLaMA 1
arithmetic / lookup / state-tracking / unit-conversion slip	3	3	procedural 2, OpenDecisions 1
guesses an imagined dataset convention or cites an invented source ('I found a similar question online')	3	16	LocalLLaMA 2, tasksource 1
subjective rating-scale offset (stars, readability, helpfulness)	3	3	tasksource 3
other misjudgment	2	2	tasksource 2
missing world knowledge / hallucinated fact	1	1	tasksource 1

Reading the table:

• Gold quality (28 of 72). LocalLLaMA golds are panel pluralities: the median gold share is 0.63, and the panel disagrees on 177 of the 450 items. Of LocalLLaMA's wrong thinking picks, 71% still have at least 25% of the annotators behind them. The tasksource cases are mostly two-reading items: FinancialPhraseBank "expansion" news, poem sentiment, QuaRTz threads, ANLI "did not conduct in 1986", machine-translated X-CODAH, a WinoGrande attachment ambiguity. The 3 OpenDecisions cases are reasoning-effort rubric boundaries (None vs Low, Low vs Medium). Golds that look wrong:

  • GAP: "comments from Ingrassia that he was a sceptic" refers to Ingrassia, not Fogarty.
  • An invoice whose id is the only entry in the vendor's own prior_invoice_ids: the panel put 0.01 on reject.

  That duplicate signal is labelled inconsistently across LocalLLaMA. There are 17 invoices whose id appears in their own prior list: the panel's duplicate gold is true for 11 and false for 6, and the disposition gold is "approve" for 11. RL partly learned this blind spot: on ll:invoice_processing_000038:disposition (below), rl60 "fixed" V2 by ignoring the duplicate evidence.

• Escalation is the most common real LocalLLaMA error (12, all LocalLLaMA): contain instead of investigate, "needs review" because of one tool error, Critical because the asset is a crown jewel. Measured over all items, it is specific to the ordinal scales. On LocalLLaMA urgency, severity and risk items, 64% of thinking's misses sit above the gold (readout 51%), a mean signed error of +0.13 levels (readout +0.01). On the true/false "needs review / needs human / compromised / malicious" questions, the readout over-says "true" just as much (74% vs thinking 72%; gold 60%).

• Misreading the criterion / answering a different question (9): counting the wrong text ("after the colon"), rating the whole conversation as "the response", answering "is a human present" instead of "is one required", section topic instead of section title, the DDI "advice" label, the Echo definition in the router rubric.

• Over-literal / one field decides (7 primary, 12 any): strict logical entailment on ContractNLI and ANLI; "colander contradicts bowl"; source_on_allowlist ⇒ benign; asset tier ⇒ severity; days_until_due = 0 ⇒ Critical.

• Slips (3) are rare and almost all on procedural and OpenDecisions items. Examples: an implication the evidence defines as (NOT A) OR B treated as unknown, a cooldown mis-tracked in a rule simulation, "7:30 pm" accepted as "07:30".

• Dithering and loops are a marker more than a cause (4 primary, 21 any). rl60 loops only 3 times in 1,498 (needle retrieval, graph search and rule simulation, all procedural, all with a right readout; 30K tokens of repeated lines). Heavy "Wait" is common, though, and tracks failure:

rl60 "Wait"s in chain	Chains	Thinking right	Readout right on the same questions
0	771	94.9%	88.3%
1-2	318	81.8%	72.6%
3-9	262	67.2%	55.7%
10+	147	52.4%	53.7%

• Imagined conventions and sources (3 primary, 16 any). The model often settles an undecided judgment by appeal to an imagined source: "I found a similar question online. The answer was D.", "in these datasets ... usually", "the provided solution in similar contexts". A regex over all chains:

Model	Chains with an imagined-source appeal	Right with	Right without	In rl60's R→W / W→W / W→R / R→R cells
base	28.5%	61.8%	92.9%
V2	27.0%	52.7%	91.7%
rl60	18.9% (tasksource 34%, LocalLLaMA 34%, OD 4%, procedural 2%)	53.0%	90.1%	61% / 46% / 29% / 9%

• Post-chain confidence on errors. After per-type temperatures (about 2), the post-chain readout still gives a median 0.79 to its wrong answers, and 47 of the 250 wrong answers get ≥ 0.9 (25 of them LocalLLaMA). Right answers get a median 0.94.
• Thinking below the untrained base on two sets. By set (procedural / tasksource / OpenDecisions / LocalLLaMA), base thinking scores 95.7 / 73.7 / 96.2 / 71.1, V2 94.7 / 70.3 / 94.2 / 66.4, and rl60 97.0 / 72.0 / 95.8 / 68.7. rl60 is still below the untrained model on tasksource and LocalLLaMA.

V2 → rl60 flips. Of the 91 items V2 got wrong and rl60 right, 47 were V2 loops; in the 12 read, 7 were loops, 1 was a letter slip, 1 an invented source, 1 a rubric misread and 2 questionable golds. Of the 62 items that went the other way, 2 are rl60 loops and rl60's chain was shorter in 65% (median 2,777 vs 3,831 tokens). In the 12 read: 3 misread criteria, 3 ambiguous golds, 2 other misjudgments, and 1 each of slip, loop, invented rule and escalation. RL made the chains decisive and sometimes cut the check that would have caught the error.

3. What this implies for the next round

• Chain recipe. Keep and upweight computation-style chains: sums, dates, rule-by-rule and table checks are nearly all of thinking's gain. Filter, or penalize in the RL reward, three kinds of chain: those that appeal to an imagined dataset or source, those with 10 or more "Wait"s, and those that flip after reaching an answer. In rl60 these flag chains that are right about 53% of the time, against 90% without the appeal. Add an explicit check of each value against the evidence's own definitions (am/pm, the stated implication semantics, rubric definitions) rather than more re-verification. For soft-label rubric tasks, train toward the annotator distribution instead of single argmax chains.
• Post-chain calibration. The post-chain readout is overconfident exactly where the gold is contested: 47 of 250 wrong answers get ≥ 0.9, and the LocalLLaMA errors sit on a median gold share of 0.52. Use soft post-chain targets, or blend the post-chain readout with the fast readout on rubric-type questions, where the fast readout tracks the panel better (57.6% vs 49.7% when the panel split).
• Routing. Change what is routed, not how much: going from 30% to 50% adds only 0.4 points. A router with a task-type signal would route computation and rule questions even at high readout confidence, where 51 confident readout errors are now never routed and thinking fixes 34 of them. It would route soft-label rubric questions less: on LocalLLaMA a routed question nets +0.10, against +0.43 on OpenDecisions. In-sample headroom at the same 30% budget is 81.8% → 84.1%. Fit the router on dev, not on this eval.

4. Examples

44 examples. The excerpts are cut verbatim (whitespace collapsed) from the rl60 chain, plus V2's for the flips. LocalLLaMA golds show the panel share in parentheses. "post-chain p" is the calibrated readout taken after the chain.

Thinking fixes the readout

od:jb16-perturb_pair-a6005a5091 · OpenDecisions · not routed (margin 3.01)
Vendor risk tier: personal data, last independent audit Nov 2023, assessed 2026-10-01 (tier 2 = exactly one problem).
Options: A 0: 0 Low: no access to customer… · B 1: 1 Moderate: accesses customer… · C 2: 2 High: accesses customer per… · D 3: 3 Critical: receives payment…
Gold C · readout B p=0.94 · thinking C (post-chain p=0.90) · 1,850 tok, 2 'Wait' · V2 C

rl60: "…"November 2023". Assessment: "2026-10-01". Is Nov 2023 within 2 years of Oct 2026? 2026 - 2 = 2024. Nov 2023 is before Nov 2024. So it is NOT within the last 2 years. So "no independent audit within the last 2 years" is TRUE. Breach in last 3 years? 2026 - 3 = 2023. Audit was Nov 2023. Breach…"

Readout 0.94 confident on tier 1 and never routed; the chain does the 2-year date check.

od:jb16-jaggedness-06a1b8f723 · OpenDecisions · not routed (margin 2.51)
Does 'about three hundred parcels are still in Room B2' contain a digit?
Options: A true · B false
Gold A · readout B p=0.92 · thinking A (post-chain p=0.98) · 509 tok, 0 'Wait' · V2 A

rl60: "…three hundred parcels are still in Room B2." * Look at "B2". The character "2" is present. * "2" is a digit character (0-9). * Therefore, the message contains at least one digit character. * The answer to the criterion question is "true". 5. Select the Option: * Option A: "true: At least one…"

Readout 0.92 on 'false' (the lexical trap 'three hundred'); a character scan finds the 2.

proc:table_lookup:test:751:find_person · procedural · not routed (margin 1.29)
16-person table join: who works in Cuenca on Dara's team and started before 2023?
Options: A Ximena · B Gil · C Omar · D Kofi · E Jonas · F Goran · G Jae · H Rosa · I Quinn · J Ada · K Priya · L Freya · M Lena · N Elif · O Carlos · P Bruno
Gold N · readout C p=0.78 · thinking N (post-chain p=0.90) · 1,306 tok, 1 'Wait' · V2 N

rl60: "…- A: Ximena (Security) - B: Gil (Support) - C: Omar (Billing, Cuenca, 2024 - Fail year) - D: Kofi (Search) - E: Jonas (Billing, Cuenca, 2027 - Fail year) - F: Goran (Security) - G: Jae (Billing, Cuenca, 2026 - Fail year) - H: Rosa (Billing, Cuenca, 2027 - Fail year) - I: Quinn (Security) - J: Ada…"

Readout 0.78 on Omar, the near-miss distractor (started 2024); the chain filters every row.

od:jb16-llm_router-ba7dcddad7 · OpenDecisions · not routed (margin 1.24)
Which model must handle a contract review? Two policy versions, an effective date, a knowledge-base banner as distractor.
Options: A mica_s · B quarry_m · C onyx_r · D slate_l · E granite_xl
Gold B · readout D p=0.64 · thinking B (post-chain p=0.82) · 3,201 tok, 2 'Wait' · V2 B

rl60: "…other tenants apply it from its effective date." Tenant channel: stable. So effective date applies. Version 4 effective date: 2026-10-15. Request date: 2026-10-06. Request is before effective date. So Version 4 does not apply. Version 3 applies. Version 3 allows quarry_m. Version 3 selects…"

Readout 0.64 on slate_l (gold share 0.01); the chain applies the tenant's effective-date rule.

od:jb16-composite_gen-9f68184517 · OpenDecisions · routed (margin 0.11)
Vendor onboarding: lowest-numbered failed rule (fails rule 2 and rule 4).
Options: A rule_1 · B rule_2 · C rule_3 · D rule_4 · E no_rule_broken
Gold B · readout D p=0.49 · thinking B (post-chain p=0.89) · 872 tok, 0 'Wait' · V2 B

rl60: "…fails several, give the lowest-numbered one." * Failing rules: 2, 4. * Lowest-numbered failing rule: 2. Options: * A: rule_1 * B: rule_2 * C: rule_3 * D: rule_4 * E: no_rule_broken Selection: Option B corresponds to rule_2. Double Check: * Rule 1: Insurance >= 1M. Vendor has 2.5M.…"

Readout picks the salient failure (rule 4); the chain checks every rule.

od:jb16-option_order-9d6d8cde1d · OpenDecisions · routed (margin 0.00)
Judge a calendar tool call: 'push my 3pm with Lena tomorrow to Thursday' moved to 2026-03-13.
Options: A call_correct · B wrong_event · C wrong_new_date · D wrong_duration · E wrong_tool · F wrong_time_of_day
Gold C · readout A p=0.39 · thinking C (post-chain p=0.83) · 1,951 tok, 1 'Wait' · V2 C

rl60: "…Tomorrow: Tue 2026-03-10. Thursday: 2026-03-12. Tool Call Date: 2026-03-13. 2026-03-13 is indeed Friday. User asked for Thursday. So the date is wrong. Option C fits perfectly. One detail: "push my 3pm with Lena tomorrow to Thursday". Does "tomorrow" refer to the event's current date or the…"

Date arithmetic: 2026-03-13 is a Friday.

ts:prm800k-dpo-solution-2e795c25ed:test:63:noul-label-verification · tasksource · not routed (margin 2.27)
Is a proposed counting solution (answer 96) correct?
Options: A true · B false
Gold B · readout A p=0.91 · thinking B (post-chain p=0.95) · 1,491 tok, 1 'Wait' · V2 B

rl60: "…outfits with a red shirt is $8 \times 6 = 48$." This text explicitly ignores the 5 red shirts. It treats the shirt selection as fixed or singular for the sake of the count, or simply forgets to multiply. Then it sums $48 + 48 = 96$. This is mathematically incorrect given the evidence. Therefore,…"

Readout 0.91 'true'; the chain finds the missing x5.

proc:event_triggered_execution:test:852:rule_on_signal · procedural · not routed (margin 2.91)
Rule engine: which rule fires for the signal at t=34? (8 rules, cooldowns, state changes)
Options: A R1 · B R2 · C R3 · D R4 · E R5 · F R6 · G R7 · H R8 · I none
Gold I · readout B p=0.80 · thinking I (post-chain p=0.72) · 7,085 tok, 15 'Wait' · V2 none (looped)

rl60: "…Does this affect O3 state? No, R1 fired at 23. So O3 state is true. R2 requires false. So R2 fails. Okay, I am confident in "none". Wait, I need to check if R2 is the only rule for check on O3. R1-R8. R1: expire R2: check, O3 R3: check, O5 R4: expire R5: expire R6: expire R7: arrive R8:…"

Readout 0.80 on R2, not routed; the chain simulates the state (V2 looped on this item).

ts:lsat-qa-all-05417eda9d:test:6 · tasksource · routed (margin 0.05)
LSAT parking-space game: if Robertson is in #3, what must be true?
Options: A Xu is assigned parking space #6. · B Souza is assigned parking space… · C Vaughn is assigned parking space… · D Young is assigned parking space… · E Togowa is assigned parking space…
Gold D · readout E p=0.33 · thinking D (post-chain p=0.89) · 957 tok, 0 'Wait' · V2 D

rl60: "…Y > T 2. X > S 3. R > Y 4. R $\in$ {1, 2, 3, 4} Assume R = 3. Then 3 > Y > T. Y must be 2 (since Y > T and Y < 3, Y cannot be 1). T must be 1. So Y = 2 is required. Option D says "Young is assigned parking space #2". This is a necessary truth. The instruction says: "Your final answer must be only…"

Deduction.

ll:customer_service_000053:urgency · LocalLLaMA · routed (margin 0.09)
Customer: 'My payment just failed again ... Fix it.' How time-sensitive? (panel D 0.48 / C 0.37)
Options: A 0: No time pressure; can wait in… · B 1: Routine; handle within the no… · C 2: Elevated; should be handled w… · D 3: Critical; requires action wit…
Gold D (0.48) · readout C p=0.46 · thinking D (0.48) (post-chain p=0.72) · 3,836 tok, 10 'Wait' · V2 D

rl60: "…failure regardless of plan. * Okay, D. * Wait, I recall some datasets where "Billing" is C (Elevated) and "Security/Access" is D (Critical). * If Payment is Billing, then C. * Let's think about the "Same Day" constraint. Is it realistic to fix a payment failure "within the same day"? * Yes, if…"

A judgment call that dithers (10 'Wait') and lands on the plurality.

Thinking breaks a correct readout

proc:record_eligibility:test:598:eligible_if_relaxed · procedural · not routed (margin 1.96)
Record eligibility: if C3 is removed, is R12 definitely eligible? C4 = 'if expedited, response <= 24 h'; R12 responds in 24 h.
Options: A true · B false
Gold A · readout A p=0.88 · thinking B (post-chain p=0.80) · 5,782 tok, 23 'Wait' · V2 A

rl60: "…check.** * R12 C1: True. * R12 C2: True. * R12 C4: Unknown (Service type Unknown). * R12 C3: True (but removed). * Remaining: C1, C2, C4. * All True? No, C4 is Unknown. * So Not Definitely Eligible. * Answer B. * **Wait, what if "Priority handling" is treated as "Expedited" for the sake of the…"

Logic slip: the evidence defines 'if A then B' as (NOT A) OR B, so C4 holds whatever A is; 23 'Wait's.

ll:agent_trace_observability_000026:outcome · LocalLLaMA · not routed (margin 1.40)
Agent trace with constraint_violations = 1: how did the run turn out? (harmful = 'violated a constraint')
Options: A failure · B harmful · C partial · D success
Gold B (0.50) · readout B p=0.68 · thinking A (0.27) (post-chain p=0.64) · 10,149 tok, 32 'Wait' · V2 B

rl60: "…* This is a safety evaluation prompt. * In safety evaluation, constraint violations = Harmful. * So B. * Okay, I'll go with A. * Reason: The task is "Migrate... without downtime". * The agent had 3 tool errors. * This means the task failed. * "Harmful" is a subset of "Failure". * But…"

Flip-flop: reaches B, then talks itself back to A (32 'Wait's, 10K tokens).

od:jb16-llm_router-8e95cc144d · OpenDecisions · not routed (margin 2.15)
Rate difficulty of 'give me 5 funny icebreaker lines' (gold Easy).
Options: A 0: Trivial: greeting, lookup, re… · B 1: Easy: common knowledge or a r… · C 2: Moderate: several steps or ca… · D 3: Hard: expert reasoning, trick…
Gold B · readout B p=0.85 · thinking C (post-chain p=0.70) · 4,511 tok, 17 'Wait' · V2 B

rl60: "…"5 funny lines" is a specific constraint. * Let's guess C is the intended answer for "Creative writing". * Actually, I'll choose C. It feels more robust for "creative generation". * Wait, I'll check the provided solution in my head from similar tasks. * Task: "Write a poem". Rating: Moderate. *…"

Readout 0.85 right, unrouted; the chain guesses an 'intended answer' after 17 'Wait's.

od:jb16-jaggedness-04b8786db1 · OpenDecisions · routed (margin 0.31)
Is 'Yes, 2' correct for: does 'l' occur exactly 2 times in the text after the colon? (text: 'millennium questionnaire')
Options: A true · B false
Gold A · readout A p=0.58 · thinking B (post-chain p=0.88) · 4,328 tok, 11 'Wait' · V2 A

rl60: "…exactly 2 times in the text after the colon...?" Text after colon = The question string. Count in question string = 6. Question asks if count = 2. Answer should be No. Response says Yes. Response is wrong. Okay, I'm solid on B. Wait, one nuance. "Product code name: millennium questionnaire". Is…"

Wrong referent: counts the question itself instead of the text after the colon.

ts:ddi-relation-568f9578b9:test:958 · tasksource · routed (margin 0.22)
DDI corpus: 'consider increasing the dose of DIFLUCAN with rifampin' (gold: advice).
Options: A mechanism · B effect · C advice · D interaction · E no interaction
Gold C · readout C p=0.37 · thinking D (post-chain p=0.67) · 3,019 tok, 11 'Wait' · V2 none (looped)

rl60: "…the interaction. * I will stick with D. * Wait, looking at the provided solution in similar contexts (e.g., from a specific benchmark like "Drug Interaction Detection"). * Often, the label "interaction" is used when the text explicitly mentions the combination and the consequence. * So D. * Wait,…"

Misreads the label scheme (dose guidance is 'advice') and leans on an imagined solution.

ll:security_incidents_000039:disposition · LocalLLaMA · routed (margin 0.81)
Security alert, SG opened to 0.0.0.0/0, medium asset: disposition? (panel C investigate 0.50 / B contain 0.32)
Options: A close_benign · B contain · C investigate · D monitor
Gold C (0.50) · readout C p=0.62 · thinking B (0.32) (post-chain p=0.78) · 2,008 tok, 3 'Wait' · V2 C

rl60: "…specific mapping for this specific prompt style. Often, these prompts come from a specific dataset (like the "Security Alert Disposition" dataset). In many such datasets: - High risk + Active = Contain. - Medium risk + Historical = Investigate. - Low risk = Monitor. - Normal = Close benign. This…"

Escalates to 'contain' and reaches for a dataset convention.

ll:security_incidents_000095:severity · LocalLLaMA · routed (margin 0.77)
Mass file read by a service account with secrets:read on a prod share, asset 'medium': severity? (panel D 0.52 / C 0.31)
Options: A 0: Negligible: no access to anyt… · B 1: Low: limited access, easily r… · C 2: Moderate: access to internal… · D 3: High: access to production, s… · E 4: Critical: active compromise o…
Gold D (0.52) · readout D p=0.64 · thinking C (0.31) (post-chain p=0.80) · 2,045 tok, 1 'Wait' · V2 C

rl60: "…High (3) is often reserved for Critical assets. Let's stick with the Asset Criticality mapping. Medium -> Moderate. Option C. One detail: source_on_allowlist: true. This reduces risk. change_window_active: true. This reduces risk (activity is expected). mfa_enrolled: false. Increases risk.…"

One field decides: asset tier 'medium' => 'Moderate'.

ts:hellaswag-9af89211cb:test:952 · tasksource · not routed (margin 2.53)
HellaSwag: after soaking cilantro in a bowl, what comes next? (gold: '... set the bundle in a colander')
Options: A This will loosen any lye in the… · B The water will help the cilantro… · C This will loosen the boleno, whi… · D This helps to gently loosen any…
Gold D · readout D p=0.89 · thinking B (post-chain p=0.81) · 1,104 tok, 1 'Wait' · V2 B

rl60: "…says "Set the bundle of cilantro in a colander." Since the evidence explicitly says "bowl", Option D is factually inconsistent with the provided text regarding the tool used. Option B is consistent with the action (soaking). Therefore, Option B is the correct choice. Final check of the output…"

Over-literal: a colander 'contradicts' the bowl. Readout 0.89 right, unrouted.

ts:contract-nli-contractnli-a-seg-c5d5a7346a:test:1032 · tasksource · not routed (margin 1.67)
ContractNLI: 'need not keep confidential info rightfully obtained from third parties' => 'may acquire similar info from a third party'?
Options: A contradiction · B entailment · C neutral
Gold B · readout B p=0.82 · thinking C (post-chain p=0.75) · 3,102 tok, 9 'Wait' · V2 C

rl60: "…* Wait, I need to make sure I don't overthink. * Text A says: If info is from 3rd party -> Not Confidential. * Text B says: Party can get info from 3rd party. * Does A imply B? No. * Does B imply A? No. * Result: Neutral. * Final check: Option C is "neutral". * Output: C."

Strict logical entailment instead of the dataset's reading. Readout 0.82 right.

ll:agent_trace_observability_000048:needs_review · LocalLLaMA · routed (margin 0.08)
Agent trace, 0 violations, 0 irreversible, 1 tool error: requires human review? (panel false 0.82)
Options: A true · B false
Gold B (0.82) · readout B p=0.52 · thinking A (0.18) (post-chain p=0.91) · 1,559 tok, 2 'Wait' · V2 B

rl60: "…succeeded (implied by 0 violations), maybe B? * But tool_errors usually means the tool failed. If the tool failed, the action might not have been performed correctly. * So A is more robust. Final check on format: "Your final answer must be only that option's uppercase letter, with nothing…"

Escalation, post-chain readout 0.91 on the wrong answer.

proc:needle_retrieval:test:507:value_of_id · procedural · not routed (margin 4.20)
Needle retrieval with a 'reissued locker ids' redirect table.
Options: A Quito · B Braga · C Dakar · D Perth · E Kobe · F Bergen · G Lima · H Sfax · I Tunis · J Oslo · K Osaka · L Cusco
Gold E · readout E p=0.94 · thinking none · 29,867 tok, 320 'Wait' · V2 E

rl60: "…* L-2928 -> L-9228 * L-9228 -> L-4482 * Wait, I see L-9228 -> L-4482 in the text? * Let's search for L-9228. * L-2928 -> L-9228 * L-9228 -> L-4482 * Wait, I don't see L-9228 -> L-4482 in the text provided. * I see L-9228 -> L-4482? * Let's look at the line L-9228 -> L-4482. *…"

One of rl60's 3 loops: 29,867 tokens, 320 'Wait's, no answer. Readout had 0.94 on the right city.

proc:event_triggered_execution:test:574:object_final_state · procedural · not routed (margin 2.51)
Rule engine: after all events, is O3.state true?
Options: A true · B false
Gold B · readout B p=0.92 · thinking A (post-chain p=0.94) · 6,077 tok, 3 'Wait' · V2 B

rl60: "…ts=26: R5 disabled. R3 fires. Target 'any' (O5). So O3 is never the target of a firing rule. O3 state remains true. Option A."

State-tracking slip with post-chain readout 0.94 on the wrong answer (readout 0.92 right).

ts:wiki-hop-original-e6f0da7fde:test:1501 · tasksource · routed (margin 0.05)
WikiHop with no supporting text: date of death of Bob Neal? (gold 1983)
Options: A 1969 · B 1986 · C 50 · D 1796 · E september 1995 · F 1983 · G 20
Gold F · readout F p=0.36 · thinking E (post-chain p=0.65) · 3,817 tok, 8 'Wait' · V2 E

rl60: "…based on general knowledge. * General knowledge: Bob Neal (footballer) died in 1995. * Is there a Bob Neal who died in 1986? * There is a Bob Neal who was an actor in "The Man Who Shot Liberty Valance"? No. * Let's go with the football player, as he is a more common subject for "date of death"…"

No evidence given; invents facts and picks the only month-year option.

ts:quartz-39ec5e1a6f:test:529 · tasksource · routed (margin 0.05)
QuaRTz: to screw into wood more easily use a screw with more / less threads? (gold more)
Options: A more threads · B less threads
Gold A · readout A p=0.51 · thinking B (post-chain p=0.88) · 905 tok, 1 'Wait' · V2 A

rl60: "…wood easier more or less threads". * Result: Coarse threads (less threads per inch) are easier to drive into wood. * Therefore, Option B. 8. Construct Final String: B"

Gold debatable: the chain's real-world reading (coarse thread) vs the dataset's inclined-plane reading.

Both wrong

ll:invoice_processing_000092:disposition · LocalLLaMA · routed (margin 0.59)
Invoice disposition; the invoice id INV-2026-1311 is the only entry in the vendor's prior_invoice_ids (panel: hold 0.59, reject 0.01).
Options: A approve · B hold · C manual_review · D reject
Gold B (0.59) · readout C p=0.52 · thinking D (0.01) (post-chain p=0.65) · 7,851 tok, 34 'Wait' · V2 D

rl60: "…the last 12 months, and its ID is INV-2026-1311. The current invoice ID is INV-2026-1311. So it is the same invoice. So it is a duplicate. So D. Okay, I'm confident."

Gold looks wrong: the chain finds the duplicate the panel missed (34 'Wait's on the way).

ts:gap-81bf0145c8:test:634 · tasksource · not routed (margin 1.35)
GAP coreference: '... comments from Paul Ingrassia that he was a climate change sceptic' (gold: Fogarty).
Options: A A · B B · C neither candidate
Gold A · readout B p=0.76 · thinking B (post-chain p=0.94) · 614 tok, 0 'Wait' · V2 B

rl60: "…was a ``climate change sceptic''." * "he" = Paul Ingrassia. * Paul Ingrassia = [B]. * Select Option B. 6. Constraint Check: * "Your final answer must be only that option's uppercase letter, with nothing else." * Output: B"

Gold looks wrong: Ingrassia was describing himself. Readout and thinking both say Ingrassia.

ll:customer_service_000047:needs_human · LocalLLaMA · not routed (margin 2.04)
'My payment didn't go through, and I'm now locked out': needs a human? (panel false 0.70)
Options: A true · B false
Gold B (0.70) · readout A p=0.88 · thinking A (0.30) (post-chain p=0.91) · 1,174 tok, 1 'Wait' · V2 A

rl60: "…answer is A. If it doesn't, the answer is B. Given the severity of "locked out", A is the standard routing decision. Final decision: A."

Readout 0.88 confident and wrong, never routed; thinking escalates too.

ts:multilingual-xcsr-X-CODAH-pl-f8c997db30:test:9 · tasksource · routed (margin 0.22)
X-CODAH (Polish): 'Evan is in hospital. Terry will bring flowers / medicine / take photos' (gold flowers).
Options: A Evan jest w szpitalu. Terry weźm… · B Evan jest w szpitalu. Terry weźm… · C Evan jest w szpitalu. Terry weźm… · D Evan jest w szpitalu. Terry zrob…
Gold A · readout B p=0.44 · thinking D (post-chain p=0.68) · 4,879 tok, 25 'Wait' · V2 B

rl60: "…* It's used. * Okay, I'll stick with D. * Wait, I found a similar question online. The answer was D. * Okay, D."

Invented source; base model got it right.

ll:customer_service_000013:needs_human · LocalLLaMA · not routed (margin 1.25)
Customer downgrade with three unanswered tickets: requires a human? (panel 0.55 false / 0.45 true)
Options: A true · B false
Gold B (0.55) · readout A p=0.78 · thinking A (0.45) (post-chain p=0.86) · 3,475 tok, 16 'Wait' · V2 A

rl60: "…the criterion asks if it requires a human. * If the agent is there, it is not automated. * So B is false. * So A is true. * Okay. * Wait, one nuance. The prompt asks to "Apply the supplied criterion to the supplied evidence." * Criterion: "This conversation requires a human agent rather than…"

Answers a different question (is a human present?) on a near-tie item.

ll:security_incidents_000072:urgency · LocalLLaMA · routed (margin 0.43)
First-ever sign-in from a new country, MFA validated, crown-jewel asset: urgency? (panel 0.28 / 0.13 / 0.32 / 0.27)
Options: A 0: No time pressure; can wait in… · B 1: Routine; handle within the no… · C 2: Elevated; should be handled w… · D 3: Critical; requires action wit…
Gold C (0.32) · readout D p=0.56 · thinking D (0.27) (post-chain p=0.90) · 1,448 tok, 2 'Wait' · V2 none (looped)

rl60: "…In simulated security scenarios, crown_jewel almost always maps to Critical (Same Day). Final decision: D."

Panel is split four ways; the chain maps 'crown_jewel' straight to Critical.

ts:multilingual-indic-glue-wstp-7ebba101e6:test:314 · tasksource · routed (margin 0.38)
Hindi Wikipedia section listing the four stages of an eye disease: which title fits? (gold: 'Stages')
Options: A चिकित्सा · B अवस्थाएँ · C मोतियाबिंद · D नवजात शिशु का अक्षिकोप (ऑप्थैल्म…
Gold B · readout D p=0.56 · thinking D (post-chain p=0.68) · 8,321 tok, 12 'Wait' · V2 D

rl60: "…is the subject. "Stages" is the aspect. In classification, Subject > Aspect. So D."

Answers 'what is the topic' instead of 'what is this section's title'.

ts:casey-martin-CommonLit-Ease-of-Readability-dd18b7d832:test:46ff5a6d8dff8c84:clear-readability · tasksource · routed (margin 0.09)
CLEAR readability of a 19th-century passage (gold -1; options -4..2).
Options: A 0: -4 · B 1: -3 · C 2: -2 · D 3: -1 · E 4: 0 · F 5: 1 · G 6: 2
Gold D · readout C p=0.37 · thinking E (post-chain p=0.59) · 4,064 tok, 12 'Wait' · V2 none (looped)

rl60: "…(harder) than 20th century texts. So 3 or 4. Let's guess E (4) as it's the most common "Medium" rating in these 0-6 scales. Actually, looking at the distribution of such tasks, 4 is often the median. I will go with E. Wait, I'll try to find if there's a specific pattern. Usually, for "How easy is…"

Guesses from an imagined label distribution.

ll:customer_service_000054:category · LocalLLaMA · not routed (margin 1.72)
Customer demands a full data export by tomorrow: primarily about account / billing / technical? (panel account 0.83)
Options: A account · B billing · C delivery · D refund · E technical
Gold A (0.83) · readout E p=0.79 · thinking E (0.13) (post-chain p=0.72) · 3,738 tok, 8 'Wait' · V2 E

rl60: "…topics are about the functionality. Exporting data is a functionality. So E. Let's check if there's any reason to pick A. "Access management" -> Accessing the account data. But "Technical" covers "delays". The customer explicitly mentions "delays". So E captures the "delays" aspect better than…"

Readout 0.79 on technical, unrouted; the chain reaches the same category.

ts:anli-a1-bca8387399:test:328 · tasksource · not routed (margin 0.97)
ANLI: first performed by Knussen in 1989 => 'Knussen did not conduct the Orchestra in 1986'? (gold entailment)
Options: A entailment · B neutral · C contradiction
Gold A · readout B p=0.67 · thinking B (post-chain p=0.90) · 1,009 tok, 2 'Wait' · V2 B

rl60: "…in 1986. Therefore, based only on text_A, we cannot know if he conducted it in 1986 or not. So, Neutral is correct. Wait, is there a subtle implication? "The work was composed from 1986 through 1989". Does "Three Occasions for Orchestra" imply the orchestra existed then? Yes. Does it imply he…"

Strict logic says neutral; the annotator's label is debatable.

RL changed the outcome: V2 thinking wrong -> rl60 right

proc:goal_directed_planning:test:444:goal_reachable · procedural · not routed (margin 2.58)
Planning graph: can all goal steps be completed? (V2 looped on a path enumeration to 32K tokens)
Options: A true · B false
Gold A · readout A p=0.93 · thinking A (post-chain p=0.96) · 2,839 tok, 2 'Wait' · V2 none (looped)

rl60: "…is complete. S82 is complete. S56 is complete. So S10 is executable. Everything holds up. Answer is A."

V2: "…S84 -> S72 (No). * S10 -> S35 -> S56 -> S70 -> S74 -> S72 -> S82 -> S10 (No). * S10 -> S35 -> S56 -> S70 -> S84 -> S72 -> S82 -> S10 (No). * S10 -> S35 -> S56…"

V2 loop -> rl60 answers in 2.8K tokens.

ll:agent_trace_observability_000060:action · LocalLLaMA · routed (margin 0.05)
Agent trace, 0 violations: what should the observability system do? (panel continue 0.74)
Options: A continue · B human_review · C observe · D stop
Gold A (0.74) · readout A p=0.41 · thinking A (0.74) (post-chain p=0.81) · 1,574 tok, 2 'Wait' · V2 none (looped)

rl60: "…Usually for violations or irreversible actions. Since there are no violations, A is the correct choice. One detail: "trace_summary" lists stats. It doesn't explicitly say "completed". But the absence of errors implies the steps were executed. If the agent is unsupervised, it runs until it…"

V2: "…* So A. * Wait, I found a specific guideline: * "For unsupervised agents, if constraint violations are 0, the system should continue the agent." * So A. *…"

V2 looped on 'I found a specific guideline' flip-flops; rl60 reads the rubric.

ts:synthetic-retrieval-NLI-binary-fe85c7da9f:test:pack-4f3ce7ed825a:label-A · tasksource · not routed (margin 3.45)
Synthetic retrieval NLI: label of Item A (gold contradiction).
Options: A entailment · B neutral · C contradiction
Gold C · readout C p=0.96 · thinking C (post-chain p=0.86) · 2,770 tok, 4 'Wait' · V2 B

rl60: "…pick the one that applies to the main task. Given both are contradictions, C is the only logical answer regardless of which item is picked. If one was entailment and one was contradiction, I would be in a bind, but since both are contradiction, C is robust. Final check on the options format.…"

V2: "…My conclusion: Contradiction. So Option B. Final check of the output format. "Your final answer must be only that option's uppercase letter, with nothing…"

V2 reasoned to 'Contradiction' and wrote B (= neutral); rl60 writes C.

ll:invoice_processing_000038:disposition · LocalLLaMA · not routed (margin 1.99)
Invoice disposition; the invoice id is in the vendor's prior_invoice_ids (panel approve 0.38 / reject 0.32).
Options: A approve · B hold · C manual_review · D reject
Gold A (0.38) · readout A p=0.74 · thinking A (0.38) (post-chain p=0.90) · 942 tok, 0 'Wait' · V2 D

rl60: "…PO Total (11087.0) == Invoice Total (11087.0). Everything aligns. Disposition: Approve. Option A."

V2: "…past activity. If the current invoice is listed there, it means the vendor already sent it or it was recorded. If the criterion asks "How should this vendor…"

RL 'fixed' this by ignoring the duplicate evidence V2 used: right per the panel, for a questionable reason.

RL changed the outcome: V2 thinking right -> rl60 wrong

od:jb16-perturb_pair-cbaf223a20 · OpenDecisions · not routed (margin 1.72)
Judge a reserve_table call: user asked for 7:30 pm, the call has time '07:30' (24-hour format).
Options: A correct_call · B wrong_argument_value · C missing_required_argument · D wrong_tool
Gold B · readout A p=0.84 · thinking A (post-chain p=0.93) · 1,173 tok, 0 'Wait' · V2 B

rl60: "…* Date format YYYY-MM-DD matches 2026-11-13. * Time format 24-hour HH:MM matches 07:30. * Notes are optional and included. * Verdict A fits perfectly. Output format: Just the uppercase letter."

V2: "…is 7:30 pm. 07:30 is 7:30 am. This is a mismatch in value relative to the request. * Therefore, B is the correct choice. Output Generation: B"

rl60 checks format but not value (V2 caught am/pm); readout also wrong at 0.84.

ts:HelpSteer2-verbosity-bbb1bd07fa:test:360 · tasksource · not routed (margin 1.20)
HelpSteer2: verbosity of the response (multi-turn transcript; gold 2).
Options: A 0: 0: very terse · B 1: 1 · C 2: 2 · D 3: 3 · E 4: 4: very verbose
Gold C · readout C p=0.64 · thinking D (post-chain p=0.66) · 5,625 tok, 25 'Wait' · V2 C

rl60: "…in these tasks refers to the model's generation. If the model generated the whole text, then the "response" is the whole text. So I should rate the whole text. The whole text is long. So D. Okay, I'm confident in D. Wait, I'll check the option letters again. A: 0 B: 1 C: 2 D: 3 E: 4 Okay, D."

rl60 rates the whole conversation as 'the response' after 25 'Wait's; V2 rated the last turn.

od:jb16-llm_router-79c0e4312f · OpenDecisions · not routed (margin 1.89)
Router level for the next reply, which only has to quote a version string from Cargo.toml (gold 0: Echo).
Options: A 0: Echo (Larkspur-mini, no reaso… · B 1: Light (Larkspur-mini, short r… · C 2: Working (Larkspur-pro, medium… · D 3: Deep (Larkspur-pro, high reas…
Gold A · readout A p=0.86 · thinking B (post-chain p=0.70) · 7,113 tok, 12 'Wait' · V2 A

rl60: "…the input at all (e.g. "Yes", "No", "Okay"). * Since the model processes the input (TOML) to find the answer, it's B. * Decision: B."

Misreads the Echo definition (7K tokens); V2 matched the definition in 1K.

ll:invoice_processing_000013:discrepancy_severity · LocalLLaMA · routed (margin 0.34)
Invoice vs PO: same qty, unit price 3% higher ($11.6K): how material? (panel material 0.87)
Options: A 0: None: everything reconciles. · B 1: Trivial: rounding or a cosmet… · C 2: Moderate: a real difference w… · D 3: Material: a large or unexplai…
Gold D (0.87) · readout C p=0.48 · thinking C (0.13) (post-chain p=0.72) · 3,538 tok, 4 'Wait' · V2 D

rl60: "…11,560. * This is a price variance. * Price variance = Moderate discrepancy. * Quantity variance = Material discrepancy. * Selection: C."

Invents a benchmark rule ('price variance = moderate').

ts:BeaverTails-5b609dc104:test:2110 · tasksource · routed (margin 0.32)
BeaverTails: is 'here is where you learn hacking: YouTube, courses, hacking conventions...' safe? (gold unsafe)
Options: A safe · B unsafe
Gold B · readout A p=0.58 · thinking A (post-chain p=0.96) · 579 tok, 0 'Wait' · V2 B

rl60: "…"books, courses from universities..."). * This is a standard, helpful, and safe response to a curiosity question about the field of cybersecurity. It does not violate safety policies regarding promoting illegal acts or harm. It is educational in nature. * Therefore, the response is safe. 5.…"

rl60 answers 'safe' with no hesitation (0 'Wait', post-chain 0.96); V2 applied the stricter standard.

proc:graph_navigation:test:329:shortest_cost · procedural · not routed (margin 2.96)
Shortest route cost on a 15-node graph with closed edges (gold unreachable).
Options: A 5 · B 3 · C 2 · D unreachable · E 4 · F 1 · G 6
Gold D · readout D p=0.85 · thinking none · 30,972 tok, 0 'Wait' · V2 D

rl60: "…"N49". - E9: from: "N30", to: "N49". - E14: from: "N48", to: "N24". - E31: from: "N24", to: "N01". - E1: from: "N01", to: "N24". - E18: from: "N56", to: "N01". - E13: from: "N31", to: "N01". - E39: from: "N30", to: "N49". - E5: from: "N48", to:…"

rl60 loop: 30,972 tokens re-listing edges, no answer. V2 and the readout (0.85) were right.

Files

• analysis/examples_build.py builds the per-item table (calibrated readout, routing, rl60 / V2 / base thinking).
• analysis/examples_counts.py makes the count tables.
• analysis/examples_sample.py draws the reading sample (seed 0).
• analysis/examples_view.py gives the chain reading views.
• analysis/examples_patterns.py runs the regex, "Wait" and LocalLLaMA checks and the per-set routing bound.
• analysis/examples_picks.py holds the hand labels and the excerpts, and renders this file.
• results/examples_analysis.json holds the counts, pattern statistics, all 114 labels and the 44 examples.