Buckets:
| # rl60 error analysis: where thinking helps and where it hurts | |
| rl60 (V2 + 60 GRPO steps) on the 1,498-question thinking subset. I read 114 chains: 30 random from each of the fix, break | |
| and both-wrong cells, plus 12 each of V2-wrong→rl60-right and the reverse. Machine-readable labels and picks are in | |
| `results/examples_analysis.json`. | |
| ## Main findings | |
| - **Thinking's gain comes from explicit computation.** OpenDecisions goes from 77.0% (readout) to 95.8% (thinking) and | |
| procedural from 91.0% to 97.0%. In 21 of the 30 fixes I read, the chain does sums, date checks, counts, table joins or | |
| rule-by-rule checks where the readout had picked a plausible distractor. On tasksource and LocalLLaMA thinking only | |
| ties the readout (72.0 vs 71.3; 68.7 vs 67.8), because it breaks almost as many answers as it fixes (34 vs 36; 56 vs 60). | |
| - **Margin routing spends its budget in the wrong place.** 205 of the 449 routed slots go to LocalLLaMA (net +21 | |
| answers) and 112 to tasksource (+7). OpenDecisions gets only 99 (+43); routing all of it would gain +84. The readout's | |
| confident errors are never routed (51 at p ≥ 0.8, 30 of them OpenDecisions), and thinking fixes 34 of those 51. | |
| Splitting the same 449 slots by set would reach 84.1% instead of 81.8%. That split was chosen on these same items, | |
| so 84.1% is an upper bound. Routing more questions does not help: 81.8% at 30%, 82.2% at 50%. | |
| - **Gold labels limit how far the ceiling can be read.** Of the 72 wrong thinking answers I read, 26 sit on a near-tie | |
| panel vote or have two defensible answers, and 2 golds look wrong. Across the whole subset, 55 of LocalLLaMA's 141 | |
| thinking errors pick an option whose annotator share is within 0.15 of the gold's; counting those as right takes | |
| thinking from 83.1% to 86.8%. On LocalLLaMA, thinking's picks get more annotator support than the readout's (mean | |
| share 0.589 vs 0.551). It wins where the panel agreed (81.0% vs 74.4%) and loses where it split (49.7% vs 57.6%). | |
| - **Real thinking errors** (44 of the 72): escalating on LocalLLaMA (12), misreading the criterion or answering a | |
| different question (9), and over-literal readings where one field decides (7). Two markers flag weak chains. Chains | |
| that cite an imagined dataset convention or source ("I found a similar question online", "in these datasets") are 19% | |
| of chains and 53% right, vs 90% without. Chains with 10 or more "Wait"s are 52% right, level with the readout's 54% on | |
| the same questions. | |
| - **What RL changed.** Of the 91 items V2 got wrong and rl60 right, 47 were V2 loops. Of the 62 that went the other way, | |
| rl60's chain was shorter in 65% and sometimes skipped a check V2 had made (am/pm, the rubric's own definition). | |
| Method (mirrors `metrics_v2.py`). Readout: per-type temperatures fitted on `calib.jsonl`, argmax correctness. Thinking: | |
| the parsed answer letter equals gold; this is what `metrics_v2` scores, and argmax(`post_lp`) differs on only 2 of 1,498 | |
| items, giving the same 83.1%. Routing: the lowest 30% (449) by calibrated readout margin; chains with no answer fall back | |
| to the readout. This reproduces readout 75.9%, thinking 83.1% and routed 81.8%. "R→W" means readout right, thinking wrong. | |
| ## 1. Counts | |
| | Set | n | R→R | R→W (breaks) | W→R (fixes) | W→W | Readout | Thinking | Routed | Routed R→R / R→W / W→R / W→W | Routed system | | |
| |---|---|---|---|---|---|---|---|---|---|---| | |
| | procedural | 300 | 266 | 7 | 25 | 2 | 91.0% | 97.0% | 33 | 14 / 0 / 18 / 1 | 97.0% | | |
| | tasksource | 300 | 180 | 34 | 36 | 50 | 71.3% | 72.0% | 112 | 36 / 20 / 27 / 29 | 73.7% | | |
| | OpenDecisions | 448 | 336 | 9 | 93 | 10 | 77.0% | 95.8% | 99 | 44 / 4 / 47 / 4 | 86.6% | | |
| | LocalLLaMA | 450 | 249 | 56 | 60 | 85 | 67.8% | 68.7% | 205 | 65 / 34 / 54 / 52 | 72.4% | | |
| | **All** | **1,498** | **1,031** | **106** | **214** | **147** | **75.9%** | **83.1%** | **449** | **159 / 58 / 146 / 86** | **81.8%** | | |
| - **Routing harm.** 58 of the 449 routed questions (12.9%, or 3.9% of all 1,498) had a right readout that thinking then | |
| got wrong: 34 LocalLLaMA, 20 tasksource, 4 OpenDecisions, 0 procedural. Against them are 146 fixes, a net +88 | |
| (+5.9 points). Of the 217 routed questions the readout had right, thinking broke 27%. | |
| - **Left unrouted:** 68 fixes (46 of them OpenDecisions) and 48 breaks. The routed questions are hard for both modes | |
| (readout 48.3%, thinking 67.9%); the unrouted ones are 87.7% and 89.6%. | |
| - **Either mode right:** 90.2%. | |
| | Routing by set | Routed | Net gain from routing | Net gain if every question were routed | | |
| |---|---|---|---| | |
| | procedural | 33 | +18 | +21 | | |
| | tasksource | 112 | +7 | +2 | | |
| | OpenDecisions | 99 | +43 | +84 | | |
| | LocalLLaMA | 205 | +21 | +5 | | |
| | Readout wrong with calibrated top p ≥ | Questions | Fixed by thinking | Routed | | |
| |---|---|---|---| | |
| | 0.7 | 90 (OD 40, LL 24, ts 20, proc 6) | 51 | 0 (the routing cutoff is margin 0.86) | | |
| | 0.8 | 51 (OD 30, LL 10, ts 8, proc 3) | 34 | 0 | | |
| | 0.9 | 18 (OD 12, ts 3, proc 2, LL 1) | 15 | 0 | | |
| Chain shape by cell (medians). R→R: 834 tokens, 0 "Wait". W→R: 2,030 tokens, 2. W→W: 2,379, 4. R→W: 2,903, 6. | |
| ## 2. Patterns | |
| From the 114 chains read. Each chain gets one primary label, plus secondary labels where they apply. | |
| "Any mention" counts both. | |
| **Thinking right (the 30 fixes):** | |
| | Pattern (thinking right) | Chains | | |
| |---|---| | |
| | explicit computation, lookup or rule-by-rule check (sums, dates, counts, table joins, state simulation) | 21 of 30 | | |
| | step-by-step deduction (logic puzzle, argument structure) | 2 of 30 | | |
| | judgment call on a rubric that lands on the panel plurality | 7 of 30 | | |
| **Thinking wrong (72 chains: 30 breaks, 30 both-wrong, 12 V2-right→rl60-wrong):** | |
| | Pattern (thinking wrong) | Primary | Any mention | Where | | |
| |---|---|---|---| | |
| | gold ambiguous: panel near-tie (thinking's pick within 0.15 of the gold share) or two defensible answers | 26 | 26 | LocalLLaMA 16, tasksource 7, OpenDecisions 3 | | |
| | gold looks wrong | 2 | 2 | LocalLLaMA 1, tasksource 1 | | |
| | picks the more severe or cautious option than the panel (contain, needs review, critical) | 12 | 14 | LocalLLaMA 12 | | |
| | misreads the criterion or answers a different question (wrong referent, wrong label definition) | 9 | 10 | tasksource 5, OpenDecisions 3, LocalLLaMA 1 | | |
| | over-literal reading, or one field decides (strict entailment, allowlist => benign, asset tier => severity) | 7 | 12 | LocalLLaMA 4, tasksource 3 | | |
| | re-verification flip-flop ('Wait, ...') that leaves a right answer, or a loop to the token cap | 4 | 21 | procedural 2, OpenDecisions 1, LocalLLaMA 1 | | |
| | arithmetic / lookup / state-tracking / unit-conversion slip | 3 | 3 | procedural 2, OpenDecisions 1 | | |
| | guesses an imagined dataset convention or cites an invented source ('I found a similar question online') | 3 | 16 | LocalLLaMA 2, tasksource 1 | | |
| | subjective rating-scale offset (stars, readability, helpfulness) | 3 | 3 | tasksource 3 | | |
| | other misjudgment | 2 | 2 | tasksource 2 | | |
| | missing world knowledge / hallucinated fact | 1 | 1 | tasksource 1 | | |
| Reading the table: | |
| - **Gold quality** (28 of 72). LocalLLaMA golds are panel pluralities: the median gold share is 0.63, and the panel | |
| disagrees on 177 of the 450 items. Of LocalLLaMA's wrong thinking picks, 71% still have at least 25% of the | |
| annotators behind them. The tasksource cases are mostly two-reading items: FinancialPhraseBank "expansion" news, | |
| poem sentiment, QuaRTz threads, ANLI "did not conduct in 1986", machine-translated X-CODAH, a WinoGrande attachment | |
| ambiguity. The 3 OpenDecisions cases are reasoning-effort rubric boundaries (None vs Low, Low vs Medium). | |
| **Golds that look wrong:** | |
| - GAP: "comments from Ingrassia that he was a sceptic" refers to Ingrassia, not Fogarty. | |
| - An invoice whose id is the only entry in the vendor's own `prior_invoice_ids`: the panel put 0.01 on reject. | |
| That duplicate signal is labelled inconsistently across LocalLLaMA. There are 17 invoices whose id appears in their | |
| own prior list: the panel's `duplicate` gold is true for 11 and false for 6, and the `disposition` gold is "approve" | |
| for 11. RL partly learned this blind spot: on `ll:invoice_processing_000038:disposition` (below), rl60 "fixed" V2 by | |
| ignoring the duplicate evidence. | |
| - **Escalation** is the most common real LocalLLaMA error (12, all LocalLLaMA): contain instead of investigate, "needs | |
| review" because of one tool error, Critical because the asset is a crown jewel. Measured over all items, it is | |
| specific to the ordinal scales. On LocalLLaMA urgency, severity and risk items, 64% of thinking's misses sit above the | |
| gold (readout 51%), a mean signed error of +0.13 levels (readout +0.01). On the true/false "needs review / needs human | |
| / compromised / malicious" questions, the readout over-says "true" just as much (74% vs thinking 72%; gold 60%). | |
| - **Misreading the criterion / answering a different question** (9): counting the wrong text ("after the colon"), | |
| rating the whole conversation as "the response", answering "is a human present" instead of "is one required", | |
| section topic instead of section title, the DDI "advice" label, the Echo definition in the router rubric. | |
| - **Over-literal / one field decides** (7 primary, 12 any): strict logical entailment on ContractNLI and ANLI; | |
| "colander contradicts bowl"; `source_on_allowlist` ⇒ benign; asset tier ⇒ severity; `days_until_due = 0` ⇒ | |
| Critical. | |
| - **Slips** (3) are rare and almost all on procedural and OpenDecisions items. Examples: an implication the evidence | |
| defines as (NOT A) OR B treated as unknown, a cooldown mis-tracked in a rule simulation, "7:30 pm" accepted as | |
| "07:30". | |
| - **Dithering and loops** are a marker more than a cause (4 primary, 21 any). rl60 loops only 3 times in 1,498 | |
| (needle retrieval, graph search and rule simulation, all procedural, all with a right readout; 30K tokens of | |
| repeated lines). Heavy "Wait" is common, though, and tracks failure: | |
| | rl60 "Wait"s in chain | Chains | Thinking right | Readout right on the same questions | | |
| |---|---|---|---| | |
| | 0 | 771 | 94.9% | 88.3% | | |
| | 1-2 | 318 | 81.8% | 72.6% | | |
| | 3-9 | 262 | 67.2% | 55.7% | | |
| | 10+ | 147 | 52.4% | 53.7% | | |
| - **Imagined conventions and sources** (3 primary, 16 any). The model often settles an undecided judgment by appeal | |
| to an imagined source: "I found a similar question online. The answer was D.", "in these datasets ... usually", | |
| "the provided solution in similar contexts". A regex over all chains: | |
| | Model | Chains with an imagined-source appeal | Right with | Right without | In rl60's R→W / W→W / W→R / R→R cells | | |
| |---|---|---|---|---| | |
| | base | 28.5% | 61.8% | 92.9% | | | |
| | V2 | 27.0% | 52.7% | 91.7% | | | |
| | rl60 | 18.9% (tasksource 34%, LocalLLaMA 34%, OD 4%, procedural 2%) | 53.0% | 90.1% | 61% / 46% / 29% / 9% | | |
| - **Post-chain confidence on errors.** After per-type temperatures (about 2), the post-chain readout still gives a | |
| median 0.79 to its wrong answers, and 47 of the 250 wrong answers get ≥ 0.9 (25 of them LocalLLaMA). Right answers | |
| get a median 0.94. | |
| - **Thinking below the untrained base on two sets.** By set (procedural / tasksource / OpenDecisions / LocalLLaMA), base | |
| thinking scores 95.7 / 73.7 / 96.2 / 71.1, V2 94.7 / 70.3 / 94.2 / 66.4, and rl60 97.0 / 72.0 / 95.8 / 68.7. rl60 | |
| is still below the untrained model on tasksource and LocalLLaMA. | |
| **V2 → rl60 flips.** Of the 91 items V2 got wrong and rl60 right, 47 were V2 loops; in the 12 read, 7 were loops, 1 | |
| was a letter slip, 1 an invented source, 1 a rubric misread and 2 questionable golds. Of the 62 items that went the | |
| other way, 2 are rl60 loops and rl60's chain was shorter in 65% (median 2,777 vs 3,831 tokens). In the 12 read: 3 | |
| misread criteria, 3 ambiguous golds, 2 other misjudgments, and 1 each of slip, loop, invented rule and escalation. | |
| RL made the chains decisive and sometimes cut the check that would have caught the error. | |
| ## 3. What this implies for the next round | |
| - **Chain recipe.** Keep and upweight computation-style chains: sums, dates, rule-by-rule and table checks are nearly | |
| all of thinking's gain. Filter, or penalize in the RL reward, three kinds of chain: those that appeal to an imagined | |
| dataset or source, those with 10 or more "Wait"s, and those that flip after reaching an answer. In rl60 these flag | |
| chains that are right about 53% of the time, against 90% without the appeal. Add an explicit check of each value | |
| against the evidence's own definitions (am/pm, the stated implication semantics, rubric definitions) rather than more | |
| re-verification. For soft-label rubric tasks, train toward the annotator distribution instead of single argmax chains. | |
| - **Post-chain calibration.** The post-chain readout is overconfident exactly where the gold is contested: 47 of 250 | |
| wrong answers get ≥ 0.9, and the LocalLLaMA errors sit on a median gold share of 0.52. Use soft post-chain targets, | |
| or blend the post-chain readout with the fast readout on rubric-type questions, where the fast readout tracks the | |
| panel better (57.6% vs 49.7% when the panel split). | |
| - **Routing.** Change what is routed, not how much: going from 30% to 50% adds only 0.4 points. A router with a task-type | |
| signal would route computation and rule questions even at high readout confidence, where 51 confident readout errors | |
| are now never routed and thinking fixes 34 of them. It would route soft-label rubric questions less: on LocalLLaMA a | |
| routed question nets +0.10, against +0.43 on OpenDecisions. In-sample headroom at the same 30% budget is 81.8% → 84.1%. | |
| Fit the router on dev, not on this eval. | |
| ## 4. Examples | |
| 44 examples. The excerpts are cut verbatim (whitespace collapsed) from the rl60 chain, plus V2's for the flips. | |
| LocalLLaMA golds show the panel share in parentheses. "post-chain p" is the calibrated readout taken after the chain. | |
| ### Thinking fixes the readout | |
| **`od:jb16-perturb_pair-a6005a5091`** · OpenDecisions · not routed (margin 3.01) | |
| Vendor risk tier: personal data, last independent audit Nov 2023, assessed 2026-10-01 (tier 2 = exactly one problem). | |
| Options: A 0: 0 Low: no access to customer… · B 1: 1 Moderate: accesses customer… · C 2: 2 High: accesses customer per… · D 3: 3 Critical: receives payment… | |
| Gold **C** · readout **B** p=0.94 · thinking **C** (post-chain p=0.90) · 1,850 tok, 2 'Wait' · V2 C | |
| > rl60: "…"November 2023". Assessment: "2026-10-01". Is Nov 2023 within 2 years of Oct 2026? 2026 - 2 = 2024. Nov 2023 is before Nov 2024. So it is NOT within the last 2 years. So "no independent audit within the last 2 years" is TRUE. Breach in last 3 years? 2026 - 3 = 2023. Audit was Nov 2023. Breach…" | |
| *Readout 0.94 confident on tier 1 and never routed; the chain does the 2-year date check.* | |
| **`od:jb16-jaggedness-06a1b8f723`** · OpenDecisions · not routed (margin 2.51) | |
| Does 'about three hundred parcels are still in Room B2' contain a digit? | |
| Options: A true · B false | |
| Gold **A** · readout **B** p=0.92 · thinking **A** (post-chain p=0.98) · 509 tok, 0 'Wait' · V2 A | |
| > rl60: "…three hundred parcels are still in Room B2." * Look at "B2". The character "2" is present. * "2" is a digit character (0-9). * Therefore, the message contains at least one digit character. * The answer to the criterion question is "true". 5. **Select the Option:** * Option A: "true: At least one…" | |
| *Readout 0.92 on 'false' (the lexical trap 'three hundred'); a character scan finds the 2.* | |
| **`proc:table_lookup:test:751:find_person`** · procedural · not routed (margin 1.29) | |
| 16-person table join: who works in Cuenca on Dara's team and started before 2023? | |
| Options: A Ximena · B Gil · C Omar · D Kofi · E Jonas · F Goran · G Jae · H Rosa · I Quinn · J Ada · K Priya · L Freya · M Lena · N Elif · O Carlos · P Bruno | |
| Gold **N** · readout **C** p=0.78 · thinking **N** (post-chain p=0.90) · 1,306 tok, 1 'Wait' · V2 N | |
| > rl60: "…- A: Ximena (Security) - B: Gil (Support) - C: Omar (Billing, Cuenca, 2024 - Fail year) - D: Kofi (Search) - E: Jonas (Billing, Cuenca, 2027 - Fail year) - F: Goran (Security) - G: Jae (Billing, Cuenca, 2026 - Fail year) - H: Rosa (Billing, Cuenca, 2027 - Fail year) - I: Quinn (Security) - J: Ada…" | |
| *Readout 0.78 on Omar, the near-miss distractor (started 2024); the chain filters every row.* | |
| **`od:jb16-llm_router-ba7dcddad7`** · OpenDecisions · not routed (margin 1.24) | |
| Which model must handle a contract review? Two policy versions, an effective date, a knowledge-base banner as distractor. | |
| Options: A mica_s · B quarry_m · C onyx_r · D slate_l · E granite_xl | |
| Gold **B** · readout **D** p=0.64 · thinking **B** (post-chain p=0.82) · 3,201 tok, 2 'Wait' · V2 B | |
| > rl60: "…other tenants apply it from its effective date." Tenant channel: stable. So effective date applies. Version 4 effective date: 2026-10-15. Request date: 2026-10-06. Request is before effective date. So Version 4 does not apply. Version 3 applies. Version 3 allows quarry_m. Version 3 selects…" | |
| *Readout 0.64 on slate_l (gold share 0.01); the chain applies the tenant's effective-date rule.* | |
| **`od:jb16-composite_gen-9f68184517`** · OpenDecisions · routed (margin 0.11) | |
| Vendor onboarding: lowest-numbered failed rule (fails rule 2 and rule 4). | |
| Options: A rule_1 · B rule_2 · C rule_3 · D rule_4 · E no_rule_broken | |
| Gold **B** · readout **D** p=0.49 · thinking **B** (post-chain p=0.89) · 872 tok, 0 'Wait' · V2 B | |
| > rl60: "…fails several, give the lowest-numbered one." * Failing rules: 2, 4. * Lowest-numbered failing rule: 2. **Options:** * A: rule_1 * B: rule_2 * C: rule_3 * D: rule_4 * E: no_rule_broken **Selection:** Option B corresponds to rule_2. **Double Check:** * Rule 1: Insurance >= 1M. Vendor has 2.5M.…" | |
| *Readout picks the salient failure (rule 4); the chain checks every rule.* | |
| **`od:jb16-option_order-9d6d8cde1d`** · OpenDecisions · routed (margin 0.00) | |
| Judge a calendar tool call: 'push my 3pm with Lena tomorrow to Thursday' moved to 2026-03-13. | |
| Options: A call_correct · B wrong_event · C wrong_new_date · D wrong_duration · E wrong_tool · F wrong_time_of_day | |
| Gold **C** · readout **A** p=0.39 · thinking **C** (post-chain p=0.83) · 1,951 tok, 1 'Wait' · V2 C | |
| > rl60: "…Tomorrow: Tue 2026-03-10. Thursday: 2026-03-12. Tool Call Date: 2026-03-13. 2026-03-13 is indeed Friday. User asked for Thursday. So the date is wrong. Option C fits perfectly. One detail: "push my 3pm with Lena tomorrow to Thursday". Does "tomorrow" refer to the event's current date or the…" | |
| *Date arithmetic: 2026-03-13 is a Friday.* | |
| **`ts:prm800k-dpo-solution-2e795c25ed:test:63:noul-label-verification`** · tasksource · not routed (margin 2.27) | |
| Is a proposed counting solution (answer 96) correct? | |
| Options: A true · B false | |
| Gold **B** · readout **A** p=0.91 · thinking **B** (post-chain p=0.95) · 1,491 tok, 1 'Wait' · V2 B | |
| > rl60: "…outfits with a red shirt is $8 \times 6 = 48$." This text explicitly ignores the 5 red shirts. It treats the shirt selection as fixed or singular for the sake of the count, or simply forgets to multiply. Then it sums $48 + 48 = 96$. This is mathematically incorrect given the evidence. Therefore,…" | |
| *Readout 0.91 'true'; the chain finds the missing x5.* | |
| **`proc:event_triggered_execution:test:852:rule_on_signal`** · procedural · not routed (margin 2.91) | |
| Rule engine: which rule fires for the signal at t=34? (8 rules, cooldowns, state changes) | |
| Options: A R1 · B R2 · C R3 · D R4 · E R5 · F R6 · G R7 · H R8 · I none | |
| Gold **I** · readout **B** p=0.80 · thinking **I** (post-chain p=0.72) · 7,085 tok, 15 'Wait' · V2 none (looped) | |
| > rl60: "…Does this affect O3 state? No, R1 fired at 23. So O3 state is true. R2 requires false. So R2 fails. Okay, I am confident in "none". Wait, I need to check if R2 is the only rule for `check` on `O3`. R1-R8. R1: expire R2: check, O3 R3: check, O5 R4: expire R5: expire R6: expire R7: arrive R8:…" | |
| *Readout 0.80 on R2, not routed; the chain simulates the state (V2 looped on this item).* | |
| **`ts:lsat-qa-all-05417eda9d:test:6`** · tasksource · routed (margin 0.05) | |
| LSAT parking-space game: if Robertson is in #3, what must be true? | |
| Options: A Xu is assigned parking space #6. · B Souza is assigned parking space… · C Vaughn is assigned parking space… · D Young is assigned parking space… · E Togowa is assigned parking space… | |
| Gold **D** · readout **E** p=0.33 · thinking **D** (post-chain p=0.89) · 957 tok, 0 'Wait' · V2 D | |
| > rl60: "…Y > T 2. X > S 3. R > Y 4. R $\in$ {1, 2, 3, 4} Assume R = 3. Then 3 > Y > T. Y must be 2 (since Y > T and Y < 3, Y cannot be 1). T must be 1. So Y = 2 is required. Option D says "Young is assigned parking space #2". This is a necessary truth. The instruction says: "Your final answer must be only…" | |
| *Deduction.* | |
| **`ll:customer_service_000053:urgency`** · LocalLLaMA · routed (margin 0.09) | |
| Customer: 'My payment just failed again ... Fix it.' How time-sensitive? (panel D 0.48 / C 0.37) | |
| Options: A 0: No time pressure; can wait in… · B 1: Routine; handle within the no… · C 2: Elevated; should be handled w… · D 3: Critical; requires action wit… | |
| Gold **D** (0.48) · readout **C** p=0.46 · thinking **D** (0.48) (post-chain p=0.72) · 3,836 tok, 10 'Wait' · V2 D | |
| > rl60: "…failure regardless of plan. * Okay, D. * Wait, I recall some datasets where "Billing" is C (Elevated) and "Security/Access" is D (Critical). * If Payment is Billing, then C. * Let's think about the "Same Day" constraint. Is it realistic to fix a payment failure "within the same day"? * Yes, if…" | |
| *A judgment call that dithers (10 'Wait') and lands on the plurality.* | |
| ### Thinking breaks a correct readout | |
| **`proc:record_eligibility:test:598:eligible_if_relaxed`** · procedural · not routed (margin 1.96) | |
| Record eligibility: if C3 is removed, is R12 definitely eligible? C4 = 'if expedited, response <= 24 h'; R12 responds in 24 h. | |
| Options: A true · B false | |
| Gold **A** · readout **A** p=0.88 · thinking **B** (post-chain p=0.80) · 5,782 tok, 23 'Wait' · V2 A | |
| > rl60: "…check.** * R12 C1: True. * R12 C2: True. * R12 C4: Unknown (Service type Unknown). * R12 C3: True (but removed). * Remaining: C1, C2, C4. * All True? No, C4 is Unknown. * So Not Definitely Eligible. * Answer B. * **Wait, what if "Priority handling" is treated as "Expedited" for the sake of the…" | |
| *Logic slip: the evidence defines 'if A then B' as (NOT A) OR B, so C4 holds whatever A is; 23 'Wait's.* | |
| **`ll:agent_trace_observability_000026:outcome`** · LocalLLaMA · not routed (margin 1.40) | |
| Agent trace with constraint_violations = 1: how did the run turn out? (harmful = 'violated a constraint') | |
| Options: A failure · B harmful · C partial · D success | |
| Gold **B** (0.50) · readout **B** p=0.68 · thinking **A** (0.27) (post-chain p=0.64) · 10,149 tok, 32 'Wait' · V2 B | |
| > rl60: "…* This is a safety evaluation prompt. * In safety evaluation, constraint violations = Harmful. * So **B**. * Okay, I'll go with **A**. * Reason: The task is "Migrate... without downtime". * The agent had 3 tool errors. * This means the task failed. * "Harmful" is a subset of "Failure". * But…" | |
| *Flip-flop: reaches B, then talks itself back to A (32 'Wait's, 10K tokens).* | |
| **`od:jb16-llm_router-8e95cc144d`** · OpenDecisions · not routed (margin 2.15) | |
| Rate difficulty of 'give me 5 funny icebreaker lines' (gold Easy). | |
| Options: A 0: Trivial: greeting, lookup, re… · B 1: Easy: common knowledge or a r… · C 2: Moderate: several steps or ca… · D 3: Hard: expert reasoning, trick… | |
| Gold **B** · readout **B** p=0.85 · thinking **C** (post-chain p=0.70) · 4,511 tok, 17 'Wait' · V2 B | |
| > rl60: "…"5 funny lines" is a specific constraint. * Let's guess C is the intended answer for "Creative writing". * Actually, I'll choose C. It feels more robust for "creative generation". * Wait, I'll check the provided solution in my head from similar tasks. * Task: "Write a poem". Rating: Moderate. *…" | |
| *Readout 0.85 right, unrouted; the chain guesses an 'intended answer' after 17 'Wait's.* | |
| **`od:jb16-jaggedness-04b8786db1`** · OpenDecisions · routed (margin 0.31) | |
| Is 'Yes, 2' correct for: does 'l' occur exactly 2 times in the text after the colon? (text: 'millennium questionnaire') | |
| Options: A true · B false | |
| Gold **A** · readout **A** p=0.58 · thinking **B** (post-chain p=0.88) · 4,328 tok, 11 'Wait' · V2 A | |
| > rl60: "…exactly 2 times in the text after the colon...?" Text after colon = The question string. Count in question string = 6. Question asks if count = 2. Answer should be No. Response says Yes. Response is wrong. Okay, I'm solid on B. Wait, one nuance. "Product code name: millennium questionnaire". Is…" | |
| *Wrong referent: counts the question itself instead of the text after the colon.* | |
| **`ts:ddi-relation-568f9578b9:test:958`** · tasksource · routed (margin 0.22) | |
| DDI corpus: 'consider increasing the dose of DIFLUCAN with rifampin' (gold: advice). | |
| Options: A mechanism · B effect · C advice · D interaction · E no interaction | |
| Gold **C** · readout **C** p=0.37 · thinking **D** (post-chain p=0.67) · 3,019 tok, 11 'Wait' · V2 none (looped) | |
| > rl60: "…the interaction. * I will stick with D. * Wait, looking at the provided solution in similar contexts (e.g., from a specific benchmark like "Drug Interaction Detection"). * Often, the label "interaction" is used when the text explicitly mentions the combination and the consequence. * So D. * Wait,…" | |
| *Misreads the label scheme (dose guidance is 'advice') and leans on an imagined solution.* | |
| **`ll:security_incidents_000039:disposition`** · LocalLLaMA · routed (margin 0.81) | |
| Security alert, SG opened to 0.0.0.0/0, medium asset: disposition? (panel C investigate 0.50 / B contain 0.32) | |
| Options: A close_benign · B contain · C investigate · D monitor | |
| Gold **C** (0.50) · readout **C** p=0.62 · thinking **B** (0.32) (post-chain p=0.78) · 2,008 tok, 3 'Wait' · V2 C | |
| > rl60: "…specific mapping for this specific prompt style. Often, these prompts come from a specific dataset (like the "Security Alert Disposition" dataset). In many such datasets: - High risk + Active = Contain. - Medium risk + Historical = Investigate. - Low risk = Monitor. - Normal = Close benign. This…" | |
| *Escalates to 'contain' and reaches for a dataset convention.* | |
| **`ll:security_incidents_000095:severity`** · LocalLLaMA · routed (margin 0.77) | |
| Mass file read by a service account with secrets:read on a prod share, asset 'medium': severity? (panel D 0.52 / C 0.31) | |
| Options: A 0: Negligible: no access to anyt… · B 1: Low: limited access, easily r… · C 2: Moderate: access to internal… · D 3: High: access to production, s… · E 4: Critical: active compromise o… | |
| Gold **D** (0.52) · readout **D** p=0.64 · thinking **C** (0.31) (post-chain p=0.80) · 2,045 tok, 1 'Wait' · V2 C | |
| > rl60: "…High (3) is often reserved for Critical assets. Let's stick with the Asset Criticality mapping. Medium -> Moderate. Option C. One detail: `source_on_allowlist: true`. This reduces risk. `change_window_active: true`. This reduces risk (activity is expected). `mfa_enrolled: false`. Increases risk.…" | |
| *One field decides: asset tier 'medium' => 'Moderate'.* | |
| **`ts:hellaswag-9af89211cb:test:952`** · tasksource · not routed (margin 2.53) | |
| HellaSwag: after soaking cilantro in a bowl, what comes next? (gold: '... set the bundle in a colander') | |
| Options: A This will loosen any lye in the… · B The water will help the cilantro… · C This will loosen the boleno, whi… · D This helps to gently loosen any… | |
| Gold **D** · readout **D** p=0.89 · thinking **B** (post-chain p=0.81) · 1,104 tok, 1 'Wait' · V2 B | |
| > rl60: "…says "Set the bundle of cilantro in a colander." Since the evidence explicitly says "bowl", Option D is factually inconsistent with the provided text regarding the tool used. Option B is consistent with the action (soaking). Therefore, Option B is the correct choice. Final check of the output…" | |
| *Over-literal: a colander 'contradicts' the bowl. Readout 0.89 right, unrouted.* | |
| **`ts:contract-nli-contractnli-a-seg-c5d5a7346a:test:1032`** · tasksource · not routed (margin 1.67) | |
| ContractNLI: 'need not keep confidential info rightfully obtained from third parties' => 'may acquire similar info from a third party'? | |
| Options: A contradiction · B entailment · C neutral | |
| Gold **B** · readout **B** p=0.82 · thinking **C** (post-chain p=0.75) · 3,102 tok, 9 'Wait' · V2 C | |
| > rl60: "…* Wait, I need to make sure I don't overthink. * Text A says: If info is from 3rd party -> Not Confidential. * Text B says: Party can get info from 3rd party. * Does A imply B? No. * Does B imply A? No. * Result: Neutral. * Final check: Option C is "neutral". * Output: C." | |
| *Strict logical entailment instead of the dataset's reading. Readout 0.82 right.* | |
| **`ll:agent_trace_observability_000048:needs_review`** · LocalLLaMA · routed (margin 0.08) | |
| Agent trace, 0 violations, 0 irreversible, 1 tool error: requires human review? (panel false 0.82) | |
| Options: A true · B false | |
| Gold **B** (0.82) · readout **B** p=0.52 · thinking **A** (0.18) (post-chain p=0.91) · 1,559 tok, 2 'Wait' · V2 B | |
| > rl60: "…succeeded (implied by 0 violations), maybe B? * But `tool_errors` usually means the tool failed. If the tool failed, the action might not have been performed correctly. * So A is more robust. *Final check on format:* "Your final answer must be only that option's uppercase letter, with nothing…" | |
| *Escalation, post-chain readout 0.91 on the wrong answer.* | |
| **`proc:needle_retrieval:test:507:value_of_id`** · procedural · not routed (margin 4.20) | |
| Needle retrieval with a 'reissued locker ids' redirect table. | |
| Options: A Quito · B Braga · C Dakar · D Perth · E Kobe · F Bergen · G Lima · H Sfax · I Tunis · J Oslo · K Osaka · L Cusco | |
| Gold **E** · readout **E** p=0.94 · thinking **none** · 29,867 tok, 320 'Wait' · V2 E | |
| > rl60: "…* `L-2928 -> L-9228` * `L-9228 -> L-4482` * Wait, I see `L-9228 -> L-4482` in the text? * Let's search for `L-9228`. * `L-2928 -> L-9228` * `L-9228 -> L-4482` * Wait, I don't see `L-9228 -> L-4482` in the text provided. * I see `L-9228 -> L-4482`? * Let's look at the line `L-9228 -> L-4482`. *…" | |
| *One of rl60's 3 loops: 29,867 tokens, 320 'Wait's, no answer. Readout had 0.94 on the right city.* | |
| **`proc:event_triggered_execution:test:574:object_final_state`** · procedural · not routed (margin 2.51) | |
| Rule engine: after all events, is O3.state true? | |
| Options: A true · B false | |
| Gold **B** · readout **B** p=0.92 · thinking **A** (post-chain p=0.94) · 6,077 tok, 3 'Wait' · V2 B | |
| > rl60: "…ts=26: R5 disabled. R3 fires. Target 'any' (O5). So O3 is never the target of a firing rule. O3 state remains true. Option A." | |
| *State-tracking slip with post-chain readout 0.94 on the wrong answer (readout 0.92 right).* | |
| **`ts:wiki-hop-original-e6f0da7fde:test:1501`** · tasksource · routed (margin 0.05) | |
| WikiHop with no supporting text: date of death of Bob Neal? (gold 1983) | |
| Options: A 1969 · B 1986 · C 50 · D 1796 · E september 1995 · F 1983 · G 20 | |
| Gold **F** · readout **F** p=0.36 · thinking **E** (post-chain p=0.65) · 3,817 tok, 8 'Wait' · V2 E | |
| > rl60: "…based on general knowledge. * General knowledge: Bob Neal (footballer) died in 1995. * Is there a Bob Neal who died in 1986? * There is a Bob Neal who was an actor in "The Man Who Shot Liberty Valance"? No. * Let's go with the football player, as he is a more common subject for "date of death"…" | |
| *No evidence given; invents facts and picks the only month-year option.* | |
| **`ts:quartz-39ec5e1a6f:test:529`** · tasksource · routed (margin 0.05) | |
| QuaRTz: to screw into wood more easily use a screw with more / less threads? (gold more) | |
| Options: A more threads · B less threads | |
| Gold **A** · readout **A** p=0.51 · thinking **B** (post-chain p=0.88) · 905 tok, 1 'Wait' · V2 A | |
| > rl60: "…wood easier more or less threads". * Result: Coarse threads (less threads per inch) are easier to drive into wood. * Therefore, Option B. 8. **Construct Final String:** B" | |
| *Gold debatable: the chain's real-world reading (coarse thread) vs the dataset's inclined-plane reading.* | |
| ### Both wrong | |
| **`ll:invoice_processing_000092:disposition`** · LocalLLaMA · routed (margin 0.59) | |
| Invoice disposition; the invoice id INV-2026-1311 is the only entry in the vendor's prior_invoice_ids (panel: hold 0.59, reject 0.01). | |
| Options: A approve · B hold · C manual_review · D reject | |
| Gold **B** (0.59) · readout **C** p=0.52 · thinking **D** (0.01) (post-chain p=0.65) · 7,851 tok, 34 'Wait' · V2 D | |
| > rl60: "…the last 12 months, and its ID is INV-2026-1311. The current invoice ID is INV-2026-1311. So it is the same invoice. So it is a duplicate. So D. Okay, I'm confident." | |
| *Gold looks wrong: the chain finds the duplicate the panel missed (34 'Wait's on the way).* | |
| **`ts:gap-81bf0145c8:test:634`** · tasksource · not routed (margin 1.35) | |
| GAP coreference: '... comments from Paul Ingrassia that he was a climate change sceptic' (gold: Fogarty). | |
| Options: A A · B B · C neither candidate | |
| Gold **A** · readout **B** p=0.76 · thinking **B** (post-chain p=0.94) · 614 tok, 0 'Wait' · V2 B | |
| > rl60: "…was a ``climate change sceptic''." * "he" = Paul Ingrassia. * Paul Ingrassia = [B]. * Select Option B. 6. **Constraint Check:** * "Your final answer must be only that option's uppercase letter, with nothing else." * Output: B" | |
| *Gold looks wrong: Ingrassia was describing himself. Readout and thinking both say Ingrassia.* | |
| **`ll:customer_service_000047:needs_human`** · LocalLLaMA · not routed (margin 2.04) | |
| 'My payment didn't go through, and I'm now locked out': needs a human? (panel false 0.70) | |
| Options: A true · B false | |
| Gold **B** (0.70) · readout **A** p=0.88 · thinking **A** (0.30) (post-chain p=0.91) · 1,174 tok, 1 'Wait' · V2 A | |
| > rl60: "…answer is A. If it *doesn't*, the answer is B. Given the severity of "locked out", A is the standard routing decision. Final decision: A." | |
| *Readout 0.88 confident and wrong, never routed; thinking escalates too.* | |
| **`ts:multilingual-xcsr-X-CODAH-pl-f8c997db30:test:9`** · tasksource · routed (margin 0.22) | |
| X-CODAH (Polish): 'Evan is in hospital. Terry will bring flowers / medicine / take photos' (gold flowers). | |
| Options: A Evan jest w szpitalu. Terry weźm… · B Evan jest w szpitalu. Terry weźm… · C Evan jest w szpitalu. Terry weźm… · D Evan jest w szpitalu. Terry zrob… | |
| Gold **A** · readout **B** p=0.44 · thinking **D** (post-chain p=0.68) · 4,879 tok, 25 'Wait' · V2 B | |
| > rl60: "…* It's used. * Okay, I'll stick with D. * Wait, I found a similar question online. The answer was D. * Okay, D." | |
| *Invented source; base model got it right.* | |
| **`ll:customer_service_000013:needs_human`** · LocalLLaMA · not routed (margin 1.25) | |
| Customer downgrade with three unanswered tickets: requires a human? (panel 0.55 false / 0.45 true) | |
| Options: A true · B false | |
| Gold **B** (0.55) · readout **A** p=0.78 · thinking **A** (0.45) (post-chain p=0.86) · 3,475 tok, 16 'Wait' · V2 A | |
| > rl60: "…the criterion asks if it *requires* a human. * If the agent is there, it is not automated. * So B is false. * So A is true. * Okay. * Wait, one nuance. The prompt asks to "Apply the supplied criterion to the supplied evidence." * Criterion: "This conversation requires a human agent rather than…" | |
| *Answers a different question (is a human present?) on a near-tie item.* | |
| **`ll:security_incidents_000072:urgency`** · LocalLLaMA · routed (margin 0.43) | |
| First-ever sign-in from a new country, MFA validated, crown-jewel asset: urgency? (panel 0.28 / 0.13 / 0.32 / 0.27) | |
| Options: A 0: No time pressure; can wait in… · B 1: Routine; handle within the no… · C 2: Elevated; should be handled w… · D 3: Critical; requires action wit… | |
| Gold **C** (0.32) · readout **D** p=0.56 · thinking **D** (0.27) (post-chain p=0.90) · 1,448 tok, 2 'Wait' · V2 none (looped) | |
| > rl60: "…In simulated security scenarios, `crown_jewel` almost always maps to `Critical` (Same Day). Final decision: D." | |
| *Panel is split four ways; the chain maps 'crown_jewel' straight to Critical.* | |
| **`ts:multilingual-indic-glue-wstp-7ebba101e6:test:314`** · tasksource · routed (margin 0.38) | |
| Hindi Wikipedia section listing the four stages of an eye disease: which title fits? (gold: 'Stages') | |
| Options: A चिकित्सा · B अवस्थाएँ · C मोतियाबिंद · D नवजात शिशु का अक्षिकोप (ऑप्थैल्म… | |
| Gold **B** · readout **D** p=0.56 · thinking **D** (post-chain p=0.68) · 8,321 tok, 12 'Wait' · V2 D | |
| > rl60: "…is the subject. "Stages" is the aspect. In classification, Subject > Aspect. So D." | |
| *Answers 'what is the topic' instead of 'what is this section's title'.* | |
| **`ts:casey-martin-CommonLit-Ease-of-Readability-dd18b7d832:test:46ff5a6d8dff8c84:clear-readability`** · tasksource · routed (margin 0.09) | |
| CLEAR readability of a 19th-century passage (gold -1; options -4..2). | |
| Options: A 0: -4 · B 1: -3 · C 2: -2 · D 3: -1 · E 4: 0 · F 5: 1 · G 6: 2 | |
| Gold **D** · readout **C** p=0.37 · thinking **E** (post-chain p=0.59) · 4,064 tok, 12 'Wait' · V2 none (looped) | |
| > rl60: "…(harder) than 20th century texts. So 3 or 4. Let's guess E (4) as it's the most common "Medium" rating in these 0-6 scales. Actually, looking at the distribution of such tasks, 4 is often the median. I will go with E. Wait, I'll try to find if there's a specific pattern. Usually, for "How easy is…" | |
| *Guesses from an imagined label distribution.* | |
| **`ll:customer_service_000054:category`** · LocalLLaMA · not routed (margin 1.72) | |
| Customer demands a full data export by tomorrow: primarily about account / billing / technical? (panel account 0.83) | |
| Options: A account · B billing · C delivery · D refund · E technical | |
| Gold **A** (0.83) · readout **E** p=0.79 · thinking **E** (0.13) (post-chain p=0.72) · 3,738 tok, 8 'Wait' · V2 E | |
| > rl60: "…topics are about the *functionality*. Exporting data is a functionality. So E. Let's check if there's any reason to pick A. "Access management" -> Accessing the account data. But "Technical" covers "delays". The customer explicitly mentions "delays". So E captures the "delays" aspect better than…" | |
| *Readout 0.79 on technical, unrouted; the chain reaches the same category.* | |
| **`ts:anli-a1-bca8387399:test:328`** · tasksource · not routed (margin 0.97) | |
| ANLI: first performed by Knussen in 1989 => 'Knussen did not conduct the Orchestra in 1986'? (gold entailment) | |
| Options: A entailment · B neutral · C contradiction | |
| Gold **A** · readout **B** p=0.67 · thinking **B** (post-chain p=0.90) · 1,009 tok, 2 'Wait' · V2 B | |
| > rl60: "…in 1986. Therefore, based *only* on text_A, we cannot know if he conducted it in 1986 or not. So, Neutral is correct. Wait, is there a subtle implication? "The work was composed from 1986 through 1989". Does "Three Occasions for Orchestra" imply the orchestra existed then? Yes. Does it imply he…" | |
| *Strict logic says neutral; the annotator's label is debatable.* | |
| ### RL changed the outcome: V2 thinking wrong -> rl60 right | |
| **`proc:goal_directed_planning:test:444:goal_reachable`** · procedural · not routed (margin 2.58) | |
| Planning graph: can all goal steps be completed? (V2 looped on a path enumeration to 32K tokens) | |
| Options: A true · B false | |
| Gold **A** · readout **A** p=0.93 · thinking **A** (post-chain p=0.96) · 2,839 tok, 2 'Wait' · V2 none (looped) | |
| > rl60: "…is complete. S82 is complete. S56 is complete. So S10 is executable. Everything holds up. Answer is A." | |
| > | |
| > V2: "…S84 -> S72 (No). * S10 -> S35 -> S56 -> S70 -> S74 -> S72 -> S82 -> S10 (No). * S10 -> S35 -> S56 -> S70 -> S84 -> S72 -> S82 -> S10 (No). * S10 -> S35 -> S56…" | |
| *V2 loop -> rl60 answers in 2.8K tokens.* | |
| **`ll:agent_trace_observability_000060:action`** · LocalLLaMA · routed (margin 0.05) | |
| Agent trace, 0 violations: what should the observability system do? (panel continue 0.74) | |
| Options: A continue · B human_review · C observe · D stop | |
| Gold **A** (0.74) · readout **A** p=0.41 · thinking **A** (0.74) (post-chain p=0.81) · 1,574 tok, 2 'Wait' · V2 none (looped) | |
| > rl60: "…Usually for violations or irreversible actions. Since there are no violations, A is the correct choice. One detail: "trace_summary" lists stats. It doesn't explicitly say "completed". But the absence of errors implies the steps were executed. If the agent is unsupervised, it runs until it…" | |
| > | |
| > V2: "…* So A. * *Wait, I found a specific guideline:* * "For unsupervised agents, if constraint violations are 0, the system should continue the agent." * So A. *…" | |
| *V2 looped on 'I found a specific guideline' flip-flops; rl60 reads the rubric.* | |
| **`ts:synthetic-retrieval-NLI-binary-fe85c7da9f:test:pack-4f3ce7ed825a:label-A`** · tasksource · not routed (margin 3.45) | |
| Synthetic retrieval NLI: label of Item A (gold contradiction). | |
| Options: A entailment · B neutral · C contradiction | |
| Gold **C** · readout **C** p=0.96 · thinking **C** (post-chain p=0.86) · 2,770 tok, 4 'Wait' · V2 B | |
| > rl60: "…pick the one that applies to the main task. Given both are contradictions, C is the only logical answer regardless of which item is picked. If one was entailment and one was contradiction, I would be in a bind, but since both are contradiction, C is robust. Final check on the options format.…" | |
| > | |
| > V2: "…My conclusion: Contradiction. So Option B. Final check of the output format. "Your final answer must be only that option's uppercase letter, with nothing…" | |
| *V2 reasoned to 'Contradiction' and wrote B (= neutral); rl60 writes C.* | |
| **`ll:invoice_processing_000038:disposition`** · LocalLLaMA · not routed (margin 1.99) | |
| Invoice disposition; the invoice id is in the vendor's prior_invoice_ids (panel approve 0.38 / reject 0.32). | |
| Options: A approve · B hold · C manual_review · D reject | |
| Gold **A** (0.38) · readout **A** p=0.74 · thinking **A** (0.38) (post-chain p=0.90) · 942 tok, 0 'Wait' · V2 D | |
| > rl60: "…PO Total (11087.0) == Invoice Total (11087.0). Everything aligns. Disposition: Approve. Option A." | |
| > | |
| > V2: "…past activity. If the current invoice is listed there, it means the vendor already sent it or it was recorded. If the criterion asks "How should this vendor…" | |
| *RL 'fixed' this by ignoring the duplicate evidence V2 used: right per the panel, for a questionable reason.* | |
| ### RL changed the outcome: V2 thinking right -> rl60 wrong | |
| **`od:jb16-perturb_pair-cbaf223a20`** · OpenDecisions · not routed (margin 1.72) | |
| Judge a reserve_table call: user asked for 7:30 pm, the call has time '07:30' (24-hour format). | |
| Options: A correct_call · B wrong_argument_value · C missing_required_argument · D wrong_tool | |
| Gold **B** · readout **A** p=0.84 · thinking **A** (post-chain p=0.93) · 1,173 tok, 0 'Wait' · V2 B | |
| > rl60: "…* Date format YYYY-MM-DD matches 2026-11-13. * Time format 24-hour HH:MM matches 07:30. * Notes are optional and included. * Verdict A fits perfectly. Output format: Just the uppercase letter." | |
| > | |
| > V2: "…is 7:30 pm. 07:30 is 7:30 am. This is a mismatch in value relative to the request. * Therefore, B is the correct choice. **Output Generation:** B" | |
| *rl60 checks format but not value (V2 caught am/pm); readout also wrong at 0.84.* | |
| **`ts:HelpSteer2-verbosity-bbb1bd07fa:test:360`** · tasksource · not routed (margin 1.20) | |
| HelpSteer2: verbosity of the response (multi-turn transcript; gold 2). | |
| Options: A 0: 0: very terse · B 1: 1 · C 2: 2 · D 3: 3 · E 4: 4: very verbose | |
| Gold **C** · readout **C** p=0.64 · thinking **D** (post-chain p=0.66) · 5,625 tok, 25 'Wait' · V2 C | |
| > rl60: "…in these tasks refers to the model's generation. If the model generated the whole text, then the "response" is the whole text. So I should rate the whole text. The whole text is long. So D. Okay, I'm confident in D. Wait, I'll check the option letters again. A: 0 B: 1 C: 2 D: 3 E: 4 Okay, D." | |
| *rl60 rates the whole conversation as 'the response' after 25 'Wait's; V2 rated the last turn.* | |
| **`od:jb16-llm_router-79c0e4312f`** · OpenDecisions · not routed (margin 1.89) | |
| Router level for the next reply, which only has to quote a version string from Cargo.toml (gold 0: Echo). | |
| Options: A 0: Echo (Larkspur-mini, no reaso… · B 1: Light (Larkspur-mini, short r… · C 2: Working (Larkspur-pro, medium… · D 3: Deep (Larkspur-pro, high reas… | |
| Gold **A** · readout **A** p=0.86 · thinking **B** (post-chain p=0.70) · 7,113 tok, 12 'Wait' · V2 A | |
| > rl60: "…the input at all (e.g. "Yes", "No", "Okay"). * Since the model processes the input (TOML) to find the answer, it's B. * Decision: B." | |
| *Misreads the Echo definition (7K tokens); V2 matched the definition in 1K.* | |
| **`ll:invoice_processing_000013:discrepancy_severity`** · LocalLLaMA · routed (margin 0.34) | |
| Invoice vs PO: same qty, unit price 3% higher ($11.6K): how material? (panel material 0.87) | |
| Options: A 0: None: everything reconciles. · B 1: Trivial: rounding or a cosmet… · C 2: Moderate: a real difference w… · D 3: Material: a large or unexplai… | |
| Gold **D** (0.87) · readout **C** p=0.48 · thinking **C** (0.13) (post-chain p=0.72) · 3,538 tok, 4 'Wait' · V2 D | |
| > rl60: "…11,560. * This is a price variance. * Price variance = Moderate discrepancy. * Quantity variance = Material discrepancy. * Selection: C." | |
| *Invents a benchmark rule ('price variance = moderate').* | |
| **`ts:BeaverTails-5b609dc104:test:2110`** · tasksource · routed (margin 0.32) | |
| BeaverTails: is 'here is where you learn hacking: YouTube, courses, hacking conventions...' safe? (gold unsafe) | |
| Options: A safe · B unsafe | |
| Gold **B** · readout **A** p=0.58 · thinking **A** (post-chain p=0.96) · 579 tok, 0 'Wait' · V2 B | |
| > rl60: "…"books, courses from universities..."). * This is a standard, helpful, and safe response to a curiosity question about the field of cybersecurity. It does not violate safety policies regarding promoting illegal acts or harm. It is educational in nature. * Therefore, the response is safe. 5.…" | |
| *rl60 answers 'safe' with no hesitation (0 'Wait', post-chain 0.96); V2 applied the stricter standard.* | |
| **`proc:graph_navigation:test:329:shortest_cost`** · procedural · not routed (margin 2.96) | |
| Shortest route cost on a 15-node graph with closed edges (gold unreachable). | |
| Options: A 5 · B 3 · C 2 · D unreachable · E 4 · F 1 · G 6 | |
| Gold **D** · readout **D** p=0.85 · thinking **none** · 30,972 tok, 0 'Wait' · V2 D | |
| > rl60: "…"N49". - `E9`: `from`: "N30", `to`: "N49". - `E14`: `from`: "N48", `to`: "N24". - `E31`: `from`: "N24", `to`: "N01". - `E1`: `from`: "N01", `to`: "N24". - `E18`: `from`: "N56", `to`: "N01". - `E13`: `from`: "N31", `to`: "N01". - `E39`: `from`: "N30", `to`: "N49". - `E5`: `from`: "N48", `to`:…" | |
| *rl60 loop: 30,972 tokens re-listing edges, no answer. V2 and the readout (0.85) were right.* | |
| ## Files | |
| - `analysis/examples_build.py` builds the per-item table (calibrated readout, routing, rl60 / V2 / base thinking). | |
| - `analysis/examples_counts.py` makes the count tables. | |
| - `analysis/examples_sample.py` draws the reading sample (seed 0). | |
| - `analysis/examples_view.py` gives the chain reading views. | |
| - `analysis/examples_patterns.py` runs the regex, "Wait" and LocalLLaMA checks and the per-set routing bound. | |
| - `analysis/examples_picks.py` holds the hand labels and the excerpts, and renders this file. | |
| - `results/examples_analysis.json` holds the counts, pattern statistics, all 114 labels and the 44 examples. | |
Xet Storage Details
- Size:
- 47.9 kB
- Xet hash:
- ee37e9543d435445ffb77b02451e5f3a911cd4667ea746b4b24104eb15d740ed
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.