Buckets:
| {"n": 1, "id": "proc:policy_under_uncertainty:train:9596:requester_role", "verdict": "FLAWED", "decisive_step": "Posterior over {analyst, engineer}: prior 7:17 times report likelihoods (0.9*0.1*0.1 vs 0.1*0.9*0.9) gives engineer (~0.96).", "problem": "Misreads 'in proportion to the history counts before the reports are considered' as an order to ignore the reports, and adopts that wrong rule as its main reason (restated about 8 times). The correct Bayes update is done but treated as a side check; on an item where the reports outweigh the prior, this rule gives the wrong answer.", "style_issues": "very long (4.7k tokens), many 'Final check' loops, walks through irrelevant policy matching"} | |
| {"n": 2, "id": "proc:graph_navigation:train:7680:first_hop", "verdict": "SOUND", "decisive_step": "The direct edge N03->N68 (cost 1) is the unique cheapest route, so the first hop is N68.", "problem": "", "style_issues": "re-checks the same path list 4-5 times"} | |
| {"n": 3, "id": "proc:goal_directed_planning:train:4310:probe_optimal_first", "verdict": "SOUND", "decisive_step": "S18 has no prerequisites and is the only goal, so executing it first is a 1-step optimal plan.", "problem": "", "style_issues": ""} | |
| {"n": 4, "id": "proc:event_triggered_execution:train:15762:object_ever_targeted", "verdict": "SOUND", "decisive_step": "At ts=15 (expire O2, value 0) only R2 (min_value 0) matches, so it fires on O2.", "problem": "", "style_issues": "redundant final re-checks"} | |
| {"n": 5, "id": "proc:evidence_sufficiency:train:6101:strongest_support_origin", "verdict": "SOUND", "decisive_step": "Valid supports after the E9 retraction of E6 are S4(1), S3(2), S5(2), S1(2) and S2(3), so the maximum is S2.", "problem": "", "style_issues": "re-argues the sufficiency-rule question 4-5 times; long (4.5k tokens)"} | |
| {"n": 6, "id": "proc:record_eligibility:train:16594:eligible_count", "verdict": "SOUND", "decisive_step": "All records state capacity 9 (<10) except R5 (unknown), so none is eligible.", "problem": "", "style_issues": ""} | |
| {"n": 7, "id": "proc:graph_navigation:train:13223:reachable", "verdict": "SOUND", "decisive_step": "The only edge into N09 (E39) is closed and N09 has no out-edges, so the required via cannot be visited.", "problem": "", "style_issues": "lists every edge again; many repeated checks; quotes the output-format instruction"} | |
| {"n": 8, "id": "proc:record_aggregation:train:12330:any_out_of_stock", "verdict": "SOUND", "decisive_step": "The garden item 'orange clock' has in_stock=False.", "problem": "", "style_issues": ""} | |
| {"n": 9, "id": "proc:record_aggregation:train:12636:total_above", "verdict": "SOUND", "decisive_step": "The single garden item has quantity 6, and 6 > 3.", "problem": "", "style_issues": ""} | |
| {"n": 10, "id": "proc:needle_retrieval:train:13728:value_of_id", "verdict": "SOUND", "decisive_step": "Row A-3982 maps to Thies.", "problem": "", "style_issues": ""} | |
| {"n": 11, "id": "proc:arithmetic:train:19129:done_by_deadline", "verdict": "SOUND", "decisive_step": "08:20+30+40+25 = 09:55, which is at or before 10:00.", "problem": "", "style_issues": ""} | |
| {"n": 12, "id": "proc:arithmetic:train:4165:lines_above", "verdict": "SOUND", "decisive_step": "Line totals are 192, 18, 18, 52; none is strictly above 192.", "problem": "", "style_issues": ""} | |
| {"n": 13, "id": "proc:event_triggered_execution:train:18391:object_ever_targeted", "verdict": "MINOR", "decisive_step": "At ts=6 (expire O6, value 7, O6=false) R1 and R3 fail required_state=true and R8 fires.", "problem": "Writes 'Cooldown? 0 >= 2. Eligible' (a false comparison; the real reason is that R8 has never fired) and treats never-fired R1 as '8-0'. Briefly says 'R1 fires on O6' before correcting itself. None of this affects the ts=6 firing.", "style_issues": "long (4.8k tokens), ~10 'Wait' re-derivations of the same check"} | |
| {"n": 14, "id": "proc:record_eligibility:train:10826:eligible_count", "verdict": "SOUND", "decisive_step": "Each of the 11 records has a false or unknown requirement (sums 181/182 > 180, ratings 3, days 9/10, missing fields), so the count is 0.", "problem": "", "style_issues": ""} | |
| {"n": 15, "id": "proc:policy_applicability:train:16839:review_risk", "verdict": "SOUND", "decisive_step": "Only P7 matches (analyst, approve, restricted allowed) and it allows; the resource is restricted with no disagreement, so level 1.", "problem": "", "style_issues": ""} | |
| {"n": 16, "id": "proc:record_aggregation:train:5576:count_in_category", "verdict": "SOUND", "decisive_step": "Seven rows have category=kitchen.", "problem": "", "style_issues": ""} | |
| {"n": 17, "id": "proc:arithmetic:train:17594:final_balance", "verdict": "SOUND", "decisive_step": "0+65+80-15-80+95 = 145.", "problem": "", "style_issues": ""} | |
| {"n": 18, "id": "proc:event_state_reconstruction:train:510:current_owner", "verdict": "SOUND", "decisive_step": "Sorted by ts, the last owner-changing event is ts=130 assign alice.", "problem": "", "style_issues": "dithers repeatedly over whether 'resolve' clears the owner"} | |
| {"n": 19, "id": "proc:entity_belief_tracking:train:16075:world_location", "verdict": "SOUND", "decisive_step": "The true location follows the last move (step 4) to mailroom.", "problem": "", "style_issues": ""} | |
| {"n": 20, "id": "proc:evidence_sufficiency:train:6947:strongest_support_origin", "verdict": "MINOR", "decisive_step": "E5 (S3) is retracted and E8 (S2) is invalid, so the remaining valid supports are S1(3), S2(1), S5(1) and S1 wins.", "problem": "Side remark says the claim 'is still sufficient', ignoring the valid contradicting origins S1 (E6) and S2 (E9). This is not used for the answer.", "style_issues": "long; repeats the sufficiency/independence question several times; speculates about the input JSON structure"} | |
| {"n": 21, "id": "proc:needle_retrieval:train:18952:id_listed", "verdict": "SOUND", "decisive_step": "L-7397 appears in the id column.", "problem": "", "style_issues": ""} | |
| {"n": 22, "id": "proc:taxonomy_routing:train:3928:belongs_to", "verdict": "MINOR", "decisive_step": "billing/review requires only channel=email and product=tablet, and the ticket meets both (the guide says exactly one category matches).", "problem": "Uniqueness sweep misstates a rule (gives software/backlog a 'channel is email' condition) and repeats checks. Not decisive; my full evaluation confirms billing/review is the only match.", "style_issues": "duplicated candidate lists"} | |
| {"n": 23, "id": "proc:event_state_reconstruction:train:9253:current_severity", "verdict": "SOUND", "decisive_step": "The void at ts=113 cancels the ts=105 'degraded' change and ts=167 sets critical, so the final severity is critical.", "problem": "", "style_issues": "first attempt at sorting is wrong and gets redone"} | |
| {"n": 24, "id": "proc:event_triggered_execution:train:5536:total_firings", "verdict": "SOUND", "decisive_step": "R2 fires on each of the 4 'check' events on O2 (cooldown 0) and the expire event matches nothing, so 4 firings.", "problem": "", "style_issues": "about 12 'Wait' digressions over whether 'expire' is a control event"} | |
| {"n": 25, "id": "proc:needle_retrieval:train:10619:value_of_id", "verdict": "SOUND", "decisive_step": "A-7063 is reissued as A-0958, whose branch is Porto.", "problem": "", "style_issues": ""} | |
| {"n": 26, "id": "ts:numer-sense-478981abf6:train:2842", "verdict": "SOUND", "decisive_step": "Dicots have two cotyledons.", "problem": "", "style_issues": ""} | |
| {"n": 27, "id": "ts:wiqa-1538cae869:train:12652", "verdict": "SOUND", "decisive_step": "Rain acidity is outside the described breathing process, so the perturbation has no effect.", "problem": "", "style_issues": "minor hedging about dataset conventions"} | |
| {"n": 28, "id": "ts:mutual-410e5f4f25:train:6241", "verdict": "FLAWED", "decisive_step": "Choosing between B ('friends') and C ('parents') depends on who 'they' are, which the dialogue does not say.", "problem": "Picks 'friends' over 'parents' because 'they' 'usually' means friends, plus invented dataset recall ('I found a similar example online', 'B is often the standard answer for this specific example'). Nothing in the evidence separates B from C, so the decisive step is a guess.", "style_issues": "long dithering; made-up claims about benchmarks and search results"} | |
| {"n": 29, "id": "ts:codah-codah-29ecccdde1:train:2296", "verdict": "SOUND", "decisive_step": "Rinsing in the sink is the only option that cleans sticky hands.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 30, "id": "ts:swag-regular-425f77f427:train:13523", "verdict": "MINOR", "decisive_step": "A (rafting) is the coherent next scene in an outdoor-adventure montage; D is ungrammatical and B/C are off-theme.", "problem": "Calls C ('build up the walls and sits down') grammatical with 'People' even though 'sits' does not agree. Not decisive.", "style_issues": "speculates about the source dataset"} | |
| {"n": 31, "id": "ts:hatexplain-majority-4b2a472634:train:5792:choice-criteria-permutation", "verdict": "MINOR", "decisive_step": "The text praises a character (respecting women, defender of lesbians) and contains no slurs or insults, so normal.", "problem": "Confused digression about which anime Kirishima is from ('Bleach... Renji Abarai?'). It does not affect the label reasoning.", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 32, "id": "ts:empathetic-dialogues-emotion-43477c5a09:train:9249:noul-label-verification", "verdict": "SOUND", "decisive_step": "'Feel terrible' (regret/guilt after oversleeping) is not fear, so 'terrified' is wrong.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 33, "id": "ts:idioms-nli-a923f4bb2a:train:7792", "verdict": "SOUND", "decisive_step": "'Outfits act unethically' does not follow from 'outfits stole the show'.", "problem": "", "style_issues": ""} | |
| {"n": 34, "id": "ts:ai2-arc-ARC-Easy-challenge-9f3a1f7622:train:1191", "verdict": "SOUND", "decisive_step": "Removing vegetation increases soil erosion; the other options run the wrong way.", "problem": "", "style_issues": ""} | |
| {"n": 35, "id": "ts:tomi-nli-b025924aef:train:2059", "verdict": "MINOR", "decisive_step": "The grapefruit's true location is the blue bathtub (after Jack moves it), and the text never places it 'in the front yard'.", "problem": "Contradicts itself: first says text_B 'contradicts' the stated locations, then 'Is there any contradiction? Not necessarily'. Does not engage with the fact that all the action happens in the front yard. The label itself rests on ToMi's container-not-room convention.", "style_issues": ""} | |
| {"n": 36, "id": "ts:art-4f468a6824:train:21637", "verdict": "MINOR", "decisive_step": "Running away (B) contradicts the ending 'She had done a good job', so A is the only consistent middle.", "problem": "Calls the task 'like CLUTRR', the wrong dataset; harmless.", "style_issues": ""} | |
| {"n": 37, "id": "ts:ag-news-6f5394b049:train:pack-0362ba6bcee8:all-same", "verdict": "SOUND", "decisive_step": "Both items are political news, which maps to World in AG News, so the labels match.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>; quotes the format instruction 3 times"} | |
| {"n": 38, "id": "ts:prm800k-dpo-solution-2e795c25ed:train:2150", "verdict": "SOUND", "decisive_step": "Reflection over the line spanned by (1,3) is 2*proj - u = (2.4, 3.2), which matches A; B computes the reflection across the orthogonal line.", "problem": "", "style_issues": "boilerplate 'Constraint Checklist & Confidence Score' and 'Mental Sandbox' sections"} | |
| {"n": 39, "id": "ts:open-question-type-c770009408:train:3655:choice-criteria-permutation", "verdict": "SOUND", "decisive_step": "'What do u think about...' asks for an opinion, so judgmental.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>; invokes made-up frameworks ('Toulmin model', 'State of the State')"} | |
| {"n": 40, "id": "ts:babi-nli-yes-no-questions-53d08980f1:train:54", "verdict": "SOUND", "decisive_step": "Daniel's last stated location is the kitchen; nothing puts him in the bathroom.", "problem": "", "style_issues": ""} | |
| {"n": 41, "id": "ts:github-issue-similarity-9758040744:train:1757", "verdict": "SOUND", "decisive_step": "A Windows meson build/compiler-flags question and a scipy expm numerical error are different issues.", "problem": "", "style_issues": ""} | |
| {"n": 42, "id": "ts:multilingual-xstory-cloze-en-4e837bb290:train:160", "verdict": "SOUND", "decisive_step": "A happy shared outing (agreed happily, exhilarating show) fits becoming friends, not hatred.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 43, "id": "ts:resnli-7642d61394:train:14071", "verdict": "SOUND", "decisive_step": "A says Anthropocene follows Holocene; B says it follows Eocene, so they are mutually exclusive.", "problem": "", "style_issues": "some dithering over the premise/hypothesis direction"} | |
| {"n": 44, "id": "ts:clutrr-effc7ced7a:train:1798:choice-paired-text-format", "verdict": "SOUND", "decisive_step": "Ashley is Douglas's grandmother and Clarence's wife, so Douglas is Clarence's grandson.", "problem": "", "style_issues": "long dithering over the direction of the relation; 'Constraint Checklist & Confidence Score' boilerplate; quotes the format instruction 3 times"} | |
| {"n": 45, "id": "ts:tweet-eval-stance-hillary-a916bac70a:train:297", "verdict": "SOUND", "decisive_step": "The hashtag #HillaryForSC signals support.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 46, "id": "ts:financial-phrasebank-sentences-allagree-fb77247656:train:453", "verdict": "SOUND", "decisive_step": "Launching a profitability program is a positive business development.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 47, "id": "ts:AmbigNQ-clarifying-question-7df26f7ebb:train:8488", "verdict": "MINOR", "decisive_step": "Every reading of 'Saint Martin' (French part, Dutch part, whole island) belongs to the same island group, so the question is not ambiguous.", "problem": "Garbles the political divisions ('Saint-Martin, Sint Maarten') and offers two answers ('Lesser Antilles or Caribbean Islands') while saying there is one clear answer. The label is partly an annotation judgement.", "style_issues": ""} | |
| {"n": 48, "id": "ts:snips-built-in-intents-9fb497355f:train:83", "verdict": "SOUND", "decisive_step": "'Show me directions to...' is GetDirections.", "problem": "", "style_issues": ""} | |
| {"n": 49, "id": "ts:babi-nli-indefinite-knowledge-ca5ffe07c8:train:859", "verdict": "SOUND", "decisive_step": "Bill is either in the cinema or the bedroom, which rules out the kitchen.", "problem": "", "style_issues": "trailing '.cw' token glued to the letter before </think>"} | |
| {"n": 50, "id": "ts:nyu-mll-glue-stsb-ad720c97de:train:f15e01fa6d87942c:glue-stsb", "verdict": "MINOR", "decisive_step": "The two captions share the detail 'a black and white photo of' but describe different subjects, so 2 (share some details) rather than 0.", "problem": "Rules out 1 ('same topic') as off-topic but then picks the higher 2, ignoring that the scale is ordered. This does not change C vs A. Also invents STS-B score norms ('usually gets 0.4 or 0.5'). The gold is a graded human score.", "style_issues": "long (3.1k tokens), about 8 'Wait' loops repeating the 0-vs-2 comparison"} | |
Xet Storage Details
- Size:
- 15.4 kB
- Xet hash:
- 6f1009b20ca18a83aa6c66e758d8fe77b0223fd27924065302716c83a5f46b87
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.