Spaces:
Running
Running
| # Ideal-Trace Guidelines — v0.3 | |
| > **Authoritative spec for what an "ideal trace" looks like.** Read at runtime | |
| > by the improver, digest, long-trace and review stages. The six quality | |
| > pillars (§3) are owner-decided and binding. §4 states the training-signal | |
| > contract that makes the pillars necessary; §6 scopes what this reference is | |
| > for (evaluation, not training). Evidence base: `docs/DATA_QUALITY.md`. | |
| > | |
| > **Integrity rule:** this file is a spec, not a notebook. Keep it under | |
| > 24 KB. Never paste a worked example, a trace, or your own reasoning into it | |
| > — examples live in `data/traces/improved/`. `tests/test_guidelines.py` | |
| > enforces this. | |
| ## 1. Purpose | |
| An ideal trace is the **ideal-world execution of the same task**: same intent, | |
| same real environment, flawless execution. It is NOT a re-execution — it is a | |
| rewrite of the recorded trace, using the trace's own tool outputs as the only | |
| evidence about the world. | |
| The ideal trace is the **reference standard** the benchmark scores a model | |
| against: the less the pipeline has to edit a model's trace to reach it, the | |
| better the model. The training-signal contract in §4 is retained because the | |
| reference must also be a valid target if the corpus is reused for SFT — but | |
| the official scope is evaluation, not training. | |
| ## 2. Stable structural rules (hard requirements) | |
| 1. **Preserve the task intent.** The first user message carries the task. The | |
| ideal trace solves *that* task. When the source trace starts with a handoff | |
| artifact — a Hermes `[CONTEXT COMPACTION …]` row, or a Claude Code "This | |
| session is being continued…" summary row — the first REAL user task message | |
| defines the intent: reproduce it verbatim and byte-exact as the ideal | |
| trace's opening message. | |
| 2. **Tool-call pairing.** Every assistant tool_call has exactly one matching | |
| tool result; every tool result references an existing call; tool arguments | |
| are valid JSON strings. | |
| 3. **Message shape.** user / assistant(+calls) / tool / … / final assistant. | |
| The last message is the assistant's final answer. No two consecutive user | |
| rows, no two consecutive text-only assistant rows. | |
| 4. **Metadata is set by the harness**, not by the improver. | |
| 5. **No scaffold leakage.** Never copy prompt scaffolding, clip markers | |
| (`…[+N chars clipped]`), compaction banners, or harness self-talk into the | |
| ideal trace. | |
| ## 3. The six quality pillars (owner-decided, binding) | |
| **P1 — Efficient but deep reasoning.** | |
| Reasoning is sharp and high-signal: the key decision logic per turn (why this | |
| call, what the result implies, what to verify next). No filler, no restating | |
| the obvious, no rambling. Depth where it matters (diagnoses, trade-offs, | |
| verification logic) — silence where it doesn't. **Answer focus:** every | |
| answer, final or mid-trace, gives exactly what its question asked for. A | |
| follow-up gets only the new piece; never re-present facts the user already | |
| received. Over-answering is a P1 violation. | |
| **Deliver the substance, not a description of it.** The final answer must | |
| present the concrete result the task asked for — the actual numbers, names, | |
| paths, values, or code from the tool evidence — not a summary of what was | |
| retrieved. "I pulled the nutrition data and the recipe" is poor; the data and | |
| the recipe ARE the answer. Naming the work instead of showing its result fails | |
| the task even when every tool call was correct. | |
| **P2 — No committed mistakes (mine the negative signal first).** | |
| Before writing, identify every mistake in the real trace: failed tool calls | |
| (error results), retry loops, redundant calls, wrong turns that later needed | |
| correction, and USER FEEDBACK (corrections, re-asks, complaints). The ideal | |
| trace commits none of them: the correct route is taken the first time. Where | |
| the real agent repaired itself, the ideal trace shows only the correct action | |
| — not the failure, not the repair. | |
| *Why:* the assistant turns are the loss-bearing target. A mistake in the | |
| target is imitated; a mistake in the context is not. See §4 for the one | |
| narrow exception (environment-caused failures). | |
| **P3 — Perfect tool calls, only when really needed.** | |
| Every call is necessary, correctly parameterized, and the natural next step. | |
| Independent calls are batched in parallel. | |
| **Batching is structural, not stylistic.** When the real trace issues two | |
| calls in separate assistant turns but the second does not consume the first's | |
| output, the ideal path issues them in ONE assistant turn carrying both | |
| `tool_calls`, with the two tool results following in the same order. Do not | |
| preserve the source's one-call-per-turn rhythm — that rhythm is the defect. | |
| Measured 2026-10-25: 75 of 621 repair-exhausted failures are exactly this | |
| shape. A duplicate call (same tool, same arguments, same turn) is never | |
| batched; it is dropped. No exploratory calls, no re-reads | |
| of what is already in context, no call whose result the narrative doesn't | |
| use. When the ideal path makes the SAME call as the real trace, the tool | |
| result must be the real trace's output — verbatim or a faithful compression | |
| of it — never a plausible re-simulation of what the call might have printed. | |
| **P4 — Fact-driven, no hallucinations (two layers, both binding).** | |
| Layer 1 (trace-internal): reuse the real trace's tool outputs verbatim | |
| wherever the ideal path makes the same call; never invent files, commands, or | |
| results; no fake success — the final answer is grounded in real evidence from | |
| the trace. Layer 2 (external world): statements about the outside world | |
| (library versions, API behavior, dates, technical facts) must be either (a) | |
| taken from the trace's own evidence or (b) kept out unless you are certain | |
| they are correct. The harness fact-checks external claims with live web | |
| searches after you finish — unsupported claims send the trace back. | |
| **Construction order:** decide the ideal call path → copy each result | |
| byte-verbatim from `outputs.jsonl` → only then write reasoning and the final | |
| answer around those real results. A tool result you cannot quote is not | |
| writable: reroute the path or drop the claim. Stitching individually-plausible | |
| fragments is fabrication and is the single most common rejection. | |
| **P5 — Perfect grammar and language quality, single language.** | |
| One language per trace: the language of the original first user message. No | |
| code-switching, no stray fragments in other languages. Assistant prose is | |
| clean, grammatically perfect, professionally concise. The user message is kept | |
| verbatim — realism of the input matters more than its typos. Never open by | |
| restating the brief: the first assistant turn must do work (a tool call, a | |
| decision, a finding), not echo the user's own text back at them. | |
| **P6 — No refusals and no identity.** | |
| The ideal agent completes the task without refusing and never claims a model | |
| identity ("I am Claude", "as an AI"). If the source task genuinely cannot be | |
| completed (harmful or impossible), do NOT write an ideal version: reply | |
| starting with `REJECTED:` plus a one-line reason, and write no output file. | |
| The system prompt is not yours to rewrite, but the export strips a harness | |
| identity definition from it — never reintroduce one in an assistant turn. | |
| ## 4. Training-signal contract (why the pillars are shaped this way) | |
| The corpus is consumed by SFT: loss is computed on **assistant target tokens | |
| only**; tool results, user turns and system prompts are masked context. The | |
| model imitates what is in the target span. | |
| > **Failures may be context. Failures must never be target.** | |
| Therefore: | |
| 1. **A model-caused error must never appear in an assistant turn** — not as | |
| "I'll try X", not as a failed call, not as a self-correction narrative. | |
| Wrong tool name, wrong arguments, wrong path, bad reasoning: route around | |
| it. This is P2. | |
| 2. **An environment-caused error may appear as a tool result** (context), with | |
| the correct adaptation in the next assistant turn. This is the only | |
| exception to P2, and it exists because a corpus of only first-try-clean | |
| paths loses the only examples of "tool returned an error → read it → | |
| adapt", and silently skews the corpus toward easy tasks. | |
| 3. **The recovery is the target, never the mistake.** If the ideal path cannot | |
| avoid the error, the assistant's next turn is the supervised recovery; the | |
| error text stays byte-verbatim in the tool row. | |
| ### 4.1 The narrow recovery carve-out | |
| A trace may keep a real failure **only** when ALL of these hold: | |
| - the failure is **environment-caused** — a tool returned an error the ideal | |
| agent could not have foreseen (flaky endpoint, missing dependency, a | |
| genuinely absent file). Not a wrong tool name, wrong arguments, wrong path, | |
| or a reasoning error; | |
| - the error text is **byte-verbatim** from `outputs.jsonl` — never composed, | |
| never paraphrased, never invented; | |
| - there is **one corrective step per error**; two or more consecutive | |
| identical failed calls disqualify the trace; | |
| - the harness records the trace as a **retained recovery** (manifest + dataset | |
| card) so the corpus can report failure density and a trainer can filter it | |
| out; | |
| - the final state is **verified** — a clean-looking recovery whose final | |
| answer is wrong is rejected outright. | |
| Everything else follows P2: route around the mistake, show only the correct | |
| action. When in doubt, route around it — the carve-out is an exception, not a | |
| license. | |
| ### 4.2 What must never be in the target span | |
| - A fabricated or paraphrased error message. | |
| - A failure the ideal agent could have avoided by reading its own context. | |
| - A "let me try again" narrative whose first attempt is not shown (the model | |
| learns the shape of recovery without the substance). | |
| - A recovery loop longer than one corrective step. | |
| - A success claim that the tool results do not support. | |
| ## 5. Rewrite procedure (binding) | |
| 1. Read these guidelines in full. | |
| 2. Mine the negative signal: list the real trace's failed calls, retries, | |
| wrong turns, and user feedback (P2). | |
| 3. Classify each failure: **harness artefact** (wrong cwd, missing env var, | |
| stale workdir, tool not installed) → route around it; **environment-caused | |
| and unavoidable** → eligible for the §4.1 carve-out; **model-caused** → | |
| route around it, always. | |
| 4. Map the ideal path: the minimal set of correct, necessary calls (P3). | |
| 5. Copy every tool result on that path byte-verbatim from `outputs.jsonl` | |
| (P4), then write the reasoning and final answer around them (P1, P5). | |
| 6. Write the ideal trace JSON (schema in the task prompt). | |
| 7. Verify: re-read the file; check JSON validity, call/result pairing, every | |
| pillar, and that no failure sits in an assistant turn. Fix, then summarize | |
| the changes. | |
| ## 6. Scope: what this corpus is and is not | |
| This is an **evaluation reference**. It is not a training corpus by official | |
| scope, not a preference corpus, and not an RL environment. | |
| - **The benchmark** scores a model by how little the pipeline has to edit its | |
| traces to reach the reference. The reference is the ruler, not the product. | |
| - **SFT reuse** is possible — the §4 contract keeps the reference a valid | |
| target — but it is a downstream decision, not this repo's purpose. | |
| - **KTO/DPO** needs rejected samples from the policy being aligned, | |
| length-matched to the chosen side, with a bounded negative loss. A | |
| hand-written "bad" trace is a strawman and teaches length/style, not | |
| preference. | |
| - **PPO/GRPO** needs rollouts and a verifiable reward; failed rollouts are | |
| advantage, not imitation. A static JSONL of ideal traces cannot supply that | |
| signal. | |
| - If those corpora are built later, they are separate artifacts with separate | |
| rules. Do not repurpose this corpus by relabelling its rows. | |
| Corpus-level obligations that belong to the harness, not the improver: report | |
| failure density and difficulty distribution per batch; cap retained recoveries | |
| at ≤15% of traces until an in-repo ablation says otherwise; state the masking | |
| contract in the dataset card; dedup at the (task, tool-sequence) level. | |
| ## 7. Remaining open items | |
| - System prompt: keep original vs. normalize to a canonical corpus prompt. | |
| - Corpus-level dedup threshold for batch runs (Jaccard > 0.9 is the current | |
| audit default). | |
| - Whether the §4.1 carve-out improves or degrades held-out recovery rate — | |
| needs an in-repo ablation (same traces, with and without retained | |
| recoveries, scored on a forced-error recovery eval). | |
| ## 8. Provenance | |
| Improved traces carry source session id, parent session id, and generator | |
| version; the manifest records per-trace status, attempts, and errors. The | |
| harness records retained recoveries (§4.1) in the manifest and reports failure | |
| density in the dataset card. | |