guidelines/IDEAL_TRACE_GUIDELINES.md

Ideal-Trace Guidelines — v0.3

Authoritative spec for what an "ideal trace" looks like. Read at runtime by the improver, digest, long-trace and review stages. The six quality pillars (§3) are owner-decided and binding. §4 states the training-signal contract that makes the pillars necessary; §6 scopes what this reference is for (evaluation, not training). Evidence base: docs/DATA_QUALITY.md.

Integrity rule: this file is a spec, not a notebook. Keep it under 24 KB. Never paste a worked example, a trace, or your own reasoning into it — examples live in data/traces/improved/. tests/test_guidelines.py enforces this.

1. Purpose

An ideal trace is the ideal-world execution of the same task: same intent, same real environment, flawless execution. It is NOT a re-execution — it is a rewrite of the recorded trace, using the trace's own tool outputs as the only evidence about the world.

The ideal trace is the reference standard the benchmark scores a model against: the less the pipeline has to edit a model's trace to reach it, the better the model. The training-signal contract in §4 is retained because the reference must also be a valid target if the corpus is reused for SFT — but the official scope is evaluation, not training.

2. Stable structural rules (hard requirements)

1. Preserve the task intent. The first user message carries the task. The ideal trace solves that task. When the source trace starts with a handoff artifact — a Hermes [CONTEXT COMPACTION …] row, or a Claude Code "This session is being continued…" summary row — the first REAL user task message defines the intent: reproduce it verbatim and byte-exact as the ideal trace's opening message.
2. Tool-call pairing. Every assistant tool_call has exactly one matching tool result; every tool result references an existing call; tool arguments are valid JSON strings.
3. Message shape. user / assistant(+calls) / tool / … / final assistant. The last message is the assistant's final answer. No two consecutive user rows, no two consecutive text-only assistant rows.
4. Metadata is set by the harness, not by the improver.
5. No scaffold leakage. Never copy prompt scaffolding, clip markers (…[+N chars clipped]), compaction banners, or harness self-talk into the ideal trace.

3. The six quality pillars (owner-decided, binding)

P1 — Efficient but deep reasoning. Reasoning is sharp and high-signal: the key decision logic per turn (why this call, what the result implies, what to verify next). No filler, no restating the obvious, no rambling. Depth where it matters (diagnoses, trade-offs, verification logic) — silence where it doesn't. Answer focus: every answer, final or mid-trace, gives exactly what its question asked for. A follow-up gets only the new piece; never re-present facts the user already received. Over-answering is a P1 violation.

Deliver the substance, not a description of it. The final answer must present the concrete result the task asked for — the actual numbers, names, paths, values, or code from the tool evidence — not a summary of what was retrieved. "I pulled the nutrition data and the recipe" is poor; the data and the recipe ARE the answer. Naming the work instead of showing its result fails the task even when every tool call was correct.

P2 — No committed mistakes (mine the negative signal first). Before writing, identify every mistake in the real trace: failed tool calls (error results), retry loops, redundant calls, wrong turns that later needed correction, and USER FEEDBACK (corrections, re-asks, complaints). The ideal trace commits none of them: the correct route is taken the first time. Where the real agent repaired itself, the ideal trace shows only the correct action — not the failure, not the repair. Why: the assistant turns are the loss-bearing target. A mistake in the target is imitated; a mistake in the context is not. See §4 for the one narrow exception (environment-caused failures).

P3 — Perfect tool calls, only when really needed. Every call is necessary, correctly parameterized, and the natural next step. Independent calls are batched in parallel.

Batching is structural, not stylistic. When the real trace issues two calls in separate assistant turns but the second does not consume the first's output, the ideal path issues them in ONE assistant turn carrying both tool_calls, with the two tool results following in the same order. Do not preserve the source's one-call-per-turn rhythm — that rhythm is the defect. Measured 2026-10-25: 75 of 621 repair-exhausted failures are exactly this shape. A duplicate call (same tool, same arguments, same turn) is never batched; it is dropped. No exploratory calls, no re-reads of what is already in context, no call whose result the narrative doesn't use. When the ideal path makes the SAME call as the real trace, the tool result must be the real trace's output — verbatim or a faithful compression of it — never a plausible re-simulation of what the call might have printed.

P4 — Fact-driven, no hallucinations (two layers, both binding). Layer 1 (trace-internal): reuse the real trace's tool outputs verbatim wherever the ideal path makes the same call; never invent files, commands, or results; no fake success — the final answer is grounded in real evidence from the trace. Layer 2 (external world): statements about the outside world (library versions, API behavior, dates, technical facts) must be either (a) taken from the trace's own evidence or (b) kept out unless you are certain they are correct. The harness fact-checks external claims with live web searches after you finish — unsupported claims send the trace back. Construction order: decide the ideal call path → copy each result byte-verbatim from outputs.jsonl → only then write reasoning and the final answer around those real results. A tool result you cannot quote is not writable: reroute the path or drop the claim. Stitching individually-plausible fragments is fabrication and is the single most common rejection.

P5 — Perfect grammar and language quality, single language. One language per trace: the language of the original first user message. No code-switching, no stray fragments in other languages. Assistant prose is clean, grammatically perfect, professionally concise. The user message is kept verbatim — realism of the input matters more than its typos. Never open by restating the brief: the first assistant turn must do work (a tool call, a decision, a finding), not echo the user's own text back at them.

P6 — No refusals and no identity. The ideal agent completes the task without refusing and never claims a model identity ("I am Claude", "as an AI"). If the source task genuinely cannot be completed (harmful or impossible), do NOT write an ideal version: reply starting with REJECTED: plus a one-line reason, and write no output file. The system prompt is not yours to rewrite, but the export strips a harness identity definition from it — never reintroduce one in an assistant turn.

4. Training-signal contract (why the pillars are shaped this way)

The corpus is consumed by SFT: loss is computed on assistant target tokens only; tool results, user turns and system prompts are masked context. The model imitates what is in the target span.

Failures may be context. Failures must never be target.

Therefore:

1. A model-caused error must never appear in an assistant turn — not as "I'll try X", not as a failed call, not as a self-correction narrative. Wrong tool name, wrong arguments, wrong path, bad reasoning: route around it. This is P2.
2. An environment-caused error may appear as a tool result (context), with the correct adaptation in the next assistant turn. This is the only exception to P2, and it exists because a corpus of only first-try-clean paths loses the only examples of "tool returned an error → read it → adapt", and silently skews the corpus toward easy tasks.
3. The recovery is the target, never the mistake. If the ideal path cannot avoid the error, the assistant's next turn is the supervised recovery; the error text stays byte-verbatim in the tool row.

4.1 The narrow recovery carve-out

A trace may keep a real failure only when ALL of these hold:

• the failure is environment-caused — a tool returned an error the ideal agent could not have foreseen (flaky endpoint, missing dependency, a genuinely absent file). Not a wrong tool name, wrong arguments, wrong path, or a reasoning error;
• the error text is byte-verbatim from outputs.jsonl — never composed, never paraphrased, never invented;
• there is one corrective step per error; two or more consecutive identical failed calls disqualify the trace;
• the harness records the trace as a retained recovery (manifest + dataset card) so the corpus can report failure density and a trainer can filter it out;
• the final state is verified — a clean-looking recovery whose final answer is wrong is rejected outright.

Everything else follows P2: route around the mistake, show only the correct action. When in doubt, route around it — the carve-out is an exception, not a license.

4.2 What must never be in the target span

• A fabricated or paraphrased error message.
• A failure the ideal agent could have avoided by reading its own context.
• A "let me try again" narrative whose first attempt is not shown (the model learns the shape of recovery without the substance).
• A recovery loop longer than one corrective step.
• A success claim that the tool results do not support.

5. Rewrite procedure (binding)

1. Read these guidelines in full.
2. Mine the negative signal: list the real trace's failed calls, retries, wrong turns, and user feedback (P2).
3. Classify each failure: harness artefact (wrong cwd, missing env var, stale workdir, tool not installed) → route around it; environment-caused and unavoidable → eligible for the §4.1 carve-out; model-caused → route around it, always.
4. Map the ideal path: the minimal set of correct, necessary calls (P3).
5. Copy every tool result on that path byte-verbatim from outputs.jsonl (P4), then write the reasoning and final answer around them (P1, P5).
6. Write the ideal trace JSON (schema in the task prompt).
7. Verify: re-read the file; check JSON validity, call/result pairing, every pillar, and that no failure sits in an assistant turn. Fix, then summarize the changes.

6. Scope: what this corpus is and is not

This is an evaluation reference. It is not a training corpus by official scope, not a preference corpus, and not an RL environment.

• The benchmark scores a model by how little the pipeline has to edit its traces to reach the reference. The reference is the ruler, not the product.
• SFT reuse is possible — the §4 contract keeps the reference a valid target — but it is a downstream decision, not this repo's purpose.
• KTO/DPO needs rejected samples from the policy being aligned, length-matched to the chosen side, with a bounded negative loss. A hand-written "bad" trace is a strawman and teaches length/style, not preference.
• PPO/GRPO needs rollouts and a verifiable reward; failed rollouts are advantage, not imitation. A static JSONL of ideal traces cannot supply that signal.
• If those corpora are built later, they are separate artifacts with separate rules. Do not repurpose this corpus by relabelling its rows.

Corpus-level obligations that belong to the harness, not the improver: report failure density and difficulty distribution per batch; cap retained recoveries at ≤15% of traces until an in-repo ablation says otherwise; state the masking contract in the dataset card; dedup at the (task, tool-sequence) level.

7. Remaining open items

• System prompt: keep original vs. normalize to a canonical corpus prompt.
• Corpus-level dedup threshold for batch runs (Jaccard > 0.9 is the current audit default).
• Whether the §4.1 carve-out improves or degrades held-out recovery rate — needs an in-repo ablation (same traces, with and without retained recoveries, scored on a forced-error recovery eval).

8. Provenance

Improved traces carry source session id, parent session id, and generator version; the manifest records per-trace status, attempts, and errors. The harness records retained recoveries (§4.1) in the manifest and reports failure density in the dataset card.