Blame for guidelines/IDEAL_TRACE_GUIDELINES.md

# Ideal-Trace Guidelines — v0.3

> **Authoritative spec for what an "ideal trace" looks like.** Read at runtime
> by the improver, digest, long-trace and review stages. The six quality
> pillars (§3) are owner-decided and binding. §4 states the training-signal
> contract that makes the pillars necessary; §6 scopes what this reference is
> for (evaluation, not training). Evidence base: `docs/DATA_QUALITY.md`.

>

> **Integrity rule:** this file is a spec, not a notebook. Keep it under

> 24 KB. Never paste a worked example, a trace, or your own reasoning into it

> — examples live in `data/traces/improved/`. `tests/test_guidelines.py`
> enforces this.

## 1. Purpose

An ideal trace is the **ideal-world execution of the same task**: same intent,
same real environment, flawless execution. It is NOT a re-execution — it is a
rewrite of the recorded trace, using the trace's own tool outputs as the only
evidence about the world.

The ideal trace is the **reference standard** the benchmark scores a model
against: the less the pipeline has to edit a model's trace to reach it, the
better the model. The training-signal contract in §4 is retained because the
reference must also be a valid target if the corpus is reused for SFT — but
the official scope is evaluation, not training.

## 2. Stable structural rules (hard requirements)

1. **Preserve the task intent.** The first user message carries the task. The
   ideal trace solves *that* task. When the source trace starts with a handoff
   artifact — a Hermes `[CONTEXT COMPACTION …]` row, or a Claude Code "This
   session is being continued…" summary row — the first REAL user task message
   defines the intent: reproduce it verbatim and byte-exact as the ideal
   trace's opening message.
2. **Tool-call pairing.** Every assistant tool_call has exactly one matching

   tool result; every tool result references an existing call; tool arguments

   are valid JSON strings.

3. **Message shape.** user / assistant(+calls) / tool / … / final assistant.

   The last message is the assistant's final answer. No two consecutive user

   rows, no two consecutive text-only assistant rows.

4. **Metadata is set by the harness**, not by the improver.

5. **No scaffold leakage.** Never copy prompt scaffolding, clip markers

   (`…[+N chars clipped]`), compaction banners, or harness self-talk into the

   ideal trace.



## 3. The six quality pillars (owner-decided, binding)



**P1 — Efficient but deep reasoning.**

Reasoning is sharp and high-signal: the key decision logic per turn (why this

call, what the result implies, what to verify next). No filler, no restating

the obvious, no rambling. Depth where it matters (diagnoses, trade-offs,

verification logic) — silence where it doesn't. **Answer focus:** every

answer, final or mid-trace, gives exactly what its question asked for. A

follow-up gets only the new piece; never re-present facts the user already

received. Over-answering is a P1 violation.



**Deliver the substance, not a description of it.** The final answer must

present the concrete result the task asked for — the actual numbers, names,

paths, values, or code from the tool evidence — not a summary of what was

retrieved. "I pulled the nutrition data and the recipe" is poor; the data and

the recipe ARE the answer. Naming the work instead of showing its result fails

the task even when every tool call was correct.



**P2 — No committed mistakes (mine the negative signal first).**

Before writing, identify every mistake in the real trace: failed tool calls

(error results), retry loops, redundant calls, wrong turns that later needed

correction, and USER FEEDBACK (corrections, re-asks, complaints). The ideal

trace commits none of them: the correct route is taken the first time. Where

the real agent repaired itself, the ideal trace shows only the correct action

— not the failure, not the repair.

*Why:* the assistant turns are the loss-bearing target. A mistake in the

target is imitated; a mistake in the context is not. See §4 for the one

narrow exception (environment-caused failures).



**P3 — Perfect tool calls, only when really needed.**

Every call is necessary, correctly parameterized, and the natural next step.

Independent calls are batched in parallel.



**Batching is structural, not stylistic.** When the real trace issues two

calls in separate assistant turns but the second does not consume the first's

output, the ideal path issues them in ONE assistant turn carrying both

`tool_calls`, with the two tool results following in the same order. Do not
preserve the source's one-call-per-turn rhythm — that rhythm is the defect.
Measured 2026-10-25: 75 of 621 repair-exhausted failures are exactly this
shape. A duplicate call (same tool, same arguments, same turn) is never
batched; it is dropped. No exploratory calls, no re-reads
of what is already in context, no call whose result the narrative doesn't
use. When the ideal path makes the SAME call as the real trace, the tool
result must be the real trace's output — verbatim or a faithful compression
of it — never a plausible re-simulation of what the call might have printed.

**P4 — Fact-driven, no hallucinations (two layers, both binding).**
Layer 1 (trace-internal): reuse the real trace's tool outputs verbatim
wherever the ideal path makes the same call; never invent files, commands, or
results; no fake success — the final answer is grounded in real evidence from
the trace. Layer 2 (external world): statements about the outside world
(library versions, API behavior, dates, technical facts) must be either (a)
taken from the trace's own evidence or (b) kept out unless you are certain
they are correct. The harness fact-checks external claims with live web
searches after you finish — unsupported claims send the trace back.
**Construction order:** decide the ideal call path → copy each result
byte-verbatim from `outputs.jsonl` → only then write reasoning and the final
answer around those real results. A tool result you cannot quote is not
writable: reroute the path or drop the claim. Stitching individually-plausible
fragments is fabrication and is the single most common rejection.

**P5 — Perfect grammar and language quality, single language.**
One language per trace: the language of the original first user message. No
code-switching, no stray fragments in other languages. Assistant prose is
clean, grammatically perfect, professionally concise. The user message is kept
verbatim — realism of the input matters more than its typos. Never open by
restating the brief: the first assistant turn must do work (a tool call, a
decision, a finding), not echo the user's own text back at them.

**P6 — No refusals and no identity.**
The ideal agent completes the task without refusing and never claims a model
identity ("I am Claude", "as an AI"). If the source task genuinely cannot be
completed (harmful or impossible), do NOT write an ideal version: reply
starting with `REJECTED:` plus a one-line reason, and write no output file.
The system prompt is not yours to rewrite, but the export strips a harness
identity definition from it — never reintroduce one in an assistant turn.

## 4. Training-signal contract (why the pillars are shaped this way)

The corpus is consumed by SFT: loss is computed on **assistant target tokens

only**; tool results, user turns and system prompts are masked context. The
model imitates what is in the target span.

> **Failures may be context. Failures must never be target.**

Therefore:

1. **A model-caused error must never appear in an assistant turn** — not as
   "I'll try X", not as a failed call, not as a self-correction narrative.
   Wrong tool name, wrong arguments, wrong path, bad reasoning: route around
   it. This is P2.
2. **An environment-caused error may appear as a tool result** (context), with
   the correct adaptation in the next assistant turn. This is the only
   exception to P2, and it exists because a corpus of only first-try-clean
   paths loses the only examples of "tool returned an error → read it →
   adapt", and silently skews the corpus toward easy tasks.
3. **The recovery is the target, never the mistake.** If the ideal path cannot
   avoid the error, the assistant's next turn is the supervised recovery; the
   error text stays byte-verbatim in the tool row.

### 4.1 The narrow recovery carve-out

A trace may keep a real failure **only** when ALL of these hold:

- the failure is **environment-caused** — a tool returned an error the ideal
  agent could not have foreseen (flaky endpoint, missing dependency, a
  genuinely absent file). Not a wrong tool name, wrong arguments, wrong path,
  or a reasoning error;
- the error text is **byte-verbatim** from `outputs.jsonl` — never composed,
  never paraphrased, never invented;
- there is **one corrective step per error**; two or more consecutive
  identical failed calls disqualify the trace;
- the harness records the trace as a **retained recovery** (manifest + dataset
  card) so the corpus can report failure density and a trainer can filter it
  out;
- the final state is **verified** — a clean-looking recovery whose final
  answer is wrong is rejected outright.

Everything else follows P2: route around the mistake, show only the correct
action. When in doubt, route around it — the carve-out is an exception, not a
license.

### 4.2 What must never be in the target span

- A fabricated or paraphrased error message.
- A failure the ideal agent could have avoided by reading its own context.
- A "let me try again" narrative whose first attempt is not shown (the model
  learns the shape of recovery without the substance).
- A recovery loop longer than one corrective step.
- A success claim that the tool results do not support.

## 5. Rewrite procedure (binding)

1. Read these guidelines in full.
2. Mine the negative signal: list the real trace's failed calls, retries,
   wrong turns, and user feedback (P2).
3. Classify each failure: **harness artefact** (wrong cwd, missing env var,
   stale workdir, tool not installed) → route around it; **environment-caused

   and unavoidable** → eligible for the §4.1 carve-out; **model-caused** →
   route around it, always.
4. Map the ideal path: the minimal set of correct, necessary calls (P3).
5. Copy every tool result on that path byte-verbatim from `outputs.jsonl`
   (P4), then write the reasoning and final answer around them (P1, P5).
6. Write the ideal trace JSON (schema in the task prompt).
7. Verify: re-read the file; check JSON validity, call/result pairing, every
   pillar, and that no failure sits in an assistant turn. Fix, then summarize
   the changes.

## 6. Scope: what this corpus is and is not

This is an **evaluation reference**. It is not a training corpus by official
scope, not a preference corpus, and not an RL environment.

- **The benchmark** scores a model by how little the pipeline has to edit its
  traces to reach the reference. The reference is the ruler, not the product.
- **SFT reuse** is possible — the §4 contract keeps the reference a valid
  target — but it is a downstream decision, not this repo's purpose.
- **KTO/DPO** needs rejected samples from the policy being aligned,
  length-matched to the chosen side, with a bounded negative loss. A
  hand-written "bad" trace is a strawman and teaches length/style, not
  preference.
- **PPO/GRPO** needs rollouts and a verifiable reward; failed rollouts are
  advantage, not imitation. A static JSONL of ideal traces cannot supply that
  signal.
- If those corpora are built later, they are separate artifacts with separate
  rules. Do not repurpose this corpus by relabelling its rows.

Corpus-level obligations that belong to the harness, not the improver: report
failure density and difficulty distribution per batch; cap retained recoveries
at ≤15% of traces until an in-repo ablation says otherwise; state the masking
contract in the dataset card; dedup at the (task, tool-sequence) level.

## 7. Remaining open items

- System prompt: keep original vs. normalize to a canonical corpus prompt.
- Corpus-level dedup threshold for batch runs (Jaccard > 0.9 is the current
  audit default).
- Whether the §4.1 carve-out improves or degrades held-out recovery rate —
  needs an in-repo ablation (same traces, with and without retained
  recoveries, scored on a forced-error recovery eval).

## 8. Provenance

Improved traces carry source session id, parent session id, and generator
version; the manifest records per-trace status, attempts, and errors. The
harness records retained recoveries (§4.1) in the manifest and reports failure
density in the dataset card.