research/07-sdpo-hint-generator.md

# SDPO Hint Generation: How to Build the Teacher's "Privileged Info" for Composer's Targeted RL with Textual Feedback

> **Research date:** 2026-05-28.
> **Scope:** Resolves the **#1 open replication question** flagged in `docs/COMPOSER_RECIPE_MAPPING.md` §1 and `research/09-composer-blog-delta-2026.md` §2: *how are the hints generated?* This doc maps OPSD/SDPO's "privileged information" onto Composer's "hint," builds a cheapest→richest **taxonomy of hint sources**, ships a **concrete template library with actual strings**, specifies the **LLM-judge fallback prompt**, aligns **error-site detection** with `ingestion/trace_examples.py`, and proposes a **layered `HintGenerator` design** that slots into the existing `CollatorConfig.hint_generator` hook.
> **Method:** Primary-source pulls (Tavily advanced) of the SDPO abstract + method/ablation HTML ([arXiv:2601.20802v2](https://arxiv.org/abs/2601.20802)), the OPSD method HTML + GitHub README ([arXiv:2601.18734v3](https://arxiv.org/abs/2601.18734), [siyan-zhao/OPSD](https://github.com/siyan-zhao/OPSD)), audited against the current `hint_generator.py`, `trainer/data_collator.py`, and `ingestion/trace_examples.py`.

---

## TL;DR

The hint **is** the teacher's privileged-information conditioning variable. Cursor never says how hints are generated, but the two cited papers bracket the answer precisely:

- **OPSD** conditions the teacher on `y⋆` = **ground-truth answer / reference CoT** — the strongest, most "privileged" signal, available only when you hold the solution.
- **SDPO** generalizes this to **environment feedback** that you *already have for free* at training time, and crucially ablates **three feedback types**: (1) a **successful sibling rollout** ("sample solution"), (2) the **environment output** (runtime errors / judge text), and (3) the **student's own original attempt**. The teacher is the *same weights* conditioned on that feedback; the student is the same weights without it; loss is a per-token KL on the student's trajectory, gradient through the student only, teacher stop-grad.

Composer's "hint" is therefore **not one thing** — it is *whatever cheap, locally-available text shifts the teacher distribution toward the correct continuation*. That reframing makes the open question tractable: build a **layered generator** that tries the cheapest source first and escalates only on miss:

```
template-by-error-kind  →  raw-tool-error-as-hint  →  LLM-judge hint  →  SDPO successful-sibling bootstrap
   (free, deterministic)      (free, structural)        (~$0.0005/site)      (free, needs a rollout group)
```

The current `hint_generator.py` implements **only the first layer** (5 templates) and is the right v0.1 starting point. This doc specifies layers 2–4 and a clean `HintGenerator` Protocol so they compose behind the existing `Callable[[str, dict], str | None]` hook with **zero collator changes**.

---

## 1. How OPSD & SDPO obtain the teacher's "privileged info" → Composer's "hint"

Both methods build teacher and student from **a single LLM** and differ only in *what extra text the teacher gets to see*. That extra text is the privileged-information variable. Composer's "hint inserted into the local context" is exactly this variable.

### 1.1 OPSD — privileged info = ground-truth answer

> *"The teacher policy is provided with privileged information `y⋆`, such as the **ground-truth answer or a reference chain-of-thought**, while the student policy conditions only on the problem `x`. … the teacher policy `p_T(·|x, y⋆)` conditions on both the problem and the privileged answer, whereas the student policy `p_S(·|x)` observes only the problem. We preserve the on-policy training paradigm by sampling trajectories `ŷ` exclusively from the student policy, which then receives dense, token-level supervision from the privileged teacher policy."* — OPSD, [arXiv:2601.18734v3](https://arxiv.org/html/2601.18734v3)

OPSD loss (Eq. 8, verbatim structure):

```
L(θ) = E_{(x, y⋆)~S} [ E_{ŷ~p_S(·|x)} [ D( p_T ‖ p_S )(ŷ | x) ] ]
```

> *"Gradients are backpropagated only through the student policy `p_S`, while the teacher `p_T` acts as a fixed full-distribution target conditioned on privileged information `(x, y⋆)`."*

**Map to Composer:** `y⋆` ≙ the **hint**. In OPSD the hint is maximally strong (the answer itself). In a coding agent you rarely have the answer at an arbitrary turn — so the OPSD form is the *upper bound* of hint strength, usable only for the subset of error sites where a reference exists (e.g. the deleted code in a Feature-Deletion task, or a known-good tool signature).

Two OPSD implementation handles that transfer directly (from the [GitHub README](https://github.com/siyan-zhao/OPSD)):
- **`--reason_first`**: *"Prepend an explicit rationalization to the teacher context before distillation."* This is OPSD's own knob for **same-model introspection** (taxonomy class (d) below) — the teacher is first asked to rationalize *why* the privileged info implies the answer, then distilled. Evidence the introspection-hint path is real and works.
- **`--jsd_token_clip`** (default `0.05`): *"Clip the JSD loss for each token … This can improve stability by preventing **stylistic tokens from dominating** the training signal."* Directly relevant to Composer's **style/communication** behavior targets — without clipping, distilling a style hint can be dominated by a few high-divergence stylistic tokens. Our collator's `sdpo_loss_mask` already isolates post-hint tokens; token-clipping is the complementary per-token stabilizer.

### 1.2 SDPO — privileged info = environment feedback (three sources, ablated)

> *"SDPO treats the current model **conditioned on feedback** as a self-teacher and distills its feedback-informed next-token predictions back into the policy. … SDPO leverages the model's ability to **retrospectively identify its own mistakes in-context**."* — SDPO abstract, [arXiv:2601.20802v2](https://arxiv.org/abs/2601.20802)

SDPO explicitly ablates **three feedback types present in a verifiable coding environment** ([SDPO method HTML](https://arxiv.org/html/2601.20802v2)):

> *"we ablate the three types of feedback present in a verifiable environment like code generation: **the sample solution** (if a successful rollout is available in the current rollout group), **the environment output** (such as runtime errors), and **the student's original attempt**."*

This is the load-bearing finding for our taxonomy. Each maps to a distinct hint source:

| SDPO feedback type | What it is | Composer "hint" equivalent | Taxonomy class (§2) |
|---|---|---|---|
| **Sample solution** | A *successful sibling rollout* from the same prompt's rollout group | Bootstrap hint: "Here is a working approach: …" | **(f)** SDPO successful-sibling bootstrap |
| **Environment output** | Runtime error / judge text returned by the env | Raw tool-error text spliced as the hint | **(b)** raw-tool-error-as-hint |
| **Student's original attempt** | The model's own failed text, re-shown | Self-introspection prompt | **(d)** same-model introspection |

The key SDPO lever for the **hint-absent case** (called out in `09-composer-blog-delta-2026.md` §3 action item 3):

> *"SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by **using successful rollouts as implicit feedback for failed attempts**."*

i.e. when there is **no external hint source**, you can still manufacture privileged info by letting the teacher condition on a *sibling rollout that passed*. This is free (you already paid for the rollout group under GRPO) and is the natural last fallback before giving up on an error site.

### 1.3 The exact mechanism nuance to preserve (from the Cursor blog, via delta doc §2)

> *"This hint **changes the probabilities for the teacher, lowering those for the wrong tool and increasing those for a valid replacement**. For that turn only, we then **update the student weights towards the new probabilities**."*

Two facts the hint generator must respect:
1. **Teacher = hint-conditioned forward pass of the same weights** (not a re-rollout, not a separate model). The generator's job is only to *produce the text spliced into the teacher context* — the collator (`_build_hint_injected_trace`) already does the splicing, and the trainer does the forward pass.
2. **Student weights are trainable; teacher is stop-grad.** The generator never touches the loss; it only conditions the teacher. So **a wrong hint is bounded-bad** — it produces a noisier teacher target at one masked turn, not a corrupted reward. This is why we can afford cheap/heuristic hints and only escalate on miss.

---

## 2. Taxonomy of hint sources — cheapest → richest

For each class: applicability, cost, and which **Composer behavior class** it covers. Composer's three stated behavior targets are **tool use, coding style, and model communication** (`09-composer-blog-delta-2026.md` §2), plus **effort calibration** (blog §"behavioral aspects"). Tool errors are the cheap, structural case; style/communication/effort are the hard cases templates can't reach.

| # | Hint source | How obtained | Cost / latency | Determinism | Tool err | Style | Comms | Effort |
|---|---|---|---|---|:--:|:--:|:--:|:--:|
| **(a)** | **Hardcoded template by error_kind** | Pattern-match `error_kind`, fill slots (`available_tools`, `tool_schema`) | **Free**, ~µs | Fully deterministic | ✅ strong | ⚠️ rigid | ⚠️ rigid | ❌ |
| **(b)** | **Raw tool-error text as hint** | Pass the env's error string through (optionally truncated) | **Free**, ~µs | Deterministic | ✅ strong | ❌ | ❌ | ❌ |
| **(c)** | **LLM-judge natural-language hint** | Call a cheap judge model with `(state, erroring_action, tool_output)` | ~$0.0003–0.001/site, ~0.5–2 s | Stochastic | ✅ | ✅ | ✅ | ✅ |
| **(d)** | **Same-model introspection** | Re-prompt the *training model* to critique its own failed turn (OPSD `--reason_first`) | **Free GPU** (1 extra gen), ~0.3–1 s | Stochastic | ✅ | ✅ | ✅ | ✅ |
| **(e)** | **Learned hint generator** | A small fine-tuned model trained to emit hints (defer to v0.2+) | Train-time cost + inference | Stochastic | ✅ | ✅ | ✅ | ✅ |
| **(f)** | **SDPO successful-sibling bootstrap** | Pick a *passing* rollout from the same prompt's GRPO group; condition teacher on it | **Free** (reuses rollout group), ~µs to select | Deterministic given group | ✅ | ✅ | ✅ | ✅ (shows a *terser* success) |

**Reading of the table:**
- **(a)+(b) cover the tool-use behavior class almost entirely** and are free + deterministic → make them the default first layer. This is the "easy case" the mapping doc warns about (`COMPOSER_RECIPE_MAPPING.md` §"Why deferring … is the right call", point 2): they *do not* validate the harder behavior cases.
- **Style / communication / effort-calibration are NOT pattern-matchable.** "This explanation was wasteful" or "this code violates house style" requires class **(c)**, **(d)**, or **(f)**. This is the real content of the open question.
- **(f) is the unique unlock** when no external hint source exists *and* you don't want an API call: it manufactures privileged info from the model's own successes. It is the natural fallback and also the cheapest source for style/comms/effort because a *successful sibling* implicitly demonstrates the desired style/terseness without anyone writing a rule.
- **(e) learned generator** is explicitly v0.2 (`COMPOSER_RECIPE_MAPPING.md` table row (d): "+ learned hint generator"). Out of scope to build now; the Protocol below makes it a drop-in later.

**Recommended escalation order (rationale):** deterministic-and-free before stochastic, structural before semantic, no-API before API. → `(a) → (b) → [(c) xor (d)] → (f)` with `(f)` as the "nothing else fired but we have a passing sibling" backstop.

---

## 3. Concrete template library (actual strings)

This **extends** the current `hint_generator.py` registry (which already ships `tool_not_found`, `json_decode`, `type_error`, `runtime_error`, `repeated_failure`). New/expanded templates below are written to the **same `HintContext` TypedDict** and same `dispatch(error_kind, ctx)` contract, so they register without touching the collator. All keep the blog's *"Reminder: …"* register (the one verbatim example Cursor published was `"Reminder: Available tools are…"`).

| error_kind | Trigger | Hint string (template) |
|---|---|---|
| `tool_not_found` | invalid tool name | `Reminder: Available tools are: {tool_list}. The tool you called does not exist — use one of these.` |
| `malformed_args` / `json_decode` | unparseable tool args / JSON | `Reminder: tool arguments must be a single valid JSON object. Common mistakes: single quotes (use double quotes), trailing commas, unescaped newlines inside strings, or wrapping the JSON in markdown fences.` |
| `schema_mismatch` / `type_error` | args parse but violate schema | `Reminder: \`{tool_name}\` expects arguments matching this schema:\n  {tool_schema}\nYour call is missing/mistyped: {bad_fields}. Re-issue with arguments matching the schema.` |
| `failing_test` | test suite returns non-zero / assertion | `Reminder: the test \`{test_name}\` is still failing: {assertion_excerpt}. Re-read the failing test's expectations and adjust the implementation to satisfy them — do not modify the test.` |
| `lint_style` | linter/formatter non-zero exit | `Reminder: this change violates the project style ({linter}: {rule_id} — {rule_msg}). Match the surrounding code's conventions (imports, naming, formatting) before proceeding.` |
| `wasteful_action` | redundant/no-op action (effort calibration) | `Reminder: this step repeated work already done (you already {prior_action}). Skip redundant reads/searches and act on what you know; prefer the most direct path to the goal.` |
| `repeated_failure` | same error_kind ≥3× consecutively | `Reminder: this approach has failed {n} times. Step back and try a different strategy: read more of the surrounding code, search for an existing working example, or decompose the task differently.` |
| `verbose_communication` | judge-flagged over-long message (comms) | `Reminder: keep the response concise and focused on the user's request. State what you did and why in 1–2 sentences; omit restating the task and step-by-step narration.` |

Notes:
- `{tool_list}`, `{tool_schema}`, `{bad_fields}`, etc. are filled from `HintContext` (`available_tools`, `tool_schema`, `tool_name`) and from new optional keys (`test_name`, `assertion_excerpt`, `linter`, `rule_id`, `rule_msg`, `prior_action`, `n`).
- `failing_test`, `lint_style`, `wasteful_action`, `verbose_communication` are **new** and extend coverage from tool-use into the style/comms/effort behavior classes at the *template tier* — but they are deliberately generic; the high-quality versions of these come from the LLM-judge (§4) or sibling-bootstrap (§2 class (f)).
- Truncate `{assertion_excerpt}` / `{tool_schema}` to ~200 chars (matches the `source_content_excerpt[:200]` convention already used in `trace_examples.py`) to keep the injected hint short — the blog stresses the hint is **local and short**.

---

## 4. LLM-judge path (class (c))

When no template fires, or when the behavior class is style/comms/effort, call a cheap judge to emit a ≤2-sentence corrective hint. The judge sees the *failed* turn and the *environment's* reaction — never the ground truth (we usually don't have it) — and is asked to produce the *minimal corrective nudge* that the teacher will then condition on.

### 4.1 Prompt template

```text
SYSTEM:
You write a single, short corrective hint for a coding agent that just made a
mistake. The hint will be inserted into the agent's context so it can retry the
SAME turn. Output ONE hint of AT MOST 2 sentences. Be concrete and actionable.
Do NOT solve the task, do NOT write code, do NOT explain your reasoning. If the
action was actually fine, output exactly: NO_HINT.

USER:
## Conversation state (last {k} turns)
{state}

## The action that went wrong
{erroring_action}

## What the environment returned
{tool_output}

## Behavior dimension to correct (one of: tool_use | style | communication | effort)
{behavior_dim}

Write the hint now (≤2 sentences, or NO_HINT):
```

- `{state}` = last `k≈3` turns (truncate to a token budget, e.g. 1.5k tokens).
- `{erroring_action}` = the assistant turn's tool call / message that failed.
- `{tool_output}` = the env error or judge text (the same string class (b) would pass raw).
- `{behavior_dim}` = routed from the error-site detector (§5): structural tool errors → `tool_use`; judge-flagged turns → `style`/`communication`/`effort`.
- `NO_HINT` sentinel maps to the generator returning `None` (skip the SDPO site), preventing the collator from minting a zero-signal row (the collator already guards "hint AND recovery content" — `data_collator.py` L308).

### 4.2 Model tier & rough cost

- **Tier:** a *small/cheap* instruct model is sufficient — the task is "spot the obvious mistake and say it in 2 sentences," not solve. Candidates: a 7–8B local model already loaded for rollouts (zero marginal $), or a hosted small model (e.g. Sonnet-class / GPT-mini-class via OpenRouter, consistent with the existing `hint_generator.py` docstring that names "Sonnet 4.6 or Opus 4.7 via OpenRouter" for v0.2).
- **Cost (hosted small model):** input ≈ 1.5k–2k tok (state + action + output), output ≤ ~60 tok. At ~$0.15/M in + ~$0.60/M out that is **≈ $0.0003–0.0006 per error site**. With error sites at, say, 1–3 per trace, this is **~$0.001–0.002/trace** — an order of magnitude cheaper than the trace-replay channel's ~$0.30/trace (`COMPOSER_RECIPE_MAPPING.md` §"three reward channels"), and only paid on **template misses**.
- **Cost (local judge):** effectively free GPU time; preferred at scale. Use the hosted path for v0.1 quality calibration, then distill to local.
- **Caching:** hints are deterministic-enough to cache keyed on `hash(erroring_action + tool_output + behavior_dim)`; repeated identical error sites across a training run reuse the hint for free.

---

## 5. Error-site detection in a trace (align with `ingestion/trace_examples.py`)

The hint generator must only fire at **error sites**. The pipeline already has two layers of structural detection that the generator must align with — do **not** invent a parallel detector.

**Existing structural detection (authoritative, do not duplicate):**
1. **Ingestion → `trace_examples.py`** sets `turn["tool_error"] = <error_kind>` on the assistant turn *immediately after* an error tool-result. It detects errors via:
   - **Structural flag first** (`_user_turn_has_error`): the ingester sets `tool_error: True` on user messages whose source JSONL had `is_error: true`. **This is the source of truth.**
   - **String-tag fallback**: matches `TOOL_ERROR_TAG = "[TOOL_RESULT (ERROR)]"` only when no structural flag is present (older traces).
   - **`error_kind` classification** (`default_classify_error`): keyword regex → `command_not_found`, `file_not_found`, `permission_denied`, `syntax_error`, `connection_error`, else `tool_error`.
2. **Collator → `data_collator.py`** (`_is_error_turn`): an error site iff `turn.get("tool_error") is not None`, AND it only mints an SDPO row when **both** a hint is produced **and** the recovery turn has content (L308) — so empty-recovery sites are skipped.

**Extending the detector for the new behavior/error classes (additions, not replacements).** Keep `error_kind` as the routing key the generator already receives, and broaden the classifier so the new templates (§3) and the judge router (§4) get the right `behavior_dim`:

| Signal in trace | Detected via | New `error_kind` → behavior_dim |
|---|---|---|
| Failed tool status / `is_error: true` | structural flag (existing) | `tool_error`/`tool_not_found`/… → `tool_use` |
| Exception traceback in tool output | regex `Traceback (most recent call last)` / `Error:` | `runtime_error` → `tool_use` |
| Malformed args / JSON | parse failure of the tool-call args | `malformed_args`/`json_decode` → `tool_use` |
| Test runner non-zero exit / assertion | regex `FAILED|AssertionError|[0-9]+ failed` in output | `failing_test` → `tool_use` (verifiable) |
| Linter/formatter non-zero exit | regex `{ruff|eslint|flake8|black}.*(error|would reformat)`; nonzero exit code | `lint_style` → `style` |
| Redundant/no-op action | heuristic: action equals a prior action's signature; or no state delta | `wasteful_action` → `effort` |
| Over-long / off-task assistant message | **LLM-judge flag only** (no structural signal) | `verbose_communication` → `communication` |

Implementation alignment rule: **add these as new `(kind, regex)` rows to `_ERROR_KIND_PATTERNS` in `trace_examples.py`** (same ordered-precedence mechanism already there — note its comment that `command_not_found` must precede `file_not_found`), so detection stays in **one** place and the generator stays a pure `error_kind → hint` function. Style/comms/effort sites that have **no structural signature** are surfaced only by the judge and should be gated (sampled, e.g. 10–20% of clean turns) to bound cost.

---

## 6. Recommended layered design + `HintGenerator` Protocol

### 6.1 Protocol

A clean, typed Protocol that subsumes the current `dispatch` and the existing `CollatorConfig.hint_generator: Callable[[str, dict], str | None]` hook. The collator calls `generator(error_kind, error_meta)`; we wrap the Protocol with a tiny adapter so **no collator change is required**.

```python
# composer_replication/hints/protocol.py
from __future__ import annotations
from typing import Protocol, TypedDict, runtime_checkable


class ErrorContext(TypedDict, total=False):
    """Everything a hint source might need. Superset of the current HintContext."""
    error_kind: str            # routing key from trace_examples classifier
    behavior_dim: str          # "tool_use" | "style" | "communication" | "effort"
    error_message: str         # raw env/tool error text  (enables class (b))
    available_tools: list[str]
    tool_name: str
    tool_schema: dict
    state_excerpt: str         # last-k turns, for the judge (class (c)/(d))
    erroring_action: str       # the failed assistant turn
    sibling_rollouts: list[dict]  # GRPO group; passing ones enable class (f)
    repeat_count: int          # for repeated_failure


@runtime_checkable
class HintGenerator(Protocol):
    """A hint source. Returns the hint text, or None to decline this site."""
    def generate(self, ctx: ErrorContext) -> str | None: ...
```

### 6.2 Layered composite (template-first → judge → sibling-bootstrap)

```python
# composer_replication/hints/layered.py
from dataclasses import dataclass, field
from .protocol import HintGenerator, ErrorContext
from . import templates          # wraps the existing HINT_TEMPLATES registry
from . import judge              # LLM-judge generator (class (c))
from . import sibling            # SDPO successful-sibling bootstrap (class (f))


@dataclass
class LayeredHintGenerator:
    """Try each source in order; first non-None wins. A wrong hint is
    bounded-bad (teacher is stop-grad), so cheap layers go first and we
    only escalate to paid/learned layers on a miss."""
    layers: list[HintGenerator] = field(default_factory=list)

    def generate(self, ctx: ErrorContext) -> str | None:
        for layer in self.layers:
            hint = layer.generate(ctx)
            if hint:                      # non-empty, non-None
                return hint
        return None                       # collator then skips this SDPO site

    # Adapter for the existing CollatorConfig.hint_generator signature.
    def as_collator_hook(self):
        def hook(error_kind: str, error_meta: dict) -> str | None:
            ctx: ErrorContext = {"error_kind": error_kind, **(error_meta or {})}
            return self.generate(ctx)
        return hook


def default_layered(*, judge_client=None, enable_judge=True) -> LayeredHintGenerator:
    layers: list[HintGenerator] = [
        templates.TemplateHintGenerator(),          # (a) free, deterministic
        templates.RawErrorHintGenerator(),          # (b) raw env error as hint
    ]
    if enable_judge and judge_client is not None:
        layers.append(judge.JudgeHintGenerator(judge_client))   # (c)
    layers.append(sibling.SiblingBootstrapGenerator())          # (f) backstop
    return LayeredHintGenerator(layers=layers)
```

Wiring (unchanged collator contract):

```python
gen = default_layered(judge_client=my_small_model_client)
config = CollatorConfig(hint_generator=gen.as_collator_hook(), enable_sdpo=True)
collator = ComposerDataCollator(tokenizer=tok, config=config)
```

### 6.3 The three new layers (sketches)

```python
# (a)+(b) templates.py — reuse the EXISTING registry verbatim
from composer_replication.hint_generator import dispatch   # current module

class TemplateHintGenerator:
    def generate(self, ctx):
        return dispatch(ctx.get("error_kind", ""), ctx)     # None on unknown kind

class RawErrorHintGenerator:
    """Class (b): SDPO 'environment output' feedback — splice the raw error."""
    def generate(self, ctx):
        msg = (ctx.get("error_message") or "").strip()
        if not msg:
            return None
        return f"Reminder: the previous action returned this error:\n{msg[:200]}\nFix the cause and retry."
```

```python
# (c) judge.py — class (c), prompt from §4.1
class JudgeHintGenerator:
    def __init__(self, client, cache=None):
        self.client, self.cache = client, (cache if cache is not None else {})
    def generate(self, ctx):
        key = hash((ctx.get("erroring_action"), ctx.get("error_message"),
                    ctx.get("behavior_dim")))
        if key in self.cache:
            return self.cache[key]
        hint = self.client.complete(_build_judge_prompt(ctx))  # §4.1 template
        hint = None if hint.strip() == "NO_HINT" else hint.strip()
        self.cache[key] = hint
        return hint
```

```python
# (f) sibling.py — SDPO 'successful rollouts as implicit feedback'
class SiblingBootstrapGenerator:
    """Class (f): when nothing else fired but a sibling rollout in the same
    GRPO group PASSED, condition the teacher on that success."""
    def generate(self, ctx):
        sibs = ctx.get("sibling_rollouts") or []
        winners = [s for s in sibs if s.get("reward", 0.0) > 0.0]
        if not winners:
            return None
        best = max(winners, key=lambda s: s["reward"])
        snippet = (best.get("solution_excerpt") or "")[:200]
        return ("Reminder: a working approach for this task looks like:\n"
                f"{snippet}\nAdapt this to the current step.")
```

> **Note on class (d):** same-model introspection (OPSD `--reason_first`) is the *training model* critiquing its own turn — best implemented inside the trainer (where the model is loaded) rather than the collator, since it needs a model forward pass. Add it as a fourth layer once the trainer exposes a `self_critique(ctx) -> str` callable; the Protocol already supports it. For v0.1, the judge (c) is the simpler stand-in for the same role.

### 6.4 Why this order (decision summary)

1. **Templates + raw-error (a/b)** are free, deterministic, and cover the **tool-use** class — the bulk of structural error sites. They reproduce Cursor's one published example (`"Reminder: Available tools are…"`) exactly.
2. **Judge (c)** is the only layer that *manufactures* a corrective for **style / communication / effort**, the behavior classes the mapping doc flags as the real test of the recipe (`COMPOSER_RECIPE_MAPPING.md` §"point 2"). Gated + cached → ~$0.0005/site, paid only on template miss.
3. **Sibling-bootstrap (f)** is the SDPO-native fallback when there's no template, no judge (or judge declined), but the rollout group contains a winner — *free* privileged info from the model's own success. This is the lever `09-composer-blog-delta-2026.md` §3 action item 3 told us to record.
4. **Learned generator (e)** drops in as a new layer in v0.2 (`COMPOSER_RECIPE_MAPPING.md` table row (d): "+ learned hint generator") without touching the Protocol or the collator.

---

## 7. Implementation handles (v0.1)

Concrete, ordered work items. Everything below preserves the existing `CollatorConfig.hint_generator: Callable[[str, dict], str | None]` contract — **no collator surgery**.

1. **Keep `hint_generator.py` as layer (a).** It already implements 5 templates with the right `dispatch(error_kind, ctx) -> str | None` signature. Add the four new templates from §3 (`failing_test`, `lint_style`, `wasteful_action`, `verbose_communication`) via `register(...)`. **Actual strings shipped in §3** — copy verbatim.
2. **Add the new error_kind regexes to `_ERROR_KIND_PATTERNS` in `trace_examples.py`** (§5 table). Single source of detection truth; preserve the ordered-precedence comment pattern (`command_not_found` before `file_not_found`). Route each `error_kind → behavior_dim` so the judge gets correct routing.
3. **Build `composer_replication/hints/`** with `protocol.py`, `layered.py`, `templates.py`, `judge.py`, `sibling.py` (§6 sketches). `templates.py` *imports the existing `dispatch`* — do not reimplement.
4. **Wire via the adapter:** `CollatorConfig(hint_generator=default_layered(...).as_collator_hook())`. The `claude_states_to_trace_examples` adapter already populates `error_meta` (`source_content_excerpt[:200]`); extend it to also stash `error_message` (for class (b)) and, when available from the GRPO loop, `sibling_rollouts` (for class (f)).
5. **Borrow OPSD stabilizers for the loss side:** when distilling style/comms hints, apply **per-token JSD clipping** (OPSD `--jsd_token_clip`, default `0.05`) so "stylistic tokens" don't dominate — the README states this is exactly why it exists. Pair with the collator's existing `sdpo_loss_mask` (post-hint tokens only).
6. **Gate the judge:** fire (c) only on (i) template miss, or (ii) a sampled fraction (~10–20%) of clean turns flagged for style/comms/effort, with hint caching keyed on `(erroring_action, error_message, behavior_dim)`. Bounds cost at ~$0.001–0.002/trace.
7. **Eval the generator independently of training** (matches `COMPOSER_RECIPE_MAPPING.md` concern that "SDPO with hardcoded templates is the easy case"): measure (a) % of error sites that get a non-None hint per layer, (b) teacher-vs-student KL *increase* at hinted turns (a good hint should *raise* divergence — it's shifting probability toward the fix, per the blog's "lowering wrong-tool, raising valid-replacement"), and (c) for style/comms, a held-out judge agreeing the hint is corrective. A hint that doesn't move the teacher distribution is a no-op and should be pruned.

---

## 8. Citations

- **SDPO** — Hübotter, Lübeck, Behric, Baumann, Bagatella, Marta, Hakimi, Shenfeld, Kleine Buening, Guestrin, Krause (ETH Zürich), *Reinforcement Learning via Self-Distillation*, [arXiv:2601.20802v2](https://arxiv.org/abs/2601.20802) (v1 28 Jan 2026, v2 16 Feb 2026), CC-BY-4.0. Abstract + method/ablation HTML ([html v2](https://arxiv.org/html/2601.20802v2)). The three-feedback-type ablation (sample solution / environment output / student's original attempt) and the "successful rollouts as implicit feedback" claim are the load-bearing sources for §1.2 and taxonomy classes (b), (d), (f).
- **OPSD** — Zhao et al., *Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models*, [arXiv:2601.18734v3](https://arxiv.org/abs/2601.18734), code [github.com/siyan-zhao/OPSD](https://github.com/siyan-zhao/OPSD) (paper CC-BY-4.0; verify code license on repo). Privileged-info teacher (`y⋆` = ground-truth/reference CoT), Eq. 8 loss, stop-grad teacher, and the `--reason_first` (introspection) + `--jsd_token_clip` (stylistic-token stabilizer) flags are the sources for §1.1 and the OPSD handles in §7.
- **Cursor blog** — [Introducing Composer 2.5](https://cursor.com/blog/composer-2-5) (2026): the "Reminder: Available tools are…" example, "hint changes the teacher probabilities / update student weights for that turn only," and the three behavior targets (tool use, coding style, model communication). Via `docs/COMPOSER_RECIPE_MAPPING.md` §1 and `research/09-composer-blog-delta-2026.md` §2.
- **Composer 2 technical report** — [arXiv:2603.24477](https://arxiv.org/abs/2603.24477) / [Composer2.pdf](https://cursor.com/resources/Composer2.pdf) (Rush et al.): flagged in the delta doc as the most likely place to resolve hint-generation directly; **still unread** — a dedicated extraction is the recommended follow-up if this design needs validation against Cursor's actual mechanism.
- **In-repo (audited):** `composer_replication/hint_generator.py` (current layer (a)), `composer_replication/trainer/data_collator.py` (`CollatorConfig.hint_generator` hook, `_build_hint_injected_trace`, hint-AND-recovery gate at L308), `composer_replication/ingestion/trace_examples.py` (structural error detection, `_ERROR_KIND_PATTERNS`, `default_classify_error`).

> **Residual gap:** Cursor still never states which hint source they use; this design *brackets* their unknown choice with the OPSD (privileged-answer) and SDPO (environment-feedback + sibling-bootstrap) endpoints and makes all of them composable. The one unread artifact that could collapse the bracket is Composer2.pdf (arXiv:2603.24477).