research: Composer 2.5 data-gen + targeted-textual-feedback deep-research wave

6049d00 about 1 month ago

33.7 kB

SDPO Hint Generation: How to Build the Teacher's "Privileged Info" for Composer's Targeted RL with Textual Feedback

Research date: 2026-05-28. Scope: Resolves the #1 open replication question flagged in docs/COMPOSER_RECIPE_MAPPING.md §1 and research/09-composer-blog-delta-2026.md §2: how are the hints generated? This doc maps OPSD/SDPO's "privileged information" onto Composer's "hint," builds a cheapest→richest taxonomy of hint sources, ships a concrete template library with actual strings, specifies the LLM-judge fallback prompt, aligns error-site detection with ingestion/trace_examples.py, and proposes a layered HintGenerator design that slots into the existing CollatorConfig.hint_generator hook. Method: Primary-source pulls (Tavily advanced) of the SDPO abstract + method/ablation HTML (arXiv:2601.20802v2), the OPSD method HTML + GitHub README (arXiv:2601.18734v3, siyan-zhao/OPSD), audited against the current hint_generator.py, trainer/data_collator.py, and ingestion/trace_examples.py.

TL;DR

The hint is the teacher's privileged-information conditioning variable. Cursor never says how hints are generated, but the two cited papers bracket the answer precisely:

OPSD conditions the teacher on y⋆ = ground-truth answer / reference CoT — the strongest, most "privileged" signal, available only when you hold the solution.
SDPO generalizes this to environment feedback that you already have for free at training time, and crucially ablates three feedback types: (1) a successful sibling rollout ("sample solution"), (2) the environment output (runtime errors / judge text), and (3) the student's own original attempt. The teacher is the same weights conditioned on that feedback; the student is the same weights without it; loss is a per-token KL on the student's trajectory, gradient through the student only, teacher stop-grad.

Composer's "hint" is therefore not one thing — it is whatever cheap, locally-available text shifts the teacher distribution toward the correct continuation. That reframing makes the open question tractable: build a layered generator that tries the cheapest source first and escalates only on miss:

template-by-error-kind  →  raw-tool-error-as-hint  →  LLM-judge hint  →  SDPO successful-sibling bootstrap
   (free, deterministic)      (free, structural)        (~$0.0005/site)      (free, needs a rollout group)

The current hint_generator.py implements only the first layer (5 templates) and is the right v0.1 starting point. This doc specifies layers 2–4 and a clean HintGenerator Protocol so they compose behind the existing Callable[[str, dict], str | None] hook with zero collator changes.

1. How OPSD & SDPO obtain the teacher's "privileged info" → Composer's "hint"

Both methods build teacher and student from a single LLM and differ only in what extra text the teacher gets to see. That extra text is the privileged-information variable. Composer's "hint inserted into the local context" is exactly this variable.

1.1 OPSD — privileged info = ground-truth answer

"The teacher policy is provided with privileged information y⋆, such as the ground-truth answer or a reference chain-of-thought, while the student policy conditions only on the problem x. … the teacher policy p_T(·|x, y⋆) conditions on both the problem and the privileged answer, whereas the student policy p_S(·|x) observes only the problem. We preserve the on-policy training paradigm by sampling trajectories ŷ exclusively from the student policy, which then receives dense, token-level supervision from the privileged teacher policy." — OPSD, arXiv:2601.18734v3

OPSD loss (Eq. 8, verbatim structure):

L(θ) = E_{(x, y⋆)~S} [ E_{ŷ~p_S(·|x)} [ D( p_T ‖ p_S )(ŷ | x) ] ]

"Gradients are backpropagated only through the student policy p_S, while the teacher p_T acts as a fixed full-distribution target conditioned on privileged information (x, y⋆)."

Map to Composer: y⋆ ≙ the hint. In OPSD the hint is maximally strong (the answer itself). In a coding agent you rarely have the answer at an arbitrary turn — so the OPSD form is the upper bound of hint strength, usable only for the subset of error sites where a reference exists (e.g. the deleted code in a Feature-Deletion task, or a known-good tool signature).

Two OPSD implementation handles that transfer directly (from the GitHub README):

--reason_first: "Prepend an explicit rationalization to the teacher context before distillation." This is OPSD's own knob for same-model introspection (taxonomy class (d) below) — the teacher is first asked to rationalize why the privileged info implies the answer, then distilled. Evidence the introspection-hint path is real and works.
--jsd_token_clip (default 0.05): "Clip the JSD loss for each token … This can improve stability by preventing stylistic tokens from dominating the training signal." Directly relevant to Composer's style/communication behavior targets — without clipping, distilling a style hint can be dominated by a few high-divergence stylistic tokens. Our collator's sdpo_loss_mask already isolates post-hint tokens; token-clipping is the complementary per-token stabilizer.

1.2 SDPO — privileged info = environment feedback (three sources, ablated)

"SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. … SDPO leverages the model's ability to retrospectively identify its own mistakes in-context." — SDPO abstract, arXiv:2601.20802v2

SDPO explicitly ablates three feedback types present in a verifiable coding environment (SDPO method HTML):

"we ablate the three types of feedback present in a verifiable environment like code generation: the sample solution (if a successful rollout is available in the current rollout group), the environment output (such as runtime errors), and the student's original attempt."

This is the load-bearing finding for our taxonomy. Each maps to a distinct hint source:

SDPO feedback type	What it is	Composer "hint" equivalent	Taxonomy class (§2)
Sample solution	A successful sibling rollout from the same prompt's rollout group	Bootstrap hint: "Here is a working approach: …"	(f) SDPO successful-sibling bootstrap
Environment output	Runtime error / judge text returned by the env	Raw tool-error text spliced as the hint	(b) raw-tool-error-as-hint
Student's original attempt	The model's own failed text, re-shown	Self-introspection prompt	(d) same-model introspection

The key SDPO lever for the hint-absent case (called out in 09-composer-blog-delta-2026.md §3 action item 3):

"SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts."

i.e. when there is no external hint source, you can still manufacture privileged info by letting the teacher condition on a sibling rollout that passed. This is free (you already paid for the rollout group under GRPO) and is the natural last fallback before giving up on an error site.

1.3 The exact mechanism nuance to preserve (from the Cursor blog, via delta doc §2)

"This hint changes the probabilities for the teacher, lowering those for the wrong tool and increasing those for a valid replacement. For that turn only, we then update the student weights towards the new probabilities."

Two facts the hint generator must respect:

Teacher = hint-conditioned forward pass of the same weights (not a re-rollout, not a separate model). The generator's job is only to produce the text spliced into the teacher context — the collator (_build_hint_injected_trace) already does the splicing, and the trainer does the forward pass.
Student weights are trainable; teacher is stop-grad. The generator never touches the loss; it only conditions the teacher. So a wrong hint is bounded-bad — it produces a noisier teacher target at one masked turn, not a corrupted reward. This is why we can afford cheap/heuristic hints and only escalate on miss.

2. Taxonomy of hint sources — cheapest → richest

For each class: applicability, cost, and which Composer behavior class it covers. Composer's three stated behavior targets are tool use, coding style, and model communication (09-composer-blog-delta-2026.md §2), plus effort calibration (blog §"behavioral aspects"). Tool errors are the cheap, structural case; style/communication/effort are the hard cases templates can't reach.

#	Hint source	How obtained	Cost / latency	Determinism	Tool err	Style	Comms	Effort
(a)	Hardcoded template by error_kind	Pattern-match `error_kind`, fill slots (`available_tools`, `tool_schema`)	Free, ~µs	Fully deterministic	✅ strong	⚠️ rigid	⚠️ rigid	❌
(b)	Raw tool-error text as hint	Pass the env's error string through (optionally truncated)	Free, ~µs	Deterministic	✅ strong	❌	❌	❌
(c)	LLM-judge natural-language hint	Call a cheap judge model with `(state, erroring_action, tool_output)`	~$0.0003–0.001/site, ~0.5–2 s	Stochastic	✅	✅	✅	✅
(d)	Same-model introspection	Re-prompt the training model to critique its own failed turn (OPSD `--reason_first`)	Free GPU (1 extra gen), ~0.3–1 s	Stochastic	✅	✅	✅	✅
(e)	Learned hint generator	A small fine-tuned model trained to emit hints (defer to v0.2+)	Train-time cost + inference	Stochastic	✅	✅	✅	✅
(f)	SDPO successful-sibling bootstrap	Pick a passing rollout from the same prompt's GRPO group; condition teacher on it	Free (reuses rollout group), ~µs to select	Deterministic given group	✅	✅	✅	✅ (shows a terser success)

Reading of the table:

(a)+(b) cover the tool-use behavior class almost entirely and are free + deterministic → make them the default first layer. This is the "easy case" the mapping doc warns about (COMPOSER_RECIPE_MAPPING.md §"Why deferring … is the right call", point 2): they do not validate the harder behavior cases.
Style / communication / effort-calibration are NOT pattern-matchable. "This explanation was wasteful" or "this code violates house style" requires class (c), (d), or (f). This is the real content of the open question.
(f) is the unique unlock when no external hint source exists and you don't want an API call: it manufactures privileged info from the model's own successes. It is the natural fallback and also the cheapest source for style/comms/effort because a successful sibling implicitly demonstrates the desired style/terseness without anyone writing a rule.
(e) learned generator is explicitly v0.2 (COMPOSER_RECIPE_MAPPING.md table row (d): "+ learned hint generator"). Out of scope to build now; the Protocol below makes it a drop-in later.

Recommended escalation order (rationale): deterministic-and-free before stochastic, structural before semantic, no-API before API. → (a) → (b) → [(c) xor (d)] → (f) with (f) as the "nothing else fired but we have a passing sibling" backstop.

3. Concrete template library (actual strings)

This extends the current hint_generator.py registry (which already ships tool_not_found, json_decode, type_error, runtime_error, repeated_failure). New/expanded templates below are written to the same HintContext TypedDict and same dispatch(error_kind, ctx) contract, so they register without touching the collator. All keep the blog's "Reminder: …" register (the one verbatim example Cursor published was "Reminder: Available tools are…").

error_kind	Trigger	Hint string (template)
`tool_not_found`	invalid tool name	`Reminder: Available tools are: {tool_list}. The tool you called does not exist — use one of these.`
`malformed_args` / `json_decode`	unparseable tool args / JSON	`Reminder: tool arguments must be a single valid JSON object. Common mistakes: single quotes (use double quotes), trailing commas, unescaped newlines inside strings, or wrapping the JSON in markdown fences.`
`schema_mismatch` / `type_error`	args parse but violate schema	`Reminder: \`{tool_name}` expects arguments matching this schema:\n {tool_schema}\nYour call is missing/mistyped: {bad_fields}. Re-issue with arguments matching the schema.`
`failing_test`	test suite returns non-zero / assertion	`Reminder: the test \`{test_name}` is still failing: {assertion_excerpt}. Re-read the failing test's expectations and adjust the implementation to satisfy them — do not modify the test.`
`lint_style`	linter/formatter non-zero exit	`Reminder: this change violates the project style ({linter}: {rule_id} — {rule_msg}). Match the surrounding code's conventions (imports, naming, formatting) before proceeding.`
`wasteful_action`	redundant/no-op action (effort calibration)	`Reminder: this step repeated work already done (you already {prior_action}). Skip redundant reads/searches and act on what you know; prefer the most direct path to the goal.`
`repeated_failure`	same error_kind ≥3× consecutively	`Reminder: this approach has failed {n} times. Step back and try a different strategy: read more of the surrounding code, search for an existing working example, or decompose the task differently.`
`verbose_communication`	judge-flagged over-long message (comms)	`Reminder: keep the response concise and focused on the user's request. State what you did and why in 1–2 sentences; omit restating the task and step-by-step narration.`

Notes:

{tool_list}, {tool_schema}, {bad_fields}, etc. are filled from HintContext (available_tools, tool_schema, tool_name) and from new optional keys (test_name, assertion_excerpt, linter, rule_id, rule_msg, prior_action, n).
failing_test, lint_style, wasteful_action, verbose_communication are new and extend coverage from tool-use into the style/comms/effort behavior classes at the template tier — but they are deliberately generic; the high-quality versions of these come from the LLM-judge (§4) or sibling-bootstrap (§2 class (f)).
Truncate {assertion_excerpt} / {tool_schema} to ~200 chars (matches the source_content_excerpt[:200] convention already used in trace_examples.py) to keep the injected hint short — the blog stresses the hint is local and short.

4. LLM-judge path (class (c))

When no template fires, or when the behavior class is style/comms/effort, call a cheap judge to emit a ≤2-sentence corrective hint. The judge sees the failed turn and the environment's reaction — never the ground truth (we usually don't have it) — and is asked to produce the minimal corrective nudge that the teacher will then condition on.

4.1 Prompt template

SYSTEM:
You write a single, short corrective hint for a coding agent that just made a
mistake. The hint will be inserted into the agent's context so it can retry the
SAME turn. Output ONE hint of AT MOST 2 sentences. Be concrete and actionable.
Do NOT solve the task, do NOT write code, do NOT explain your reasoning. If the
action was actually fine, output exactly: NO_HINT.

USER:
## Conversation state (last {k} turns)
{state}

## The action that went wrong
{erroring_action}

## What the environment returned
{tool_output}

## Behavior dimension to correct (one of: tool_use | style | communication | effort)
{behavior_dim}

Write the hint now (≤2 sentences, or NO_HINT):

{state} = last k≈3 turns (truncate to a token budget, e.g. 1.5k tokens).
{erroring_action} = the assistant turn's tool call / message that failed.
{tool_output} = the env error or judge text (the same string class (b) would pass raw).
{behavior_dim} = routed from the error-site detector (§5): structural tool errors → tool_use; judge-flagged turns → style/communication/effort.
NO_HINT sentinel maps to the generator returning None (skip the SDPO site), preventing the collator from minting a zero-signal row (the collator already guards "hint AND recovery content" — data_collator.py L308).

4.2 Model tier & rough cost

Tier: a small/cheap instruct model is sufficient — the task is "spot the obvious mistake and say it in 2 sentences," not solve. Candidates: a 7–8B local model already loaded for rollouts (zero marginal $), or a hosted small model (e.g. Sonnet-class / GPT-mini-class via OpenRouter, consistent with the existing hint_generator.py docstring that names "Sonnet 4.6 or Opus 4.7 via OpenRouter" for v0.2).
Cost (hosted small model): input ≈ 1.5k–2k tok (state + action + output), output ≤ 60 tok. At ~$0.15/M in + ~$0.60/M out that is ≈ $0.0003–0.0006 per error site. With error sites at, say, 1–3 per trace, this is **$0.001–0.002/trace** — an order of magnitude cheaper than the trace-replay channel's ~$0.30/trace (COMPOSER_RECIPE_MAPPING.md §"three reward channels"), and only paid on template misses.
Cost (local judge): effectively free GPU time; preferred at scale. Use the hosted path for v0.1 quality calibration, then distill to local.
Caching: hints are deterministic-enough to cache keyed on hash(erroring_action + tool_output + behavior_dim); repeated identical error sites across a training run reuse the hint for free.

5. Error-site detection in a trace (align with `ingestion/trace_examples.py`)

The hint generator must only fire at error sites. The pipeline already has two layers of structural detection that the generator must align with — do not invent a parallel detector.

Existing structural detection (authoritative, do not duplicate):

Ingestion → trace_examples.py sets turn["tool_error"] = <error_kind> on the assistant turn immediately after an error tool-result. It detects errors via:
- Structural flag first (_user_turn_has_error): the ingester sets tool_error: True on user messages whose source JSONL had is_error: true. This is the source of truth.
- String-tag fallback: matches TOOL_ERROR_TAG = "[TOOL_RESULT (ERROR)]" only when no structural flag is present (older traces).
- error_kind classification (default_classify_error): keyword regex → command_not_found, file_not_found, permission_denied, syntax_error, connection_error, else tool_error.
Collator → data_collator.py (_is_error_turn): an error site iff turn.get("tool_error") is not None, AND it only mints an SDPO row when both a hint is produced and the recovery turn has content (L308) — so empty-recovery sites are skipped.

Extending the detector for the new behavior/error classes (additions, not replacements). Keep error_kind as the routing key the generator already receives, and broaden the classifier so the new templates (§3) and the judge router (§4) get the right behavior_dim:

Signal in trace	Detected via	New `error_kind` → behavior_dim
Failed tool status / `is_error: true`	structural flag (existing)	`tool_error`/`tool_not_found`/… → `tool_use`
Exception traceback in tool output	regex `Traceback (most recent call last)` / `Error:`	`runtime_error` → `tool_use`
Malformed args / JSON	parse failure of the tool-call args	`malformed_args`/`json_decode` → `tool_use`
Test runner non-zero exit / assertion	regex `FAILED	AssertionError
Linter/formatter non-zero exit	regex `{ruff	eslint
Redundant/no-op action	heuristic: action equals a prior action's signature; or no state delta	`wasteful_action` → `effort`
Over-long / off-task assistant message	LLM-judge flag only (no structural signal)	`verbose_communication` → `communication`

Implementation alignment rule: add these as new (kind, regex) rows to _ERROR_KIND_PATTERNS in trace_examples.py (same ordered-precedence mechanism already there — note its comment that command_not_found must precede file_not_found), so detection stays in one place and the generator stays a pure error_kind → hint function. Style/comms/effort sites that have no structural signature are surfaced only by the judge and should be gated (sampled, e.g. 10–20% of clean turns) to bound cost.

6. Recommended layered design + `HintGenerator` Protocol

6.1 Protocol

A clean, typed Protocol that subsumes the current dispatch and the existing CollatorConfig.hint_generator: Callable[[str, dict], str | None] hook. The collator calls generator(error_kind, error_meta); we wrap the Protocol with a tiny adapter so no collator change is required.

# composer_replication/hints/protocol.py
from __future__ import annotations
from typing import Protocol, TypedDict, runtime_checkable


class ErrorContext(TypedDict, total=False):
    """Everything a hint source might need. Superset of the current HintContext."""
    error_kind: str            # routing key from trace_examples classifier
    behavior_dim: str          # "tool_use" | "style" | "communication" | "effort"
    error_message: str         # raw env/tool error text  (enables class (b))
    available_tools: list[str]
    tool_name: str
    tool_schema: dict
    state_excerpt: str         # last-k turns, for the judge (class (c)/(d))
    erroring_action: str       # the failed assistant turn
    sibling_rollouts: list[dict]  # GRPO group; passing ones enable class (f)
    repeat_count: int          # for repeated_failure


@runtime_checkable
class HintGenerator(Protocol):
    """A hint source. Returns the hint text, or None to decline this site."""
    def generate(self, ctx: ErrorContext) -> str | None: ...

6.2 Layered composite (template-first → judge → sibling-bootstrap)

# composer_replication/hints/layered.py
from dataclasses import dataclass, field
from .protocol import HintGenerator, ErrorContext
from . import templates          # wraps the existing HINT_TEMPLATES registry
from . import judge              # LLM-judge generator (class (c))
from . import sibling            # SDPO successful-sibling bootstrap (class (f))


@dataclass
class LayeredHintGenerator:
    """Try each source in order; first non-None wins. A wrong hint is
    bounded-bad (teacher is stop-grad), so cheap layers go first and we
    only escalate to paid/learned layers on a miss."""
    layers: list[HintGenerator] = field(default_factory=list)

    def generate(self, ctx: ErrorContext) -> str | None:
        for layer in self.layers:
            hint = layer.generate(ctx)
            if hint:                      # non-empty, non-None
                return hint
        return None                       # collator then skips this SDPO site

    # Adapter for the existing CollatorConfig.hint_generator signature.
    def as_collator_hook(self):
        def hook(error_kind: str, error_meta: dict) -> str | None:
            ctx: ErrorContext = {"error_kind": error_kind, **(error_meta or {})}
            return self.generate(ctx)
        return hook


def default_layered(*, judge_client=None, enable_judge=True) -> LayeredHintGenerator:
    layers: list[HintGenerator] = [
        templates.TemplateHintGenerator(),          # (a) free, deterministic
        templates.RawErrorHintGenerator(),          # (b) raw env error as hint
    ]
    if enable_judge and judge_client is not None:
        layers.append(judge.JudgeHintGenerator(judge_client))   # (c)
    layers.append(sibling.SiblingBootstrapGenerator())          # (f) backstop
    return LayeredHintGenerator(layers=layers)

Wiring (unchanged collator contract):

gen = default_layered(judge_client=my_small_model_client)
config = CollatorConfig(hint_generator=gen.as_collator_hook(), enable_sdpo=True)
collator = ComposerDataCollator(tokenizer=tok, config=config)

6.3 The three new layers (sketches)

# (a)+(b) templates.py — reuse the EXISTING registry verbatim
from composer_replication.hint_generator import dispatch   # current module

class TemplateHintGenerator:
    def generate(self, ctx):
        return dispatch(ctx.get("error_kind", ""), ctx)     # None on unknown kind

class RawErrorHintGenerator:
    """Class (b): SDPO 'environment output' feedback — splice the raw error."""
    def generate(self, ctx):
        msg = (ctx.get("error_message") or "").strip()
        if not msg:
            return None
        return f"Reminder: the previous action returned this error:\n{msg[:200]}\nFix the cause and retry."

# (c) judge.py — class (c), prompt from §4.1
class JudgeHintGenerator:
    def __init__(self, client, cache=None):
        self.client, self.cache = client, (cache if cache is not None else {})
    def generate(self, ctx):
        key = hash((ctx.get("erroring_action"), ctx.get("error_message"),
                    ctx.get("behavior_dim")))
        if key in self.cache:
            return self.cache[key]
        hint = self.client.complete(_build_judge_prompt(ctx))  # §4.1 template
        hint = None if hint.strip() == "NO_HINT" else hint.strip()
        self.cache[key] = hint
        return hint

# (f) sibling.py — SDPO 'successful rollouts as implicit feedback'
class SiblingBootstrapGenerator:
    """Class (f): when nothing else fired but a sibling rollout in the same
    GRPO group PASSED, condition the teacher on that success."""
    def generate(self, ctx):
        sibs = ctx.get("sibling_rollouts") or []
        winners = [s for s in sibs if s.get("reward", 0.0) > 0.0]
        if not winners:
            return None
        best = max(winners, key=lambda s: s["reward"])
        snippet = (best.get("solution_excerpt") or "")[:200]
        return ("Reminder: a working approach for this task looks like:\n"
                f"{snippet}\nAdapt this to the current step.")

Note on class (d): same-model introspection (OPSD --reason_first) is the training model critiquing its own turn — best implemented inside the trainer (where the model is loaded) rather than the collator, since it needs a model forward pass. Add it as a fourth layer once the trainer exposes a self_critique(ctx) -> str callable; the Protocol already supports it. For v0.1, the judge (c) is the simpler stand-in for the same role.

6.4 Why this order (decision summary)

Templates + raw-error (a/b) are free, deterministic, and cover the tool-use class — the bulk of structural error sites. They reproduce Cursor's one published example ("Reminder: Available tools are…") exactly.
Judge (c) is the only layer that manufactures a corrective for style / communication / effort, the behavior classes the mapping doc flags as the real test of the recipe (COMPOSER_RECIPE_MAPPING.md §"point 2"). Gated + cached → ~$0.0005/site, paid only on template miss.
Sibling-bootstrap (f) is the SDPO-native fallback when there's no template, no judge (or judge declined), but the rollout group contains a winner — free privileged info from the model's own success. This is the lever 09-composer-blog-delta-2026.md §3 action item 3 told us to record.
Learned generator (e) drops in as a new layer in v0.2 (COMPOSER_RECIPE_MAPPING.md table row (d): "+ learned hint generator") without touching the Protocol or the collator.

7. Implementation handles (v0.1)

Concrete, ordered work items. Everything below preserves the existing CollatorConfig.hint_generator: Callable[[str, dict], str | None] contract — no collator surgery.

Keep hint_generator.py as layer (a). It already implements 5 templates with the right dispatch(error_kind, ctx) -> str | None signature. Add the four new templates from §3 (failing_test, lint_style, wasteful_action, verbose_communication) via register(...). Actual strings shipped in §3 — copy verbatim.
Add the new error_kind regexes to _ERROR_KIND_PATTERNS in trace_examples.py (§5 table). Single source of detection truth; preserve the ordered-precedence comment pattern (command_not_found before file_not_found). Route each error_kind → behavior_dim so the judge gets correct routing.
Build composer_replication/hints/ with protocol.py, layered.py, templates.py, judge.py, sibling.py (§6 sketches). templates.py imports the existing dispatch — do not reimplement.
Wire via the adapter: CollatorConfig(hint_generator=default_layered(...).as_collator_hook()). The claude_states_to_trace_examples adapter already populates error_meta (source_content_excerpt[:200]); extend it to also stash error_message (for class (b)) and, when available from the GRPO loop, sibling_rollouts (for class (f)).
Borrow OPSD stabilizers for the loss side: when distilling style/comms hints, apply per-token JSD clipping (OPSD --jsd_token_clip, default 0.05) so "stylistic tokens" don't dominate — the README states this is exactly why it exists. Pair with the collator's existing sdpo_loss_mask (post-hint tokens only).
Gate the judge: fire (c) only on (i) template miss, or (ii) a sampled fraction (~10–20%) of clean turns flagged for style/comms/effort, with hint caching keyed on (erroring_action, error_message, behavior_dim). Bounds cost at ~$0.001–0.002/trace.
Eval the generator independently of training (matches COMPOSER_RECIPE_MAPPING.md concern that "SDPO with hardcoded templates is the easy case"): measure (a) % of error sites that get a non-None hint per layer, (b) teacher-vs-student KL increase at hinted turns (a good hint should raise divergence — it's shifting probability toward the fix, per the blog's "lowering wrong-tool, raising valid-replacement"), and (c) for style/comms, a held-out judge agreeing the hint is corrective. A hint that doesn't move the teacher distribution is a no-op and should be pruned.

8. Citations

SDPO — Hübotter, Lübeck, Behric, Baumann, Bagatella, Marta, Hakimi, Shenfeld, Kleine Buening, Guestrin, Krause (ETH Zürich), Reinforcement Learning via Self-Distillation, arXiv:2601.20802v2 (v1 28 Jan 2026, v2 16 Feb 2026), CC-BY-4.0. Abstract + method/ablation HTML (html v2). The three-feedback-type ablation (sample solution / environment output / student's original attempt) and the "successful rollouts as implicit feedback" claim are the load-bearing sources for §1.2 and taxonomy classes (b), (d), (f).
OPSD — Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, arXiv:2601.18734v3, code github.com/siyan-zhao/OPSD (paper CC-BY-4.0; verify code license on repo). Privileged-info teacher (y⋆ = ground-truth/reference CoT), Eq. 8 loss, stop-grad teacher, and the --reason_first (introspection) + --jsd_token_clip (stylistic-token stabilizer) flags are the sources for §1.1 and the OPSD handles in §7.
Cursor blog — Introducing Composer 2.5 (2026): the "Reminder: Available tools are…" example, "hint changes the teacher probabilities / update student weights for that turn only," and the three behavior targets (tool use, coding style, model communication). Via docs/COMPOSER_RECIPE_MAPPING.md §1 and research/09-composer-blog-delta-2026.md §2.
Composer 2 technical report — arXiv:2603.24477 / Composer2.pdf (Rush et al.): flagged in the delta doc as the most likely place to resolve hint-generation directly; still unread — a dedicated extraction is the recommended follow-up if this design needs validation against Cursor's actual mechanism.
In-repo (audited): composer_replication/hint_generator.py (current layer (a)), composer_replication/trainer/data_collator.py (CollatorConfig.hint_generator hook, _build_hint_injected_trace, hint-AND-recovery gate at L308), composer_replication/ingestion/trace_examples.py (structural error detection, _ERROR_KIND_PATTERNS, default_classify_error).

Residual gap: Cursor still never states which hint source they use; this design brackets their unknown choice with the OPSD (privileged-answer) and SDPO (environment-feedback + sibling-bootstrap) endpoints and makes all of them composable. The one unread artifact that could collapse the bracket is Composer2.pdf (arXiv:2603.24477).