research/deepread/03-sdpo-opsd.md

# Deep-Read: SDPO / OPSD — Critical Audit of Cluster 2

**Date**: 2026-06-09
**Sources fetched from primary HTML**:
- SDPO: arXiv:2601.20802v2 (Hübotter et al., ETH Zürich / MIT / Stanford)
  — note `reinforcement-learning-via-self-distillation-2` (148 K body)
- OPSD: arXiv:2601.18734v3 (Zhao et al., ICML 2026)
  — note `self-distilled-reasoner-on-policy-self-distillation-for-large-language-models-2` (52 K body)
- OPSD code repo: github.com/siyan-zhao/OPSD (README + key args)
- SDPO code repo: github.com/lasgroup/SDPO (listed in abstract; fetch returned empty body)

**Repo files audited**: `composer_replication/opsd.py`, `composer_replication/trainer/composer_trainer.py::_compute_sdpo_loss`, `composer_replication/hint_generator.py`, `research/07-sdpo-hint-generator.md`, `research/11-sdpo-alignment-indices.md`, `docs/adrs/ADR-007`, `ADR-008`, `ADR-009`, `ADR-011`.

---

## 1. What the primary sources actually say

### 1.1 SDPO (arXiv:2601.20802v2) — core method

**Exact loss (Eq. 1)**:

```
L_SDPO(θ) := Σ_t KL( π_θ(·|x, y_{<t}) ‖ stopgrad(π_θ(·|x, f, y_{<t})) )
```

KL direction is **forward KL — KL(student ‖ teacher)**, i.e. `KL(π_θ || q_θ)`. The
student is in the first argument. This is the "reverse KL from the teacher's
perspective" but forward from the student's perspective (student wants to match teacher).
The paper writes it `KL(π_θ ‖ stopgrad(q_θ))`.

**Stability improvements (§2.3)**:
1. Regularized teacher: EMA of student params OR interpolation between current teacher
   and initial teacher `q_{θ_ref}`. The paper calls these "trust-region" and "EMA"
   teachers (Table 4). Non-regularized teacher (`q_θ`, the live student) diverges.
2. Symmetric JSD: the paper adopts JSD as the distillation loss for stability —
   citing Agarwal et al. 2024 on-policy distillation. The pseudocode (Fig 14) calls
   this `divergence(logprobs_student, logprobs_teacher)` with no fixed default — the
   paper reports using JSD.

**Top-K approximation (Appendix A.3)**:
The paper approximates the full-vocab KL with top-K tokens of the **student** distribution:
```
L_SDPO ≈ Σ_t [ Σ_{ŷ_t ∈ topK(π_θ)} π_θ(ŷ_t|x,y_{<t}) · log(π_θ(ŷ_t) / q_θ(ŷ_t))
              + tail_term ]
```
The tail term aggregates the remaining probability mass. Default K=100.

**Teacher context construction (Table 2, verbatim)**:
```
User:  {prompt}
Correct solution: {successful_previous_rollout}    (skipped if unavailable)
The following is feedback from your unsuccessful earlier attempt:
{environment_output}                               (skipped if no env output or if solved)
Correctly solve the original question.
Assistant: {original_response}                     (the student's original attempt, for log-prob re-eval)
```

Critical nuance: the `original_response` is placed in the teacher context so the
model can re-evaluate log-probs of `y` under the teacher. **The student's original
attempt is always appended as the response the teacher evaluates** — this is how both
student and teacher evaluate log-probs of the same token sequence `y`.

Token alignment: **there is no shift / hint-insertion at an "error turn."** The teacher
sees `[prompt, feedback, original_response]` and both student and teacher evaluate
log-probs of the SAME `original_response` tokens. No additional tokens are inserted
into the response; the prefix is longer for the teacher (it has feedback), but the
response token sequence evaluated is identical.

**Three feedback types ablated (§4.6, Table 6)**:
1. **Sample solution** (`f = own solution`): a successful sibling rollout from the GRPO
   group; always student-generated (no expert model). Teacher accuracy: 42.4%.
2. **Environment output** (`f = output`): runtime errors, failing unit tests, etc.
   Teacher accuracy: 32.5%.
3. **Student's original attempt** (`f = y`): the repo calls out that including the
   original attempt in the feedback (not just in the response slot) **reduces teacher
   diversity** (biases teacher toward student; "Same output": 30% vs ~10–13%).
4. **Combined** (`f = output + own solution`): best trained student accuracy (48.3%).
   Excluding `f = y` (the original attempt as part of conditioning) is key.

**Failure modes reported**:
- Non-regularized teacher (`q_θ`) diverges / training collapses (Table 4: 36.1% vs
  50.6% for trust-region teacher).
- Performance depends on model in-context learning ability: SDPO underperforms GRPO on
  weaker models (Qwen3-0.6B); hybrid SDPO+GRPO (λ=0.9 GRPO + 0.1 SDPO) is more
  robust (§4.5).
- Uninformative or misleading environment feedback: SDPO cannot learn from it.
- SDPO adds small compute overhead (additional forward for log-prob re-computation of
  teacher context); minor for large models, non-negligible for small models.
- Including the student's own attempt in the teacher conditioning (not just as the
  response to re-evaluate) reduces diversity; the correct template excludes it from
  the conditioning prefix.

**SDPO operates over the full rollout, not at isolated "error turns"**. The loss
sums over all tokens `t` in the response `y`. There is no error-site detection
step in the SDPO paper.

### 1.2 OPSD (arXiv:2601.18734v3) — core method

**Exact loss (Eq. 6–8)**:
```
L_OPSD(θ) = E_{(x,y*)~S} [ E_{ŷ~p_S(·|x)} [ D(p_T ‖ p_S)(ŷ|x) ] ]
where
D(p_T ‖ p_S)(ŷ|x) := (1/|ŷ|) Σ_{n=1}^{|ŷ|} D( p_T(·|x, y*, ŷ_{<n}) ‖ p_S(·|x, ŷ_{<n}) )
```

**Divergence D** can be: forward KL, reverse KL, or JSD_β. The paper defines:
```
JSD_β(p_T ‖ p_S) = β·KL(p_T ‖ m) + (1-β)·KL(p_S ‖ m)
m = β·p_T + (1-β)·p_S
```

**Direction convention**: `D(p_T ‖ p_S)` — teacher in first arg, student in second. For the JSD:
- β=0 → KL(p_S ‖ m) (approaches pure KL(p_S ‖ p_T) as m→p_T; forward KL w.r.t. teacher)
- β=1 → KL(p_T ‖ m) (approaches pure KL(p_T ‖ p_S); reverse KL w.r.t. teacher / forward KL w.r.t. student)

The GKD paper (arXiv:2306.13649) that OPSD cites defines JSD_β with the **same
convention**: `JSD_β(p ‖ q) = β·KL(p||M) + (1-β)·KL(q||M)`.

**Per-token pointwise clipping**: OPSD introduces this explicitly:
```
D_clip^(f)(p_T ‖ p_S) = (1/|ŷ|) Σ_n Σ_v min(l_{n,v}^(f), τ)
where l_{n,v}^(f) = p_T(v|·) · f( p_S(v|·) / p_T(v|·) )
```
This clips per vocab-entry contribution. Default τ=0.05 (from README: `--jsd_token_clip 0.05`).
Non-thinking mode results in README use 1e-7 (Qwen3-8B) and 1e-6 (Qwen3-4B, 1.7B).

**Teacher context**: `p_T(·|x, y*, ŷ_{<n})` — teacher sees the ground-truth answer `y*`
(a reference CoT / verified reasoning trace from the dataset) prepended to the problem,
then evaluates the student's prefix `ŷ_{<n}`. Same token sequence for both distributions
evaluated at each step `n`.

**`--reason_first` flag (from GitHub README)**: Prepend an explicit rationalization to the
teacher context before distillation. This is OPSD's self-introspection lever: the teacher
is first asked to rationalize why `y*` is correct, then that rationalization is folded into
the conditioning. Not the main results configuration; requires `--use_peft`.

**Results**: On Qwen3-1.7B (AIME24/25/HMMT25), OPSD +OPSD vs. base: 37.1% → 43.4%
(Avg@12). Outperforms GRPO (37.7%) and SFT (35.8%). Token-efficient: generation capped
at 1024 tokens vs. GRPO's 16K.

---

## 2. Audit: `composer_replication/opsd.py` — "byte-for-byte OPSD parity" claim

### 2.1 JSD formula — CORRECT, with a subtle direction note

The code implements:
```python
JSD = β·KL(teacher||M) + (1-β)·KL(student||M)
M = logsumexp([ log p_student + log(1-β), log p_teacher + log(β) ])
```

The OPSD paper (Eq. 7) defines:
```
JSD_β(p_T ‖ p_S) = β·KL(p_T‖m) + (1-β)·KL(p_S‖m)
```
where `m = β·p_T + (1-β)·p_S`.

The code `kl_teacher = F.kl_div(mixture_log_probs, teacher_log_probs, ...)` uses
PyTorch semantics where `F.kl_div(input=log_q, target=log_p, log_target=True)`
computes `KL(p||q) = Σ p(x)·(log p(x) - log q(x))`. So `kl_teacher` computes
`KL(teacher||mixture)` and `kl_student` computes `KL(student||mixture)`.

The final JSD: `β·kl_teacher + (1-β)·kl_student` = `β·KL(teacher||M) + (1-β)·KL(student||M)`.

This matches the OPSD paper's `JSD_β(p_T ‖ p_S)` exactly. **CORRECT.**

### 2.2 β convention docstring — INVERTED vs. both papers

The `opsd.py` docstring says:
```
β = 0  → KL(teacher || student)  (reverse KL — mode-covering for student)
β = 1  → KL(student || teacher)  (forward KL — mode-seeking for student)
```

From the OPSD GitHub README:
> `--beta`: Interpolation weight for the JSD mixture distribution.
> **Beta=0 means forward KL and 1 means reverse KL.**

The repo docstring has the β=0 and β=1 labels **swapped** relative to the OPSD upstream.
When β=0: `JSD_0 = 0·KL(teacher||M) + 1·KL(student||M)`. In the limit (degenerate β=0),
M approaches p_teacher, so this approaches `KL(student||teacher)` — which is **forward KL**
(student → teacher), **mode-seeking for student**. The README says "Beta=0 means forward KL"
which matches this analysis.

The code *implementation* is correct (the formula computes the right mixture). The *docstring*
labels β=0 as "reverse KL" and β=1 as "forward KL", which contradicts both the upstream README
and the mathematical analysis. This is a documentation error, not a numerical error.

**VERDICT**: Implementation is numerically correct. Docstring direction labels are inverted.

### 2.3 `reduction="batchmean"` behavior — MINOR DIVERGENCE from upstream

The repo `opsd.py` comment says:
> "batchmean" matches upstream OPSD: divides by `mask.sum()` when labels are given,
> else by the leading dim of jsd (= batch size).

The OPSD paper (Algorithm 1) normalizes by `|ŷ|` (sequence length, token-mean):
```
ℓ(x,y*) = D(p_T‖p_S)(ŷ|x) = (1/|ŷ|) Σ_n D(...)
L_OPSD(θ) = (1/|B|) Σ_{(x,y*)∈B} ℓ(x,y*)
```

The repo divides by `mask.sum()` (number of valid/masked tokens in the batch), which is
equivalent to OPSD's normalization only when every example has the same number of
error-turn tokens. When batch sizes vary (real training), this differs from the paper's
per-sequence average followed by batch average. In practice this difference is negligible
for stability, but it is technically not byte-for-byte OPSD parity on the reduction.

**VERDICT**: The `reduction="batchmean"` logic is borrowed from the OPSD upstream code
(which uses the same `mask.sum()` convention). The docstring's "matches upstream" claim
is accurate for the code, but the code diverges from the paper's stated per-sequence
normalization. Not a material issue.

### 2.4 `token_clip` parameter — CORRECT semantics, but per-token vs. per-(token,vocab) distinction

The repo implements `token_clip` as a **per-position** JSD clip:
```python
jsd = jsd.clamp(max=token_clip)  # jsd shape is (B, T, V) or (n_valid, V)
```

The OPSD paper's pointwise clipping (Section 3.3) clips **per-(position, vocab-entry)**:
`min(l_{n,v}^(f), τ)` for each vocab entry v at each position n.

The upstream OPSD code (`--jsd_token_clip`) appears to apply the same per-(position,vocab)
clip. The repo's clamp on the jsd tensor before reduction would clip the full-vocab
contribution per position (since jsd has shape (B,T,V) before masking) — this is
equivalent to per-(position,vocab) clipping, which is correct.

**VERDICT**: Implementation appears correct. The parameter name (`token_clip`) is slightly
misleading (it clips per-token-vocab-entry, not just per-token), but the semantics match.

---

## 3. Critical structural mismatch: Composer/repo framing vs. SDPO mechanics

### 3.1 ERROR-TURN MASKING — NOT IN SDPO

The repo implements SDPO as an error-turn-masked loss:
- `_compute_sdpo_loss` applies JSD only at `error-turn tokens` (via `sdpo_loss_mask`).
- The data collator detects error sites in a trace and constructs a teacher context
  with a hint inserted at the error turn (`ctx_teacher = ctx_student + hint`).
- The hint shifts teacher response tokens right, requiring explicit alignment indices
  (ADR-011).

**The SDPO paper has no error-turn masking.** SDPO applies the KL loss to ALL tokens `t`
in the rollout response:
> "L_SDPO(θ) := Σ_t KL(π_θ(·|x, y_{<t}) ‖ stopgrad(π_θ(·|x, f, y_{<t})))"

The SDPO teacher context includes the full feedback; both student and teacher evaluate
log-probs of the **same response tokens** `y`. There is no "hint inserted into the
response" — the feedback is in the conditioning prefix, not intercalated into the
response sequence. Therefore the teacher response tokens are **not shifted** and token
alignment is trivially preserved: both contexts evaluate the same sequence `y`.

**The repo's architecture (hint at error turn → response token shift → alignment indices)**
is an interpretation of Composer 2.5's "hint" mechanism, not a feature of SDPO. SDPO's
feedback is in the prompt/conditioning context; it does not intercalate text into the
middle of a response.

**VERDICT**: The repo's error-turn-masking design is a reasonable extension of the
Composer blog's described mechanism ("insert hint at error turn") but is **NOT**
SDPO as described in the paper. The Composer blog's mechanism is itself not fully
described and may or may not match SDPO mechanics.

### 3.2 TEACHER CONTEXT — CRITICAL DIFFERENCE

**SDPO teacher context** (Table 2):
```
[prompt, feedback_f, original_response_y]
```
The teacher evaluates log-probs of `original_response_y` given `[prompt, feedback_f]`.
Teacher prefix = `[prompt, feedback_f]`. Response = `y` (same as student). No hint is
inserted *into* `y`.

**Repo teacher context**:
```
ctx_teacher = ctx_student + hint_at_error_turn
```
The hint is *intercalated* into the response sequence at an error turn. Teacher prefix
= student prefix. Response = `y_before_error + hint + y_after_error`. Teacher response
tokens are LONGER than student response tokens and SHIFTED.

This is architecturally different from SDPO. The alignment problem (ADR-008, ADR-011)
arises precisely because the repo's teacher context design inserts hint text into the
response, which SDPO does not do.

**VERDICT**: The repo's teacher context construction is a novel design inspired by the
Composer blog. It is not what SDPO does. The ADR-008 "trust-gap" and the entire
ADR-011 alignment index complexity are artifacts of this departure from SDPO, not
corrections to SDPO.

### 3.3 OPSD vs. SDPO as the loss source

The repo header in `opsd.py` says the loss is:
> "lifted from siyan-zhao/OPSD::OPSDTrainer.generalized_jsd_loss (MIT)"

And:
> "SDPO paper: Hübotter et al. … formalizes the same loss as Composer 2.5's
> 'Targeted RL with Textual Feedback.'"

These are TWO DIFFERENT methods with related but not identical losses:

- **OPSD loss** (Eq. 7–8): `JSD_β(p_T ‖ p_S)` with teacher having `y*` (ground truth).
  Normalization: per-sequence average then batch average. Pointwise vocab clipping.
  Training runs ~100 steps. Fixed teacher (initial checkpoint, not live).

- **SDPO loss** (Eq. 1): `KL(π_θ(·|x,y_{<t}) ‖ stopgrad(π_θ(·|x,f,y_{<t})))` where
  KL is applied per-position over the full response. The paper adopts JSD as a stability
  improvement (§2.3) but the base formulation is reverse KL (student ‖ teacher).
  Teacher is regularized via EMA or trust-region. No per-vocab clipping in the paper
  (top-K approximation instead).

The repo correctly implements the OPSD JSD formula (which SDPO also uses for stability).
The claim "verified port of siyan-zhao/OPSD::OPSDTrainer.generalized_jsd_loss" is
accurate for the loss kernel. The claim "Composer 2.5's 'Targeted RL with Textual Feedback'"
is an assertion that Composer uses the same loss — this is not confirmed anywhere in the
Cursor blog or Composer 2 tech report.

---

## 4. Audit: `_compute_sdpo_loss` in `composer_trainer.py`

### 4.1 Gradient flow — CORRECT

```python
student_logits = model(input_ids=inputs["input_ids"]).logits
with torch.no_grad():
    teacher_logits = model(input_ids=inputs["ctx_teacher_input_ids"]).logits
```

Teacher is `no_grad` — matches SDPO's `stopgrad(π_θ(·|x,f,y_{<t}))`. Student has
gradient. Correct.

### 4.2 Alignment index machinery — NECESSARY GIVEN THE DESIGN, BUT NOT FROM SDPO

The `student_response_idx` / `teacher_response_idx` machinery (ADR-011) is needed
because the hint is inserted into the teacher response sequence. This complexity does
not exist in SDPO or OPSD because those methods never insert text into the response.
The repo's `strict_sdpo_alignment` guard is a correct defense against the problem it
has created for itself.

### 4.3 Batch-level masking — CORRECT for the repo's error-turn interpretation

The loss is masked to error-turn tokens only (`aligned_labels` with -100 elsewhere).
This means the SDPO channel only trains on error recovery tokens, not the full rollout.
SDPO trains on the full rollout. For Composer's intent (correcting error turns), the
masking is reasonable, but it produces a loss that is more like a targeted distillation
at error sites than SDPO's full-rollout advantage assignment.

---

## 5. Audit: `research/07-sdpo-hint-generator.md` — Accuracy check

### 5.1 Three feedback types from SDPO paper — CORRECTLY REPORTED

research/07 correctly identifies the three types:
1. Sample solution (successful sibling rollout)
2. Environment output (runtime errors)
3. Student's original attempt

The paper (Table 6 results, which research/07 did NOT have access to) shows:
- Best configuration: `f = output + own solution` (48.3% accuracy)
- Including `f = y` (original attempt as conditioning, not as response) **hurts diversity**
  and slightly reduces final accuracy (44.5% vs 48.3%)

Research/07 correctly notes the sibling rollout is "always generated by the student, not
an expert model" — confirmed in the paper: "We emphasize that these sample solutions are
always generated by the student, as in GRPO, and do not require an expert model."

### 5.2 "Successful sibling rollout as implicit feedback" claim — CORRECTLY REPORTED

The abstract: "SDPO also outperforms baselines in standard RLVR environments that only
return scalar feedback by using successful rollouts as implicit feedback for failed attempts."

Research/07 cites this correctly and uses it as the basis for the `SiblingBootstrapGenerator`.

### 5.3 OPSD `--reason_first` flag — CORRECTLY DESCRIBED

The OPSD README confirms: `--reason_first False: Prepend an explicit rationalization to
the teacher context before distillation.` Research/07 correctly calls this "OPSD's own
knob for same-model introspection."

### 5.4 `--jsd_token_clip default 0.05` — CORRECTLY CITED

Confirmed from OPSD README: `--jsd_token_clip 0.05` is the default.

---

## 6. Audit: `SiblingBootstrapGenerator` — Is it supported by the papers?

The repo's `hint_generator.py` sketch (lines 319–331) and `research/07` §6.3:
```python
class SiblingBootstrapGenerator:
    def generate(self, ctx):
        sibs = ctx.get("sibling_rollouts") or []
        winners = [s for s in sibs if s.get("reward", 0.0) > 0.0]
        if not winners:
            return None
        best = max(winners, key=lambda s: s["reward"])
        snippet = (best.get("solution_excerpt") or "")[:200]
        return ("Reminder: a working approach for this task looks like:\n"
                f"{snippet}\nAdapt this to the current step.")
```

**What the SDPO paper actually does** (Table 2 template):
```
Correct solution: {successful_previous_rollout}
```
The successful rollout is passed as the full solution (or relevant excerpt) in the
teacher context. The teacher then evaluates log-probs of the student's original
response given this context.

**Key difference**: In SDPO, the sibling solution goes into the teacher's conditioning
prefix. The teacher does not generate a new hint; it just re-evaluates the student's
response log-probs with the solution visible. In the repo, the sibling solution is
used to *generate a hint string* that gets inserted into the response sequence.

This is an extrapolation beyond what the SDPO paper supports. SDPO's "successful rollout
as implicit feedback" mechanism does NOT:
1. Generate a "Reminder: a working approach..." hint string.
2. Insert text into the student's response sequence.
3. Require error-turn detection.

The SDPO sibling mechanism IS:
1. Condition the teacher on the full successful solution.
2. Re-evaluate ALL student response token log-probs under that teacher.
3. Apply the KL loss across the entire response.

**VERDICT**: The `SiblingBootstrapGenerator` as sketched is an extrapolation from SDPO's
mechanism, not a faithful implementation of it. The paper supports using a sibling rollout
as teacher conditioning context; it does not support generating a textual hint from it
to splice into the response. The Composer blog's "hint" framing is the source of this
architectural decision; SDPO is cited as inspiration but is not the mechanism.

Research/07 acknowledges this at several points ("A working approach looks like: …" in
the class comment vs the actual SDPO template) but does not flag it as a divergence — it
presents the sibling-bootstrap hint approach as if it naturally follows from SDPO.

---

## 7. Audit: `research/11-sdpo-alignment-indices.md`

### 7.1 Problem correctly identified

ADR-011 correctly identifies that inserting a hint into the teacher context shifts teacher
response tokens right. The alignment indices machinery (`_mask_to_padded_indices`,
`student_response_idx`, `teacher_response_idx`, sentinel handling) is a sound engineering
solution to the problem the repo's design creates.

### 7.2 Root cause attribution — MISLEADING

ADR-011 and the trainer comments frame the alignment problem as an SDPO issue that the
papers fail to address ("the exact trust-gap flagged in ADR-008"). This is not accurate.
The alignment problem does not exist in SDPO or OPSD because those methods never insert
text into the response sequence. The alignment problem is entirely self-created by the
repo's decision to implement the Composer blog's "hint at error turn" as a text insertion
into the teacher's response sequence.

---

## 8. Audit: ADR-007, ADR-008, ADR-009 — Key claims

### ADR-007 — JSD as "the kernel of SDPO arXiv:2601.20802"

The ADR says `generalized_jsd_loss` is "verified port of siyan-zhao/OPSD, the kernel of
SDPO arXiv:2601.20802." This telescopes two papers. The JSD is the kernel of OPSD
(the Zhao et al. paper, 2601.18734). SDPO (Hübotter et al., 2601.20802) uses JSD as a
**stability improvement** over the base KL loss; the primary SDPO loss is the KL. Both
papers use the same JSD formula (citing GKD paper 2306.13649). The conflation is not
consequential for the loss code but creates confusion in documentation.

### ADR-008 — "SDPO needs full vocabulary logits"

Confirmed. SDPO Appendix A.3 discusses top-K approximation of the KL because "naively
computing the KL divergence between student and teacher requires holding full logits of
both models in memory." The repo's claim about needing full logits is correct; PRIME-RL's
log-probs-only interface is correctly identified as incompatible with the SDPO channel.

### ADR-008 — Dr. GRPO as the Composer algorithm

This is sourced from research/10 (Composer 2 tech report mining). Not audited here (out
of scope for this cluster).

### ADR-009 — "How Cursor generates that hint is unstated"

Confirmed true. The Composer 2 tech report (arXiv:2603.24477) is cited as unread in
research/07 §8 and ADR-009. ADR-009 correctly acknowledges the open question.

---

## 9. Summary of findings

| Claim | Source | Verdict |
|---|---|---|
| JSD formula in opsd.py is numerically correct | OPSD Eq. 7 | CORRECT |
| β=0 = "reverse KL" in docstring | OPSD README: "β=0 = forward KL" | INVERTED label |
| "byte-for-byte OPSD parity" | OPSD code | Mostly correct; β direction label wrong; reduction differs from paper's per-sequence normalization; otherwise matches upstream code |
| Error-turn masking is from SDPO | SDPO paper | FALSE — SDPO applies loss to full rollout, no error-turn detection |
| Teacher context = ctx_student + hint_at_error_turn | SDPO paper | FALSE — SDPO teacher = [prompt, feedback, student_response]; feedback is in prefix, not intercalated |
| SiblingBootstrapGenerator follows from SDPO "successful rollout as implicit feedback" | SDPO §4.6 | EXTRAPOLATION — SDPO conditions teacher on full solution; repo generates a hint string and inserts it into response sequence |
| Alignment indices machinery (ADR-011) addresses SDPO misalignment | SDPO paper | MISLEADING — problem is self-created by hint-insertion design; does not exist in SDPO |
| SDPO needs full vocabulary logits (ADR-008) | SDPO Appendix A.3 | CORRECT |
| Three feedback types in research/07 | SDPO §4.6 | CORRECTLY REPORTED |
| --jsd_token_clip default 0.05 | OPSD README | CORRECT |
| --reason_first flag | OPSD README | CORRECTLY DESCRIBED |
| "Successful rollouts as implicit feedback" claim | SDPO abstract | CORRECTLY CITED |
| Teacher is stop-grad, student has gradient | SDPO Eq. 1 | CORRECT in opsd.py and composer_trainer.py |

---

## 10. Recommendations

1. **Fix the β docstring** in `opsd.py` to match the OPSD upstream convention:
   β=0 → forward KL (KL(student‖teacher)), β=1 → reverse KL (KL(teacher‖student)).

2. **Clarify the architectural departure from SDPO** in `composer_trainer.py` docstring
   and `research/07`: the repo implements a Composer-blog-inspired error-turn hint
   injection, which is an extension beyond SDPO. SDPO uses the feedback in the prompt
   prefix and evaluates the full response; the repo intercalates text into the response.

3. **Reconsider framing of `SiblingBootstrapGenerator`**: it is an original design choice,
   not an SDPO mechanism. The SDPO "sibling as implicit feedback" mechanism would look like:
   build a teacher context `[prompt, successful_sibling_rollout, original_response]` and
   apply KL over the whole original response — without generating a hint string or
   error-turn detection. This would be simpler and more faithful to SDPO.

4. **Teacher regularization is not implemented**: the SDPO paper shows a non-regularized
   teacher diverges (Table 4: 36.1% vs. 50.6%). The repo's teacher is the live model
   weights at each step with no EMA or trust-region regularization. For production SDPO
   runs this is a gap. (The `sdpo_jsd_beta` default of 0.5 uses symmetric JSD which is
   one of SDPO's stability improvements, but the teacher regularization is absent.)

5. **SDPO's original attempt placement**: the paper includes the student's original
   response as the sequence being log-prob-evaluated (i.e., the "response" slot in the
   teacher context). The repo's collator instead masks specific error-turn tokens within
   a modified response. These are architecturally different. The paper-accurate approach
   would re-evaluate log-probs of the entire original response under the hint-conditioned
   teacher, not just the tokens after the error.

6. **Failure mode from SDPO paper**: the strongest limitation is model capability
   dependence — SDPO underperforms GRPO on weak models (Qwen3-0.6B); SDPO+GRPO with
   λ=0.9 is recommended for weaker base models. This is not documented in the repo's
   SDPO usage guidance.

---

## 11. What the papers do NOT say (repo-claimed but unconfirmed in sources)

- That Composer 2.5's "Targeted RL with Textual Feedback" is SDPO specifically (the
  Cursor blog does not cite SDPO; it describes a mechanism consistent with SDPO but
  the connection is an inference, not a citation).
- That error-turn masking is part of SDPO.
- That the repo's hint-at-error-turn teacher context is the SDPO mechanism.
- That the alignment index problem (ADR-011) is an issue in SDPO.
- How Cursor generates the hint (confirmed absent in all Cursor artifacts).