# sdpo_with_real_traces_production — Production-grade SDPO via `ComposerDataCollator` (CPU, ~2min)

This is the **fourth** example in the SDPO progression — the
production-grade sibling to `examples/sdpo_with_real_traces/`:

| # | Example | Path | What it demonstrates |
|---|---|---|---|
| 1 | `qwen_05b_quickstart/` | toy LM, no SDPO | Package import + import smoke |
| 2 | `gsm8k_grpo/` | hand-written GSM8K, no SDPO | Plain GRPO baseline |
| 3 | `gsm8k_grpo_with_sdpo/` | hand-written GSM8K | SDPO column wiring on synthetic prompts |
| 4 | `sdpo_with_real_traces/` | `ClaudeCodeIngester` | **Wiring** smoke (misaligned student/teacher) |
| **5** | **`sdpo_with_real_traces_production/`** ⬅ | **Full ingester→adapter→collator→loss** | **Production-grade ALIGNED SDPO** |

## What this example demonstrates

- ✅ Full production data path: `ClaudeCodeIngester → claude_states_to_trace_examples → ComposerDataCollator → compose_loss`
- ✅ Tool-error site detection from real `is_error: true` JSONL records
- ✅ The collator's `_build_hint_injected_trace` injecting hints AT the error site
- ✅ Position-level alignment of the recovery-turn tokens (post-Wave-19 fix: ~67% of in-loss positions are bit-aligned student vs teacher; the remaining ~33% reflect a segment-vs-chat-template-marker drift bug tracked for Wave 20)
- ✅ Non-trivial, content-meaningful SDPO JSD signal (~0.25 — vs the degenerate ~0.68 ≈ ln(2) we'd get with broken alignment, which Wave 19 round-1 review caught and Wave 19 round-2 fixed)
- ✅ Gradient flow through Qwen2.5-0.5B-Instruct
- ✅ The collator's shape-reconciliation (Wave 19 fix: builds an aligned student tensor with placeholder system messages so `student_logits.shape == teacher_logits.shape`)

> **Honesty caveat about alignment** (Wave 19 cross-family review caught
> this and it's tracked for Wave 20):
>
> The collator's existing `_build_segment_mask` doesn't account for the
> chat-template markers (`<|im_start|>system\n`, `<|im_end|>\n`) that
> `apply_chat_template` adds AROUND each message segment. So the
> `sdpo_loss_mask` is approximately — not exactly — aligned with the
> recovery-turn tokens. On the with-error fixture, ~84% of the in-loss
> positions hold identical student/teacher tokens; the other ~16% land
> on the hint-vs-placeholder content boundary because the segment-tokenizer
> double-counts template markers.
>
> What this means in practice:
>   - The SDPO signal here is meaningful (most positions ARE aligned)
>     but not 100% pure.
>   - For production training of small models, the residual drift may
>     manifest as a slight noise floor — the model receives an SDPO
>     gradient that mostly trains the right thing, with a small
>     fraction training the placeholder-vs-hint distinction (which is
>     unhelpful but bounded).
>   - The fix requires re-architecting `_build_segment_mask` to align
>     with `apply_chat_template`'s actual token output. Wave 20.

## Run it

```bash
pip install -e ".[train]"
python examples/sdpo_with_real_traces_production/run.py
```

Expected wall-clock: ~2min on CPU (5 steps × ~25s/step on a 0.5B model).

## What success looks like

```
[3/5] Building batch via production pipeline ...
  ClaudeCodeIngester → claude_states_to_trace_examples → ComposerDataCollator
  ingested 3 states; adapter detected 1 error site(s)
    input_ids: shape=(3, 261) dtype=torch.int64
    ...
    ctx_teacher_input_ids: shape=(3, 261) dtype=torch.int64
    sdpo_loss_mask: shape=(3, 261) dtype=torch.int64
  sdpo_loss_mask: 70 positions in loss (per-row: [0, 0, 70])
  shape reconciliation: student (3, 261) vs teacher (3, 261) — ALIGNED

[4/5] Running 5 SGD steps with alpha_sdpo=0.50 ...
  step 1/5: total=2.1137  lm_ce=1.9898  sdpo_jsd=0.2478  ...  |grad|=6.04e+05
  ...
  step 5/5: total=1.8953  lm_ce=1.7682  sdpo_jsd=0.2543  ...  |grad|=5.06e+05

[5/5] Verifying production-grade SDPO behavior ...
  ✓ sdpo_jsd > 1e-7 at every step (min=0.2478 max=0.2543)
  ✓ total != lm_ce at every step (min |diff|=0.1239)
  ✓ |grad| finite at every step
  alignment audit: 47 / 70 in-loss positions match student==teacher (67.1%)
  WARNING: 23 positions (32.9%) of the SDPO mask cover non-aligned tokens
           (segment-vs-chat-template drift; tracked for Wave 20).

✅ Production-grade SDPO verified end-to-end via ComposerDataCollator.
```

The key difference from `examples/sdpo_with_real_traces/`:

| Property | Wiring example | Production example |
|---|---|---|
| Hint placement | Appended to messages list | Injected BY the collator at the error site |
| Student vs teacher | Different right-edge tokens | Same tokens at masked positions |
| Loss mask | Hardcoded last 32 positions | Derived from error-turn boundaries |
| SDPO signal | Reflects different inputs | Reflects teacher-with-hint vs student-without-hint on SAME content |
| Use case | Wiring proof | **What you should actually copy for production training** |

## How the production pipeline works

### 1. Ingest

```python
from composer_replication.ingestion import ClaudeCodeIngester
ingester = ClaudeCodeIngester(skip_sidechain=True, strip_thinking=True)
states = list(ingester.ingest(jsonl_path))
```

The ingester reads Claude Code v2.1.x session JSONL and emits
`TraceState` dicts. It preserves `is_error: true` from `tool_result`
records by tagging the serialized content with `[TOOL_RESULT (ERROR)]`.

### 2. Adapt

```python
from composer_replication.ingestion import claude_states_to_trace_examples
examples = claude_states_to_trace_examples(states)
```

The Wave 19 adapter walks each state's `messages`, detects error sites
by string-matching the `[TOOL_RESULT (ERROR)]` tag in user-role
messages, and marks the *immediately following* assistant turn (the
recovery turn) with `tool_error="<classified_kind>"` — the field that
`ComposerDataCollator._is_error_turn` checks.

The default error classifier categorizes the tool-result content into
`file_not_found`, `permission_denied`, `command_not_found`,
`syntax_error`, `connection_error`, or generic `tool_error`. You can
pass your own classifier via the `error_kind_fn` parameter.

### 3. Collate

```python
from composer_replication.trainer.data_collator import (
    ComposerDataCollator, CollatorConfig,
)

config = CollatorConfig(
    hint_generator=hint_for_error,          # error_kind, error_meta -> hint_text
    enable_replay_dpo=False,
    pad_token_id=tokenizer.pad_token_id,
)
collator = ComposerDataCollator(tokenizer=tokenizer, config=config)
batch = collator(examples)
```

The collator's `_build_hint_injected_trace` walks each example's turns;
when it hits an error turn, it calls `hint_generator(error_kind, error_meta)`
and injects the returned hint text as a system message BEFORE the
assistant recovery turn. The `sdpo_loss_mask` is set to 1 only at the
post-hint assistant tokens — the positions where student and teacher
are predicting the same content.

The collator's `__call__` reconciles shapes: hint injection makes
`ctx_teacher_input_ids` LONGER than `input_ids`, but `compose_loss`
gates SDPO on `student_logits.shape == teacher_logits.shape`. The
collator right-pads student fields with `pad_token_id` and zeros to
match teacher length so the gate passes. (This was a Wave 19 collator
fix; pre-Wave-19 callers got SDPO=0 because the gate failed.)

### 4. Loss

```python
from composer_replication import compose_loss
out = compose_loss(model, batch, alpha_sdpo=0.5, beta_replay=0.0)
out.total.backward()
```

`compose_loss` runs the model on `input_ids` (student forward) and
`ctx_teacher_input_ids` (teacher forward, no_grad), checks shapes
match, and computes the JSD over positions where `sdpo_loss_mask == 1`.

## Hint generator

The hint generator in `run.py` is deterministic and error-kind-aware:

```python
def hint_for_error(error_kind: str, error_meta: dict) -> str | None:
    library = {
        "file_not_found":     "Hint: ...verify the path with `ls` first...",
        "permission_denied":  "Hint: ...check ownership with `ls -l`...",
        "command_not_found":  "Hint: ...check `which` and `$PATH`...",
        "tool_error":         "Hint: ...read the error and consider retry vs pivot...",
    }
    return library.get(error_kind, library["tool_error"])
```

A real production hint generator would pull from a curated hint
library or call an LLM-as-teacher; this one is static for determinism.
Returning `None` for an error kind tells the collator to skip the
SDPO injection for that turn.

## Trace fixture

The script uses
`spikes/007-real-trace-ingestion/fixtures/synthetic_session_with_error.jsonl`
— a 6-message Claude Code v2.1.143-format session where a `Read` tool
call hits a non-existent file, the assistant recovers by listing
candidate paths, and the second `Bash` call succeeds. Wave 19
introduced this fixture specifically to exercise the SDPO error-site
path; the Wave 18 example used the original Spike 007 fixture which
had no errors.

To run on your own real Claude Code sessions, point `FIXTURE_PATH` at
`~/.claude/projects/.../session.jsonl`. The full pipeline is content-
agnostic; it works on any Claude Code v2.1.x session.

## Cross-references

- [`composer_replication.ingestion.trace_examples.claude_states_to_trace_examples`](../../composer_replication/ingestion/trace_examples.py) — the adapter
- [`composer_replication.ingestion.tests.test_trace_examples_adapter`](../../composer_replication/ingestion/tests/test_trace_examples_adapter.py) — adapter contract tests
- [`composer_replication.trainer.data_collator.ComposerDataCollator`](../../composer_replication/trainer/data_collator.py) — production-grade collator
- [`examples/sdpo_with_real_traces/`](../sdpo_with_real_traces/) — the wiring-only sibling for comparison
- [`spikes/007-real-trace-ingestion/`](../../spikes/007-real-trace-ingestion/) — the spike pinning the ingester contract