Hugging Face's logo

Codeseys
/
composer-replication-framework

Reinforcement Learning
Transformers
English
post-training
distillation
agentic-coding
composer-2.5
cursor
kimi-k2
grpo
dapo
diloco
openenv
trl
verl
research
methodology
Model card Files
xet
 Community
composer-replication-framework / examples /sdpo_with_real_traces /README.md
Codeseys's picture
Codeseys
Wave 19: production-grade SDPO via ComposerDataCollator + adapter + collator fixes
03bf323
preview code
|
Raw
History Blame Contribute Delete
5.69 kB

sdpo_with_real_traces — SDPO column wiring smoke on ClaudeCodeIngester-format traces (CPU, ~30s)

This is the third example in the SDPO progression. It demonstrates that the SDPO hint-distillation column survives end-to-end when its inputs come from ClaudeCodeIngester (vs hand-written prompts) on Qwen2.5-0.5B-Instruct, on CPU.

Honesty caveat (read this first):

The fixture this example uses (spikes/007-real-trace-ingestion/fixtures/synthetic_session.jsonl) is hand-authored — it matches the actual Claude Code v2.1.143 wire format exactly, but the conversational content is synthetic (we can't ship a real user session in the repo for PII reasons). So this example proves:

✅ The ClaudeCodeIngester → compose_loss plumbing works. ✅ The SDPO column fires (gradient flows, loss decreases). ❌ NOT that the framework has been validated on a real-content Claude Code session. To do that, point FIXTURE_PATH at one of your own ~/.claude/projects/.../session.jsonl files.

The docs/VISION_VALIDATION.md § 4.3 V5 gap is partially closed by this example: the ingestion pipeline is end-to-end-validated; the real-data run still requires a user with their own session JSONL files.

How it differs from the other examples

Example Trace source What it demonstrates
examples/gsm8k_grpo/ hand-written GSM8K problems Plain GRPO baseline (alpha_sdpo=0)
examples/gsm8k_grpo_with_sdpo/ hand-written GSM8K problems SDPO column wiring on synthetic prompts
examples/sdpo_with_real_traces/ synthetic-content Claude Code-format JSONL ingested via ClaudeCodeIngester SDPO column wiring through the production ingestion path

Run it 

pip install -e ".[train]"
python examples/sdpo_with_real_traces/run.py

Expected wall-clock: ~30s on CPU (after a one-time HF model download).

What success looks like 

[3/5] Ingesting trace + building SDPO batch (T=64) ...
  input_ids: shape=(2, 64) dtype=torch.int64
  ...
[4/5] Running 5 SGD steps with alpha_sdpo=0.50 on B=2 ...
  step 1/5: total=4.3004  lm_ce=3.9899  sdpo_jsd=0.6210  ...
  ...
  step 5/5: total=3.6877  lm_ce=3.3800  sdpo_jsd=0.6155  ...
[5/5] Verifying SDPO column wiring on real trace ...
  ✓ sdpo_jsd > 0 at every step (min=0.6155 max=0.6210)
  ✓ total != lm_ce at every step (min |diff|=0.3077 max=0.3105)
  ✓ |grad| > 0 and finite at every step (min=4.95e+05 max=6.35e+05)
✅ SDPO column wiring verified end-to-end on REAL agent trace.

Why the SDPO signal here is "wiring proof," not "production-quality"

The script appends the HINT as a system turn at the END of the messages list, so the chat template renders student vs teacher contexts that share most tokens but diverge near the right edge. The SDPO mask (rightmost 32 of 64 positions) therefore covers DIFFERENT token CONTENT in student vs teacher — student's last tokens are the user's tool-result, teacher's last tokens are the HINT.

That means the JSD signal we measure (sdpo_jsd ≈ 0.62) reflects the model's prediction divergence on different inputs, not a clean per-position teacher-vs-student divergence on the same content at an error site. This is acceptable for a wiring smoke test (proves the channel fires on real-trace input, gradients flow, the code path doesn't crash). It is NOT a production training signal.

A more rigorous demonstration would: (a) take the assistant turn from the trace as the "target action" the student is predicting, (b) align student/teacher contexts so the assistant turn lands at the same position in both, (c) place the HINT before that turn in the teacher only, (d) mask only the assistant-response positions. That's what composer_replication/trainer/data_collator.py:_build_sdpo_fields does in production. Out of scope for this wiring proof.

Trace fixture

The script uses spikes/007-real-trace-ingestion/fixtures/synthetic_session.jsonl — an 8-message Claude Code v2.1.143-format session pinned by Spike 007's test suite. The fixture matches the actual wire format Claude Code emits, with all the fields ClaudeCodeIngester reads (parentUuid, uuid, sessionId, type: user|assistant, message.content with text / tool_use / tool_result blocks, etc.).

To run against your own Claude Code sessions, point FIXTURE_PATH at ~/.claude/projects/.../session.jsonl. The ingester handles the real format identically — that's exactly the contract ClaudeCodeIngester is pinned to maintain.

Cross-references