Commit History
-
Wave 4: data collator + loss composition smoke (38/38 tests pass)
baladithyabSpike 005's biggest engineering gap was: composer_trainer.py described the inputs it expected (ctx_teacher_input_ids, sdpo_loss_mask, dpo_chosen_input_ids, etc.) but nothing constructed them from raw traces. This wave fills that gap and adds an end-to-end gradient-step smoke test on a real model. Added: 1. trl_path/data_collator.py - ComposerDataCollator turns raw TraceExample into the exact dict shape ComposerReplicationTrainer._compute_loss expects. Channel 1: input_ids, attention_mask, response_mask, rewards. Channel 2: ctx_teacher_input_ids, sdpo_loss_mask (post-hint = 1, else -100). Channel 3: dpo_chosen_input_ids, dpo_rejected_input_ids, response_masks. Hint injection, error-site detection, multi-turn DPO tokenization, padding. 2. tests/test_data_collator.py (15 tests, all pass): verifies SDPO is skipped when no error sites or no hint generator, post-hint mask correctly marks 1 vs ignore_index, DPO response masks zero prompt tokens, padding handles mixe…157cdba -
Wave 3: integration architecture + spike-005 trainer skeleton (16 tests pass)
baladithyabUser asked: can RLVR + Composer-SDPO + N-teacher-replay all integrate complementarily with the PyTorch agentic-RL stack (Monarch / TorchForge / OpenEnv / VeRL / TRL)? Answer (verified, not extrapolated): YES, with concrete extension points: 1. Verified ground-truth integration surfaces via DeepWiki audits: - TRL: subclass GRPOTrainer, override _compute_loss(model, inputs); plug OpenEnv via environment_factory= kwarg. - VeRL: register a new advantage estimator via @register_adv_est(name); attach extra fields to DataProto.batch / non_tensor_batch (precedent exists — distillation already uses 'teacher_log_probs' the same way). - OPSD: generalized_jsd_loss is a self-contained static method, MIT, directly liftable. FlashAttention-2 compatible, standard PyTorch. 2. NEW docs/INTEGRATION_ARCHITECTURE.md (30KB): - Per-framework integration matrix (TRL / VeRL / TorchForge / Monarch / OpenEnv) × per-channel (RLVR / SDPO / N-teacher-replay). - Sequence di…fd77f74