Spaces:
Sleeping
Sleeping
Commit History
-
fix(v4): increase T4 SFT to 50 eps, add action trace to eval output
N_SFT_EPISODES 20→50 on T4: 20 eps gave loss≈2.0 which is too high for GRPO to improve from. 50 eps targets loss<1.0 as SFT floor. Eval now prints the action sequence per episode in brackets so it's immediately visible whether the model is: (a) not generating valid JSON (marked with !), (b) skipping assign_ticket, (c) posting to wrong channel. This makes it easy to diagnose score=0.000 episodes without rerunning. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>16705aa -
fix(v4): reduce MAX_COMP_LEN 384→192, fix eval hanging with for_inference
MAX_COMP_LEN=384 caused clipped_ratio=1.000 every step → all completion gradients masked → Training Loss=0.000. JSON actions are 80-120 tokens; 192 gives headroom without filling TRL's buffer. Eval cell hung because Unsloth's training hooks were still active after trainer.train(). FastLanguageModel.for_inference(model) is required before model.generate() post-training. Also reduced eval max_new_tokens 256→128. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2f02c01 -
feat(training): add clean v4 notebook with all reward fixes
Single-pass, no dead code. Key improvements over v3/pink notebooks: - gen_slot correctly offsets env seed per GRPO generation - runbook-compliance reward (label/priority/team/channel vs org config) - compute_rollout_reward imported from training/rewards.py - temperature 1.1 + top_k=50 for diverse rollouts - PatchFastRL called once before trl imports - No duplicate git-pull or dead rollout cells Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>4b0fc8a -
fix(notebook): rewrite run_grpo_episode with runbook-compliance reward
- Fix gen_slot=0 hardcoded override — now correctly offsets env seed per GRPO generation so parallel rollouts explore different episodes - Fix double env.step bug (was called twice per step in try/except) - Add full runbook-compliance tracking: ticket_label, ticket_priority, assigned_team, posted_channels, valid_labels, valid_priorities, valid_teams, oncall_channels extracted from meta.read_runbook response - Replace old reward formula (env*0.70 + json*0.10 + diversity) with compute_rollout_reward from training/rewards.py — reward now varies per org config so GRPO gets non-zero advantage signal - Raise temperature 0.85→1.1, add top_k=50 for generation diversity - Track prompt_seen in grpo_rollout_func to pass correct gen_offset - Update log format to show label=✓/✗ priority=✓/✗ team=✓/✗ channel=✓/✗ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>094fade -
fix(grpo): fresh reward design — runbook-compliance scoring
Root cause analysis ------------------- env_score was always 0.0 for two independent reasons: 1. create_ticket(label='sec-issue', priority='P1') fails TicketingApp validation for most org configs. The app returns ok=False, no ticket is stored, and the grader returns 0.0 immediately (no new tickets found). 2. Dataset seeds produced triage briefs but PMOpsEnvironment.reset(seed) independently ran rng.choice(_TASK_TYPES) — silently generating release_notes or dep_update tasks for the same seed. The triage grader was never called; the wrong grader always returned 0.0. Fixes ----- dataset.py: generate_triage_dataset now pre-simulates the env's first two RNG calls (difficulty + task_type) and only includes seeds where the env WILL run a triage episode, using the exact same difficulty and org params. rewards.py: replaced all previous reward formulas with compute_rollout_reward, a single function based on RUNBOOK COMPLIANCE: read_runbook +0.10 — process: read runbook fi…1c6aad2 -
fix(grpo): eliminate zero-advantage collapse from stale rewards
Three root causes all produced reward=const → advantage=0 → zero gradient: 1. generate_rollout_completions uses near-greedy decoding, making every rollout for the same prompt produce identical outputs. Replaced with _generate_no_vllm at temperature=1.1 (top_p=0.95, top_k=50). 2. Reward formula env_score*0.85 + json_ratio*0.15 collapses to 0.15 whenever env_score=0 and json_ratio=1.0 (always true after SFT). Added ok_ratio (fraction of steps env accepted) as the variance source when the task isn't yet solved; ok_ratio differs across rollouts even with identical JSON output. 3. All num_generations copies of the same prompt hit the same env seed → same starting state → same reward. Added gen_offset (tracked per prompt in grpo_rollout_func) so each generation resets the env with seed+offset, exploring a distinct org config and task variant. Also fixed _calculate_rewards tile-mod (cache[i % k] gave every generation of the same prompt an identical reward even i…38457df -
fix: format runbook as readable key-value pairs for model
fff1405 -
fix: increase runbook truncation limit 800->2000 chars
b9588df -
fix(grpo): harden rollout reward injection on cloud runtime
Add direct rollout probe fallback in PMOpsGRPOTrainer when cache is empty, and update train_v3 GRPO setup for rollout compatibility (disable fast RL patch/vLLM path, add trainer assertions and preflight probe).962ea9c -
fix(pm_ops_trainer): use direct closure capture instead of _self_ref
The _self_ref forward-reference trick was the bug — _self_ref was populated after super().__init__() but the cache assignment still used it unnecessarily. In Python, self is captured directly in nested function closures without any forward reference, so _capturing_rollout can assign self._rollout_reward_cache straight away. Added diagnostic prints to confirm capture/inject on each step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0ddcea5 -
fix(pm_ops_trainer): use direct closure capture instead of _self_ref
The _self_ref forward-reference trick was the bug — _self_ref was populated after super().__init__() but the cache assignment still used it unnecessarily. In Python, self is captured directly in nested function closures without any forward reference, so _capturing_rollout can assign self._rollout_reward_cache straight away. Added diagnostic prints to confirm capture/inject on each step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>8d3837e -
fix(pm_ops_trainer): capture rewards via rollout_func wrapper, not inputs
The original approach checked inputs[] for the 'reward' key, but TRL only puts dataset columns into inputs — rollout_func extra keys never arrive there. Fix: wrap rollout_func in __init__ to capture the reward list into self._rollout_reward_cache, then inject it in _calculate_rewards before the standard reward_funcs path runs. Also handles num_generations > 1 by repeating rewards to match completions count. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>dc36ffd
-
fix(train_v3): use PMOpsGRPOTrainer to fix reward kwarg plumbing
PatchFastRL strips custom rollout keys before calling reward_func, so the 'reward' key never arrived. PMOpsGRPOTrainer overrides _calculate_rewards to inject pre-computed rewards from the rollout batch directly, bypassing the broken kwargs path entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>22d40fb -
Merge remote-tracking branch 'hf/main' into train/v1
1b68f93 -
fix(training): fix GRPO reward bugs + add v3 notebook with SFT warmup
rollout.py fixes: - step_aware_fallback now takes max_steps param so meta.finish is reachable within the 12/15-step training cap (was checking >= 37) - read_runbook_done only set when model outputs valid JSON, not fallback - diagnostic print on step=0 JSON parse failure train_v3.ipynb (new): - SFT warmup on 60 baseline-agent traces before GRPO to fix json=0.00 - Simplified 2-signal reward: env_score*0.85 + json_ratio*0.15 - Unsloth + use_vllm=True for fast generation (was eager+HF generate) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>703b668 -
fix(reward): harsh -1.0 for zero JSON output, -0.30 for zero task completion
Aditya Guntur5bcbe7f -
fix(rollout): safe model unwrapping for accelerate/PEFT/DDP in _generate_no_vllm
Aditya Guntur606e517 -
fix(rollout): replace generate_rollout_completions (vLLM-only) with direct model.generate()
Aditya Gunturgenerate_rollout_completions hard-crashes when use_vllm=False. Replace with _generate_no_vllm() which uses HF model.generate() + output_scores=True to get prompt_ids, completion_ids, logprobs, and text without vLLM.5d330b7 -
feat(training): PMOpsGRPOTrainer — override _calculate_rewards to inject rollout rewards directly
Aditya GunturBypasses TRL 1.2.0 kwargs-drop bug: instead of reward_funcs receiving pre-computed rewards via broken **kwargs, _calculate_rewards reads the 'reward' key from the input batch and returns it as a [batch_size, 1] tensor. Falls back to standard GRPOTrainer if key is absent. - training/pm_ops_trainer.py: new PMOpsGRPOTrainer subclass - training/train_v2.ipynb: swap GRPOTrainer -> PMOpsGRPOTrainer in cell 2429d6757 -
fix(train_v2): drop Unsloth, use bitsandbytes+PEFT to fix dep conflicts
Unsloth 2026.4.8 requires torch<2.11.0 and trl<=0.24.0, both incompatible with Colab's current torch 2.11.0 and the TRL 1.2.0 we need for the rollout_func kwargs fix. Switch to stock bitsandbytes+PEFT for 4-bit+LoRA — same memory profile, no ecosystem conflicts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>14d4979 -
fix(training): reward signal flow + new v2 notebook
- rewards.py: add defensive _extract warning when kwargs are empty; add combined_reward() as single-function fallback; add COMBINED_REWARD_FUNC list - rollout.py: print per-episode reward breakdown; compute combined reward inline - train_v2.ipynb: new notebook addressing all v1 issues: * pins TRL 1.2.0 stable (no git install) * removes PatchFastRL (incompatible with TRL 1.2.0) * single 'reward' key from rollout (bypasses kwargs-flow bug) * auto-detects T4 vs A100, adapts grad_accum/num_gen/max_comp * smoke test cell before training Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>5e6d53a -
modifying the training script to be more memory efficient: Unsloth, also reduced the number of steps to 15 instead of 40
860d7e4 -
writing the GRPO script to train qwen1.7 on A100 GRPO
5bce7ac