Spaces:
Paused
Paused
Commit History
-
Add evaluation, minor updates to HF space
8ad6382 -
Add per-batch generation liveness logging
siddeshwar-kagatikarThe generator and answerer samplers print their progress *after* each batch completes. With prompts=24, generation_batch_size=16, max_new_tokens=640 on an L40S, the very first batch can sit silently for 60-90 seconds because of CUDA kernel autotuning and KV cache allocation. That made the user think training had hung after the "model_loaded" line. Print a "generating..." line *before* each model.generate() call and a "generate_elapsed=Ts" line right after, in all three samplers (_sample_swarm_v2_completion_texts_with_model, _sample_generated_tasks_with_model, _generate_answerer_completion_texts_with_model). Now every batch has a visible start and end with timings, so liveness is obvious even during the slow first-batch warmup. Made-with: Cursor55c5f82 -
Upload intermediate GRPO checkpoints to HF Hub on every save
siddeshwar-kagatikarThe resume-from-HF-Hub fallback in _train_grpo_phase only helps if HF Hub actually has a fresh checkpoint when a Space restarts mid-phase. Until now uploads only happened at the end of each phase, so a crash at e.g. step 70/120 would leave the Hub stuck at the start-of-phase weights and force training to restart from scratch. Add a Trainer callback _HfHubCheckpointUploadCallback registered via trainer_kwargs["callbacks"] in _train_grpo_phase. On every Trainer save (driven by phase.save_steps) it locates the newest local checkpoint-* directory and pushes it to OSINT_HF_CHECKPOINT_REPO_ID through the existing _maybe_upload_folder_to_hf helper, reusing the ignore patterns that strip optimizer/scheduler state to keep uploads manageable. The callback is built lazily so transformers stays an optional dependency. Failures are caught and logged; training never stops because of an upload error, and the next save_steps retries. Behavior is opt-out via OSINT_HF_UPLOAD_ON_SAVE=0 and only activat…b0b586a -
Make self-play training resilient to HF Space restarts
siddeshwar-kagatikarThe L40S full run was getting killed at ~step 84/120 of the generator phase because uvicorn (PID 1) caught a SIGTERM from the Space and the backgrounded training subshell died with the container, with no recovery path. The answerer reward graph was empty because training never reached the answerer phase. scripts/space_start.sh: invert the lifecycle. uvicorn now runs in the background (so HF healthchecks pass) and self-play training runs in the foreground as the script's blocking child. A SIGTERM from the platform now propagates to the trainer instead of cleanly tearing down PID 1, and the EXIT trap stops uvicorn afterwards. src/osint_env/training/self_play.py: _train_grpo_phase now resumes work on every restart. It (1) skips the phase entirely if final_model is already on disk, (2) auto-detects the latest local checkpoint-* and calls trainer.train(resume_from_checkpoint=...) when full state (optimizer.pt + trainer_state.json) is present, (3) falls back to downloading the latest check…2e14f6d -
feat(training): improve self-play progress visibility and reward diagnostics
siddeshwar-kagatikarBatch generation/sampling and add explicit reward instrumentation so HF Space runs do not appear stalled and answerer reward signals remain continuously observable in logs and W&B. Made-with: Cursor4aca4f5
-
Minor config changes
3e893cd -
Update training config, add checkpointing on HF
e44cdee -
test hf space commit
commit 8ddb8edbeb82f8d1b73954dbdee4854f4792210b Merge: 20d6040 fe1f842 Author: Ritish Shrirao <ritishshrirao@gmail.com> Date: Sun Apr 26 02:33:06 2026 +0530 Merge HF Space history into local main commit 20d6040f8ddacfbd24251186cea73b05b4514f84 Author: Ritish Shrirao <ritishshrirao@gmail.com> Date: Sun Apr 26 01:32:55 2026 +0530 Update rewards, training scripts, add HF jobs for training commit 6ea85753f94f96755595dfe5d9652437b0d4c0b7 Author: siddeshwar-kagatikar <siddeshwar2004@gamil.com> Date: Sat Apr 25 22:34:30 2026 +0530 fix(rewards): never crash GRPO on malformed completions Trainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary stri…d822755 -
Sync current main to Hugging Face Space
siddeshwar-kagatikarMade-with: Cursorfe1f842 -
fix(rewards): never crash GRPO on malformed completions
siddeshwar-kagatikarTrainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary strings (``"none"``, ``"N/A"``, ``"2 agents"``). Use it for every orchestrator integer. - Wrap ``decode_completion_text`` and ``parse_generated_task_completion`` in defensive ``try/except`` inside ``GeneratorRewardFunction.__call__``; on any parsing/reward exception, fall through to a graded floor reward so GRPO keeps training and the failure is logged via ``[reward_debug][parse_error]`` / ``[reward_debug][reward_error]``. - Add regression tests for both behaviours. Made-with: Cursord814291