Spaces:
Paused
Paused
Commit History
-
Fix training rc=127 by using python -m fallback and tee logs to stdout
siddeshwar-kagatikarMade-with: Cursor8828fdd -
Stop training failures from killing the API server (fixes 500 on Space)
siddeshwar-kagatikarAfter the previous restructure, scripts/space_start.sh ran with `set -eu` and an EXIT trap that killed uvicorn. If `osint-env train-self-play` crashed for any reason (e.g. hardware mismatch with the L40S config, missing GPU, transient HF Hub download error) the script exited, the trap fired, and uvicorn died with it. Visitors then saw an internal server error / Space unavailable response on every route, including the dashboards which are completely independent of training. Rewrite the start script so: - `set -e` is dropped; non-zero exits no longer cascade. - uvicorn runs as the persistent process the script `wait`s on, so the container stays alive as long as the API is healthy. - training runs in a backgrounded supervised subshell that logs its exit code to /tmp/self_play_training.log and never propagates failure to the parent script. - The signal trap is INT/TERM only (real platform shutdown), forwarding signals to both children, instead of EXIT which fired on every script…04ad851 -
Add evaluation, minor updates to HF space
8ad6382 -
Make self-play training resilient to HF Space restarts
siddeshwar-kagatikarThe L40S full run was getting killed at ~step 84/120 of the generator phase because uvicorn (PID 1) caught a SIGTERM from the Space and the backgrounded training subshell died with the container, with no recovery path. The answerer reward graph was empty because training never reached the answerer phase. scripts/space_start.sh: invert the lifecycle. uvicorn now runs in the background (so HF healthchecks pass) and self-play training runs in the foreground as the script's blocking child. A SIGTERM from the platform now propagates to the trainer instead of cleanly tearing down PID 1, and the EXIT trap stops uvicorn afterwards. src/osint_env/training/self_play.py: _train_grpo_phase now resumes work on every restart. It (1) skips the phase entirely if final_model is already on disk, (2) auto-detects the latest local checkpoint-* and calls trainer.train(resume_from_checkpoint=...) when full state (optimizer.pt + trainer_state.json) is present, (3) falls back to downloading the latest check…2e14f6d
-
Update training config, add checkpointing on HF
e44cdee -
Sync current main to Hugging Face Space
siddeshwar-kagatikarMade-with: Cursorfe1f842 -
fix(rewards): never crash GRPO on malformed completions
siddeshwar-kagatikarTrainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary strings (``"none"``, ``"N/A"``, ``"2 agents"``). Use it for every orchestrator integer. - Wrap ``decode_completion_text`` and ``parse_generated_task_completion`` in defensive ``try/except`` inside ``GeneratorRewardFunction.__call__``; on any parsing/reward exception, fall through to a graded floor reward so GRPO keeps training and the failure is logged via ``[reward_debug][parse_error]`` / ``[reward_debug][reward_error]``. - Add regression tests for both behaviours. Made-with: Cursord814291