Spaces:
Paused
Paused
Commit History
-
Add W&B training run link to blog
siddeshwar-kagatikarMade-with: Cursor0738dc1 -
Fix training rc=127 by using python -m fallback and tee logs to stdout
siddeshwar-kagatikarMade-with: Cursor8828fdd -
Document all Space variables and secrets in README
siddeshwar-kagatikarMade-with: Cursor274f638 -
Fix LaTeX math rendering in blog (use \$\$ and \$ delimiters)
siddeshwar-kagatikarMade-with: Cursor9e3e5ff -
Add Trace Net blog post with final checkpoint link
siddeshwar-kagatikarMade-with: Cursor72cba6b -
Add TraceNet slide deck link to README
siddeshwar-kagatikarMade-with: Cursor0a16175 -
Remove stale COMPARE_FINETUNED_DASHBOARD references (fix 500 on Space)
siddeshwar-kagatikarMade-with: Cursor55d75e0 -
Stop training failures from killing the API server (fixes 500 on Space)
siddeshwar-kagatikarAfter the previous restructure, scripts/space_start.sh ran with `set -eu` and an EXIT trap that killed uvicorn. If `osint-env train-self-play` crashed for any reason (e.g. hardware mismatch with the L40S config, missing GPU, transient HF Hub download error) the script exited, the trap fired, and uvicorn died with it. Visitors then saw an internal server error / Space unavailable response on every route, including the dashboards which are completely independent of training. Rewrite the start script so: - `set -e` is dropped; non-zero exits no longer cascade. - uvicorn runs as the persistent process the script `wait`s on, so the container stays alive as long as the API is healthy. - training runs in a backgrounded supervised subshell that logs its exit code to /tmp/self_play_training.log and never propagates failure to the parent script. - The signal trap is INT/TERM only (real platform shutdown), forwarding signals to both children, instead of EXIT which fired on every script…04ad851 -
Fix dashboard 404 by shipping artifacts/ into the Space image
siddeshwar-kagatikarThe Space homepage exposed Post-Training and Pre-Training dashboard buttons but every click returned {"detail":"Post-training dashboard not found"} because the Dockerfile never copied the artifacts/ tree into the image and .dockerignore explicitly excluded it. The html files were on hf/main but absent from the runtime container fs. .dockerignore: keep ignoring artifacts/* by default but un-ignore the small evaluation/baseline/dashboard files that the Space UI loads: artifacts/baselines/**, artifacts/local_hf_eval_round_006/**, artifacts/latest_evaluation.json, artifacts/leaderboard.json. Dockerfile: COPY --chown=user artifacts $HOME/app/artifacts so the selectively-allowed files actually land inside the image. Heavy training outputs stay excluded by the artifacts/* rule above. Also rename the dashboard html titles to "OSINT Benchmark Dashboard post Training" / "OSINT Benchmark Dashboard pre Training" so the two views are distinguishable inside the iframe view. Made-with: Cursor7e4ee5e -
Add missing post_training_benchmark_dashboard.html to HF Space
siddeshwar-kagatikarMade-with: Cursor07286e6 -
Always render Pre/Post-Training dashboard buttons; drop snapshot+difficulty cards
siddeshwar-kagatikarRender the Post-Training and Pre-Training dashboard buttons unconditionally on the Space homepage so visitors can always pick between the two benchmark dashboards. Each route still serves a 404 when the underlying html is missing, so behavior is safe. Remove the "Included Snapshot" stat card (Tasks/Provider/Score/Success) and the "Difficulty Mix" card from the homepage as requested. Update the lower grid to 2 columns now that only Action Space and Task Families remain. Made-with: Cursor243dfa6 -
Add pre/post benchmark dashboard options in Space UI
siddeshwar-kagatikarReplace the single default dashboard entry on the Space home page with explicit Post-Training and Pre-Training dashboard links, and wire those to dedicated routes. Update the compare-dashboard path mapping to use pre_training_benchmark_dashboard_original.html and keep compatibility aliases for existing /dashboard/finetuned and /dashboard/base URLs. Made-with: Cursor9f98669 -
added images (hf-space text-only subset)
siddeshwar-kagatikarMade-with: Cursor957b4b2 -
Add evaluation, minor updates to HF space
8ad6382 -
Add per-batch generation liveness logging
siddeshwar-kagatikarThe generator and answerer samplers print their progress *after* each batch completes. With prompts=24, generation_batch_size=16, max_new_tokens=640 on an L40S, the very first batch can sit silently for 60-90 seconds because of CUDA kernel autotuning and KV cache allocation. That made the user think training had hung after the "model_loaded" line. Print a "generating..." line *before* each model.generate() call and a "generate_elapsed=Ts" line right after, in all three samplers (_sample_swarm_v2_completion_texts_with_model, _sample_generated_tasks_with_model, _generate_answerer_completion_texts_with_model). Now every batch has a visible start and end with timings, so liveness is obvious even during the slow first-batch warmup. Made-with: Cursor55c5f82 -
Upload intermediate GRPO checkpoints to HF Hub on every save
siddeshwar-kagatikarThe resume-from-HF-Hub fallback in _train_grpo_phase only helps if HF Hub actually has a fresh checkpoint when a Space restarts mid-phase. Until now uploads only happened at the end of each phase, so a crash at e.g. step 70/120 would leave the Hub stuck at the start-of-phase weights and force training to restart from scratch. Add a Trainer callback _HfHubCheckpointUploadCallback registered via trainer_kwargs["callbacks"] in _train_grpo_phase. On every Trainer save (driven by phase.save_steps) it locates the newest local checkpoint-* directory and pushes it to OSINT_HF_CHECKPOINT_REPO_ID through the existing _maybe_upload_folder_to_hf helper, reusing the ignore patterns that strip optimizer/scheduler state to keep uploads manageable. The callback is built lazily so transformers stays an optional dependency. Failures are caught and logged; training never stops because of an upload error, and the next save_steps retries. Behavior is opt-out via OSINT_HF_UPLOAD_ON_SAVE=0 and only activat…b0b586a -
Make self-play training resilient to HF Space restarts
siddeshwar-kagatikarThe L40S full run was getting killed at ~step 84/120 of the generator phase because uvicorn (PID 1) caught a SIGTERM from the Space and the backgrounded training subshell died with the container, with no recovery path. The answerer reward graph was empty because training never reached the answerer phase. scripts/space_start.sh: invert the lifecycle. uvicorn now runs in the background (so HF healthchecks pass) and self-play training runs in the foreground as the script's blocking child. A SIGTERM from the platform now propagates to the trainer instead of cleanly tearing down PID 1, and the EXIT trap stops uvicorn afterwards. src/osint_env/training/self_play.py: _train_grpo_phase now resumes work on every restart. It (1) skips the phase entirely if final_model is already on disk, (2) auto-detects the latest local checkpoint-* and calls trainer.train(resume_from_checkpoint=...) when full state (optimizer.pt + trainer_state.json) is present, (3) falls back to downloading the latest check…2e14f6d -
feat(training): improve self-play progress visibility and reward diagnostics
siddeshwar-kagatikarBatch generation/sampling and add explicit reward instrumentation so HF Space runs do not appear stalled and answerer reward signals remain continuously observable in logs and W&B. Made-with: Cursor4aca4f5
-
Minor config changes
3e893cd -
Update training config, add checkpointing on HF
e44cdee -
test hf space commit
commit 8ddb8edbeb82f8d1b73954dbdee4854f4792210b Merge: 20d6040 fe1f842 Author: Ritish Shrirao <ritishshrirao@gmail.com> Date: Sun Apr 26 02:33:06 2026 +0530 Merge HF Space history into local main commit 20d6040f8ddacfbd24251186cea73b05b4514f84 Author: Ritish Shrirao <ritishshrirao@gmail.com> Date: Sun Apr 26 01:32:55 2026 +0530 Update rewards, training scripts, add HF jobs for training commit 6ea85753f94f96755595dfe5d9652437b0d4c0b7 Author: siddeshwar-kagatikar <siddeshwar2004@gamil.com> Date: Sat Apr 25 22:34:30 2026 +0530 fix(rewards): never crash GRPO on malformed completions Trainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary stri…d822755 -
Sync current main to Hugging Face Space
siddeshwar-kagatikarMade-with: Cursorfe1f842 -
fix(rewards): never crash GRPO on malformed completions
siddeshwar-kagatikarTrainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary strings (``"none"``, ``"N/A"``, ``"2 agents"``). Use it for every orchestrator integer. - Wrap ``decode_completion_text`` and ``parse_generated_task_completion`` in defensive ``try/except`` inside ``GeneratorRewardFunction.__call__``; on any parsing/reward exception, fall through to a graded floor reward so GRPO keeps training and the failure is logged via ``[reward_debug][parse_error]`` / ``[reward_debug][reward_error]``. - Add regression tests for both behaviours. Made-with: Cursord814291