| # SkyRL Standard-GRPO Training Metrics Analysis |
|
|
| Generated from 1 log file(s) (`--format standard`) |
|
|
| ## Overview |
|
|
| | Log File | Max Step | Train Steps | Final Reward | Max Reward | Final Entropy | |
| |----------|----------|-------------|--------------|------------|---------------| |
| | finelog | 21 | 11 | 0.4062 | 0.5684 | 0.7739 | |
|
|
| ## Async Metrics |
|
|
| | | Mean | Std | Min | Max | Count | |
| |:------------------------------|-----------:|---------:|----------:|----------:|--------:| |
| | async/discard_rate | 0 | 0 | 0 | 0 | 11 | |
| | async/discarded_count | 0 | 0 | 0 | 0 | 11 | |
| | async/effective_batch_groups | 64 | 0 | 64 | 64 | 11 | |
| | async/effective_batch_samples | 512 | 0 | 512 | 512 | 11 | |
| | async/staleness_max | 3 | 1.78885 | 1 | 7 | 11 | |
| | async/staleness_mean | 0.919034 | 0.514099 | 0.03125 | 1.82812 | 11 | |
| | async/staleness_min | 0.272727 | 0.467099 | 0 | 1 | 11 | |
| | async/staleness_ratio | 0.721591 | 0.300863 | 0.03125 | 1 | 11 | |
|
|
| ## Generate Metrics |
|
|
| | | Mean | Std | Min | Max | Count | |
| |:-------------------------------------|-----------------:|----------------:|--------------:|-----------------:|--------:| |
| | generate/avg_num_tokens | 2902.98 | 599.241 | 2234.61 | 3771.75 | 11 | |
| | generate/avg_tokens_non_zero_rewards | 2580.54 | 627.47 | 1941.57 | 3374.06 | 11 | |
| | generate/avg_tokens_zero_rewards | 3138.75 | 579.896 | 2427.73 | 4067.32 | 11 | |
| | generate/max_num_tokens | 23268.2 | 2191.1 | 20143 | 26854 | 11 | |
| | generate/min_num_tokens | 379.636 | 115.639 | 247 | 651 | 11 | |
| | generate/std_num_tokens | 2697.28 | 365.539 | 2265.94 | 3377.54 | 11 | |
| | generate/tis/aligned_tokens | 713531 | 149297 | 566081 | 955649 | 11 | |
| | generate/tis/alignment_fail_count | 0.0909091 | 0.301511 | 0 | 1 | 11 | |
| | generate/tis/exact_match_fraction | 0.999999 | 1.6768e-06 | 0.999994 | 1 | 11 | |
| | generate/tis/lcs_fallback_alert | 0 | 0 | 0 | 0 | 11 | |
| | generate/tis/lcs_fallback_fraction | 0 | 0 | 0 | 0 | 11 | |
| | generate/tis/lcs_fallback_messages | 0 | 0 | 0 | 0 | 11 | |
| | generate/tis/unaligned_fraction | 5.05574e-07 | 1.6768e-06 | 0 | 5.56131e-06 | 11 | |
| |
| ## Loss Metrics |
| |
| | | Mean | Std | Min | Max | Count | |
| |:----------------------------|-----------:|----------:|-----------:|-----------:|--------:| |
| | loss/avg_final_rewards | 0.407138 | 0.0823485 | 0.257812 | 0.568359 | 11 | |
| | loss/avg_raw_advantages | -0.0153546 | 0.0118293 | -0.0296332 | 0.00130317 | 11 | |
| | loss/avg_raw_advantages_abs | 0.117088 | 0.0249498 | 0.0684489 | 0.153624 | 11 | |
|
|
| ## Policy Metrics |
|
|
| | | Mean | Std | Min | Max | Count | |
| |:------------------------------------|----------------:|---------------:|-------------:|---------------:|--------:| |
| | policy/final_loss | -9.3325e-05 | 0.000147465 | -0.000340611 | 0.000204686 | 11 | |
| | policy/policy_entropy | 0.907788 | 0.0857953 | 0.773902 | 1.01393 | 11 | |
| | policy/policy_loss | -9.3325e-05 | 0.000147465 | -0.000340611 | 0.000204686 | 11 | |
| | policy/policy_lr | 8e-06 | 0 | 8e-06 | 8e-06 | 11 | |
| | policy/policy_update_steps | 1 | 0 | 1 | 1 | 11 | |
| | policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 11 | |
| | policy/raw_grad_norm | 0.0641336 | 0.00992899 | 0.0436851 | 0.0750673 | 11 | |
| | policy/rollout_train_prob_diff_mean | 497.177 | 1578.13 | 1.0394 | 5253.32 | 11 | |
| | policy/rollout_train_prob_diff_std | 455497 | 1.46949e+06 | 0.722785 | 4.88546e+06 | 11 | |
|
|
| ## Reward Metrics |
|
|
| | | Mean | Std | Min | Max | Count | |
| |:----------------------|---------:|----------:|---------:|---------:|--------:| |
| | reward/avg_pass_at_8 | 0.53267 | 0.055663 | 0.453125 | 0.625 | 11 | |
| | reward/avg_raw_reward | 0.407138 | 0.0823485 | 0.257812 | 0.568359 | 11 | |
| |
| ## System Metrics |
| |
| | | Mean | Std | Min | Max | Count | |
| |:------------------------|----------:|---------:|----------:|----------:|--------:| |
| | system/process_rss_gb | 20.8444 | 1.41878 | 17.5353 | 22.0891 | 11 | |
| | system/process_vms_gb | 138.577 | 9.70567 | 120.923 | 149.985 | 11 | |
| | system/ram_available_gb | 1865.23 | 3.1764 | 1862.12 | 1872.6 | 11 | |
| | system/ram_percent | 7.4 | 0.173205 | 7 | 7.6 | 11 | |
| | system/ram_total_gb | 2014.43 | 0 | 2014.43 | 2014.43 | 11 | |
| | system/ram_used_gb | 149.202 | 3.1764 | 141.826 | 152.307 | 11 | |
|
|
| ## Timing Metrics |
|
|
| | | Mean | Std | Min | Max | Count | |
| |:--------------------------------------|------------:|-------------:|--------------:|-------------:|--------:| |
| | timing/cleanup_old_checkpoints | 0.518482 | 1.69636 | 0.00617218 | 5.63319 | 11 | |
| | timing/compute_advantages_and_returns | 0.0781249 | 0.00819244 | 0.0662737 | 0.0914588 | 11 | |
| | timing/convert_to_training_input | 3.14866 | 0.373154 | 2.52558 | 3.63984 | 11 | |
| | timing/fwd_logprobs_values_reward | 46.9547 | 7.59618 | 41.8199 | 69.1615 | 11 | |
| | timing/policy_train | 148.799 | 13.4953 | 135.556 | 183.641 | 11 | |
| | timing/run_training | 195.926 | 20.8822 | 177.516 | 253.015 | 11 | |
| | timing/save_checkpoints | 464.877 | 6.8193 | 455.88 | 479.929 | 11 | |
| | timing/step | 789.831 | 479.948 | 293.359 | 1965.64 | 11 | |
| | timing/sync_weights | 123.259 | 67.5995 | 81.1427 | 264.975 | 11 | |
| | timing/train_critic_and_policy | 148.872 | 13.5064 | 135.61 | 183.741 | 11 | |
| | timing/wait_for_generation_buffer | 467.458 | 507.172 | 0.000506878 | 1703.85 | 11 | |
| | timing/save_hf_model | 37.1397 | 2.46155 | 35.3991 | 38.8803 | 2 | |
| |
| ## Tis Metrics |
| |
| | | Mean | Std | Min | Max | Count | |
| |:------------------------------|-------:|------:|------:|------:|--------:| |
| | tis/batch_skipped_no_logprobs | 0 | 0 | 0 | 0 | 11 | |
| | tis/skipped_fraction | 0 | 0 | 0 | 0 | 11 | |
| |
| ## Trainer Metrics |
| |
| | | Mean | Std | Min | Max | Count | |
| |:--------------------|-------:|--------:|------:|------:|--------:| |
| | trainer/epoch | 0 | 0 | 0 | 0 | 11 | |
| | trainer/global_step | 16 | 3.31662 | 11 | 21 | 11 | |
|
|
| ## Training Progression (collapse signals) |
|
|
| ### finelog |
|
|
| | Step | Epoch | Reward | avg_pass_at_8 | Entropy | GradNorm | PPOClip | PolicyLoss | logRatio_mean | Step Time (s) | |
| |------|-------|--------|--------|---------|----------|---------|------------|---------------|---------------| |
| | 11 | 0 | 0.4492 | 0.5156 | 0.9170 | 0.0738 | 0.00000 | -0.00011 | nan | 877.4 | |
| | 12 | 0 | 0.2578 | 0.4531 | 0.9270 | 0.0723 | 0.00000 | -0.00001 | nan | 462.7 | |
| | 13 | 0 | 0.3828 | 0.5000 | 0.9662 | 0.0748 | 0.00000 | 0.00020 | nan | 293.4 | |
| | 14 | 0 | 0.3965 | 0.5312 | 1.0139 | 0.0654 | 0.00000 | 0.00004 | nan | 312.2 | |
| | 15 | 0 | 0.3730 | 0.5312 | 1.0018 | 0.0640 | 0.00000 | -0.00017 | nan | 672.2 | |
| | 16 | 0 | 0.4941 | 0.6250 | 1.0002 | 0.0586 | 0.00000 | -0.00034 | nan | 985.8 | |
| | 17 | 0 | 0.3184 | 0.4688 | 0.9017 | 0.0541 | 0.00000 | -0.00023 | nan | 428.3 | |
| | 18 | 0 | 0.4043 | 0.5625 | 0.8676 | 0.0751 | 0.00000 | -0.00001 | nan | 693.6 | |
| | 19 | 0 | 0.4277 | 0.5469 | 0.8370 | 0.0580 | 0.00000 | -0.00007 | nan | 836.6 | |
| | 20 | 0 | 0.5684 | 0.6250 | 0.7793 | 0.0437 | 0.00000 | -0.00018 | nan | 1160.3 | |
| | 21 | 0 | 0.4062 | 0.5000 | 0.7739 | 0.0657 | 0.00000 | -0.00015 | nan | 1965.6 | |
|
|
| ## vLLM Inference Engine Analysis (per-engine) |
|
|
| | Log | Avg Running/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % | |
| |-----|-------------------|--------------------------|----------------|------------------| |
| | finelog | 2.0 | 232.1 tok/s | 0.7% | 90.2% | |
|
|
| ## Best Checkpoint (trailing-5 EMA of reward/avg_raw_reward) |
|
|
| No step chosen: No saved-aligned checkpoint eligible (after cap + exports intersection). available_exports=[], cap_step=None, save_every=20 |
| |
| - available exports: `[]` |
| - cap (latest_ckpt_global_step.txt): `None` |
|
|
| | Step | Reward | EMA | Eligible | |
| |------|--------|-----|----------| |
| | 11 | 0.4492 | 0.4492 | | |
| | 12 | 0.2578 | 0.3854 | | |
| | 13 | 0.3828 | 0.3845 | | |
| | 14 | 0.3965 | 0.3885 | | |
| | 15 | 0.3730 | 0.3834 | | |
| | 16 | 0.4941 | 0.4203 | | |
| | 17 | 0.3184 | 0.3863 | | |
| | 18 | 0.4043 | 0.3923 | | |
| | 19 | 0.4277 | 0.4041 | | |
| | 20 | 0.5684 | 0.4589 | | |
| | 21 | 0.4062 | 0.4413 | | |
|
|
|
|
