Hugging Face's logo

Asilarkness
/
testgeniy

Text Generation
Transformers
Safetensors
English
testgeniy
causal-lm
reasoning
mathematics
logic
long-context
4k-context
small-language-model
Model card Files
xet
 Community
Asilarkness commited on
Commit
014c05f
·
verified ·
1 Parent(s): fe54262

Math-augmented RL v2: constrained joint source with gradient projection

Browse files
Files changed (30) hide show
  1. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/FINAL_REPORT.md +38 -0
  2. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/INFORMATION_RL_POLICY.md +65 -0
  3. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/MATH_AUGMENTED_RL_POLICY.md +13 -0
  4. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/STAGE1_REPORT.md +13 -0
  5. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cases.jsonl +0 -0
  6. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cheap_dev_gate.json +121 -0
  7. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cheap_dev_more.json +104 -0
  8. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/constrained_training_report.json +1409 -0
  9. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/curriculum_manifest.json +20 -0
  10. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/encoded_rollouts.jsonl +0 -0
  11. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/encoding_manifest.json +7 -0
  12. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/final_decision.json +101 -0
  13. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_dev_gate.json +206 -0
  14. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_test_gate.json +53 -0
  15. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_training_report.json +768 -0
  16. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/info_math_constrained_source.pt +3 -0
  17. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/info_rl_v1.py +268 -0
  18. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/long_dev_gate.json +189 -0
  19. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_encoded_rollouts.jsonl +0 -0
  20. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_encoding_manifest.json +7 -0
  21. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rl_manifest.json +13 -0
  22. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rl_pool.jsonl +0 -0
  23. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rollout_manifest.json +17 -0
  24. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rollouts_scored.jsonl +0 -0
  25. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/reason_gate_scores.json +264 -0
  26. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/rl_rollouts.jsonl +0 -0
  27. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/rollout_manifest.json +12 -0
  28. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/stage1_decision.json +187 -0
  29. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/training_report.json +728 -0
  30. candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/training_report_long.json +2386 -0
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/FINAL_REPORT.md ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Information-management RL v1 — final decision
2
+
3
+ ## Decision
4
+
5
+ **Rejected. `verified-math-a025` remains the leader.** The public root was not modified and model size stayed at 487,800,064 parameters.
6
+
7
+ ## Work completed
8
+
9
+ - 105 manually specified long multi-turn cases: 63 train, 21 dev, 21 held-out test.
10
+ - Seven skills: current-state updates, stale-step avoidance, selective retrieval, delta summaries, conflict clarification, source boundaries, and accumulated constraints.
11
+ - 252 stochastic Budgie rollouts and 63 manual ideal demonstrations.
12
+ - Qwen3.8-27B group rewards with explicit stale-fact, missing-state, format, honesty, and repetition penalties.
13
+ - Offline clipped token-level GRPO/PPO with immutable-leader KL.
14
+ - A longer three-epoch GRPO run plus manual-demonstration behavior cloning.
15
+ - An eight-epoch high-quality ideal-response consolidation after RL.
16
+
17
+ ## Held-out information-management results
18
+
19
+ | Model | Dev pass | Dev loops | Test pass | Test loops |
20
+ |---|---:|---:|---:|---:|
21
+ | leader | 3/21 | 0 | 1/21 | 1 |
22
+ | hybrid alpha .025 | 4/21 | 0 | 2/21 | 1 |
23
+ | full hybrid source | 7/21 | 2 | 5/21 | 2 |
24
+
25
+ The full source learns conflict clarification and some selective/source-boundary behavior, but increases repetition and still fails all held-out current-state-update cases. The small alpha gives a modest no-loop transfer gain.
26
+
27
+ ## Protected reasoning gate
28
+
29
+ | Model | GSM /30 | MATH /15 | ARC /30 | FOLIO /30 |
30
+ |---|---:|---:|---:|---:|
31
+ | leader | 5 | 3 | 11 | 13 |
32
+ | hybrid alpha .025 | 3 | 1 | 11 | 13 |
33
+
34
+ The only low-drift information candidate regresses GSM and MATH sharply, so it is rejected before HelpSteer/expanded gates. Smaller alphas do not improve the information holdout.
35
+
36
+ ## Conclusion
37
+
38
+ Manual long-form RL produced measurable specialist behavior, but not a Pareto-safe general assistant. Do not promote or continue from this branch. The durable curriculum, rewards, rollouts, code, reports, and one reconstructable hybrid source are retained for research; future work must restart from `verified-math-a025`.
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/INFORMATION_RL_POLICY.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Budgie information-management RL v1
2
+
3
+ ## Objective
4
+
5
+ Improve long English dialogue without increasing Budgie's 487,800,064 parameters. The target behavior is to maintain a compact internal conversation state: use the newest correction, retain active constraints, answer only the requested slice of information, distinguish supplied facts from unknown facts, ask one focused clarification on unresolved conflicts, avoid repeating attempted actions, and stop after a complete answer.
6
+
7
+ Immutable base: `verified-math-a025` / `ba2_vmath_a025`, SHA-256 `7404de2c08df968f41800d1362704a0ad4106ea0116cd0cd4b1709612a2a66be`.
8
+
9
+ ## Manual curriculum
10
+
11
+ Seven independently scored skills:
12
+
13
+ 1. state updates and stale-fact suppression;
14
+ 2. avoiding previously attempted troubleshooting steps;
15
+ 3. selective retrieval from distracting context;
16
+ 4. delta-only summaries after updates;
17
+ 5. focused clarification under conflicting records;
18
+ 6. source boundaries and explicit uncertainty;
19
+ 7. accumulated exact constraints.
20
+
21
+ There are 105 manually specified multi-turn cases: 63 train, 21 development, and 21 held-out test. Topic instances are disjoint across splits. Every case contains an ideal response, explicit required/forbidden state anchors, format checks, and repetition checks.
22
+
23
+ ## Reward
24
+
25
+ Each rollout receives:
26
+
27
+ - deterministic state/constraint reward;
28
+ - stale-information penalties;
29
+ - missing-current-fact penalties;
30
+ - repetition and duplicate-sentence penalties;
31
+ - canned-closing penalty;
32
+ - Qwen3.8-27B semantic scores for state accuracy, instruction following, relevance, honesty, and non-repetition;
33
+ - a hard cap when an explicit current fact or format constraint is violated.
34
+
35
+ The semantic judge sees the full conversation, ideal reference, checks, and all candidate answers at once. Qwen thinking is disabled. Candidate order is rotated.
36
+
37
+ ## RL algorithm
38
+
39
+ Use offline one-epoch clipped group-relative policy optimization:
40
+
41
+ - four stochastic Budgie rollouts per train case plus one verified teacher demonstration;
42
+ - rewards normalized within each case;
43
+ - token-level PPO clipping against stored leader log-probabilities;
44
+ - sampled-action KL penalty to the immutable leader;
45
+ - entropy regularization;
46
+ - FP32 master weights and BF16 autocast;
47
+ - gradient clipping;
48
+ - protected random/synthetic reasoning replay;
49
+ - conservative updates and task-vector interpolation.
50
+
51
+ This is not DPO. Strong and weak DPO previously failed; the new run uses explicit per-rollout scalar rewards, group advantages, PPO clipping, and leader KL.
52
+
53
+ ## Stop rules
54
+
55
+ A candidate is rejected immediately if any of these occur:
56
+
57
+ - held-out information-management reward or pass rate does not improve;
58
+ - repetition-loop rate rises;
59
+ - honesty/source-boundary pass rate falls;
60
+ - fixed GSM8K, MATH-500, ARC-Challenge, FOLIO, or HelpSteer2 falls below `5/30`, `3/15`, `11/30`, `13/30`, or `97/200`;
61
+ - the 36-prompt dialogue gate loses honesty or increases loops;
62
+ - parameters, tokenizer, or architecture change;
63
+ - NaN/Inf or malformed weights appear.
64
+
65
+ Only a candidate passing the fixed gate may run the expanded/full gate. Rejected branches never become the next base, and the public root is not overwritten automatically.
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/MATH_AUGMENTED_RL_POLICY.md ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Budgie information + mathematics RL v2
2
+
3
+ Start from `verified-math-a025`, not from the rejected information specialist.
4
+
5
+ Joint groups:
6
+
7
+ - 63 manual long-dialogue information-management groups retained from v1;
8
+ - 96 random, decontaminated, symbolically verified mathematics problems from `open-r1/OpenR1-Math-220k`;
9
+ - no GSM8K, MATH-500, Hendrycks MATH, ARC, or FOLIO training rows.
10
+
11
+ Each math group contains stochastic Budgie solutions plus a verified reference solution. Rewards are +1 for symbolic equivalence, negative for incorrect/malformed/repetitive solutions, and group-normalized. The policy objective is clipped token-level GRPO/PPO with KL to the immutable leader. Information and math groups are mixed at no less than 40:60, so dialogue gradients cannot dominate protected reasoning. Manual ideal responses and verified math references may receive a small behavior-cloning anchor.
12
+
13
+ Candidates must improve held-out information-management tests without increasing loops, and must preserve at least GSM 5/30, MATH 3/15, ARC 11/30, FOLIO 13/30, and HelpSteer 97/200 before expanded evaluation. Model size and tokenizer remain unchanged.
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/STAGE1_REPORT.md ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Information-management RL v1 — preliminary decision
2
+
3
+ A manual long-dialogue curriculum of 105 cases was built across state updates, stale-step avoidance, selective retrieval, delta summaries, conflict clarification, source boundaries, and accumulated constraints. Sixty-three train cases produced 252 stochastic Budgie rollouts plus 63 manual ideal demonstrations. Qwen3.8-27B assigned group rewards. Training used clipped token-level GRPO/PPO with leader KL, followed by a longer GRPO run and manual high-quality consolidation. Model size stayed unchanged.
4
+
5
+ ## Held-out deterministic gate
6
+
7
+ | Model | Dev pass | Dev loops | Test pass | Test loops |
8
+ |---|---:|---:|---:|---:|
9
+ | leader | 3/21 | 0 | 1/21 | 1 |
10
+ | hybrid alpha 0.025 | 4/21 | 0 | 2/21 | 1 |
11
+ | full hybrid source | 7/21 | 2 | 5/21 | 2 |
12
+
13
+ The small alpha has a modest transfer gain without extra loops. The full source improves conflict handling and selective retrieval but increases repetitions. Neither is promoted yet. `hybrid-a0p025` must pass fixed GSM/MATH/ARC/FOLIO/HelpSteer and Qwen pairwise dialogue gates before retention.
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cases.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cheap_dev_gate.json ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "leader": {
3
+ "alpha": 0,
4
+ "pass": 3,
5
+ "n": 21,
6
+ "mean_reward": 0.3333333333333334,
7
+ "loops": 0,
8
+ "mean_words": 8.857142857142858,
9
+ "by_category": {
10
+ "already_tried": 0,
11
+ "conflict_clarification": 0,
12
+ "constraint_stack": 2,
13
+ "delta_summary": 0,
14
+ "selective_answer": 1,
15
+ "source_boundary": 0,
16
+ "state_update": 0
17
+ }
18
+ },
19
+ "a0p001": {
20
+ "alpha": 0.001,
21
+ "pass": 3,
22
+ "n": 21,
23
+ "mean_reward": 0.3333333333333334,
24
+ "loops": 0,
25
+ "mean_words": 8.857142857142858,
26
+ "by_category": {
27
+ "already_tried": 0,
28
+ "conflict_clarification": 0,
29
+ "constraint_stack": 2,
30
+ "delta_summary": 0,
31
+ "selective_answer": 1,
32
+ "source_boundary": 0,
33
+ "state_update": 0
34
+ }
35
+ },
36
+ "a0p0025": {
37
+ "alpha": 0.0025,
38
+ "pass": 3,
39
+ "n": 21,
40
+ "mean_reward": 0.3333333333333334,
41
+ "loops": 0,
42
+ "mean_words": 8.857142857142858,
43
+ "by_category": {
44
+ "already_tried": 0,
45
+ "conflict_clarification": 0,
46
+ "constraint_stack": 2,
47
+ "delta_summary": 0,
48
+ "selective_answer": 1,
49
+ "source_boundary": 0,
50
+ "state_update": 0
51
+ }
52
+ },
53
+ "a0p005": {
54
+ "alpha": 0.005,
55
+ "pass": 3,
56
+ "n": 21,
57
+ "mean_reward": 0.3333333333333334,
58
+ "loops": 0,
59
+ "mean_words": 9.0,
60
+ "by_category": {
61
+ "already_tried": 0,
62
+ "conflict_clarification": 0,
63
+ "constraint_stack": 2,
64
+ "delta_summary": 0,
65
+ "selective_answer": 1,
66
+ "source_boundary": 0,
67
+ "state_update": 0
68
+ }
69
+ },
70
+ "a0p01": {
71
+ "alpha": 0.01,
72
+ "pass": 3,
73
+ "n": 21,
74
+ "mean_reward": 0.3333333333333334,
75
+ "loops": 0,
76
+ "mean_words": 9.238095238095237,
77
+ "by_category": {
78
+ "already_tried": 0,
79
+ "conflict_clarification": 0,
80
+ "constraint_stack": 2,
81
+ "delta_summary": 0,
82
+ "selective_answer": 1,
83
+ "source_boundary": 0,
84
+ "state_update": 0
85
+ }
86
+ },
87
+ "a0p025": {
88
+ "alpha": 0.025,
89
+ "pass": 3,
90
+ "n": 21,
91
+ "mean_reward": 0.3333333333333334,
92
+ "loops": 0,
93
+ "mean_words": 9.047619047619047,
94
+ "by_category": {
95
+ "already_tried": 0,
96
+ "conflict_clarification": 0,
97
+ "constraint_stack": 2,
98
+ "delta_summary": 0,
99
+ "selective_answer": 1,
100
+ "source_boundary": 0,
101
+ "state_update": 0
102
+ }
103
+ },
104
+ "a0p05": {
105
+ "alpha": 0.05,
106
+ "pass": 3,
107
+ "n": 21,
108
+ "mean_reward": 0.3333333333333334,
109
+ "loops": 0,
110
+ "mean_words": 8.80952380952381,
111
+ "by_category": {
112
+ "already_tried": 0,
113
+ "conflict_clarification": 0,
114
+ "constraint_stack": 2,
115
+ "delta_summary": 0,
116
+ "selective_answer": 1,
117
+ "source_boundary": 0,
118
+ "state_update": 0
119
+ }
120
+ }
121
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/cheap_dev_more.json ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "a0p1": {
3
+ "alpha": 0.1,
4
+ "pass": 3,
5
+ "n": 21,
6
+ "mean_reward": 0.3333333333333334,
7
+ "loops": 0,
8
+ "mean_words": 8.761904761904763,
9
+ "by_category": {
10
+ "already_tried": 0,
11
+ "conflict_clarification": 0,
12
+ "constraint_stack": 2,
13
+ "delta_summary": 0,
14
+ "selective_answer": 1,
15
+ "source_boundary": 0,
16
+ "state_update": 0
17
+ }
18
+ },
19
+ "a0p2": {
20
+ "alpha": 0.2,
21
+ "pass": 4,
22
+ "n": 21,
23
+ "mean_reward": 0.3500000000000001,
24
+ "loops": 0,
25
+ "mean_words": 8.714285714285714,
26
+ "by_category": {
27
+ "already_tried": 0,
28
+ "conflict_clarification": 0,
29
+ "constraint_stack": 3,
30
+ "delta_summary": 0,
31
+ "selective_answer": 1,
32
+ "source_boundary": 0,
33
+ "state_update": 0
34
+ }
35
+ },
36
+ "a0p35": {
37
+ "alpha": 0.35,
38
+ "pass": 4,
39
+ "n": 21,
40
+ "mean_reward": 0.3500000000000001,
41
+ "loops": 0,
42
+ "mean_words": 8.571428571428571,
43
+ "by_category": {
44
+ "already_tried": 0,
45
+ "conflict_clarification": 0,
46
+ "constraint_stack": 3,
47
+ "delta_summary": 0,
48
+ "selective_answer": 1,
49
+ "source_boundary": 0,
50
+ "state_update": 0
51
+ }
52
+ },
53
+ "a0p5": {
54
+ "alpha": 0.5,
55
+ "pass": 3,
56
+ "n": 21,
57
+ "mean_reward": 0.3500000000000001,
58
+ "loops": 0,
59
+ "mean_words": 8.80952380952381,
60
+ "by_category": {
61
+ "already_tried": 0,
62
+ "conflict_clarification": 0,
63
+ "constraint_stack": 2,
64
+ "delta_summary": 0,
65
+ "selective_answer": 1,
66
+ "source_boundary": 0,
67
+ "state_update": 0
68
+ }
69
+ },
70
+ "a0p75": {
71
+ "alpha": 0.75,
72
+ "pass": 3,
73
+ "n": 21,
74
+ "mean_reward": 0.3333333333333334,
75
+ "loops": 0,
76
+ "mean_words": 8.619047619047619,
77
+ "by_category": {
78
+ "already_tried": 0,
79
+ "conflict_clarification": 0,
80
+ "constraint_stack": 2,
81
+ "delta_summary": 0,
82
+ "selective_answer": 1,
83
+ "source_boundary": 0,
84
+ "state_update": 0
85
+ }
86
+ },
87
+ "a1p0": {
88
+ "alpha": 1.0,
89
+ "pass": 3,
90
+ "n": 21,
91
+ "mean_reward": 0.3333333333333334,
92
+ "loops": 0,
93
+ "mean_words": 8.904761904761905,
94
+ "by_category": {
95
+ "already_tried": 0,
96
+ "conflict_clarification": 0,
97
+ "constraint_stack": 2,
98
+ "delta_summary": 0,
99
+ "selective_answer": 1,
100
+ "source_boundary": 0,
101
+ "state_update": 0
102
+ }
103
+ }
104
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/constrained_training_report.json ADDED
@@ -0,0 +1,1409 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "algorithm": "anchor-first constrained GRPO with global gradient projection",
3
+ "base": "verified-math-a025",
4
+ "info_groups": 63,
5
+ "math_groups": 72,
6
+ "info_samples": 126,
7
+ "math_samples": 144,
8
+ "updates": 126,
9
+ "math_to_info_per_update": "2:1",
10
+ "info_gradient_weight": 0.45,
11
+ "peak_lr": 3e-07,
12
+ "ppo_clip": 0.1,
13
+ "kl_beta": 0.06,
14
+ "math_bc": 0.12,
15
+ "info_bc": 0.08,
16
+ "gradient_conflict_rate": 0.373015873015873,
17
+ "history": [
18
+ {
19
+ "update": 1,
20
+ "math_pg": -1.414245367050171,
21
+ "math_kl": 0.0004710083594545722,
22
+ "info_pg": -2.002326726913452,
23
+ "info_kl": 0.0010085623944178224,
24
+ "conflict": true,
25
+ "cos_dot": -5.22668981552124,
26
+ "grad_norm": 115.39231872558594,
27
+ "lr": 3.75e-08
28
+ },
29
+ {
30
+ "update": 2,
31
+ "math_pg": -0.3563050925731659,
32
+ "math_kl": 0.0005872978072147816,
33
+ "info_pg": -1.9616996049880981,
34
+ "info_kl": 0.0008686535293236375,
35
+ "conflict": false,
36
+ "cos_dot": 0.10240434110164642,
37
+ "grad_norm": 51.239524841308594,
38
+ "lr": 7.5e-08
39
+ },
40
+ {
41
+ "update": 3,
42
+ "math_pg": -1.4148276448249817,
43
+ "math_kl": 0.0005600455042440444,
44
+ "info_pg": 0.8755850791931152,
45
+ "info_kl": 0.000935528427362442,
46
+ "conflict": false,
47
+ "cos_dot": 0.1080315038561821,
48
+ "grad_norm": 19.87111473083496,
49
+ "lr": 1.125e-07
50
+ },
51
+ {
52
+ "update": 4,
53
+ "math_pg": -1.3954129219055176,
54
+ "math_kl": 0.0007679596601519734,
55
+ "info_pg": 0.6024690866470337,
56
+ "info_kl": 0.0005373635212890804,
57
+ "conflict": false,
58
+ "cos_dot": 0.3637424409389496,
59
+ "grad_norm": 17.212766647338867,
60
+ "lr": 1.5e-07
61
+ },
62
+ {
63
+ "update": 5,
64
+ "math_pg": -0.020817935466766357,
65
+ "math_kl": 0.00048575906839687377,
66
+ "info_pg": 0.49411332607269287,
67
+ "info_kl": 0.00046509335516020656,
68
+ "conflict": false,
69
+ "cos_dot": 1.1652194261550903,
70
+ "grad_norm": 31.187753677368164,
71
+ "lr": 1.875e-07
72
+ },
73
+ {
74
+ "update": 6,
75
+ "math_pg": -0.35406866669654846,
76
+ "math_kl": 0.0003101894399151206,
77
+ "info_pg": 0.5011839270591736,
78
+ "info_kl": 0.0007691028295084834,
79
+ "conflict": false,
80
+ "cos_dot": 0.3225843608379364,
81
+ "grad_norm": 25.5515193939209,
82
+ "lr": 2.25e-07
83
+ },
84
+ {
85
+ "update": 7,
86
+ "math_pg": -0.35563796758651733,
87
+ "math_kl": 0.0006047477363608778,
88
+ "info_pg": 0.6523590087890625,
89
+ "info_kl": 0.001441694563254714,
90
+ "conflict": true,
91
+ "cos_dot": -0.02110724337399006,
92
+ "grad_norm": 19.03196144104004,
93
+ "lr": 2.625e-07
94
+ },
95
+ {
96
+ "update": 8,
97
+ "math_pg": -0.351636677980423,
98
+ "math_kl": 0.0006871302030049264,
99
+ "info_pg": -1.9481141567230225,
100
+ "info_kl": 0.00010081132495542988,
101
+ "conflict": true,
102
+ "cos_dot": -9.251835823059082,
103
+ "grad_norm": 569.4281616210938,
104
+ "lr": 3e-07
105
+ },
106
+ {
107
+ "update": 9,
108
+ "math_pg": -1.3936036825180054,
109
+ "math_kl": 0.0008557917608413845,
110
+ "info_pg": 0.503322422504425,
111
+ "info_kl": 0.00035435924655757844,
112
+ "conflict": true,
113
+ "cos_dot": -0.36513951420783997,
114
+ "grad_norm": 13.045738220214844,
115
+ "lr": 3e-07
116
+ },
117
+ {
118
+ "update": 10,
119
+ "math_pg": -0.35181188583374023,
120
+ "math_kl": 0.0004529313155217096,
121
+ "info_pg": 0.49829238653182983,
122
+ "info_kl": 0.0005901436670683324,
123
+ "conflict": true,
124
+ "cos_dot": -0.17399394512176514,
125
+ "grad_norm": 16.9553165435791,
126
+ "lr": 2.999540397780379e-07
127
+ },
128
+ {
129
+ "update": 11,
130
+ "math_pg": -0.3314424455165863,
131
+ "math_kl": 0.0006046635680831969,
132
+ "info_pg": 0.5906732678413391,
133
+ "info_kl": 0.0006312000914476812,
134
+ "conflict": false,
135
+ "cos_dot": 0.2825433313846588,
136
+ "grad_norm": 31.446027755737305,
137
+ "lr": 2.9981619224692805e-07
138
+ },
139
+ {
140
+ "update": 12,
141
+ "math_pg": -0.21694588661193848,
142
+ "math_kl": 0.0005262371123535559,
143
+ "info_pg": 0.5000734925270081,
144
+ "info_kl": 0.0006105701322667301,
145
+ "conflict": false,
146
+ "cos_dot": 0.45223528146743774,
147
+ "grad_norm": 17.342531204223633,
148
+ "lr": 2.9958655678711177e-07
149
+ },
150
+ {
151
+ "update": 13,
152
+ "math_pg": -0.3516656458377838,
153
+ "math_kl": 0.0004019683983642608,
154
+ "info_pg": -1.9916791915893555,
155
+ "info_kl": 0.002047985792160034,
156
+ "conflict": false,
157
+ "cos_dot": 3.7522664070129395,
158
+ "grad_norm": 138.60952758789062,
159
+ "lr": 2.9926529895304706e-07
160
+ },
161
+ {
162
+ "update": 14,
163
+ "math_pg": -0.21506553888320923,
164
+ "math_kl": 0.0006292707985267043,
165
+ "info_pg": 0.4956199824810028,
166
+ "info_kl": 0.00034475047141313553,
167
+ "conflict": true,
168
+ "cos_dot": -0.4814872741699219,
169
+ "grad_norm": 15.364961624145508,
170
+ "lr": 2.9885265035385334e-07
171
+ },
172
+ {
173
+ "update": 15,
174
+ "math_pg": -1.3724873065948486,
175
+ "math_kl": 0.0006703575782012194,
176
+ "info_pg": 0.5003576278686523,
177
+ "info_kl": 0.0004253766965121031,
178
+ "conflict": true,
179
+ "cos_dot": -0.9974016547203064,
180
+ "grad_norm": 28.29279136657715,
181
+ "lr": 2.9834890848633384e-07
182
+ },
183
+ {
184
+ "update": 16,
185
+ "math_pg": 0.84393709897995,
186
+ "math_kl": 0.00040049983363132924,
187
+ "info_pg": -1.9926856756210327,
188
+ "info_kl": 0.0004945237888023257,
189
+ "conflict": false,
190
+ "cos_dot": 2.033108711242676,
191
+ "grad_norm": 75.14515686035156,
192
+ "lr": 2.977544365204972e-07
193
+ },
194
+ {
195
+ "update": 17,
196
+ "math_pg": 0.8449388146400452,
197
+ "math_kl": 0.0003727652074303478,
198
+ "info_pg": -1.9866124391555786,
199
+ "info_kl": 0.0008818792412057519,
200
+ "conflict": false,
201
+ "cos_dot": 2.231381893157959,
202
+ "grad_norm": 67.16155242919922,
203
+ "lr": 2.9706966303773166e-07
204
+ },
205
+ {
206
+ "update": 18,
207
+ "math_pg": -1.4159849286079407,
208
+ "math_kl": 0.0008810267318040133,
209
+ "info_pg": -1.9998441934585571,
210
+ "info_kl": 0.0007015059818513691,
211
+ "conflict": false,
212
+ "cos_dot": 0.012045023031532764,
213
+ "grad_norm": 75.43449401855469,
214
+ "lr": 2.962950817218216e-07
215
+ },
216
+ {
217
+ "update": 19,
218
+ "math_pg": -0.3534339666366577,
219
+ "math_kl": 0.0008122214785544202,
220
+ "info_pg": 0.49167418479919434,
221
+ "info_kl": 0.0010795834241434932,
222
+ "conflict": false,
223
+ "cos_dot": 0.3844939172267914,
224
+ "grad_norm": 25.814733505249023,
225
+ "lr": 2.954312510030292e-07
226
+ },
227
+ {
228
+ "update": 20,
229
+ "math_pg": -0.35436299443244934,
230
+ "math_kl": 0.0008246892539318651,
231
+ "info_pg": -1.9157607555389404,
232
+ "info_kl": 0.000298593717161566,
233
+ "conflict": false,
234
+ "cos_dot": 0.4496350586414337,
235
+ "grad_norm": 73.1011962890625,
236
+ "lr": 2.944787936554967e-07
237
+ },
238
+ {
239
+ "update": 21,
240
+ "math_pg": -0.35409772396087646,
241
+ "math_kl": 0.0005890416505280882,
242
+ "info_pg": -2.0084068775177,
243
+ "info_kl": 0.0018628461984917521,
244
+ "conflict": false,
245
+ "cos_dot": 0.7556353807449341,
246
+ "grad_norm": 36.533634185791016,
247
+ "lr": 2.93438396348261e-07
248
+ },
249
+ {
250
+ "update": 22,
251
+ "math_pg": -0.3519936501979828,
252
+ "math_kl": 0.0007213769713416696,
253
+ "info_pg": -1.9961791038513184,
254
+ "info_kl": 0.0011058369418606162,
255
+ "conflict": true,
256
+ "cos_dot": -5.899958610534668,
257
+ "grad_norm": 156.04544067382812,
258
+ "lr": 2.923108091502033e-07
259
+ },
260
+ {
261
+ "update": 23,
262
+ "math_pg": -0.3462774157524109,
263
+ "math_kl": 0.0009997647139243782,
264
+ "info_pg": -2.0173258781433105,
265
+ "info_kl": 0.001874180044978857,
266
+ "conflict": false,
267
+ "cos_dot": 0.709530770778656,
268
+ "grad_norm": 58.112430572509766,
269
+ "lr": 2.9109684498929064e-07
270
+ },
271
+ {
272
+ "update": 24,
273
+ "math_pg": -1.1982656121253967,
274
+ "math_kl": 0.0005401678936323151,
275
+ "info_pg": 0.5816492438316345,
276
+ "info_kl": 0.0006900060107000172,
277
+ "conflict": false,
278
+ "cos_dot": 0.39639952778816223,
279
+ "grad_norm": 17.205760955810547,
280
+ "lr": 2.8979737906650006e-07
281
+ },
282
+ {
283
+ "update": 25,
284
+ "math_pg": -1.4126299023628235,
285
+ "math_kl": 0.0005536000244319439,
286
+ "info_pg": 0.5042392611503601,
287
+ "info_kl": 0.0006763922865502536,
288
+ "conflict": false,
289
+ "cos_dot": 0.19436289370059967,
290
+ "grad_norm": 24.87588119506836,
291
+ "lr": 2.8841334822484727e-07
292
+ },
293
+ {
294
+ "update": 26,
295
+ "math_pg": 0.7083474397659302,
296
+ "math_kl": 0.0004929077113047242,
297
+ "info_pg": -1.9827852249145508,
298
+ "info_kl": 0.00044063234236091375,
299
+ "conflict": false,
300
+ "cos_dot": 3.6086432933807373,
301
+ "grad_norm": 72.60540008544922,
302
+ "lr": 2.869457502739743e-07
303
+ },
304
+ {
305
+ "update": 27,
306
+ "math_pg": 0.8433120548725128,
307
+ "math_kl": 0.00040260852983919904,
308
+ "info_pg": 0.8501220941543579,
309
+ "info_kl": 0.0028083028737455606,
310
+ "conflict": false,
311
+ "cos_dot": 0.6576541066169739,
312
+ "grad_norm": 29.20465850830078,
313
+ "lr": 2.8539564327078423e-07
314
+ },
315
+ {
316
+ "update": 28,
317
+ "math_pg": -1.398414433002472,
318
+ "math_kl": 0.0009098303853534162,
319
+ "info_pg": 0.6001955270767212,
320
+ "info_kl": 0.0009855404496192932,
321
+ "conflict": true,
322
+ "cos_dot": -0.8426816463470459,
323
+ "grad_norm": 28.01599884033203,
324
+ "lr": 2.837641447566408e-07
325
+ },
326
+ {
327
+ "update": 29,
328
+ "math_pg": -0.35457849502563477,
329
+ "math_kl": 0.0007221649138955399,
330
+ "info_pg": -1.964111089706421,
331
+ "info_kl": 0.0011894062627106905,
332
+ "conflict": false,
333
+ "cos_dot": 0.020252790302038193,
334
+ "grad_norm": 66.01853942871094,
335
+ "lr": 2.8205243095168264e-07
336
+ },
337
+ {
338
+ "update": 30,
339
+ "math_pg": 1.0696012079715729,
340
+ "math_kl": 0.0005223805856076069,
341
+ "info_pg": 0.8586828708648682,
342
+ "info_kl": 0.0006360396509990096,
343
+ "conflict": false,
344
+ "cos_dot": 0.1707495003938675,
345
+ "grad_norm": 22.324607849121094,
346
+ "lr": 2.8026173590683383e-07
347
+ },
348
+ {
349
+ "update": 31,
350
+ "math_pg": 0.8441479504108429,
351
+ "math_kl": 0.0003469506336841732,
352
+ "info_pg": 0.5130608677864075,
353
+ "info_kl": 0.00047360575990751386,
354
+ "conflict": false,
355
+ "cos_dot": 0.44103577733039856,
356
+ "grad_norm": 36.234214782714844,
357
+ "lr": 2.783933506141215e-07
358
+ },
359
+ {
360
+ "update": 32,
361
+ "math_pg": 1.0623615384101868,
362
+ "math_kl": 0.00035507082066033036,
363
+ "info_pg": 0.7313033938407898,
364
+ "info_kl": 0.0002846356073860079,
365
+ "conflict": true,
366
+ "cos_dot": -0.5288285613059998,
367
+ "grad_norm": 31.93411636352539,
368
+ "lr": 2.764486220759419e-07
369
+ },
370
+ {
371
+ "update": 33,
372
+ "math_pg": -0.35352638363838196,
373
+ "math_kl": 0.0004669608024414629,
374
+ "info_pg": 0.519494891166687,
375
+ "info_kl": 0.0014579734997823834,
376
+ "conflict": false,
377
+ "cos_dot": 0.011693264357745647,
378
+ "grad_norm": 108.5232925415039,
379
+ "lr": 2.744289523339464e-07
380
+ },
381
+ {
382
+ "update": 34,
383
+ "math_pg": -0.21674281358718872,
384
+ "math_kl": 0.0005225432978477329,
385
+ "info_pg": 0.4997045397758484,
386
+ "info_kl": 0.0005960784619674087,
387
+ "conflict": false,
388
+ "cos_dot": 0.59102463722229,
389
+ "grad_norm": 21.525070190429688,
390
+ "lr": 2.723357974582468e-07
391
+ },
392
+ {
393
+ "update": 35,
394
+ "math_pg": -0.35195502638816833,
395
+ "math_kl": 0.000507119155372493,
396
+ "info_pg": -1.9802346229553223,
397
+ "info_kl": 0.0009624784579500556,
398
+ "conflict": true,
399
+ "cos_dot": -2.3963537216186523,
400
+ "grad_norm": 182.28607177734375,
401
+ "lr": 2.701706664976697e-07
402
+ },
403
+ {
404
+ "update": 36,
405
+ "math_pg": -0.35456258058547974,
406
+ "math_kl": 0.0005204900226090103,
407
+ "info_pg": -2.0107288360595703,
408
+ "info_kl": 0.0006918454309925437,
409
+ "conflict": false,
410
+ "cos_dot": 0.8455079793930054,
411
+ "grad_norm": 80.75829315185547,
412
+ "lr": 2.6793512039181535e-07
413
+ },
414
+ {
415
+ "update": 37,
416
+ "math_pg": -0.35494887828826904,
417
+ "math_kl": 0.0005331943975761533,
418
+ "info_pg": -2.024609088897705,
419
+ "info_kl": 0.000992915011011064,
420
+ "conflict": false,
421
+ "cos_dot": 1.3504447937011719,
422
+ "grad_norm": 212.3436737060547,
423
+ "lr": 2.6563077084570715e-07
424
+ },
425
+ {
426
+ "update": 38,
427
+ "math_pg": 0.8439383506774902,
428
+ "math_kl": 0.0002651476679602638,
429
+ "info_pg": 0.7607088088989258,
430
+ "info_kl": 0.000789453973993659,
431
+ "conflict": false,
432
+ "cos_dot": 0.8688093423843384,
433
+ "grad_norm": 32.27117919921875,
434
+ "lr": 2.632592791678409e-07
435
+ },
436
+ {
437
+ "update": 39,
438
+ "math_pg": -0.3553324043750763,
439
+ "math_kl": 0.00044912427256349474,
440
+ "info_pg": -1.9010255336761475,
441
+ "info_kl": 0.0010899155167862773,
442
+ "conflict": false,
443
+ "cos_dot": 2.832444429397583,
444
+ "grad_norm": 91.93256378173828,
445
+ "lr": 2.6082235507247393e-07
446
+ },
447
+ {
448
+ "update": 40,
449
+ "math_pg": -0.3365023136138916,
450
+ "math_kl": 0.00042378922807984054,
451
+ "info_pg": 0.5034157633781433,
452
+ "info_kl": 0.0017099202377721667,
453
+ "conflict": false,
454
+ "cos_dot": 0.11300937831401825,
455
+ "grad_norm": 28.836551666259766,
456
+ "lr": 2.583217554470152e-07
457
+ },
458
+ {
459
+ "update": 41,
460
+ "math_pg": -1.4112279415130615,
461
+ "math_kl": 0.0008031792822293937,
462
+ "info_pg": -2.055854320526123,
463
+ "info_kl": 0.004961743950843811,
464
+ "conflict": false,
465
+ "cos_dot": 2.5167598724365234,
466
+ "grad_norm": 267.05828857421875,
467
+ "lr": 2.557592830854071e-07
468
+ },
469
+ {
470
+ "update": 42,
471
+ "math_pg": -0.3527606427669525,
472
+ "math_kl": 0.0005684931529685855,
473
+ "info_pg": -2.0148816108703613,
474
+ "info_kl": 0.0005933092907071114,
475
+ "conflict": false,
476
+ "cos_dot": 0.2901570200920105,
477
+ "grad_norm": 46.17240524291992,
478
+ "lr": 2.5313678538841054e-07
479
+ },
480
+ {
481
+ "update": 43,
482
+ "math_pg": -0.35361573100090027,
483
+ "math_kl": 0.000454138993518427,
484
+ "info_pg": -2.050567388534546,
485
+ "info_kl": 0.00426877848803997,
486
+ "conflict": true,
487
+ "cos_dot": -0.8272485733032227,
488
+ "grad_norm": 249.3991241455078,
489
+ "lr": 2.5045615303173155e-07
490
+ },
491
+ {
492
+ "update": 44,
493
+ "math_pg": -1.399952232837677,
494
+ "math_kl": 0.001077802269719541,
495
+ "info_pg": -2.0122785568237305,
496
+ "info_kl": 0.0007408169913105667,
497
+ "conflict": false,
498
+ "cos_dot": 1.8629937171936035,
499
+ "grad_norm": 198.93035888671875,
500
+ "lr": 2.477193186029482e-07
501
+ },
502
+ {
503
+ "update": 45,
504
+ "math_pg": -1.3931199312210083,
505
+ "math_kl": 0.0006107337540015578,
506
+ "info_pg": 0.8710179328918457,
507
+ "info_kl": 0.0014073881320655346,
508
+ "conflict": false,
509
+ "cos_dot": 1.057770848274231,
510
+ "grad_norm": 48.98581314086914,
511
+ "lr": 2.449282552082224e-07
512
+ },
513
+ {
514
+ "update": 46,
515
+ "math_pg": -1.413301706314087,
516
+ "math_kl": 0.0008238591544795781,
517
+ "info_pg": -2.0439000129699707,
518
+ "info_kl": 0.0025257533416152,
519
+ "conflict": true,
520
+ "cos_dot": -1.454201102256775,
521
+ "grad_norm": 175.6990203857422,
522
+ "lr": 2.4208497504979935e-07
523
+ },
524
+ {
525
+ "update": 47,
526
+ "math_pg": -1.413575828075409,
527
+ "math_kl": 0.0002799725698423572,
528
+ "info_pg": -1.9826511144638062,
529
+ "info_kl": 0.00046889917575754225,
530
+ "conflict": false,
531
+ "cos_dot": 1.5837894678115845,
532
+ "grad_norm": 73.69894409179688,
533
+ "lr": 2.391915279753211e-07
534
+ },
535
+ {
536
+ "update": 48,
537
+ "math_pg": -0.35528507828712463,
538
+ "math_kl": 0.0004670460621127859,
539
+ "info_pg": 0.4998255968093872,
540
+ "info_kl": 0.0003060665330849588,
541
+ "conflict": false,
542
+ "cos_dot": 0.3493826985359192,
543
+ "grad_norm": 14.255271911621094,
544
+ "lr": 2.3625e-07
545
+ },
546
+ {
547
+ "update": 49,
548
+ "math_pg": -0.0009347200393676758,
549
+ "math_kl": 0.0003360901027917862,
550
+ "info_pg": 0.5128419399261475,
551
+ "info_kl": 0.0029443702660501003,
552
+ "conflict": true,
553
+ "cos_dot": -0.23802708089351654,
554
+ "grad_norm": 52.6982307434082,
555
+ "lr": 2.3326251180271673e-07
556
+ },
557
+ {
558
+ "update": 50,
559
+ "math_pg": 0.7067213952541351,
560
+ "math_kl": 0.00028348278283374384,
561
+ "info_pg": -2.0147366523742676,
562
+ "info_kl": 0.00124279351439327,
563
+ "conflict": false,
564
+ "cos_dot": 2.4110872745513916,
565
+ "grad_norm": 82.09896087646484,
566
+ "lr": 2.3023121719712877e-07
567
+ },
568
+ {
569
+ "update": 51,
570
+ "math_pg": -0.21675291657447815,
571
+ "math_kl": 0.0004914215387543663,
572
+ "info_pg": 0.6852624416351318,
573
+ "info_kl": 0.00031384528847411275,
574
+ "conflict": true,
575
+ "cos_dot": -0.09636291861534119,
576
+ "grad_norm": 49.133567810058594,
577
+ "lr": 2.271583015788894e-07
578
+ },
579
+ {
580
+ "update": 52,
581
+ "math_pg": 0.844824880361557,
582
+ "math_kl": 0.00041445178794674575,
583
+ "info_pg": 0.7044641375541687,
584
+ "info_kl": 0.0027901604771614075,
585
+ "conflict": true,
586
+ "cos_dot": -0.19440996646881104,
587
+ "grad_norm": 61.585411071777344,
588
+ "lr": 2.240459803500989e-07
589
+ },
590
+ {
591
+ "update": 53,
592
+ "math_pg": 0.7068532109260559,
593
+ "math_kl": 0.00028252354968572035,
594
+ "info_pg": 0.8017353415489197,
595
+ "info_kl": 0.0005963122239336371,
596
+ "conflict": false,
597
+ "cos_dot": 0.11658605933189392,
598
+ "grad_norm": 9.532983779907227,
599
+ "lr": 2.208964973221218e-07
600
+ },
601
+ {
602
+ "update": 54,
603
+ "math_pg": -0.3333965837955475,
604
+ "math_kl": 0.0005686318254447542,
605
+ "info_pg": -1.987859845161438,
606
+ "info_kl": 0.0007142906542867422,
607
+ "conflict": false,
608
+ "cos_dot": 0.8052766919136047,
609
+ "grad_norm": 142.16346740722656,
610
+ "lr": 2.1771212309792326e-07
611
+ },
612
+ {
613
+ "update": 55,
614
+ "math_pg": -1.4141443371772766,
615
+ "math_kl": 0.00042046300222864375,
616
+ "info_pg": 0.5043144226074219,
617
+ "info_kl": 0.003299224656075239,
618
+ "conflict": false,
619
+ "cos_dot": 1.1584422588348389,
620
+ "grad_norm": 38.20326614379883,
621
+ "lr": 2.1449515343509014e-07
622
+ },
623
+ {
624
+ "update": 56,
625
+ "math_pg": 0.706267774105072,
626
+ "math_kl": 0.0002570558135630563,
627
+ "info_pg": 0.5038169622421265,
628
+ "info_kl": 0.0002705130900721997,
629
+ "conflict": false,
630
+ "cos_dot": 0.00986660085618496,
631
+ "grad_norm": 28.509937286376953,
632
+ "lr": 2.1124790759071646e-07
633
+ },
634
+ {
635
+ "update": 57,
636
+ "math_pg": -0.3504635691642761,
637
+ "math_kl": 0.0005283812060952187,
638
+ "info_pg": -2.0043039321899414,
639
+ "info_kl": 0.001836988958530128,
640
+ "conflict": false,
641
+ "cos_dot": 0.7231704592704773,
642
+ "grad_norm": 124.19365692138672,
643
+ "lr": 2.079727266493477e-07
644
+ },
645
+ {
646
+ "update": 58,
647
+ "math_pg": -1.0381224751472473,
648
+ "math_kl": 0.0008741390774957836,
649
+ "info_pg": -1.9399632215499878,
650
+ "info_kl": 0.0010055579477921128,
651
+ "conflict": true,
652
+ "cos_dot": -0.6529443264007568,
653
+ "grad_norm": 135.59664916992188,
654
+ "lr": 2.046719718351883e-07
655
+ },
656
+ {
657
+ "update": 59,
658
+ "math_pg": -0.35290685296058655,
659
+ "math_kl": 0.00047231429925886914,
660
+ "info_pg": 0.5019758939743042,
661
+ "info_kl": 0.000835469807498157,
662
+ "conflict": true,
663
+ "cos_dot": -0.5326172113418579,
664
+ "grad_norm": 44.98362731933594,
665
+ "lr": 2.0134802280978953e-07
666
+ },
667
+ {
668
+ "update": 60,
669
+ "math_pg": 0.844897985458374,
670
+ "math_kl": 0.00030389138555619866,
671
+ "info_pg": 0.4953189790248871,
672
+ "info_kl": 0.0004451630520634353,
673
+ "conflict": false,
674
+ "cos_dot": 1.0321533679962158,
675
+ "grad_norm": 43.91508865356445,
676
+ "lr": 1.9800327595644567e-07
677
+ },
678
+ {
679
+ "update": 61,
680
+ "math_pg": -0.3346695005893707,
681
+ "math_kl": 0.0006728900625603274,
682
+ "info_pg": 0.4955896735191345,
683
+ "info_kl": 0.0005714573198929429,
684
+ "conflict": true,
685
+ "cos_dot": -0.37378668785095215,
686
+ "grad_norm": 27.225486755371094,
687
+ "lr": 1.946401426525336e-07
688
+ },
689
+ {
690
+ "update": 62,
691
+ "math_pg": 1.174721598625183,
692
+ "math_kl": 0.0005097126850159839,
693
+ "info_pg": -1.8735204935073853,
694
+ "info_kl": 0.000989772961474955,
695
+ "conflict": false,
696
+ "cos_dot": 2.935145616531372,
697
+ "grad_norm": 149.9563446044922,
698
+ "lr": 1.9126104753104378e-07
699
+ },
700
+ {
701
+ "update": 63,
702
+ "math_pg": -0.143155038356781,
703
+ "math_kl": 0.0007695616950513795,
704
+ "info_pg": 0.5903162956237793,
705
+ "info_kl": 0.0013044251827523112,
706
+ "conflict": false,
707
+ "cos_dot": 1.0455175638198853,
708
+ "grad_norm": 47.751583099365234,
709
+ "lr": 1.878684267325537e-07
710
+ },
711
+ {
712
+ "update": 64,
713
+ "math_pg": -1.3990607857704163,
714
+ "math_kl": 0.0011470018071122468,
715
+ "info_pg": 0.49906235933303833,
716
+ "info_kl": 0.0005691951955668628,
717
+ "conflict": false,
718
+ "cos_dot": 0.10516241937875748,
719
+ "grad_norm": 63.60158920288086,
720
+ "lr": 1.8446472614890536e-07
721
+ },
722
+ {
723
+ "update": 65,
724
+ "math_pg": -0.2169962227344513,
725
+ "math_kl": 0.0004642999265342951,
726
+ "info_pg": 0.4983784258365631,
727
+ "info_kl": 0.0015330531168729067,
728
+ "conflict": false,
729
+ "cos_dot": 0.22806529700756073,
730
+ "grad_norm": 21.154802322387695,
731
+ "lr": 1.810523996598528e-07
732
+ },
733
+ {
734
+ "update": 66,
735
+ "math_pg": -0.3531833291053772,
736
+ "math_kl": 0.00029802579956594855,
737
+ "info_pg": -2.0290822982788086,
738
+ "info_kl": 0.004916084930300713,
739
+ "conflict": false,
740
+ "cos_dot": 2.748922348022461,
741
+ "grad_norm": 133.350830078125,
742
+ "lr": 1.776339073639504e-07
743
+ },
744
+ {
745
+ "update": 67,
746
+ "math_pg": 0.7071178257465363,
747
+ "math_kl": 0.00038849063275847584,
748
+ "info_pg": -1.9958614110946655,
749
+ "info_kl": 0.0022443837951868773,
750
+ "conflict": false,
751
+ "cos_dot": 4.758065700531006,
752
+ "grad_norm": 139.9806671142578,
753
+ "lr": 1.7421171380495816e-07
754
+ },
755
+ {
756
+ "update": 68,
757
+ "math_pg": -0.3518122732639313,
758
+ "math_kl": 0.0006191963038872927,
759
+ "info_pg": -1.9818499088287354,
760
+ "info_kl": 0.001085887080989778,
761
+ "conflict": false,
762
+ "cos_dot": 4.365755558013916,
763
+ "grad_norm": 82.95382690429688,
764
+ "lr": 1.7078828619504182e-07
765
+ },
766
+ {
767
+ "update": 69,
768
+ "math_pg": -0.33516058325767517,
769
+ "math_kl": 0.0006429424684029073,
770
+ "info_pg": -2.0100555419921875,
771
+ "info_kl": 0.005229127127677202,
772
+ "conflict": true,
773
+ "cos_dot": -2.2270474433898926,
774
+ "grad_norm": 144.986328125,
775
+ "lr": 1.673660926360496e-07
776
+ },
777
+ {
778
+ "update": 70,
779
+ "math_pg": -0.2207457721233368,
780
+ "math_kl": 0.0006097159930504858,
781
+ "info_pg": -2.0236034393310547,
782
+ "info_kl": 0.0009856321848928928,
783
+ "conflict": false,
784
+ "cos_dot": 2.2895517349243164,
785
+ "grad_norm": 129.53277587890625,
786
+ "lr": 1.639476003401472e-07
787
+ },
788
+ {
789
+ "update": 71,
790
+ "math_pg": 0.7076331377029419,
791
+ "math_kl": 0.0003346075100125745,
792
+ "info_pg": -2.001629590988159,
793
+ "info_kl": 0.001025490928441286,
794
+ "conflict": true,
795
+ "cos_dot": -4.607853889465332,
796
+ "grad_norm": 250.16641235351562,
797
+ "lr": 1.6053527385109464e-07
798
+ },
799
+ {
800
+ "update": 72,
801
+ "math_pg": 0.7069272994995117,
802
+ "math_kl": 0.00031706265144748613,
803
+ "info_pg": -2.0206899642944336,
804
+ "info_kl": 0.0010109726572409272,
805
+ "conflict": true,
806
+ "cos_dot": -1.4142712354660034,
807
+ "grad_norm": 60.06268310546875,
808
+ "lr": 1.571315732674463e-07
809
+ },
810
+ {
811
+ "update": 73,
812
+ "math_pg": -1.4164837002754211,
813
+ "math_kl": 0.0005633192195091397,
814
+ "info_pg": 0.4997780919075012,
815
+ "info_kl": 0.0026831747964024544,
816
+ "conflict": true,
817
+ "cos_dot": -0.3786667287349701,
818
+ "grad_norm": 21.893260955810547,
819
+ "lr": 1.5373895246895617e-07
820
+ },
821
+ {
822
+ "update": 74,
823
+ "math_pg": -0.3554006814956665,
824
+ "math_kl": 0.0005086237797513604,
825
+ "info_pg": 0.5048808455467224,
826
+ "info_kl": 0.00039290383574552834,
827
+ "conflict": false,
828
+ "cos_dot": 0.027060039341449738,
829
+ "grad_norm": 9.763389587402344,
830
+ "lr": 1.503598573474664e-07
831
+ },
832
+ {
833
+ "update": 75,
834
+ "math_pg": -1.4157336950302124,
835
+ "math_kl": 0.000499681307701394,
836
+ "info_pg": -1.8622864484786987,
837
+ "info_kl": 0.0009393274085596204,
838
+ "conflict": false,
839
+ "cos_dot": 2.0534262657165527,
840
+ "grad_norm": 94.29167175292969,
841
+ "lr": 1.4699672404355434e-07
842
+ },
843
+ {
844
+ "update": 76,
845
+ "math_pg": -1.4002699851989746,
846
+ "math_kl": 0.000864204514073208,
847
+ "info_pg": -2.003434896469116,
848
+ "info_kl": 0.0012200098717585206,
849
+ "conflict": true,
850
+ "cos_dot": -0.5801063776016235,
851
+ "grad_norm": 84.75376892089844,
852
+ "lr": 1.4365197719021045e-07
853
+ },
854
+ {
855
+ "update": 77,
856
+ "math_pg": -0.02031916379928589,
857
+ "math_kl": 0.0005882248806301504,
858
+ "info_pg": 0.8518873453140259,
859
+ "info_kl": 0.000349417474353686,
860
+ "conflict": false,
861
+ "cos_dot": 0.49166345596313477,
862
+ "grad_norm": 15.659280776977539,
863
+ "lr": 1.403280281648117e-07
864
+ },
865
+ {
866
+ "update": 78,
867
+ "math_pg": -0.35432207584381104,
868
+ "math_kl": 0.0002840360102709383,
869
+ "info_pg": -1.8767015933990479,
870
+ "info_kl": 0.0018594146240502596,
871
+ "conflict": true,
872
+ "cos_dot": -1.6519439220428467,
873
+ "grad_norm": 445.5745544433594,
874
+ "lr": 1.3702727335065228e-07
875
+ },
876
+ {
877
+ "update": 79,
878
+ "math_pg": -0.35593679547309875,
879
+ "math_kl": 0.0007267271139426157,
880
+ "info_pg": -1.9401602745056152,
881
+ "info_kl": 0.002190536353737116,
882
+ "conflict": false,
883
+ "cos_dot": 1.5718234777450562,
884
+ "grad_norm": 146.9812774658203,
885
+ "lr": 1.3375209240928352e-07
886
+ },
887
+ {
888
+ "update": 80,
889
+ "math_pg": -0.3570924401283264,
890
+ "math_kl": 0.0008298760221805423,
891
+ "info_pg": -1.991286039352417,
892
+ "info_kl": 0.0014814657624810934,
893
+ "conflict": true,
894
+ "cos_dot": -4.118163108825684,
895
+ "grad_norm": 50.51897048950195,
896
+ "lr": 1.3050484656490984e-07
897
+ },
898
+ {
899
+ "update": 81,
900
+ "math_pg": -1.3957706689834595,
901
+ "math_kl": 0.0009473387035541236,
902
+ "info_pg": -1.9323502779006958,
903
+ "info_kl": 0.0037279108073562384,
904
+ "conflict": true,
905
+ "cos_dot": -6.036247730255127,
906
+ "grad_norm": 356.2556457519531,
907
+ "lr": 1.2728787690207672e-07
908
+ },
909
+ {
910
+ "update": 82,
911
+ "math_pg": -0.3538176119327545,
912
+ "math_kl": 0.0004930246213916689,
913
+ "info_pg": -1.9800235033035278,
914
+ "info_kl": 0.0005016137147322297,
915
+ "conflict": false,
916
+ "cos_dot": 4.5368123054504395,
917
+ "grad_norm": 82.59356689453125,
918
+ "lr": 1.2410350267787822e-07
919
+ },
920
+ {
921
+ "update": 83,
922
+ "math_pg": -0.3355676829814911,
923
+ "math_kl": 0.0007027465617284179,
924
+ "info_pg": -2.0288822650909424,
925
+ "info_kl": 0.000736443733330816,
926
+ "conflict": false,
927
+ "cos_dot": 0.19580082595348358,
928
+ "grad_norm": 129.6429443359375,
929
+ "lr": 1.209540196499011e-07
930
+ },
931
+ {
932
+ "update": 84,
933
+ "math_pg": -0.2171933352947235,
934
+ "math_kl": 0.0004248647455824539,
935
+ "info_pg": -1.9413702487945557,
936
+ "info_kl": 0.002054561860859394,
937
+ "conflict": false,
938
+ "cos_dot": 2.000851631164551,
939
+ "grad_norm": 389.7947082519531,
940
+ "lr": 1.1784169842111063e-07
941
+ },
942
+ {
943
+ "update": 85,
944
+ "math_pg": -0.3549956679344177,
945
+ "math_kl": 0.0004383653576951474,
946
+ "info_pg": 0.9000182747840881,
947
+ "info_kl": 0.0011344983940944076,
948
+ "conflict": false,
949
+ "cos_dot": 0.228316992521286,
950
+ "grad_norm": 27.502059936523438,
951
+ "lr": 1.1476878280287126e-07
952
+ },
953
+ {
954
+ "update": 86,
955
+ "math_pg": -0.21836236119270325,
956
+ "math_kl": 0.0007169026066549122,
957
+ "info_pg": 0.4954044222831726,
958
+ "info_kl": 0.0013972355518490076,
959
+ "conflict": true,
960
+ "cos_dot": -1.1419477462768555,
961
+ "grad_norm": 18.96268653869629,
962
+ "lr": 1.1173748819728322e-07
963
+ },
964
+ {
965
+ "update": 87,
966
+ "math_pg": -1.3769994378089905,
967
+ "math_kl": 0.0006275597552303225,
968
+ "info_pg": 0.5858305096626282,
969
+ "info_kl": 0.0006182853248901665,
970
+ "conflict": false,
971
+ "cos_dot": 0.22649362683296204,
972
+ "grad_norm": 15.290396690368652,
973
+ "lr": 1.0875e-07
974
+ },
975
+ {
976
+ "update": 88,
977
+ "math_pg": 0.8436494767665863,
978
+ "math_kl": 0.000336490644258447,
979
+ "info_pg": 0.505917489528656,
980
+ "info_kl": 0.0006245188415050507,
981
+ "conflict": false,
982
+ "cos_dot": 0.7073910236358643,
983
+ "grad_norm": 17.559791564941406,
984
+ "lr": 1.0580847202467887e-07
985
+ },
986
+ {
987
+ "update": 89,
988
+ "math_pg": 0.8442975580692291,
989
+ "math_kl": 0.0003102580376435071,
990
+ "info_pg": 0.5221441388130188,
991
+ "info_kl": 0.0018197218887507915,
992
+ "conflict": false,
993
+ "cos_dot": 0.1648477464914322,
994
+ "grad_norm": 13.363994598388672,
995
+ "lr": 1.0291502495020066e-07
996
+ },
997
+ {
998
+ "update": 90,
999
+ "math_pg": -1.4144253730773926,
1000
+ "math_kl": 0.0009068460785783827,
1001
+ "info_pg": -2.0117101669311523,
1002
+ "info_kl": 0.0010430136462673545,
1003
+ "conflict": false,
1004
+ "cos_dot": 1.5450851917266846,
1005
+ "grad_norm": 255.55413818359375,
1006
+ "lr": 1.0007174479177764e-07
1007
+ },
1008
+ {
1009
+ "update": 91,
1010
+ "math_pg": -0.3582771122455597,
1011
+ "math_kl": 0.0008150976791512221,
1012
+ "info_pg": -1.9988183975219727,
1013
+ "info_kl": 0.0013158313231542706,
1014
+ "conflict": false,
1015
+ "cos_dot": 1.4043487310409546,
1016
+ "grad_norm": 99.26768493652344,
1017
+ "lr": 9.728068139705186e-08
1018
+ },
1019
+ {
1020
+ "update": 92,
1021
+ "math_pg": -0.35473671555519104,
1022
+ "math_kl": 0.0007046105456538498,
1023
+ "info_pg": -2.0266201496124268,
1024
+ "info_kl": 0.0016591703752055764,
1025
+ "conflict": true,
1026
+ "cos_dot": -0.21418018639087677,
1027
+ "grad_norm": 64.37957000732422,
1028
+ "lr": 9.454384696826843e-08
1029
+ },
1030
+ {
1031
+ "update": 93,
1032
+ "math_pg": -0.35462045669555664,
1033
+ "math_kl": 0.0005181815504329279,
1034
+ "info_pg": -1.9866762161254883,
1035
+ "info_kl": 0.0012570773251354694,
1036
+ "conflict": false,
1037
+ "cos_dot": 0.5214288830757141,
1038
+ "grad_norm": 90.71595764160156,
1039
+ "lr": 9.186321461158941e-08
1040
+ },
1041
+ {
1042
+ "update": 94,
1043
+ "math_pg": -0.3529442548751831,
1044
+ "math_kl": 0.0005923639255343005,
1045
+ "info_pg": 0.5814455151557922,
1046
+ "info_kl": 0.0013405797071754932,
1047
+ "conflict": false,
1048
+ "cos_dot": 0.8188340663909912,
1049
+ "grad_norm": 21.23854637145996,
1050
+ "lr": 8.924071691459288e-08
1051
+ },
1052
+ {
1053
+ "update": 95,
1054
+ "math_pg": -0.34893208742141724,
1055
+ "math_kl": 0.0005561230209423229,
1056
+ "info_pg": 0.7646831274032593,
1057
+ "info_kl": 0.00035598085378296673,
1058
+ "conflict": false,
1059
+ "cos_dot": 0.44678640365600586,
1060
+ "grad_norm": 25.49726676940918,
1061
+ "lr": 8.667824455298483e-08
1062
+ },
1063
+ {
1064
+ "update": 96,
1065
+ "math_pg": -1.2010241746902466,
1066
+ "math_kl": 0.0005347385013010353,
1067
+ "info_pg": 0.49213749170303345,
1068
+ "info_kl": 0.0011111663188785315,
1069
+ "conflict": true,
1070
+ "cos_dot": -0.1073569655418396,
1071
+ "grad_norm": 14.688800811767578,
1072
+ "lr": 8.417764492752611e-08
1073
+ },
1074
+ {
1075
+ "update": 97,
1076
+ "math_pg": -1.4169167876243591,
1077
+ "math_kl": 0.0006012196536175907,
1078
+ "info_pg": -1.9376145601272583,
1079
+ "info_kl": 0.0014446377754211426,
1080
+ "conflict": true,
1081
+ "cos_dot": -6.530718803405762,
1082
+ "grad_norm": 353.1114501953125,
1083
+ "lr": 8.174072083215908e-08
1084
+ },
1085
+ {
1086
+ "update": 98,
1087
+ "math_pg": 0.7074518799781799,
1088
+ "math_kl": 0.00030174081621225923,
1089
+ "info_pg": -2.001893997192383,
1090
+ "info_kl": 0.00041985948337242007,
1091
+ "conflict": true,
1092
+ "cos_dot": -4.359536647796631,
1093
+ "grad_norm": 91.67293548583984,
1094
+ "lr": 7.936922915429285e-08
1095
+ },
1096
+ {
1097
+ "update": 99,
1098
+ "math_pg": 0.8427177965641022,
1099
+ "math_kl": 0.00023393851006403565,
1100
+ "info_pg": -1.956587791442871,
1101
+ "info_kl": 0.0006078661535866559,
1102
+ "conflict": true,
1103
+ "cos_dot": -24.04313850402832,
1104
+ "grad_norm": 426.7361145019531,
1105
+ "lr": 7.706487960818459e-08
1106
+ },
1107
+ {
1108
+ "update": 100,
1109
+ "math_pg": -1.4004024267196655,
1110
+ "math_kl": 0.001008993509458378,
1111
+ "info_pg": 0.4988733232021332,
1112
+ "info_kl": 8.740524936001748e-05,
1113
+ "conflict": true,
1114
+ "cos_dot": -0.21304668486118317,
1115
+ "grad_norm": 23.444772720336914,
1116
+ "lr": 7.482933350233031e-08
1117
+ },
1118
+ {
1119
+ "update": 101,
1120
+ "math_pg": -0.36065253615379333,
1121
+ "math_kl": 0.0007438795582856983,
1122
+ "info_pg": 0.5021178722381592,
1123
+ "info_kl": 0.0026513838674873114,
1124
+ "conflict": true,
1125
+ "cos_dot": -1.6447150707244873,
1126
+ "grad_norm": 61.75741958618164,
1127
+ "lr": 7.266420254175317e-08
1128
+ },
1129
+ {
1130
+ "update": 102,
1131
+ "math_pg": 1.0589343011379242,
1132
+ "math_kl": 0.0005180509760975838,
1133
+ "info_pg": 0.6590686440467834,
1134
+ "info_kl": 0.0005759750492870808,
1135
+ "conflict": true,
1136
+ "cos_dot": -0.46784743666648865,
1137
+ "grad_norm": 24.019821166992188,
1138
+ "lr": 7.057104766605363e-08
1139
+ },
1140
+ {
1141
+ "update": 103,
1142
+ "math_pg": 0.8438460528850555,
1143
+ "math_kl": 0.00028825862682424486,
1144
+ "info_pg": 0.485715389251709,
1145
+ "info_kl": 0.0014983154833316803,
1146
+ "conflict": false,
1147
+ "cos_dot": 1.2194383144378662,
1148
+ "grad_norm": 13.870468139648438,
1149
+ "lr": 6.85513779240581e-08
1150
+ },
1151
+ {
1152
+ "update": 104,
1153
+ "math_pg": 1.0586774051189423,
1154
+ "math_kl": 0.0002850003838830162,
1155
+ "info_pg": 0.6773384213447571,
1156
+ "info_kl": 0.0013752492377534509,
1157
+ "conflict": false,
1158
+ "cos_dot": 0.4305083751678467,
1159
+ "grad_norm": 20.46723747253418,
1160
+ "lr": 6.66066493858785e-08
1161
+ },
1162
+ {
1163
+ "update": 105,
1164
+ "math_pg": -0.3530276417732239,
1165
+ "math_kl": 0.000495817672344856,
1166
+ "info_pg": -2.0334830284118652,
1167
+ "info_kl": 0.0010761487064883113,
1168
+ "conflict": true,
1169
+ "cos_dot": -1.5735609531402588,
1170
+ "grad_norm": 100.43695831298828,
1171
+ "lr": 6.473826409316618e-08
1172
+ },
1173
+ {
1174
+ "update": 106,
1175
+ "math_pg": -0.21759682893753052,
1176
+ "math_kl": 0.0005268521053949371,
1177
+ "info_pg": -1.99626886844635,
1178
+ "info_kl": 0.004420326091349125,
1179
+ "conflict": true,
1180
+ "cos_dot": -2.1750681400299072,
1181
+ "grad_norm": 283.7276306152344,
1182
+ "lr": 6.294756904831735e-08
1183
+ },
1184
+ {
1185
+ "update": 107,
1186
+ "math_pg": -0.3547899127006531,
1187
+ "math_kl": 0.000621954706730321,
1188
+ "info_pg": -2.0421130657196045,
1189
+ "info_kl": 0.007852323353290558,
1190
+ "conflict": false,
1191
+ "cos_dot": 4.720099925994873,
1192
+ "grad_norm": 545.143310546875,
1193
+ "lr": 6.123585524335919e-08
1194
+ },
1195
+ {
1196
+ "update": 108,
1197
+ "math_pg": -0.35723552107810974,
1198
+ "math_kl": 0.0006870804645586759,
1199
+ "info_pg": -1.9974678754806519,
1200
+ "info_kl": 0.0029452084563672543,
1201
+ "conflict": true,
1202
+ "cos_dot": -3.9602081775665283,
1203
+ "grad_norm": 102.61061096191406,
1204
+ "lr": 5.960435672921572e-08
1205
+ },
1206
+ {
1207
+ "update": 109,
1208
+ "math_pg": -0.35298752784729004,
1209
+ "math_kl": 0.00048777814663480967,
1210
+ "info_pg": 1.1429667472839355,
1211
+ "info_kl": 0.0005623303004540503,
1212
+ "conflict": true,
1213
+ "cos_dot": -0.08116637915372849,
1214
+ "grad_norm": 29.92388916015625,
1215
+ "lr": 5.805424972602571e-08
1216
+ },
1217
+ {
1218
+ "update": 110,
1219
+ "math_pg": 0.8428821861743927,
1220
+ "math_kl": 0.0003461098822299391,
1221
+ "info_pg": 0.762768030166626,
1222
+ "info_kl": 0.0007351161912083626,
1223
+ "conflict": true,
1224
+ "cos_dot": -0.9354484677314758,
1225
+ "grad_norm": 24.29819107055664,
1226
+ "lr": 5.658665177515274e-08
1227
+ },
1228
+ {
1229
+ "update": 111,
1230
+ "math_pg": -0.35667508840560913,
1231
+ "math_kl": 0.00040777110552880913,
1232
+ "info_pg": 0.8494613766670227,
1233
+ "info_kl": 0.00041002529906108975,
1234
+ "conflict": false,
1235
+ "cos_dot": 0.08765782415866852,
1236
+ "grad_norm": 22.337888717651367,
1237
+ "lr": 5.520262093349991e-08
1238
+ },
1239
+ {
1240
+ "update": 112,
1241
+ "math_pg": -0.33756327629089355,
1242
+ "math_kl": 0.00047108410217333585,
1243
+ "info_pg": 0.7194754481315613,
1244
+ "info_kl": 0.000937680306378752,
1245
+ "conflict": true,
1246
+ "cos_dot": -1.3454258441925049,
1247
+ "grad_norm": 46.24418640136719,
1248
+ "lr": 5.390315501070938e-08
1249
+ },
1250
+ {
1251
+ "update": 113,
1252
+ "math_pg": -1.4137396216392517,
1253
+ "math_kl": 0.0010276496759615839,
1254
+ "info_pg": -2.020637035369873,
1255
+ "info_kl": 0.0007347759674303234,
1256
+ "conflict": true,
1257
+ "cos_dot": -0.962862491607666,
1258
+ "grad_norm": 184.51568603515625,
1259
+ "lr": 5.2689190849796674e-08
1260
+ },
1261
+ {
1262
+ "update": 114,
1263
+ "math_pg": -0.3564266562461853,
1264
+ "math_kl": 0.000645190360955894,
1265
+ "info_pg": 0.49314773082733154,
1266
+ "info_kl": 0.0008213946712203324,
1267
+ "conflict": false,
1268
+ "cos_dot": 0.3529888093471527,
1269
+ "grad_norm": 17.641721725463867,
1270
+ "lr": 5.156160365173895e-08
1271
+ },
1272
+ {
1273
+ "update": 115,
1274
+ "math_pg": -0.35436150431632996,
1275
+ "math_kl": 0.00048513715591980144,
1276
+ "info_pg": -2.0095951557159424,
1277
+ "info_kl": 0.0013731971848756075,
1278
+ "conflict": false,
1279
+ "cos_dot": 5.06558084487915,
1280
+ "grad_norm": 146.51138305664062,
1281
+ "lr": 5.0521206344503294e-08
1282
+ },
1283
+ {
1284
+ "update": 116,
1285
+ "math_pg": -1.402031421661377,
1286
+ "math_kl": 0.001435663434676826,
1287
+ "info_pg": -2.0335991382598877,
1288
+ "info_kl": 0.0013365073828026652,
1289
+ "conflict": true,
1290
+ "cos_dot": -2.6764097213745117,
1291
+ "grad_norm": 93.7044677734375,
1292
+ "lr": 4.95687489969708e-08
1293
+ },
1294
+ {
1295
+ "update": 117,
1296
+ "math_pg": -1.3935039043426514,
1297
+ "math_kl": 0.0005805671389680356,
1298
+ "info_pg": 0.9774270057678223,
1299
+ "info_kl": 0.0005040190299041569,
1300
+ "conflict": false,
1301
+ "cos_dot": 0.44716283679008484,
1302
+ "grad_norm": 15.049036979675293,
1303
+ "lr": 4.870491827817838e-08
1304
+ },
1305
+ {
1306
+ "update": 118,
1307
+ "math_pg": -1.4149820804595947,
1308
+ "math_kl": 0.0006915488338563591,
1309
+ "info_pg": -2.006382942199707,
1310
+ "info_kl": 0.02558968774974346,
1311
+ "conflict": false,
1312
+ "cos_dot": 0.8905575275421143,
1313
+ "grad_norm": 83.24064636230469,
1314
+ "lr": 4.7930336962268326e-08
1315
+ },
1316
+ {
1317
+ "update": 119,
1318
+ "math_pg": -1.4151860475540161,
1319
+ "math_kl": 0.00024495362595189363,
1320
+ "info_pg": -2.0316572189331055,
1321
+ "info_kl": 0.0006327910232357681,
1322
+ "conflict": false,
1323
+ "cos_dot": 0.15848509967327118,
1324
+ "grad_norm": 76.53851318359375,
1325
+ "lr": 4.724556347950274e-08
1326
+ },
1327
+ {
1328
+ "update": 120,
1329
+ "math_pg": -0.3555457592010498,
1330
+ "math_kl": 0.0005058542737970129,
1331
+ "info_pg": 0.4941679537296295,
1332
+ "info_kl": 0.0015830148477107286,
1333
+ "conflict": true,
1334
+ "cos_dot": -0.1935790479183197,
1335
+ "grad_norm": 18.782440185546875,
1336
+ "lr": 4.665109151366611e-08
1337
+ },
1338
+ {
1339
+ "update": 121,
1340
+ "math_pg": -0.0008659064769744873,
1341
+ "math_kl": 0.00040810131758917123,
1342
+ "info_pg": -2.0065274238586426,
1343
+ "info_kl": 0.0010482475627213717,
1344
+ "conflict": false,
1345
+ "cos_dot": 2.3563873767852783,
1346
+ "grad_norm": 156.1387481689453,
1347
+ "lr": 4.614734964614667e-08
1348
+ },
1349
+ {
1350
+ "update": 122,
1351
+ "math_pg": 0.7062605023384094,
1352
+ "math_kl": 0.00027502219018060714,
1353
+ "info_pg": 0.4960906505584717,
1354
+ "info_kl": 0.0006710522575303912,
1355
+ "conflict": false,
1356
+ "cos_dot": 0.49717646837234497,
1357
+ "grad_norm": 15.7587251663208,
1358
+ "lr": 4.573470104695294e-08
1359
+ },
1360
+ {
1361
+ "update": 123,
1362
+ "math_pg": -0.21748623251914978,
1363
+ "math_kl": 0.0004485274257604033,
1364
+ "info_pg": 0.900920033454895,
1365
+ "info_kl": 0.0014183461898937821,
1366
+ "conflict": true,
1367
+ "cos_dot": -0.44856420159339905,
1368
+ "grad_norm": 120.2303237915039,
1369
+ "lr": 4.541344321288823e-08
1370
+ },
1371
+ {
1372
+ "update": 124,
1373
+ "math_pg": 0.842811793088913,
1374
+ "math_kl": 0.000415054673794657,
1375
+ "info_pg": -2.056959629058838,
1376
+ "info_kl": 0.017879236489534378,
1377
+ "conflict": true,
1378
+ "cos_dot": -1.0193002223968506,
1379
+ "grad_norm": 276.0547790527344,
1380
+ "lr": 4.518380775307192e-08
1381
+ },
1382
+ {
1383
+ "update": 125,
1384
+ "math_pg": 0.7079161703586578,
1385
+ "math_kl": 0.0003035137051483616,
1386
+ "info_pg": 0.5000783205032349,
1387
+ "info_kl": 0.0008649383089505136,
1388
+ "conflict": false,
1389
+ "cos_dot": 0.4524635970592499,
1390
+ "grad_norm": 10.145975112915039,
1391
+ "lr": 4.504596022196211e-08
1392
+ },
1393
+ {
1394
+ "update": 126,
1395
+ "math_pg": -0.3334178328514099,
1396
+ "math_kl": 0.0005072159547125921,
1397
+ "info_pg": -2.016545295715332,
1398
+ "info_kl": 0.0005507194437086582,
1399
+ "conflict": false,
1400
+ "cos_dot": 3.870191812515259,
1401
+ "grad_norm": 59.78828811645508,
1402
+ "lr": 4.5e-08
1403
+ }
1404
+ ],
1405
+ "source_path": "/marimo/budgie_information_rl_v1/info_math_constrained_source.pt",
1406
+ "seconds": 29.05026912689209,
1407
+ "parameter_count_change": 0,
1408
+ "benchmark_family_rows": 0
1409
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/curriculum_manifest.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "rows": 105,
3
+ "splits": {
4
+ "train": 63,
5
+ "dev": 21,
6
+ "test": 21
7
+ },
8
+ "categories": {
9
+ "state_update": 15,
10
+ "already_tried": 15,
11
+ "selective_answer": 15,
12
+ "delta_summary": 15,
13
+ "conflict_clarification": 15,
14
+ "source_boundary": 15,
15
+ "constraint_stack": 15
16
+ },
17
+ "ideal_reference_pass": 105,
18
+ "benchmark_family_rows": 0,
19
+ "parameter_count_change": 0
20
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/encoded_rollouts.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/encoding_manifest.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "encoded": 315,
3
+ "mean_tokens": 170.52380952380952,
4
+ "mean_response_tokens": 18.444444444444443,
5
+ "max_tokens": 341,
6
+ "seconds": 4.603764295578003
7
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/final_decision.json ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "date": "2026-08-26",
3
+ "stage": "information-management RL v1",
4
+ "status": "rejected; leader unchanged",
5
+ "leader": "verified-math-a025",
6
+ "curriculum": {
7
+ "rows": 105,
8
+ "splits": {
9
+ "train": 63,
10
+ "dev": 21,
11
+ "test": 21
12
+ },
13
+ "categories": {
14
+ "state_update": 15,
15
+ "already_tried": 15,
16
+ "selective_answer": 15,
17
+ "delta_summary": 15,
18
+ "conflict_clarification": 15,
19
+ "source_boundary": 15,
20
+ "constraint_stack": 15
21
+ },
22
+ "ideal_reference_pass": 105,
23
+ "benchmark_family_rows": 0,
24
+ "parameter_count_change": 0
25
+ },
26
+ "reward_data": {
27
+ "cases": 63,
28
+ "budgie_rollouts": 252,
29
+ "manual_demonstrations": 63,
30
+ "mean_reward_budgie": -0.21686507936507934,
31
+ "mean_reward_ideal": 0.9992063492063492,
32
+ "positive_budgie": 4,
33
+ "budgie_deterministic_pass": 21,
34
+ "budgie_loops": 4,
35
+ "parse_errors": [],
36
+ "seconds": 935.0733463764191
37
+ },
38
+ "methods": [
39
+ {
40
+ "name": "conservative GRPO",
41
+ "result": "No deterministic dev improvement through alpha 1.0."
42
+ },
43
+ {
44
+ "name": "long GRPO plus manual BC",
45
+ "result": "Full source dev 4/21 vs 3/21; no core state-update transfer."
46
+ },
47
+ {
48
+ "name": "hybrid GRPO plus 8-epoch ideal consolidation",
49
+ "result": "Full source dev 7/21 and test 5/21 but loops increased; alpha .025 dev 4/21 and test 2/21 with loops unchanged."
50
+ }
51
+ ],
52
+ "heldout": {
53
+ "leader": {
54
+ "dev_pass": 3,
55
+ "dev_total": 21,
56
+ "dev_loops": 0,
57
+ "test_pass": 1,
58
+ "test_total": 21,
59
+ "test_loops": 1
60
+ },
61
+ "hybrid_a0p025": {
62
+ "dev_pass": 4,
63
+ "dev_total": 21,
64
+ "dev_loops": 0,
65
+ "test_pass": 2,
66
+ "test_total": 21,
67
+ "test_loops": 1
68
+ },
69
+ "full_hybrid_source": {
70
+ "dev_pass": 7,
71
+ "dev_total": 21,
72
+ "dev_loops": 2,
73
+ "test_pass": 5,
74
+ "test_total": 21,
75
+ "test_loops": 2
76
+ }
77
+ },
78
+ "fixed_reasoning_gate": {
79
+ "leader": {
80
+ "gsm8k": "5/30",
81
+ "math500": "3/15",
82
+ "arc": "11/30",
83
+ "folio": "13/30"
84
+ },
85
+ "hybrid_a0p025": {
86
+ "gsm8k": "3/30",
87
+ "math500": "1/15",
88
+ "arc": "11/30",
89
+ "folio": "13/30"
90
+ }
91
+ },
92
+ "decision": "Reject hybrid-a0p025 because GSM8K and MATH regress. Reject full hybrid source because repetitions increase and it is more distant. Smaller alphas do not improve the held-out information gate. Keep verified-math-a025 active and public root unchanged.",
93
+ "parameter_count_change": 0,
94
+ "public_root_changed": false,
95
+ "recovery_commits": [
96
+ "349550db2916bc27969dd56d7f8c3fa07cdc68dc",
97
+ "5112b94abff81b17297671b8e48751b2336d0839",
98
+ "02342edca539f218c9ab6e462616562f04fb7a5c",
99
+ "d53a83fc50ebc4481410e566f195610db438430d"
100
+ ]
101
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_dev_gate.json ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "leader": {
3
+ "alpha": 0,
4
+ "pass": 3,
5
+ "n": 21,
6
+ "mean_reward": 0.3333333333333334,
7
+ "loops": 0,
8
+ "mean_words": 8.857142857142858,
9
+ "by_category": {
10
+ "already_tried": 0,
11
+ "conflict_clarification": 0,
12
+ "constraint_stack": 2,
13
+ "delta_summary": 0,
14
+ "selective_answer": 1,
15
+ "source_boundary": 0,
16
+ "state_update": 0
17
+ }
18
+ },
19
+ "a0p005": {
20
+ "alpha": 0.005,
21
+ "pass": 3,
22
+ "n": 21,
23
+ "mean_reward": 0.3333333333333334,
24
+ "loops": 0,
25
+ "mean_words": 9.380952380952381,
26
+ "by_category": {
27
+ "already_tried": 0,
28
+ "conflict_clarification": 0,
29
+ "constraint_stack": 2,
30
+ "delta_summary": 0,
31
+ "selective_answer": 1,
32
+ "source_boundary": 0,
33
+ "state_update": 0
34
+ }
35
+ },
36
+ "a0p01": {
37
+ "alpha": 0.01,
38
+ "pass": 3,
39
+ "n": 21,
40
+ "mean_reward": 0.3333333333333334,
41
+ "loops": 0,
42
+ "mean_words": 9.142857142857142,
43
+ "by_category": {
44
+ "already_tried": 0,
45
+ "conflict_clarification": 0,
46
+ "constraint_stack": 2,
47
+ "delta_summary": 0,
48
+ "selective_answer": 1,
49
+ "source_boundary": 0,
50
+ "state_update": 0
51
+ }
52
+ },
53
+ "a0p025": {
54
+ "alpha": 0.025,
55
+ "pass": 4,
56
+ "n": 21,
57
+ "mean_reward": 0.36666666666666675,
58
+ "loops": 0,
59
+ "mean_words": 8.571428571428571,
60
+ "by_category": {
61
+ "already_tried": 0,
62
+ "conflict_clarification": 0,
63
+ "constraint_stack": 3,
64
+ "delta_summary": 0,
65
+ "selective_answer": 1,
66
+ "source_boundary": 0,
67
+ "state_update": 0
68
+ }
69
+ },
70
+ "a0p05": {
71
+ "alpha": 0.05,
72
+ "pass": 3,
73
+ "n": 21,
74
+ "mean_reward": 0.3500000000000001,
75
+ "loops": 0,
76
+ "mean_words": 9.047619047619047,
77
+ "by_category": {
78
+ "already_tried": 0,
79
+ "conflict_clarification": 0,
80
+ "constraint_stack": 2,
81
+ "delta_summary": 0,
82
+ "selective_answer": 1,
83
+ "source_boundary": 0,
84
+ "state_update": 0
85
+ }
86
+ },
87
+ "a0p075": {
88
+ "alpha": 0.075,
89
+ "pass": 3,
90
+ "n": 21,
91
+ "mean_reward": 0.3500000000000001,
92
+ "loops": 0,
93
+ "mean_words": 8.666666666666666,
94
+ "by_category": {
95
+ "already_tried": 0,
96
+ "conflict_clarification": 0,
97
+ "constraint_stack": 2,
98
+ "delta_summary": 0,
99
+ "selective_answer": 1,
100
+ "source_boundary": 0,
101
+ "state_update": 0
102
+ }
103
+ },
104
+ "a0p1": {
105
+ "alpha": 0.1,
106
+ "pass": 3,
107
+ "n": 21,
108
+ "mean_reward": 0.3333333333333334,
109
+ "loops": 0,
110
+ "mean_words": 9.047619047619047,
111
+ "by_category": {
112
+ "already_tried": 0,
113
+ "conflict_clarification": 0,
114
+ "constraint_stack": 2,
115
+ "delta_summary": 0,
116
+ "selective_answer": 1,
117
+ "source_boundary": 0,
118
+ "state_update": 0
119
+ }
120
+ },
121
+ "a0p15": {
122
+ "alpha": 0.15,
123
+ "pass": 3,
124
+ "n": 21,
125
+ "mean_reward": 0.3333333333333334,
126
+ "loops": 0,
127
+ "mean_words": 8.952380952380953,
128
+ "by_category": {
129
+ "already_tried": 0,
130
+ "conflict_clarification": 0,
131
+ "constraint_stack": 2,
132
+ "delta_summary": 0,
133
+ "selective_answer": 1,
134
+ "source_boundary": 0,
135
+ "state_update": 0
136
+ }
137
+ },
138
+ "a0p2": {
139
+ "alpha": 0.2,
140
+ "pass": 3,
141
+ "n": 21,
142
+ "mean_reward": 0.3333333333333334,
143
+ "loops": 0,
144
+ "mean_words": 9.095238095238095,
145
+ "by_category": {
146
+ "already_tried": 0,
147
+ "conflict_clarification": 0,
148
+ "constraint_stack": 2,
149
+ "delta_summary": 0,
150
+ "selective_answer": 1,
151
+ "source_boundary": 0,
152
+ "state_update": 0
153
+ }
154
+ },
155
+ "a0p3": {
156
+ "alpha": 0.3,
157
+ "pass": 3,
158
+ "n": 21,
159
+ "mean_reward": 0.3500000000000001,
160
+ "loops": 0,
161
+ "mean_words": 9.238095238095237,
162
+ "by_category": {
163
+ "already_tried": 0,
164
+ "conflict_clarification": 0,
165
+ "constraint_stack": 2,
166
+ "delta_summary": 0,
167
+ "selective_answer": 1,
168
+ "source_boundary": 0,
169
+ "state_update": 0
170
+ }
171
+ },
172
+ "a0p5": {
173
+ "alpha": 0.5,
174
+ "pass": 3,
175
+ "n": 21,
176
+ "mean_reward": 0.2666666666666667,
177
+ "loops": 3,
178
+ "mean_words": 21.61904761904762,
179
+ "by_category": {
180
+ "already_tried": 0,
181
+ "conflict_clarification": 0,
182
+ "constraint_stack": 2,
183
+ "delta_summary": 0,
184
+ "selective_answer": 1,
185
+ "source_boundary": 0,
186
+ "state_update": 0
187
+ }
188
+ },
189
+ "a1p0": {
190
+ "alpha": 1.0,
191
+ "pass": 7,
192
+ "n": 21,
193
+ "mean_reward": 0.5166666666666667,
194
+ "loops": 2,
195
+ "mean_words": 13.952380952380953,
196
+ "by_category": {
197
+ "already_tried": 0,
198
+ "conflict_clarification": 3,
199
+ "constraint_stack": 2,
200
+ "delta_summary": 0,
201
+ "selective_answer": 2,
202
+ "source_boundary": 0,
203
+ "state_update": 0
204
+ }
205
+ }
206
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_test_gate.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "leader": {
3
+ "alpha": 0,
4
+ "pass": 1,
5
+ "n": 21,
6
+ "mean_reward": 0.23333333333333345,
7
+ "loops": 1,
8
+ "mean_words": 12.761904761904763,
9
+ "by_category": {
10
+ "already_tried": 0,
11
+ "conflict_clarification": 0,
12
+ "constraint_stack": 1,
13
+ "delta_summary": 0,
14
+ "selective_answer": 0,
15
+ "source_boundary": 0,
16
+ "state_update": 0
17
+ }
18
+ },
19
+ "hybrid_a0p025": {
20
+ "alpha": 0.025,
21
+ "pass": 2,
22
+ "n": 21,
23
+ "mean_reward": 0.2500000000000001,
24
+ "loops": 1,
25
+ "mean_words": 13.714285714285714,
26
+ "by_category": {
27
+ "already_tried": 0,
28
+ "conflict_clarification": 0,
29
+ "constraint_stack": 1,
30
+ "delta_summary": 0,
31
+ "selective_answer": 0,
32
+ "source_boundary": 1,
33
+ "state_update": 0
34
+ }
35
+ },
36
+ "hybrid_a1p0": {
37
+ "alpha": 1.0,
38
+ "pass": 5,
39
+ "n": 21,
40
+ "mean_reward": 0.4214285714285715,
41
+ "loops": 2,
42
+ "mean_words": 11.666666666666666,
43
+ "by_category": {
44
+ "already_tried": 0,
45
+ "conflict_clarification": 2,
46
+ "constraint_stack": 1,
47
+ "delta_summary": 0,
48
+ "selective_answer": 1,
49
+ "source_boundary": 1,
50
+ "state_update": 0
51
+ }
52
+ }
53
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/hybrid_training_report.json ADDED
@@ -0,0 +1,768 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "base": "long GRPO source from verified-math-a025",
3
+ "method": "8-epoch manual ideal-response consolidation after GRPO",
4
+ "rows": 63,
5
+ "updates": 126,
6
+ "peak_lr": 2e-06,
7
+ "history": [
8
+ {
9
+ "update": 1,
10
+ "loss": 4.039825260639191,
11
+ "grad_norm": 444.040283203125,
12
+ "lr": 2.5e-07
13
+ },
14
+ {
15
+ "update": 2,
16
+ "loss": 3.743570566177368,
17
+ "grad_norm": 145.93502807617188,
18
+ "lr": 5e-07
19
+ },
20
+ {
21
+ "update": 3,
22
+ "loss": 3.3991292119026184,
23
+ "grad_norm": 205.69931030273438,
24
+ "lr": 7.5e-07
25
+ },
26
+ {
27
+ "update": 4,
28
+ "loss": 4.1453264355659485,
29
+ "grad_norm": 128.31797790527344,
30
+ "lr": 1e-06
31
+ },
32
+ {
33
+ "update": 5,
34
+ "loss": 3.7671092748641968,
35
+ "grad_norm": 82.89564514160156,
36
+ "lr": 1.2499999999999999e-06
37
+ },
38
+ {
39
+ "update": 6,
40
+ "loss": 3.2989422082901,
41
+ "grad_norm": 60.31236267089844,
42
+ "lr": 1.5e-06
43
+ },
44
+ {
45
+ "update": 7,
46
+ "loss": 2.5751004815101624,
47
+ "grad_norm": 77.47644805908203,
48
+ "lr": 1.75e-06
49
+ },
50
+ {
51
+ "update": 8,
52
+ "loss": 2.7045833468437195,
53
+ "grad_norm": 48.908607482910156,
54
+ "lr": 2e-06
55
+ },
56
+ {
57
+ "update": 9,
58
+ "loss": 3.2889516949653625,
59
+ "grad_norm": 160.1339874267578,
60
+ "lr": 2e-06
61
+ },
62
+ {
63
+ "update": 10,
64
+ "loss": 4.074514925479889,
65
+ "grad_norm": 95.43360137939453,
66
+ "lr": 1.9996755749037967e-06
67
+ },
68
+ {
69
+ "update": 11,
70
+ "loss": 2.6491937041282654,
71
+ "grad_norm": 81.17024993896484,
72
+ "lr": 1.9987025335077274e-06
73
+ },
74
+ {
75
+ "update": 12,
76
+ "loss": 2.528728663921356,
77
+ "grad_norm": 67.7745132446289,
78
+ "lr": 1.997081577320789e-06
79
+ },
80
+ {
81
+ "update": 13,
82
+ "loss": 3.1969401240348816,
83
+ "grad_norm": 166.1395263671875,
84
+ "lr": 1.9948138749626852e-06
85
+ },
86
+ {
87
+ "update": 14,
88
+ "loss": 3.378227114677429,
89
+ "grad_norm": 70.27755737304688,
90
+ "lr": 1.9919010613213176e-06
91
+ },
92
+ {
93
+ "update": 15,
94
+ "loss": 2.9134928584098816,
95
+ "grad_norm": 69.02920532226562,
96
+ "lr": 1.988345236374121e-06
97
+ },
98
+ {
99
+ "update": 16,
100
+ "loss": 3.122160702943802,
101
+ "grad_norm": 168.7936553955078,
102
+ "lr": 1.984148963674098e-06
103
+ },
104
+ {
105
+ "update": 17,
106
+ "loss": 2.56716126203537,
107
+ "grad_norm": 341.92889404296875,
108
+ "lr": 1.979315268501635e-06
109
+ },
110
+ {
111
+ "update": 18,
112
+ "loss": 2.512660175561905,
113
+ "grad_norm": 67.98460388183594,
114
+ "lr": 1.973847635683447e-06
115
+ },
116
+ {
117
+ "update": 19,
118
+ "loss": 3.56206876039505,
119
+ "grad_norm": 66.12527465820312,
120
+ "lr": 1.967750007080206e-06
121
+ },
122
+ {
123
+ "update": 20,
124
+ "loss": 3.0908075273036957,
125
+ "grad_norm": 62.41841506958008,
126
+ "lr": 1.961026778744683e-06
127
+ },
128
+ {
129
+ "update": 21,
130
+ "loss": 2.302137613296509,
131
+ "grad_norm": 107.56423950195312,
132
+ "lr": 1.953682797752431e-06
133
+ },
134
+ {
135
+ "update": 22,
136
+ "loss": 3.143071234226227,
137
+ "grad_norm": 69.44225311279297,
138
+ "lr": 1.9457233587073175e-06
139
+ },
140
+ {
141
+ "update": 23,
142
+ "loss": 2.8047977089881897,
143
+ "grad_norm": 139.52978515625,
144
+ "lr": 1.9371541999244046e-06
145
+ },
146
+ {
147
+ "update": 24,
148
+ "loss": 2.235747903585434,
149
+ "grad_norm": 89.59989166259766,
150
+ "lr": 1.9279814992929416e-06
151
+ },
152
+ {
153
+ "update": 25,
154
+ "loss": 3.05132919549942,
155
+ "grad_norm": 116.39580535888672,
156
+ "lr": 1.9182118698224515e-06
157
+ },
158
+ {
159
+ "update": 26,
160
+ "loss": 3.206944167613983,
161
+ "grad_norm": 64.20648193359375,
162
+ "lr": 1.9078523548751127e-06
163
+ },
164
+ {
165
+ "update": 27,
166
+ "loss": 3.0517847537994385,
167
+ "grad_norm": 146.65589904785156,
168
+ "lr": 1.8969104230878888e-06
169
+ },
170
+ {
171
+ "update": 28,
172
+ "loss": 2.915673926472664,
173
+ "grad_norm": 90.91871643066406,
174
+ "lr": 1.8853939629880528e-06
175
+ },
176
+ {
177
+ "update": 29,
178
+ "loss": 2.9550198018550873,
179
+ "grad_norm": 39.75973129272461,
180
+ "lr": 1.8733112773059952e-06
181
+ },
182
+ {
183
+ "update": 30,
184
+ "loss": 2.1965317726135254,
185
+ "grad_norm": 45.64877700805664,
186
+ "lr": 1.860671076989415e-06
187
+ },
188
+ {
189
+ "update": 31,
190
+ "loss": 3.3446889221668243,
191
+ "grad_norm": 155.02618408203125,
192
+ "lr": 1.8474824749232105e-06
193
+ },
194
+ {
195
+ "update": 32,
196
+ "loss": 3.4126275181770325,
197
+ "grad_norm": 161.667236328125,
198
+ "lr": 1.8337549793595896e-06
199
+ },
200
+ {
201
+ "update": 33,
202
+ "loss": 2.6579933762550354,
203
+ "grad_norm": 46.23124694824219,
204
+ "lr": 1.8194984870631512e-06
205
+ },
206
+ {
207
+ "update": 34,
208
+ "loss": 2.6678313612937927,
209
+ "grad_norm": 338.75823974609375,
210
+ "lr": 1.8047232761758597e-06
211
+ },
212
+ {
213
+ "update": 35,
214
+ "loss": 2.6663589477539062,
215
+ "grad_norm": 49.884552001953125,
216
+ "lr": 1.78943999880708e-06
217
+ },
218
+ {
219
+ "update": 36,
220
+ "loss": 2.426388740539551,
221
+ "grad_norm": 58.36009216308594,
222
+ "lr": 1.7736596733539907e-06
223
+ },
224
+ {
225
+ "update": 37,
226
+ "loss": 2.2160482071340084,
227
+ "grad_norm": 78.70022583007812,
228
+ "lr": 1.7573936765579326e-06
229
+ },
230
+ {
231
+ "update": 38,
232
+ "loss": 2.1053349375724792,
233
+ "grad_norm": 136.91580200195312,
234
+ "lr": 1.7406537353024065e-06
235
+ },
236
+ {
237
+ "update": 39,
238
+ "loss": 2.1104882806539536,
239
+ "grad_norm": 82.48108673095703,
240
+ "lr": 1.7234519181586395e-06
241
+ },
242
+ {
243
+ "update": 40,
244
+ "loss": 2.1138173639774323,
245
+ "grad_norm": 50.78386688232422,
246
+ "lr": 1.705800626684813e-06
247
+ },
248
+ {
249
+ "update": 41,
250
+ "loss": 2.7158827781677246,
251
+ "grad_norm": 82.92627716064453,
252
+ "lr": 1.6877125864852265e-06
253
+ },
254
+ {
255
+ "update": 42,
256
+ "loss": 2.569867432117462,
257
+ "grad_norm": 109.22274780273438,
258
+ "lr": 1.6692008380358394e-06
259
+ },
260
+ {
261
+ "update": 43,
262
+ "loss": 2.459792248904705,
263
+ "grad_norm": 92.05064392089844,
264
+ "lr": 1.6502787272828109e-06
265
+ },
266
+ {
267
+ "update": 44,
268
+ "loss": 2.9284843504428864,
269
+ "grad_norm": 54.5689811706543,
270
+ "lr": 1.6309598960208107e-06
271
+ },
272
+ {
273
+ "update": 45,
274
+ "loss": 3.131755091249943,
275
+ "grad_norm": 82.51221466064453,
276
+ "lr": 1.6112582720580404e-06
277
+ },
278
+ {
279
+ "update": 46,
280
+ "loss": 2.290101081132889,
281
+ "grad_norm": 43.71992111206055,
282
+ "lr": 1.5911880591750539e-06
283
+ },
284
+ {
285
+ "update": 47,
286
+ "loss": 2.498674988746643,
287
+ "grad_norm": 49.3583984375,
288
+ "lr": 1.5707637268846196e-06
289
+ },
290
+ {
291
+ "update": 48,
292
+ "loss": 2.5337125714868307,
293
+ "grad_norm": 39.9047737121582,
294
+ "lr": 1.55e-06
295
+ },
296
+ {
297
+ "update": 49,
298
+ "loss": 1.6494463682174683,
299
+ "grad_norm": 95.84458923339844,
300
+ "lr": 1.528911848019177e-06
301
+ },
302
+ {
303
+ "update": 50,
304
+ "loss": 1.9845543056726456,
305
+ "grad_norm": 43.6153678894043,
306
+ "lr": 1.5075144743326738e-06
307
+ },
308
+ {
309
+ "update": 51,
310
+ "loss": 2.660667508840561,
311
+ "grad_norm": 230.04049682617188,
312
+ "lr": 1.4858233052627487e-06
313
+ },
314
+ {
315
+ "update": 52,
316
+ "loss": 2.148329734802246,
317
+ "grad_norm": 40.60578155517578,
318
+ "lr": 1.4638539789418744e-06
319
+ },
320
+ {
321
+ "update": 53,
322
+ "loss": 2.109841048717499,
323
+ "grad_norm": 55.406494140625,
324
+ "lr": 1.4416223340385068e-06
325
+ },
326
+ {
327
+ "update": 54,
328
+ "loss": 2.467424362897873,
329
+ "grad_norm": 46.915855407714844,
330
+ "lr": 1.419144398338282e-06
331
+ },
332
+ {
333
+ "update": 55,
334
+ "loss": 2.5224798917770386,
335
+ "grad_norm": 96.59014892578125,
336
+ "lr": 1.3964363771888716e-06
337
+ },
338
+ {
339
+ "update": 56,
340
+ "loss": 2.026753395795822,
341
+ "grad_norm": 40.28848648071289,
342
+ "lr": 1.3735146418168222e-06
343
+ },
344
+ {
345
+ "update": 57,
346
+ "loss": 2.1092909276485443,
347
+ "grad_norm": 38.93667221069336,
348
+ "lr": 1.3503957175248073e-06
349
+ },
350
+ {
351
+ "update": 58,
352
+ "loss": 2.344885379076004,
353
+ "grad_norm": 56.196800231933594,
354
+ "lr": 1.3270962717777997e-06
355
+ },
356
+ {
357
+ "update": 59,
358
+ "loss": 3.3393632769584656,
359
+ "grad_norm": 88.70268249511719,
360
+ "lr": 1.3036331021867497e-06
361
+ },
362
+ {
363
+ "update": 60,
364
+ "loss": 1.9308308400213718,
365
+ "grad_norm": 61.54011917114258,
366
+ "lr": 1.2800231243984401e-06
367
+ },
368
+ {
369
+ "update": 61,
370
+ "loss": 2.1644031703472137,
371
+ "grad_norm": 31.750835418701172,
372
+ "lr": 1.2562833599002372e-06
373
+ },
374
+ {
375
+ "update": 62,
376
+ "loss": 1.9471352398395538,
377
+ "grad_norm": 80.13998413085938,
378
+ "lr": 1.2324309237485445e-06
379
+ },
380
+ {
381
+ "update": 63,
382
+ "loss": 2.4662376046180725,
383
+ "grad_norm": 63.7134895324707,
384
+ "lr": 1.2084830122297909e-06
385
+ },
386
+ {
387
+ "update": 64,
388
+ "loss": 2.5796701312065125,
389
+ "grad_norm": 53.398380279541016,
390
+ "lr": 1.1844568904628613e-06
391
+ },
392
+ {
393
+ "update": 65,
394
+ "loss": 1.8910281211137772,
395
+ "grad_norm": 47.025291442871094,
396
+ "lr": 1.1603698799519022e-06
397
+ },
398
+ {
399
+ "update": 66,
400
+ "loss": 2.5808085501194,
401
+ "grad_norm": 89.11796569824219,
402
+ "lr": 1.1362393460984735e-06
403
+ },
404
+ {
405
+ "update": 67,
406
+ "loss": 2.51431205868721,
407
+ "grad_norm": 47.033138275146484,
408
+ "lr": 1.1120826856820577e-06
409
+ },
410
+ {
411
+ "update": 68,
412
+ "loss": 2.231712064705789,
413
+ "grad_norm": 56.363197326660156,
414
+ "lr": 1.0879173143179426e-06
415
+ },
416
+ {
417
+ "update": 69,
418
+ "loss": 2.5181360244750977,
419
+ "grad_norm": 38.62602996826172,
420
+ "lr": 1.0637606539015268e-06
421
+ },
422
+ {
423
+ "update": 70,
424
+ "loss": 1.3502467274665833,
425
+ "grad_norm": 77.26298522949219,
426
+ "lr": 1.0396301200480979e-06
427
+ },
428
+ {
429
+ "update": 71,
430
+ "loss": 1.4006303995847702,
431
+ "grad_norm": 30.853120803833008,
432
+ "lr": 1.0155431095371388e-06
433
+ },
434
+ {
435
+ "update": 72,
436
+ "loss": 1.8782568871974945,
437
+ "grad_norm": 43.32051086425781,
438
+ "lr": 9.915169877702092e-07
439
+ },
440
+ {
441
+ "update": 73,
442
+ "loss": 2.083909863140434,
443
+ "grad_norm": 53.801822662353516,
444
+ "lr": 9.675690762514554e-07
445
+ },
446
+ {
447
+ "update": 74,
448
+ "loss": 1.6123994588851929,
449
+ "grad_norm": 31.710891723632812,
450
+ "lr": 9.437166400997629e-07
451
+ },
452
+ {
453
+ "update": 75,
454
+ "loss": 1.9556108973920345,
455
+ "grad_norm": 97.12733459472656,
456
+ "lr": 9.199768756015602e-07
457
+ },
458
+ {
459
+ "update": 76,
460
+ "loss": 1.4689449667930603,
461
+ "grad_norm": 76.20115661621094,
462
+ "lr": 8.963668978132502e-07
463
+ },
464
+ {
465
+ "update": 77,
466
+ "loss": 1.6270627975463867,
467
+ "grad_norm": 29.44704818725586,
468
+ "lr": 8.729037282222005e-07
469
+ },
470
+ {
471
+ "update": 78,
472
+ "loss": 2.9926183158531785,
473
+ "grad_norm": 65.76885986328125,
474
+ "lr": 8.496042824751925e-07
475
+ },
476
+ {
477
+ "update": 79,
478
+ "loss": 1.7530170381069183,
479
+ "grad_norm": 37.15497970581055,
480
+ "lr": 8.264853581831778e-07
481
+ },
482
+ {
483
+ "update": 80,
484
+ "loss": 2.6701021790504456,
485
+ "grad_norm": 63.23847579956055,
486
+ "lr": 8.035636228111283e-07
487
+ },
488
+ {
489
+ "update": 81,
490
+ "loss": 1.8949713874608278,
491
+ "grad_norm": 38.718841552734375,
492
+ "lr": 7.80855601661718e-07
493
+ },
494
+ {
495
+ "update": 82,
496
+ "loss": 2.097111356444657,
497
+ "grad_norm": 37.66083526611328,
498
+ "lr": 7.583776659614936e-07
499
+ },
500
+ {
501
+ "update": 83,
502
+ "loss": 2.1271576583385468,
503
+ "grad_norm": 44.62977981567383,
504
+ "lr": 7.361460210581254e-07
505
+ },
506
+ {
507
+ "update": 84,
508
+ "loss": 1.3919870406389236,
509
+ "grad_norm": 33.02729034423828,
510
+ "lr": 7.141766947372514e-07
511
+ },
512
+ {
513
+ "update": 85,
514
+ "loss": 2.1023717522621155,
515
+ "grad_norm": 60.720882415771484,
516
+ "lr": 6.924855256673266e-07
517
+ },
518
+ {
519
+ "update": 86,
520
+ "loss": 1.9150399267673492,
521
+ "grad_norm": 55.23069763183594,
522
+ "lr": 6.710881519808229e-07
523
+ },
524
+ {
525
+ "update": 87,
526
+ "loss": 1.001864439342171,
527
+ "grad_norm": 31.232582092285156,
528
+ "lr": 6.500000000000002e-07
529
+ },
530
+ {
531
+ "update": 88,
532
+ "loss": 1.6875469982624054,
533
+ "grad_norm": 39.78133010864258,
534
+ "lr": 6.292362731153802e-07
535
+ },
536
+ {
537
+ "update": 89,
538
+ "loss": 1.8454711437225342,
539
+ "grad_norm": 37.15188980102539,
540
+ "lr": 6.088119408249458e-07
541
+ },
542
+ {
543
+ "update": 90,
544
+ "loss": 1.969850331544876,
545
+ "grad_norm": 40.388031005859375,
546
+ "lr": 5.887417279419599e-07
547
+ },
548
+ {
549
+ "update": 91,
550
+ "loss": 1.447516760788858,
551
+ "grad_norm": 80.65613555908203,
552
+ "lr": 5.690401039791896e-07
553
+ },
554
+ {
555
+ "update": 92,
556
+ "loss": 2.345772683620453,
557
+ "grad_norm": 79.79951477050781,
558
+ "lr": 5.497212727171889e-07
559
+ },
560
+ {
561
+ "update": 93,
562
+ "loss": 2.299563229084015,
563
+ "grad_norm": 41.189239501953125,
564
+ "lr": 5.307991619641605e-07
565
+ },
566
+ {
567
+ "update": 94,
568
+ "loss": 1.796433463692665,
569
+ "grad_norm": 62.8557243347168,
570
+ "lr": 5.122874135147733e-07
571
+ },
572
+ {
573
+ "update": 95,
574
+ "loss": 1.721599355340004,
575
+ "grad_norm": 36.86030578613281,
576
+ "lr": 4.94199373315187e-07
577
+ },
578
+ {
579
+ "update": 96,
580
+ "loss": 2.2367631793022156,
581
+ "grad_norm": 36.5849723815918,
582
+ "lr": 4.765480818413609e-07
583
+ },
584
+ {
585
+ "update": 97,
586
+ "loss": 0.8942152983509004,
587
+ "grad_norm": 22.681598663330078,
588
+ "lr": 4.593462646975936e-07
589
+ },
590
+ {
591
+ "update": 98,
592
+ "loss": 3.0119372606277466,
593
+ "grad_norm": 65.02761840820312,
594
+ "lr": 4.4260632344206726e-07
595
+ },
596
+ {
597
+ "update": 99,
598
+ "loss": 1.8733932375907898,
599
+ "grad_norm": 85.15650177001953,
600
+ "lr": 4.2634032664600896e-07
601
+ },
602
+ {
603
+ "update": 100,
604
+ "loss": 1.313596561551094,
605
+ "grad_norm": 32.66768264770508,
606
+ "lr": 4.105600011929199e-07
607
+ },
608
+ {
609
+ "update": 101,
610
+ "loss": 1.2862253584899008,
611
+ "grad_norm": 30.563064575195312,
612
+ "lr": 3.9527672382414003e-07
613
+ },
614
+ {
615
+ "update": 102,
616
+ "loss": 1.8459213376045227,
617
+ "grad_norm": 43.4942741394043,
618
+ "lr": 3.805015129368492e-07
619
+ },
620
+ {
621
+ "update": 103,
622
+ "loss": 1.356388971209526,
623
+ "grad_norm": 25.93343734741211,
624
+ "lr": 3.662450206404102e-07
625
+ },
626
+ {
627
+ "update": 104,
628
+ "loss": 2.4240047335624695,
629
+ "grad_norm": 35.70319747924805,
630
+ "lr": 3.525175250767896e-07
631
+ },
632
+ {
633
+ "update": 105,
634
+ "loss": 1.2130699353292584,
635
+ "grad_norm": 32.22932434082031,
636
+ "lr": 3.3932892301058486e-07
637
+ },
638
+ {
639
+ "update": 106,
640
+ "loss": 2.5364281833171844,
641
+ "grad_norm": 61.31160354614258,
642
+ "lr": 3.2668872269400484e-07
643
+ },
644
+ {
645
+ "update": 107,
646
+ "loss": 1.9427461326122284,
647
+ "grad_norm": 36.784542083740234,
648
+ "lr": 3.146060370119473e-07
649
+ },
650
+ {
651
+ "update": 108,
652
+ "loss": 2.5215356796979904,
653
+ "grad_norm": 55.17816162109375,
654
+ "lr": 3.03089576912111e-07
655
+ },
656
+ {
657
+ "update": 109,
658
+ "loss": 1.3408555090427399,
659
+ "grad_norm": 33.4058723449707,
660
+ "lr": 2.921476451248874e-07
661
+ },
662
+ {
663
+ "update": 110,
664
+ "loss": 1.2249134834855795,
665
+ "grad_norm": 68.6984634399414,
666
+ "lr": 2.817881301775488e-07
667
+ },
668
+ {
669
+ "update": 111,
670
+ "loss": 1.1046768948435783,
671
+ "grad_norm": 58.63383865356445,
672
+ "lr": 2.7201850070705825e-07
673
+ },
674
+ {
675
+ "update": 112,
676
+ "loss": 1.2363624721765518,
677
+ "grad_norm": 28.14107894897461,
678
+ "lr": 2.6284580007559564e-07
679
+ },
680
+ {
681
+ "update": 113,
682
+ "loss": 2.4891226440668106,
683
+ "grad_norm": 62.42837142944336,
684
+ "lr": 2.542766412926824e-07
685
+ },
686
+ {
687
+ "update": 114,
688
+ "loss": 1.191118311136961,
689
+ "grad_norm": 33.83761978149414,
690
+ "lr": 2.4631720224756906e-07
691
+ },
692
+ {
693
+ "update": 115,
694
+ "loss": 2.057683438062668,
695
+ "grad_norm": 39.86688995361328,
696
+ "lr": 2.389732212553174e-07
697
+ },
698
+ {
699
+ "update": 116,
700
+ "loss": 2.4992255568504333,
701
+ "grad_norm": 40.132606506347656,
702
+ "lr": 2.322499929197939e-07
703
+ },
704
+ {
705
+ "update": 117,
706
+ "loss": 3.0710437297821045,
707
+ "grad_norm": 60.52458572387695,
708
+ "lr": 2.261523643165533e-07
709
+ },
710
+ {
711
+ "update": 118,
712
+ "loss": 1.9181353747844696,
713
+ "grad_norm": 66.15868377685547,
714
+ "lr": 2.206847314983647e-07
715
+ },
716
+ {
717
+ "update": 119,
718
+ "loss": 1.4751406162977219,
719
+ "grad_norm": 79.93660736083984,
720
+ "lr": 2.1585103632590171e-07
721
+ },
722
+ {
723
+ "update": 120,
724
+ "loss": 1.3226167259272188,
725
+ "grad_norm": 47.1295280456543,
726
+ "lr": 2.1165476362587845e-07
727
+ },
728
+ {
729
+ "update": 121,
730
+ "loss": 1.1483375281095505,
731
+ "grad_norm": 29.44586944580078,
732
+ "lr": 2.080989386786824e-07
733
+ },
734
+ {
735
+ "update": 122,
736
+ "loss": 1.2816440318711102,
737
+ "grad_norm": 34.2610969543457,
738
+ "lr": 2.0518612503731492e-07
739
+ },
740
+ {
741
+ "update": 123,
742
+ "loss": 1.402582824230194,
743
+ "grad_norm": 36.53142547607422,
744
+ "lr": 2.0291842267921106e-07
745
+ },
746
+ {
747
+ "update": 124,
748
+ "loss": 2.162113755941391,
749
+ "grad_norm": 34.86579513549805,
750
+ "lr": 2.0129746649227242e-07
751
+ },
752
+ {
753
+ "update": 125,
754
+ "loss": 1.664678193628788,
755
+ "grad_norm": 61.73565673828125,
756
+ "lr": 2.0032442509620316e-07
757
+ },
758
+ {
759
+ "update": 126,
760
+ "loss": 1.4778528213500977,
761
+ "grad_norm": 28.797748565673828,
762
+ "lr": 2e-07
763
+ }
764
+ ],
765
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_hybrid_source.pt",
766
+ "seconds": 33.85362410545349,
767
+ "parameter_count_change": 0
768
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/info_math_constrained_source.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:00a4221a276258561c0150faad4f5aa8861eae3535ae137047b7d73aa2b0c443
3
+ size 1101695710
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/info_rl_v1.py ADDED
@@ -0,0 +1,268 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Manual long-dialogue information-management curriculum for Budgie-500M.
2
+
3
+ The cases are synthetic and unrelated to held-out reasoning benchmarks. Each case
4
+ contains explicit state/constraint checks suitable for deterministic reward plus
5
+ an independent semantic judge. Training/dev/test topics are disjoint.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import json
10
+ import re
11
+ from collections import Counter
12
+ from typing import Any
13
+
14
+ SYSTEM = (
15
+ "You are Budgie, a compact AI assistant created by Asilarkness. Track the "
16
+ "conversation's current facts and constraints carefully. Prefer the newest "
17
+ "explicit correction over stale information. Answer the latest request "
18
+ "directly, do not repeat the whole conversation, and do not invent missing "
19
+ "facts. Ask one focused clarification only when a contradiction or missing "
20
+ "detail blocks a useful answer. Avoid canned closings and repetition."
21
+ )
22
+
23
+ # Fifteen semantically distinct variants per skill. Indices 0-8 train, 9-11 dev,
24
+ # and 12-14 test, so no topic instance appears in more than one split.
25
+ STATE_UPDATES = [
26
+ dict(topic="community workshop", old="Saturday at 10:00", new="Sunday at 14:00", fixed="Room Cedar", ask="Give the final time and room in one sentence.", ideal="The community workshop is Sunday at 14:00 in Room Cedar."),
27
+ dict(topic="software release", old="version 3.4 on May 8", new="version 3.5 on May 10", fixed="staged rollout", ask="State the current version, date, and rollout method in one sentence.", ideal="Version 3.5 will launch on May 10 through a staged rollout."),
28
+ dict(topic="supplier order", old="240 blue folders", new="180 green folders", fixed="deliver to Dock 2", ask="Write the final order instruction in one sentence.", ideal="Order 180 green folders and deliver them to Dock 2."),
29
+ dict(topic="interview schedule", old="Mira at 09:30", new="Jon at 11:15", fixed="video call", ask="Give only the current candidate, time, and format.", ideal="Jon at 11:15 by video call."),
30
+ dict(topic="museum visit", old="12 adult tickets", new="9 adult tickets and 3 child tickets", fixed="entry at 16:20", ask="Summarize the final booking in one sentence.", ideal="Book 9 adult tickets and 3 child tickets for entry at 16:20."),
31
+ dict(topic="newsletter campaign", old="send Tuesday to 8,000 readers", new="send Thursday to 6,500 readers", fixed="subject line: Spring Notes", ask="Give the current send plan in one sentence.", ideal="Send “Spring Notes” on Thursday to 6,500 readers."),
32
+ dict(topic="lab sample", old="label K-17 at 4°C", new="label K-19 at -20°C", fixed="amber container", ask="State the final storage record in one sentence.", ideal="Store sample K-19 at -20°C in an amber container."),
33
+ dict(topic="training session", old="90 minutes for 25 people", new="60 minutes for 18 people", fixed="beginner level", ask="Give the final duration, audience size, and level.", ideal="Run a 60-minute beginner session for 18 people."),
34
+ dict(topic="catering request", old="vegetarian lunch for 40", new="vegan lunch for 32", fixed="no peanuts", ask="State the current catering requirement in one sentence.", ideal="Provide a peanut-free vegan lunch for 32 people."),
35
+ dict(topic="archive transfer", old="Box A7 to Shelf 4", new="Box C2 to Shelf 9", fixed="scan the index first", ask="Give the current transfer instruction only.", ideal="Scan the index, then move Box C2 to Shelf 9."),
36
+ dict(topic="podcast recording", old="episode 18 on Wednesday", new="episode 21 on Friday", fixed="guest: Dr. Vale", ask="State the latest recording plan in one sentence.", ideal="Record episode 21 with Dr. Vale on Friday."),
37
+ dict(topic="bike rental", old="two city bikes for 3 hours", new="three cargo bikes for 2 hours", fixed="pickup at North Gate", ask="Give the final rental details in one sentence.", ideal="Rent three cargo bikes for two hours, with pickup at North Gate."),
38
+ dict(topic="library display", old="history books in June", new="science magazines in July", fixed="second-floor lobby", ask="State the final display plan without mentioning the old one.", ideal="Display science magazines in the second-floor lobby in July."),
39
+ dict(topic="garden delivery", old="six rose bushes on Monday", new="four maple saplings on Wednesday", fixed="use the east entrance", ask="Give the updated delivery instruction in one sentence.", ideal="Deliver four maple saplings on Wednesday through the east entrance."),
40
+ dict(topic="translation job", old="French, 1,200 words, formal", new="Spanish, 900 words, conversational", fixed="deadline 18:00 UTC", ask="Give only the current language, length, tone, and deadline.", ideal="Translate 900 words into conversational Spanish by 18:00 UTC."),
41
+ ]
42
+
43
+ ALREADY_TRIED = [
44
+ dict(problem="a laptop that connects to Wi-Fi but cannot reach websites", tried=["restarted the laptop", "reconnected to the network"], next=["renew the DHCP lease", "test with a different DNS resolver"]),
45
+ dict(problem="a printer producing faint pages", tried=["replaced the paper", "restarted the printer"], next=["check the ink or toner level", "run the print-head cleaning utility"]),
46
+ dict(problem="a Python service returning a port-in-use error", tried=["restarted the service", "changed the browser"], next=["identify the process holding the port", "stop it or configure a different port"]),
47
+ dict(problem="a phone that will not charge", tried=["rebooted the phone", "tried another wall outlet"], next=["inspect and gently clean the charging port", "test a known-good cable and adapter"]),
48
+ dict(problem="a spreadsheet with formulas showing as text", tried=["reopened the file", "changed the font"], next=["set the cells to General format", "re-enter each formula beginning with an equals sign"]),
49
+ dict(problem="a web page whose new CSS is not visible", tried=["refreshed the page", "restarted the editor"], next=["perform a hard refresh or clear the browser cache", "verify that the edited stylesheet is the one loaded by the page"]),
50
+ dict(problem="a Bluetooth keyboard that pairs but sends no keystrokes", tried=["replaced the batteries", "paired it again"], next=["remove conflicting keyboard profiles", "test the keyboard on another device"]),
51
+ dict(problem="a Git push rejected as non-fast-forward", tried=["retried the push", "checked the internet connection"], next=["fetch and inspect the remote changes", "rebase or merge before pushing again"]),
52
+ dict(problem="a microwave that powers on but does not heat", tried=["used a different outlet", "changed the timer"], next=["stop using it and check the door latch for visible damage", "contact a qualified repair technician"]),
53
+ dict(problem="headphones playing sound on only one side", tried=["restarted the music app", "raised the volume"], next=["check the device balance setting", "test the headphones with another source"]),
54
+ dict(problem="a calendar invitation showing the wrong time zone", tried=["reopened the invitation", "changed the event color"], next=["verify the calendar account time zone", "recreate the event with an explicit zone"]),
55
+ dict(problem="a Docker container exiting immediately", tried=["restarted Docker", "pulled the image again"], next=["inspect the container logs", "run the image with its entrypoint overridden for diagnosis"]),
56
+ dict(problem="a camera whose photos look unexpectedly dark", tried=["charged the battery", "cleaned the lens"], next=["check exposure compensation", "test in aperture-priority mode with a wider aperture"]),
57
+ dict(problem="an e-reader that cannot open one book", tried=["restarted the e-reader", "increased screen brightness"], next=["remove and download the book again", "verify that its format is supported and not corrupted"]),
58
+ dict(problem="a smart speaker that hears commands but plays no audio", tried=["restarted the speaker", "raised the spoken volume"], next=["check whether audio is routed to another device", "unlink and relink the music service"]),
59
+ ]
60
+
61
+ SELECTIVE = [
62
+ dict(context="The Cedar project owner is Nila. Its budget is €18,400, its review is on 12 June, and the prototype color is orange.", question="What is the review date? Answer with the date only.", required=["12 June"], forbidden=["Nila", "18,400", "orange"], ideal="12 June"),
63
+ dict(context="Order RX-4 contains 36 valves, ships by rail, arrives at Terminal B, and is insured for $9,000.", question="Which terminal receives the order? Answer with the terminal only.", required=["Terminal B"], forbidden=["36", "rail", "9,000"], ideal="Terminal B"),
64
+ dict(context="Leah chose the elm desk, Omar chose the glass table, Priya chose the oak shelf, and Zane chose the steel cabinet.", question="What did Priya choose? Give only the item.", required=["oak shelf"], forbidden=["elm", "glass", "steel"], ideal="oak shelf"),
65
+ dict(context="The rehearsal starts at 17:40 in Studio 3. Doors open at 17:10, the conductor is Ivo, and parking closes at 22:00.", question="Where is the rehearsal? Answer with the location only.", required=["Studio 3"], forbidden=["17:40", "Ivo", "22:00"], ideal="Studio 3"),
66
+ dict(context="Dataset Lumen has 8,200 rows, uses a CC BY 4.0 license, was updated in March, and has 14 columns.", question="What license does Dataset Lumen use? Give only the license.", required=["CC BY 4.0"], forbidden=["8,200", "March", "14"], ideal="CC BY 4.0"),
67
+ dict(context="The blue key opens the west cabinet, the brass key opens the archive, and the small black key opens the equipment locker.", question="Which key opens the archive? Answer with the key only.", required=["brass key"], forbidden=["blue", "black", "west"], ideal="brass key"),
68
+ dict(context="Flight 608 departs at 06:25 from Gate 11, boards at 05:50, and allows one 8 kg cabin bag.", question="What is the departure gate? Give only the gate.", required=["Gate 11"], forbidden=["06:25", "05:50", "8 kg"], ideal="Gate 11"),
69
+ dict(context="The Orion room seats 18, the Maple room seats 10, and the Harbor room seats 26. Only Harbor has a projector.", question="Which room has a projector? Give only its name.", required=["Harbor"], forbidden=["18", "10", "26"], ideal="Harbor"),
70
+ dict(context="Invoice 771 is due on 4 August, totals £620, belongs to Kestrel Ltd, and uses purchase order PO-19.", question="What is the purchase-order code? Return only the code.", required=["PO-19"], forbidden=["August", "620", "Kestrel"], ideal="PO-19"),
71
+ dict(context="The north sensor reports humidity, the east sensor reports pressure, and the south sensor reports temperature.", question="Which sensor reports pressure? Answer with the direction only.", required=["east"], forbidden=["north", "south", "temperature"], ideal="east"),
72
+ dict(context="Package M2 weighs 4.3 kg, uses priority shipping, has tracking code TR-882, and contains textiles.", question="Give the tracking code only.", required=["TR-882"], forbidden=["4.3", "priority", "textiles"], ideal="TR-882"),
73
+ dict(context="Aster Café opens at 07:30, stops serving breakfast at 11:00, closes at 19:00, and is closed on Mondays.", question="On which day is the café closed? One word only.", required=["Monday"], forbidden=["07:30", "11:00", "19:00"], ideal="Monday"),
74
+ dict(context="The field notebook is in Drawer 6, the spare battery is in Drawer 2, and the lens cloth is in Drawer 9.", question="Where is the spare battery? Give only the drawer.", required=["Drawer 2"], forbidden=["Drawer 6", "Drawer 9", "notebook"], ideal="Drawer 2"),
75
+ dict(context="Team Quartz uses Matrix for chat, Forgejo for code, MinIO for files, and Jitsi for calls.", question="What does Team Quartz use for code? Answer with the product only.", required=["Forgejo"], forbidden=["Matrix", "MinIO", "Jitsi"], ideal="Forgejo"),
76
+ dict(context="The safety drill coordinator is Hana, the assembly point is Courtyard C, the alarm test is at 15:05, and attendance is 44.", question="Give the assembly point only.", required=["Courtyard C"], forbidden=["Hana", "15:05", "44"], ideal="Courtyard C"),
77
+ ]
78
+
79
+ DELTA = [
80
+ dict(base="The launch plan used a $12,000 budget, a Monday release, and email promotion.", updates=["Budget is now $9,500.", "Release moves to Wednesday."], ideal=["Budget: $9,500", "Release: Wednesday"]),
81
+ dict(base="The course had 10 lessons, weekly quizzes, and a final presentation.", updates=["The course now has 8 lessons.", "The final presentation is replaced by a written reflection."], ideal=["Length: 8 lessons", "Final task: written reflection"]),
82
+ dict(base="The trip used Hotel Nova, train travel, and a four-night stay.", updates=["Accommodation changes to Harbor Hostel.", "The stay is shortened to three nights."], ideal=["Accommodation: Harbor Hostel", "Stay: three nights"]),
83
+ dict(base="The app supported English and German and required Android 12.", updates=["Spanish support is added.", "Minimum Android version changes to 13."], ideal=["Language added: Spanish", "Minimum Android: 13"]),
84
+ dict(base="The event expected 75 guests, used Hall A, and started at 18:00.", updates=["Attendance estimate rises to 90.", "The venue changes to Hall C."], ideal=["Expected guests: 90", "Venue: Hall C"]),
85
+ dict(base="The report covered Q1, used PDF format, and was due Friday.", updates=["Coverage expands to Q1 and Q2.", "Delivery format changes to HTML."], ideal=["Coverage: Q1 and Q2", "Format: HTML"]),
86
+ dict(base="The device used a 2,000 mAh battery, USB-C charging, and a black enclosure.", updates=["Battery capacity increases to 2,600 mAh.", "The enclosure changes to silver."], ideal=["Battery: 2,600 mAh", "Enclosure: silver"]),
87
+ dict(base="The study sampled 120 adults, used phone interviews, and ran for six weeks.", updates=["Sample size increases to 150 adults.", "Interviews move online."], ideal=["Sample: 150 adults", "Interview mode: online"]),
88
+ dict(base="The menu offered soup, pasta, and cake, with service at noon.", updates=["Cake is replaced by fruit.", "Service moves to 13:00."], ideal=["Dessert: fruit", "Service time: 13:00"]),
89
+ dict(base="The backup ran daily at 01:00 and retained files for 14 days.", updates=["The schedule changes to every six hours.", "Retention increases to 30 days."], ideal=["Schedule: every six hours", "Retention: 30 days"]),
90
+ dict(base="The exhibit featured ceramics, opened in April, and occupied Gallery 2.", updates=["Opening moves to May.", "The exhibit moves to Gallery 5."], ideal=["Opening: May", "Location: Gallery 5"]),
91
+ dict(base="The subscription cost $18 monthly and included five team seats.", updates=["Price changes to $20 monthly.", "The plan now includes eight seats."], ideal=["Monthly price: $20", "Seats: eight"]),
92
+ dict(base="The greenhouse target was 22°C, 55% humidity, and morning watering.", updates=["Temperature target drops to 20°C.", "Watering moves to evening."], ideal=["Temperature: 20°C", "Watering: evening"]),
93
+ dict(base="The reading group met biweekly, discussed two chapters, and used Room 8.", updates=["Meetings become weekly.", "The group will discuss one chapter per meeting."], ideal=["Frequency: weekly", "Reading: one chapter per meeting"]),
94
+ dict(base="The API allowed 100 requests per minute and returned XML by default.", updates=["The limit rises to 160 requests per minute.", "The default response changes to JSON."], ideal=["Rate limit: 160 requests per minute", "Default format: JSON"]),
95
+ ]
96
+
97
+ CONFLICTS = [
98
+ ("The venue manager says the room holds 40 people, while the signed booking says 28.", "Which capacity should I put on the invitation?", "Should I use the venue manager’s figure of 40 or the signed booking’s figure of 28?"),
99
+ ("The project brief names Lina as owner, but the approval email names Marcus.", "Who should I list as owner?", "Should Lina or Marcus be treated as the authoritative project owner?"),
100
+ ("One itinerary says the train leaves at 08:10; the ticket says 08:45.", "Tell me when to arrive.", "Which departure time is current: 08:10 on the itinerary or 08:45 on the ticket?"),
101
+ ("The spreadsheet total is €4,720, but the signed quote is €4,270.", "What amount should I approve?", "Which amount is authoritative: €4,720 in the spreadsheet or €4,270 in the signed quote?"),
102
+ ("The label says refrigerate the sample, while the protocol says store it at room temperature.", "Where should I put it?", "Should the sample follow the label’s refrigeration instruction or the protocol’s room-temperature instruction?"),
103
+ ("The calendar lists Tuesday, but the client’s last message says Thursday.", "Schedule the call.", "Should I schedule the call for Tuesday or for Thursday from the client’s latest message?"),
104
+ ("The inventory system shows 16 units, while the physical count found 11.", "How many can I promise?", "Which quantity should govern the promise: 16 in the system or 11 physically counted?"),
105
+ ("The style guide requires sentence case, but the campaign brief asks for title case.", "Which capitalization should I use?", "Should the style guide’s sentence case or the campaign brief’s title case take priority?"),
106
+ ("The map marks the path open, but a ranger’s notice marks it closed.", "Can I plan the route through it?", "Is the ranger’s closure notice current and authoritative over the map?"),
107
+ ("The database record gives Ana’s region as West, while her submitted form says North.", "Which region belongs in the report?", "Should the report use West from the database or North from Ana’s submitted form?"),
108
+ ("The product page promises a two-year warranty, but the box insert says one year.", "What warranty should support quote?", "Which warranty term is authoritative: two years on the product page or one year in the box insert?"),
109
+ ("The teacher’s rubric allows 1,000 words, but the assignment portal shows 800.", "How long should my essay be?", "Should the essay follow the 1,000-word rubric or the 800-word portal limit?"),
110
+ ("The maintenance log says filter type F7, while the installed-unit plate specifies F9.", "Which replacement should I order?", "Should the replacement match F7 in the log or F9 on the installed unit’s plate?"),
111
+ ("The grant call says applications close at 17:00 local time, but the portal countdown implies 16:00.", "Give me the deadline.", "Which deadline is authoritative: 17:00 in the grant call or 16:00 implied by the portal?"),
112
+ ("The recipe card calls for 4 grams of yeast, while the chef’s correction says 7 grams.", "How much yeast should I use?", "Should I follow the original 4 grams or the chef’s corrected 7 grams?"),
113
+ ]
114
+
115
+ SOURCE_BOUNDARY = [
116
+ ("The supplied memo says the pilot had 48 participants and lasted six weeks. It does not report the dropout count.", "How many participants dropped out?", "The dropout count is not provided in the memo, so it cannot be determined from the supplied information."),
117
+ ("The inspection note confirms a cracked seal but gives no cause.", "What caused the seal to crack?", "The note confirms the crack but does not identify its cause."),
118
+ ("The article excerpt says the town planted 2,000 trees. It gives no survival survey.", "How many trees survived?", "The excerpt does not provide survival data, so the number cannot be determined."),
119
+ ("The invoice lists labor and materials but no payment date.", "On what date was it paid?", "No payment date is provided, so the payment date cannot be verified."),
120
+ ("The experiment log records a temperature rise after minute 12 but does not identify the mechanism.", "Which mechanism caused the rise?", "The log shows when the rise occurred but does not establish its mechanism."),
121
+ ("The meeting minutes approve the proposal without recording individual votes.", "Who voted against it?", "Individual votes were not recorded, so the minutes cannot identify who voted against it."),
122
+ ("The product sheet gives a 500-cycle rating under lab conditions but no field-life estimate.", "How many years will it last in normal use?", "The sheet does not provide enough usage data to convert the lab cycle rating into years of normal use."),
123
+ ("The survey summary reports an average score of 7.2 but omits the response count.", "How many people answered?", "The response count is missing, so it cannot be determined from the summary."),
124
+ ("The shipping notice says the parcel left the depot yesterday but has no live tracking update.", "Where is it right now?", "The notice does not contain live location data, so the parcel’s current location cannot be verified."),
125
+ ("The biography states that Noor studied chemistry and later founded a company. It does not name the university.", "Which university did Noor attend?", "The university is not named in the supplied biography."),
126
+ ("The budget shows projected revenue and costs but no actual results.", "What profit did the project make?", "Actual profit cannot be calculated because only projections are provided."),
127
+ ("The case summary says the server failed after an update but contains no logs.", "Which component failed first?", "The summary does not include logs or a failure sequence, so the first failed component cannot be identified."),
128
+ ("The catalog dates the vase to the 1880s but does not name its maker.", "Who made the vase?", "The maker is not identified in the catalog entry."),
129
+ ("The clinical abstract reports improvement but omits the control group’s numbers.", "How much better was treatment than control?", "The difference cannot be calculated without the control group’s results."),
130
+ ("The energy report gives annual consumption but no building floor area.", "What is consumption per square meter?", "Consumption per square meter cannot be calculated without the floor area."),
131
+ ]
132
+
133
+ CONSTRAINTS = [
134
+ ("Draft a reminder about tomorrow’s 09:00 safety briefing.", ["exactly two sentences", "include 'Room 4'", "do not use 'urgent'"], "Tomorrow’s safety briefing begins at 09:00 in Room 4. Please arrive five minutes early."),
135
+ ("List ways to protect a laptop while traveling.", ["exactly three bullet points", "each under eight words", "do not mention insurance"], "- Use a padded, water-resistant sleeve.\n- Keep it in your carry-on.\n- Enable encryption and a screen lock."),
136
+ ("Write a status line for a completed data import.", ["exactly eight words", "start with 'Import complete:'"], "Import complete: all records passed validation without errors."),
137
+ ("Give a compact JSON object for a green task due Friday.", ["valid JSON only", "keys exactly title, color, due"], '{"title":"task","color":"green","due":"Friday"}'),
138
+ ("Decline a lunch invitation politely.", ["one sentence", "include 'thank you'", "do not give an excuse"], "Thank you for inviting me, but I’ll have to decline."),
139
+ ("Summarize the action: Eli will send the draft by noon.", ["start with 'Owner:'", "one line", "include the deadline"], "Owner: Eli — send the draft by noon."),
140
+ ("Describe a calm workspace.", ["exactly six words", "include 'light'"], "Soft light, clear desk, quiet focus."),
141
+ ("Answer whether backups reduce data-loss risk.", ["begin with Yes or No", "then exactly one explanatory sentence"], "Yes. Backups provide a separate copy when original data is lost or damaged."),
142
+ ("Give two reasons to label cables.", ["exactly two numbered items", "no introduction"], "1. Faster troubleshooting.\n2. Safer equipment changes."),
143
+ ("Write a meeting subject about reviewing the Atlas budget.", ["under seven words", "include 'Atlas'", "do not use punctuation"], "Atlas budget review"),
144
+ ("Explain version control to a beginner.", ["one sentence", "use the word history", "under 20 words"], "Version control keeps a history of changes so work can be compared, shared, or restored."),
145
+ ("Provide a CSV header for name, role, and start date.", ["one line only", "exactly three comma-separated fields", "lowercase"], "name,role,start_date"),
146
+ ("Write a friendly confirmation for a 15:30 call.", ["exactly two sentences", "include Tuesday", "do not start with Sure"], "Tuesday at 15:30 works for me. I’ll be ready for the call."),
147
+ ("Give a four-word warning about a wet floor.", ["exactly four words", "end with 'carefully'"], "Wet floor; walk carefully."),
148
+ ("Return the enabled state and retry count.", ["valid JSON only", "keys exactly enabled and retries", "enabled true", "retries 3"], '{"enabled":true,"retries":3}'),
149
+ ]
150
+
151
+
152
+ def split_for(i: int) -> str:
153
+ return "train" if i < 9 else "dev" if i < 12 else "test"
154
+
155
+
156
+ def make_cases() -> list[dict[str, Any]]:
157
+ cases: list[dict[str, Any]] = []
158
+ state_anchors = [
159
+ ["Sunday", "14:00", "Room Cedar"],
160
+ ["3.5", "May 10", "staged rollout"],
161
+ ["180", "green folders", "Dock 2"],
162
+ ["Jon", "11:15", "video call"],
163
+ ["9 adult", "3 child", "16:20"],
164
+ ["Spring Notes", "Thursday", "6,500"],
165
+ ["K-19", "-20", "amber"],
166
+ ["60-minute", "18 people", "beginner"],
167
+ ["vegan", "32", "peanut-free"],
168
+ ["scan the index", "Box C2", "Shelf 9"],
169
+ ["episode 21", "Dr. Vale", "Friday"],
170
+ ["three cargo bikes", "two hours", "North Gate"],
171
+ ["science magazines", "July", "second-floor lobby"],
172
+ ["four maple saplings", "Wednesday", "east entrance"],
173
+ ["900 words", "Spanish", "conversational", "18:00 UTC"],
174
+ ]
175
+ for i, x in enumerate(STATE_UPDATES):
176
+ messages = [
177
+ {"role": "system", "content": SYSTEM},
178
+ {"role": "user", "content": f"Track this {x['topic']}. The initial record is {x['old']}; {x['fixed']}."},
179
+ {"role": "assistant", "content": f"Recorded the initial {x['topic']} details."},
180
+ {"role": "user", "content": f"Correction: replace {x['old']} with {x['new']}. Keep the other detail unchanged."},
181
+ {"role": "assistant", "content": "Understood; I’ll use the corrected value from now on."},
182
+ {"role": "user", "content": x["ask"]},
183
+ ]
184
+ cases.append(dict(id=f"state_update_{i:02d}", split=split_for(i), category="state_update", messages=messages, ideal=x["ideal"], required=state_anchors[i], forbidden=[x["old"]], checks={"max_words": 30, "no_canned_closing": True}))
185
+ for i, x in enumerate(ALREADY_TRIED):
186
+ tried_text = "; ".join(x["tried"])
187
+ messages = [
188
+ {"role": "system", "content": SYSTEM},
189
+ {"role": "user", "content": f"I need help with {x['problem']}. So far I have already {tried_text}. Neither changed the problem."},
190
+ {"role": "assistant", "content": "I’ll avoid sending you through those same steps again."},
191
+ {"role": "user", "content": "Give exactly two next diagnostic actions, as numbered steps. Do not repeat anything I already tried."},
192
+ ]
193
+ ideal = f"1. {x['next'][0].capitalize()}.\n2. {x['next'][1].capitalize()}."
194
+ cases.append(dict(id=f"already_tried_{i:02d}", split=split_for(i), category="already_tried", messages=messages, ideal=ideal, required=x["next"], forbidden=x["tried"], checks={"numbered_items": 2, "max_words": 55, "no_canned_closing": True}))
195
+ for i, x in enumerate(SELECTIVE):
196
+ messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": x["context"]}, {"role": "assistant", "content": "I have the details."}, {"role": "user", "content": x["question"]}]
197
+ cases.append(dict(id=f"selective_{i:02d}", split=split_for(i), category="selective_answer", messages=messages, ideal=x["ideal"], required=x["required"], forbidden=x["forbidden"], checks={"max_words": 8, "no_canned_closing": True}))
198
+ for i, x in enumerate(DELTA):
199
+ messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"Initial summary: {x['base']}"}, {"role": "assistant", "content": "Initial summary recorded."}]
200
+ for u in x["updates"]:
201
+ messages.extend([{"role": "user", "content": u}, {"role": "assistant", "content": "Updated."}])
202
+ messages.append({"role": "user", "content": "List only what changed from the initial summary, using exactly two bullet points. Do not repeat unchanged details."})
203
+ ideal = "\n".join(f"- {z}." for z in x["ideal"])
204
+ unchanged = [p.strip() for p in re.split(r",| and ", x["base"]) if len(p.strip()) > 5]
205
+ cases.append(dict(id=f"delta_{i:02d}", split=split_for(i), category="delta_summary", messages=messages, ideal=ideal, required=x["ideal"], forbidden=unchanged[-1:], checks={"bullet_count": 2, "max_words": 40, "no_canned_closing": True}))
206
+ for i, (ctx, q, ideal) in enumerate(CONFLICTS):
207
+ messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ctx}, {"role": "assistant", "content": "Those two records conflict."}, {"role": "user", "content": q + " Ask one focused clarification instead of choosing a value."}]
208
+ cases.append(dict(id=f"conflict_{i:02d}", split=split_for(i), category="conflict_clarification", messages=messages, ideal=ideal, required=[], forbidden=[], checks={"must_end_question": True, "question_marks": 1, "max_words": 35, "no_canned_closing": True}))
209
+ for i, (ctx, q, ideal) in enumerate(SOURCE_BOUNDARY):
210
+ messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ctx}, {"role": "assistant", "content": "I’ll use only the information in that material."}, {"role": "user", "content": q + " Answer directly without guessing."}]
211
+ cases.append(dict(id=f"source_boundary_{i:02d}", split=split_for(i), category="source_boundary", messages=messages, ideal=ideal, required=[], forbidden=[], checks={"uncertainty_language": True, "max_words": 45, "no_canned_closing": True}))
212
+ for i, (q, constraints, ideal) in enumerate(CONSTRAINTS):
213
+ messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": q}, {"role": "assistant", "content": "What constraints should I follow?"}, {"role": "user", "content": "Constraints: " + "; ".join(constraints) + ". Return only the requested content."}]
214
+ checks: dict[str, Any] = {"no_canned_closing": True}
215
+ text = " ".join(constraints).lower()
216
+ for pat, key in [(r"exactly (\w+) sentences?", "sentence_count"), (r"exactly (\w+) bullet", "bullet_count"), (r"exactly (\w+) numbered", "numbered_items"), (r"exactly (\w+) words?", "exact_words"), (r"(?<!each )under (\w+) words?", "max_words")]:
217
+ m = re.search(pat, text)
218
+ if m:
219
+ nums = {"one": 1, "two": 2, "three": 3, "four": 4, "six": 6, "seven": 7, "eight": 8, "twenty": 20}
220
+ checks[key] = nums.get(m.group(1), int(m.group(1)) if m.group(1).isdigit() else None)
221
+ cases.append(dict(id=f"constraint_{i:02d}", split=split_for(i), category="constraint_stack", messages=messages, ideal=ideal, required=[], forbidden=[], checks=checks))
222
+ return cases
223
+
224
+
225
+ def normalize(text: str) -> str:
226
+ return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9]+", " ", text.lower())).strip()
227
+
228
+
229
+ def repetition_metrics(text: str) -> dict[str, Any]:
230
+ words = re.findall(r"[A-Za-z0-9']+", text.lower())
231
+ grams = Counter(tuple(words[i:i+4]) for i in range(max(0, len(words)-3)))
232
+ sentences = [normalize(x) for x in re.split(r"(?<=[.!?])\s+", text.strip()) if normalize(x)]
233
+ sent_counts = Counter(sentences)
234
+ return {"words": len(words), "unique_ratio": len(set(words))/max(1,len(words)), "max_4gram_count": max(grams.values(), default=1), "max_sentence_count": max(sent_counts.values(), default=1), "loop": max(grams.values(), default=1)>2 or max(sent_counts.values(), default=1)>1 or (len(words)>=25 and len(set(words))/len(words)<0.45)}
235
+
236
+
237
+ def deterministic_reward(case: dict[str, Any], answer: str) -> dict[str, Any]:
238
+ a = answer.strip(); low = a.lower(); failures: list[str] = []
239
+ for req in case.get("required", []):
240
+ if normalize(str(req)) not in normalize(a): failures.append("missing:"+str(req))
241
+ for ban in case.get("forbidden", []):
242
+ if normalize(str(ban)) and normalize(str(ban)) in normalize(a): failures.append("stale_or_irrelevant:"+str(ban))
243
+ c = case.get("checks", {}); words = re.findall(r"\b[\w'-]+\b", a)
244
+ bullets = [x for x in a.splitlines() if re.match(r"^\s*[-*•]\s+", x)]
245
+ numbered = [x for x in a.splitlines() if re.match(r"^\s*\d+[.)]\s+", x)]
246
+ sentences = [x for x in re.split(r"(?<=[.!?])(?:[\"']*)\s+", a) if x.strip()]
247
+ if c.get("max_words") and len(words) > int(c["max_words"]): failures.append("max_words")
248
+ if c.get("exact_words") and len(words) != int(c["exact_words"]): failures.append("exact_words")
249
+ if c.get("bullet_count") and len(bullets) != int(c["bullet_count"]): failures.append("bullet_count")
250
+ if c.get("numbered_items") and len(numbered) != int(c["numbered_items"]): failures.append("numbered_items")
251
+ if c.get("sentence_count") and len(sentences) != int(c["sentence_count"]): failures.append("sentence_count")
252
+ if c.get("must_end_question") and not a.endswith("?"): failures.append("must_end_question")
253
+ if c.get("question_marks") and a.count("?") != int(c["question_marks"]): failures.append("question_marks")
254
+ if c.get("uncertainty_language") and not any(x in low for x in ["not provided", "cannot", "can't", "does not", "missing", "not identified", "not recorded", "not named", "no "]): failures.append("uncertainty_language")
255
+ if c.get("no_canned_closing") and any(x in low for x in ["let me know if", "i hope this helps", "feel free to ask"]): failures.append("canned_closing")
256
+ rep = repetition_metrics(a)
257
+ if rep["loop"]: failures.append("repetition_loop")
258
+ hard = len(failures)
259
+ score = max(-1.0, 1.0 - 0.35*hard)
260
+ return {"score": score, "pass": hard == 0, "failures": failures, **rep}
261
+
262
+
263
+ JUDGE_SYSTEM = """You are a strict evaluator of a compact assistant's information management in a multi-turn dialogue. Compare candidate answers against the conversation, the latest user request, explicit state updates, stale facts, constraints, and the supplied ideal reference. Penalize repeating old information, repeating attempted steps, inventing missing facts, ignoring corrections, verbose restatement, canned closings, and sentence loops. Reward concise direct use of current information. Score each candidate 0-10 for state accuracy, instruction following, relevance, honesty, and non-repetition. Return JSON only."""
264
+
265
+
266
+ def export_jsonl(path: str, rows: list[dict[str, Any]]) -> None:
267
+ with open(path, "w", encoding="utf-8") as f:
268
+ for row in rows: f.write(json.dumps(row, ensure_ascii=False)+"\n")
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/long_dev_gate.json ADDED
@@ -0,0 +1,189 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "leader": {
3
+ "alpha": 0,
4
+ "pass": 3,
5
+ "n": 21,
6
+ "mean_reward": 0.3333333333333334,
7
+ "loops": 0,
8
+ "mean_words": 8.857142857142858,
9
+ "by_category": {
10
+ "already_tried": 0,
11
+ "conflict_clarification": 0,
12
+ "constraint_stack": 2,
13
+ "delta_summary": 0,
14
+ "selective_answer": 1,
15
+ "source_boundary": 0,
16
+ "state_update": 0
17
+ }
18
+ },
19
+ "a0p005": {
20
+ "alpha": 0.005,
21
+ "pass": 3,
22
+ "n": 21,
23
+ "mean_reward": 0.3333333333333334,
24
+ "loops": 0,
25
+ "mean_words": 8.904761904761905,
26
+ "by_category": {
27
+ "already_tried": 0,
28
+ "conflict_clarification": 0,
29
+ "constraint_stack": 2,
30
+ "delta_summary": 0,
31
+ "selective_answer": 1,
32
+ "source_boundary": 0,
33
+ "state_update": 0
34
+ }
35
+ },
36
+ "a0p01": {
37
+ "alpha": 0.01,
38
+ "pass": 3,
39
+ "n": 21,
40
+ "mean_reward": 0.3333333333333334,
41
+ "loops": 0,
42
+ "mean_words": 8.857142857142858,
43
+ "by_category": {
44
+ "already_tried": 0,
45
+ "conflict_clarification": 0,
46
+ "constraint_stack": 2,
47
+ "delta_summary": 0,
48
+ "selective_answer": 1,
49
+ "source_boundary": 0,
50
+ "state_update": 0
51
+ }
52
+ },
53
+ "a0p025": {
54
+ "alpha": 0.025,
55
+ "pass": 3,
56
+ "n": 21,
57
+ "mean_reward": 0.3333333333333334,
58
+ "loops": 0,
59
+ "mean_words": 9.142857142857142,
60
+ "by_category": {
61
+ "already_tried": 0,
62
+ "conflict_clarification": 0,
63
+ "constraint_stack": 2,
64
+ "delta_summary": 0,
65
+ "selective_answer": 1,
66
+ "source_boundary": 0,
67
+ "state_update": 0
68
+ }
69
+ },
70
+ "a0p05": {
71
+ "alpha": 0.05,
72
+ "pass": 3,
73
+ "n": 21,
74
+ "mean_reward": 0.3333333333333334,
75
+ "loops": 0,
76
+ "mean_words": 9.047619047619047,
77
+ "by_category": {
78
+ "already_tried": 0,
79
+ "conflict_clarification": 0,
80
+ "constraint_stack": 2,
81
+ "delta_summary": 0,
82
+ "selective_answer": 1,
83
+ "source_boundary": 0,
84
+ "state_update": 0
85
+ }
86
+ },
87
+ "a0p1": {
88
+ "alpha": 0.1,
89
+ "pass": 3,
90
+ "n": 21,
91
+ "mean_reward": 0.3333333333333334,
92
+ "loops": 0,
93
+ "mean_words": 8.571428571428571,
94
+ "by_category": {
95
+ "already_tried": 0,
96
+ "conflict_clarification": 0,
97
+ "constraint_stack": 2,
98
+ "delta_summary": 0,
99
+ "selective_answer": 1,
100
+ "source_boundary": 0,
101
+ "state_update": 0
102
+ }
103
+ },
104
+ "a0p2": {
105
+ "alpha": 0.2,
106
+ "pass": 3,
107
+ "n": 21,
108
+ "mean_reward": 0.3500000000000001,
109
+ "loops": 0,
110
+ "mean_words": 8.761904761904763,
111
+ "by_category": {
112
+ "already_tried": 0,
113
+ "conflict_clarification": 0,
114
+ "constraint_stack": 2,
115
+ "delta_summary": 0,
116
+ "selective_answer": 1,
117
+ "source_boundary": 0,
118
+ "state_update": 0
119
+ }
120
+ },
121
+ "a0p35": {
122
+ "alpha": 0.35,
123
+ "pass": 3,
124
+ "n": 21,
125
+ "mean_reward": 0.3500000000000001,
126
+ "loops": 0,
127
+ "mean_words": 9.285714285714286,
128
+ "by_category": {
129
+ "already_tried": 0,
130
+ "conflict_clarification": 0,
131
+ "constraint_stack": 2,
132
+ "delta_summary": 0,
133
+ "selective_answer": 1,
134
+ "source_boundary": 0,
135
+ "state_update": 0
136
+ }
137
+ },
138
+ "a0p5": {
139
+ "alpha": 0.5,
140
+ "pass": 3,
141
+ "n": 21,
142
+ "mean_reward": 0.3500000000000001,
143
+ "loops": 0,
144
+ "mean_words": 9.857142857142858,
145
+ "by_category": {
146
+ "already_tried": 0,
147
+ "conflict_clarification": 0,
148
+ "constraint_stack": 2,
149
+ "delta_summary": 0,
150
+ "selective_answer": 1,
151
+ "source_boundary": 0,
152
+ "state_update": 0
153
+ }
154
+ },
155
+ "a0p75": {
156
+ "alpha": 0.75,
157
+ "pass": 3,
158
+ "n": 21,
159
+ "mean_reward": 0.3500000000000001,
160
+ "loops": 0,
161
+ "mean_words": 9.761904761904763,
162
+ "by_category": {
163
+ "already_tried": 0,
164
+ "conflict_clarification": 0,
165
+ "constraint_stack": 2,
166
+ "delta_summary": 0,
167
+ "selective_answer": 1,
168
+ "source_boundary": 0,
169
+ "state_update": 0
170
+ }
171
+ },
172
+ "a1p0": {
173
+ "alpha": 1.0,
174
+ "pass": 4,
175
+ "n": 21,
176
+ "mean_reward": 0.36666666666666675,
177
+ "loops": 0,
178
+ "mean_words": 9.142857142857142,
179
+ "by_category": {
180
+ "already_tried": 0,
181
+ "conflict_clarification": 0,
182
+ "constraint_stack": 3,
183
+ "delta_summary": 0,
184
+ "selective_answer": 1,
185
+ "source_boundary": 0,
186
+ "state_update": 0
187
+ }
188
+ }
189
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_encoded_rollouts.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_encoding_manifest.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "groups": 72,
3
+ "encoded": 216,
4
+ "mean_tokens": 443.5925925925926,
5
+ "mean_response_tokens": 307.73148148148147,
6
+ "seconds": 3.2618279457092285
7
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rl_manifest.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "scanned": 341,
3
+ "symbolically_verified_available": 120,
4
+ "selected": 96,
5
+ "rejections": {
6
+ "verify": 213,
7
+ "length": 7
8
+ },
9
+ "benchmark_family_rows": 0,
10
+ "decontamination": "normalized exact + fixed-gate shared 12-token windows",
11
+ "source": "open-r1/OpenR1-Math-220k",
12
+ "verification": "math_verify(reference solution, reference answer)"
13
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rl_pool.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rollout_manifest.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "groups": 96,
3
+ "splits": {
4
+ "math_train": 72,
5
+ "math_dev": 24
6
+ },
7
+ "correct": 6,
8
+ "incorrect": 186,
9
+ "loops": 221,
10
+ "parse_fail": 0,
11
+ "benchmark_family_rows": 0,
12
+ "reward": "math_verify exact symbolic equivalence; reference +1, correct rollout +1, wrong -0.5, loop extra -0.5",
13
+ "budgie_correct": 6,
14
+ "budgie_incorrect": 186,
15
+ "budgie_loops": 150,
16
+ "reference_loops_ignored": 71
17
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/math_rollouts_scored.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/reason_gate_scores.json ADDED
@@ -0,0 +1,264 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "leader": {
3
+ "gsm8k": {
4
+ "correct": 5,
5
+ "total": 30,
6
+ "items": [
7
+ true,
8
+ false,
9
+ false,
10
+ false,
11
+ false,
12
+ false,
13
+ true,
14
+ false,
15
+ false,
16
+ true,
17
+ false,
18
+ false,
19
+ false,
20
+ false,
21
+ false,
22
+ false,
23
+ false,
24
+ false,
25
+ false,
26
+ false,
27
+ false,
28
+ true,
29
+ false,
30
+ false,
31
+ false,
32
+ false,
33
+ false,
34
+ false,
35
+ false,
36
+ true
37
+ ]
38
+ },
39
+ "math500": {
40
+ "correct": 3,
41
+ "total": 15,
42
+ "items": [
43
+ false,
44
+ false,
45
+ false,
46
+ false,
47
+ false,
48
+ false,
49
+ true,
50
+ true,
51
+ false,
52
+ false,
53
+ false,
54
+ false,
55
+ false,
56
+ true,
57
+ false
58
+ ]
59
+ },
60
+ "arc_challenge": {
61
+ "correct": 11,
62
+ "total": 30,
63
+ "items": [
64
+ true,
65
+ false,
66
+ false,
67
+ false,
68
+ false,
69
+ false,
70
+ false,
71
+ true,
72
+ false,
73
+ false,
74
+ true,
75
+ true,
76
+ true,
77
+ false,
78
+ false,
79
+ false,
80
+ true,
81
+ false,
82
+ false,
83
+ true,
84
+ false,
85
+ true,
86
+ true,
87
+ false,
88
+ false,
89
+ true,
90
+ false,
91
+ true,
92
+ false,
93
+ false
94
+ ]
95
+ },
96
+ "folio": {
97
+ "correct": 13,
98
+ "total": 30,
99
+ "items": [
100
+ true,
101
+ true,
102
+ true,
103
+ true,
104
+ true,
105
+ false,
106
+ false,
107
+ true,
108
+ false,
109
+ true,
110
+ false,
111
+ true,
112
+ true,
113
+ false,
114
+ false,
115
+ false,
116
+ false,
117
+ true,
118
+ false,
119
+ false,
120
+ true,
121
+ false,
122
+ false,
123
+ true,
124
+ false,
125
+ true,
126
+ false,
127
+ false,
128
+ false,
129
+ false
130
+ ]
131
+ }
132
+ },
133
+ "candidate": {
134
+ "gsm8k": {
135
+ "correct": 3,
136
+ "total": 30,
137
+ "items": [
138
+ false,
139
+ false,
140
+ false,
141
+ false,
142
+ false,
143
+ false,
144
+ true,
145
+ false,
146
+ false,
147
+ true,
148
+ false,
149
+ false,
150
+ false,
151
+ false,
152
+ false,
153
+ false,
154
+ false,
155
+ false,
156
+ false,
157
+ false,
158
+ false,
159
+ false,
160
+ false,
161
+ false,
162
+ false,
163
+ false,
164
+ false,
165
+ false,
166
+ false,
167
+ true
168
+ ]
169
+ },
170
+ "math500": {
171
+ "correct": 1,
172
+ "total": 15,
173
+ "items": [
174
+ false,
175
+ false,
176
+ false,
177
+ false,
178
+ false,
179
+ false,
180
+ true,
181
+ false,
182
+ false,
183
+ false,
184
+ false,
185
+ false,
186
+ false,
187
+ false,
188
+ false
189
+ ]
190
+ },
191
+ "arc_challenge": {
192
+ "correct": 11,
193
+ "total": 30,
194
+ "items": [
195
+ true,
196
+ false,
197
+ false,
198
+ false,
199
+ false,
200
+ false,
201
+ false,
202
+ true,
203
+ false,
204
+ false,
205
+ true,
206
+ true,
207
+ true,
208
+ false,
209
+ false,
210
+ false,
211
+ true,
212
+ false,
213
+ false,
214
+ true,
215
+ false,
216
+ true,
217
+ true,
218
+ false,
219
+ false,
220
+ true,
221
+ false,
222
+ true,
223
+ false,
224
+ false
225
+ ]
226
+ },
227
+ "folio": {
228
+ "correct": 13,
229
+ "total": 30,
230
+ "items": [
231
+ true,
232
+ true,
233
+ true,
234
+ true,
235
+ true,
236
+ false,
237
+ false,
238
+ true,
239
+ false,
240
+ true,
241
+ false,
242
+ true,
243
+ true,
244
+ false,
245
+ false,
246
+ false,
247
+ false,
248
+ true,
249
+ false,
250
+ false,
251
+ true,
252
+ false,
253
+ false,
254
+ true,
255
+ false,
256
+ true,
257
+ false,
258
+ false,
259
+ false,
260
+ false
261
+ ]
262
+ }
263
+ }
264
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/rl_rollouts.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/rollout_manifest.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cases": 63,
3
+ "budgie_rollouts": 252,
4
+ "manual_demonstrations": 63,
5
+ "mean_reward_budgie": -0.21686507936507934,
6
+ "mean_reward_ideal": 0.9992063492063492,
7
+ "positive_budgie": 4,
8
+ "budgie_deterministic_pass": 21,
9
+ "budgie_loops": 4,
10
+ "parse_errors": [],
11
+ "seconds": 935.0733463764191
12
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/stage1_decision.json ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "date": "2026-08-26",
3
+ "stage": "information-management RL v1 pilot",
4
+ "base": "verified-math-a025",
5
+ "status": "preliminary candidate; protected reasoning gate pending; do not promote",
6
+ "curriculum": {
7
+ "rows": 105,
8
+ "splits": {
9
+ "train": 63,
10
+ "dev": 21,
11
+ "test": 21
12
+ },
13
+ "categories": {
14
+ "state_update": 15,
15
+ "already_tried": 15,
16
+ "selective_answer": 15,
17
+ "delta_summary": 15,
18
+ "conflict_clarification": 15,
19
+ "source_boundary": 15,
20
+ "constraint_stack": 15
21
+ },
22
+ "ideal_reference_pass": 105,
23
+ "benchmark_family_rows": 0,
24
+ "parameter_count_change": 0
25
+ },
26
+ "rollouts": {
27
+ "cases": 63,
28
+ "budgie_rollouts": 252,
29
+ "manual_demonstrations": 63,
30
+ "mean_reward_budgie": -0.21686507936507934,
31
+ "mean_reward_ideal": 0.9992063492063492,
32
+ "positive_budgie": 4,
33
+ "budgie_deterministic_pass": 21,
34
+ "budgie_loops": 4,
35
+ "parse_errors": [],
36
+ "seconds": 935.0733463764191
37
+ },
38
+ "training": {
39
+ "first_grpo": {
40
+ "algorithm": "offline clipped token-level GRPO/PPO",
41
+ "base": "verified-math-a025",
42
+ "base_sha256": "7404de2c08df968f41800d1362704a0ad4106ea0116cd0cd4b1709612a2a66be",
43
+ "samples": 315,
44
+ "groups": 63,
45
+ "updates": 79,
46
+ "grad_accum": 4,
47
+ "peak_lr": 1e-07,
48
+ "clip": 0.1,
49
+ "kl_beta": 0.03,
50
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_grpo_source.pt",
51
+ "seconds": 22.73498511314392,
52
+ "parameter_count_change": 0
53
+ },
54
+ "long_grpo": {
55
+ "algorithm": "3-epoch clipped GRPO/PPO plus 0.15 manual-demonstration BC",
56
+ "base": "verified-math-a025",
57
+ "samples_per_epoch": 315,
58
+ "epochs": 3,
59
+ "updates": 237,
60
+ "peak_lr": 5e-07,
61
+ "clip": 0.1,
62
+ "kl_beta": 0.05,
63
+ "bc_coef": 0.15,
64
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_grpo_long_source.pt",
65
+ "seconds": 63.36139249801636,
66
+ "parameter_count_change": 0
67
+ },
68
+ "hybrid_manual_consolidation": {
69
+ "base": "long GRPO source from verified-math-a025",
70
+ "method": "8-epoch manual ideal-response consolidation after GRPO",
71
+ "rows": 63,
72
+ "updates": 126,
73
+ "peak_lr": 2e-06,
74
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_hybrid_source.pt",
75
+ "seconds": 33.85362410545349,
76
+ "parameter_count_change": 0
77
+ }
78
+ },
79
+ "dev": {
80
+ "leader": {
81
+ "alpha": 0,
82
+ "pass": 3,
83
+ "n": 21,
84
+ "mean_reward": 0.3333333333333334,
85
+ "loops": 0,
86
+ "mean_words": 8.857142857142858,
87
+ "by_category": {
88
+ "already_tried": 0,
89
+ "conflict_clarification": 0,
90
+ "constraint_stack": 2,
91
+ "delta_summary": 0,
92
+ "selective_answer": 1,
93
+ "source_boundary": 0,
94
+ "state_update": 0
95
+ }
96
+ },
97
+ "a0p025": {
98
+ "alpha": 0.025,
99
+ "pass": 4,
100
+ "n": 21,
101
+ "mean_reward": 0.36666666666666675,
102
+ "loops": 0,
103
+ "mean_words": 8.571428571428571,
104
+ "by_category": {
105
+ "already_tried": 0,
106
+ "conflict_clarification": 0,
107
+ "constraint_stack": 3,
108
+ "delta_summary": 0,
109
+ "selective_answer": 1,
110
+ "source_boundary": 0,
111
+ "state_update": 0
112
+ }
113
+ },
114
+ "full_source": {
115
+ "alpha": 1.0,
116
+ "pass": 7,
117
+ "n": 21,
118
+ "mean_reward": 0.5166666666666667,
119
+ "loops": 2,
120
+ "mean_words": 13.952380952380953,
121
+ "by_category": {
122
+ "already_tried": 0,
123
+ "conflict_clarification": 3,
124
+ "constraint_stack": 2,
125
+ "delta_summary": 0,
126
+ "selective_answer": 2,
127
+ "source_boundary": 0,
128
+ "state_update": 0
129
+ }
130
+ }
131
+ },
132
+ "test": {
133
+ "leader": {
134
+ "alpha": 0,
135
+ "pass": 1,
136
+ "n": 21,
137
+ "mean_reward": 0.23333333333333345,
138
+ "loops": 1,
139
+ "mean_words": 12.761904761904763,
140
+ "by_category": {
141
+ "already_tried": 0,
142
+ "conflict_clarification": 0,
143
+ "constraint_stack": 1,
144
+ "delta_summary": 0,
145
+ "selective_answer": 0,
146
+ "source_boundary": 0,
147
+ "state_update": 0
148
+ }
149
+ },
150
+ "hybrid_a0p025": {
151
+ "alpha": 0.025,
152
+ "pass": 2,
153
+ "n": 21,
154
+ "mean_reward": 0.2500000000000001,
155
+ "loops": 1,
156
+ "mean_words": 13.714285714285714,
157
+ "by_category": {
158
+ "already_tried": 0,
159
+ "conflict_clarification": 0,
160
+ "constraint_stack": 1,
161
+ "delta_summary": 0,
162
+ "selective_answer": 0,
163
+ "source_boundary": 1,
164
+ "state_update": 0
165
+ }
166
+ },
167
+ "hybrid_a1p0": {
168
+ "alpha": 1.0,
169
+ "pass": 5,
170
+ "n": 21,
171
+ "mean_reward": 0.4214285714285715,
172
+ "loops": 2,
173
+ "mean_words": 11.666666666666666,
174
+ "by_category": {
175
+ "already_tried": 0,
176
+ "conflict_clarification": 2,
177
+ "constraint_stack": 1,
178
+ "delta_summary": 0,
179
+ "selective_answer": 1,
180
+ "source_boundary": 1,
181
+ "state_update": 0
182
+ }
183
+ }
184
+ },
185
+ "decision": "Retain a0p025 only as a preliminary low-drift candidate. It improves deterministic dev 3/21 to 4/21 and test 1/21 to 2/21 without increasing loop counts. Full source improves more but increases loops and is unprotected. No root change before GSM/MATH/ARC/FOLIO/HelpSteer gates.",
186
+ "parameter_count_change": 0
187
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/training_report.json ADDED
@@ -0,0 +1,728 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "algorithm": "offline clipped token-level GRPO/PPO",
3
+ "base": "verified-math-a025",
4
+ "base_sha256": "7404de2c08df968f41800d1362704a0ad4106ea0116cd0cd4b1709612a2a66be",
5
+ "samples": 315,
6
+ "groups": 63,
7
+ "updates": 79,
8
+ "grad_accum": 4,
9
+ "peak_lr": 1e-07,
10
+ "clip": 0.1,
11
+ "kl_beta": 0.03,
12
+ "history": [
13
+ {
14
+ "update": 1,
15
+ "policy_loss": -0.23037529736757278,
16
+ "kl": 0.0011916453659068793,
17
+ "clipfrac": 0.04411764722317457,
18
+ "ratio": 0.9927704781293869,
19
+ "grad_norm": 30.026437759399414,
20
+ "lr": 1.6666666666666664e-08
21
+ },
22
+ {
23
+ "update": 2,
24
+ "policy_loss": 0.6626787334680557,
25
+ "kl": 0.0010778388532344252,
26
+ "clipfrac": 0.0470588244497776,
27
+ "ratio": 1.0042662024497986,
28
+ "grad_norm": 36.82639694213867,
29
+ "lr": 3.333333333333333e-08
30
+ },
31
+ {
32
+ "update": 3,
33
+ "policy_loss": 0.47056204825639725,
34
+ "kl": 0.0007909220657893457,
35
+ "clipfrac": 0.059259260073304176,
36
+ "ratio": 1.0064422488212585,
37
+ "grad_norm": 55.595428466796875,
38
+ "lr": 5e-08
39
+ },
40
+ {
41
+ "update": 4,
42
+ "policy_loss": -0.20021970197558403,
43
+ "kl": 0.0012040091241942719,
44
+ "clipfrac": 0.05199212767183781,
45
+ "ratio": 1.003753125667572,
46
+ "grad_norm": 78.22858428955078,
47
+ "lr": 6.666666666666665e-08
48
+ },
49
+ {
50
+ "update": 5,
51
+ "policy_loss": 0.559862419962883,
52
+ "kl": 0.0005764995075878687,
53
+ "clipfrac": 0.019047619309276342,
54
+ "ratio": 1.0058188140392303,
55
+ "grad_norm": 23.515090942382812,
56
+ "lr": 8.333333333333334e-08
57
+ },
58
+ {
59
+ "update": 6,
60
+ "policy_loss": 0.3895067647099495,
61
+ "kl": 0.00040326562339032535,
62
+ "clipfrac": 0.0,
63
+ "ratio": 0.996701791882515,
64
+ "grad_norm": 15.766783714294434,
65
+ "lr": 1e-07
66
+ },
67
+ {
68
+ "update": 7,
69
+ "policy_loss": 0.5787125676870346,
70
+ "kl": 0.0006672931049251929,
71
+ "clipfrac": 0.035511364694684744,
72
+ "ratio": 1.004628300666809,
73
+ "grad_norm": 20.49469566345215,
74
+ "lr": 1e-07
75
+ },
76
+ {
77
+ "update": 8,
78
+ "policy_loss": 0.5522924363613129,
79
+ "kl": 0.0006476888374891132,
80
+ "clipfrac": 0.022727273404598236,
81
+ "ratio": 1.0048147141933441,
82
+ "grad_norm": 29.801023483276367,
83
+ "lr": 9.995954941722896e-08
84
+ },
85
+ {
86
+ "update": 9,
87
+ "policy_loss": -0.7444212883710861,
88
+ "kl": 0.0006780616167816333,
89
+ "clipfrac": 0.027472528629004955,
90
+ "ratio": 0.998538613319397,
91
+ "grad_norm": 67.31665802001953,
92
+ "lr": 9.983827466889918e-08
93
+ },
94
+ {
95
+ "update": 10,
96
+ "policy_loss": -0.7285518050193787,
97
+ "kl": 0.0006317672196018975,
98
+ "clipfrac": 0.030555556528270245,
99
+ "ratio": 1.0039662569761276,
100
+ "grad_norm": 301.2376708984375,
101
+ "lr": 9.963640660838694e-08
102
+ },
103
+ {
104
+ "update": 11,
105
+ "policy_loss": -0.10094080120325089,
106
+ "kl": 0.0004800117530976422,
107
+ "clipfrac": 0.0,
108
+ "ratio": 0.993068590760231,
109
+ "grad_norm": 124.36708068847656,
110
+ "lr": 9.935432950301885e-08
111
+ },
112
+ {
113
+ "update": 12,
114
+ "policy_loss": 0.34706523083150387,
115
+ "kl": 0.0010870432306546718,
116
+ "clipfrac": 0.05890485271811485,
117
+ "ratio": 1.0028632283210754,
118
+ "grad_norm": 22.80665397644043,
119
+ "lr": 9.899258030259716e-08
120
+ },
121
+ {
122
+ "update": 13,
123
+ "policy_loss": -0.14341779053211212,
124
+ "kl": 0.0010440356709295884,
125
+ "clipfrac": 0.06430905871093273,
126
+ "ratio": 1.0097874402999878,
127
+ "grad_norm": 71.0000228881836,
128
+ "lr": 9.85518476172854e-08
129
+ },
130
+ {
131
+ "update": 14,
132
+ "policy_loss": -0.14554723352193832,
133
+ "kl": 0.00042200443931506015,
134
+ "clipfrac": 0.0,
135
+ "ratio": 1.0001006871461868,
136
+ "grad_norm": 97.95014953613281,
137
+ "lr": 9.803297040679963e-08
138
+ },
139
+ {
140
+ "update": 15,
141
+ "policy_loss": -0.05999794602394104,
142
+ "kl": 0.004065853987412993,
143
+ "clipfrac": 0.10000000149011612,
144
+ "ratio": 1.0324535220861435,
145
+ "grad_norm": 481.89080810546875,
146
+ "lr": 9.74369363834011e-08
147
+ },
148
+ {
149
+ "update": 16,
150
+ "policy_loss": -0.09696632623672485,
151
+ "kl": 0.0011448013974586502,
152
+ "clipfrac": 0.047619049437344074,
153
+ "ratio": 1.0043786019086838,
154
+ "grad_norm": 80.94847106933594,
155
+ "lr": 9.676488013172969e-08
156
+ },
157
+ {
158
+ "update": 17,
159
+ "policy_loss": -0.7218086570501328,
160
+ "kl": 0.0009463204260100611,
161
+ "clipfrac": 0.03333333507180214,
162
+ "ratio": 0.993640273809433,
163
+ "grad_norm": 202.5080108642578,
164
+ "lr": 9.601808094905762e-08
165
+ },
166
+ {
167
+ "update": 18,
168
+ "policy_loss": -0.07248872518539429,
169
+ "kl": 0.0003838754564640112,
170
+ "clipfrac": 0.0,
171
+ "ratio": 0.9967044144868851,
172
+ "grad_norm": 218.25973510742188,
173
+ "lr": 9.519796041007442e-08
174
+ },
175
+ {
176
+ "update": 19,
177
+ "policy_loss": 0.4126771241426468,
178
+ "kl": 0.0008437502183369361,
179
+ "clipfrac": 0.03125,
180
+ "ratio": 1.0107713341712952,
181
+ "grad_norm": 44.18994140625,
182
+ "lr": 9.430607966083865e-08
183
+ },
184
+ {
185
+ "update": 20,
186
+ "policy_loss": 0.5212756395339966,
187
+ "kl": 0.0004093618626939133,
188
+ "clipfrac": 0.0,
189
+ "ratio": 1.0007718205451965,
190
+ "grad_norm": 22.770496368408203,
191
+ "lr": 9.334413644704764e-08
192
+ },
193
+ {
194
+ "update": 21,
195
+ "policy_loss": 0.6261144056916237,
196
+ "kl": 0.0006374888398568146,
197
+ "clipfrac": 0.03125,
198
+ "ratio": 0.9935241192579269,
199
+ "grad_norm": 22.040185928344727,
200
+ "lr": 9.231396188228215e-08
201
+ },
202
+ {
203
+ "update": 22,
204
+ "policy_loss": -1.3446397483348846,
205
+ "kl": 0.002206062665209174,
206
+ "clipfrac": 0.078125,
207
+ "ratio": 0.9838357716798782,
208
+ "grad_norm": 233.2508544921875,
209
+ "lr": 9.121751696237751e-08
210
+ },
211
+ {
212
+ "update": 23,
213
+ "policy_loss": -0.17602678388357162,
214
+ "kl": 0.0007956790504977107,
215
+ "clipfrac": 0.038352273404598236,
216
+ "ratio": 1.0005210638046265,
217
+ "grad_norm": 46.34673309326172,
218
+ "lr": 9.005688883255656e-08
219
+ },
220
+ {
221
+ "update": 24,
222
+ "policy_loss": -0.7793388739228249,
223
+ "kl": 0.0013599027952295728,
224
+ "clipfrac": 0.0833333358168602,
225
+ "ratio": 1.0021197348833084,
226
+ "grad_norm": 81.56828308105469,
227
+ "lr": 8.883428681443027e-08
228
+ },
229
+ {
230
+ "update": 25,
231
+ "policy_loss": -0.6576565504074097,
232
+ "kl": 0.0006211905056261458,
233
+ "clipfrac": 0.018295403569936752,
234
+ "ratio": 0.9970162659883499,
235
+ "grad_norm": 97.28546142578125,
236
+ "lr": 8.755203820042827e-08
237
+ },
238
+ {
239
+ "update": 26,
240
+ "policy_loss": 0.4109273701906204,
241
+ "kl": 0.0009198562474921346,
242
+ "clipfrac": 0.02083333395421505,
243
+ "ratio": 0.999998077750206,
244
+ "grad_norm": 15.58299732208252,
245
+ "lr": 8.621258382366556e-08
246
+ },
247
+ {
248
+ "update": 27,
249
+ "policy_loss": 0.3947326987981796,
250
+ "kl": 0.00088205451174872,
251
+ "clipfrac": 0.03818785585463047,
252
+ "ratio": 1.0055690109729767,
253
+ "grad_norm": 13.190741539001465,
254
+ "lr": 8.481847341167791e-08
255
+ },
256
+ {
257
+ "update": 28,
258
+ "policy_loss": -0.17237670719623566,
259
+ "kl": 0.0007001994817983359,
260
+ "clipfrac": 0.008928571827709675,
261
+ "ratio": 0.9994656294584274,
262
+ "grad_norm": 70.97956848144531,
263
+ "lr": 8.337236073287062e-08
264
+ },
265
+ {
266
+ "update": 29,
267
+ "policy_loss": -0.7233755439519882,
268
+ "kl": 0.0011043671402148902,
269
+ "clipfrac": 0.02475845441222191,
270
+ "ratio": 0.9997147768735886,
271
+ "grad_norm": 61.6943473815918,
272
+ "lr": 8.187699854491945e-08
273
+ },
274
+ {
275
+ "update": 30,
276
+ "policy_loss": -0.11639179289340973,
277
+ "kl": 0.0014169361893436871,
278
+ "clipfrac": 0.07696007937192917,
279
+ "ratio": 1.0042703598737717,
280
+ "grad_norm": 64.11062622070312,
281
+ "lr": 8.033523335474001e-08
282
+ },
283
+ {
284
+ "update": 31,
285
+ "policy_loss": 0.4638165980577469,
286
+ "kl": 0.0011379004718037322,
287
+ "clipfrac": 0.08125000074505806,
288
+ "ratio": 0.9928393661975861,
289
+ "grad_norm": 20.34388542175293,
290
+ "lr": 7.875e-08
291
+ },
292
+ {
293
+ "update": 32,
294
+ "policy_loss": 0.5675623491406441,
295
+ "kl": 0.0006073993426980451,
296
+ "clipfrac": 0.01923076994717121,
297
+ "ratio": 0.9995690137147903,
298
+ "grad_norm": 47.69286346435547,
299
+ "lr": 7.712431606248893e-08
300
+ },
301
+ {
302
+ "update": 33,
303
+ "policy_loss": 0.5257612317800522,
304
+ "kl": 0.0006966667206143029,
305
+ "clipfrac": 0.03125,
306
+ "ratio": 1.0064667463302612,
307
+ "grad_norm": 29.840736389160156,
308
+ "lr": 7.546127612397972e-08
309
+ },
310
+ {
311
+ "update": 34,
312
+ "policy_loss": -0.793354719877243,
313
+ "kl": 0.00046434551404672675,
314
+ "clipfrac": 0.015625,
315
+ "ratio": 0.9924948662519455,
316
+ "grad_norm": 213.46080017089844,
317
+ "lr": 7.376404587551631e-08
318
+ },
319
+ {
320
+ "update": 35,
321
+ "policy_loss": 0.4756254553794861,
322
+ "kl": 0.0007708487610216253,
323
+ "clipfrac": 0.04973118379712105,
324
+ "ratio": 1.008134737610817,
325
+ "grad_norm": 27.62037467956543,
326
+ "lr": 7.203585609134093e-08
327
+ },
328
+ {
329
+ "update": 36,
330
+ "policy_loss": -0.8083227314054966,
331
+ "kl": 0.004024053458124399,
332
+ "clipfrac": 0.08997252862900496,
333
+ "ratio": 1.0261076390743256,
334
+ "grad_norm": 50.15277862548828,
335
+ "lr": 7.02799964789316e-08
336
+ },
337
+ {
338
+ "update": 37,
339
+ "policy_loss": 0.5223310142755508,
340
+ "kl": 0.001121183086070232,
341
+ "clipfrac": 0.05694444663822651,
342
+ "ratio": 0.9989928007125854,
343
+ "grad_norm": 42.304046630859375,
344
+ "lr": 6.849980941685713e-08
345
+ },
346
+ {
347
+ "update": 38,
348
+ "policy_loss": -0.1972136739641428,
349
+ "kl": 0.0007333776520681567,
350
+ "clipfrac": 0.01923076994717121,
351
+ "ratio": 1.004689410328865,
352
+ "grad_norm": 233.6256866455078,
353
+ "lr": 6.669868359236936e-08
354
+ },
355
+ {
356
+ "update": 39,
357
+ "policy_loss": 0.43520450592041016,
358
+ "kl": 0.0013158429501345381,
359
+ "clipfrac": 0.05902777798473835,
360
+ "ratio": 0.9904992878437042,
361
+ "grad_norm": 32.244903564453125,
362
+ "lr": 6.488004755084454e-08
363
+ },
364
+ {
365
+ "update": 40,
366
+ "policy_loss": 0.5257853269577026,
367
+ "kl": 0.000648263670882443,
368
+ "clipfrac": 0.007575757801532745,
369
+ "ratio": 1.0029462575912476,
370
+ "grad_norm": 27.229291915893555,
371
+ "lr": 6.304736316935218e-08
372
+ },
373
+ {
374
+ "update": 41,
375
+ "policy_loss": 0.6121181920170784,
376
+ "kl": 0.0007790233794366941,
377
+ "clipfrac": 0.03467908967286348,
378
+ "ratio": 1.003097340464592,
379
+ "grad_norm": 20.22237777709961,
380
+ "lr": 6.120411906677546e-08
381
+ },
382
+ {
383
+ "update": 42,
384
+ "policy_loss": 0.4218008816242218,
385
+ "kl": 0.0008195128320949152,
386
+ "clipfrac": 0.02083333395421505,
387
+ "ratio": 0.9937786608934402,
388
+ "grad_norm": 26.536725997924805,
389
+ "lr": 5.935382396302678e-08
390
+ },
391
+ {
392
+ "update": 43,
393
+ "policy_loss": -0.1556526944041252,
394
+ "kl": 0.0006042122186045162,
395
+ "clipfrac": 0.0,
396
+ "ratio": 0.9939344972372055,
397
+ "grad_norm": 73.46481323242188,
398
+ "lr": 5.749999999999999e-08
399
+ },
400
+ {
401
+ "update": 44,
402
+ "policy_loss": -0.15321169793605804,
403
+ "kl": 0.0016226090519921854,
404
+ "clipfrac": 0.0634319398086518,
405
+ "ratio": 1.0012177377939224,
406
+ "grad_norm": 52.92159652709961,
407
+ "lr": 5.5646176036973224e-08
408
+ },
409
+ {
410
+ "update": 45,
411
+ "policy_loss": -0.03745938837528229,
412
+ "kl": 0.0008766282699070871,
413
+ "clipfrac": 0.03510518837720156,
414
+ "ratio": 1.0012139827013016,
415
+ "grad_norm": 59.688804626464844,
416
+ "lr": 5.379588093322453e-08
417
+ },
418
+ {
419
+ "update": 46,
420
+ "policy_loss": 0.671097069978714,
421
+ "kl": 0.00051483995775925,
422
+ "clipfrac": 0.020869565196335316,
423
+ "ratio": 1.0009578615427017,
424
+ "grad_norm": 21.906808853149414,
425
+ "lr": 5.195263683064781e-08
426
+ },
427
+ {
428
+ "update": 47,
429
+ "policy_loss": -0.13393858075141907,
430
+ "kl": 0.0006431687688746024,
431
+ "clipfrac": 0.0,
432
+ "ratio": 1.0134539604187012,
433
+ "grad_norm": 44.802223205566406,
434
+ "lr": 5.011995244915546e-08
435
+ },
436
+ {
437
+ "update": 48,
438
+ "policy_loss": 0.3311524875462055,
439
+ "kl": 0.0007468888070434332,
440
+ "clipfrac": 0.01785714365541935,
441
+ "ratio": 0.9904734790325165,
442
+ "grad_norm": 18.64971351623535,
443
+ "lr": 4.8301316407630634e-08
444
+ },
445
+ {
446
+ "update": 49,
447
+ "policy_loss": 0.46640586853027344,
448
+ "kl": 0.0006929267983650789,
449
+ "clipfrac": 0.025921659544110298,
450
+ "ratio": 0.9968485832214355,
451
+ "grad_norm": 33.951053619384766,
452
+ "lr": 4.6500190583142864e-08
453
+ },
454
+ {
455
+ "update": 50,
456
+ "policy_loss": 0.5548064038157463,
457
+ "kl": 0.0006275954219745472,
458
+ "clipfrac": 0.02380952425301075,
459
+ "ratio": 1.001102477312088,
460
+ "grad_norm": 29.941875457763672,
461
+ "lr": 4.4720003521068395e-08
462
+ },
463
+ {
464
+ "update": 51,
465
+ "policy_loss": 0.7612060010433197,
466
+ "kl": 0.0006664514767180663,
467
+ "clipfrac": 0.01785714365541935,
468
+ "ratio": 1.0138370990753174,
469
+ "grad_norm": 66.48155975341797,
470
+ "lr": 4.296414390865909e-08
471
+ },
472
+ {
473
+ "update": 52,
474
+ "policy_loss": -0.08108541369438171,
475
+ "kl": 0.0020505651373241562,
476
+ "clipfrac": 0.05263157933950424,
477
+ "ratio": 0.9996013641357422,
478
+ "grad_norm": 138.5041046142578,
479
+ "lr": 4.1235954124483686e-08
480
+ },
481
+ {
482
+ "update": 53,
483
+ "policy_loss": -0.8076933175325394,
484
+ "kl": 0.0024614080612082034,
485
+ "clipfrac": 0.12132352963089943,
486
+ "ratio": 1.017799660563469,
487
+ "grad_norm": 212.42625427246094,
488
+ "lr": 3.9538723876020276e-08
489
+ },
490
+ {
491
+ "update": 54,
492
+ "policy_loss": 0.4771292805671692,
493
+ "kl": 0.000937689357670024,
494
+ "clipfrac": 0.015476190950721502,
495
+ "ratio": 1.0008769780397415,
496
+ "grad_norm": 26.762781143188477,
497
+ "lr": 3.787568393751105e-08
498
+ },
499
+ {
500
+ "update": 55,
501
+ "policy_loss": -0.33603056613355875,
502
+ "kl": 0.001213077048305422,
503
+ "clipfrac": 0.04554655961692333,
504
+ "ratio": 0.9982029944658279,
505
+ "grad_norm": 41.6628532409668,
506
+ "lr": 3.6250000000000004e-08
507
+ },
508
+ {
509
+ "update": 56,
510
+ "policy_loss": -1.3424333781003952,
511
+ "kl": 0.0010098835336975753,
512
+ "clipfrac": 0.07500000298023224,
513
+ "ratio": 0.9831256121397018,
514
+ "grad_norm": 105.74688720703125,
515
+ "lr": 3.4664766645259996e-08
516
+ },
517
+ {
518
+ "update": 57,
519
+ "policy_loss": 0.5214753597974777,
520
+ "kl": 0.00044564168274519034,
521
+ "clipfrac": 0.027529762126505375,
522
+ "ratio": 1.0039145648479462,
523
+ "grad_norm": 20.75997543334961,
524
+ "lr": 3.3123001455080536e-08
525
+ },
526
+ {
527
+ "update": 58,
528
+ "policy_loss": -0.1294296830892563,
529
+ "kl": 0.0009099526942009106,
530
+ "clipfrac": 0.008928571827709675,
531
+ "ratio": 1.0007587224245071,
532
+ "grad_norm": 47.566001892089844,
533
+ "lr": 3.162763926712937e-08
534
+ },
535
+ {
536
+ "update": 59,
537
+ "policy_loss": -0.11015290021896362,
538
+ "kl": 0.0014910743702785112,
539
+ "clipfrac": 0.011904762126505375,
540
+ "ratio": 0.9988532066345215,
541
+ "grad_norm": 74.82030487060547,
542
+ "lr": 3.018152658832208e-08
543
+ },
544
+ {
545
+ "update": 60,
546
+ "policy_loss": -0.026217177510261536,
547
+ "kl": 0.0006827369434176944,
548
+ "clipfrac": 0.0243055559694767,
549
+ "ratio": 1.0057201236486435,
550
+ "grad_norm": 54.335819244384766,
551
+ "lr": 2.8787416176334434e-08
552
+ },
553
+ {
554
+ "update": 61,
555
+ "policy_loss": -0.14155735075473785,
556
+ "kl": 0.0009369388862978667,
557
+ "clipfrac": 0.04880952462553978,
558
+ "ratio": 0.9952247589826584,
559
+ "grad_norm": 35.135799407958984,
560
+ "lr": 2.744796179957173e-08
561
+ },
562
+ {
563
+ "update": 62,
564
+ "policy_loss": -0.0782356783747673,
565
+ "kl": 0.0008571091893827543,
566
+ "clipfrac": 0.02500000037252903,
567
+ "ratio": 1.0032548010349274,
568
+ "grad_norm": 68.35108184814453,
569
+ "lr": 2.6165713185569727e-08
570
+ },
571
+ {
572
+ "update": 63,
573
+ "policy_loss": -0.12117467820644379,
574
+ "kl": 0.0009687186102382839,
575
+ "clipfrac": 0.05000000074505806,
576
+ "ratio": 1.0031340569257736,
577
+ "grad_norm": 31.594810485839844,
578
+ "lr": 2.4943111167443438e-08
579
+ },
580
+ {
581
+ "update": 64,
582
+ "policy_loss": 0.3857679981738329,
583
+ "kl": 0.0006399283593054861,
584
+ "clipfrac": 0.01515151560306549,
585
+ "ratio": 1.0012485682964325,
586
+ "grad_norm": 19.391820907592773,
587
+ "lr": 2.378248303762251e-08
588
+ },
589
+ {
590
+ "update": 65,
591
+ "policy_loss": -0.7923163399100304,
592
+ "kl": 0.001449741903343238,
593
+ "clipfrac": 0.06985780596733093,
594
+ "ratio": 1.0021595358848572,
595
+ "grad_norm": 348.50360107421875,
596
+ "lr": 2.2686038117717844e-08
597
+ },
598
+ {
599
+ "update": 66,
600
+ "policy_loss": -0.08538608253002167,
601
+ "kl": 0.0008700684120412916,
602
+ "clipfrac": 0.056845239363610744,
603
+ "ratio": 0.995424211025238,
604
+ "grad_norm": 76.93407440185547,
605
+ "lr": 2.1655863552952364e-08
606
+ },
607
+ {
608
+ "update": 67,
609
+ "policy_loss": 0.44711967557668686,
610
+ "kl": 0.0011243958651903085,
611
+ "clipfrac": 0.05357143096625805,
612
+ "ratio": 1.017716869711876,
613
+ "grad_norm": 45.443870544433594,
614
+ "lr": 2.0693920339161363e-08
615
+ },
616
+ {
617
+ "update": 68,
618
+ "policy_loss": 0.35332551412284374,
619
+ "kl": 0.0007005828865658259,
620
+ "clipfrac": 0.016826923470944166,
621
+ "ratio": 0.9864731729030609,
622
+ "grad_norm": 38.95060348510742,
623
+ "lr": 1.9802039589925576e-08
624
+ },
625
+ {
626
+ "update": 69,
627
+ "policy_loss": 0.5939304530620575,
628
+ "kl": 0.0016302597941830754,
629
+ "clipfrac": 0.16043956158682704,
630
+ "ratio": 0.9868166595697403,
631
+ "grad_norm": 45.89197540283203,
632
+ "lr": 1.8981919050942376e-08
633
+ },
634
+ {
635
+ "update": 70,
636
+ "policy_loss": -0.6884057372808456,
637
+ "kl": 0.0006987877422943711,
638
+ "clipfrac": 0.02380952425301075,
639
+ "ratio": 1.0099818706512451,
640
+ "grad_norm": 61.384559631347656,
641
+ "lr": 1.8235119868270312e-08
642
+ },
643
+ {
644
+ "update": 71,
645
+ "policy_loss": -0.6742468401789665,
646
+ "kl": 0.000681552934111096,
647
+ "clipfrac": 0.01923076994717121,
648
+ "ratio": 0.9964191764593124,
649
+ "grad_norm": 68.5340347290039,
650
+ "lr": 1.7563063616598895e-08
651
+ },
652
+ {
653
+ "update": 72,
654
+ "policy_loss": -0.04078532010316849,
655
+ "kl": 0.001274814650969347,
656
+ "clipfrac": 0.0773809552192688,
657
+ "ratio": 0.9921253472566605,
658
+ "grad_norm": 112.31976318359375,
659
+ "lr": 1.696702959320036e-08
660
+ },
661
+ {
662
+ "update": 73,
663
+ "policy_loss": -0.14819200336933136,
664
+ "kl": 0.0008460474491585046,
665
+ "clipfrac": 0.05357143096625805,
666
+ "ratio": 1.0018796622753143,
667
+ "grad_norm": 99.58033752441406,
668
+ "lr": 1.6448152382714594e-08
669
+ },
670
+ {
671
+ "update": 74,
672
+ "policy_loss": 0.4682491421699524,
673
+ "kl": 0.0009040829463629052,
674
+ "clipfrac": 0.05000000447034836,
675
+ "ratio": 1.0006756633520126,
676
+ "grad_norm": 28.670822143554688,
677
+ "lr": 1.6007419697402834e-08
678
+ },
679
+ {
680
+ "update": 75,
681
+ "policy_loss": -0.8038802891969681,
682
+ "kl": 0.0013436266162898391,
683
+ "clipfrac": 0.0714285746216774,
684
+ "ratio": 1.0070986449718475,
685
+ "grad_norm": 147.66311645507812,
686
+ "lr": 1.5645670496981157e-08
687
+ },
688
+ {
689
+ "update": 76,
690
+ "policy_loss": 0.4944037050008774,
691
+ "kl": 0.0009207027251250111,
692
+ "clipfrac": 0.01666666753590107,
693
+ "ratio": 0.9882079660892487,
694
+ "grad_norm": 23.254777908325195,
695
+ "lr": 1.5363593391613058e-08
696
+ },
697
+ {
698
+ "update": 77,
699
+ "policy_loss": -0.7182111665606499,
700
+ "kl": 0.0009854048839770257,
701
+ "clipfrac": 0.04699754994362593,
702
+ "ratio": 0.9965574443340302,
703
+ "grad_norm": 175.54769897460938,
704
+ "lr": 1.5161725331100814e-08
705
+ },
706
+ {
707
+ "update": 78,
708
+ "policy_loss": -0.7862310260534286,
709
+ "kl": 0.0005922885611653328,
710
+ "clipfrac": 0.01315789483487606,
711
+ "ratio": 1.0051915794610977,
712
+ "grad_norm": 120.22238159179688,
713
+ "lr": 1.504045058277104e-08
714
+ },
715
+ {
716
+ "update": 79,
717
+ "policy_loss": 0.5465165674686432,
718
+ "kl": 0.0009215092868544161,
719
+ "clipfrac": 0.031746032337347664,
720
+ "ratio": 1.005193551381429,
721
+ "grad_norm": 18.83639144897461,
722
+ "lr": 1.5e-08
723
+ }
724
+ ],
725
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_grpo_source.pt",
726
+ "seconds": 22.73498511314392,
727
+ "parameter_count_change": 0
728
+ }
candidates/budgie-alignment-v2/information-rl-v2-math-augmented/recovery/training_report_long.json ADDED
@@ -0,0 +1,2386 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "algorithm": "3-epoch clipped GRPO/PPO plus 0.15 manual-demonstration BC",
3
+ "base": "verified-math-a025",
4
+ "samples_per_epoch": 315,
5
+ "epochs": 3,
6
+ "updates": 237,
7
+ "peak_lr": 5e-07,
8
+ "clip": 0.1,
9
+ "kl_beta": 0.05,
10
+ "bc_coef": 0.15,
11
+ "history": [
12
+ {
13
+ "update": 1,
14
+ "policy_loss": -0.19510458782315254,
15
+ "kl": 0.000876154939760454,
16
+ "bc": 1.027942419052124,
17
+ "clipfrac": 0.025921659544110298,
18
+ "ratio": 0.9908683747053146,
19
+ "grad_norm": 206.4007568359375,
20
+ "lr": 4.166666666666666e-08
21
+ },
22
+ {
23
+ "update": 2,
24
+ "policy_loss": -0.13415861129760742,
25
+ "kl": 0.006434725124563556,
26
+ "bc": 1.4706554412841797,
27
+ "clipfrac": 0.12272727489471436,
28
+ "ratio": 1.016227051615715,
29
+ "grad_norm": 511.6988220214844,
30
+ "lr": 8.333333333333333e-08
31
+ },
32
+ {
33
+ "update": 3,
34
+ "policy_loss": -0.12622825801372528,
35
+ "kl": 0.0008369503266294487,
36
+ "bc": 0.6043471693992615,
37
+ "clipfrac": 0.015625,
38
+ "ratio": 0.9986923336982727,
39
+ "grad_norm": 78.35072326660156,
40
+ "lr": 1.25e-07
41
+ },
42
+ {
43
+ "update": 4,
44
+ "policy_loss": 0.5154320076107979,
45
+ "kl": 0.0005285190381982829,
46
+ "bc": 0.0,
47
+ "clipfrac": 0.006097560748457909,
48
+ "ratio": 0.9943587332963943,
49
+ "grad_norm": 32.61154556274414,
50
+ "lr": 1.6666666666666665e-07
51
+ },
52
+ {
53
+ "update": 5,
54
+ "policy_loss": -0.02217152714729309,
55
+ "kl": 0.00104370761255268,
56
+ "bc": 0.8066266179084778,
57
+ "clipfrac": 0.053571430034935474,
58
+ "ratio": 1.0076594948768616,
59
+ "grad_norm": 66.17955017089844,
60
+ "lr": 2.0833333333333333e-07
61
+ },
62
+ {
63
+ "update": 6,
64
+ "policy_loss": 0.4610743671655655,
65
+ "kl": 0.0006572127495019231,
66
+ "bc": 0.0,
67
+ "clipfrac": 0.030789826065301895,
68
+ "ratio": 1.0029787868261337,
69
+ "grad_norm": 31.670940399169922,
70
+ "lr": 2.5e-07
71
+ },
72
+ {
73
+ "update": 7,
74
+ "policy_loss": -0.10410520434379578,
75
+ "kl": 0.0010717859913711436,
76
+ "bc": 0.4193834662437439,
77
+ "clipfrac": 0.01785714365541935,
78
+ "ratio": 1.0043146163225174,
79
+ "grad_norm": 49.3795166015625,
80
+ "lr": 2.916666666666667e-07
81
+ },
82
+ {
83
+ "update": 8,
84
+ "policy_loss": 0.4617234840989113,
85
+ "kl": 0.0006714024202665314,
86
+ "bc": 0.0,
87
+ "clipfrac": 0.009259259328246117,
88
+ "ratio": 0.9883570224046707,
89
+ "grad_norm": 65.1924819946289,
90
+ "lr": 3.333333333333333e-07
91
+ },
92
+ {
93
+ "update": 9,
94
+ "policy_loss": -0.16966338455677032,
95
+ "kl": 0.000871529700816609,
96
+ "bc": 0.5899983644485474,
97
+ "clipfrac": 0.05637255031615496,
98
+ "ratio": 1.0094109177589417,
99
+ "grad_norm": 72.21329498291016,
100
+ "lr": 3.75e-07
101
+ },
102
+ {
103
+ "update": 10,
104
+ "policy_loss": -0.7109477370977402,
105
+ "kl": 0.0012198839540360495,
106
+ "bc": 2.374994218349457,
107
+ "clipfrac": 0.06362007185816765,
108
+ "ratio": 0.9968485832214355,
109
+ "grad_norm": 332.5988464355469,
110
+ "lr": 4.1666666666666667e-07
111
+ },
112
+ {
113
+ "update": 11,
114
+ "policy_loss": -0.17466256767511368,
115
+ "kl": 0.0008736935051274486,
116
+ "bc": 0.9465745091438293,
117
+ "clipfrac": 0.05248366016894579,
118
+ "ratio": 0.9966766387224197,
119
+ "grad_norm": 50.857933044433594,
120
+ "lr": 4.5833333333333327e-07
121
+ },
122
+ {
123
+ "update": 12,
124
+ "policy_loss": -0.09500440955162048,
125
+ "kl": 0.0010176949144806713,
126
+ "bc": 0.7076413631439209,
127
+ "clipfrac": 0.05929487384855747,
128
+ "ratio": 0.9911515712738037,
129
+ "grad_norm": 41.16560363769531,
130
+ "lr": 5e-07
131
+ },
132
+ {
133
+ "update": 13,
134
+ "policy_loss": -0.08254086226224899,
135
+ "kl": 0.0007406689837807789,
136
+ "bc": 0.5545374751091003,
137
+ "clipfrac": 0.015625,
138
+ "ratio": 0.9955743849277496,
139
+ "grad_norm": 38.700313568115234,
140
+ "lr": 5e-07
141
+ },
142
+ {
143
+ "update": 14,
144
+ "policy_loss": 0.028862856328487396,
145
+ "kl": 0.0005022259974793997,
146
+ "bc": 0.47250425815582275,
147
+ "clipfrac": 0.033928572200238705,
148
+ "ratio": 1.0003439635038376,
149
+ "grad_norm": 38.674442291259766,
150
+ "lr": 4.999778716459044e-07
151
+ },
152
+ {
153
+ "update": 15,
154
+ "policy_loss": 0.5371713787317276,
155
+ "kl": 0.0004163368994340999,
156
+ "bc": 0.0,
157
+ "clipfrac": 0.0071428571827709675,
158
+ "ratio": 1.0013226717710495,
159
+ "grad_norm": 28.877824783325195,
160
+ "lr": 4.999114909361869e-07
161
+ },
162
+ {
163
+ "update": 16,
164
+ "policy_loss": -0.09978614747524261,
165
+ "kl": 0.0006174262780405115,
166
+ "bc": 0.5324281454086304,
167
+ "clipfrac": 0.03333333507180214,
168
+ "ratio": 1.0074394196271896,
169
+ "grad_norm": 61.868385314941406,
170
+ "lr": 4.998008709276995e-07
171
+ },
172
+ {
173
+ "update": 17,
174
+ "policy_loss": 0.5966428965330124,
175
+ "kl": 0.000747586087527452,
176
+ "bc": 0.0,
177
+ "clipfrac": 0.0,
178
+ "ratio": 1.0091132670640945,
179
+ "grad_norm": 46.40936279296875,
180
+ "lr": 4.996460333790088e-07
181
+ },
182
+ {
183
+ "update": 18,
184
+ "policy_loss": 0.47971052676439285,
185
+ "kl": 0.000941236074140761,
186
+ "bc": 0.0,
187
+ "clipfrac": 0.061607143841683865,
188
+ "ratio": 0.9984636008739471,
189
+ "grad_norm": 21.705615997314453,
190
+ "lr": 4.994470087461153e-07
191
+ },
192
+ {
193
+ "update": 19,
194
+ "policy_loss": -0.13076525926589966,
195
+ "kl": 0.0008292395359603688,
196
+ "bc": 1.0263206958770752,
197
+ "clipfrac": 0.01666666753590107,
198
+ "ratio": 1.0024433732032776,
199
+ "grad_norm": 54.934539794921875,
200
+ "lr": 4.992038361764643e-07
201
+ },
202
+ {
203
+ "update": 20,
204
+ "policy_loss": 0.500286839902401,
205
+ "kl": 0.0006831871287431568,
206
+ "bc": 0.0,
207
+ "clipfrac": 0.0,
208
+ "ratio": 1.0011813342571259,
209
+ "grad_norm": 30.321924209594727,
210
+ "lr": 4.989165635012443e-07
211
+ },
212
+ {
213
+ "update": 21,
214
+ "policy_loss": 0.4278707131743431,
215
+ "kl": 0.0007060934440232813,
216
+ "bc": 0.0,
217
+ "clipfrac": 0.008928571827709675,
218
+ "ratio": 1.001550242304802,
219
+ "grad_norm": 13.077935218811035,
220
+ "lr": 4.985852472259796e-07
221
+ },
222
+ {
223
+ "update": 22,
224
+ "policy_loss": -0.7952130883932114,
225
+ "kl": 0.0009766756847966462,
226
+ "bc": 1.3902219235897064,
227
+ "clipfrac": 0.032142858020961285,
228
+ "ratio": 0.995733231306076,
229
+ "grad_norm": 88.3668441772461,
230
+ "lr": 4.982099525194155e-07
231
+ },
232
+ {
233
+ "update": 23,
234
+ "policy_loss": -0.12327355146408081,
235
+ "kl": 0.0008326557872351259,
236
+ "bc": 0.8241127729415894,
237
+ "clipfrac": 0.04478609748184681,
238
+ "ratio": 0.9986664354801178,
239
+ "grad_norm": 59.80543518066406,
240
+ "lr": 4.977907532007003e-07
241
+ },
242
+ {
243
+ "update": 24,
244
+ "policy_loss": -1.3888931721448898,
245
+ "kl": 0.003256130214140285,
246
+ "bc": 2.8387696146965027,
247
+ "clipfrac": 0.08630952425301075,
248
+ "ratio": 0.9948488175868988,
249
+ "grad_norm": 240.91549682617188,
250
+ "lr": 4.973277317248644e-07
251
+ },
252
+ {
253
+ "update": 25,
254
+ "policy_loss": -0.7882493212819099,
255
+ "kl": 0.0012935253034811467,
256
+ "bc": 2.6234980821609497,
257
+ "clipfrac": 0.0357142873108387,
258
+ "ratio": 1.0014193952083588,
259
+ "grad_norm": 166.27757263183594,
260
+ "lr": 4.96820979166603e-07
261
+ },
262
+ {
263
+ "update": 26,
264
+ "policy_loss": -0.8596315551549196,
265
+ "kl": 0.0020427469862625003,
266
+ "bc": 1.9046548008918762,
267
+ "clipfrac": 0.08378623332828283,
268
+ "ratio": 1.0095646679401398,
269
+ "grad_norm": 50.48497009277344,
270
+ "lr": 4.962705952023609e-07
271
+ },
272
+ {
273
+ "update": 27,
274
+ "policy_loss": -0.1284533217549324,
275
+ "kl": 0.0007647274469491094,
276
+ "bc": 0.8432860374450684,
277
+ "clipfrac": 0.03125,
278
+ "ratio": 0.9976484179496765,
279
+ "grad_norm": 65.6868667602539,
280
+ "lr": 4.956766880907269e-07
281
+ },
282
+ {
283
+ "update": 28,
284
+ "policy_loss": -0.7519287168979645,
285
+ "kl": 0.0017238830187125131,
286
+ "bc": 1.9211148619651794,
287
+ "clipfrac": 0.09415584616363049,
288
+ "ratio": 1.0046603828668594,
289
+ "grad_norm": 91.02025604248047,
290
+ "lr": 4.950393746511397e-07
291
+ },
292
+ {
293
+ "update": 29,
294
+ "policy_loss": 0.520062830299139,
295
+ "kl": 0.001096149775548838,
296
+ "bc": 0.0,
297
+ "clipfrac": 0.039682540111243725,
298
+ "ratio": 1.0004958510398865,
299
+ "grad_norm": 18.666715621948242,
300
+ "lr": 4.943587802409103e-07
301
+ },
302
+ {
303
+ "update": 30,
304
+ "policy_loss": -0.16757307946681976,
305
+ "kl": 0.0007558927609352395,
306
+ "bc": 0.6297526955604553,
307
+ "clipfrac": 0.06030701659619808,
308
+ "ratio": 0.9938405305147171,
309
+ "grad_norm": 48.397377014160156,
310
+ "lr": 4.93635038730564e-07
311
+ },
312
+ {
313
+ "update": 31,
314
+ "policy_loss": 0.5658809170126915,
315
+ "kl": 0.0010287038603564724,
316
+ "bc": 0.0,
317
+ "clipfrac": 0.04022258287295699,
318
+ "ratio": 0.9991264492273331,
319
+ "grad_norm": 17.148656845092773,
320
+ "lr": 4.928682924775088e-07
321
+ },
322
+ {
323
+ "update": 32,
324
+ "policy_loss": -0.08903001993894577,
325
+ "kl": 0.0015731347375549376,
326
+ "bc": 0.6057514548301697,
327
+ "clipfrac": 0.061607143841683865,
328
+ "ratio": 1.0142334699630737,
329
+ "grad_norm": 73.3297119140625,
330
+ "lr": 4.920586922980346e-07
331
+ },
332
+ {
333
+ "update": 33,
334
+ "policy_loss": -0.17615246772766113,
335
+ "kl": 0.0006532101906486787,
336
+ "bc": 0.6488139629364014,
337
+ "clipfrac": 0.0357142873108387,
338
+ "ratio": 1.0043449252843857,
339
+ "grad_norm": 55.56952667236328,
340
+ "lr": 4.912063974376474e-07
341
+ },
342
+ {
343
+ "update": 34,
344
+ "policy_loss": 0.4746376797556877,
345
+ "kl": 0.0005684795105480589,
346
+ "bc": 0.0,
347
+ "clipfrac": 0.015625,
348
+ "ratio": 0.9930563271045685,
349
+ "grad_norm": 22.345109939575195,
350
+ "lr": 4.903115755397469e-07
351
+ },
352
+ {
353
+ "update": 35,
354
+ "policy_loss": -0.8364970535039902,
355
+ "kl": 0.002653615902090678,
356
+ "bc": 1.9397691488265991,
357
+ "clipfrac": 0.18377976212650537,
358
+ "ratio": 1.015764757990837,
359
+ "grad_norm": 165.18174743652344,
360
+ "lr": 4.893744026126517e-07
361
+ },
362
+ {
363
+ "update": 36,
364
+ "policy_loss": -0.06222192943096161,
365
+ "kl": 0.003575598617317155,
366
+ "bc": 0.688238799571991,
367
+ "clipfrac": 0.3273809552192688,
368
+ "ratio": 1.0048000812530518,
369
+ "grad_norm": 78.1209487915039,
370
+ "lr": 4.883950629949781e-07
371
+ },
372
+ {
373
+ "update": 37,
374
+ "policy_loss": -0.16398538649082184,
375
+ "kl": 0.001642811723286286,
376
+ "bc": 1.10191810131073,
377
+ "clipfrac": 0.1016106503084302,
378
+ "ratio": 0.9897035658359528,
379
+ "grad_norm": 58.051239013671875,
380
+ "lr": 4.873737493193827e-07
381
+ },
382
+ {
383
+ "update": 38,
384
+ "policy_loss": 0.49943089485168457,
385
+ "kl": 0.0008657122089061886,
386
+ "bc": 0.0,
387
+ "clipfrac": 0.060714286752045155,
388
+ "ratio": 0.992439404129982,
389
+ "grad_norm": 26.804126739501953,
390
+ "lr": 4.863106624746713e-07
391
+ },
392
+ {
393
+ "update": 39,
394
+ "policy_loss": -0.7487267479300499,
395
+ "kl": 0.0007124825242499355,
396
+ "bc": 1.300371527671814,
397
+ "clipfrac": 0.034615385346114635,
398
+ "ratio": 0.9999753832817078,
399
+ "grad_norm": 111.66492462158203,
400
+ "lr": 4.85206011566285e-07
401
+ },
402
+ {
403
+ "update": 40,
404
+ "policy_loss": -0.06982056796550751,
405
+ "kl": 0.00041918083297787234,
406
+ "bc": 1.1426799297332764,
407
+ "clipfrac": 0.0,
408
+ "ratio": 0.9936048835515976,
409
+ "grad_norm": 89.8308334350586,
410
+ "lr": 4.840600138751701e-07
411
+ },
412
+ {
413
+ "update": 41,
414
+ "policy_loss": -0.22436698898673058,
415
+ "kl": 0.0022047627571737394,
416
+ "bc": 2.1092965602874756,
417
+ "clipfrac": 0.07936508394777775,
418
+ "ratio": 1.0159265995025635,
419
+ "grad_norm": 140.4076385498047,
420
+ "lr": 4.828728948150396e-07
421
+ },
422
+ {
423
+ "update": 42,
424
+ "policy_loss": 0.5093231052160263,
425
+ "kl": 0.0005924126307945698,
426
+ "bc": 0.0,
427
+ "clipfrac": 0.013888888992369175,
428
+ "ratio": 0.9994108527898788,
429
+ "grad_norm": 24.26468849182129,
430
+ "lr": 4.816448878880347e-07
431
+ },
432
+ {
433
+ "update": 43,
434
+ "policy_loss": 0.4842234645038843,
435
+ "kl": 0.001008763603749685,
436
+ "bc": 0.0,
437
+ "clipfrac": 0.04032257944345474,
438
+ "ratio": 1.0002600401639938,
439
+ "grad_norm": 25.505123138427734,
440
+ "lr": 4.803762346387967e-07
441
+ },
442
+ {
443
+ "update": 44,
444
+ "policy_loss": 0.4704095721244812,
445
+ "kl": 0.0012127027730457485,
446
+ "bc": 0.0,
447
+ "clipfrac": 0.05113636516034603,
448
+ "ratio": 1.000778317451477,
449
+ "grad_norm": 22.0725154876709,
450
+ "lr": 4.790671846069551e-07
451
+ },
452
+ {
453
+ "update": 45,
454
+ "policy_loss": 0.4555321254301816,
455
+ "kl": 0.0007328471983782947,
456
+ "bc": 0.0,
457
+ "clipfrac": 0.02777777798473835,
458
+ "ratio": 0.990035817027092,
459
+ "grad_norm": 33.223751068115234,
460
+ "lr": 4.777179952780442e-07
461
+ },
462
+ {
463
+ "update": 46,
464
+ "policy_loss": -0.13467103987932205,
465
+ "kl": 0.0016522360310773365,
466
+ "bc": 0.718216061592102,
467
+ "clipfrac": 0.0625,
468
+ "ratio": 1.0132396072149277,
469
+ "grad_norm": 52.98504638671875,
470
+ "lr": 4.7632893203285733e-07
471
+ },
472
+ {
473
+ "update": 47,
474
+ "policy_loss": -0.721901535987854,
475
+ "kl": 0.0021116837742738426,
476
+ "bc": 1.6638635993003845,
477
+ "clipfrac": 0.17417582869529724,
478
+ "ratio": 1.016477346420288,
479
+ "grad_norm": 174.0909423828125,
480
+ "lr": 4.7490026809524617e-07
481
+ },
482
+ {
483
+ "update": 48,
484
+ "policy_loss": 0.5309353247284889,
485
+ "kl": 0.000814796207123436,
486
+ "bc": 0.0,
487
+ "clipfrac": 0.01785714365541935,
488
+ "ratio": 0.9963850229978561,
489
+ "grad_norm": 25.491992950439453,
490
+ "lr": 4.7343228447837985e-07
491
+ },
492
+ {
493
+ "update": 49,
494
+ "policy_loss": 0.4972270205616951,
495
+ "kl": 0.0023522967094322667,
496
+ "bc": 0.0,
497
+ "clipfrac": 0.11309524159878492,
498
+ "ratio": 0.9854845106601715,
499
+ "grad_norm": 25.150253295898438,
500
+ "lr": 4.719252699294696e-07
501
+ },
502
+ {
503
+ "update": 50,
504
+ "policy_loss": 0.40502023696899414,
505
+ "kl": 0.0007564941079181153,
506
+ "bc": 0.0,
507
+ "clipfrac": 0.03125,
508
+ "ratio": 0.9968298375606537,
509
+ "grad_norm": 14.627226829528809,
510
+ "lr": 4.703795208729737e-07
511
+ },
512
+ {
513
+ "update": 51,
514
+ "policy_loss": 0.5723534002900124,
515
+ "kl": 0.000959977405727841,
516
+ "bc": 0.0,
517
+ "clipfrac": 0.03574561467394233,
518
+ "ratio": 0.9942629486322403,
519
+ "grad_norm": 38.04493713378906,
520
+ "lr": 4.68795341352292e-07
521
+ },
522
+ {
523
+ "update": 52,
524
+ "policy_loss": 0.3527619857341051,
525
+ "kl": 0.0014499610115308315,
526
+ "bc": 0.0,
527
+ "clipfrac": 0.10768398642539978,
528
+ "ratio": 0.9992406070232391,
529
+ "grad_norm": 14.932208061218262,
530
+ "lr": 4.6717304296996114e-07
531
+ },
532
+ {
533
+ "update": 53,
534
+ "policy_loss": -0.1164940893650055,
535
+ "kl": 0.005587249164818786,
536
+ "bc": 0.40276047587394714,
537
+ "clipfrac": 0.11656746361404657,
538
+ "ratio": 1.017602875828743,
539
+ "grad_norm": 36.721534729003906,
540
+ "lr": 4.655129448263639e-07
541
+ },
542
+ {
543
+ "update": 54,
544
+ "policy_loss": 0.34909214451909065,
545
+ "kl": 0.0014042896509636194,
546
+ "bc": 0.0,
547
+ "clipfrac": 0.08756868354976177,
548
+ "ratio": 0.996032327413559,
549
+ "grad_norm": 15.396754264831543,
550
+ "lr": 4.63815373456963e-07
551
+ },
552
+ {
553
+ "update": 55,
554
+ "policy_loss": 0.4464123770594597,
555
+ "kl": 0.0013048123946646228,
556
+ "bc": 0.0,
557
+ "clipfrac": 0.051060606725513935,
558
+ "ratio": 0.9773492217063904,
559
+ "grad_norm": 28.954362869262695,
560
+ "lr": 4.620806627680727e-07
561
+ },
562
+ {
563
+ "update": 56,
564
+ "policy_loss": -0.7736191302537918,
565
+ "kl": 0.001302291791944299,
566
+ "bc": 1.364084005355835,
567
+ "clipfrac": 0.10833333805203438,
568
+ "ratio": 0.9943810254335403,
569
+ "grad_norm": 102.35198974609375,
570
+ "lr": 4.603091539711808e-07
571
+ },
572
+ {
573
+ "update": 57,
574
+ "policy_loss": 0.49199698120355606,
575
+ "kl": 0.0014561986245098524,
576
+ "bc": 0.0,
577
+ "clipfrac": 0.05733082629740238,
578
+ "ratio": 0.9814591258764267,
579
+ "grad_norm": 15.39022445678711,
580
+ "lr": 4.585011955158335e-07
581
+ },
582
+ {
583
+ "update": 58,
584
+ "policy_loss": -1.3895136043429375,
585
+ "kl": 0.009901580662699416,
586
+ "bc": 2.2219923436641693,
587
+ "clipfrac": 0.1713497918099165,
588
+ "ratio": 1.0234117805957794,
589
+ "grad_norm": 309.3123474121094,
590
+ "lr": 4.5665714302109655e-07
591
+ },
592
+ {
593
+ "update": 59,
594
+ "policy_loss": -0.13466782867908478,
595
+ "kl": 0.006685108528472483,
596
+ "bc": 0.8375793099403381,
597
+ "clipfrac": 0.15054945833981037,
598
+ "ratio": 0.9952787309885025,
599
+ "grad_norm": 60.0086669921875,
600
+ "lr": 4.547773592056061e-07
601
+ },
602
+ {
603
+ "update": 60,
604
+ "policy_loss": -0.10036198794841766,
605
+ "kl": 0.0011401836964068934,
606
+ "bc": 0.6442959904670715,
607
+ "clipfrac": 0.10687134601175785,
608
+ "ratio": 0.996518149971962,
609
+ "grad_norm": 47.838436126708984,
610
+ "lr": 4.5286221381622326e-07
611
+ },
612
+ {
613
+ "update": 61,
614
+ "policy_loss": 0.536563865840435,
615
+ "kl": 0.0012018798006465659,
616
+ "bc": 0.0,
617
+ "clipfrac": 0.05119733791798353,
618
+ "ratio": 0.9935150146484375,
619
+ "grad_norm": 20.38047981262207,
620
+ "lr": 4.509120835553067e-07
621
+ },
622
+ {
623
+ "update": 62,
624
+ "policy_loss": 0.5667472332715988,
625
+ "kl": 0.0015989259918569587,
626
+ "bc": 0.0,
627
+ "clipfrac": 0.08279569633305073,
628
+ "ratio": 0.9950767457485199,
629
+ "grad_norm": 20.32708168029785,
630
+ "lr": 4.489273520066158e-07
631
+ },
632
+ {
633
+ "update": 63,
634
+ "policy_loss": -0.053504444658756256,
635
+ "kl": 0.0023575219383928925,
636
+ "bc": 1.2979912757873535,
637
+ "clipfrac": 0.13363095745444298,
638
+ "ratio": 0.9901792854070663,
639
+ "grad_norm": 227.2769012451172,
640
+ "lr": 4.4690840955986164e-07
641
+ },
642
+ {
643
+ "update": 64,
644
+ "policy_loss": -0.17155902832746506,
645
+ "kl": 0.002278537838719785,
646
+ "bc": 0.9187172651290894,
647
+ "clipfrac": 0.09214743785560131,
648
+ "ratio": 1.0070673376321793,
649
+ "grad_norm": 57.718162536621094,
650
+ "lr": 4.4485565333391863e-07
651
+ },
652
+ {
653
+ "update": 65,
654
+ "policy_loss": 0.4981924146413803,
655
+ "kl": 0.0015484071773244068,
656
+ "bc": 0.0,
657
+ "clipfrac": 0.05208333395421505,
658
+ "ratio": 0.9932793974876404,
659
+ "grad_norm": 37.10612869262695,
660
+ "lr": 4.427694870987122e-07
661
+ },
662
+ {
663
+ "update": 66,
664
+ "policy_loss": -0.15755303762853146,
665
+ "kl": 0.0020522976119536906,
666
+ "bc": 0.6041543483734131,
667
+ "clipfrac": 0.08350421767681837,
668
+ "ratio": 1.0014862716197968,
669
+ "grad_norm": 45.1898078918457,
670
+ "lr": 4.4065032119579957e-07
671
+ },
672
+ {
673
+ "update": 67,
674
+ "policy_loss": -0.110604427754879,
675
+ "kl": 0.0024213973665609956,
676
+ "bc": 0.8314558863639832,
677
+ "clipfrac": 0.1458333395421505,
678
+ "ratio": 0.9971132874488831,
679
+ "grad_norm": 51.38134765625,
680
+ "lr": 4.384985724576561e-07
681
+ },
682
+ {
683
+ "update": 68,
684
+ "policy_loss": -0.1660739928483963,
685
+ "kl": 0.003218230587663129,
686
+ "bc": 0.5636866092681885,
687
+ "clipfrac": 0.12724171951413155,
688
+ "ratio": 0.9715565890073776,
689
+ "grad_norm": 44.52268600463867,
690
+ "lr": 4.3631466412568614e-07
691
+ },
692
+ {
693
+ "update": 69,
694
+ "policy_loss": -0.15945792198181152,
695
+ "kl": 0.0013489263074006885,
696
+ "bc": 0.5274693369865417,
697
+ "clipfrac": 0.08333333488553762,
698
+ "ratio": 1.0054049789905548,
699
+ "grad_norm": 34.03193283081055,
700
+ "lr": 4.3409902576697315e-07
701
+ },
702
+ {
703
+ "update": 70,
704
+ "policy_loss": 0.5136360824108124,
705
+ "kl": 0.0010616784711601213,
706
+ "bc": 0.0,
707
+ "clipfrac": 0.09649122972041368,
708
+ "ratio": 0.9840033799409866,
709
+ "grad_norm": 29.517765045166016,
710
+ "lr": 4.318520931897852e-07
711
+ },
712
+ {
713
+ "update": 71,
714
+ "policy_loss": -0.13222867250442505,
715
+ "kl": 0.003160769381793216,
716
+ "bc": 1.0121378898620605,
717
+ "clipfrac": 0.07638888992369175,
718
+ "ratio": 0.9837389290332794,
719
+ "grad_norm": 86.48680114746094,
720
+ "lr": 4.295743083578529e-07
721
+ },
722
+ {
723
+ "update": 72,
724
+ "policy_loss": 0.5410062298178673,
725
+ "kl": 0.00268736548605375,
726
+ "bc": 0.0,
727
+ "clipfrac": 0.09263393096625805,
728
+ "ratio": 0.9868515729904175,
729
+ "grad_norm": 15.604569435119629,
730
+ "lr": 4.272661193034367e-07
731
+ },
732
+ {
733
+ "update": 73,
734
+ "policy_loss": -0.17949698865413666,
735
+ "kl": 0.004443807149073109,
736
+ "bc": 0.6238439083099365,
737
+ "clipfrac": 0.09690656699240208,
738
+ "ratio": 1.0063030123710632,
739
+ "grad_norm": 68.43749237060547,
740
+ "lr": 4.249279800392009e-07
741
+ },
742
+ {
743
+ "update": 74,
744
+ "policy_loss": -0.15009494870901108,
745
+ "kl": 0.0030391460459213704,
746
+ "bc": 0.7174612879753113,
747
+ "clipfrac": 0.13389845471829176,
748
+ "ratio": 0.9814443588256836,
749
+ "grad_norm": 184.6975860595703,
750
+ "lr": 4.225603504689104e-07
751
+ },
752
+ {
753
+ "update": 75,
754
+ "policy_loss": -0.13638710975646973,
755
+ "kl": 0.0038440215575974435,
756
+ "bc": 1.1852185726165771,
757
+ "clipfrac": 0.14078675396740437,
758
+ "ratio": 0.9772951751947403,
759
+ "grad_norm": 100.4109878540039,
760
+ "lr": 4.2016369629696974e-07
761
+ },
762
+ {
763
+ "update": 76,
764
+ "policy_loss": -0.6719670966267586,
765
+ "kl": 0.0016136056510731578,
766
+ "bc": 2.4694125652313232,
767
+ "clipfrac": 0.0843802047893405,
768
+ "ratio": 1.0086623281240463,
769
+ "grad_norm": 104.93734741210938,
770
+ "lr": 4.1773848893682024e-07
771
+ },
772
+ {
773
+ "update": 77,
774
+ "policy_loss": 0.05180861055850983,
775
+ "kl": 0.0014200602017808706,
776
+ "bc": 0.6797431707382202,
777
+ "clipfrac": 0.07604166772216558,
778
+ "ratio": 0.9969923943281174,
779
+ "grad_norm": 94.22496032714844,
780
+ "lr": 4.1528520541821503e-07
781
+ },
782
+ {
783
+ "update": 78,
784
+ "policy_loss": 0.5485774800181389,
785
+ "kl": 0.0017155272216768935,
786
+ "bc": 0.0,
787
+ "clipfrac": 0.0456349216401577,
788
+ "ratio": 1.0091137439012527,
789
+ "grad_norm": 36.68336868286133,
790
+ "lr": 4.1280432829338873e-07
791
+ },
792
+ {
793
+ "update": 79,
794
+ "policy_loss": -0.22321119531989098,
795
+ "kl": 0.0028132829756941646,
796
+ "bc": 0.7811753749847412,
797
+ "clipfrac": 0.05602241028100252,
798
+ "ratio": 1.0050344169139862,
799
+ "grad_norm": 78.10454559326172,
800
+ "lr": 4.1029634554214126e-07
801
+ },
802
+ {
803
+ "update": 80,
804
+ "policy_loss": 0.5951048731803894,
805
+ "kl": 0.002644898631842807,
806
+ "bc": 0.0,
807
+ "clipfrac": 0.12291666865348816,
808
+ "ratio": 0.9599624574184418,
809
+ "grad_norm": 73.50647735595703,
810
+ "lr": 4.0776175047585383e-07
811
+ },
812
+ {
813
+ "update": 81,
814
+ "policy_loss": -0.8109649568796158,
815
+ "kl": 0.036508842022158206,
816
+ "bc": 2.0294277667999268,
817
+ "clipfrac": 0.2873337250202894,
818
+ "ratio": 1.128624364733696,
819
+ "grad_norm": 89.3708724975586,
820
+ "lr": 4.0520104164045624e-07
821
+ },
822
+ {
823
+ "update": 82,
824
+ "policy_loss": -0.913199407979846,
825
+ "kl": 0.005278062162688002,
826
+ "bc": 2.9137173891067505,
827
+ "clipfrac": 0.2524961605668068,
828
+ "ratio": 1.030241072177887,
829
+ "grad_norm": 102.12661743164062,
830
+ "lr": 4.0261472271836505e-07
831
+ },
832
+ {
833
+ "update": 83,
834
+ "policy_loss": -0.1692982092499733,
835
+ "kl": 0.004277061874745414,
836
+ "bc": 1.1456223726272583,
837
+ "clipfrac": 0.18038303032517433,
838
+ "ratio": 0.9872328639030457,
839
+ "grad_norm": 32.475040435791016,
840
+ "lr": 4.000033024294105e-07
841
+ },
842
+ {
843
+ "update": 84,
844
+ "policy_loss": -0.2224792167544365,
845
+ "kl": 0.003205167828127742,
846
+ "bc": 0.7932528853416443,
847
+ "clipfrac": 0.10499066486954689,
848
+ "ratio": 0.9940780848264694,
849
+ "grad_norm": 50.824398040771484,
850
+ "lr": 3.973672944307734e-07
851
+ },
852
+ {
853
+ "update": 85,
854
+ "policy_loss": -0.2956040073186159,
855
+ "kl": 0.007550611830083653,
856
+ "bc": 0.5936598777770996,
857
+ "clipfrac": 0.17083333805203438,
858
+ "ratio": 0.987587183713913,
859
+ "grad_norm": 67.58879852294922,
860
+ "lr": 3.9470721721595066e-07
861
+ },
862
+ {
863
+ "update": 86,
864
+ "policy_loss": -0.04095003753900528,
865
+ "kl": 0.0061000274727120996,
866
+ "bc": 0.8317468166351318,
867
+ "clipfrac": 0.18536069802939892,
868
+ "ratio": 0.984009861946106,
869
+ "grad_norm": 39.99055862426758,
870
+ "lr": 3.920235940127693e-07
871
+ },
872
+ {
873
+ "update": 87,
874
+ "policy_loss": 0.48022887855768204,
875
+ "kl": 0.0021361262624850497,
876
+ "bc": 0.0,
877
+ "clipfrac": 0.20238095708191395,
878
+ "ratio": 0.9574830085039139,
879
+ "grad_norm": 54.10400390625,
880
+ "lr": 3.893169526804691e-07
881
+ },
882
+ {
883
+ "update": 88,
884
+ "policy_loss": 0.3618209194391966,
885
+ "kl": 0.0032076271600089967,
886
+ "bc": 0.0,
887
+ "clipfrac": 0.07631257828325033,
888
+ "ratio": 0.9837832748889923,
889
+ "grad_norm": 15.709806442260742,
890
+ "lr": 3.865878256058753e-07
891
+ },
892
+ {
893
+ "update": 89,
894
+ "policy_loss": -0.16095273941755295,
895
+ "kl": 0.003962355607654899,
896
+ "bc": 0.5929303765296936,
897
+ "clipfrac": 0.20745342783629894,
898
+ "ratio": 1.003268837928772,
899
+ "grad_norm": 25.1063175201416,
900
+ "lr": 3.8383674959867896e-07
901
+ },
902
+ {
903
+ "update": 90,
904
+ "policy_loss": 0.523290142416954,
905
+ "kl": 0.00310191616154043,
906
+ "bc": 0.0,
907
+ "clipfrac": 0.20673013385385275,
908
+ "ratio": 0.964552715420723,
909
+ "grad_norm": 13.397140502929688,
910
+ "lr": 3.8106426578584944e-07
911
+ },
912
+ {
913
+ "update": 91,
914
+ "policy_loss": 0.5284171476960182,
915
+ "kl": 0.002698375523323193,
916
+ "bc": 0.0,
917
+ "clipfrac": 0.12680653110146523,
918
+ "ratio": 0.9659009873867035,
919
+ "grad_norm": 14.952335357666016,
920
+ "lr": 3.78270919505196e-07
921
+ },
922
+ {
923
+ "update": 92,
924
+ "policy_loss": -0.11489962786436081,
925
+ "kl": 0.0031867561920080334,
926
+ "bc": 0.4634056091308594,
927
+ "clipfrac": 0.13580827601253986,
928
+ "ratio": 0.9861327707767487,
929
+ "grad_norm": 16.886022567749023,
930
+ "lr": 3.7545726019810165e-07
931
+ },
932
+ {
933
+ "update": 93,
934
+ "policy_loss": -0.1677750051021576,
935
+ "kl": 0.003100484929746017,
936
+ "bc": 0.9139189124107361,
937
+ "clipfrac": 0.1448412761092186,
938
+ "ratio": 0.9896708875894547,
939
+ "grad_norm": 46.46292495727539,
940
+ "lr": 3.726238413014505e-07
941
+ },
942
+ {
943
+ "update": 94,
944
+ "policy_loss": -0.17248625308275223,
945
+ "kl": 0.005315756628988311,
946
+ "bc": 0.681207537651062,
947
+ "clipfrac": 0.08830165304243565,
948
+ "ratio": 1.0014788955450058,
949
+ "grad_norm": 69.89939880371094,
950
+ "lr": 3.6977122013876844e-07
951
+ },
952
+ {
953
+ "update": 95,
954
+ "policy_loss": -0.607801266014576,
955
+ "kl": 0.004958546414854936,
956
+ "bc": 1.7870023846626282,
957
+ "clipfrac": 0.1955954870209098,
958
+ "ratio": 1.0164655148983002,
959
+ "grad_norm": 74.3194580078125,
960
+ "lr": 3.6689995781059913e-07
961
+ },
962
+ {
963
+ "update": 96,
964
+ "policy_loss": -0.19851364940404892,
965
+ "kl": 0.007386021374259144,
966
+ "bc": 1.2842053174972534,
967
+ "clipfrac": 0.23482907377183437,
968
+ "ratio": 0.9818010628223419,
969
+ "grad_norm": 91.5948715209961,
970
+ "lr": 3.6401061908413823e-07
971
+ },
972
+ {
973
+ "update": 97,
974
+ "policy_loss": -0.12129638344049454,
975
+ "kl": 0.005685700103640556,
976
+ "bc": 0.5157343149185181,
977
+ "clipfrac": 0.17057387717068195,
978
+ "ratio": 0.9947812706232071,
979
+ "grad_norm": 29.41790199279785,
980
+ "lr": 3.611037722821452e-07
981
+ },
982
+ {
983
+ "update": 98,
984
+ "policy_loss": 0.45163997262716293,
985
+ "kl": 0.005724758317228407,
986
+ "bc": 0.0,
987
+ "clipfrac": 0.29286859929561615,
988
+ "ratio": 0.9485321193933487,
989
+ "grad_norm": 16.036773681640625,
990
+ "lr": 3.581799891711563e-07
991
+ },
992
+ {
993
+ "update": 99,
994
+ "policy_loss": -0.7246200144290924,
995
+ "kl": 0.007142169604776427,
996
+ "bc": 0.7870274186134338,
997
+ "clipfrac": 0.1130952425301075,
998
+ "ratio": 1.0211147964000702,
999
+ "grad_norm": 32.951419830322266,
1000
+ "lr": 3.552398448490204e-07
1001
+ },
1002
+ {
1003
+ "update": 100,
1004
+ "policy_loss": 0.38352592661976814,
1005
+ "kl": 0.0031510748085565865,
1006
+ "bc": 0.0,
1007
+ "clipfrac": 0.13436411321163177,
1008
+ "ratio": 0.9780140519142151,
1009
+ "grad_norm": 26.024097442626953,
1010
+ "lr": 3.5228391763177874e-07
1011
+ },
1012
+ {
1013
+ "update": 101,
1014
+ "policy_loss": -0.16108163446187973,
1015
+ "kl": 0.02177758899051696,
1016
+ "bc": 0.6768733263015747,
1017
+ "clipfrac": 0.22738095931708813,
1018
+ "ratio": 1.03679259121418,
1019
+ "grad_norm": 32.56053161621094,
1020
+ "lr": 3.493127889399126e-07
1021
+ },
1022
+ {
1023
+ "update": 102,
1024
+ "policy_loss": -0.7280345037579536,
1025
+ "kl": 0.0035929589648731053,
1026
+ "bc": 1.4555137753486633,
1027
+ "clipfrac": 0.21956731379032135,
1028
+ "ratio": 1.0029985904693604,
1029
+ "grad_norm": 141.55015563964844,
1030
+ "lr": 3.4632704318397976e-07
1031
+ },
1032
+ {
1033
+ "update": 103,
1034
+ "policy_loss": 0.46010279655456543,
1035
+ "kl": 0.006097094068536535,
1036
+ "bc": 0.0,
1037
+ "clipfrac": 0.18975257128477097,
1038
+ "ratio": 0.9574123322963715,
1039
+ "grad_norm": 16.979164123535156,
1040
+ "lr": 3.433272676496633e-07
1041
+ },
1042
+ {
1043
+ "update": 104,
1044
+ "policy_loss": -0.18419775366783142,
1045
+ "kl": 0.00766886945348233,
1046
+ "bc": 1.084028720855713,
1047
+ "clipfrac": 0.29858193919062614,
1048
+ "ratio": 0.9862464219331741,
1049
+ "grad_norm": 59.71857452392578,
1050
+ "lr": 3.4031405238225405e-07
1051
+ },
1052
+ {
1053
+ "update": 105,
1054
+ "policy_loss": -0.16689463704824448,
1055
+ "kl": 0.006816658744355664,
1056
+ "bc": 0.5334615707397461,
1057
+ "clipfrac": 0.3062500059604645,
1058
+ "ratio": 0.9733274728059769,
1059
+ "grad_norm": 25.801488876342773,
1060
+ "lr": 3.372879900705911e-07
1061
+ },
1062
+ {
1063
+ "update": 106,
1064
+ "policy_loss": -0.817687377333641,
1065
+ "kl": 0.004712065681815147,
1066
+ "bc": 1.2418087422847748,
1067
+ "clipfrac": 0.14629121124744415,
1068
+ "ratio": 1.005902647972107,
1069
+ "grad_norm": 44.91917037963867,
1070
+ "lr": 3.3424967593048216e-07
1071
+ },
1072
+ {
1073
+ "update": 107,
1074
+ "policy_loss": -0.14527320861816406,
1075
+ "kl": 0.011452957231085747,
1076
+ "bc": 0.8884074091911316,
1077
+ "clipfrac": 0.2732142936438322,
1078
+ "ratio": 1.0341073274612427,
1079
+ "grad_norm": 102.02629852294922,
1080
+ "lr": 3.3119970758762603e-07
1081
+ },
1082
+ {
1083
+ "update": 108,
1084
+ "policy_loss": -0.15353950299322605,
1085
+ "kl": 0.0032168398320209235,
1086
+ "bc": 0.4798267185688019,
1087
+ "clipfrac": 0.22415967471897602,
1088
+ "ratio": 0.9889881759881973,
1089
+ "grad_norm": 138.17477416992188,
1090
+ "lr": 3.281386849600627e-07
1091
+ },
1092
+ {
1093
+ "update": 109,
1094
+ "policy_loss": 0.4849424436688423,
1095
+ "kl": 0.0022496887104352936,
1096
+ "bc": 0.0,
1097
+ "clipfrac": 0.11387844756245613,
1098
+ "ratio": 0.9658464044332504,
1099
+ "grad_norm": 35.76771545410156,
1100
+ "lr": 3.250672101401707e-07
1101
+ },
1102
+ {
1103
+ "update": 110,
1104
+ "policy_loss": -0.10173454880714417,
1105
+ "kl": 0.005458016035845503,
1106
+ "bc": 0.5199418663978577,
1107
+ "clipfrac": 0.24873738177120686,
1108
+ "ratio": 0.9752708375453949,
1109
+ "grad_norm": 29.09611701965332,
1110
+ "lr": 3.219858872762382e-07
1111
+ },
1112
+ {
1113
+ "update": 111,
1114
+ "policy_loss": 0.5427076146006584,
1115
+ "kl": 0.007444275222951546,
1116
+ "bc": 0.0,
1117
+ "clipfrac": 0.30333646945655346,
1118
+ "ratio": 0.9394504725933075,
1119
+ "grad_norm": 21.594669342041016,
1120
+ "lr": 3.1889532245362887e-07
1121
+ },
1122
+ {
1123
+ "update": 112,
1124
+ "policy_loss": 0.5717221051454544,
1125
+ "kl": 0.0025899992760969326,
1126
+ "bc": 0.0,
1127
+ "clipfrac": 0.12324134632945061,
1128
+ "ratio": 0.9609829783439636,
1129
+ "grad_norm": 13.80060863494873,
1130
+ "lr": 3.1579612357556707e-07
1131
+ },
1132
+ {
1133
+ "update": 113,
1134
+ "policy_loss": 0.6477869376540184,
1135
+ "kl": 0.001633069732633885,
1136
+ "bc": 0.0,
1137
+ "clipfrac": 0.07341270055621862,
1138
+ "ratio": 0.9736277312040329,
1139
+ "grad_norm": 17.753814697265625,
1140
+ "lr": 3.126889002435657e-07
1141
+ },
1142
+ {
1143
+ "update": 114,
1144
+ "policy_loss": -0.8016757145524025,
1145
+ "kl": 0.008439437166089192,
1146
+ "bc": 2.1725412011146545,
1147
+ "clipfrac": 0.2721291854977608,
1148
+ "ratio": 1.0386827141046524,
1149
+ "grad_norm": 76.680908203125,
1150
+ "lr": 3.095742636375192e-07
1151
+ },
1152
+ {
1153
+ "update": 115,
1154
+ "policy_loss": 0.4861496388912201,
1155
+ "kl": 0.0032499938242835924,
1156
+ "bc": 0.0,
1157
+ "clipfrac": 0.1388888917863369,
1158
+ "ratio": 0.9632406979799271,
1159
+ "grad_norm": 19.49519157409668,
1160
+ "lr": 3.064528263954874e-07
1161
+ },
1162
+ {
1163
+ "update": 116,
1164
+ "policy_loss": -0.7699894160032272,
1165
+ "kl": 0.008691574796102941,
1166
+ "bc": 1.3369492292404175,
1167
+ "clipfrac": 0.24355359748005867,
1168
+ "ratio": 1.0383476465940475,
1169
+ "grad_norm": 43.7039680480957,
1170
+ "lr": 3.033252024931907e-07
1171
+ },
1172
+ {
1173
+ "update": 117,
1174
+ "policy_loss": -0.8034671694040298,
1175
+ "kl": 0.02327919940580614,
1176
+ "bc": 1.254778265953064,
1177
+ "clipfrac": 0.12777778320014477,
1178
+ "ratio": 1.070195510983467,
1179
+ "grad_norm": 35.76602554321289,
1180
+ "lr": 3.001920071232442e-07
1181
+ },
1182
+ {
1183
+ "update": 118,
1184
+ "policy_loss": -0.06427973508834839,
1185
+ "kl": 0.008403046333114617,
1186
+ "bc": 0.6116286516189575,
1187
+ "clipfrac": 0.08773561660200357,
1188
+ "ratio": 1.0153762847185135,
1189
+ "grad_norm": 45.92771911621094,
1190
+ "lr": 2.970538565741511e-07
1191
+ },
1192
+ {
1193
+ "update": 119,
1194
+ "policy_loss": 0.47180959582328796,
1195
+ "kl": 0.0057956767559517175,
1196
+ "bc": 0.0,
1197
+ "clipfrac": 0.16285104490816593,
1198
+ "ratio": 0.951073631644249,
1199
+ "grad_norm": 24.0487117767334,
1200
+ "lr": 2.939113681090807e-07
1201
+ },
1202
+ {
1203
+ "update": 120,
1204
+ "policy_loss": 0.6154569238424301,
1205
+ "kl": 0.0024104636977426708,
1206
+ "bc": 0.0,
1207
+ "clipfrac": 0.13694084249436855,
1208
+ "ratio": 0.963427260518074,
1209
+ "grad_norm": 18.855512619018555,
1210
+ "lr": 2.907651598444552e-07
1211
+ },
1212
+ {
1213
+ "update": 121,
1214
+ "policy_loss": -0.13249771296977997,
1215
+ "kl": 0.011257881356868893,
1216
+ "bc": 1.2037131786346436,
1217
+ "clipfrac": 0.34440476819872856,
1218
+ "ratio": 1.0314175635576248,
1219
+ "grad_norm": 103.34041595458984,
1220
+ "lr": 2.876158506283681e-07
1221
+ },
1222
+ {
1223
+ "update": 122,
1224
+ "policy_loss": 0.4815194681286812,
1225
+ "kl": 0.0027667283720802516,
1226
+ "bc": 0.0,
1227
+ "clipfrac": 0.20337302051484585,
1228
+ "ratio": 0.9528466016054153,
1229
+ "grad_norm": 14.214773178100586,
1230
+ "lr": 2.844640599188597e-07
1231
+ },
1232
+ {
1233
+ "update": 123,
1234
+ "policy_loss": -0.09313835203647614,
1235
+ "kl": 0.01670134239248,
1236
+ "bc": 0.9120008945465088,
1237
+ "clipfrac": 0.17897727340459824,
1238
+ "ratio": 1.0345669239759445,
1239
+ "grad_norm": 50.784725189208984,
1240
+ "lr": 2.8131040766207056e-07
1241
+ },
1242
+ {
1243
+ "update": 124,
1244
+ "policy_loss": 0.34709276631474495,
1245
+ "kl": 0.0025999247445724905,
1246
+ "bc": 0.0,
1247
+ "clipfrac": 0.11725484952330589,
1248
+ "ratio": 0.9622242897748947,
1249
+ "grad_norm": 7.116175651550293,
1250
+ "lr": 2.781555141703015e-07
1251
+ },
1252
+ {
1253
+ "update": 125,
1254
+ "policy_loss": -0.008863568305969238,
1255
+ "kl": 0.005911520114750601,
1256
+ "bc": 0.5235967636108398,
1257
+ "clipfrac": 0.17906746361404657,
1258
+ "ratio": 0.9610164761543274,
1259
+ "grad_norm": 43.19078063964844,
1260
+ "lr": 2.75e-07
1261
+ },
1262
+ {
1263
+ "update": 126,
1264
+ "policy_loss": -0.0914047509431839,
1265
+ "kl": 0.013187595526687801,
1266
+ "bc": 0.6700361967086792,
1267
+ "clipfrac": 0.2492559663951397,
1268
+ "ratio": 0.9613917917013168,
1269
+ "grad_norm": 48.50836181640625,
1270
+ "lr": 2.7184448582969847e-07
1271
+ },
1272
+ {
1273
+ "update": 127,
1274
+ "policy_loss": -0.8338740579783916,
1275
+ "kl": 0.012471641966840252,
1276
+ "bc": 1.4417413473129272,
1277
+ "clipfrac": 0.23245798982679844,
1278
+ "ratio": 1.0373221337795258,
1279
+ "grad_norm": 30.907865524291992,
1280
+ "lr": 2.6868959233792947e-07
1281
+ },
1282
+ {
1283
+ "update": 128,
1284
+ "policy_loss": -0.6662788987159729,
1285
+ "kl": 0.0037896056019235402,
1286
+ "bc": 1.6417542695999146,
1287
+ "clipfrac": 0.2107536792755127,
1288
+ "ratio": 1.0234609991312027,
1289
+ "grad_norm": 45.24131393432617,
1290
+ "lr": 2.6553594008114033e-07
1291
+ },
1292
+ {
1293
+ "update": 129,
1294
+ "policy_loss": -0.9540773779153824,
1295
+ "kl": 0.009773703292012215,
1296
+ "bc": 2.2668979167938232,
1297
+ "clipfrac": 0.22341270372271538,
1298
+ "ratio": 1.0332483351230621,
1299
+ "grad_norm": 76.16902923583984,
1300
+ "lr": 2.623841493716319e-07
1301
+ },
1302
+ {
1303
+ "update": 130,
1304
+ "policy_loss": -0.7758631259202957,
1305
+ "kl": 0.02479085954837501,
1306
+ "bc": 1.775077760219574,
1307
+ "clipfrac": 0.38942307978868484,
1308
+ "ratio": 1.0425720810890198,
1309
+ "grad_norm": 122.66915130615234,
1310
+ "lr": 2.592348401555448e-07
1311
+ },
1312
+ {
1313
+ "update": 131,
1314
+ "policy_loss": -0.2660554125905037,
1315
+ "kl": 0.00948268175125122,
1316
+ "bc": 0.7654210329055786,
1317
+ "clipfrac": 0.1985294222831726,
1318
+ "ratio": 1.0060230195522308,
1319
+ "grad_norm": 38.174949645996094,
1320
+ "lr": 2.5608863189091933e-07
1321
+ },
1322
+ {
1323
+ "update": 132,
1324
+ "policy_loss": -0.7939306050539017,
1325
+ "kl": 0.014540722128003836,
1326
+ "bc": 1.2794857621192932,
1327
+ "clipfrac": 0.3013995476067066,
1328
+ "ratio": 1.025377780199051,
1329
+ "grad_norm": 34.53009796142578,
1330
+ "lr": 2.529461434258489e-07
1331
+ },
1332
+ {
1333
+ "update": 133,
1334
+ "policy_loss": -0.15211518108844757,
1335
+ "kl": 0.00449422569363378,
1336
+ "bc": 0.8219987154006958,
1337
+ "clipfrac": 0.19543651584535837,
1338
+ "ratio": 0.9902503490447998,
1339
+ "grad_norm": 50.23661422729492,
1340
+ "lr": 2.498079928767558e-07
1341
+ },
1342
+ {
1343
+ "update": 134,
1344
+ "policy_loss": 0.37242793664336205,
1345
+ "kl": 0.00482619037211407,
1346
+ "bc": 0.0,
1347
+ "clipfrac": 0.1782200187444687,
1348
+ "ratio": 0.9596484303474426,
1349
+ "grad_norm": 19.877368927001953,
1350
+ "lr": 2.4667479750680926e-07
1351
+ },
1352
+ {
1353
+ "update": 135,
1354
+ "policy_loss": 0.4948083385825157,
1355
+ "kl": 0.007523117703385651,
1356
+ "bc": 0.0,
1357
+ "clipfrac": 0.28761574253439903,
1358
+ "ratio": 0.9336472749710083,
1359
+ "grad_norm": 22.844087600708008,
1360
+ "lr": 2.4354717360451264e-07
1361
+ },
1362
+ {
1363
+ "update": 136,
1364
+ "policy_loss": 0.5161232352256775,
1365
+ "kl": 0.004681747595896013,
1366
+ "bc": 0.0,
1367
+ "clipfrac": 0.2857142947614193,
1368
+ "ratio": 0.9386581778526306,
1369
+ "grad_norm": 20.81160545349121,
1370
+ "lr": 2.4042573636248074e-07
1371
+ },
1372
+ {
1373
+ "update": 137,
1374
+ "policy_loss": 0.4499662443995476,
1375
+ "kl": 0.002458634233335033,
1376
+ "bc": 0.0,
1377
+ "clipfrac": 0.13785885088145733,
1378
+ "ratio": 0.9691950082778931,
1379
+ "grad_norm": 12.595650672912598,
1380
+ "lr": 2.3731109975643432e-07
1381
+ },
1382
+ {
1383
+ "update": 138,
1384
+ "policy_loss": -0.19027069211006165,
1385
+ "kl": 0.012539578819996677,
1386
+ "bc": 0.6115367412567139,
1387
+ "clipfrac": 0.16837607137858868,
1388
+ "ratio": 0.9997258335351944,
1389
+ "grad_norm": 36.07073974609375,
1390
+ "lr": 2.3420387642443296e-07
1391
+ },
1392
+ {
1393
+ "update": 139,
1394
+ "policy_loss": 0.5539462938904762,
1395
+ "kl": 0.006019645894411951,
1396
+ "bc": 0.0,
1397
+ "clipfrac": 0.20351431146264076,
1398
+ "ratio": 0.9465714991092682,
1399
+ "grad_norm": 18.255661010742188,
1400
+ "lr": 2.3110467754637113e-07
1401
+ },
1402
+ {
1403
+ "update": 140,
1404
+ "policy_loss": 0.4358837269246578,
1405
+ "kl": 0.0026838294288609177,
1406
+ "bc": 0.0,
1407
+ "clipfrac": 0.15010302886366844,
1408
+ "ratio": 0.9660115838050842,
1409
+ "grad_norm": 17.377384185791016,
1410
+ "lr": 2.2801411272376176e-07
1411
+ },
1412
+ {
1413
+ "update": 141,
1414
+ "policy_loss": 0.42817045003175735,
1415
+ "kl": 0.002246610034490004,
1416
+ "bc": 0.0,
1417
+ "clipfrac": 0.11329186707735062,
1418
+ "ratio": 0.9656824320554733,
1419
+ "grad_norm": 13.245376586914062,
1420
+ "lr": 2.2493278985982928e-07
1421
+ },
1422
+ {
1423
+ "update": 142,
1424
+ "policy_loss": -0.29643074376508594,
1425
+ "kl": 0.004381276725325733,
1426
+ "bc": 0.7114024758338928,
1427
+ "clipfrac": 0.28686862252652645,
1428
+ "ratio": 0.9799467623233795,
1429
+ "grad_norm": 40.080745697021484,
1430
+ "lr": 2.2186131503993732e-07
1431
+ },
1432
+ {
1433
+ "update": 143,
1434
+ "policy_loss": 0.5071671903133392,
1435
+ "kl": 0.00395230995491147,
1436
+ "bc": 0.0,
1437
+ "clipfrac": 0.22227565199136734,
1438
+ "ratio": 0.9591687172651291,
1439
+ "grad_norm": 24.720291137695312,
1440
+ "lr": 2.1880029241237397e-07
1441
+ },
1442
+ {
1443
+ "update": 144,
1444
+ "policy_loss": 0.4829065129160881,
1445
+ "kl": 0.00674260058440268,
1446
+ "bc": 0.0,
1447
+ "clipfrac": 0.1753246821463108,
1448
+ "ratio": 0.9471403062343597,
1449
+ "grad_norm": 12.745595932006836,
1450
+ "lr": 2.1575032406951787e-07
1451
+ },
1452
+ {
1453
+ "update": 145,
1454
+ "policy_loss": 0.5927018225193024,
1455
+ "kl": 0.002160707605071366,
1456
+ "bc": 0.0,
1457
+ "clipfrac": 0.10492979642003775,
1458
+ "ratio": 0.9670459926128387,
1459
+ "grad_norm": 16.321918487548828,
1460
+ "lr": 2.127120099294089e-07
1461
+ },
1462
+ {
1463
+ "update": 146,
1464
+ "policy_loss": 0.41514017432928085,
1465
+ "kl": 0.005255778087303042,
1466
+ "bc": 0.0,
1467
+ "clipfrac": 0.29935066029429436,
1468
+ "ratio": 0.9448337852954865,
1469
+ "grad_norm": 9.928765296936035,
1470
+ "lr": 2.0968594761774593e-07
1471
+ },
1472
+ {
1473
+ "update": 147,
1474
+ "policy_loss": 0.4216472189873457,
1475
+ "kl": 0.0032482502865605056,
1476
+ "bc": 0.0,
1477
+ "clipfrac": 0.13842674531042576,
1478
+ "ratio": 0.9664092659950256,
1479
+ "grad_norm": 9.996636390686035,
1480
+ "lr": 2.0667273235033666e-07
1481
+ },
1482
+ {
1483
+ "update": 148,
1484
+ "policy_loss": -0.8132306933403015,
1485
+ "kl": 0.00899909669533372,
1486
+ "bc": 2.1240413188934326,
1487
+ "clipfrac": 0.3263392895460129,
1488
+ "ratio": 1.0167763829231262,
1489
+ "grad_norm": 74.27102661132812,
1490
+ "lr": 2.036729568160202e-07
1491
+ },
1492
+ {
1493
+ "update": 149,
1494
+ "policy_loss": -0.2107883244752884,
1495
+ "kl": 0.02516083201044239,
1496
+ "bc": 1.640141487121582,
1497
+ "clipfrac": 0.3869047686457634,
1498
+ "ratio": 1.082545056939125,
1499
+ "grad_norm": 22.927894592285156,
1500
+ "lr": 2.006872110600874e-07
1501
+ },
1502
+ {
1503
+ "update": 150,
1504
+ "policy_loss": 0.42088213935494423,
1505
+ "kl": 0.0036219899775460362,
1506
+ "bc": 0.0,
1507
+ "clipfrac": 0.12379934638738632,
1508
+ "ratio": 0.9665167331695557,
1509
+ "grad_norm": 25.35317039489746,
1510
+ "lr": 1.9771608236822124e-07
1511
+ },
1512
+ {
1513
+ "update": 151,
1514
+ "policy_loss": -0.1866942048072815,
1515
+ "kl": 0.0056472577271051705,
1516
+ "bc": 0.9997265338897705,
1517
+ "clipfrac": 0.2994124013930559,
1518
+ "ratio": 0.9758753329515457,
1519
+ "grad_norm": 59.30940628051758,
1520
+ "lr": 1.947601551509796e-07
1521
+ },
1522
+ {
1523
+ "update": 152,
1524
+ "policy_loss": -0.13033290207386017,
1525
+ "kl": 0.005829456611536443,
1526
+ "bc": 1.1727460622787476,
1527
+ "clipfrac": 0.17038690857589245,
1528
+ "ratio": 1.013729840517044,
1529
+ "grad_norm": 73.14572143554688,
1530
+ "lr": 1.9182001082884365e-07
1531
+ },
1532
+ {
1533
+ "update": 153,
1534
+ "policy_loss": -0.7866647019982338,
1535
+ "kl": 0.008185772690922022,
1536
+ "bc": 1.90780907869339,
1537
+ "clipfrac": 0.37666666880249977,
1538
+ "ratio": 0.9774237424135208,
1539
+ "grad_norm": 81.19231414794922,
1540
+ "lr": 1.888962277178548e-07
1541
+ },
1542
+ {
1543
+ "update": 154,
1544
+ "policy_loss": -0.1924322471022606,
1545
+ "kl": 0.01776626252103597,
1546
+ "bc": 0.8112168908119202,
1547
+ "clipfrac": 0.41607143357396126,
1548
+ "ratio": 0.9187407791614532,
1549
+ "grad_norm": 26.095949172973633,
1550
+ "lr": 1.859893809158618e-07
1551
+ },
1552
+ {
1553
+ "update": 155,
1554
+ "policy_loss": -0.7093217894434929,
1555
+ "kl": 0.012317015512962826,
1556
+ "bc": 2.044274687767029,
1557
+ "clipfrac": 0.22624412551522255,
1558
+ "ratio": 1.0467184782028198,
1559
+ "grad_norm": 59.9600715637207,
1560
+ "lr": 1.831000421894009e-07
1561
+ },
1562
+ {
1563
+ "update": 156,
1564
+ "policy_loss": 0.4846528694033623,
1565
+ "kl": 0.005428007745649666,
1566
+ "bc": 0.0,
1567
+ "clipfrac": 0.23739178106188774,
1568
+ "ratio": 0.9541454613208771,
1569
+ "grad_norm": 11.248618125915527,
1570
+ "lr": 1.8022877986123159e-07
1571
+ },
1572
+ {
1573
+ "update": 157,
1574
+ "policy_loss": 0.33932487666606903,
1575
+ "kl": 0.002773384127067402,
1576
+ "bc": 0.0,
1577
+ "clipfrac": 0.13767176307737827,
1578
+ "ratio": 0.971508115530014,
1579
+ "grad_norm": 13.81155776977539,
1580
+ "lr": 1.7737615869854945e-07
1581
+ },
1582
+ {
1583
+ "update": 158,
1584
+ "policy_loss": -0.18958817422389984,
1585
+ "kl": 0.004370117385406047,
1586
+ "bc": 0.8111776113510132,
1587
+ "clipfrac": 0.24940477311611176,
1588
+ "ratio": 0.9839509725570679,
1589
+ "grad_norm": 23.803937911987305,
1590
+ "lr": 1.7454273980189838e-07
1591
+ },
1592
+ {
1593
+ "update": 159,
1594
+ "policy_loss": 0.5237517580389977,
1595
+ "kl": 0.0035155022487742826,
1596
+ "bc": 0.0,
1597
+ "clipfrac": 0.1502635097131133,
1598
+ "ratio": 0.9562225043773651,
1599
+ "grad_norm": 11.149166107177734,
1600
+ "lr": 1.7172908049480408e-07
1601
+ },
1602
+ {
1603
+ "update": 160,
1604
+ "policy_loss": -0.6242316365242004,
1605
+ "kl": 0.02590106730349362,
1606
+ "bc": 1.3923507928848267,
1607
+ "clipfrac": 0.30669643357396126,
1608
+ "ratio": 1.023161768913269,
1609
+ "grad_norm": 51.60231018066406,
1610
+ "lr": 1.6893573421415054e-07
1611
+ },
1612
+ {
1613
+ "update": 161,
1614
+ "policy_loss": -0.1317959576845169,
1615
+ "kl": 0.009016798925586045,
1616
+ "bc": 0.5791808366775513,
1617
+ "clipfrac": 0.36071430146694183,
1618
+ "ratio": 0.9944023787975311,
1619
+ "grad_norm": 15.098554611206055,
1620
+ "lr": 1.6616325040132104e-07
1621
+ },
1622
+ {
1623
+ "update": 162,
1624
+ "policy_loss": -0.06118093430995941,
1625
+ "kl": 0.005359444650821388,
1626
+ "bc": 1.0066962242126465,
1627
+ "clipfrac": 0.2936166934669018,
1628
+ "ratio": 0.9862392842769623,
1629
+ "grad_norm": 31.295148849487305,
1630
+ "lr": 1.634121743941248e-07
1631
+ },
1632
+ {
1633
+ "update": 163,
1634
+ "policy_loss": -0.2664652206003666,
1635
+ "kl": 0.0057262409645773005,
1636
+ "bc": 0.6303243637084961,
1637
+ "clipfrac": 0.1934523843228817,
1638
+ "ratio": 1.0052708834409714,
1639
+ "grad_norm": 29.24091148376465,
1640
+ "lr": 1.6068304731953092e-07
1641
+ },
1642
+ {
1643
+ "update": 164,
1644
+ "policy_loss": -0.17575354129076004,
1645
+ "kl": 0.08974440195015632,
1646
+ "bc": 0.6282806992530823,
1647
+ "clipfrac": 0.3697612099349499,
1648
+ "ratio": 1.1549614369869232,
1649
+ "grad_norm": 7.301316261291504,
1650
+ "lr": 1.579764059872307e-07
1651
+ },
1652
+ {
1653
+ "update": 165,
1654
+ "policy_loss": -0.21272777765989304,
1655
+ "kl": 0.04105747409630567,
1656
+ "bc": 0.9015478491783142,
1657
+ "clipfrac": 0.4281250014901161,
1658
+ "ratio": 0.9885951429605484,
1659
+ "grad_norm": 45.02065658569336,
1660
+ "lr": 1.552927827840493e-07
1661
+ },
1662
+ {
1663
+ "update": 166,
1664
+ "policy_loss": -1.4214999228715897,
1665
+ "kl": 0.0761992103070952,
1666
+ "bc": 3.4340218901634216,
1667
+ "clipfrac": 0.45822369307279587,
1668
+ "ratio": 1.2402956783771515,
1669
+ "grad_norm": 62.728172302246094,
1670
+ "lr": 1.5263270556922658e-07
1671
+ },
1672
+ {
1673
+ "update": 167,
1674
+ "policy_loss": -0.7767997086048126,
1675
+ "kl": 0.00891138892620802,
1676
+ "bc": 1.196827471256256,
1677
+ "clipfrac": 0.5452381148934364,
1678
+ "ratio": 1.0012703984975815,
1679
+ "grad_norm": 21.129199981689453,
1680
+ "lr": 1.4999669757058948e-07
1681
+ },
1682
+ {
1683
+ "update": 168,
1684
+ "policy_loss": -0.781811848282814,
1685
+ "kl": 0.005839640740305185,
1686
+ "bc": 2.309096336364746,
1687
+ "clipfrac": 0.2661490701138973,
1688
+ "ratio": 1.0389018505811691,
1689
+ "grad_norm": 56.6461181640625,
1690
+ "lr": 1.473852772816349e-07
1691
+ },
1692
+ {
1693
+ "update": 169,
1694
+ "policy_loss": 0.5326934233307838,
1695
+ "kl": 0.012893855746369809,
1696
+ "bc": 0.0,
1697
+ "clipfrac": 0.45019230991601944,
1698
+ "ratio": 0.9189207255840302,
1699
+ "grad_norm": 28.584409713745117,
1700
+ "lr": 1.447989583595437e-07
1701
+ },
1702
+ {
1703
+ "update": 170,
1704
+ "policy_loss": -0.22834327816963196,
1705
+ "kl": 0.035727742419112474,
1706
+ "bc": 1.6214936971664429,
1707
+ "clipfrac": 0.37973630614578724,
1708
+ "ratio": 1.1067697703838348,
1709
+ "grad_norm": 21.884521484375,
1710
+ "lr": 1.422382495241462e-07
1711
+ },
1712
+ {
1713
+ "update": 171,
1714
+ "policy_loss": -0.20136379823088646,
1715
+ "kl": 0.02210965153062716,
1716
+ "bc": 1.1826050281524658,
1717
+ "clipfrac": 0.44047620333731174,
1718
+ "ratio": 1.0474232733249664,
1719
+ "grad_norm": 26.58038902282715,
1720
+ "lr": 1.3970365445785864e-07
1721
+ },
1722
+ {
1723
+ "update": 172,
1724
+ "policy_loss": -0.8181442692875862,
1725
+ "kl": 0.01831141274305992,
1726
+ "bc": 1.1787208914756775,
1727
+ "clipfrac": 0.2714460864663124,
1728
+ "ratio": 1.038434088230133,
1729
+ "grad_norm": 31.50794219970703,
1730
+ "lr": 1.3719567170661122e-07
1731
+ },
1732
+ {
1733
+ "update": 173,
1734
+ "policy_loss": 0.4464041143655777,
1735
+ "kl": 0.0042122508166357875,
1736
+ "bc": 0.0,
1737
+ "clipfrac": 0.22420635726302862,
1738
+ "ratio": 0.9468116909265518,
1739
+ "grad_norm": 11.695716857910156,
1740
+ "lr": 1.3471479458178497e-07
1741
+ },
1742
+ {
1743
+ "update": 174,
1744
+ "policy_loss": 0.48929116874933243,
1745
+ "kl": 0.006243520474527031,
1746
+ "bc": 0.0,
1747
+ "clipfrac": 0.32730015739798546,
1748
+ "ratio": 0.9353606551885605,
1749
+ "grad_norm": 15.101021766662598,
1750
+ "lr": 1.322615110631798e-07
1751
+ },
1752
+ {
1753
+ "update": 175,
1754
+ "policy_loss": 0.4729524776339531,
1755
+ "kl": 0.005075858847703785,
1756
+ "bc": 0.0,
1757
+ "clipfrac": 0.39121419563889503,
1758
+ "ratio": 0.930628776550293,
1759
+ "grad_norm": 13.126051902770996,
1760
+ "lr": 1.2983630370303034e-07
1761
+ },
1762
+ {
1763
+ "update": 176,
1764
+ "policy_loss": 0.36995284259319305,
1765
+ "kl": 0.005287194217089564,
1766
+ "bc": 0.0,
1767
+ "clipfrac": 0.25008481182157993,
1768
+ "ratio": 0.939659059047699,
1769
+ "grad_norm": 12.411783218383789,
1770
+ "lr": 1.2743964953108966e-07
1771
+ },
1772
+ {
1773
+ "update": 177,
1774
+ "policy_loss": -0.7814064994454384,
1775
+ "kl": 0.009147423086687922,
1776
+ "bc": 2.074175000190735,
1777
+ "clipfrac": 0.40119048953056335,
1778
+ "ratio": 1.0117540210485458,
1779
+ "grad_norm": 65.57577514648438,
1780
+ "lr": 1.2507201996079915e-07
1781
+ },
1782
+ {
1783
+ "update": 178,
1784
+ "policy_loss": -0.2150591015815735,
1785
+ "kl": 0.005755379475886002,
1786
+ "bc": 0.721364438533783,
1787
+ "clipfrac": 0.31062273122370243,
1788
+ "ratio": 0.998631164431572,
1789
+ "grad_norm": 17.59869956970215,
1790
+ "lr": 1.2273388069656332e-07
1791
+ },
1792
+ {
1793
+ "update": 179,
1794
+ "policy_loss": -0.20176464319229126,
1795
+ "kl": 0.012139830039814115,
1796
+ "bc": 0.9259139895439148,
1797
+ "clipfrac": 0.36629629880189896,
1798
+ "ratio": 0.9358293116092682,
1799
+ "grad_norm": 52.51183319091797,
1800
+ "lr": 1.2042569164214713e-07
1801
+ },
1802
+ {
1803
+ "update": 180,
1804
+ "policy_loss": 0.5985569655895233,
1805
+ "kl": 0.003611046908190474,
1806
+ "bc": 0.0,
1807
+ "clipfrac": 0.19003851898014545,
1808
+ "ratio": 0.9506955444812775,
1809
+ "grad_norm": 20.74603271484375,
1810
+ "lr": 1.1814790681021478e-07
1811
+ },
1812
+ {
1813
+ "update": 181,
1814
+ "policy_loss": -1.45585098862648,
1815
+ "kl": 0.02253301558084786,
1816
+ "bc": 2.4986151456832886,
1817
+ "clipfrac": 0.3331766948103905,
1818
+ "ratio": 1.109479159116745,
1819
+ "grad_norm": 132.5094451904297,
1820
+ "lr": 1.1590097423302682e-07
1821
+ },
1822
+ {
1823
+ "update": 182,
1824
+ "policy_loss": -0.13990433514118195,
1825
+ "kl": 0.012288626399822533,
1826
+ "bc": 1.118018627166748,
1827
+ "clipfrac": 0.45476191118359566,
1828
+ "ratio": 0.9595870226621628,
1829
+ "grad_norm": 45.27135467529297,
1830
+ "lr": 1.1368533587431388e-07
1831
+ },
1832
+ {
1833
+ "update": 183,
1834
+ "policy_loss": 0.5084870085120201,
1835
+ "kl": 0.0038982827099971473,
1836
+ "bc": 0.0,
1837
+ "clipfrac": 0.1877705715596676,
1838
+ "ratio": 0.9459741413593292,
1839
+ "grad_norm": 12.422896385192871,
1840
+ "lr": 1.1150142754234391e-07
1841
+ },
1842
+ {
1843
+ "update": 184,
1844
+ "policy_loss": -0.16922210156917572,
1845
+ "kl": 0.01880211930256337,
1846
+ "bc": 0.6023637056350708,
1847
+ "clipfrac": 0.4326073247939348,
1848
+ "ratio": 0.9767763614654541,
1849
+ "grad_norm": 23.138647079467773,
1850
+ "lr": 1.093496788042004e-07
1851
+ },
1852
+ {
1853
+ "update": 185,
1854
+ "policy_loss": -0.22641828656196594,
1855
+ "kl": 0.011414454027544707,
1856
+ "bc": 0.46532905101776123,
1857
+ "clipfrac": 0.4263305440545082,
1858
+ "ratio": 0.9784712493419647,
1859
+ "grad_norm": 29.720129013061523,
1860
+ "lr": 1.0723051290128773e-07
1861
+ },
1862
+ {
1863
+ "update": 186,
1864
+ "policy_loss": -0.1238740086555481,
1865
+ "kl": 0.011308514745905995,
1866
+ "bc": 0.7013340592384338,
1867
+ "clipfrac": 0.4669659212231636,
1868
+ "ratio": 0.9545435458421707,
1869
+ "grad_norm": 38.85552978515625,
1870
+ "lr": 1.0514434666608143e-07
1871
+ },
1872
+ {
1873
+ "update": 187,
1874
+ "policy_loss": 0.38045115768909454,
1875
+ "kl": 0.007569728069938719,
1876
+ "bc": 0.0,
1877
+ "clipfrac": 0.2996031828224659,
1878
+ "ratio": 0.9507016688585281,
1879
+ "grad_norm": 11.247143745422363,
1880
+ "lr": 1.0309159044013837e-07
1881
+ },
1882
+ {
1883
+ "update": 188,
1884
+ "policy_loss": -0.14893551915884018,
1885
+ "kl": 0.005075723864138126,
1886
+ "bc": 0.8231615424156189,
1887
+ "clipfrac": 0.23244048096239567,
1888
+ "ratio": 0.9861000925302505,
1889
+ "grad_norm": 32.03021240234375,
1890
+ "lr": 1.0107264799338425e-07
1891
+ },
1892
+ {
1893
+ "update": 189,
1894
+ "policy_loss": -0.8858140781521797,
1895
+ "kl": 0.033990635245572776,
1896
+ "bc": 1.607565462589264,
1897
+ "clipfrac": 0.2529411781579256,
1898
+ "ratio": 1.096856489777565,
1899
+ "grad_norm": 31.50334358215332,
1900
+ "lr": 9.908791644469336e-08
1901
+ },
1902
+ {
1903
+ "update": 190,
1904
+ "policy_loss": -0.800799123942852,
1905
+ "kl": 0.007067363825626671,
1906
+ "bc": 1.3741395473480225,
1907
+ "clipfrac": 0.3175790272653103,
1908
+ "ratio": 1.0140328705310822,
1909
+ "grad_norm": 37.9118766784668,
1910
+ "lr": 9.713778618377669e-08
1911
+ },
1912
+ {
1913
+ "update": 191,
1914
+ "policy_loss": -0.09286663681268692,
1915
+ "kl": 0.007438857166562229,
1916
+ "bc": 1.3837733268737793,
1917
+ "clipfrac": 0.3009875677525997,
1918
+ "ratio": 0.9923033267259598,
1919
+ "grad_norm": 23.500221252441406,
1920
+ "lr": 9.522264079439394e-08
1921
+ },
1922
+ {
1923
+ "update": 192,
1924
+ "policy_loss": 0.5343745201826096,
1925
+ "kl": 0.005179949919693172,
1926
+ "bc": 0.0,
1927
+ "clipfrac": 0.24186057783663273,
1928
+ "ratio": 0.9456040859222412,
1929
+ "grad_norm": 11.598170280456543,
1930
+ "lr": 9.334285697890345e-08
1931
+ },
1932
+ {
1933
+ "update": 193,
1934
+ "policy_loss": -0.11494488269090652,
1935
+ "kl": 0.004718655138276517,
1936
+ "bc": 0.3813433051109314,
1937
+ "clipfrac": 0.1675824224948883,
1938
+ "ratio": 0.9922601282596588,
1939
+ "grad_norm": 20.438251495361328,
1940
+ "lr": 9.149880448416648e-08
1941
+ },
1942
+ {
1943
+ "update": 194,
1944
+ "policy_loss": -0.1975642666220665,
1945
+ "kl": 0.009758722793776542,
1946
+ "bc": 0.5255277752876282,
1947
+ "clipfrac": 0.35676805675029755,
1948
+ "ratio": 0.9759582281112671,
1949
+ "grad_norm": 17.81694793701172,
1950
+ "lr": 8.969084602881921e-08
1951
+ },
1952
+ {
1953
+ "update": 195,
1954
+ "policy_loss": -0.16617629677057266,
1955
+ "kl": 0.015513387537794188,
1956
+ "bc": 1.0714190006256104,
1957
+ "clipfrac": 0.37380953319370747,
1958
+ "ratio": 0.9813934415578842,
1959
+ "grad_norm": 17.91180992126465,
1960
+ "lr": 8.791933723192737e-08
1961
+ },
1962
+ {
1963
+ "update": 196,
1964
+ "policy_loss": 0.42807020992040634,
1965
+ "kl": 0.007878909585997462,
1966
+ "bc": 0.0,
1967
+ "clipfrac": 0.3611111156642437,
1968
+ "ratio": 0.9152966439723969,
1969
+ "grad_norm": 12.735188484191895,
1970
+ "lr": 8.618462654303701e-08
1971
+ },
1972
+ {
1973
+ "update": 197,
1974
+ "policy_loss": 0.3960537314414978,
1975
+ "kl": 0.00400349564733915,
1976
+ "bc": 0.0,
1977
+ "clipfrac": 0.24238202162086964,
1978
+ "ratio": 0.9503706693649292,
1979
+ "grad_norm": 16.328781127929688,
1980
+ "lr": 8.448705517363608e-08
1981
+ },
1982
+ {
1983
+ "update": 198,
1984
+ "policy_loss": 0.18183963117189705,
1985
+ "kl": 0.003471619333140552,
1986
+ "bc": 0.0,
1987
+ "clipfrac": 0.13838661834597588,
1988
+ "ratio": 0.9633630812168121,
1989
+ "grad_norm": 12.24112319946289,
1990
+ "lr": 8.282695703003884e-08
1991
+ },
1992
+ {
1993
+ "update": 199,
1994
+ "policy_loss": -0.22901365906000137,
1995
+ "kl": 0.0068230390606913716,
1996
+ "bc": 0.810291588306427,
1997
+ "clipfrac": 0.23414820805191994,
1998
+ "ratio": 0.9976735562086105,
1999
+ "grad_norm": 25.992286682128906,
2000
+ "lr": 8.120465864770804e-08
2001
+ },
2002
+ {
2003
+ "update": 200,
2004
+ "policy_loss": 0.5745870694518089,
2005
+ "kl": 0.002352958734263666,
2006
+ "bc": 0.0,
2007
+ "clipfrac": 0.11659356765449047,
2008
+ "ratio": 0.9621504545211792,
2009
+ "grad_norm": 12.282270431518555,
2010
+ "lr": 7.962047912702629e-08
2011
+ },
2012
+ {
2013
+ "update": 201,
2014
+ "policy_loss": -0.19391578435897827,
2015
+ "kl": 0.00762225576909259,
2016
+ "bc": 0.7799859642982483,
2017
+ "clipfrac": 0.2762179262936115,
2018
+ "ratio": 0.987635999917984,
2019
+ "grad_norm": 22.866992950439453,
2020
+ "lr": 7.807473007053042e-08
2021
+ },
2022
+ {
2023
+ "update": 202,
2024
+ "policy_loss": -0.18833769112825394,
2025
+ "kl": 0.017759457463398576,
2026
+ "bc": 0.9773730635643005,
2027
+ "clipfrac": 0.3374338671565056,
2028
+ "ratio": 0.9495762884616852,
2029
+ "grad_norm": 64.86711120605469,
2030
+ "lr": 7.656771552162008e-08
2031
+ },
2032
+ {
2033
+ "update": 203,
2034
+ "policy_loss": -0.26496498100459576,
2035
+ "kl": 0.010653663310222328,
2036
+ "bc": 1.145985722541809,
2037
+ "clipfrac": 0.4002463072538376,
2038
+ "ratio": 0.9764825999736786,
2039
+ "grad_norm": 29.16523551940918,
2040
+ "lr": 7.509973190475371e-08
2041
+ },
2042
+ {
2043
+ "update": 204,
2044
+ "policy_loss": -0.04434698820114136,
2045
+ "kl": 0.015695500187575817,
2046
+ "bc": 0.5019158124923706,
2047
+ "clipfrac": 0.30999599676579237,
2048
+ "ratio": 0.9886054545640945,
2049
+ "grad_norm": 12.579261779785156,
2050
+ "lr": 7.367106796714263e-08
2051
+ },
2052
+ {
2053
+ "update": 205,
2054
+ "policy_loss": -0.08587321639060974,
2055
+ "kl": 0.003963867842685431,
2056
+ "bc": 0.5191554427146912,
2057
+ "clipfrac": 0.24201128631830215,
2058
+ "ratio": 0.9821504652500153,
2059
+ "grad_norm": 35.572242736816406,
2060
+ "lr": 7.228200472195572e-08
2061
+ },
2062
+ {
2063
+ "update": 206,
2064
+ "policy_loss": 0.4585028253495693,
2065
+ "kl": 0.003865327234962024,
2066
+ "bc": 0.0,
2067
+ "clipfrac": 0.17550505138933659,
2068
+ "ratio": 0.9499169290065765,
2069
+ "grad_norm": 10.823963165283203,
2070
+ "lr": 7.093281539304492e-08
2071
+ },
2072
+ {
2073
+ "update": 207,
2074
+ "policy_loss": -0.12399858236312866,
2075
+ "kl": 0.00591503331088461,
2076
+ "bc": 1.0452237129211426,
2077
+ "clipfrac": 0.26981872878968716,
2078
+ "ratio": 0.9851301461458206,
2079
+ "grad_norm": 55.08019256591797,
2080
+ "lr": 6.962376536120327e-08
2081
+ },
2082
+ {
2083
+ "update": 208,
2084
+ "policy_loss": -0.7821331918239594,
2085
+ "kl": 0.025322271860204637,
2086
+ "bc": 1.4819339513778687,
2087
+ "clipfrac": 0.44929973408579826,
2088
+ "ratio": 1.0769125372171402,
2089
+ "grad_norm": 57.30143356323242,
2090
+ "lr": 6.83551121119653e-08
2091
+ },
2092
+ {
2093
+ "update": 209,
2094
+ "policy_loss": 0.5633153319358826,
2095
+ "kl": 0.0035923445248045027,
2096
+ "bc": 0.0,
2097
+ "clipfrac": 0.13523253984749317,
2098
+ "ratio": 0.9675768613815308,
2099
+ "grad_norm": 25.936527252197266,
2100
+ "lr": 6.712710518496048e-08
2101
+ },
2102
+ {
2103
+ "update": 210,
2104
+ "policy_loss": 0.44904763251543045,
2105
+ "kl": 0.0021378881647251546,
2106
+ "bc": 0.0,
2107
+ "clipfrac": 0.0861330721527338,
2108
+ "ratio": 0.9566548019647598,
2109
+ "grad_norm": 10.572012901306152,
2110
+ "lr": 6.593998612482993e-08
2111
+ },
2112
+ {
2113
+ "update": 211,
2114
+ "policy_loss": 0.5997125804424286,
2115
+ "kl": 0.00534545021946542,
2116
+ "bc": 0.0,
2117
+ "clipfrac": 0.3321764525026083,
2118
+ "ratio": 0.9320560693740845,
2119
+ "grad_norm": 16.28018569946289,
2120
+ "lr": 6.479398843371502e-08
2121
+ },
2122
+ {
2123
+ "update": 212,
2124
+ "policy_loss": 0.41220862604677677,
2125
+ "kl": 0.0069093137281015515,
2126
+ "bc": 0.0,
2127
+ "clipfrac": 0.31602565199136734,
2128
+ "ratio": 0.9343459159135818,
2129
+ "grad_norm": 13.204140663146973,
2130
+ "lr": 6.368933752532874e-08
2131
+ },
2132
+ {
2133
+ "update": 213,
2134
+ "policy_loss": -0.08928822726011276,
2135
+ "kl": 0.013354651164263487,
2136
+ "bc": 0.6552591323852539,
2137
+ "clipfrac": 0.30000000447034836,
2138
+ "ratio": 0.9808294326066971,
2139
+ "grad_norm": 59.38974380493164,
2140
+ "lr": 6.262625068061729e-08
2141
+ },
2142
+ {
2143
+ "update": 214,
2144
+ "policy_loss": 0.44168490171432495,
2145
+ "kl": 0.006905473885126412,
2146
+ "bc": 0.0,
2147
+ "clipfrac": 0.3164842650294304,
2148
+ "ratio": 0.92221999168396,
2149
+ "grad_norm": 13.673892974853516,
2150
+ "lr": 6.16049370050219e-08
2151
+ },
2152
+ {
2153
+ "update": 215,
2154
+ "policy_loss": -0.31594472005963326,
2155
+ "kl": 0.012427664041751996,
2156
+ "bc": 0.5998329520225525,
2157
+ "clipfrac": 0.18316625244915485,
2158
+ "ratio": 1.0253362655639648,
2159
+ "grad_norm": 14.029699325561523,
2160
+ "lr": 6.062559738734832e-08
2161
+ },
2162
+ {
2163
+ "update": 216,
2164
+ "policy_loss": 0.4765186756849289,
2165
+ "kl": 0.006880268163513392,
2166
+ "bc": 0.0,
2167
+ "clipfrac": 0.3195767290890217,
2168
+ "ratio": 0.9289685785770416,
2169
+ "grad_norm": 11.629363059997559,
2170
+ "lr": 5.968842446025302e-08
2171
+ },
2172
+ {
2173
+ "update": 217,
2174
+ "policy_loss": -0.1911810263991356,
2175
+ "kl": 0.008599135500844568,
2176
+ "bc": 0.40800967812538147,
2177
+ "clipfrac": 0.30998170375823975,
2178
+ "ratio": 0.9529530853033066,
2179
+ "grad_norm": 20.554332733154297,
2180
+ "lr": 5.8793602562352614e-08
2181
+ },
2182
+ {
2183
+ "update": 218,
2184
+ "policy_loss": -0.31329971365630627,
2185
+ "kl": 0.017185386328492314,
2186
+ "bc": 0.6616475582122803,
2187
+ "clipfrac": 0.28882576525211334,
2188
+ "ratio": 1.0050361156463623,
2189
+ "grad_norm": 27.733449935913086,
2190
+ "lr": 5.7941307701965444e-08
2191
+ },
2192
+ {
2193
+ "update": 219,
2194
+ "policy_loss": 0.3968544602394104,
2195
+ "kl": 0.0034502759808674455,
2196
+ "bc": 0.0,
2197
+ "clipfrac": 0.21412907727062702,
2198
+ "ratio": 0.9545861482620239,
2199
+ "grad_norm": 12.406554222106934,
2200
+ "lr": 5.7131707522491196e-08
2201
+ },
2202
+ {
2203
+ "update": 220,
2204
+ "policy_loss": -0.19086821377277374,
2205
+ "kl": 0.005171792930923402,
2206
+ "bc": 0.907887876033783,
2207
+ "clipfrac": 0.211805559694767,
2208
+ "ratio": 0.9823577702045441,
2209
+ "grad_norm": 39.891273498535156,
2210
+ "lr": 5.636496126943605e-08
2211
+ },
2212
+ {
2213
+ "update": 221,
2214
+ "policy_loss": 0.7168167605996132,
2215
+ "kl": 0.0034180156071670353,
2216
+ "bc": 0.0,
2217
+ "clipfrac": 0.19162088632583618,
2218
+ "ratio": 0.9539076387882233,
2219
+ "grad_norm": 26.670072555541992,
2220
+ "lr": 5.564121975908966e-08
2221
+ },
2222
+ {
2223
+ "update": 222,
2224
+ "policy_loss": 0.4930870980024338,
2225
+ "kl": 0.007206271751783788,
2226
+ "bc": 0.0,
2227
+ "clipfrac": 0.3430555574595928,
2228
+ "ratio": 0.9236617833375931,
2229
+ "grad_norm": 16.749170303344727,
2230
+ "lr": 5.496062534886026e-08
2231
+ },
2232
+ {
2233
+ "update": 223,
2234
+ "policy_loss": -0.8105712458491325,
2235
+ "kl": 0.02402468817308545,
2236
+ "bc": 1.3979585766792297,
2237
+ "clipfrac": 0.4750000089406967,
2238
+ "ratio": 1.0091509073972702,
2239
+ "grad_norm": 77.14275360107422,
2240
+ "lr": 5.432331190927315e-08
2241
+ },
2242
+ {
2243
+ "update": 224,
2244
+ "policy_loss": -0.10476350039243698,
2245
+ "kl": 0.009918495343299583,
2246
+ "bc": 1.4061574935913086,
2247
+ "clipfrac": 0.2722884640097618,
2248
+ "ratio": 1.025970220565796,
2249
+ "grad_norm": 19.39459228515625,
2250
+ "lr": 5.372940479763911e-08
2251
+ },
2252
+ {
2253
+ "update": 225,
2254
+ "policy_loss": 0.6387261599302292,
2255
+ "kl": 0.005038053554017097,
2256
+ "bc": 0.0,
2257
+ "clipfrac": 0.2683905679732561,
2258
+ "ratio": 0.9582431763410568,
2259
+ "grad_norm": 10.114340782165527,
2260
+ "lr": 5.3179020833396924e-08
2261
+ },
2262
+ {
2263
+ "update": 226,
2264
+ "policy_loss": -0.21396557986736298,
2265
+ "kl": 0.009834777156356722,
2266
+ "bc": 0.6272738575935364,
2267
+ "clipfrac": 0.37755682319402695,
2268
+ "ratio": 0.9561335891485214,
2269
+ "grad_norm": 26.264175415039062,
2270
+ "lr": 5.267226827513552e-08
2271
+ },
2272
+ {
2273
+ "update": 227,
2274
+ "policy_loss": 0.5056896954774857,
2275
+ "kl": 0.004480219562537968,
2276
+ "bc": 0.0,
2277
+ "clipfrac": 0.1971726231276989,
2278
+ "ratio": 0.9605302661657333,
2279
+ "grad_norm": 22.489315032958984,
2280
+ "lr": 5.2209246799299756e-08
2281
+ },
2282
+ {
2283
+ "update": 228,
2284
+ "policy_loss": -0.11348351091146469,
2285
+ "kl": 0.0046789165935479105,
2286
+ "bc": 0.6506944894790649,
2287
+ "clipfrac": 0.2508416883647442,
2288
+ "ratio": 0.9789642989635468,
2289
+ "grad_norm": 27.75130271911621,
2290
+ "lr": 5.179004748058445e-08
2291
+ },
2292
+ {
2293
+ "update": 229,
2294
+ "policy_loss": 0.47661878168582916,
2295
+ "kl": 0.009887116961181164,
2296
+ "bc": 0.0,
2297
+ "clipfrac": 0.43086082488298416,
2298
+ "ratio": 0.9217001497745514,
2299
+ "grad_norm": 31.38336181640625,
2300
+ "lr": 5.1414752774020416e-08
2301
+ },
2302
+ {
2303
+ "update": 230,
2304
+ "policy_loss": -0.259795306250453,
2305
+ "kl": 0.031804178084712476,
2306
+ "bc": 0.376176655292511,
2307
+ "clipfrac": 0.1592421941459179,
2308
+ "ratio": 1.0570754557847977,
2309
+ "grad_norm": 12.38129711151123,
2310
+ "lr": 5.108343649875572e-08
2311
+ },
2312
+ {
2313
+ "update": 231,
2314
+ "policy_loss": -0.7767976820468903,
2315
+ "kl": 0.008196834853151813,
2316
+ "bc": 1.5988484621047974,
2317
+ "clipfrac": 0.2288961075246334,
2318
+ "ratio": 1.0292986184358597,
2319
+ "grad_norm": 44.38459777832031,
2320
+ "lr": 5.079616382353571e-08
2321
+ },
2322
+ {
2323
+ "update": 232,
2324
+ "policy_loss": 0.45308832824230194,
2325
+ "kl": 0.00577791512478143,
2326
+ "bc": 0.0,
2327
+ "clipfrac": 0.2626923155039549,
2328
+ "ratio": 0.9330124258995056,
2329
+ "grad_norm": 10.215874671936035,
2330
+ "lr": 5.055299125388464e-08
2331
+ },
2332
+ {
2333
+ "update": 233,
2334
+ "policy_loss": -0.1293880194425583,
2335
+ "kl": 0.04098668703227304,
2336
+ "bc": 0.5506393313407898,
2337
+ "clipfrac": 0.22456597443670034,
2338
+ "ratio": 1.0735907554626465,
2339
+ "grad_norm": 8.662755012512207,
2340
+ "lr": 5.0353966620991265e-08
2341
+ },
2342
+ {
2343
+ "update": 234,
2344
+ "policy_loss": 0.47508352994918823,
2345
+ "kl": 0.006448009284213185,
2346
+ "bc": 0.0,
2347
+ "clipfrac": 0.292316023260355,
2348
+ "ratio": 0.9280217289924622,
2349
+ "grad_norm": 12.319525718688965,
2350
+ "lr": 5.019912907230045e-08
2351
+ },
2352
+ {
2353
+ "update": 235,
2354
+ "policy_loss": -0.17280161380767822,
2355
+ "kl": 0.007811228220816702,
2356
+ "bc": 2.0807101726531982,
2357
+ "clipfrac": 0.4126984253525734,
2358
+ "ratio": 1.0033964812755585,
2359
+ "grad_norm": 41.84281539916992,
2360
+ "lr": 5.008850906381311e-08
2361
+ },
2362
+ {
2363
+ "update": 236,
2364
+ "policy_loss": 0.4406544119119644,
2365
+ "kl": 0.010217002243734896,
2366
+ "bc": 0.0,
2367
+ "clipfrac": 0.38307179883122444,
2368
+ "ratio": 0.9113075137138367,
2369
+ "grad_norm": 18.533588409423828,
2370
+ "lr": 5.002212835409562e-08
2371
+ },
2372
+ {
2373
+ "update": 237,
2374
+ "policy_loss": 0.4860967695713043,
2375
+ "kl": 0.002052091294899583,
2376
+ "bc": 0.0,
2377
+ "clipfrac": 0.1111111119389534,
2378
+ "ratio": 0.9653682708740234,
2379
+ "grad_norm": 4.0564093589782715,
2380
+ "lr": 5e-08
2381
+ }
2382
+ ],
2383
+ "source_path": "/marimo/budgie_information_rl_v1/info_rl_grpo_long_source.pt",
2384
+ "seconds": 63.36139249801636,
2385
+ "parameter_count_change": 0
2386
+ }