Hugging Face's logo

psp-dada
/
Qwen2.5-Math-7B-Uni-DPO

Text Generation
Transformers
Safetensors
English
qwen2
bf16
qwen
RLHF
DPO
text-generation-inference
Model card Files
xet
 Community
psp-dada commited on
Commit
d3f8911
·
0 Parent(s):

first commit

Browse files
Files changed (20) hide show
  1. .gitattributes +37 -0
  2. .gitignore +1 -0
  3. README.md +117 -0
  4. added_tokens.json +24 -0
  5. config.json +29 -0
  6. generation_config.json +6 -0
  7. merges.txt +0 -0
  8. model-00001-of-00004.safetensors +3 -0
  9. model-00002-of-00004.safetensors +3 -0
  10. model-00003-of-00004.safetensors +3 -0
  11. model-00004-of-00004.safetensors +3 -0
  12. model.safetensors.index.json +346 -0
  13. special_tokens_map.json +26 -0
  14. tokenizer.json +3 -0
  15. tokenizer_config.json +3 -0
  16. train_results/all_results.json +8 -0
  17. train_results/train_results.json +8 -0
  18. train_results/trainer_state.json +1836 -0
  19. training_args.bin +3 -0
  20. vocab.json +0 -0
.gitattributes ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer_config.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ ./checkpoint-450/*
README.md ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - psp-dada/Uni-DPO
5
+ language:
6
+ - en
7
+ base_model:
8
+ - Qwen/Qwen2.5-Math-7B
9
+ pipeline_tag: text-generation
10
+ library_name: transformers
11
+ tags:
12
+ - bf16
13
+ - qwen
14
+ - RLHF
15
+ - DPO
16
+ ---
17
+
18
+ # Model Card for `psp-dada/Qwen2.5-Math-7B-Uni-DPO` | ICLR 2026 | Uni-DPO: <br> A Unified Paradigm for Dynamic Preference Optimization of LLMs <!-- omit in toc -->
19
+
20
+ <a href='https://arxiv.org/abs/2506.10054'>
21
+ <img src='https://img.shields.io/badge/Paper-Arxiv-purple'></a>
22
+ <a href='https://huggingface.co/datasets/psp-dada/Uni-DPO'>
23
+ <img src='https://img.shields.io/badge/Datasets-HF-Green'></a>
24
+ <a href='https://huggingface.co/collections/psp-dada/uni-dpo'>
25
+ <img src='https://img.shields.io/badge/Models-HF-orange'></a>
26
+ <a href='https://huggingface.co/papers/2506.10054'>
27
+ <img src='https://img.shields.io/badge/Discussion-HF-blue'></a>
28
+ <a href='https://github.com/pspdada/Uni-DPO/blob/main/LICENSE'>
29
+ <img src='https://img.shields.io/badge/LICENSE-Apache_2.0-yellow'></a>
30
+
31
+ <a href='https://modelscope.cn/datasets/pspdada/Uni-DPO'>
32
+ <img src='https://img.shields.io/badge/Datasets-🤖ModelScope-pink'></a>
33
+ <a href='https://modelscope.cn/collections/pspdada/Uni-DPO'>
34
+ <img src='https://img.shields.io/badge/Models-🤖ModelScope-red'></a>
35
+
36
+ ## 🎊 News <!-- omit in toc -->
37
+
38
+ - [2026.02.16] 📖 Code, data, and models are released!
39
+ - [2026.01.26] 🎉 Our Uni-DPO is accepted by **ICLR 2026**!
40
+
41
+ ## 🚀 Overview <!-- omit in toc -->
42
+
43
+ **Uni-DPO** introduces a unified dynamic preference optimization paradigm for training large language models (LLMs) from preference data. Unlike prior DPO-based methods that treat all preference pairs equally, Uni-DPO jointly considers **intrinsic data quality** and **model learning dynamics**, enabling more effective and robust preference learning.
44
+
45
+ **Key advantages:**
46
+
47
+ - **Quality-aware**: Adaptively prioritizes high-quality preference pairs while down-weighting ambiguous ones.
48
+ - **Dynamics-aware**: Shifts training focus toward under-fitted samples to mitigate overfitting.
49
+ - **Unified & lightweight**: Seamlessly integrates dual-perspective weighting and calibrated NLL into standard DPO with minimal overhead.
50
+
51
+ ## 🔑 Key Features
52
+
53
+ - **Dual-perspective dynamic weighting for preference optimization**.
54
+ Uni-DPO jointly models _what data is worth learning_ (intrinsic quality) and _what the model still struggles with_ (learning dynamics). By combining a quality-aware weight and a performance-aware weight, Uni-DPO dynamically reallocates training focus throughout optimization.
55
+
56
+ <table align="center">
57
+ <p align="center">
58
+ <img src="https://github.com/pspdada/Uni-DPO/raw/main/docs/figures/figure1.png" width="80%" />
59
+ </p>
60
+ </table>
61
+
62
+ - **Quality-aware weighting filters ambiguous preference pairs**.
63
+ Preference data varies widely in reliability. Uni-DPO leverages score margins between preferred and rejected responses to assign higher weights to clear, high-quality pairs while suppressing noisy or ambiguous ones.
64
+
65
+ <table align="center">
66
+ <p align="center">
67
+ <img src="https://github.com/pspdada/Uni-DPO/raw/main/docs/figures/figure2.png" width="80%" />
68
+ </p>
69
+ </table>
70
+
71
+ - **Performance-aware weighting mitigates overfitting during training**.
72
+ High-quality samples are not always the most informative once the model has already mastered them. Uni-DPO introduces a stabilized focal-style performance weight that down-weights well-fitted pairs and emphasizes hard-but-informative examples, effectively reducing overfitting.
73
+
74
+ <table align="center">
75
+ <p align="center">
76
+ <img src="https://github.com/pspdada/Uni-DPO/raw/main/docs/figures/figure3.png" width="80%" />
77
+ </p>
78
+ </table>
79
+
80
+ - **Decoupling data quality from learning difficulty**.
81
+ Empirical analysis reveals that data quality (score margin) and learning difficulty (reward margin) are weakly correlated. Uni-DPO explicitly models this mismatch, ensuring that optimization is guided by both dimensions rather than relying on either alone.
82
+
83
+ <table align="center">
84
+ <p align="center">
85
+ <img src="https://github.com/pspdada/Uni-DPO/raw/main/docs/figures/figure4.png" width="80%" />
86
+ </p>
87
+ </table>
88
+
89
+ - **State-of-the-art performance across text, math, and multimodal benchmarks**.
90
+ Uni-DPO consistently outperforms DPO and SimPO across diverse settings.
91
+
92
+ <table align="center">
93
+ <p align="center">
94
+ <img src="https://github.com/pspdada/Uni-DPO/raw/main/docs/figures/table1.png" width="80%" />
95
+ </p>
96
+ </table>
97
+
98
+ ## How to use
99
+
100
+ For the details of this model, please refer to the [documentation](https://github.com/pspdada/Uni-DPO?tab=readme-ov-file#-model-weights) of the GitHub repo.
101
+
102
+ ## 📝 Citation
103
+
104
+ If you find our model/code/data/paper helpful, please consider citing our papers 📝 and starring us ⭐️！
105
+
106
+ ```bibtex
107
+ @article{peng2025omni,
108
+ title={Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs},
109
+ author={Peng, Shangpin and Wang, Weinong and Tian, Zhuotao and Yang, Senqiao and Wu, Xing and Xu, Haotian and Zhang, Chengquan and Isobe, Takashi and Hu, Baotian and Zhang, Min},
110
+ journal={arXiv preprint arXiv:2506.10054},
111
+ year={2025}
112
+ }
113
+ ```
114
+
115
+ ## 📧 Contact us <!-- omit in toc -->
116
+
117
+ If you have any questions, comments, or suggestions, please do not hesitate to submit an issue or PR to help advance research in this area.
added_tokens.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</tool_call>": 151658,
3
+ "<tool_call>": 151657,
4
+ "<|box_end|>": 151649,
5
+ "<|box_start|>": 151648,
6
+ "<|endoftext|>": 151643,
7
+ "<|file_sep|>": 151664,
8
+ "<|fim_middle|>": 151660,
9
+ "<|fim_pad|>": 151662,
10
+ "<|fim_prefix|>": 151659,
11
+ "<|fim_suffix|>": 151661,
12
+ "<|im_end|>": 151645,
13
+ "<|im_start|>": 151644,
14
+ "<|image_pad|>": 151655,
15
+ "<|object_ref_end|>": 151647,
16
+ "<|object_ref_start|>": 151646,
17
+ "<|quad_end|>": 151651,
18
+ "<|quad_start|>": 151650,
19
+ "<|repo_name|>": 151663,
20
+ "<|video_pad|>": 151656,
21
+ "<|vision_end|>": 151653,
22
+ "<|vision_pad|>": 151654,
23
+ "<|vision_start|>": 151652
24
+ }
config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "Qwen/Qwen2.5-Math-7B",
3
+ "architectures": [
4
+ "Qwen2ForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 151643,
8
+ "eos_token_id": 151643,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 3584,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 18944,
13
+ "max_position_embeddings": 4096,
14
+ "max_window_layers": 28,
15
+ "model_type": "qwen2",
16
+ "num_attention_heads": 28,
17
+ "num_hidden_layers": 28,
18
+ "num_key_value_heads": 4,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_theta": 10000,
21
+ "sliding_window": 4096,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "bfloat16",
24
+ "transformers_version": "4.42.2",
25
+ "use_cache": false,
26
+ "use_mrope": false,
27
+ "use_sliding_window": false,
28
+ "vocab_size": 152064
29
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "eos_token_id": 151643,
4
+ "max_new_tokens": 2048,
5
+ "transformers_version": "4.42.2"
6
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9763c2ce74a1f5bba642b967d0194d3f67dbc65f1fb5b58b3191679b0147ed11
3
+ size 4877660776
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d2f8c5e98ced651a784a470fffde5f734c3f420ec70f8c445e8268fb410cebc
3
+ size 4932751008
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:132d0043af569fd695a4f5bc1779cb1f64a3c89ad387eece3dfe4314fd9e5503
3
+ size 4330865200
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:342915d354304f8f5f614f325c253c314e54ec23a68d2b458f821bc3cedca4d6
3
+ size 1089994880
model.safetensors.index.json ADDED
@@ -0,0 +1,346 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 15231233024
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00004-of-00004.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
8
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
9
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
10
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
11
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
12
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
13
+ "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
14
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
15
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
16
+ "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
17
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
18
+ "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
19
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
20
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
21
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
22
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
23
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
24
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
25
+ "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
26
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
27
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
28
+ "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
29
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
30
+ "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
31
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
32
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
33
+ "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
34
+ "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
35
+ "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
36
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
37
+ "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
38
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
39
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
40
+ "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
41
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
42
+ "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
43
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
44
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
45
+ "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
46
+ "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
47
+ "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
48
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
49
+ "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
50
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
51
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
52
+ "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
53
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
54
+ "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
55
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
56
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
57
+ "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
58
+ "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
59
+ "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
60
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
61
+ "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
62
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
63
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
64
+ "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
65
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
66
+ "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
67
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
68
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
69
+ "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
70
+ "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
71
+ "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
72
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
73
+ "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
74
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
75
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
76
+ "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
77
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
78
+ "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
79
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
80
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
81
+ "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
82
+ "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
83
+ "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
84
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
85
+ "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
86
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
87
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
88
+ "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
89
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
90
+ "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
91
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
92
+ "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
93
+ "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
94
+ "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
95
+ "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
96
+ "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
97
+ "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
98
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
99
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
100
+ "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
101
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
102
+ "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
103
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
104
+ "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
105
+ "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
106
+ "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
107
+ "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
108
+ "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
109
+ "model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
110
+ "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
111
+ "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
112
+ "model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
113
+ "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
114
+ "model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
115
+ "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
116
+ "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
117
+ "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
118
+ "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
119
+ "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
120
+ "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
121
+ "model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
122
+ "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
123
+ "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
124
+ "model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
125
+ "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
126
+ "model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
127
+ "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
128
+ "model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
129
+ "model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
130
+ "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
131
+ "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
132
+ "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
133
+ "model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
134
+ "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
135
+ "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
136
+ "model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
137
+ "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
138
+ "model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
139
+ "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
140
+ "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
141
+ "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
142
+ "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
143
+ "model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
144
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
145
+ "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
146
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
147
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
148
+ "model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
149
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
150
+ "model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
151
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
152
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
153
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
154
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
155
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
156
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
157
+ "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
158
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
159
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
160
+ "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
161
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
162
+ "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
163
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
164
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
165
+ "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
166
+ "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
167
+ "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
168
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
169
+ "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
170
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
171
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
172
+ "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
173
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
174
+ "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
175
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
176
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
177
+ "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
178
+ "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
179
+ "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
180
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
181
+ "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
182
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
183
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
184
+ "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
185
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
186
+ "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
187
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
188
+ "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
189
+ "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
190
+ "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
191
+ "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
192
+ "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
193
+ "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
194
+ "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
195
+ "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
196
+ "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
197
+ "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
198
+ "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
199
+ "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
200
+ "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
201
+ "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
202
+ "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
203
+ "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
204
+ "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
205
+ "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
206
+ "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
207
+ "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
208
+ "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
209
+ "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
210
+ "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
211
+ "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
212
+ "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
213
+ "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
214
+ "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
215
+ "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
216
+ "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
217
+ "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
218
+ "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
219
+ "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
220
+ "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
221
+ "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
222
+ "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
223
+ "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
224
+ "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
225
+ "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
226
+ "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
227
+ "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
228
+ "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
229
+ "model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
230
+ "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
231
+ "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
232
+ "model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
233
+ "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
234
+ "model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
235
+ "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
236
+ "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
237
+ "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
238
+ "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
239
+ "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
240
+ "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
241
+ "model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
242
+ "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
243
+ "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
244
+ "model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
245
+ "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
246
+ "model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
247
+ "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
248
+ "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
249
+ "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
250
+ "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
251
+ "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
252
+ "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
253
+ "model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
254
+ "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
255
+ "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
256
+ "model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
257
+ "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
258
+ "model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
259
+ "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
260
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
261
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
262
+ "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
263
+ "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
264
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
265
+ "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
266
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
267
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
268
+ "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
269
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
270
+ "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
271
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
272
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
273
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
274
+ "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
275
+ "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
276
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
277
+ "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
278
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
279
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
280
+ "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
281
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
282
+ "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
283
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
284
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
285
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
286
+ "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
287
+ "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
288
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
289
+ "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
290
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
291
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
292
+ "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
293
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
294
+ "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
295
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
296
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
297
+ "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
298
+ "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
299
+ "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
300
+ "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
301
+ "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
302
+ "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
303
+ "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
304
+ "model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
305
+ "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
306
+ "model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
307
+ "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
308
+ "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
309
+ "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
310
+ "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
311
+ "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
312
+ "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
313
+ "model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
314
+ "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
315
+ "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
316
+ "model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
317
+ "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
318
+ "model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
319
+ "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
320
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
321
+ "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
322
+ "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
323
+ "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
324
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
325
+ "model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
326
+ "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
327
+ "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
328
+ "model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
329
+ "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
330
+ "model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
331
+ "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
332
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
333
+ "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
334
+ "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
335
+ "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
336
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
337
+ "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
338
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
339
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
340
+ "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
341
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
342
+ "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
343
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
344
+ "model.norm.weight": "model-00003-of-00004.safetensors"
345
+ }
346
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|im_start|>",
4
+ "<|im_end|>",
5
+ "<|object_ref_start|>",
6
+ "<|object_ref_end|>",
7
+ "<|box_start|>",
8
+ "<|box_end|>",
9
+ "<|quad_start|>",
10
+ "<|quad_end|>",
11
+ "<|vision_start|>",
12
+ "<|vision_end|>",
13
+ "<|vision_pad|>",
14
+ "<|image_pad|>",
15
+ "<|video_pad|>"
16
+ ],
17
+ "bos_token": "<|endoftext|>",
18
+ "eos_token": {
19
+ "content": "<|endoftext|>",
20
+ "lstrip": false,
21
+ "normalized": false,
22
+ "rstrip": false,
23
+ "single_word": false
24
+ },
25
+ "pad_token": "<|endoftext|>"
26
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a8506e7111b80c6d8635951a02eab0f4e1a8e4e5772da83846579e97b16f61bf
3
+ size 7031673
tokenizer_config.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d384a9421a1194545013e2154c06818533869f19ddaa573b670b0c29eee94ceb
3
+ size 7330
train_results/all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 0.9984,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.3464691298894393,
5
+ "train_runtime": 1464.0997,
6
+ "train_samples_per_second": 6.83,
7
+ "train_steps_per_second": 0.053
8
+ }
train_results/train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 0.9984,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.3464691298894393,
5
+ "train_runtime": 1464.0997,
6
+ "train_samples_per_second": 6.83,
7
+ "train_steps_per_second": 0.053
8
+ }
train_results/trainer_state.json ADDED
@@ -0,0 +1,1836 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.9984,
5
+ "eval_steps": 500,
6
+ "global_step": 78,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0128,
13
+ "fix_coeffi": 0.9835841059684753,
14
+ "focal_item_coeffi": 0.5089281797409058,
15
+ "grad_norm": 1.4328081247358297,
16
+ "learning_rate": 9.995944990857847e-07,
17
+ "logits": -0.00044327322393655777,
18
+ "loss": 0.3523,
19
+ "policy_logits/chosen": 0.3756556212902069,
20
+ "policy_logits/rejected": 0.6191669702529907,
21
+ "policy_logps/chosen": -0.1075749397277832,
22
+ "policy_logps/rejected": -0.11915800720453262,
23
+ "policy_logratios": 0.011583071202039719,
24
+ "positive_sft_coeffi": 0.125,
25
+ "ref_logps/chosen": -0.10703779011964798,
26
+ "ref_logps/rejected": -0.11906413733959198,
27
+ "ref_logratios": 0.012026344425976276,
28
+ "rewards/accuracies": 0.25,
29
+ "rewards/chosen": -0.0010743052698671818,
30
+ "rewards/margins": -0.0008865464478731155,
31
+ "rewards/rejected": -0.00018775882199406624,
32
+ "step": 1
33
+ },
34
+ {
35
+ "epoch": 0.0256,
36
+ "fix_coeffi": 0.986102283000946,
37
+ "focal_item_coeffi": 0.5028802156448364,
38
+ "grad_norm": 1.2467623214489967,
39
+ "learning_rate": 9.98378654067105e-07,
40
+ "logits": 0.013724885880947113,
41
+ "loss": 0.3472,
42
+ "policy_logits/chosen": -0.24451640248298645,
43
+ "policy_logits/rejected": 0.3242242634296417,
44
+ "policy_logps/chosen": -0.0901331827044487,
45
+ "policy_logps/rejected": -0.11135070770978928,
46
+ "policy_logratios": 0.021217525005340576,
47
+ "positive_sft_coeffi": 0.012500000186264515,
48
+ "ref_logps/chosen": -0.09265745431184769,
49
+ "ref_logps/rejected": -0.10015010833740234,
50
+ "ref_logratios": 0.007492639124393463,
51
+ "rewards/accuracies": 0.625,
52
+ "rewards/chosen": 0.005048542283475399,
53
+ "rewards/margins": 0.027449771761894226,
54
+ "rewards/rejected": -0.022401228547096252,
55
+ "step": 2
56
+ },
57
+ {
58
+ "epoch": 0.0384,
59
+ "fix_coeffi": 0.9856858849525452,
60
+ "focal_item_coeffi": 0.499173104763031,
61
+ "grad_norm": 1.8077456554356028,
62
+ "learning_rate": 9.963544370490268e-07,
63
+ "logits": 0.00021568406373262405,
64
+ "loss": 0.3485,
65
+ "policy_logits/chosen": -0.6639317870140076,
66
+ "policy_logits/rejected": 0.31396517157554626,
67
+ "policy_logps/chosen": -0.08218777179718018,
68
+ "policy_logps/rejected": -0.10962922871112823,
69
+ "policy_logratios": 0.027441462501883507,
70
+ "positive_sft_coeffi": 0.08750000596046448,
71
+ "ref_logps/chosen": -0.08004841208457947,
72
+ "ref_logps/rejected": -0.10727418959140778,
73
+ "ref_logratios": 0.027225779369473457,
74
+ "rewards/accuracies": 0.5625,
75
+ "rewards/chosen": -0.004278717562556267,
76
+ "rewards/margins": 0.0004313681274652481,
77
+ "rewards/rejected": -0.004710085690021515,
78
+ "step": 3
79
+ },
80
+ {
81
+ "epoch": 0.0512,
82
+ "fix_coeffi": 0.9775933623313904,
83
+ "focal_item_coeffi": 0.5193935632705688,
84
+ "grad_norm": 0.8840649467251522,
85
+ "learning_rate": 9.935251313189563e-07,
86
+ "logits": -0.0006601079367101192,
87
+ "loss": 0.349,
88
+ "policy_logits/chosen": -0.6437878012657166,
89
+ "policy_logits/rejected": -0.7638754844665527,
90
+ "policy_logps/chosen": -0.09489614516496658,
91
+ "policy_logps/rejected": -0.0894932746887207,
92
+ "policy_logratios": -0.005402863956987858,
93
+ "positive_sft_coeffi": 0.0625,
94
+ "ref_logps/chosen": -0.09494853764772415,
95
+ "ref_logps/rejected": -0.09020577371120453,
96
+ "ref_logratios": -0.004742756951600313,
97
+ "rewards/accuracies": 0.4375,
98
+ "rewards/chosen": 0.00010479474440217018,
99
+ "rewards/margins": -0.0013202158734202385,
100
+ "rewards/rejected": 0.0014250106178224087,
101
+ "step": 4
102
+ },
103
+ {
104
+ "epoch": 0.064,
105
+ "fix_coeffi": 0.9787843823432922,
106
+ "focal_item_coeffi": 0.5132390260696411,
107
+ "grad_norm": 0.6750680831319114,
108
+ "learning_rate": 9.898953260211337e-07,
109
+ "logits": 0.007043382618576288,
110
+ "loss": 0.3474,
111
+ "policy_logits/chosen": -0.049934305250644684,
112
+ "policy_logits/rejected": 0.12815727293491364,
113
+ "policy_logps/chosen": -0.1009887307882309,
114
+ "policy_logps/rejected": -0.10561253875494003,
115
+ "policy_logratios": 0.0046238102950155735,
116
+ "positive_sft_coeffi": 0.03750000149011612,
117
+ "ref_logps/chosen": -0.10768668353557587,
118
+ "ref_logps/rejected": -0.10526710748672485,
119
+ "ref_logratios": -0.002419572789222002,
120
+ "rewards/accuracies": 0.6875,
121
+ "rewards/chosen": 0.013395914807915688,
122
+ "rewards/margins": 0.014086765237152576,
123
+ "rewards/rejected": -0.0006908504292368889,
124
+ "step": 5
125
+ },
126
+ {
127
+ "epoch": 0.0768,
128
+ "fix_coeffi": 0.9762604236602783,
129
+ "focal_item_coeffi": 0.5024773478507996,
130
+ "grad_norm": 5.807173565939531,
131
+ "learning_rate": 9.85470908713026e-07,
132
+ "logits": 0.009615638293325901,
133
+ "loss": 0.3673,
134
+ "policy_logits/chosen": 2.769953489303589,
135
+ "policy_logits/rejected": 3.4124929904937744,
136
+ "policy_logps/chosen": -0.1529441624879837,
137
+ "policy_logps/rejected": -0.17491097748279572,
138
+ "policy_logratios": 0.02196682058274746,
139
+ "positive_sft_coeffi": 0.17500001192092896,
140
+ "ref_logps/chosen": -0.0930604487657547,
141
+ "ref_logps/rejected": -0.10541163384914398,
142
+ "ref_logratios": 0.012351179495453835,
143
+ "rewards/accuracies": 0.5,
144
+ "rewards/chosen": -0.11976741254329681,
145
+ "rewards/margins": 0.019231276586651802,
146
+ "rewards/rejected": -0.13899868726730347,
147
+ "step": 6
148
+ },
149
+ {
150
+ "epoch": 0.0896,
151
+ "fix_coeffi": 0.9845889806747437,
152
+ "focal_item_coeffi": 0.5101208686828613,
153
+ "grad_norm": 2.3245065529804956,
154
+ "learning_rate": 9.802590558156861e-07,
155
+ "logits": 0.0036336404737085104,
156
+ "loss": 0.345,
157
+ "policy_logits/chosen": -0.17252862453460693,
158
+ "policy_logits/rejected": 0.6632210612297058,
159
+ "policy_logps/chosen": -0.10324347764253616,
160
+ "policy_logps/rejected": -0.11281944811344147,
161
+ "policy_logratios": 0.009575962089002132,
162
+ "positive_sft_coeffi": 0.012500000186264515,
163
+ "ref_logps/chosen": -0.10653240978717804,
164
+ "ref_logps/rejected": -0.112474724650383,
165
+ "ref_logratios": 0.005942321382462978,
166
+ "rewards/accuracies": 0.75,
167
+ "rewards/chosen": 0.006577867548912764,
168
+ "rewards/margins": 0.007267280016094446,
169
+ "rewards/rejected": -0.0006894129328429699,
170
+ "step": 7
171
+ },
172
+ {
173
+ "epoch": 0.1024,
174
+ "fix_coeffi": 0.985831618309021,
175
+ "focal_item_coeffi": 0.47545546293258667,
176
+ "grad_norm": 4.372192168844234,
177
+ "learning_rate": 9.742682209735727e-07,
178
+ "logits": 0.037635721266269684,
179
+ "loss": 0.3507,
180
+ "policy_logits/chosen": 2.0098564624786377,
181
+ "policy_logits/rejected": 1.936082124710083,
182
+ "policy_logps/chosen": -0.12014156579971313,
183
+ "policy_logps/rejected": -0.18671229481697083,
184
+ "policy_logratios": 0.06657072901725769,
185
+ "positive_sft_coeffi": 0.15000000596046448,
186
+ "ref_logps/chosen": -0.10108990222215652,
187
+ "ref_logps/rejected": -0.13002490997314453,
188
+ "ref_logratios": 0.028935005888342857,
189
+ "rewards/accuracies": 0.625,
190
+ "rewards/chosen": -0.03810329735279083,
191
+ "rewards/margins": 0.07527144253253937,
192
+ "rewards/rejected": -0.1133747398853302,
193
+ "step": 8
194
+ },
195
+ {
196
+ "epoch": 0.1152,
197
+ "fix_coeffi": 0.977953314781189,
198
+ "focal_item_coeffi": 0.4957093596458435,
199
+ "grad_norm": 2.8612971673498526,
200
+ "learning_rate": 9.675081213427074e-07,
201
+ "logits": 0.013627536594867706,
202
+ "loss": 0.3586,
203
+ "policy_logits/chosen": 1.1523056030273438,
204
+ "policy_logits/rejected": 1.831128716468811,
205
+ "policy_logps/chosen": -0.13890302181243896,
206
+ "policy_logps/rejected": -0.172043576836586,
207
+ "policy_logratios": 0.033140555024147034,
208
+ "positive_sft_coeffi": 0.1875000149011612,
209
+ "ref_logps/chosen": -0.10710667073726654,
210
+ "ref_logps/rejected": -0.12661968171596527,
211
+ "ref_logratios": 0.019513018429279327,
212
+ "rewards/accuracies": 0.625,
213
+ "rewards/chosen": -0.06359270960092545,
214
+ "rewards/margins": 0.027255073189735413,
215
+ "rewards/rejected": -0.09084778279066086,
216
+ "step": 9
217
+ },
218
+ {
219
+ "epoch": 0.128,
220
+ "fix_coeffi": 0.984481930732727,
221
+ "focal_item_coeffi": 0.5143294334411621,
222
+ "grad_norm": 1.4193573115178666,
223
+ "learning_rate": 9.599897218294121e-07,
224
+ "logits": 0.0020799911580979824,
225
+ "loss": 0.357,
226
+ "policy_logits/chosen": 0.8174508213996887,
227
+ "policy_logits/rejected": 1.350592017173767,
228
+ "policy_logps/chosen": -0.11057809740304947,
229
+ "policy_logps/rejected": -0.11332608759403229,
230
+ "policy_logratios": 0.002747981809079647,
231
+ "positive_sft_coeffi": 0.11250000447034836,
232
+ "ref_logps/chosen": -0.10403745621442795,
233
+ "ref_logps/rejected": -0.10470543801784515,
234
+ "ref_logratios": 0.0006679906509816647,
235
+ "rewards/accuracies": 0.625,
236
+ "rewards/chosen": -0.013081288896501064,
237
+ "rewards/margins": 0.004159982316195965,
238
+ "rewards/rejected": -0.01724127121269703,
239
+ "step": 10
240
+ },
241
+ {
242
+ "epoch": 0.1408,
243
+ "fix_coeffi": 0.9826814532279968,
244
+ "focal_item_coeffi": 0.5101717710494995,
245
+ "grad_norm": 1.8035610967185876,
246
+ "learning_rate": 9.517252173051911e-07,
247
+ "logits": 0.00645341444760561,
248
+ "loss": 0.3476,
249
+ "policy_logits/chosen": -0.3606153130531311,
250
+ "policy_logits/rejected": -0.07814718782901764,
251
+ "policy_logps/chosen": -0.10171571373939514,
252
+ "policy_logps/rejected": -0.11131560057401657,
253
+ "policy_logratios": 0.009599885903298855,
254
+ "positive_sft_coeffi": 0.11250000447034836,
255
+ "ref_logps/chosen": -0.09868644177913666,
256
+ "ref_logps/rejected": -0.10183291137218475,
257
+ "ref_logratios": 0.003146471455693245,
258
+ "rewards/accuracies": 0.625,
259
+ "rewards/chosen": -0.006058532744646072,
260
+ "rewards/margins": 0.01290682889521122,
261
+ "rewards/rejected": -0.018965359777212143,
262
+ "step": 11
263
+ },
264
+ {
265
+ "epoch": 0.1536,
266
+ "fix_coeffi": 0.9853373169898987,
267
+ "focal_item_coeffi": 0.498578816652298,
268
+ "grad_norm": 1.4655337732955462,
269
+ "learning_rate": 9.427280128266049e-07,
270
+ "logits": 0.006295318715274334,
271
+ "loss": 0.3562,
272
+ "policy_logits/chosen": 1.193603754043579,
273
+ "policy_logits/rejected": 1.666297197341919,
274
+ "policy_logps/chosen": -0.12213902175426483,
275
+ "policy_logps/rejected": -0.1502697467803955,
276
+ "policy_logratios": 0.02813071198761463,
277
+ "positive_sft_coeffi": 0.1250000149011612,
278
+ "ref_logps/chosen": -0.10439996421337128,
279
+ "ref_logps/rejected": -0.12623536586761475,
280
+ "ref_logratios": 0.021835394203662872,
281
+ "rewards/accuracies": 0.625,
282
+ "rewards/chosen": -0.03547811880707741,
283
+ "rewards/margins": 0.012590636499226093,
284
+ "rewards/rejected": -0.048068758100271225,
285
+ "step": 12
286
+ },
287
+ {
288
+ "epoch": 0.1664,
289
+ "fix_coeffi": 0.9800107479095459,
290
+ "focal_item_coeffi": 0.5178880095481873,
291
+ "grad_norm": 1.0648419090854426,
292
+ "learning_rate": 9.330127018922193e-07,
293
+ "logits": 0.001285820733755827,
294
+ "loss": 0.3493,
295
+ "policy_logits/chosen": 0.5962052941322327,
296
+ "policy_logits/rejected": 0.804395318031311,
297
+ "policy_logps/chosen": -0.11390722543001175,
298
+ "policy_logps/rejected": -0.11087796092033386,
299
+ "policy_logratios": -0.0030292561277747154,
300
+ "positive_sft_coeffi": 0.13750000298023224,
301
+ "ref_logps/chosen": -0.09365612268447876,
302
+ "ref_logps/rejected": -0.08934105187654495,
303
+ "ref_logratios": -0.004315076861530542,
304
+ "rewards/accuracies": 0.4375,
305
+ "rewards/chosen": -0.04050220549106598,
306
+ "rewards/margins": 0.002571641467511654,
307
+ "rewards/rejected": -0.04307384416460991,
308
+ "step": 13
309
+ },
310
+ {
311
+ "epoch": 0.1792,
312
+ "fix_coeffi": 0.9837765097618103,
313
+ "focal_item_coeffi": 0.514049232006073,
314
+ "grad_norm": 1.822873650046738,
315
+ "learning_rate": 9.225950427718974e-07,
316
+ "logits": -0.013207457959651947,
317
+ "loss": 0.3595,
318
+ "policy_logits/chosen": -0.45941129326820374,
319
+ "policy_logits/rejected": 0.15231606364250183,
320
+ "policy_logps/chosen": -0.1455477923154831,
321
+ "policy_logps/rejected": -0.1451178789138794,
322
+ "policy_logratios": -0.0004299040883779526,
323
+ "positive_sft_coeffi": 0.15000000596046448,
324
+ "ref_logps/chosen": -0.10130025446414948,
325
+ "ref_logps/rejected": -0.11407780647277832,
326
+ "ref_logratios": 0.012777555733919144,
327
+ "rewards/accuracies": 0.625,
328
+ "rewards/chosen": -0.08849507570266724,
329
+ "rewards/margins": -0.026414915919303894,
330
+ "rewards/rejected": -0.062080152332782745,
331
+ "step": 14
332
+ },
333
+ {
334
+ "epoch": 0.192,
335
+ "fix_coeffi": 0.9864974021911621,
336
+ "focal_item_coeffi": 0.49726036190986633,
337
+ "grad_norm": 1.7917662537278625,
338
+ "learning_rate": 9.114919329468282e-07,
339
+ "logits": 0.014411830343306065,
340
+ "loss": 0.3529,
341
+ "policy_logits/chosen": 0.5780079364776611,
342
+ "policy_logits/rejected": 1.1044719219207764,
343
+ "policy_logps/chosen": -0.10612262040376663,
344
+ "policy_logps/rejected": -0.1366795152425766,
345
+ "policy_logratios": 0.030556898564100266,
346
+ "positive_sft_coeffi": 0.17500001192092896,
347
+ "ref_logps/chosen": -0.0779130756855011,
348
+ "ref_logps/rejected": -0.09405814856290817,
349
+ "ref_logratios": 0.016145067289471626,
350
+ "rewards/accuracies": 0.5625,
351
+ "rewards/chosen": -0.056419096887111664,
352
+ "rewards/margins": 0.02882366068661213,
353
+ "rewards/rejected": -0.08524274826049805,
354
+ "step": 15
355
+ },
356
+ {
357
+ "epoch": 0.2048,
358
+ "fix_coeffi": 0.9797394871711731,
359
+ "focal_item_coeffi": 0.5063139200210571,
360
+ "grad_norm": 1.2569781846546355,
361
+ "learning_rate": 8.997213817017506e-07,
362
+ "logits": 0.005497605539858341,
363
+ "loss": 0.3501,
364
+ "policy_logits/chosen": -1.6107919216156006,
365
+ "policy_logits/rejected": -0.9289680123329163,
366
+ "policy_logps/chosen": -0.08765186369419098,
367
+ "policy_logps/rejected": -0.1035788506269455,
368
+ "policy_logratios": 0.01592697948217392,
369
+ "positive_sft_coeffi": 0.07500000298023224,
370
+ "ref_logps/chosen": -0.08843531459569931,
371
+ "ref_logps/rejected": -0.09886468201875687,
372
+ "ref_logratios": 0.010429373942315578,
373
+ "rewards/accuracies": 0.75,
374
+ "rewards/chosen": 0.0015668817795813084,
375
+ "rewards/margins": 0.010995211079716682,
376
+ "rewards/rejected": -0.009428328834474087,
377
+ "step": 16
378
+ },
379
+ {
380
+ "epoch": 0.2176,
381
+ "fix_coeffi": 0.9732264876365662,
382
+ "focal_item_coeffi": 0.4989505410194397,
383
+ "grad_norm": 2.0397423178261036,
384
+ "learning_rate": 8.873024809138272e-07,
385
+ "logits": 0.018774695694446564,
386
+ "loss": 0.3522,
387
+ "policy_logits/chosen": -1.351194143295288,
388
+ "policy_logits/rejected": -0.7183037996292114,
389
+ "policy_logps/chosen": -0.09203340113162994,
390
+ "policy_logps/rejected": -0.11978676915168762,
391
+ "policy_logratios": 0.02775336243212223,
392
+ "positive_sft_coeffi": 0.07500000298023224,
393
+ "ref_logps/chosen": -0.08847107738256454,
394
+ "ref_logps/rejected": -0.09744974225759506,
395
+ "ref_logratios": 0.008978668600320816,
396
+ "rewards/accuracies": 0.875,
397
+ "rewards/chosen": -0.007124642841517925,
398
+ "rewards/margins": 0.03754939138889313,
399
+ "rewards/rejected": -0.04467403516173363,
400
+ "step": 17
401
+ },
402
+ {
403
+ "epoch": 0.2304,
404
+ "fix_coeffi": 0.9812570810317993,
405
+ "focal_item_coeffi": 0.5031195282936096,
406
+ "grad_norm": 1.8548290134218681,
407
+ "learning_rate": 8.742553740855505e-07,
408
+ "logits": 0.00520216254517436,
409
+ "loss": 0.3571,
410
+ "policy_logits/chosen": 0.3951745331287384,
411
+ "policy_logits/rejected": 1.1887165307998657,
412
+ "policy_logps/chosen": -0.10930471122264862,
413
+ "policy_logps/rejected": -0.13044798374176025,
414
+ "policy_logratios": 0.021143266931176186,
415
+ "positive_sft_coeffi": 0.13750000298023224,
416
+ "ref_logps/chosen": -0.08903292566537857,
417
+ "ref_logps/rejected": -0.10497403144836426,
418
+ "ref_logratios": 0.015941105782985687,
419
+ "rewards/accuracies": 0.5,
420
+ "rewards/chosen": -0.040543586015701294,
421
+ "rewards/margins": 0.01040432509034872,
422
+ "rewards/rejected": -0.05094791203737259,
423
+ "step": 18
424
+ },
425
+ {
426
+ "epoch": 0.2432,
427
+ "fix_coeffi": 0.9801645874977112,
428
+ "focal_item_coeffi": 0.4744604229927063,
429
+ "grad_norm": 2.9472468604548356,
430
+ "learning_rate": 8.606012236719073e-07,
431
+ "logits": 0.09759598970413208,
432
+ "loss": 0.3865,
433
+ "policy_logits/chosen": -0.25552576780319214,
434
+ "policy_logits/rejected": 0.3762688934803009,
435
+ "policy_logps/chosen": -0.13322323560714722,
436
+ "policy_logps/rejected": -0.2592465281486511,
437
+ "policy_logratios": 0.1260233074426651,
438
+ "positive_sft_coeffi": 0.16250000894069672,
439
+ "ref_logps/chosen": -0.09152866154909134,
440
+ "ref_logps/rejected": -0.11995595693588257,
441
+ "ref_logratios": 0.028427300974726677,
442
+ "rewards/accuracies": 0.6875,
443
+ "rewards/chosen": -0.08338914811611176,
444
+ "rewards/margins": 0.19519197940826416,
445
+ "rewards/rejected": -0.2785811722278595,
446
+ "step": 19
447
+ },
448
+ {
449
+ "epoch": 0.256,
450
+ "fix_coeffi": 0.9848646521568298,
451
+ "focal_item_coeffi": 0.49854862689971924,
452
+ "grad_norm": 9.693652692781011,
453
+ "learning_rate": 8.463621767547997e-07,
454
+ "logits": 0.014589127153158188,
455
+ "loss": 0.3586,
456
+ "policy_logits/chosen": 0.06237606704235077,
457
+ "policy_logits/rejected": 0.7283445000648499,
458
+ "policy_logps/chosen": -0.11030930280685425,
459
+ "policy_logps/rejected": -0.13866950571537018,
460
+ "policy_logratios": 0.02836019918322563,
461
+ "positive_sft_coeffi": 0.13750000298023224,
462
+ "ref_logps/chosen": -0.0953645408153534,
463
+ "ref_logps/rejected": -0.10913561284542084,
464
+ "ref_logratios": 0.013771074824035168,
465
+ "rewards/accuracies": 0.75,
466
+ "rewards/chosen": -0.02988952025771141,
467
+ "rewards/margins": 0.029178254306316376,
468
+ "rewards/rejected": -0.059067778289318085,
469
+ "step": 20
470
+ },
471
+ {
472
+ "epoch": 0.2688,
473
+ "fix_coeffi": 0.9794363379478455,
474
+ "focal_item_coeffi": 0.5036914348602295,
475
+ "grad_norm": 1.197765449030972,
476
+ "learning_rate": 8.315613291203976e-07,
477
+ "logits": 0.011058323085308075,
478
+ "loss": 0.3436,
479
+ "policy_logits/chosen": -0.9592422246932983,
480
+ "policy_logits/rejected": -0.4262007772922516,
481
+ "policy_logps/chosen": -0.09527543932199478,
482
+ "policy_logps/rejected": -0.11527076363563538,
483
+ "policy_logratios": 0.0199953131377697,
484
+ "positive_sft_coeffi": 0.13750000298023224,
485
+ "ref_logps/chosen": -0.08988872915506363,
486
+ "ref_logps/rejected": -0.09882570803165436,
487
+ "ref_logratios": 0.008936991915106773,
488
+ "rewards/accuracies": 0.75,
489
+ "rewards/chosen": -0.01077345572412014,
490
+ "rewards/margins": 0.02211664617061615,
491
+ "rewards/rejected": -0.03289010375738144,
492
+ "step": 21
493
+ },
494
+ {
495
+ "epoch": 0.2816,
496
+ "fix_coeffi": 0.9824039936065674,
497
+ "focal_item_coeffi": 0.4995446801185608,
498
+ "grad_norm": 2.2819727130349396,
499
+ "learning_rate": 8.162226877976886e-07,
500
+ "logits": 0.013251272030174732,
501
+ "loss": 0.3547,
502
+ "policy_logits/chosen": -0.08443702757358551,
503
+ "policy_logits/rejected": 0.8897062540054321,
504
+ "policy_logps/chosen": -0.10860311985015869,
505
+ "policy_logps/rejected": -0.13532036542892456,
506
+ "policy_logratios": 0.026717234402894974,
507
+ "positive_sft_coeffi": 0.08750000596046448,
508
+ "ref_logps/chosen": -0.09806712716817856,
509
+ "ref_logps/rejected": -0.11153308302164078,
510
+ "ref_logratios": 0.013465962372720242,
511
+ "rewards/accuracies": 0.875,
512
+ "rewards/chosen": -0.02107200399041176,
513
+ "rewards/margins": 0.026502544060349464,
514
+ "rewards/rejected": -0.047574546188116074,
515
+ "step": 22
516
+ },
517
+ {
518
+ "epoch": 0.2944,
519
+ "fix_coeffi": 0.981361985206604,
520
+ "focal_item_coeffi": 0.4879322052001953,
521
+ "grad_norm": 2.6270223473212098,
522
+ "learning_rate": 8.003711321189895e-07,
523
+ "logits": 0.031471338123083115,
524
+ "loss": 0.3452,
525
+ "policy_logits/chosen": -0.3838559687137604,
526
+ "policy_logits/rejected": 0.0620497427880764,
527
+ "policy_logps/chosen": -0.11237306147813797,
528
+ "policy_logps/rejected": -0.15777207911014557,
529
+ "policy_logratios": 0.04539903998374939,
530
+ "positive_sft_coeffi": 0.125,
531
+ "ref_logps/chosen": -0.10156327486038208,
532
+ "ref_logps/rejected": -0.11549098044633865,
533
+ "ref_logratios": 0.013927703723311424,
534
+ "rewards/accuracies": 0.8125,
535
+ "rewards/chosen": -0.021619543433189392,
536
+ "rewards/margins": 0.06294267624616623,
537
+ "rewards/rejected": -0.08456221967935562,
538
+ "step": 23
539
+ },
540
+ {
541
+ "epoch": 0.3072,
542
+ "fix_coeffi": 0.976114809513092,
543
+ "focal_item_coeffi": 0.483175128698349,
544
+ "grad_norm": 1.1880384252088747,
545
+ "learning_rate": 7.840323733655778e-07,
546
+ "logits": 0.03420798480510712,
547
+ "loss": 0.35,
548
+ "policy_logits/chosen": 0.22115886211395264,
549
+ "policy_logits/rejected": 0.989837646484375,
550
+ "policy_logps/chosen": -0.12655988335609436,
551
+ "policy_logps/rejected": -0.18053291738033295,
552
+ "policy_logratios": 0.05397304520010948,
553
+ "positive_sft_coeffi": 0.17500001192092896,
554
+ "ref_logps/chosen": -0.10451289266347885,
555
+ "ref_logps/rejected": -0.12427795678377151,
556
+ "ref_logratios": 0.019765062257647514,
557
+ "rewards/accuracies": 0.8125,
558
+ "rewards/chosen": -0.04409395903348923,
559
+ "rewards/margins": 0.06841596961021423,
560
+ "rewards/rejected": -0.11250994354486465,
561
+ "step": 24
562
+ },
563
+ {
564
+ "epoch": 0.32,
565
+ "fix_coeffi": 0.981358528137207,
566
+ "focal_item_coeffi": 0.49852997064590454,
567
+ "grad_norm": 8.947243621007857,
568
+ "learning_rate": 7.672329130639005e-07,
569
+ "logits": 0.007318189833313227,
570
+ "loss": 0.3567,
571
+ "policy_logits/chosen": -0.24950921535491943,
572
+ "policy_logits/rejected": 0.8151937127113342,
573
+ "policy_logps/chosen": -0.13857412338256836,
574
+ "policy_logps/rejected": -0.16683626174926758,
575
+ "policy_logratios": 0.028262142091989517,
576
+ "positive_sft_coeffi": 0.15000000596046448,
577
+ "ref_logps/chosen": -0.11754828691482544,
578
+ "ref_logps/rejected": -0.13849222660064697,
579
+ "ref_logratios": 0.020943952724337578,
580
+ "rewards/accuracies": 0.5625,
581
+ "rewards/chosen": -0.04205168038606644,
582
+ "rewards/margins": 0.014636380597949028,
583
+ "rewards/rejected": -0.056688059121370316,
584
+ "step": 25
585
+ },
586
+ {
587
+ "epoch": 0.3328,
588
+ "fix_coeffi": 0.9824901819229126,
589
+ "focal_item_coeffi": 0.494914710521698,
590
+ "grad_norm": 1.716842312272706,
591
+ "learning_rate": 7.5e-07,
592
+ "logits": 0.01641026884317398,
593
+ "loss": 0.3487,
594
+ "policy_logits/chosen": -0.6850305795669556,
595
+ "policy_logits/rejected": -0.12782959640026093,
596
+ "policy_logps/chosen": -0.10488370060920715,
597
+ "policy_logps/rejected": -0.13926003873348236,
598
+ "policy_logratios": 0.0343763493001461,
599
+ "positive_sft_coeffi": 0.125,
600
+ "ref_logps/chosen": -0.09121353179216385,
601
+ "ref_logps/rejected": -0.10917960852384567,
602
+ "ref_logratios": 0.017966080456972122,
603
+ "rewards/accuracies": 0.75,
604
+ "rewards/chosen": -0.02734033204615116,
605
+ "rewards/margins": 0.03282053768634796,
606
+ "rewards/rejected": -0.06016086786985397,
607
+ "step": 26
608
+ },
609
+ {
610
+ "epoch": 0.3456,
611
+ "fix_coeffi": 0.9830281734466553,
612
+ "focal_item_coeffi": 0.468192994594574,
613
+ "grad_norm": 13.202327835208298,
614
+ "learning_rate": 7.323615860218842e-07,
615
+ "logits": 0.07923495769500732,
616
+ "loss": 0.3519,
617
+ "policy_logits/chosen": -0.770702600479126,
618
+ "policy_logits/rejected": -0.05294930189847946,
619
+ "policy_logps/chosen": -0.1078248918056488,
620
+ "policy_logps/rejected": -0.2335628867149353,
621
+ "policy_logratios": 0.1257379949092865,
622
+ "positive_sft_coeffi": 0.16250000894069672,
623
+ "ref_logps/chosen": -0.0905764102935791,
624
+ "ref_logps/rejected": -0.13707944750785828,
625
+ "ref_logratios": 0.046503033488988876,
626
+ "rewards/accuracies": 0.75,
627
+ "rewards/chosen": -0.0344969779253006,
628
+ "rewards/margins": 0.15846990048885345,
629
+ "rewards/rejected": -0.19296687841415405,
630
+ "step": 27
631
+ },
632
+ {
633
+ "epoch": 0.3584,
634
+ "fix_coeffi": 0.9840790033340454,
635
+ "focal_item_coeffi": 0.5058746337890625,
636
+ "grad_norm": 7.87857468619277,
637
+ "learning_rate": 7.14346280701527e-07,
638
+ "logits": 0.0038224519230425358,
639
+ "loss": 0.347,
640
+ "policy_logits/chosen": -0.5358918309211731,
641
+ "policy_logits/rejected": -0.0709114521741867,
642
+ "policy_logps/chosen": -0.12395370006561279,
643
+ "policy_logps/rejected": -0.1404440850019455,
644
+ "policy_logratios": 0.016490377485752106,
645
+ "positive_sft_coeffi": 0.15000000596046448,
646
+ "ref_logps/chosen": -0.10481671988964081,
647
+ "ref_logps/rejected": -0.11748463660478592,
648
+ "ref_logratios": 0.012667926028370857,
649
+ "rewards/accuracies": 0.5,
650
+ "rewards/chosen": -0.03827396780252457,
651
+ "rewards/margins": 0.0076449038460850716,
652
+ "rewards/rejected": -0.04591887444257736,
653
+ "step": 28
654
+ },
655
+ {
656
+ "epoch": 0.3712,
657
+ "fix_coeffi": 0.9845030903816223,
658
+ "focal_item_coeffi": 0.508884847164154,
659
+ "grad_norm": 3.2541019955957102,
660
+ "learning_rate": 6.959833049300375e-07,
661
+ "logits": -0.012878144159913063,
662
+ "loss": 0.3543,
663
+ "policy_logits/chosen": -1.2571114301681519,
664
+ "policy_logits/rejected": -0.3266504406929016,
665
+ "policy_logps/chosen": -0.1532968431711197,
666
+ "policy_logps/rejected": -0.14601215720176697,
667
+ "policy_logratios": -0.007284687831997871,
668
+ "positive_sft_coeffi": 0.11250000447034836,
669
+ "ref_logps/chosen": -0.11947188526391983,
670
+ "ref_logps/rejected": -0.12506534159183502,
671
+ "ref_logratios": 0.005593459587544203,
672
+ "rewards/accuracies": 0.8125,
673
+ "rewards/chosen": -0.06764990091323853,
674
+ "rewards/margins": -0.025756295770406723,
675
+ "rewards/rejected": -0.041893601417541504,
676
+ "step": 29
677
+ },
678
+ {
679
+ "epoch": 0.384,
680
+ "fix_coeffi": 0.9828195571899414,
681
+ "focal_item_coeffi": 0.495877742767334,
682
+ "grad_norm": 2.4765885115069097,
683
+ "learning_rate": 6.773024435212677e-07,
684
+ "logits": 0.024792660027742386,
685
+ "loss": 0.3496,
686
+ "policy_logits/chosen": -1.0900813341140747,
687
+ "policy_logits/rejected": -0.7720285654067993,
688
+ "policy_logps/chosen": -0.09298589825630188,
689
+ "policy_logps/rejected": -0.1255740076303482,
690
+ "policy_logratios": 0.03258810192346573,
691
+ "positive_sft_coeffi": 0.13750000298023224,
692
+ "ref_logps/chosen": -0.08834590762853622,
693
+ "ref_logps/rejected": -0.09614134579896927,
694
+ "ref_logratios": 0.007795444689691067,
695
+ "rewards/accuracies": 0.9375,
696
+ "rewards/chosen": -0.009280010126531124,
697
+ "rewards/margins": 0.04958532005548477,
698
+ "rewards/rejected": -0.05886532738804817,
699
+ "step": 30
700
+ },
701
+ {
702
+ "epoch": 0.3968,
703
+ "fix_coeffi": 0.98390793800354,
704
+ "focal_item_coeffi": 0.49783796072006226,
705
+ "grad_norm": 7.830009160036184,
706
+ "learning_rate": 6.583339969007363e-07,
707
+ "logits": 0.015777666121721268,
708
+ "loss": 0.3411,
709
+ "policy_logits/chosen": -1.290731430053711,
710
+ "policy_logits/rejected": -0.9378005862236023,
711
+ "policy_logps/chosen": -0.10924562811851501,
712
+ "policy_logps/rejected": -0.13851290941238403,
713
+ "policy_logratios": 0.029267292469739914,
714
+ "positive_sft_coeffi": 0.1875000149011612,
715
+ "ref_logps/chosen": -0.08980346471071243,
716
+ "ref_logps/rejected": -0.10329308360815048,
717
+ "ref_logratios": 0.013489626348018646,
718
+ "rewards/accuracies": 0.6875,
719
+ "rewards/chosen": -0.038884300738573074,
720
+ "rewards/margins": 0.031555332243442535,
721
+ "rewards/rejected": -0.0704396441578865,
722
+ "step": 31
723
+ },
724
+ {
725
+ "epoch": 0.4096,
726
+ "fix_coeffi": 0.9768311977386475,
727
+ "focal_item_coeffi": 0.5076443552970886,
728
+ "grad_norm": 5.151117259551285,
729
+ "learning_rate": 6.391087319582263e-07,
730
+ "logits": 0.013762960210442543,
731
+ "loss": 0.3548,
732
+ "policy_logits/chosen": -0.7259964346885681,
733
+ "policy_logits/rejected": -0.39904287457466125,
734
+ "policy_logps/chosen": -0.1493164747953415,
735
+ "policy_logps/rejected": -0.16279977560043335,
736
+ "policy_logratios": 0.0134833212941885,
737
+ "positive_sft_coeffi": 0.16250000894069672,
738
+ "ref_logps/chosen": -0.11615272611379623,
739
+ "ref_logps/rejected": -0.11587309092283249,
740
+ "ref_logratios": -0.00027963845059275627,
741
+ "rewards/accuracies": 0.75,
742
+ "rewards/chosen": -0.06632747501134872,
743
+ "rewards/margins": 0.027525920420885086,
744
+ "rewards/rejected": -0.09385339170694351,
745
+ "step": 32
746
+ },
747
+ {
748
+ "epoch": 0.4224,
749
+ "fix_coeffi": 0.9803541898727417,
750
+ "focal_item_coeffi": 0.4924553632736206,
751
+ "grad_norm": 2.257869723143706,
752
+ "learning_rate": 6.196578321437789e-07,
753
+ "logits": 0.01572483591735363,
754
+ "loss": 0.3528,
755
+ "policy_logits/chosen": -0.8784639835357666,
756
+ "policy_logits/rejected": -0.534101128578186,
757
+ "policy_logps/chosen": -0.11288098245859146,
758
+ "policy_logps/rejected": -0.1508454978466034,
759
+ "policy_logratios": 0.03796451538801193,
760
+ "positive_sft_coeffi": 0.1625000238418579,
761
+ "ref_logps/chosen": -0.0914713591337204,
762
+ "ref_logps/rejected": -0.11371105909347534,
763
+ "ref_logratios": 0.022239675745368004,
764
+ "rewards/accuracies": 0.8125,
765
+ "rewards/chosen": -0.042819250375032425,
766
+ "rewards/margins": 0.03144967183470726,
767
+ "rewards/rejected": -0.07426892220973969,
768
+ "step": 33
769
+ },
770
+ {
771
+ "epoch": 0.4352,
772
+ "fix_coeffi": 0.9828526377677917,
773
+ "focal_item_coeffi": 0.4904533326625824,
774
+ "grad_norm": 4.581806909629723,
775
+ "learning_rate": 6.000128468880222e-07,
776
+ "logits": 0.02504175342619419,
777
+ "loss": 0.3413,
778
+ "policy_logits/chosen": -1.755483627319336,
779
+ "policy_logits/rejected": -0.7362502217292786,
780
+ "policy_logps/chosen": -0.10036768764257431,
781
+ "policy_logps/rejected": -0.14162124693393707,
782
+ "policy_logratios": 0.04125357046723366,
783
+ "positive_sft_coeffi": 0.125,
784
+ "ref_logps/chosen": -0.0899302214384079,
785
+ "ref_logps/rejected": -0.10614202916622162,
786
+ "ref_logratios": 0.016211815178394318,
787
+ "rewards/accuracies": 0.6875,
788
+ "rewards/chosen": -0.02087494358420372,
789
+ "rewards/margins": 0.05008350685238838,
790
+ "rewards/rejected": -0.0709584504365921,
791
+ "step": 34
792
+ },
793
+ {
794
+ "epoch": 0.448,
795
+ "fix_coeffi": 0.9819279313087463,
796
+ "focal_item_coeffi": 0.5074229836463928,
797
+ "grad_norm": 4.030681754161456,
798
+ "learning_rate": 5.802056404288801e-07,
799
+ "logits": 0.012121563777327538,
800
+ "loss": 0.3458,
801
+ "policy_logits/chosen": -1.5923280715942383,
802
+ "policy_logits/rejected": -1.0361722707748413,
803
+ "policy_logps/chosen": -0.11964356899261475,
804
+ "policy_logps/rejected": -0.13315045833587646,
805
+ "policy_logratios": 0.013506898656487465,
806
+ "positive_sft_coeffi": 0.125,
807
+ "ref_logps/chosen": -0.09653420746326447,
808
+ "ref_logps/rejected": -0.0979195386171341,
809
+ "ref_logratios": 0.0013853348791599274,
810
+ "rewards/accuracies": 0.625,
811
+ "rewards/chosen": -0.046218715608119965,
812
+ "rewards/margins": 0.024243127554655075,
813
+ "rewards/rejected": -0.07046184688806534,
814
+ "step": 35
815
+ },
816
+ {
817
+ "epoch": 0.4608,
818
+ "fix_coeffi": 0.982082188129425,
819
+ "focal_item_coeffi": 0.48990198969841003,
820
+ "grad_norm": 2.44420773756206,
821
+ "learning_rate": 5.602683401276614e-07,
822
+ "logits": 0.022583654150366783,
823
+ "loss": 0.3423,
824
+ "policy_logits/chosen": -1.4836114645004272,
825
+ "policy_logits/rejected": -0.9023423194885254,
826
+ "policy_logps/chosen": -0.12841860949993134,
827
+ "policy_logps/rejected": -0.1705304980278015,
828
+ "policy_logratios": 0.04211188107728958,
829
+ "positive_sft_coeffi": 0.15000002086162567,
830
+ "ref_logps/chosen": -0.10194506496191025,
831
+ "ref_logps/rejected": -0.1214732825756073,
832
+ "ref_logratios": 0.019528226926922798,
833
+ "rewards/accuracies": 0.6875,
834
+ "rewards/chosen": -0.05294708162546158,
835
+ "rewards/margins": 0.045167308300733566,
836
+ "rewards/rejected": -0.09811439365148544,
837
+ "step": 36
838
+ },
839
+ {
840
+ "epoch": 0.4736,
841
+ "fix_coeffi": 0.9839043617248535,
842
+ "focal_item_coeffi": 0.4923628270626068,
843
+ "grad_norm": 1.9552292129762348,
844
+ "learning_rate": 5.40233284358363e-07,
845
+ "logits": 0.028175394982099533,
846
+ "loss": 0.353,
847
+ "policy_logits/chosen": -0.9158037900924683,
848
+ "policy_logits/rejected": -0.43341314792633057,
849
+ "policy_logps/chosen": -0.10895893722772598,
850
+ "policy_logps/rejected": -0.1472579538822174,
851
+ "policy_logratios": 0.03829900547862053,
852
+ "positive_sft_coeffi": 0.15000000596046448,
853
+ "ref_logps/chosen": -0.08450745046138763,
854
+ "ref_logps/rejected": -0.09463104605674744,
855
+ "ref_logratios": 0.010123610496520996,
856
+ "rewards/accuracies": 0.9375,
857
+ "rewards/chosen": -0.04890299588441849,
858
+ "rewards/margins": 0.05635078623890877,
859
+ "rewards/rejected": -0.10525378584861755,
860
+ "step": 37
861
+ },
862
+ {
863
+ "epoch": 0.4864,
864
+ "fix_coeffi": 0.9782541394233704,
865
+ "focal_item_coeffi": 0.49288293719291687,
866
+ "grad_norm": 10.411782320207385,
867
+ "learning_rate": 5.201329700547076e-07,
868
+ "logits": 0.024269087240099907,
869
+ "loss": 0.3607,
870
+ "policy_logits/chosen": -0.5079262256622314,
871
+ "policy_logits/rejected": 0.3136002719402313,
872
+ "policy_logps/chosen": -0.1414806842803955,
873
+ "policy_logps/rejected": -0.17897126078605652,
874
+ "policy_logratios": 0.03749057278037071,
875
+ "positive_sft_coeffi": 0.16250000894069672,
876
+ "ref_logps/chosen": -0.10568848252296448,
877
+ "ref_logps/rejected": -0.11890997737646103,
878
+ "ref_logratios": 0.013221489265561104,
879
+ "rewards/accuracies": 0.75,
880
+ "rewards/chosen": -0.07158437371253967,
881
+ "rewards/margins": 0.048538174480199814,
882
+ "rewards/rejected": -0.12012255191802979,
883
+ "step": 38
884
+ },
885
+ {
886
+ "epoch": 0.4992,
887
+ "fix_coeffi": 0.9775339365005493,
888
+ "focal_item_coeffi": 0.49986129999160767,
889
+ "grad_norm": 2.96696055884824,
890
+ "learning_rate": 5e-07,
891
+ "logits": 0.019692786037921906,
892
+ "loss": 0.3459,
893
+ "policy_logits/chosen": -1.6474698781967163,
894
+ "policy_logits/rejected": -1.140698790550232,
895
+ "policy_logps/chosen": -0.10526902973651886,
896
+ "policy_logps/rejected": -0.1312849372625351,
897
+ "policy_logratios": 0.026015909388661385,
898
+ "positive_sft_coeffi": 0.1875000149011612,
899
+ "ref_logps/chosen": -0.07764188200235367,
900
+ "ref_logps/rejected": -0.083965003490448,
901
+ "ref_logratios": 0.006323125213384628,
902
+ "rewards/accuracies": 0.75,
903
+ "rewards/chosen": -0.055254317820072174,
904
+ "rewards/margins": 0.03938557207584381,
905
+ "rewards/rejected": -0.09463988244533539,
906
+ "step": 39
907
+ },
908
+ {
909
+ "epoch": 0.512,
910
+ "fix_coeffi": 0.985194981098175,
911
+ "focal_item_coeffi": 0.4898516535758972,
912
+ "grad_norm": 4.71337044212431,
913
+ "learning_rate": 4.798670299452926e-07,
914
+ "logits": 0.0194325502961874,
915
+ "loss": 0.3482,
916
+ "policy_logits/chosen": -1.2630760669708252,
917
+ "policy_logits/rejected": -0.14844384789466858,
918
+ "policy_logps/chosen": -0.11292049288749695,
919
+ "policy_logps/rejected": -0.1549755185842514,
920
+ "policy_logratios": 0.04205501452088356,
921
+ "positive_sft_coeffi": 0.16250000894069672,
922
+ "ref_logps/chosen": -0.08799314498901367,
923
+ "ref_logps/rejected": -0.11061559617519379,
924
+ "ref_logratios": 0.02262245863676071,
925
+ "rewards/accuracies": 0.6875,
926
+ "rewards/chosen": -0.04985470324754715,
927
+ "rewards/margins": 0.0388651005923748,
928
+ "rewards/rejected": -0.08871980756521225,
929
+ "step": 40
930
+ },
931
+ {
932
+ "epoch": 0.5248,
933
+ "fix_coeffi": 0.9816229343414307,
934
+ "focal_item_coeffi": 0.5054661631584167,
935
+ "grad_norm": 4.8891176866891435,
936
+ "learning_rate": 4.5976671564163703e-07,
937
+ "logits": 0.014489522203803062,
938
+ "loss": 0.3389,
939
+ "policy_logits/chosen": -1.7239389419555664,
940
+ "policy_logits/rejected": -1.1534802913665771,
941
+ "policy_logps/chosen": -0.127137690782547,
942
+ "policy_logps/rejected": -0.14405909180641174,
943
+ "policy_logratios": 0.016921401023864746,
944
+ "positive_sft_coeffi": 0.15000000596046448,
945
+ "ref_logps/chosen": -0.0958910584449768,
946
+ "ref_logps/rejected": -0.09832293540239334,
947
+ "ref_logratios": 0.002431879285722971,
948
+ "rewards/accuracies": 0.5,
949
+ "rewards/chosen": -0.062493257224559784,
950
+ "rewards/margins": 0.028979042544960976,
951
+ "rewards/rejected": -0.09147229790687561,
952
+ "step": 41
953
+ },
954
+ {
955
+ "epoch": 0.5376,
956
+ "fix_coeffi": 0.9873742461204529,
957
+ "focal_item_coeffi": 0.4957110285758972,
958
+ "grad_norm": 2.8981544380738735,
959
+ "learning_rate": 4.397316598723385e-07,
960
+ "logits": 0.016218703240156174,
961
+ "loss": 0.3414,
962
+ "policy_logits/chosen": -2.1547656059265137,
963
+ "policy_logits/rejected": -1.366182804107666,
964
+ "policy_logps/chosen": -0.11845016479492188,
965
+ "policy_logps/rejected": -0.14999765157699585,
966
+ "policy_logratios": 0.031547483056783676,
967
+ "positive_sft_coeffi": 0.20000001788139343,
968
+ "ref_logps/chosen": -0.07900828123092651,
969
+ "ref_logps/rejected": -0.09433706104755402,
970
+ "ref_logratios": 0.015328779816627502,
971
+ "rewards/accuracies": 0.8125,
972
+ "rewards/chosen": -0.07888374477624893,
973
+ "rewards/margins": 0.03243740648031235,
974
+ "rewards/rejected": -0.11132115870714188,
975
+ "step": 42
976
+ },
977
+ {
978
+ "epoch": 0.5504,
979
+ "fix_coeffi": 0.9815526604652405,
980
+ "focal_item_coeffi": 0.5066700577735901,
981
+ "grad_norm": 3.801456300252324,
982
+ "learning_rate": 4.197943595711198e-07,
983
+ "logits": 0.013502766378223896,
984
+ "loss": 0.3476,
985
+ "policy_logits/chosen": -1.5803419351577759,
986
+ "policy_logits/rejected": -1.0385318994522095,
987
+ "policy_logps/chosen": -0.137198805809021,
988
+ "policy_logps/rejected": -0.15185044705867767,
989
+ "policy_logratios": 0.014651626348495483,
990
+ "positive_sft_coeffi": 0.15000000596046448,
991
+ "ref_logps/chosen": -0.10526979714632034,
992
+ "ref_logps/rejected": -0.10641864687204361,
993
+ "ref_logratios": 0.0011488597374409437,
994
+ "rewards/accuracies": 0.6875,
995
+ "rewards/chosen": -0.0638580396771431,
996
+ "rewards/margins": 0.027005532756447792,
997
+ "rewards/rejected": -0.09086357057094574,
998
+ "step": 43
999
+ },
1000
+ {
1001
+ "epoch": 0.5632,
1002
+ "fix_coeffi": 0.989657998085022,
1003
+ "focal_item_coeffi": 0.5009726881980896,
1004
+ "grad_norm": 5.690155691593627,
1005
+ "learning_rate": 3.9998715311197783e-07,
1006
+ "logits": 0.024562392383813858,
1007
+ "loss": 0.3446,
1008
+ "policy_logits/chosen": -1.5592479705810547,
1009
+ "policy_logits/rejected": -0.6050398349761963,
1010
+ "policy_logps/chosen": -0.14265373349189758,
1011
+ "policy_logps/rejected": -0.16672557592391968,
1012
+ "policy_logratios": 0.024071838706731796,
1013
+ "positive_sft_coeffi": 0.15000000596046448,
1014
+ "ref_logps/chosen": -0.09081384539604187,
1015
+ "ref_logps/rejected": -0.09032328426837921,
1016
+ "ref_logratios": -0.0004905527457594872,
1017
+ "rewards/accuracies": 0.5,
1018
+ "rewards/chosen": -0.10367979109287262,
1019
+ "rewards/margins": 0.049124784767627716,
1020
+ "rewards/rejected": -0.15280458331108093,
1021
+ "step": 44
1022
+ },
1023
+ {
1024
+ "epoch": 0.576,
1025
+ "fix_coeffi": 0.9779914617538452,
1026
+ "focal_item_coeffi": 0.5143808722496033,
1027
+ "grad_norm": 2.479932367892572,
1028
+ "learning_rate": 3.8034216785622125e-07,
1029
+ "logits": 0.010076407343149185,
1030
+ "loss": 0.3523,
1031
+ "policy_logits/chosen": -1.3979623317718506,
1032
+ "policy_logits/rejected": -1.4125741720199585,
1033
+ "policy_logps/chosen": -0.17873534560203552,
1034
+ "policy_logps/rejected": -0.1802450716495514,
1035
+ "policy_logratios": 0.0015097390860319138,
1036
+ "positive_sft_coeffi": 0.16250000894069672,
1037
+ "ref_logps/chosen": -0.11275173723697662,
1038
+ "ref_logps/rejected": -0.1041850745677948,
1039
+ "ref_logratios": -0.008566668257117271,
1040
+ "rewards/accuracies": 0.6875,
1041
+ "rewards/chosen": -0.1319671869277954,
1042
+ "rewards/margins": 0.020152810961008072,
1043
+ "rewards/rejected": -0.15212000906467438,
1044
+ "step": 45
1045
+ },
1046
+ {
1047
+ "epoch": 0.5888,
1048
+ "fix_coeffi": 0.9825418591499329,
1049
+ "focal_item_coeffi": 0.49965623021125793,
1050
+ "grad_norm": 2.4946190988147783,
1051
+ "learning_rate": 3.6089126804177364e-07,
1052
+ "logits": 0.02326827496290207,
1053
+ "loss": 0.3521,
1054
+ "policy_logits/chosen": -1.3168985843658447,
1055
+ "policy_logits/rejected": -0.9142702221870422,
1056
+ "policy_logps/chosen": -0.14688363671302795,
1057
+ "policy_logps/rejected": -0.17319604754447937,
1058
+ "policy_logratios": 0.026312392204999924,
1059
+ "positive_sft_coeffi": 0.16250000894069672,
1060
+ "ref_logps/chosen": -0.10597966611385345,
1061
+ "ref_logps/rejected": -0.10902378708124161,
1062
+ "ref_logratios": 0.0030441193375736475,
1063
+ "rewards/accuracies": 0.625,
1064
+ "rewards/chosen": -0.081807941198349,
1065
+ "rewards/margins": 0.04653654247522354,
1066
+ "rewards/rejected": -0.12834449112415314,
1067
+ "step": 46
1068
+ },
1069
+ {
1070
+ "epoch": 0.6016,
1071
+ "fix_coeffi": 0.9810597896575928,
1072
+ "focal_item_coeffi": 0.5131548047065735,
1073
+ "grad_norm": 3.060084071904193,
1074
+ "learning_rate": 3.4166600309926387e-07,
1075
+ "logits": 0.0006668753921985626,
1076
+ "loss": 0.3421,
1077
+ "policy_logits/chosen": -2.144726514816284,
1078
+ "policy_logits/rejected": -1.6465920209884644,
1079
+ "policy_logps/chosen": -0.13193106651306152,
1080
+ "policy_logps/rejected": -0.1363188922405243,
1081
+ "policy_logratios": 0.004387810826301575,
1082
+ "positive_sft_coeffi": 0.16250000894069672,
1083
+ "ref_logps/chosen": -0.0907638818025589,
1084
+ "ref_logps/rejected": -0.09448482096195221,
1085
+ "ref_logratios": 0.003720934735611081,
1086
+ "rewards/accuracies": 0.625,
1087
+ "rewards/chosen": -0.08233436197042465,
1088
+ "rewards/margins": 0.0013337507843971252,
1089
+ "rewards/rejected": -0.08366811275482178,
1090
+ "step": 47
1091
+ },
1092
+ {
1093
+ "epoch": 0.6144,
1094
+ "fix_coeffi": 0.9838154315948486,
1095
+ "focal_item_coeffi": 0.49068188667297363,
1096
+ "grad_norm": 2.2699255066538777,
1097
+ "learning_rate": 3.2269755647873214e-07,
1098
+ "logits": 0.037335123866796494,
1099
+ "loss": 0.3435,
1100
+ "policy_logits/chosen": -1.771597146987915,
1101
+ "policy_logits/rejected": -0.7590209245681763,
1102
+ "policy_logps/chosen": -0.13523265719413757,
1103
+ "policy_logps/rejected": -0.1755005419254303,
1104
+ "policy_logratios": 0.04026788845658302,
1105
+ "positive_sft_coeffi": 0.15000000596046448,
1106
+ "ref_logps/chosen": -0.10191057622432709,
1107
+ "ref_logps/rejected": -0.10484332591295242,
1108
+ "ref_logratios": 0.0029327599331736565,
1109
+ "rewards/accuracies": 0.8125,
1110
+ "rewards/chosen": -0.06664416939020157,
1111
+ "rewards/margins": 0.07467024773359299,
1112
+ "rewards/rejected": -0.14131443202495575,
1113
+ "step": 48
1114
+ },
1115
+ {
1116
+ "epoch": 0.6272,
1117
+ "fix_coeffi": 0.9797801375389099,
1118
+ "focal_item_coeffi": 0.4789538085460663,
1119
+ "grad_norm": 2.871402283313492,
1120
+ "learning_rate": 3.040166950699625e-07,
1121
+ "logits": 0.040210362523794174,
1122
+ "loss": 0.3331,
1123
+ "policy_logits/chosen": -2.0251073837280273,
1124
+ "policy_logits/rejected": -1.0917900800704956,
1125
+ "policy_logps/chosen": -0.1262364536523819,
1126
+ "policy_logps/rejected": -0.1857452392578125,
1127
+ "policy_logratios": 0.0595087856054306,
1128
+ "positive_sft_coeffi": 0.11250000447034836,
1129
+ "ref_logps/chosen": -0.09439139068126678,
1130
+ "ref_logps/rejected": -0.11368982493877411,
1131
+ "ref_logratios": 0.019298434257507324,
1132
+ "rewards/accuracies": 0.8125,
1133
+ "rewards/chosen": -0.06369014084339142,
1134
+ "rewards/margins": 0.08042072504758835,
1135
+ "rewards/rejected": -0.14411085844039917,
1136
+ "step": 49
1137
+ },
1138
+ {
1139
+ "epoch": 0.64,
1140
+ "fix_coeffi": 0.980614423751831,
1141
+ "focal_item_coeffi": 0.46539896726608276,
1142
+ "grad_norm": 3.26096015041,
1143
+ "learning_rate": 2.856537192984728e-07,
1144
+ "logits": 0.24540501832962036,
1145
+ "loss": 0.3521,
1146
+ "policy_logits/chosen": -2.0591659545898438,
1147
+ "policy_logits/rejected": -1.4475215673446655,
1148
+ "policy_logps/chosen": -0.1279754340648651,
1149
+ "policy_logps/rejected": -0.4230402410030365,
1150
+ "policy_logratios": 0.2950647473335266,
1151
+ "positive_sft_coeffi": 0.13750000298023224,
1152
+ "ref_logps/chosen": -0.08821889013051987,
1153
+ "ref_logps/rejected": -0.1378786861896515,
1154
+ "ref_logratios": 0.049659788608551025,
1155
+ "rewards/accuracies": 0.875,
1156
+ "rewards/chosen": -0.07951308786869049,
1157
+ "rewards/margins": 0.4908100366592407,
1158
+ "rewards/rejected": -0.5703231692314148,
1159
+ "step": 50
1160
+ },
1161
+ {
1162
+ "epoch": 0.6528,
1163
+ "fix_coeffi": 0.9841020107269287,
1164
+ "focal_item_coeffi": 0.4874262511730194,
1165
+ "grad_norm": 4.295010970631194,
1166
+ "learning_rate": 2.676384139781157e-07,
1167
+ "logits": 0.03224971145391464,
1168
+ "loss": 0.3411,
1169
+ "policy_logits/chosen": -1.7304267883300781,
1170
+ "policy_logits/rejected": -1.3776047229766846,
1171
+ "policy_logps/chosen": -0.12312814593315125,
1172
+ "policy_logps/rejected": -0.16908591985702515,
1173
+ "policy_logratios": 0.045957766473293304,
1174
+ "positive_sft_coeffi": 0.15000000596046448,
1175
+ "ref_logps/chosen": -0.09287795424461365,
1176
+ "ref_logps/rejected": -0.1065860241651535,
1177
+ "ref_logratios": 0.01370805874466896,
1178
+ "rewards/accuracies": 0.875,
1179
+ "rewards/chosen": -0.0605003647506237,
1180
+ "rewards/margins": 0.06449942290782928,
1181
+ "rewards/rejected": -0.12499978393316269,
1182
+ "step": 51
1183
+ },
1184
+ {
1185
+ "epoch": 0.6656,
1186
+ "fix_coeffi": 0.9823248386383057,
1187
+ "focal_item_coeffi": 0.5049876570701599,
1188
+ "grad_norm": 8.682374559650592,
1189
+ "learning_rate": 2.500000000000001e-07,
1190
+ "logits": 0.004311848431825638,
1191
+ "loss": 0.3496,
1192
+ "policy_logits/chosen": -2.3149898052215576,
1193
+ "policy_logits/rejected": -1.8041744232177734,
1194
+ "policy_logps/chosen": -0.19820746779441833,
1195
+ "policy_logps/rejected": -0.19096598029136658,
1196
+ "policy_logratios": -0.007241487503051758,
1197
+ "positive_sft_coeffi": 0.17500001192092896,
1198
+ "ref_logps/chosen": -0.12392059713602066,
1199
+ "ref_logps/rejected": -0.11236726492643356,
1200
+ "ref_logratios": -0.01155333872884512,
1201
+ "rewards/accuracies": 0.625,
1202
+ "rewards/chosen": -0.14857371151447296,
1203
+ "rewards/margins": 0.00862371176481247,
1204
+ "rewards/rejected": -0.15719741582870483,
1205
+ "step": 52
1206
+ },
1207
+ {
1208
+ "epoch": 0.6784,
1209
+ "fix_coeffi": 0.9784698486328125,
1210
+ "focal_item_coeffi": 0.5000934600830078,
1211
+ "grad_norm": 3.2899635222445567,
1212
+ "learning_rate": 2.3276708693609942e-07,
1213
+ "logits": 0.01874784380197525,
1214
+ "loss": 0.3322,
1215
+ "policy_logits/chosen": -2.42234468460083,
1216
+ "policy_logits/rejected": -1.9896959066390991,
1217
+ "policy_logps/chosen": -0.128046452999115,
1218
+ "policy_logps/rejected": -0.15386544167995453,
1219
+ "policy_logratios": 0.025818975642323494,
1220
+ "positive_sft_coeffi": 0.13750000298023224,
1221
+ "ref_logps/chosen": -0.0999474972486496,
1222
+ "ref_logps/rejected": -0.10701862722635269,
1223
+ "ref_logratios": 0.007071132306009531,
1224
+ "rewards/accuracies": 0.8125,
1225
+ "rewards/chosen": -0.05619791895151138,
1226
+ "rewards/margins": 0.0374956876039505,
1227
+ "rewards/rejected": -0.09369360655546188,
1228
+ "step": 53
1229
+ },
1230
+ {
1231
+ "epoch": 0.6912,
1232
+ "fix_coeffi": 0.977252721786499,
1233
+ "focal_item_coeffi": 0.4751685857772827,
1234
+ "grad_norm": 5.216218870308387,
1235
+ "learning_rate": 2.1596762663442213e-07,
1236
+ "logits": 0.04333459958434105,
1237
+ "loss": 0.3484,
1238
+ "policy_logits/chosen": -2.3006138801574707,
1239
+ "policy_logits/rejected": -1.436716914176941,
1240
+ "policy_logps/chosen": -0.13173560798168182,
1241
+ "policy_logps/rejected": -0.20424073934555054,
1242
+ "policy_logratios": 0.07250512391328812,
1243
+ "positive_sft_coeffi": 0.16250000894069672,
1244
+ "ref_logps/chosen": -0.0926876813173294,
1245
+ "ref_logps/rejected": -0.12185820192098618,
1246
+ "ref_logratios": 0.02917051501572132,
1247
+ "rewards/accuracies": 0.75,
1248
+ "rewards/chosen": -0.07809583097696304,
1249
+ "rewards/margins": 0.08666921406984329,
1250
+ "rewards/rejected": -0.16476503014564514,
1251
+ "step": 54
1252
+ },
1253
+ {
1254
+ "epoch": 0.704,
1255
+ "fix_coeffi": 0.9889881014823914,
1256
+ "focal_item_coeffi": 0.5077127814292908,
1257
+ "grad_norm": 7.76441906556794,
1258
+ "learning_rate": 1.9962886788101047e-07,
1259
+ "logits": 0.01651328057050705,
1260
+ "loss": 0.3548,
1261
+ "policy_logits/chosen": -1.377277135848999,
1262
+ "policy_logits/rejected": -1.0159400701522827,
1263
+ "policy_logps/chosen": -0.15637433528900146,
1264
+ "policy_logps/rejected": -0.1691744476556778,
1265
+ "policy_logratios": 0.012800119817256927,
1266
+ "positive_sft_coeffi": 0.16250000894069672,
1267
+ "ref_logps/chosen": -0.11394304037094116,
1268
+ "ref_logps/rejected": -0.11022987961769104,
1269
+ "ref_logratios": -0.003713162150233984,
1270
+ "rewards/accuracies": 0.5,
1271
+ "rewards/chosen": -0.08486258238554001,
1272
+ "rewards/margins": 0.0330265611410141,
1273
+ "rewards/rejected": -0.1178891509771347,
1274
+ "step": 55
1275
+ },
1276
+ {
1277
+ "epoch": 0.7168,
1278
+ "fix_coeffi": 0.981607973575592,
1279
+ "focal_item_coeffi": 0.4884195327758789,
1280
+ "grad_norm": 5.396219562910484,
1281
+ "learning_rate": 1.837773122023114e-07,
1282
+ "logits": 0.03817833587527275,
1283
+ "loss": 0.3241,
1284
+ "policy_logits/chosen": -1.8968136310577393,
1285
+ "policy_logits/rejected": -0.9397016763687134,
1286
+ "policy_logps/chosen": -0.1473933309316635,
1287
+ "policy_logps/rejected": -0.19120770692825317,
1288
+ "policy_logratios": 0.04381437599658966,
1289
+ "positive_sft_coeffi": 0.15000000596046448,
1290
+ "ref_logps/chosen": -0.10809781402349472,
1291
+ "ref_logps/rejected": -0.11373385787010193,
1292
+ "ref_logratios": 0.0056360443122684956,
1293
+ "rewards/accuracies": 0.6875,
1294
+ "rewards/chosen": -0.07859104871749878,
1295
+ "rewards/margins": 0.0763566717505455,
1296
+ "rewards/rejected": -0.15494771301746368,
1297
+ "step": 56
1298
+ },
1299
+ {
1300
+ "epoch": 0.7296,
1301
+ "fix_coeffi": 0.9756256341934204,
1302
+ "focal_item_coeffi": 0.5005915760993958,
1303
+ "grad_norm": 5.133006633233018,
1304
+ "learning_rate": 1.6843867087960251e-07,
1305
+ "logits": 0.030851740390062332,
1306
+ "loss": 0.3508,
1307
+ "policy_logits/chosen": -2.888317108154297,
1308
+ "policy_logits/rejected": -2.919947624206543,
1309
+ "policy_logps/chosen": -0.15259821712970734,
1310
+ "policy_logps/rejected": -0.1766326129436493,
1311
+ "policy_logratios": 0.024034399539232254,
1312
+ "positive_sft_coeffi": 0.15000000596046448,
1313
+ "ref_logps/chosen": -0.10026225447654724,
1314
+ "ref_logps/rejected": -0.09344490617513657,
1315
+ "ref_logratios": -0.006817337591201067,
1316
+ "rewards/accuracies": 0.6875,
1317
+ "rewards/chosen": -0.10467193275690079,
1318
+ "rewards/margins": 0.061703480780124664,
1319
+ "rewards/rejected": -0.16637542843818665,
1320
+ "step": 57
1321
+ },
1322
+ {
1323
+ "epoch": 0.7424,
1324
+ "fix_coeffi": 0.9847123622894287,
1325
+ "focal_item_coeffi": 0.46967267990112305,
1326
+ "grad_norm": 4.082738784883245,
1327
+ "learning_rate": 1.536378232452003e-07,
1328
+ "logits": 0.05128350853919983,
1329
+ "loss": 0.3276,
1330
+ "policy_logits/chosen": -1.7085278034210205,
1331
+ "policy_logits/rejected": -0.9004208445549011,
1332
+ "policy_logps/chosen": -0.13805311918258667,
1333
+ "policy_logps/rejected": -0.21297375857830048,
1334
+ "policy_logratios": 0.07492061704397202,
1335
+ "positive_sft_coeffi": 0.16250000894069672,
1336
+ "ref_logps/chosen": -0.10160607099533081,
1337
+ "ref_logps/rejected": -0.1252431869506836,
1338
+ "ref_logratios": 0.023637104779481888,
1339
+ "rewards/accuracies": 0.9375,
1340
+ "rewards/chosen": -0.07289408892393112,
1341
+ "rewards/margins": 0.10256701707839966,
1342
+ "rewards/rejected": -0.17546111345291138,
1343
+ "step": 58
1344
+ },
1345
+ {
1346
+ "epoch": 0.7552,
1347
+ "fix_coeffi": 0.9828384518623352,
1348
+ "focal_item_coeffi": 0.49018165469169617,
1349
+ "grad_norm": 2.3142464667942098,
1350
+ "learning_rate": 1.3939877632809278e-07,
1351
+ "logits": 0.03899291902780533,
1352
+ "loss": 0.3383,
1353
+ "policy_logits/chosen": -2.1067862510681152,
1354
+ "policy_logits/rejected": -1.2315044403076172,
1355
+ "policy_logps/chosen": -0.15005925297737122,
1356
+ "policy_logps/rejected": -0.19155989587306976,
1357
+ "policy_logratios": 0.04150063917040825,
1358
+ "positive_sft_coeffi": 0.17500001192092896,
1359
+ "ref_logps/chosen": -0.10577111691236496,
1360
+ "ref_logps/rejected": -0.10827884078025818,
1361
+ "ref_logratios": 0.002507720608264208,
1362
+ "rewards/accuracies": 0.8125,
1363
+ "rewards/chosen": -0.08857627213001251,
1364
+ "rewards/margins": 0.07798583805561066,
1365
+ "rewards/rejected": -0.16656211018562317,
1366
+ "step": 59
1367
+ },
1368
+ {
1369
+ "epoch": 0.768,
1370
+ "fix_coeffi": 0.9874282479286194,
1371
+ "focal_item_coeffi": 0.5020555257797241,
1372
+ "grad_norm": 3.5053286992560704,
1373
+ "learning_rate": 1.257446259144494e-07,
1374
+ "logits": 0.021277405321598053,
1375
+ "loss": 0.3396,
1376
+ "policy_logits/chosen": -1.7301051616668701,
1377
+ "policy_logits/rejected": -1.4250915050506592,
1378
+ "policy_logps/chosen": -0.1746881902217865,
1379
+ "policy_logps/rejected": -0.19680778682231903,
1380
+ "policy_logratios": 0.022119589149951935,
1381
+ "positive_sft_coeffi": 0.16250000894069672,
1382
+ "ref_logps/chosen": -0.13134512305259705,
1383
+ "ref_logps/rejected": -0.13218730688095093,
1384
+ "ref_logratios": 0.0008421875536441803,
1385
+ "rewards/accuracies": 0.625,
1386
+ "rewards/chosen": -0.0866861492395401,
1387
+ "rewards/margins": 0.042554810643196106,
1388
+ "rewards/rejected": -0.1292409598827362,
1389
+ "step": 60
1390
+ },
1391
+ {
1392
+ "epoch": 0.7808,
1393
+ "fix_coeffi": 0.9786032438278198,
1394
+ "focal_item_coeffi": 0.481638103723526,
1395
+ "grad_norm": 5.174851065100917,
1396
+ "learning_rate": 1.1269751908617275e-07,
1397
+ "logits": 0.0354713499546051,
1398
+ "loss": 0.3379,
1399
+ "policy_logits/chosen": -2.1576731204986572,
1400
+ "policy_logits/rejected": -1.5160058736801147,
1401
+ "policy_logps/chosen": -0.14598117768764496,
1402
+ "policy_logps/rejected": -0.2012050449848175,
1403
+ "policy_logratios": 0.05522387847304344,
1404
+ "positive_sft_coeffi": 0.15000000596046448,
1405
+ "ref_logps/chosen": -0.09668442606925964,
1406
+ "ref_logps/rejected": -0.11643696576356888,
1407
+ "ref_logratios": 0.019752532243728638,
1408
+ "rewards/accuracies": 0.8125,
1409
+ "rewards/chosen": -0.09859348833560944,
1410
+ "rewards/margins": 0.07094269245862961,
1411
+ "rewards/rejected": -0.16953618824481964,
1412
+ "step": 61
1413
+ },
1414
+ {
1415
+ "epoch": 0.7936,
1416
+ "fix_coeffi": 0.9856793880462646,
1417
+ "focal_item_coeffi": 0.4770285487174988,
1418
+ "grad_norm": 2.971259491354303,
1419
+ "learning_rate": 1.0027861829824952e-07,
1420
+ "logits": 0.03819984942674637,
1421
+ "loss": 0.3421,
1422
+ "policy_logits/chosen": -2.104520082473755,
1423
+ "policy_logits/rejected": -1.6713505983352661,
1424
+ "policy_logps/chosen": -0.14258340001106262,
1425
+ "policy_logps/rejected": -0.20525400340557098,
1426
+ "policy_logratios": 0.06267061829566956,
1427
+ "positive_sft_coeffi": 0.13750000298023224,
1428
+ "ref_logps/chosen": -0.10176616907119751,
1429
+ "ref_logps/rejected": -0.1262369304895401,
1430
+ "ref_logratios": 0.024470770731568336,
1431
+ "rewards/accuracies": 0.6875,
1432
+ "rewards/chosen": -0.08163446187973022,
1433
+ "rewards/margins": 0.07639969885349274,
1434
+ "rewards/rejected": -0.15803414583206177,
1435
+ "step": 62
1436
+ },
1437
+ {
1438
+ "epoch": 0.8064,
1439
+ "fix_coeffi": 0.9773581027984619,
1440
+ "focal_item_coeffi": 0.4988199472427368,
1441
+ "grad_norm": 3.672860621163625,
1442
+ "learning_rate": 8.850806705317182e-08,
1443
+ "logits": 0.025526590645313263,
1444
+ "loss": 0.3292,
1445
+ "policy_logits/chosen": -2.535458564758301,
1446
+ "policy_logits/rejected": -2.444281578063965,
1447
+ "policy_logps/chosen": -0.1547073870897293,
1448
+ "policy_logps/rejected": -0.18234416842460632,
1449
+ "policy_logratios": 0.027636803686618805,
1450
+ "positive_sft_coeffi": 0.13750000298023224,
1451
+ "ref_logps/chosen": -0.10086537897586823,
1452
+ "ref_logps/rejected": -0.10297559946775436,
1453
+ "ref_logratios": 0.002110213739797473,
1454
+ "rewards/accuracies": 0.625,
1455
+ "rewards/chosen": -0.10768399387598038,
1456
+ "rewards/margins": 0.051053181290626526,
1457
+ "rewards/rejected": -0.1587371826171875,
1458
+ "step": 63
1459
+ },
1460
+ {
1461
+ "epoch": 0.8192,
1462
+ "fix_coeffi": 0.9807887673377991,
1463
+ "focal_item_coeffi": 0.4760918617248535,
1464
+ "grad_norm": 3.5701391239929565,
1465
+ "learning_rate": 7.740495722810269e-08,
1466
+ "logits": 0.06111777573823929,
1467
+ "loss": 0.3393,
1468
+ "policy_logits/chosen": -2.5813114643096924,
1469
+ "policy_logits/rejected": -2.126056671142578,
1470
+ "policy_logps/chosen": -0.14353778958320618,
1471
+ "policy_logps/rejected": -0.20789176225662231,
1472
+ "policy_logratios": 0.06435397267341614,
1473
+ "positive_sft_coeffi": 0.16250000894069672,
1474
+ "ref_logps/chosen": -0.10706878453493118,
1475
+ "ref_logps/rejected": -0.11030498147010803,
1476
+ "ref_logratios": 0.003236193675547838,
1477
+ "rewards/accuracies": 0.8125,
1478
+ "rewards/chosen": -0.07293801009654999,
1479
+ "rewards/margins": 0.12223555892705917,
1480
+ "rewards/rejected": -0.19517356157302856,
1481
+ "step": 64
1482
+ },
1483
+ {
1484
+ "epoch": 0.832,
1485
+ "fix_coeffi": 0.9829193949699402,
1486
+ "focal_item_coeffi": 0.4961282014846802,
1487
+ "grad_norm": 5.916810835601197,
1488
+ "learning_rate": 6.698729810778064e-08,
1489
+ "logits": 0.022682160139083862,
1490
+ "loss": 0.3428,
1491
+ "policy_logits/chosen": -2.7547287940979004,
1492
+ "policy_logits/rejected": -1.9564570188522339,
1493
+ "policy_logps/chosen": -0.1532108336687088,
1494
+ "policy_logps/rejected": -0.18559576570987701,
1495
+ "policy_logratios": 0.03238494321703911,
1496
+ "positive_sft_coeffi": 0.13750000298023224,
1497
+ "ref_logps/chosen": -0.09967292845249176,
1498
+ "ref_logps/rejected": -0.1093757152557373,
1499
+ "ref_logratios": 0.009702783077955246,
1500
+ "rewards/accuracies": 0.875,
1501
+ "rewards/chosen": -0.10707581043243408,
1502
+ "rewards/margins": 0.045364320278167725,
1503
+ "rewards/rejected": -0.1524401307106018,
1504
+ "step": 65
1505
+ },
1506
+ {
1507
+ "epoch": 0.8448,
1508
+ "fix_coeffi": 0.9772939682006836,
1509
+ "focal_item_coeffi": 0.4869677722454071,
1510
+ "grad_norm": 6.855589998194368,
1511
+ "learning_rate": 5.72719871733951e-08,
1512
+ "logits": 0.04695258289575577,
1513
+ "loss": 0.3473,
1514
+ "policy_logits/chosen": -2.0387773513793945,
1515
+ "policy_logits/rejected": -1.5787204504013062,
1516
+ "policy_logps/chosen": -0.14013159275054932,
1517
+ "policy_logps/rejected": -0.1868254691362381,
1518
+ "policy_logratios": 0.04669388383626938,
1519
+ "positive_sft_coeffi": 0.13750000298023224,
1520
+ "ref_logps/chosen": -0.10522621870040894,
1521
+ "ref_logps/rejected": -0.10496751219034195,
1522
+ "ref_logratios": -0.0002587023191154003,
1523
+ "rewards/accuracies": 0.75,
1524
+ "rewards/chosen": -0.06981074810028076,
1525
+ "rewards/margins": 0.09390517324209213,
1526
+ "rewards/rejected": -0.1637159138917923,
1527
+ "step": 66
1528
+ },
1529
+ {
1530
+ "epoch": 0.8576,
1531
+ "fix_coeffi": 0.9817394614219666,
1532
+ "focal_item_coeffi": 0.4786987602710724,
1533
+ "grad_norm": 2.6975498742648316,
1534
+ "learning_rate": 4.827478269480895e-08,
1535
+ "logits": 0.050806205719709396,
1536
+ "loss": 0.3442,
1537
+ "policy_logits/chosen": -2.305556297302246,
1538
+ "policy_logits/rejected": -1.8619630336761475,
1539
+ "policy_logps/chosen": -0.14199405908584595,
1540
+ "policy_logps/rejected": -0.20300914347171783,
1541
+ "policy_logratios": 0.06101508438587189,
1542
+ "positive_sft_coeffi": 0.1875000149011612,
1543
+ "ref_logps/chosen": -0.09993252158164978,
1544
+ "ref_logps/rejected": -0.11014139652252197,
1545
+ "ref_logratios": 0.010208876803517342,
1546
+ "rewards/accuracies": 0.75,
1547
+ "rewards/chosen": -0.08412308990955353,
1548
+ "rewards/margins": 0.10161241143941879,
1549
+ "rewards/rejected": -0.18573549389839172,
1550
+ "step": 67
1551
+ },
1552
+ {
1553
+ "epoch": 0.8704,
1554
+ "fix_coeffi": 0.9803918600082397,
1555
+ "focal_item_coeffi": 0.4953024387359619,
1556
+ "grad_norm": 5.592981578505752,
1557
+ "learning_rate": 4.0010278170587884e-08,
1558
+ "logits": 0.019841268658638,
1559
+ "loss": 0.345,
1560
+ "policy_logits/chosen": -2.4927449226379395,
1561
+ "policy_logits/rejected": -1.6543523073196411,
1562
+ "policy_logps/chosen": -0.14626118540763855,
1563
+ "policy_logps/rejected": -0.17983616888523102,
1564
+ "policy_logratios": 0.033574994653463364,
1565
+ "positive_sft_coeffi": 0.1875000149011612,
1566
+ "ref_logps/chosen": -0.10440389066934586,
1567
+ "ref_logps/rejected": -0.11813762038946152,
1568
+ "ref_logratios": 0.013733725994825363,
1569
+ "rewards/accuracies": 0.625,
1570
+ "rewards/chosen": -0.08371458947658539,
1571
+ "rewards/margins": 0.039682537317276,
1572
+ "rewards/rejected": -0.12339711934328079,
1573
+ "step": 68
1574
+ },
1575
+ {
1576
+ "epoch": 0.8832,
1577
+ "fix_coeffi": 0.983547031879425,
1578
+ "focal_item_coeffi": 0.47107985615730286,
1579
+ "grad_norm": 4.117455699123021,
1580
+ "learning_rate": 3.2491878657292635e-08,
1581
+ "logits": 0.06389674544334412,
1582
+ "loss": 0.3294,
1583
+ "policy_logits/chosen": -3.0289320945739746,
1584
+ "policy_logits/rejected": -2.6473026275634766,
1585
+ "policy_logps/chosen": -0.13031131029129028,
1586
+ "policy_logps/rejected": -0.20251160860061646,
1587
+ "policy_logratios": 0.07220031321048737,
1588
+ "positive_sft_coeffi": 0.17500001192092896,
1589
+ "ref_logps/chosen": -0.1000053882598877,
1590
+ "ref_logps/rejected": -0.10830894857645035,
1591
+ "ref_logratios": 0.0083035659044981,
1592
+ "rewards/accuracies": 0.875,
1593
+ "rewards/chosen": -0.06061182916164398,
1594
+ "rewards/margins": 0.12779349088668823,
1595
+ "rewards/rejected": -0.18840530514717102,
1596
+ "step": 69
1597
+ },
1598
+ {
1599
+ "epoch": 0.896,
1600
+ "fix_coeffi": 0.9875835180282593,
1601
+ "focal_item_coeffi": 0.4833886921405792,
1602
+ "grad_norm": 5.501583323515586,
1603
+ "learning_rate": 2.5731779026427257e-08,
1604
+ "logits": 0.04149772599339485,
1605
+ "loss": 0.3383,
1606
+ "policy_logits/chosen": -2.2594168186187744,
1607
+ "policy_logits/rejected": -1.790095567703247,
1608
+ "policy_logps/chosen": -0.1648138463497162,
1609
+ "policy_logps/rejected": -0.21739351749420166,
1610
+ "policy_logratios": 0.052579667419195175,
1611
+ "positive_sft_coeffi": 0.16250000894069672,
1612
+ "ref_logps/chosen": -0.11531969904899597,
1613
+ "ref_logps/rejected": -0.1264016330242157,
1614
+ "ref_logratios": 0.011081943288445473,
1615
+ "rewards/accuracies": 0.8125,
1616
+ "rewards/chosen": -0.09898830950260162,
1617
+ "rewards/margins": 0.0829954519867897,
1618
+ "rewards/rejected": -0.18198375403881073,
1619
+ "step": 70
1620
+ },
1621
+ {
1622
+ "epoch": 0.9088,
1623
+ "fix_coeffi": 0.9863916635513306,
1624
+ "focal_item_coeffi": 0.4777306616306305,
1625
+ "grad_norm": 2.6990624593275676,
1626
+ "learning_rate": 1.974094418431388e-08,
1627
+ "logits": 0.052217770367860794,
1628
+ "loss": 0.3289,
1629
+ "policy_logits/chosen": -2.2797064781188965,
1630
+ "policy_logits/rejected": -2.0593111515045166,
1631
+ "policy_logps/chosen": -0.15526652336120605,
1632
+ "policy_logps/rejected": -0.2166626900434494,
1633
+ "policy_logratios": 0.06139618158340454,
1634
+ "positive_sft_coeffi": 0.16250000894069672,
1635
+ "ref_logps/chosen": -0.10544908791780472,
1636
+ "ref_logps/rejected": -0.11462749540805817,
1637
+ "ref_logratios": 0.00917841400951147,
1638
+ "rewards/accuracies": 0.6875,
1639
+ "rewards/chosen": -0.09963487088680267,
1640
+ "rewards/margins": 0.10443554073572159,
1641
+ "rewards/rejected": -0.20407040417194366,
1642
+ "step": 71
1643
+ },
1644
+ {
1645
+ "epoch": 0.9216,
1646
+ "fix_coeffi": 0.9806711673736572,
1647
+ "focal_item_coeffi": 0.5071695446968079,
1648
+ "grad_norm": 3.123774821973762,
1649
+ "learning_rate": 1.4529091286973993e-08,
1650
+ "logits": -0.03165913373231888,
1651
+ "loss": 0.3542,
1652
+ "policy_logits/chosen": -2.540611505508423,
1653
+ "policy_logits/rejected": -2.129610776901245,
1654
+ "policy_logps/chosen": -0.2025057077407837,
1655
+ "policy_logps/rejected": -0.18385374546051025,
1656
+ "policy_logratios": -0.018651969730854034,
1657
+ "positive_sft_coeffi": 0.15000000596046448,
1658
+ "ref_logps/chosen": -0.09205804765224457,
1659
+ "ref_logps/rejected": -0.10506521165370941,
1660
+ "ref_logratios": 0.013007167726755142,
1661
+ "rewards/accuracies": 0.6875,
1662
+ "rewards/chosen": -0.22089532017707825,
1663
+ "rewards/margins": -0.06331825256347656,
1664
+ "rewards/rejected": -0.1575770378112793,
1665
+ "step": 72
1666
+ },
1667
+ {
1668
+ "epoch": 0.9344,
1669
+ "fix_coeffi": 0.984578549861908,
1670
+ "focal_item_coeffi": 0.47575342655181885,
1671
+ "grad_norm": 3.9065582771005425,
1672
+ "learning_rate": 1.0104673978866163e-08,
1673
+ "logits": 0.042688433080911636,
1674
+ "loss": 0.3397,
1675
+ "policy_logits/chosen": -2.628143072128296,
1676
+ "policy_logits/rejected": -1.52046537399292,
1677
+ "policy_logps/chosen": -0.1213134154677391,
1678
+ "policy_logps/rejected": -0.18583202362060547,
1679
+ "policy_logratios": 0.06451861560344696,
1680
+ "positive_sft_coeffi": 0.16250000894069672,
1681
+ "ref_logps/chosen": -0.09047695994377136,
1682
+ "ref_logps/rejected": -0.11230714619159698,
1683
+ "ref_logratios": 0.02183018997311592,
1684
+ "rewards/accuracies": 0.6875,
1685
+ "rewards/chosen": -0.061672914773225784,
1686
+ "rewards/margins": 0.08537686616182327,
1687
+ "rewards/rejected": -0.14704976975917816,
1688
+ "step": 73
1689
+ },
1690
+ {
1691
+ "epoch": 0.9472,
1692
+ "fix_coeffi": 0.9837413430213928,
1693
+ "focal_item_coeffi": 0.49963295459747314,
1694
+ "grad_norm": 2.9462642527275844,
1695
+ "learning_rate": 6.474868681043577e-09,
1696
+ "logits": 0.017393849790096283,
1697
+ "loss": 0.3434,
1698
+ "policy_logits/chosen": -3.072700262069702,
1699
+ "policy_logits/rejected": -2.5184197425842285,
1700
+ "policy_logps/chosen": -0.1256764531135559,
1701
+ "policy_logps/rejected": -0.15233443677425385,
1702
+ "policy_logratios": 0.02665800042450428,
1703
+ "positive_sft_coeffi": 0.1875000149011612,
1704
+ "ref_logps/chosen": -0.08265330642461777,
1705
+ "ref_logps/rejected": -0.09191745519638062,
1706
+ "ref_logratios": 0.009264146909117699,
1707
+ "rewards/accuracies": 0.625,
1708
+ "rewards/chosen": -0.08604628592729568,
1709
+ "rewards/margins": 0.034787699580192566,
1710
+ "rewards/rejected": -0.12083399295806885,
1711
+ "step": 74
1712
+ },
1713
+ {
1714
+ "epoch": 0.96,
1715
+ "fix_coeffi": 0.9899287223815918,
1716
+ "focal_item_coeffi": 0.49144142866134644,
1717
+ "grad_norm": 4.616196389415246,
1718
+ "learning_rate": 3.6455629509730135e-09,
1719
+ "logits": 0.03687784820795059,
1720
+ "loss": 0.3423,
1721
+ "policy_logits/chosen": -2.1187071800231934,
1722
+ "policy_logits/rejected": -1.3435924053192139,
1723
+ "policy_logps/chosen": -0.1363033503293991,
1724
+ "policy_logps/rejected": -0.17593124508857727,
1725
+ "policy_logratios": 0.03962789103388786,
1726
+ "positive_sft_coeffi": 0.1875000149011612,
1727
+ "ref_logps/chosen": -0.10113701224327087,
1728
+ "ref_logps/rejected": -0.10388705134391785,
1729
+ "ref_logratios": 0.0027500393334776163,
1730
+ "rewards/accuracies": 0.8125,
1731
+ "rewards/chosen": -0.07033268362283707,
1732
+ "rewards/margins": 0.07375569641590118,
1733
+ "rewards/rejected": -0.14408838748931885,
1734
+ "step": 75
1735
+ },
1736
+ {
1737
+ "epoch": 0.9728,
1738
+ "fix_coeffi": 0.9790648221969604,
1739
+ "focal_item_coeffi": 0.5043489336967468,
1740
+ "grad_norm": 2.5732518774405997,
1741
+ "learning_rate": 1.6213459328950352e-09,
1742
+ "logits": 0.008588922210037708,
1743
+ "loss": 0.3209,
1744
+ "policy_logits/chosen": -1.9825241565704346,
1745
+ "policy_logits/rejected": -1.5509438514709473,
1746
+ "policy_logps/chosen": -0.16658538579940796,
1747
+ "policy_logps/rejected": -0.1852148473262787,
1748
+ "policy_logratios": 0.01862948387861252,
1749
+ "positive_sft_coeffi": 0.13750000298023224,
1750
+ "ref_logps/chosen": -0.11572954058647156,
1751
+ "ref_logps/rejected": -0.12577009201049805,
1752
+ "ref_logratios": 0.010040560737252235,
1753
+ "rewards/accuracies": 0.6875,
1754
+ "rewards/chosen": -0.10171166807413101,
1755
+ "rewards/margins": 0.017177844420075417,
1756
+ "rewards/rejected": -0.11888951063156128,
1757
+ "step": 76
1758
+ },
1759
+ {
1760
+ "epoch": 0.9856,
1761
+ "fix_coeffi": 0.9805728197097778,
1762
+ "focal_item_coeffi": 0.4998331665992737,
1763
+ "grad_norm": 4.018731140340865,
1764
+ "learning_rate": 4.055009142152066e-10,
1765
+ "logits": 0.014980219304561615,
1766
+ "loss": 0.3359,
1767
+ "policy_logits/chosen": -2.5756959915161133,
1768
+ "policy_logits/rejected": -2.020047426223755,
1769
+ "policy_logps/chosen": -0.12740713357925415,
1770
+ "policy_logps/rejected": -0.1532643437385559,
1771
+ "policy_logratios": 0.02585720457136631,
1772
+ "positive_sft_coeffi": 0.17500001192092896,
1773
+ "ref_logps/chosen": -0.0916062742471695,
1774
+ "ref_logps/rejected": -0.10248326510190964,
1775
+ "ref_logratios": 0.01087698619812727,
1776
+ "rewards/accuracies": 0.6875,
1777
+ "rewards/chosen": -0.07160171866416931,
1778
+ "rewards/margins": 0.02996043860912323,
1779
+ "rewards/rejected": -0.10156215727329254,
1780
+ "step": 77
1781
+ },
1782
+ {
1783
+ "epoch": 0.9984,
1784
+ "fix_coeffi": 0.9791566729545593,
1785
+ "focal_item_coeffi": 0.49183690547943115,
1786
+ "grad_norm": 5.549344233642899,
1787
+ "learning_rate": 0.0,
1788
+ "logits": 0.02598106488585472,
1789
+ "loss": 0.3295,
1790
+ "policy_logits/chosen": -2.2902348041534424,
1791
+ "policy_logits/rejected": -1.7178772687911987,
1792
+ "policy_logps/chosen": -0.1306924968957901,
1793
+ "policy_logps/rejected": -0.16928984224796295,
1794
+ "policy_logratios": 0.03859733045101166,
1795
+ "positive_sft_coeffi": 0.16250000894069672,
1796
+ "ref_logps/chosen": -0.09151959419250488,
1797
+ "ref_logps/rejected": -0.10413586348295212,
1798
+ "ref_logratios": 0.012616267427802086,
1799
+ "rewards/accuracies": 0.75,
1800
+ "rewards/chosen": -0.07834579050540924,
1801
+ "rewards/margins": 0.05196212977170944,
1802
+ "rewards/rejected": -0.13030791282653809,
1803
+ "step": 78
1804
+ },
1805
+ {
1806
+ "epoch": 0.9984,
1807
+ "step": 78,
1808
+ "total_flos": 0.0,
1809
+ "train_loss": 0.3464691298894393,
1810
+ "train_runtime": 1464.0997,
1811
+ "train_samples_per_second": 6.83,
1812
+ "train_steps_per_second": 0.053
1813
+ }
1814
+ ],
1815
+ "logging_steps": 1,
1816
+ "max_steps": 78,
1817
+ "num_input_tokens_seen": 0,
1818
+ "num_train_epochs": 1,
1819
+ "save_steps": 500,
1820
+ "stateful_callbacks": {
1821
+ "TrainerControl": {
1822
+ "args": {
1823
+ "should_epoch_stop": false,
1824
+ "should_evaluate": false,
1825
+ "should_log": false,
1826
+ "should_save": false,
1827
+ "should_training_stop": false
1828
+ },
1829
+ "attributes": {}
1830
+ }
1831
+ },
1832
+ "total_flos": 0.0,
1833
+ "train_batch_size": 1,
1834
+ "trial_name": null,
1835
+ "trial_params": null
1836
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3863e35127b0d3337531a566409bc2b799e2e00de4bc7df95488c2ae8f36fbbe
3
+ size 7992
vocab.json ADDED
The diff for this file is too large to render. See raw diff