Hugging Face's logo

Pretam
/
NIOS_Trilingual_Decoder_Biased

PEFT
Safetensors
Transformers
lora
Model card Files
xet
 Community
Pretam commited on
Commit
39085d5
·
verified ·
1 Parent(s): 153adf5

Upload folder using huggingface_hub

Browse files
Files changed (22) hide show
  1. .gitattributes +2 -0
  2. README.md +206 -0
  3. adapter_config.json +237 -0
  4. adapter_model.safetensors +3 -0
  5. all_results.json +16 -0
  6. checkpoint-30630/README.md +206 -0
  7. checkpoint-30630/adapter_config.json +237 -0
  8. checkpoint-30630/adapter_model.safetensors +3 -0
  9. checkpoint-30630/optimizer.pt +3 -0
  10. checkpoint-30630/rng_state.pth +3 -0
  11. checkpoint-30630/scaler.pt +3 -0
  12. checkpoint-30630/scheduler.pt +3 -0
  13. checkpoint-30630/tokenizer.json +3 -0
  14. checkpoint-30630/tokenizer_config.json +221 -0
  15. checkpoint-30630/trainer_state.json +461 -0
  16. checkpoint-30630/training_args.bin +3 -0
  17. eval_results.json +10 -0
  18. tokenizer.json +3 -0
  19. tokenizer_config.json +221 -0
  20. train_results.json +9 -0
  21. trainer_state.json +470 -0
  22. training_args.bin +3 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ checkpoint-30630/tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: facebook/nllb-200-distilled-600M
3
+ library_name: peft
4
+ tags:
5
+ - base_model:adapter:facebook/nllb-200-distilled-600M
6
+ - lora
7
+ - transformers
8
+ ---
9
+
10
+ # Model Card for Model ID
11
+
12
+ <!-- Provide a quick summary of what the model is/does. -->
13
+
14
+
15
+
16
+ ## Model Details
17
+
18
+ ### Model Description
19
+
20
+ <!-- Provide a longer summary of what this model is. -->
21
+
22
+
23
+
24
+ - **Developed by:** [More Information Needed]
25
+ - **Funded by [optional]:** [More Information Needed]
26
+ - **Shared by [optional]:** [More Information Needed]
27
+ - **Model type:** [More Information Needed]
28
+ - **Language(s) (NLP):** [More Information Needed]
29
+ - **License:** [More Information Needed]
30
+ - **Finetuned from model [optional]:** [More Information Needed]
31
+
32
+ ### Model Sources [optional]
33
+
34
+ <!-- Provide the basic links for the model. -->
35
+
36
+ - **Repository:** [More Information Needed]
37
+ - **Paper [optional]:** [More Information Needed]
38
+ - **Demo [optional]:** [More Information Needed]
39
+
40
+ ## Uses
41
+
42
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
43
+
44
+ ### Direct Use
45
+
46
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
47
+
48
+ [More Information Needed]
49
+
50
+ ### Downstream Use [optional]
51
+
52
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
53
+
54
+ [More Information Needed]
55
+
56
+ ### Out-of-Scope Use
57
+
58
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
59
+
60
+ [More Information Needed]
61
+
62
+ ## Bias, Risks, and Limitations
63
+
64
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
65
+
66
+ [More Information Needed]
67
+
68
+ ### Recommendations
69
+
70
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
71
+
72
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
73
+
74
+ ## How to Get Started with the Model
75
+
76
+ Use the code below to get started with the model.
77
+
78
+ [More Information Needed]
79
+
80
+ ## Training Details
81
+
82
+ ### Training Data
83
+
84
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
85
+
86
+ [More Information Needed]
87
+
88
+ ### Training Procedure
89
+
90
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
91
+
92
+ #### Preprocessing [optional]
93
+
94
+ [More Information Needed]
95
+
96
+
97
+ #### Training Hyperparameters
98
+
99
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
100
+
101
+ #### Speeds, Sizes, Times [optional]
102
+
103
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
104
+
105
+ [More Information Needed]
106
+
107
+ ## Evaluation
108
+
109
+ <!-- This section describes the evaluation protocols and provides the results. -->
110
+
111
+ ### Testing Data, Factors & Metrics
112
+
113
+ #### Testing Data
114
+
115
+ <!-- This should link to a Dataset Card if possible. -->
116
+
117
+ [More Information Needed]
118
+
119
+ #### Factors
120
+
121
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
122
+
123
+ [More Information Needed]
124
+
125
+ #### Metrics
126
+
127
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
128
+
129
+ [More Information Needed]
130
+
131
+ ### Results
132
+
133
+ [More Information Needed]
134
+
135
+ #### Summary
136
+
137
+
138
+
139
+ ## Model Examination [optional]
140
+
141
+ <!-- Relevant interpretability work for the model goes here -->
142
+
143
+ [More Information Needed]
144
+
145
+ ## Environmental Impact
146
+
147
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
148
+
149
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
150
+
151
+ - **Hardware Type:** [More Information Needed]
152
+ - **Hours used:** [More Information Needed]
153
+ - **Cloud Provider:** [More Information Needed]
154
+ - **Compute Region:** [More Information Needed]
155
+ - **Carbon Emitted:** [More Information Needed]
156
+
157
+ ## Technical Specifications [optional]
158
+
159
+ ### Model Architecture and Objective
160
+
161
+ [More Information Needed]
162
+
163
+ ### Compute Infrastructure
164
+
165
+ [More Information Needed]
166
+
167
+ #### Hardware
168
+
169
+ [More Information Needed]
170
+
171
+ #### Software
172
+
173
+ [More Information Needed]
174
+
175
+ ## Citation [optional]
176
+
177
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
178
+
179
+ **BibTeX:**
180
+
181
+ [More Information Needed]
182
+
183
+ **APA:**
184
+
185
+ [More Information Needed]
186
+
187
+ ## Glossary [optional]
188
+
189
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
190
+
191
+ [More Information Needed]
192
+
193
+ ## More Information [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Authors [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Contact
202
+
203
+ [More Information Needed]
204
+ ### Framework versions
205
+
206
+ - PEFT 0.18.1
adapter_config.json ADDED
@@ -0,0 +1,237 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {
4
+ "model.decoder.layers.0.encoder_attn.k_proj": 128,
5
+ "model.decoder.layers.0.encoder_attn.out_proj": 128,
6
+ "model.decoder.layers.0.encoder_attn.q_proj": 128,
7
+ "model.decoder.layers.0.encoder_attn.v_proj": 128,
8
+ "model.decoder.layers.0.self_attn.k_proj": 64,
9
+ "model.decoder.layers.0.self_attn.out_proj": 64,
10
+ "model.decoder.layers.0.self_attn.q_proj": 64,
11
+ "model.decoder.layers.0.self_attn.v_proj": 64,
12
+ "model.decoder.layers.1.encoder_attn.k_proj": 128,
13
+ "model.decoder.layers.1.encoder_attn.out_proj": 128,
14
+ "model.decoder.layers.1.encoder_attn.q_proj": 128,
15
+ "model.decoder.layers.1.encoder_attn.v_proj": 128,
16
+ "model.decoder.layers.1.self_attn.k_proj": 64,
17
+ "model.decoder.layers.1.self_attn.out_proj": 64,
18
+ "model.decoder.layers.1.self_attn.q_proj": 64,
19
+ "model.decoder.layers.1.self_attn.v_proj": 64,
20
+ "model.decoder.layers.10.encoder_attn.k_proj": 128,
21
+ "model.decoder.layers.10.encoder_attn.out_proj": 128,
22
+ "model.decoder.layers.10.encoder_attn.q_proj": 128,
23
+ "model.decoder.layers.10.encoder_attn.v_proj": 128,
24
+ "model.decoder.layers.10.self_attn.k_proj": 64,
25
+ "model.decoder.layers.10.self_attn.out_proj": 64,
26
+ "model.decoder.layers.10.self_attn.q_proj": 64,
27
+ "model.decoder.layers.10.self_attn.v_proj": 64,
28
+ "model.decoder.layers.11.encoder_attn.k_proj": 128,
29
+ "model.decoder.layers.11.encoder_attn.out_proj": 128,
30
+ "model.decoder.layers.11.encoder_attn.q_proj": 128,
31
+ "model.decoder.layers.11.encoder_attn.v_proj": 128,
32
+ "model.decoder.layers.11.self_attn.k_proj": 64,
33
+ "model.decoder.layers.11.self_attn.out_proj": 64,
34
+ "model.decoder.layers.11.self_attn.q_proj": 64,
35
+ "model.decoder.layers.11.self_attn.v_proj": 64,
36
+ "model.decoder.layers.2.encoder_attn.k_proj": 128,
37
+ "model.decoder.layers.2.encoder_attn.out_proj": 128,
38
+ "model.decoder.layers.2.encoder_attn.q_proj": 128,
39
+ "model.decoder.layers.2.encoder_attn.v_proj": 128,
40
+ "model.decoder.layers.2.self_attn.k_proj": 64,
41
+ "model.decoder.layers.2.self_attn.out_proj": 64,
42
+ "model.decoder.layers.2.self_attn.q_proj": 64,
43
+ "model.decoder.layers.2.self_attn.v_proj": 64,
44
+ "model.decoder.layers.3.encoder_attn.k_proj": 128,
45
+ "model.decoder.layers.3.encoder_attn.out_proj": 128,
46
+ "model.decoder.layers.3.encoder_attn.q_proj": 128,
47
+ "model.decoder.layers.3.encoder_attn.v_proj": 128,
48
+ "model.decoder.layers.3.self_attn.k_proj": 64,
49
+ "model.decoder.layers.3.self_attn.out_proj": 64,
50
+ "model.decoder.layers.3.self_attn.q_proj": 64,
51
+ "model.decoder.layers.3.self_attn.v_proj": 64,
52
+ "model.decoder.layers.4.encoder_attn.k_proj": 128,
53
+ "model.decoder.layers.4.encoder_attn.out_proj": 128,
54
+ "model.decoder.layers.4.encoder_attn.q_proj": 128,
55
+ "model.decoder.layers.4.encoder_attn.v_proj": 128,
56
+ "model.decoder.layers.4.self_attn.k_proj": 64,
57
+ "model.decoder.layers.4.self_attn.out_proj": 64,
58
+ "model.decoder.layers.4.self_attn.q_proj": 64,
59
+ "model.decoder.layers.4.self_attn.v_proj": 64,
60
+ "model.decoder.layers.5.encoder_attn.k_proj": 128,
61
+ "model.decoder.layers.5.encoder_attn.out_proj": 128,
62
+ "model.decoder.layers.5.encoder_attn.q_proj": 128,
63
+ "model.decoder.layers.5.encoder_attn.v_proj": 128,
64
+ "model.decoder.layers.5.self_attn.k_proj": 64,
65
+ "model.decoder.layers.5.self_attn.out_proj": 64,
66
+ "model.decoder.layers.5.self_attn.q_proj": 64,
67
+ "model.decoder.layers.5.self_attn.v_proj": 64,
68
+ "model.decoder.layers.6.encoder_attn.k_proj": 128,
69
+ "model.decoder.layers.6.encoder_attn.out_proj": 128,
70
+ "model.decoder.layers.6.encoder_attn.q_proj": 128,
71
+ "model.decoder.layers.6.encoder_attn.v_proj": 128,
72
+ "model.decoder.layers.6.self_attn.k_proj": 64,
73
+ "model.decoder.layers.6.self_attn.out_proj": 64,
74
+ "model.decoder.layers.6.self_attn.q_proj": 64,
75
+ "model.decoder.layers.6.self_attn.v_proj": 64,
76
+ "model.decoder.layers.7.encoder_attn.k_proj": 128,
77
+ "model.decoder.layers.7.encoder_attn.out_proj": 128,
78
+ "model.decoder.layers.7.encoder_attn.q_proj": 128,
79
+ "model.decoder.layers.7.encoder_attn.v_proj": 128,
80
+ "model.decoder.layers.7.self_attn.k_proj": 64,
81
+ "model.decoder.layers.7.self_attn.out_proj": 64,
82
+ "model.decoder.layers.7.self_attn.q_proj": 64,
83
+ "model.decoder.layers.7.self_attn.v_proj": 64,
84
+ "model.decoder.layers.8.encoder_attn.k_proj": 128,
85
+ "model.decoder.layers.8.encoder_attn.out_proj": 128,
86
+ "model.decoder.layers.8.encoder_attn.q_proj": 128,
87
+ "model.decoder.layers.8.encoder_attn.v_proj": 128,
88
+ "model.decoder.layers.8.self_attn.k_proj": 64,
89
+ "model.decoder.layers.8.self_attn.out_proj": 64,
90
+ "model.decoder.layers.8.self_attn.q_proj": 64,
91
+ "model.decoder.layers.8.self_attn.v_proj": 64,
92
+ "model.decoder.layers.9.encoder_attn.k_proj": 128,
93
+ "model.decoder.layers.9.encoder_attn.out_proj": 128,
94
+ "model.decoder.layers.9.encoder_attn.q_proj": 128,
95
+ "model.decoder.layers.9.encoder_attn.v_proj": 128,
96
+ "model.decoder.layers.9.self_attn.k_proj": 64,
97
+ "model.decoder.layers.9.self_attn.out_proj": 64,
98
+ "model.decoder.layers.9.self_attn.q_proj": 64,
99
+ "model.decoder.layers.9.self_attn.v_proj": 64
100
+ },
101
+ "arrow_config": null,
102
+ "auto_mapping": null,
103
+ "base_model_name_or_path": "facebook/nllb-200-distilled-600M",
104
+ "bias": "none",
105
+ "corda_config": null,
106
+ "ensure_weight_tying": false,
107
+ "eva_config": null,
108
+ "exclude_modules": null,
109
+ "fan_in_fan_out": false,
110
+ "inference_mode": true,
111
+ "init_lora_weights": true,
112
+ "layer_replication": null,
113
+ "layers_pattern": null,
114
+ "layers_to_transform": null,
115
+ "loftq_config": {},
116
+ "lora_alpha": 16,
117
+ "lora_bias": false,
118
+ "lora_dropout": 0.1,
119
+ "megatron_config": null,
120
+ "megatron_core": "megatron.core",
121
+ "modules_to_save": null,
122
+ "peft_type": "LORA",
123
+ "peft_version": "0.18.1",
124
+ "qalora_group_size": 16,
125
+ "r": 8,
126
+ "rank_pattern": {
127
+ "model.decoder.layers.0.encoder_attn.k_proj": 64,
128
+ "model.decoder.layers.0.encoder_attn.out_proj": 64,
129
+ "model.decoder.layers.0.encoder_attn.q_proj": 64,
130
+ "model.decoder.layers.0.encoder_attn.v_proj": 64,
131
+ "model.decoder.layers.0.self_attn.k_proj": 32,
132
+ "model.decoder.layers.0.self_attn.out_proj": 32,
133
+ "model.decoder.layers.0.self_attn.q_proj": 32,
134
+ "model.decoder.layers.0.self_attn.v_proj": 32,
135
+ "model.decoder.layers.1.encoder_attn.k_proj": 64,
136
+ "model.decoder.layers.1.encoder_attn.out_proj": 64,
137
+ "model.decoder.layers.1.encoder_attn.q_proj": 64,
138
+ "model.decoder.layers.1.encoder_attn.v_proj": 64,
139
+ "model.decoder.layers.1.self_attn.k_proj": 32,
140
+ "model.decoder.layers.1.self_attn.out_proj": 32,
141
+ "model.decoder.layers.1.self_attn.q_proj": 32,
142
+ "model.decoder.layers.1.self_attn.v_proj": 32,
143
+ "model.decoder.layers.10.encoder_attn.k_proj": 64,
144
+ "model.decoder.layers.10.encoder_attn.out_proj": 64,
145
+ "model.decoder.layers.10.encoder_attn.q_proj": 64,
146
+ "model.decoder.layers.10.encoder_attn.v_proj": 64,
147
+ "model.decoder.layers.10.self_attn.k_proj": 32,
148
+ "model.decoder.layers.10.self_attn.out_proj": 32,
149
+ "model.decoder.layers.10.self_attn.q_proj": 32,
150
+ "model.decoder.layers.10.self_attn.v_proj": 32,
151
+ "model.decoder.layers.11.encoder_attn.k_proj": 64,
152
+ "model.decoder.layers.11.encoder_attn.out_proj": 64,
153
+ "model.decoder.layers.11.encoder_attn.q_proj": 64,
154
+ "model.decoder.layers.11.encoder_attn.v_proj": 64,
155
+ "model.decoder.layers.11.self_attn.k_proj": 32,
156
+ "model.decoder.layers.11.self_attn.out_proj": 32,
157
+ "model.decoder.layers.11.self_attn.q_proj": 32,
158
+ "model.decoder.layers.11.self_attn.v_proj": 32,
159
+ "model.decoder.layers.2.encoder_attn.k_proj": 64,
160
+ "model.decoder.layers.2.encoder_attn.out_proj": 64,
161
+ "model.decoder.layers.2.encoder_attn.q_proj": 64,
162
+ "model.decoder.layers.2.encoder_attn.v_proj": 64,
163
+ "model.decoder.layers.2.self_attn.k_proj": 32,
164
+ "model.decoder.layers.2.self_attn.out_proj": 32,
165
+ "model.decoder.layers.2.self_attn.q_proj": 32,
166
+ "model.decoder.layers.2.self_attn.v_proj": 32,
167
+ "model.decoder.layers.3.encoder_attn.k_proj": 64,
168
+ "model.decoder.layers.3.encoder_attn.out_proj": 64,
169
+ "model.decoder.layers.3.encoder_attn.q_proj": 64,
170
+ "model.decoder.layers.3.encoder_attn.v_proj": 64,
171
+ "model.decoder.layers.3.self_attn.k_proj": 32,
172
+ "model.decoder.layers.3.self_attn.out_proj": 32,
173
+ "model.decoder.layers.3.self_attn.q_proj": 32,
174
+ "model.decoder.layers.3.self_attn.v_proj": 32,
175
+ "model.decoder.layers.4.encoder_attn.k_proj": 64,
176
+ "model.decoder.layers.4.encoder_attn.out_proj": 64,
177
+ "model.decoder.layers.4.encoder_attn.q_proj": 64,
178
+ "model.decoder.layers.4.encoder_attn.v_proj": 64,
179
+ "model.decoder.layers.4.self_attn.k_proj": 32,
180
+ "model.decoder.layers.4.self_attn.out_proj": 32,
181
+ "model.decoder.layers.4.self_attn.q_proj": 32,
182
+ "model.decoder.layers.4.self_attn.v_proj": 32,
183
+ "model.decoder.layers.5.encoder_attn.k_proj": 64,
184
+ "model.decoder.layers.5.encoder_attn.out_proj": 64,
185
+ "model.decoder.layers.5.encoder_attn.q_proj": 64,
186
+ "model.decoder.layers.5.encoder_attn.v_proj": 64,
187
+ "model.decoder.layers.5.self_attn.k_proj": 32,
188
+ "model.decoder.layers.5.self_attn.out_proj": 32,
189
+ "model.decoder.layers.5.self_attn.q_proj": 32,
190
+ "model.decoder.layers.5.self_attn.v_proj": 32,
191
+ "model.decoder.layers.6.encoder_attn.k_proj": 64,
192
+ "model.decoder.layers.6.encoder_attn.out_proj": 64,
193
+ "model.decoder.layers.6.encoder_attn.q_proj": 64,
194
+ "model.decoder.layers.6.encoder_attn.v_proj": 64,
195
+ "model.decoder.layers.6.self_attn.k_proj": 32,
196
+ "model.decoder.layers.6.self_attn.out_proj": 32,
197
+ "model.decoder.layers.6.self_attn.q_proj": 32,
198
+ "model.decoder.layers.6.self_attn.v_proj": 32,
199
+ "model.decoder.layers.7.encoder_attn.k_proj": 64,
200
+ "model.decoder.layers.7.encoder_attn.out_proj": 64,
201
+ "model.decoder.layers.7.encoder_attn.q_proj": 64,
202
+ "model.decoder.layers.7.encoder_attn.v_proj": 64,
203
+ "model.decoder.layers.7.self_attn.k_proj": 32,
204
+ "model.decoder.layers.7.self_attn.out_proj": 32,
205
+ "model.decoder.layers.7.self_attn.q_proj": 32,
206
+ "model.decoder.layers.7.self_attn.v_proj": 32,
207
+ "model.decoder.layers.8.encoder_attn.k_proj": 64,
208
+ "model.decoder.layers.8.encoder_attn.out_proj": 64,
209
+ "model.decoder.layers.8.encoder_attn.q_proj": 64,
210
+ "model.decoder.layers.8.encoder_attn.v_proj": 64,
211
+ "model.decoder.layers.8.self_attn.k_proj": 32,
212
+ "model.decoder.layers.8.self_attn.out_proj": 32,
213
+ "model.decoder.layers.8.self_attn.q_proj": 32,
214
+ "model.decoder.layers.8.self_attn.v_proj": 32,
215
+ "model.decoder.layers.9.encoder_attn.k_proj": 64,
216
+ "model.decoder.layers.9.encoder_attn.out_proj": 64,
217
+ "model.decoder.layers.9.encoder_attn.q_proj": 64,
218
+ "model.decoder.layers.9.encoder_attn.v_proj": 64,
219
+ "model.decoder.layers.9.self_attn.k_proj": 32,
220
+ "model.decoder.layers.9.self_attn.out_proj": 32,
221
+ "model.decoder.layers.9.self_attn.q_proj": 32,
222
+ "model.decoder.layers.9.self_attn.v_proj": 32
223
+ },
224
+ "revision": null,
225
+ "target_modules": [
226
+ "q_proj",
227
+ "v_proj",
228
+ "out_proj",
229
+ "k_proj"
230
+ ],
231
+ "target_parameters": null,
232
+ "task_type": "SEQ_2_SEQ_LM",
233
+ "trainable_token_indices": null,
234
+ "use_dora": false,
235
+ "use_qalora": false,
236
+ "use_rslora": false
237
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e1cc8a85a0283346f7d8aa04c1856816217c932f254c7c54980e97719c5a24c
3
+ size 40935672
all_results.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 10.0,
3
+ "eval_avg_bleu": 20.0189,
4
+ "eval_gen_len": 27.1703,
5
+ "eval_loss": 3.1360061168670654,
6
+ "eval_runtime": 3043.4197,
7
+ "eval_samples": 10370,
8
+ "eval_samples_per_second": 3.407,
9
+ "eval_steps_per_second": 0.426,
10
+ "total_flos": 1.2426966463650202e+17,
11
+ "train_loss": 5.026907345643823,
12
+ "train_runtime": 19932.1348,
13
+ "train_samples": 98002,
14
+ "train_samples_per_second": 49.168,
15
+ "train_steps_per_second": 1.537
16
+ }
checkpoint-30630/README.md ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: facebook/nllb-200-distilled-600M
3
+ library_name: peft
4
+ tags:
5
+ - base_model:adapter:facebook/nllb-200-distilled-600M
6
+ - lora
7
+ - transformers
8
+ ---
9
+
10
+ # Model Card for Model ID
11
+
12
+ <!-- Provide a quick summary of what the model is/does. -->
13
+
14
+
15
+
16
+ ## Model Details
17
+
18
+ ### Model Description
19
+
20
+ <!-- Provide a longer summary of what this model is. -->
21
+
22
+
23
+
24
+ - **Developed by:** [More Information Needed]
25
+ - **Funded by [optional]:** [More Information Needed]
26
+ - **Shared by [optional]:** [More Information Needed]
27
+ - **Model type:** [More Information Needed]
28
+ - **Language(s) (NLP):** [More Information Needed]
29
+ - **License:** [More Information Needed]
30
+ - **Finetuned from model [optional]:** [More Information Needed]
31
+
32
+ ### Model Sources [optional]
33
+
34
+ <!-- Provide the basic links for the model. -->
35
+
36
+ - **Repository:** [More Information Needed]
37
+ - **Paper [optional]:** [More Information Needed]
38
+ - **Demo [optional]:** [More Information Needed]
39
+
40
+ ## Uses
41
+
42
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
43
+
44
+ ### Direct Use
45
+
46
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
47
+
48
+ [More Information Needed]
49
+
50
+ ### Downstream Use [optional]
51
+
52
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
53
+
54
+ [More Information Needed]
55
+
56
+ ### Out-of-Scope Use
57
+
58
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
59
+
60
+ [More Information Needed]
61
+
62
+ ## Bias, Risks, and Limitations
63
+
64
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
65
+
66
+ [More Information Needed]
67
+
68
+ ### Recommendations
69
+
70
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
71
+
72
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
73
+
74
+ ## How to Get Started with the Model
75
+
76
+ Use the code below to get started with the model.
77
+
78
+ [More Information Needed]
79
+
80
+ ## Training Details
81
+
82
+ ### Training Data
83
+
84
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
85
+
86
+ [More Information Needed]
87
+
88
+ ### Training Procedure
89
+
90
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
91
+
92
+ #### Preprocessing [optional]
93
+
94
+ [More Information Needed]
95
+
96
+
97
+ #### Training Hyperparameters
98
+
99
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
100
+
101
+ #### Speeds, Sizes, Times [optional]
102
+
103
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
104
+
105
+ [More Information Needed]
106
+
107
+ ## Evaluation
108
+
109
+ <!-- This section describes the evaluation protocols and provides the results. -->
110
+
111
+ ### Testing Data, Factors & Metrics
112
+
113
+ #### Testing Data
114
+
115
+ <!-- This should link to a Dataset Card if possible. -->
116
+
117
+ [More Information Needed]
118
+
119
+ #### Factors
120
+
121
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
122
+
123
+ [More Information Needed]
124
+
125
+ #### Metrics
126
+
127
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
128
+
129
+ [More Information Needed]
130
+
131
+ ### Results
132
+
133
+ [More Information Needed]
134
+
135
+ #### Summary
136
+
137
+
138
+
139
+ ## Model Examination [optional]
140
+
141
+ <!-- Relevant interpretability work for the model goes here -->
142
+
143
+ [More Information Needed]
144
+
145
+ ## Environmental Impact
146
+
147
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
148
+
149
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
150
+
151
+ - **Hardware Type:** [More Information Needed]
152
+ - **Hours used:** [More Information Needed]
153
+ - **Cloud Provider:** [More Information Needed]
154
+ - **Compute Region:** [More Information Needed]
155
+ - **Carbon Emitted:** [More Information Needed]
156
+
157
+ ## Technical Specifications [optional]
158
+
159
+ ### Model Architecture and Objective
160
+
161
+ [More Information Needed]
162
+
163
+ ### Compute Infrastructure
164
+
165
+ [More Information Needed]
166
+
167
+ #### Hardware
168
+
169
+ [More Information Needed]
170
+
171
+ #### Software
172
+
173
+ [More Information Needed]
174
+
175
+ ## Citation [optional]
176
+
177
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
178
+
179
+ **BibTeX:**
180
+
181
+ [More Information Needed]
182
+
183
+ **APA:**
184
+
185
+ [More Information Needed]
186
+
187
+ ## Glossary [optional]
188
+
189
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
190
+
191
+ [More Information Needed]
192
+
193
+ ## More Information [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Authors [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Contact
202
+
203
+ [More Information Needed]
204
+ ### Framework versions
205
+
206
+ - PEFT 0.18.1
checkpoint-30630/adapter_config.json ADDED
@@ -0,0 +1,237 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {
4
+ "model.decoder.layers.0.encoder_attn.k_proj": 128,
5
+ "model.decoder.layers.0.encoder_attn.out_proj": 128,
6
+ "model.decoder.layers.0.encoder_attn.q_proj": 128,
7
+ "model.decoder.layers.0.encoder_attn.v_proj": 128,
8
+ "model.decoder.layers.0.self_attn.k_proj": 64,
9
+ "model.decoder.layers.0.self_attn.out_proj": 64,
10
+ "model.decoder.layers.0.self_attn.q_proj": 64,
11
+ "model.decoder.layers.0.self_attn.v_proj": 64,
12
+ "model.decoder.layers.1.encoder_attn.k_proj": 128,
13
+ "model.decoder.layers.1.encoder_attn.out_proj": 128,
14
+ "model.decoder.layers.1.encoder_attn.q_proj": 128,
15
+ "model.decoder.layers.1.encoder_attn.v_proj": 128,
16
+ "model.decoder.layers.1.self_attn.k_proj": 64,
17
+ "model.decoder.layers.1.self_attn.out_proj": 64,
18
+ "model.decoder.layers.1.self_attn.q_proj": 64,
19
+ "model.decoder.layers.1.self_attn.v_proj": 64,
20
+ "model.decoder.layers.10.encoder_attn.k_proj": 128,
21
+ "model.decoder.layers.10.encoder_attn.out_proj": 128,
22
+ "model.decoder.layers.10.encoder_attn.q_proj": 128,
23
+ "model.decoder.layers.10.encoder_attn.v_proj": 128,
24
+ "model.decoder.layers.10.self_attn.k_proj": 64,
25
+ "model.decoder.layers.10.self_attn.out_proj": 64,
26
+ "model.decoder.layers.10.self_attn.q_proj": 64,
27
+ "model.decoder.layers.10.self_attn.v_proj": 64,
28
+ "model.decoder.layers.11.encoder_attn.k_proj": 128,
29
+ "model.decoder.layers.11.encoder_attn.out_proj": 128,
30
+ "model.decoder.layers.11.encoder_attn.q_proj": 128,
31
+ "model.decoder.layers.11.encoder_attn.v_proj": 128,
32
+ "model.decoder.layers.11.self_attn.k_proj": 64,
33
+ "model.decoder.layers.11.self_attn.out_proj": 64,
34
+ "model.decoder.layers.11.self_attn.q_proj": 64,
35
+ "model.decoder.layers.11.self_attn.v_proj": 64,
36
+ "model.decoder.layers.2.encoder_attn.k_proj": 128,
37
+ "model.decoder.layers.2.encoder_attn.out_proj": 128,
38
+ "model.decoder.layers.2.encoder_attn.q_proj": 128,
39
+ "model.decoder.layers.2.encoder_attn.v_proj": 128,
40
+ "model.decoder.layers.2.self_attn.k_proj": 64,
41
+ "model.decoder.layers.2.self_attn.out_proj": 64,
42
+ "model.decoder.layers.2.self_attn.q_proj": 64,
43
+ "model.decoder.layers.2.self_attn.v_proj": 64,
44
+ "model.decoder.layers.3.encoder_attn.k_proj": 128,
45
+ "model.decoder.layers.3.encoder_attn.out_proj": 128,
46
+ "model.decoder.layers.3.encoder_attn.q_proj": 128,
47
+ "model.decoder.layers.3.encoder_attn.v_proj": 128,
48
+ "model.decoder.layers.3.self_attn.k_proj": 64,
49
+ "model.decoder.layers.3.self_attn.out_proj": 64,
50
+ "model.decoder.layers.3.self_attn.q_proj": 64,
51
+ "model.decoder.layers.3.self_attn.v_proj": 64,
52
+ "model.decoder.layers.4.encoder_attn.k_proj": 128,
53
+ "model.decoder.layers.4.encoder_attn.out_proj": 128,
54
+ "model.decoder.layers.4.encoder_attn.q_proj": 128,
55
+ "model.decoder.layers.4.encoder_attn.v_proj": 128,
56
+ "model.decoder.layers.4.self_attn.k_proj": 64,
57
+ "model.decoder.layers.4.self_attn.out_proj": 64,
58
+ "model.decoder.layers.4.self_attn.q_proj": 64,
59
+ "model.decoder.layers.4.self_attn.v_proj": 64,
60
+ "model.decoder.layers.5.encoder_attn.k_proj": 128,
61
+ "model.decoder.layers.5.encoder_attn.out_proj": 128,
62
+ "model.decoder.layers.5.encoder_attn.q_proj": 128,
63
+ "model.decoder.layers.5.encoder_attn.v_proj": 128,
64
+ "model.decoder.layers.5.self_attn.k_proj": 64,
65
+ "model.decoder.layers.5.self_attn.out_proj": 64,
66
+ "model.decoder.layers.5.self_attn.q_proj": 64,
67
+ "model.decoder.layers.5.self_attn.v_proj": 64,
68
+ "model.decoder.layers.6.encoder_attn.k_proj": 128,
69
+ "model.decoder.layers.6.encoder_attn.out_proj": 128,
70
+ "model.decoder.layers.6.encoder_attn.q_proj": 128,
71
+ "model.decoder.layers.6.encoder_attn.v_proj": 128,
72
+ "model.decoder.layers.6.self_attn.k_proj": 64,
73
+ "model.decoder.layers.6.self_attn.out_proj": 64,
74
+ "model.decoder.layers.6.self_attn.q_proj": 64,
75
+ "model.decoder.layers.6.self_attn.v_proj": 64,
76
+ "model.decoder.layers.7.encoder_attn.k_proj": 128,
77
+ "model.decoder.layers.7.encoder_attn.out_proj": 128,
78
+ "model.decoder.layers.7.encoder_attn.q_proj": 128,
79
+ "model.decoder.layers.7.encoder_attn.v_proj": 128,
80
+ "model.decoder.layers.7.self_attn.k_proj": 64,
81
+ "model.decoder.layers.7.self_attn.out_proj": 64,
82
+ "model.decoder.layers.7.self_attn.q_proj": 64,
83
+ "model.decoder.layers.7.self_attn.v_proj": 64,
84
+ "model.decoder.layers.8.encoder_attn.k_proj": 128,
85
+ "model.decoder.layers.8.encoder_attn.out_proj": 128,
86
+ "model.decoder.layers.8.encoder_attn.q_proj": 128,
87
+ "model.decoder.layers.8.encoder_attn.v_proj": 128,
88
+ "model.decoder.layers.8.self_attn.k_proj": 64,
89
+ "model.decoder.layers.8.self_attn.out_proj": 64,
90
+ "model.decoder.layers.8.self_attn.q_proj": 64,
91
+ "model.decoder.layers.8.self_attn.v_proj": 64,
92
+ "model.decoder.layers.9.encoder_attn.k_proj": 128,
93
+ "model.decoder.layers.9.encoder_attn.out_proj": 128,
94
+ "model.decoder.layers.9.encoder_attn.q_proj": 128,
95
+ "model.decoder.layers.9.encoder_attn.v_proj": 128,
96
+ "model.decoder.layers.9.self_attn.k_proj": 64,
97
+ "model.decoder.layers.9.self_attn.out_proj": 64,
98
+ "model.decoder.layers.9.self_attn.q_proj": 64,
99
+ "model.decoder.layers.9.self_attn.v_proj": 64
100
+ },
101
+ "arrow_config": null,
102
+ "auto_mapping": null,
103
+ "base_model_name_or_path": "facebook/nllb-200-distilled-600M",
104
+ "bias": "none",
105
+ "corda_config": null,
106
+ "ensure_weight_tying": false,
107
+ "eva_config": null,
108
+ "exclude_modules": null,
109
+ "fan_in_fan_out": false,
110
+ "inference_mode": true,
111
+ "init_lora_weights": true,
112
+ "layer_replication": null,
113
+ "layers_pattern": null,
114
+ "layers_to_transform": null,
115
+ "loftq_config": {},
116
+ "lora_alpha": 16,
117
+ "lora_bias": false,
118
+ "lora_dropout": 0.1,
119
+ "megatron_config": null,
120
+ "megatron_core": "megatron.core",
121
+ "modules_to_save": null,
122
+ "peft_type": "LORA",
123
+ "peft_version": "0.18.1",
124
+ "qalora_group_size": 16,
125
+ "r": 8,
126
+ "rank_pattern": {
127
+ "model.decoder.layers.0.encoder_attn.k_proj": 64,
128
+ "model.decoder.layers.0.encoder_attn.out_proj": 64,
129
+ "model.decoder.layers.0.encoder_attn.q_proj": 64,
130
+ "model.decoder.layers.0.encoder_attn.v_proj": 64,
131
+ "model.decoder.layers.0.self_attn.k_proj": 32,
132
+ "model.decoder.layers.0.self_attn.out_proj": 32,
133
+ "model.decoder.layers.0.self_attn.q_proj": 32,
134
+ "model.decoder.layers.0.self_attn.v_proj": 32,
135
+ "model.decoder.layers.1.encoder_attn.k_proj": 64,
136
+ "model.decoder.layers.1.encoder_attn.out_proj": 64,
137
+ "model.decoder.layers.1.encoder_attn.q_proj": 64,
138
+ "model.decoder.layers.1.encoder_attn.v_proj": 64,
139
+ "model.decoder.layers.1.self_attn.k_proj": 32,
140
+ "model.decoder.layers.1.self_attn.out_proj": 32,
141
+ "model.decoder.layers.1.self_attn.q_proj": 32,
142
+ "model.decoder.layers.1.self_attn.v_proj": 32,
143
+ "model.decoder.layers.10.encoder_attn.k_proj": 64,
144
+ "model.decoder.layers.10.encoder_attn.out_proj": 64,
145
+ "model.decoder.layers.10.encoder_attn.q_proj": 64,
146
+ "model.decoder.layers.10.encoder_attn.v_proj": 64,
147
+ "model.decoder.layers.10.self_attn.k_proj": 32,
148
+ "model.decoder.layers.10.self_attn.out_proj": 32,
149
+ "model.decoder.layers.10.self_attn.q_proj": 32,
150
+ "model.decoder.layers.10.self_attn.v_proj": 32,
151
+ "model.decoder.layers.11.encoder_attn.k_proj": 64,
152
+ "model.decoder.layers.11.encoder_attn.out_proj": 64,
153
+ "model.decoder.layers.11.encoder_attn.q_proj": 64,
154
+ "model.decoder.layers.11.encoder_attn.v_proj": 64,
155
+ "model.decoder.layers.11.self_attn.k_proj": 32,
156
+ "model.decoder.layers.11.self_attn.out_proj": 32,
157
+ "model.decoder.layers.11.self_attn.q_proj": 32,
158
+ "model.decoder.layers.11.self_attn.v_proj": 32,
159
+ "model.decoder.layers.2.encoder_attn.k_proj": 64,
160
+ "model.decoder.layers.2.encoder_attn.out_proj": 64,
161
+ "model.decoder.layers.2.encoder_attn.q_proj": 64,
162
+ "model.decoder.layers.2.encoder_attn.v_proj": 64,
163
+ "model.decoder.layers.2.self_attn.k_proj": 32,
164
+ "model.decoder.layers.2.self_attn.out_proj": 32,
165
+ "model.decoder.layers.2.self_attn.q_proj": 32,
166
+ "model.decoder.layers.2.self_attn.v_proj": 32,
167
+ "model.decoder.layers.3.encoder_attn.k_proj": 64,
168
+ "model.decoder.layers.3.encoder_attn.out_proj": 64,
169
+ "model.decoder.layers.3.encoder_attn.q_proj": 64,
170
+ "model.decoder.layers.3.encoder_attn.v_proj": 64,
171
+ "model.decoder.layers.3.self_attn.k_proj": 32,
172
+ "model.decoder.layers.3.self_attn.out_proj": 32,
173
+ "model.decoder.layers.3.self_attn.q_proj": 32,
174
+ "model.decoder.layers.3.self_attn.v_proj": 32,
175
+ "model.decoder.layers.4.encoder_attn.k_proj": 64,
176
+ "model.decoder.layers.4.encoder_attn.out_proj": 64,
177
+ "model.decoder.layers.4.encoder_attn.q_proj": 64,
178
+ "model.decoder.layers.4.encoder_attn.v_proj": 64,
179
+ "model.decoder.layers.4.self_attn.k_proj": 32,
180
+ "model.decoder.layers.4.self_attn.out_proj": 32,
181
+ "model.decoder.layers.4.self_attn.q_proj": 32,
182
+ "model.decoder.layers.4.self_attn.v_proj": 32,
183
+ "model.decoder.layers.5.encoder_attn.k_proj": 64,
184
+ "model.decoder.layers.5.encoder_attn.out_proj": 64,
185
+ "model.decoder.layers.5.encoder_attn.q_proj": 64,
186
+ "model.decoder.layers.5.encoder_attn.v_proj": 64,
187
+ "model.decoder.layers.5.self_attn.k_proj": 32,
188
+ "model.decoder.layers.5.self_attn.out_proj": 32,
189
+ "model.decoder.layers.5.self_attn.q_proj": 32,
190
+ "model.decoder.layers.5.self_attn.v_proj": 32,
191
+ "model.decoder.layers.6.encoder_attn.k_proj": 64,
192
+ "model.decoder.layers.6.encoder_attn.out_proj": 64,
193
+ "model.decoder.layers.6.encoder_attn.q_proj": 64,
194
+ "model.decoder.layers.6.encoder_attn.v_proj": 64,
195
+ "model.decoder.layers.6.self_attn.k_proj": 32,
196
+ "model.decoder.layers.6.self_attn.out_proj": 32,
197
+ "model.decoder.layers.6.self_attn.q_proj": 32,
198
+ "model.decoder.layers.6.self_attn.v_proj": 32,
199
+ "model.decoder.layers.7.encoder_attn.k_proj": 64,
200
+ "model.decoder.layers.7.encoder_attn.out_proj": 64,
201
+ "model.decoder.layers.7.encoder_attn.q_proj": 64,
202
+ "model.decoder.layers.7.encoder_attn.v_proj": 64,
203
+ "model.decoder.layers.7.self_attn.k_proj": 32,
204
+ "model.decoder.layers.7.self_attn.out_proj": 32,
205
+ "model.decoder.layers.7.self_attn.q_proj": 32,
206
+ "model.decoder.layers.7.self_attn.v_proj": 32,
207
+ "model.decoder.layers.8.encoder_attn.k_proj": 64,
208
+ "model.decoder.layers.8.encoder_attn.out_proj": 64,
209
+ "model.decoder.layers.8.encoder_attn.q_proj": 64,
210
+ "model.decoder.layers.8.encoder_attn.v_proj": 64,
211
+ "model.decoder.layers.8.self_attn.k_proj": 32,
212
+ "model.decoder.layers.8.self_attn.out_proj": 32,
213
+ "model.decoder.layers.8.self_attn.q_proj": 32,
214
+ "model.decoder.layers.8.self_attn.v_proj": 32,
215
+ "model.decoder.layers.9.encoder_attn.k_proj": 64,
216
+ "model.decoder.layers.9.encoder_attn.out_proj": 64,
217
+ "model.decoder.layers.9.encoder_attn.q_proj": 64,
218
+ "model.decoder.layers.9.encoder_attn.v_proj": 64,
219
+ "model.decoder.layers.9.self_attn.k_proj": 32,
220
+ "model.decoder.layers.9.self_attn.out_proj": 32,
221
+ "model.decoder.layers.9.self_attn.q_proj": 32,
222
+ "model.decoder.layers.9.self_attn.v_proj": 32
223
+ },
224
+ "revision": null,
225
+ "target_modules": [
226
+ "q_proj",
227
+ "v_proj",
228
+ "out_proj",
229
+ "k_proj"
230
+ ],
231
+ "target_parameters": null,
232
+ "task_type": "SEQ_2_SEQ_LM",
233
+ "trainable_token_indices": null,
234
+ "use_dora": false,
235
+ "use_qalora": false,
236
+ "use_rslora": false
237
+ }
checkpoint-30630/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e1cc8a85a0283346f7d8aa04c1856816217c932f254c7c54980e97719c5a24c
3
+ size 40935672
checkpoint-30630/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d45d1c4b76aeb1c6ef05fcce5cf1813a5441e8eb73b66d6246da690cb69a65a
3
+ size 82035403
checkpoint-30630/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:78c1e839435b55b7c773dae70cca280dd8194ba10696b1b21540d154cf0afb5b
3
+ size 14645
checkpoint-30630/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:649e3804b030ab1f66d55ef8fd90c60a566a4717f1c26cf181257ec1f7a23fe8
3
+ size 1383
checkpoint-30630/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6642c76b7769138267a75426cbd3867ef9491ff080d20e1c416f4a684ea5d72
3
+ size 1465
checkpoint-30630/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3be18cc91c94d4a1d83731ace4dac0b90a7db024edecdeb9fe7d19ec01ce901
3
+ size 32240136
checkpoint-30630/tokenizer_config.json ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<s>",
4
+ "cls_token": "<s>",
5
+ "eos_token": "</s>",
6
+ "extra_special_tokens": [
7
+ "ace_Arab",
8
+ "ace_Latn",
9
+ "acm_Arab",
10
+ "acq_Arab",
11
+ "aeb_Arab",
12
+ "afr_Latn",
13
+ "ajp_Arab",
14
+ "aka_Latn",
15
+ "amh_Ethi",
16
+ "apc_Arab",
17
+ "arb_Arab",
18
+ "ars_Arab",
19
+ "ary_Arab",
20
+ "arz_Arab",
21
+ "asm_Beng",
22
+ "ast_Latn",
23
+ "awa_Deva",
24
+ "ayr_Latn",
25
+ "azb_Arab",
26
+ "azj_Latn",
27
+ "bak_Cyrl",
28
+ "bam_Latn",
29
+ "ban_Latn",
30
+ "bel_Cyrl",
31
+ "bem_Latn",
32
+ "ben_Beng",
33
+ "bho_Deva",
34
+ "bjn_Arab",
35
+ "bjn_Latn",
36
+ "bod_Tibt",
37
+ "bos_Latn",
38
+ "bug_Latn",
39
+ "bul_Cyrl",
40
+ "cat_Latn",
41
+ "ceb_Latn",
42
+ "ces_Latn",
43
+ "cjk_Latn",
44
+ "ckb_Arab",
45
+ "crh_Latn",
46
+ "cym_Latn",
47
+ "dan_Latn",
48
+ "deu_Latn",
49
+ "dik_Latn",
50
+ "dyu_Latn",
51
+ "dzo_Tibt",
52
+ "ell_Grek",
53
+ "eng_Latn",
54
+ "epo_Latn",
55
+ "est_Latn",
56
+ "eus_Latn",
57
+ "ewe_Latn",
58
+ "fao_Latn",
59
+ "pes_Arab",
60
+ "fij_Latn",
61
+ "fin_Latn",
62
+ "fon_Latn",
63
+ "fra_Latn",
64
+ "fur_Latn",
65
+ "fuv_Latn",
66
+ "gla_Latn",
67
+ "gle_Latn",
68
+ "glg_Latn",
69
+ "grn_Latn",
70
+ "guj_Gujr",
71
+ "hat_Latn",
72
+ "hau_Latn",
73
+ "heb_Hebr",
74
+ "hin_Deva",
75
+ "hne_Deva",
76
+ "hrv_Latn",
77
+ "hun_Latn",
78
+ "hye_Armn",
79
+ "ibo_Latn",
80
+ "ilo_Latn",
81
+ "ind_Latn",
82
+ "isl_Latn",
83
+ "ita_Latn",
84
+ "jav_Latn",
85
+ "jpn_Jpan",
86
+ "kab_Latn",
87
+ "kac_Latn",
88
+ "kam_Latn",
89
+ "kan_Knda",
90
+ "kas_Arab",
91
+ "kas_Deva",
92
+ "kat_Geor",
93
+ "knc_Arab",
94
+ "knc_Latn",
95
+ "kaz_Cyrl",
96
+ "kbp_Latn",
97
+ "kea_Latn",
98
+ "khm_Khmr",
99
+ "kik_Latn",
100
+ "kin_Latn",
101
+ "kir_Cyrl",
102
+ "kmb_Latn",
103
+ "kon_Latn",
104
+ "kor_Hang",
105
+ "kmr_Latn",
106
+ "lao_Laoo",
107
+ "lvs_Latn",
108
+ "lij_Latn",
109
+ "lim_Latn",
110
+ "lin_Latn",
111
+ "lit_Latn",
112
+ "lmo_Latn",
113
+ "ltg_Latn",
114
+ "ltz_Latn",
115
+ "lua_Latn",
116
+ "lug_Latn",
117
+ "luo_Latn",
118
+ "lus_Latn",
119
+ "mag_Deva",
120
+ "mai_Deva",
121
+ "mal_Mlym",
122
+ "mar_Deva",
123
+ "min_Latn",
124
+ "mkd_Cyrl",
125
+ "plt_Latn",
126
+ "mlt_Latn",
127
+ "mni_Beng",
128
+ "khk_Cyrl",
129
+ "mos_Latn",
130
+ "mri_Latn",
131
+ "zsm_Latn",
132
+ "mya_Mymr",
133
+ "nld_Latn",
134
+ "nno_Latn",
135
+ "nob_Latn",
136
+ "npi_Deva",
137
+ "nso_Latn",
138
+ "nus_Latn",
139
+ "nya_Latn",
140
+ "oci_Latn",
141
+ "gaz_Latn",
142
+ "ory_Orya",
143
+ "pag_Latn",
144
+ "pan_Guru",
145
+ "pap_Latn",
146
+ "pol_Latn",
147
+ "por_Latn",
148
+ "prs_Arab",
149
+ "pbt_Arab",
150
+ "quy_Latn",
151
+ "ron_Latn",
152
+ "run_Latn",
153
+ "rus_Cyrl",
154
+ "sag_Latn",
155
+ "san_Deva",
156
+ "sat_Beng",
157
+ "scn_Latn",
158
+ "shn_Mymr",
159
+ "sin_Sinh",
160
+ "slk_Latn",
161
+ "slv_Latn",
162
+ "smo_Latn",
163
+ "sna_Latn",
164
+ "snd_Arab",
165
+ "som_Latn",
166
+ "sot_Latn",
167
+ "spa_Latn",
168
+ "als_Latn",
169
+ "srd_Latn",
170
+ "srp_Cyrl",
171
+ "ssw_Latn",
172
+ "sun_Latn",
173
+ "swe_Latn",
174
+ "swh_Latn",
175
+ "szl_Latn",
176
+ "tam_Taml",
177
+ "tat_Cyrl",
178
+ "tel_Telu",
179
+ "tgk_Cyrl",
180
+ "tgl_Latn",
181
+ "tha_Thai",
182
+ "tir_Ethi",
183
+ "taq_Latn",
184
+ "taq_Tfng",
185
+ "tpi_Latn",
186
+ "tsn_Latn",
187
+ "tso_Latn",
188
+ "tuk_Latn",
189
+ "tum_Latn",
190
+ "tur_Latn",
191
+ "twi_Latn",
192
+ "tzm_Tfng",
193
+ "uig_Arab",
194
+ "ukr_Cyrl",
195
+ "umb_Latn",
196
+ "urd_Arab",
197
+ "uzn_Latn",
198
+ "vec_Latn",
199
+ "vie_Latn",
200
+ "war_Latn",
201
+ "wol_Latn",
202
+ "xho_Latn",
203
+ "ydd_Hebr",
204
+ "yor_Latn",
205
+ "yue_Hant",
206
+ "zho_Hans",
207
+ "zho_Hant",
208
+ "zul_Latn"
209
+ ],
210
+ "is_local": false,
211
+ "legacy_behaviour": false,
212
+ "mask_token": "<mask>",
213
+ "model_max_length": 1024,
214
+ "pad_token": "<pad>",
215
+ "sep_token": "</s>",
216
+ "sp_model_kwargs": {},
217
+ "src_lang": "eng_Latn",
218
+ "tgt_lang": null,
219
+ "tokenizer_class": "NllbTokenizer",
220
+ "unk_token": "<unk>"
221
+ }
checkpoint-30630/trainer_state.json ADDED
@@ -0,0 +1,461 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 10.0,
6
+ "eval_steps": 500.0,
7
+ "global_step": 30630,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.1632519794302506,
14
+ "grad_norm": 0.42205774784088135,
15
+ "learning_rate": 0.0001996,
16
+ "loss": 9.0337958984375,
17
+ "step": 500
18
+ },
19
+ {
20
+ "epoch": 0.3265039588605012,
21
+ "grad_norm": 0.4745626747608185,
22
+ "learning_rate": 0.00019668768669100565,
23
+ "loss": 7.18489794921875,
24
+ "step": 1000
25
+ },
26
+ {
27
+ "epoch": 0.48975593829075176,
28
+ "grad_norm": 0.5245969295501709,
29
+ "learning_rate": 0.00019336873547958847,
30
+ "loss": 6.76750439453125,
31
+ "step": 1500
32
+ },
33
+ {
34
+ "epoch": 0.6530079177210024,
35
+ "grad_norm": 0.5147088170051575,
36
+ "learning_rate": 0.00019004978426817126,
37
+ "loss": 6.49613671875,
38
+ "step": 2000
39
+ },
40
+ {
41
+ "epoch": 0.8162598971512529,
42
+ "grad_norm": 0.5029264688491821,
43
+ "learning_rate": 0.00018673083305675408,
44
+ "loss": 6.30743798828125,
45
+ "step": 2500
46
+ },
47
+ {
48
+ "epoch": 0.9795118765815035,
49
+ "grad_norm": 0.52044677734375,
50
+ "learning_rate": 0.00018341188184533688,
51
+ "loss": 6.20166357421875,
52
+ "step": 3000
53
+ },
54
+ {
55
+ "epoch": 1.142682230022039,
56
+ "grad_norm": 0.5113121867179871,
57
+ "learning_rate": 0.0001800929306339197,
58
+ "loss": 5.97177001953125,
59
+ "step": 3500
60
+ },
61
+ {
62
+ "epoch": 1.3059342094522897,
63
+ "grad_norm": 0.5511940121650696,
64
+ "learning_rate": 0.0001767739794225025,
65
+ "loss": 5.921890625,
66
+ "step": 4000
67
+ },
68
+ {
69
+ "epoch": 1.4691861888825402,
70
+ "grad_norm": 0.4959582984447479,
71
+ "learning_rate": 0.0001734550282110853,
72
+ "loss": 5.78601318359375,
73
+ "step": 4500
74
+ },
75
+ {
76
+ "epoch": 1.6324381683127909,
77
+ "grad_norm": 0.527199387550354,
78
+ "learning_rate": 0.0001701360769996681,
79
+ "loss": 5.76173681640625,
80
+ "step": 5000
81
+ },
82
+ {
83
+ "epoch": 1.7956901477430414,
84
+ "grad_norm": 0.49316641688346863,
85
+ "learning_rate": 0.00016681712578825092,
86
+ "loss": 5.67507861328125,
87
+ "step": 5500
88
+ },
89
+ {
90
+ "epoch": 1.9589421271732919,
91
+ "grad_norm": 0.4699156880378723,
92
+ "learning_rate": 0.00016349817457683371,
93
+ "loss": 5.567580078125,
94
+ "step": 6000
95
+ },
96
+ {
97
+ "epoch": 2.1221124806138274,
98
+ "grad_norm": 0.5287707448005676,
99
+ "learning_rate": 0.00016017922336541653,
100
+ "loss": 5.44504248046875,
101
+ "step": 6500
102
+ },
103
+ {
104
+ "epoch": 2.285364460044078,
105
+ "grad_norm": 1.1574363708496094,
106
+ "learning_rate": 0.00015686027215399935,
107
+ "loss": 5.31701904296875,
108
+ "step": 7000
109
+ },
110
+ {
111
+ "epoch": 2.4486164394743284,
112
+ "grad_norm": 0.45680922269821167,
113
+ "learning_rate": 0.00015354132094258215,
114
+ "loss": 5.35315771484375,
115
+ "step": 7500
116
+ },
117
+ {
118
+ "epoch": 2.6118684189045793,
119
+ "grad_norm": 0.5091222524642944,
120
+ "learning_rate": 0.00015022236973116497,
121
+ "loss": 5.32297998046875,
122
+ "step": 8000
123
+ },
124
+ {
125
+ "epoch": 2.77512039833483,
126
+ "grad_norm": 0.5736968517303467,
127
+ "learning_rate": 0.00014690341851974776,
128
+ "loss": 5.2701904296875,
129
+ "step": 8500
130
+ },
131
+ {
132
+ "epoch": 2.9383723777650803,
133
+ "grad_norm": 0.5231903791427612,
134
+ "learning_rate": 0.00014358446730833058,
135
+ "loss": 5.30401953125,
136
+ "step": 9000
137
+ },
138
+ {
139
+ "epoch": 3.101542731205616,
140
+ "grad_norm": 0.5694177746772766,
141
+ "learning_rate": 0.00014026551609691337,
142
+ "loss": 5.1482998046875,
143
+ "step": 9500
144
+ },
145
+ {
146
+ "epoch": 3.2647947106358663,
147
+ "grad_norm": 0.5935769081115723,
148
+ "learning_rate": 0.0001369465648854962,
149
+ "loss": 5.08368701171875,
150
+ "step": 10000
151
+ },
152
+ {
153
+ "epoch": 3.4280466900661173,
154
+ "grad_norm": 0.6495661735534668,
155
+ "learning_rate": 0.000133627613674079,
156
+ "loss": 5.0398076171875,
157
+ "step": 10500
158
+ },
159
+ {
160
+ "epoch": 3.591298669496368,
161
+ "grad_norm": 0.5465214252471924,
162
+ "learning_rate": 0.0001303086624626618,
163
+ "loss": 5.07168017578125,
164
+ "step": 11000
165
+ },
166
+ {
167
+ "epoch": 3.7545506489266183,
168
+ "grad_norm": 0.5718339681625366,
169
+ "learning_rate": 0.00012698971125124463,
170
+ "loss": 5.0271943359375,
171
+ "step": 11500
172
+ },
173
+ {
174
+ "epoch": 3.9178026283568688,
175
+ "grad_norm": 0.607941746711731,
176
+ "learning_rate": 0.00012367076003982742,
177
+ "loss": 5.05531640625,
178
+ "step": 12000
179
+ },
180
+ {
181
+ "epoch": 4.080972981797404,
182
+ "grad_norm": 0.5361756682395935,
183
+ "learning_rate": 0.00012035180882841021,
184
+ "loss": 4.88087744140625,
185
+ "step": 12500
186
+ },
187
+ {
188
+ "epoch": 4.244224961227655,
189
+ "grad_norm": 0.5884597301483154,
190
+ "learning_rate": 0.00011703285761699303,
191
+ "loss": 4.826615234375,
192
+ "step": 13000
193
+ },
194
+ {
195
+ "epoch": 4.407476940657905,
196
+ "grad_norm": 0.6183493137359619,
197
+ "learning_rate": 0.00011371390640557584,
198
+ "loss": 4.8345087890625,
199
+ "step": 13500
200
+ },
201
+ {
202
+ "epoch": 4.570728920088156,
203
+ "grad_norm": 0.4895070493221283,
204
+ "learning_rate": 0.00011039495519415866,
205
+ "loss": 4.808126953125,
206
+ "step": 14000
207
+ },
208
+ {
209
+ "epoch": 4.733980899518406,
210
+ "grad_norm": 0.5796904563903809,
211
+ "learning_rate": 0.00010707600398274147,
212
+ "loss": 4.8475546875,
213
+ "step": 14500
214
+ },
215
+ {
216
+ "epoch": 4.897232878948657,
217
+ "grad_norm": 0.5432486534118652,
218
+ "learning_rate": 0.00010375705277132426,
219
+ "loss": 4.81570703125,
220
+ "step": 15000
221
+ },
222
+ {
223
+ "epoch": 5.060403232389192,
224
+ "grad_norm": 0.6771642565727234,
225
+ "learning_rate": 0.00010043810155990707,
226
+ "loss": 4.76427978515625,
227
+ "step": 15500
228
+ },
229
+ {
230
+ "epoch": 5.223655211819444,
231
+ "grad_norm": 0.6445333957672119,
232
+ "learning_rate": 9.711915034848989e-05,
233
+ "loss": 4.65180908203125,
234
+ "step": 16000
235
+ },
236
+ {
237
+ "epoch": 5.386907191249694,
238
+ "grad_norm": 0.599590539932251,
239
+ "learning_rate": 9.380019913707268e-05,
240
+ "loss": 4.70791796875,
241
+ "step": 16500
242
+ },
243
+ {
244
+ "epoch": 5.550159170679945,
245
+ "grad_norm": 0.6182076334953308,
246
+ "learning_rate": 9.04812479256555e-05,
247
+ "loss": 4.666,
248
+ "step": 17000
249
+ },
250
+ {
251
+ "epoch": 5.713411150110195,
252
+ "grad_norm": 0.601693332195282,
253
+ "learning_rate": 8.71622967142383e-05,
254
+ "loss": 4.6578359375,
255
+ "step": 17500
256
+ },
257
+ {
258
+ "epoch": 5.876663129540446,
259
+ "grad_norm": 0.6478536128997803,
260
+ "learning_rate": 8.384334550282111e-05,
261
+ "loss": 4.64422509765625,
262
+ "step": 18000
263
+ },
264
+ {
265
+ "epoch": 6.039833482980981,
266
+ "grad_norm": 0.6848897337913513,
267
+ "learning_rate": 8.052439429140392e-05,
268
+ "loss": 4.63602099609375,
269
+ "step": 18500
270
+ },
271
+ {
272
+ "epoch": 6.203085462411232,
273
+ "grad_norm": 0.599690854549408,
274
+ "learning_rate": 7.720544307998673e-05,
275
+ "loss": 4.57735595703125,
276
+ "step": 19000
277
+ },
278
+ {
279
+ "epoch": 6.366337441841482,
280
+ "grad_norm": 0.567436158657074,
281
+ "learning_rate": 7.388649186856955e-05,
282
+ "loss": 4.51408740234375,
283
+ "step": 19500
284
+ },
285
+ {
286
+ "epoch": 6.529589421271733,
287
+ "grad_norm": 0.9956502914428711,
288
+ "learning_rate": 7.056754065715234e-05,
289
+ "loss": 4.52962158203125,
290
+ "step": 20000
291
+ },
292
+ {
293
+ "epoch": 6.692841400701983,
294
+ "grad_norm": 0.6528770923614502,
295
+ "learning_rate": 6.724858944573516e-05,
296
+ "loss": 4.47805078125,
297
+ "step": 20500
298
+ },
299
+ {
300
+ "epoch": 6.856093380132235,
301
+ "grad_norm": 0.5711560249328613,
302
+ "learning_rate": 6.392963823431795e-05,
303
+ "loss": 4.5910791015625,
304
+ "step": 21000
305
+ },
306
+ {
307
+ "epoch": 7.019263733572769,
308
+ "grad_norm": 0.5683479309082031,
309
+ "learning_rate": 6.061068702290077e-05,
310
+ "loss": 4.54874267578125,
311
+ "step": 21500
312
+ },
313
+ {
314
+ "epoch": 7.182515713003021,
315
+ "grad_norm": 0.6191678643226624,
316
+ "learning_rate": 5.729173581148357e-05,
317
+ "loss": 4.418267578125,
318
+ "step": 22000
319
+ },
320
+ {
321
+ "epoch": 7.345767692433271,
322
+ "grad_norm": 0.6200758814811707,
323
+ "learning_rate": 5.3972784600066386e-05,
324
+ "loss": 4.46817431640625,
325
+ "step": 22500
326
+ },
327
+ {
328
+ "epoch": 7.509019671863522,
329
+ "grad_norm": 0.5810480713844299,
330
+ "learning_rate": 5.0653833388649185e-05,
331
+ "loss": 4.4288740234375,
332
+ "step": 23000
333
+ },
334
+ {
335
+ "epoch": 7.672271651293772,
336
+ "grad_norm": 0.6930559277534485,
337
+ "learning_rate": 4.7334882177232e-05,
338
+ "loss": 4.42653369140625,
339
+ "step": 23500
340
+ },
341
+ {
342
+ "epoch": 7.835523630724023,
343
+ "grad_norm": 0.6396164298057556,
344
+ "learning_rate": 4.4015930965814805e-05,
345
+ "loss": 4.49230419921875,
346
+ "step": 24000
347
+ },
348
+ {
349
+ "epoch": 7.998775610154273,
350
+ "grad_norm": 0.5532464981079102,
351
+ "learning_rate": 4.069697975439761e-05,
352
+ "loss": 4.39004931640625,
353
+ "step": 24500
354
+ },
355
+ {
356
+ "epoch": 8.161945963594809,
357
+ "grad_norm": 0.6885871291160583,
358
+ "learning_rate": 3.737802854298042e-05,
359
+ "loss": 4.382765625,
360
+ "step": 25000
361
+ },
362
+ {
363
+ "epoch": 8.32519794302506,
364
+ "grad_norm": 0.6912867426872253,
365
+ "learning_rate": 3.4059077331563225e-05,
366
+ "loss": 4.396109375,
367
+ "step": 25500
368
+ },
369
+ {
370
+ "epoch": 8.48844992245531,
371
+ "grad_norm": 0.6827392578125,
372
+ "learning_rate": 3.074012612014603e-05,
373
+ "loss": 4.40128125,
374
+ "step": 26000
375
+ },
376
+ {
377
+ "epoch": 8.651701901885561,
378
+ "grad_norm": 0.6782070398330688,
379
+ "learning_rate": 2.742117490872884e-05,
380
+ "loss": 4.35358203125,
381
+ "step": 26500
382
+ },
383
+ {
384
+ "epoch": 8.81495388131581,
385
+ "grad_norm": 0.7143053412437439,
386
+ "learning_rate": 2.410222369731165e-05,
387
+ "loss": 4.339625,
388
+ "step": 27000
389
+ },
390
+ {
391
+ "epoch": 8.978205860746062,
392
+ "grad_norm": 0.684012234210968,
393
+ "learning_rate": 2.0783272485894458e-05,
394
+ "loss": 4.3393515625,
395
+ "step": 27500
396
+ },
397
+ {
398
+ "epoch": 9.141376214186597,
399
+ "grad_norm": 0.6413611769676208,
400
+ "learning_rate": 1.7464321274477265e-05,
401
+ "loss": 4.32941748046875,
402
+ "step": 28000
403
+ },
404
+ {
405
+ "epoch": 9.304628193616848,
406
+ "grad_norm": 0.6120012402534485,
407
+ "learning_rate": 1.4145370063060073e-05,
408
+ "loss": 4.31831884765625,
409
+ "step": 28500
410
+ },
411
+ {
412
+ "epoch": 9.467880173047098,
413
+ "grad_norm": 0.6299709677696228,
414
+ "learning_rate": 1.0826418851642881e-05,
415
+ "loss": 4.28150341796875,
416
+ "step": 29000
417
+ },
418
+ {
419
+ "epoch": 9.631132152477349,
420
+ "grad_norm": 0.6235489845275879,
421
+ "learning_rate": 7.5074676402256894e-06,
422
+ "loss": 4.31871484375,
423
+ "step": 29500
424
+ },
425
+ {
426
+ "epoch": 9.794384131907599,
427
+ "grad_norm": 0.7088222503662109,
428
+ "learning_rate": 4.188516428808497e-06,
429
+ "loss": 4.35441064453125,
430
+ "step": 30000
431
+ },
432
+ {
433
+ "epoch": 9.95763611133785,
434
+ "grad_norm": 0.636341392993927,
435
+ "learning_rate": 8.695652173913044e-07,
436
+ "loss": 4.2993779296875,
437
+ "step": 30500
438
+ }
439
+ ],
440
+ "logging_steps": 500,
441
+ "max_steps": 30630,
442
+ "num_input_tokens_seen": 0,
443
+ "num_train_epochs": 10,
444
+ "save_steps": 2000,
445
+ "stateful_callbacks": {
446
+ "TrainerControl": {
447
+ "args": {
448
+ "should_epoch_stop": false,
449
+ "should_evaluate": false,
450
+ "should_log": false,
451
+ "should_save": true,
452
+ "should_training_stop": true
453
+ },
454
+ "attributes": {}
455
+ }
456
+ },
457
+ "total_flos": 1.2426966463650202e+17,
458
+ "train_batch_size": 8,
459
+ "trial_name": null,
460
+ "trial_params": null
461
+ }
checkpoint-30630/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1aef21c170dea1ff74d8ca0080f3f628809ef5bc7ac895de7fc736581881577
3
+ size 5329
eval_results.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 10.0,
3
+ "eval_avg_bleu": 20.0189,
4
+ "eval_gen_len": 27.1703,
5
+ "eval_loss": 3.1360061168670654,
6
+ "eval_runtime": 3043.4197,
7
+ "eval_samples": 10370,
8
+ "eval_samples_per_second": 3.407,
9
+ "eval_steps_per_second": 0.426
10
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3be18cc91c94d4a1d83731ace4dac0b90a7db024edecdeb9fe7d19ec01ce901
3
+ size 32240136
tokenizer_config.json ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<s>",
4
+ "cls_token": "<s>",
5
+ "eos_token": "</s>",
6
+ "extra_special_tokens": [
7
+ "ace_Arab",
8
+ "ace_Latn",
9
+ "acm_Arab",
10
+ "acq_Arab",
11
+ "aeb_Arab",
12
+ "afr_Latn",
13
+ "ajp_Arab",
14
+ "aka_Latn",
15
+ "amh_Ethi",
16
+ "apc_Arab",
17
+ "arb_Arab",
18
+ "ars_Arab",
19
+ "ary_Arab",
20
+ "arz_Arab",
21
+ "asm_Beng",
22
+ "ast_Latn",
23
+ "awa_Deva",
24
+ "ayr_Latn",
25
+ "azb_Arab",
26
+ "azj_Latn",
27
+ "bak_Cyrl",
28
+ "bam_Latn",
29
+ "ban_Latn",
30
+ "bel_Cyrl",
31
+ "bem_Latn",
32
+ "ben_Beng",
33
+ "bho_Deva",
34
+ "bjn_Arab",
35
+ "bjn_Latn",
36
+ "bod_Tibt",
37
+ "bos_Latn",
38
+ "bug_Latn",
39
+ "bul_Cyrl",
40
+ "cat_Latn",
41
+ "ceb_Latn",
42
+ "ces_Latn",
43
+ "cjk_Latn",
44
+ "ckb_Arab",
45
+ "crh_Latn",
46
+ "cym_Latn",
47
+ "dan_Latn",
48
+ "deu_Latn",
49
+ "dik_Latn",
50
+ "dyu_Latn",
51
+ "dzo_Tibt",
52
+ "ell_Grek",
53
+ "eng_Latn",
54
+ "epo_Latn",
55
+ "est_Latn",
56
+ "eus_Latn",
57
+ "ewe_Latn",
58
+ "fao_Latn",
59
+ "pes_Arab",
60
+ "fij_Latn",
61
+ "fin_Latn",
62
+ "fon_Latn",
63
+ "fra_Latn",
64
+ "fur_Latn",
65
+ "fuv_Latn",
66
+ "gla_Latn",
67
+ "gle_Latn",
68
+ "glg_Latn",
69
+ "grn_Latn",
70
+ "guj_Gujr",
71
+ "hat_Latn",
72
+ "hau_Latn",
73
+ "heb_Hebr",
74
+ "hin_Deva",
75
+ "hne_Deva",
76
+ "hrv_Latn",
77
+ "hun_Latn",
78
+ "hye_Armn",
79
+ "ibo_Latn",
80
+ "ilo_Latn",
81
+ "ind_Latn",
82
+ "isl_Latn",
83
+ "ita_Latn",
84
+ "jav_Latn",
85
+ "jpn_Jpan",
86
+ "kab_Latn",
87
+ "kac_Latn",
88
+ "kam_Latn",
89
+ "kan_Knda",
90
+ "kas_Arab",
91
+ "kas_Deva",
92
+ "kat_Geor",
93
+ "knc_Arab",
94
+ "knc_Latn",
95
+ "kaz_Cyrl",
96
+ "kbp_Latn",
97
+ "kea_Latn",
98
+ "khm_Khmr",
99
+ "kik_Latn",
100
+ "kin_Latn",
101
+ "kir_Cyrl",
102
+ "kmb_Latn",
103
+ "kon_Latn",
104
+ "kor_Hang",
105
+ "kmr_Latn",
106
+ "lao_Laoo",
107
+ "lvs_Latn",
108
+ "lij_Latn",
109
+ "lim_Latn",
110
+ "lin_Latn",
111
+ "lit_Latn",
112
+ "lmo_Latn",
113
+ "ltg_Latn",
114
+ "ltz_Latn",
115
+ "lua_Latn",
116
+ "lug_Latn",
117
+ "luo_Latn",
118
+ "lus_Latn",
119
+ "mag_Deva",
120
+ "mai_Deva",
121
+ "mal_Mlym",
122
+ "mar_Deva",
123
+ "min_Latn",
124
+ "mkd_Cyrl",
125
+ "plt_Latn",
126
+ "mlt_Latn",
127
+ "mni_Beng",
128
+ "khk_Cyrl",
129
+ "mos_Latn",
130
+ "mri_Latn",
131
+ "zsm_Latn",
132
+ "mya_Mymr",
133
+ "nld_Latn",
134
+ "nno_Latn",
135
+ "nob_Latn",
136
+ "npi_Deva",
137
+ "nso_Latn",
138
+ "nus_Latn",
139
+ "nya_Latn",
140
+ "oci_Latn",
141
+ "gaz_Latn",
142
+ "ory_Orya",
143
+ "pag_Latn",
144
+ "pan_Guru",
145
+ "pap_Latn",
146
+ "pol_Latn",
147
+ "por_Latn",
148
+ "prs_Arab",
149
+ "pbt_Arab",
150
+ "quy_Latn",
151
+ "ron_Latn",
152
+ "run_Latn",
153
+ "rus_Cyrl",
154
+ "sag_Latn",
155
+ "san_Deva",
156
+ "sat_Beng",
157
+ "scn_Latn",
158
+ "shn_Mymr",
159
+ "sin_Sinh",
160
+ "slk_Latn",
161
+ "slv_Latn",
162
+ "smo_Latn",
163
+ "sna_Latn",
164
+ "snd_Arab",
165
+ "som_Latn",
166
+ "sot_Latn",
167
+ "spa_Latn",
168
+ "als_Latn",
169
+ "srd_Latn",
170
+ "srp_Cyrl",
171
+ "ssw_Latn",
172
+ "sun_Latn",
173
+ "swe_Latn",
174
+ "swh_Latn",
175
+ "szl_Latn",
176
+ "tam_Taml",
177
+ "tat_Cyrl",
178
+ "tel_Telu",
179
+ "tgk_Cyrl",
180
+ "tgl_Latn",
181
+ "tha_Thai",
182
+ "tir_Ethi",
183
+ "taq_Latn",
184
+ "taq_Tfng",
185
+ "tpi_Latn",
186
+ "tsn_Latn",
187
+ "tso_Latn",
188
+ "tuk_Latn",
189
+ "tum_Latn",
190
+ "tur_Latn",
191
+ "twi_Latn",
192
+ "tzm_Tfng",
193
+ "uig_Arab",
194
+ "ukr_Cyrl",
195
+ "umb_Latn",
196
+ "urd_Arab",
197
+ "uzn_Latn",
198
+ "vec_Latn",
199
+ "vie_Latn",
200
+ "war_Latn",
201
+ "wol_Latn",
202
+ "xho_Latn",
203
+ "ydd_Hebr",
204
+ "yor_Latn",
205
+ "yue_Hant",
206
+ "zho_Hans",
207
+ "zho_Hant",
208
+ "zul_Latn"
209
+ ],
210
+ "is_local": false,
211
+ "legacy_behaviour": false,
212
+ "mask_token": "<mask>",
213
+ "model_max_length": 1024,
214
+ "pad_token": "<pad>",
215
+ "sep_token": "</s>",
216
+ "sp_model_kwargs": {},
217
+ "src_lang": "eng_Latn",
218
+ "tgt_lang": null,
219
+ "tokenizer_class": "NllbTokenizer",
220
+ "unk_token": "<unk>"
221
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 10.0,
3
+ "total_flos": 1.2426966463650202e+17,
4
+ "train_loss": 5.026907345643823,
5
+ "train_runtime": 19932.1348,
6
+ "train_samples": 98002,
7
+ "train_samples_per_second": 49.168,
8
+ "train_steps_per_second": 1.537
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,470 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 10.0,
6
+ "eval_steps": 500.0,
7
+ "global_step": 30630,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.1632519794302506,
14
+ "grad_norm": 0.42205774784088135,
15
+ "learning_rate": 0.0001996,
16
+ "loss": 9.0337958984375,
17
+ "step": 500
18
+ },
19
+ {
20
+ "epoch": 0.3265039588605012,
21
+ "grad_norm": 0.4745626747608185,
22
+ "learning_rate": 0.00019668768669100565,
23
+ "loss": 7.18489794921875,
24
+ "step": 1000
25
+ },
26
+ {
27
+ "epoch": 0.48975593829075176,
28
+ "grad_norm": 0.5245969295501709,
29
+ "learning_rate": 0.00019336873547958847,
30
+ "loss": 6.76750439453125,
31
+ "step": 1500
32
+ },
33
+ {
34
+ "epoch": 0.6530079177210024,
35
+ "grad_norm": 0.5147088170051575,
36
+ "learning_rate": 0.00019004978426817126,
37
+ "loss": 6.49613671875,
38
+ "step": 2000
39
+ },
40
+ {
41
+ "epoch": 0.8162598971512529,
42
+ "grad_norm": 0.5029264688491821,
43
+ "learning_rate": 0.00018673083305675408,
44
+ "loss": 6.30743798828125,
45
+ "step": 2500
46
+ },
47
+ {
48
+ "epoch": 0.9795118765815035,
49
+ "grad_norm": 0.52044677734375,
50
+ "learning_rate": 0.00018341188184533688,
51
+ "loss": 6.20166357421875,
52
+ "step": 3000
53
+ },
54
+ {
55
+ "epoch": 1.142682230022039,
56
+ "grad_norm": 0.5113121867179871,
57
+ "learning_rate": 0.0001800929306339197,
58
+ "loss": 5.97177001953125,
59
+ "step": 3500
60
+ },
61
+ {
62
+ "epoch": 1.3059342094522897,
63
+ "grad_norm": 0.5511940121650696,
64
+ "learning_rate": 0.0001767739794225025,
65
+ "loss": 5.921890625,
66
+ "step": 4000
67
+ },
68
+ {
69
+ "epoch": 1.4691861888825402,
70
+ "grad_norm": 0.4959582984447479,
71
+ "learning_rate": 0.0001734550282110853,
72
+ "loss": 5.78601318359375,
73
+ "step": 4500
74
+ },
75
+ {
76
+ "epoch": 1.6324381683127909,
77
+ "grad_norm": 0.527199387550354,
78
+ "learning_rate": 0.0001701360769996681,
79
+ "loss": 5.76173681640625,
80
+ "step": 5000
81
+ },
82
+ {
83
+ "epoch": 1.7956901477430414,
84
+ "grad_norm": 0.49316641688346863,
85
+ "learning_rate": 0.00016681712578825092,
86
+ "loss": 5.67507861328125,
87
+ "step": 5500
88
+ },
89
+ {
90
+ "epoch": 1.9589421271732919,
91
+ "grad_norm": 0.4699156880378723,
92
+ "learning_rate": 0.00016349817457683371,
93
+ "loss": 5.567580078125,
94
+ "step": 6000
95
+ },
96
+ {
97
+ "epoch": 2.1221124806138274,
98
+ "grad_norm": 0.5287707448005676,
99
+ "learning_rate": 0.00016017922336541653,
100
+ "loss": 5.44504248046875,
101
+ "step": 6500
102
+ },
103
+ {
104
+ "epoch": 2.285364460044078,
105
+ "grad_norm": 1.1574363708496094,
106
+ "learning_rate": 0.00015686027215399935,
107
+ "loss": 5.31701904296875,
108
+ "step": 7000
109
+ },
110
+ {
111
+ "epoch": 2.4486164394743284,
112
+ "grad_norm": 0.45680922269821167,
113
+ "learning_rate": 0.00015354132094258215,
114
+ "loss": 5.35315771484375,
115
+ "step": 7500
116
+ },
117
+ {
118
+ "epoch": 2.6118684189045793,
119
+ "grad_norm": 0.5091222524642944,
120
+ "learning_rate": 0.00015022236973116497,
121
+ "loss": 5.32297998046875,
122
+ "step": 8000
123
+ },
124
+ {
125
+ "epoch": 2.77512039833483,
126
+ "grad_norm": 0.5736968517303467,
127
+ "learning_rate": 0.00014690341851974776,
128
+ "loss": 5.2701904296875,
129
+ "step": 8500
130
+ },
131
+ {
132
+ "epoch": 2.9383723777650803,
133
+ "grad_norm": 0.5231903791427612,
134
+ "learning_rate": 0.00014358446730833058,
135
+ "loss": 5.30401953125,
136
+ "step": 9000
137
+ },
138
+ {
139
+ "epoch": 3.101542731205616,
140
+ "grad_norm": 0.5694177746772766,
141
+ "learning_rate": 0.00014026551609691337,
142
+ "loss": 5.1482998046875,
143
+ "step": 9500
144
+ },
145
+ {
146
+ "epoch": 3.2647947106358663,
147
+ "grad_norm": 0.5935769081115723,
148
+ "learning_rate": 0.0001369465648854962,
149
+ "loss": 5.08368701171875,
150
+ "step": 10000
151
+ },
152
+ {
153
+ "epoch": 3.4280466900661173,
154
+ "grad_norm": 0.6495661735534668,
155
+ "learning_rate": 0.000133627613674079,
156
+ "loss": 5.0398076171875,
157
+ "step": 10500
158
+ },
159
+ {
160
+ "epoch": 3.591298669496368,
161
+ "grad_norm": 0.5465214252471924,
162
+ "learning_rate": 0.0001303086624626618,
163
+ "loss": 5.07168017578125,
164
+ "step": 11000
165
+ },
166
+ {
167
+ "epoch": 3.7545506489266183,
168
+ "grad_norm": 0.5718339681625366,
169
+ "learning_rate": 0.00012698971125124463,
170
+ "loss": 5.0271943359375,
171
+ "step": 11500
172
+ },
173
+ {
174
+ "epoch": 3.9178026283568688,
175
+ "grad_norm": 0.607941746711731,
176
+ "learning_rate": 0.00012367076003982742,
177
+ "loss": 5.05531640625,
178
+ "step": 12000
179
+ },
180
+ {
181
+ "epoch": 4.080972981797404,
182
+ "grad_norm": 0.5361756682395935,
183
+ "learning_rate": 0.00012035180882841021,
184
+ "loss": 4.88087744140625,
185
+ "step": 12500
186
+ },
187
+ {
188
+ "epoch": 4.244224961227655,
189
+ "grad_norm": 0.5884597301483154,
190
+ "learning_rate": 0.00011703285761699303,
191
+ "loss": 4.826615234375,
192
+ "step": 13000
193
+ },
194
+ {
195
+ "epoch": 4.407476940657905,
196
+ "grad_norm": 0.6183493137359619,
197
+ "learning_rate": 0.00011371390640557584,
198
+ "loss": 4.8345087890625,
199
+ "step": 13500
200
+ },
201
+ {
202
+ "epoch": 4.570728920088156,
203
+ "grad_norm": 0.4895070493221283,
204
+ "learning_rate": 0.00011039495519415866,
205
+ "loss": 4.808126953125,
206
+ "step": 14000
207
+ },
208
+ {
209
+ "epoch": 4.733980899518406,
210
+ "grad_norm": 0.5796904563903809,
211
+ "learning_rate": 0.00010707600398274147,
212
+ "loss": 4.8475546875,
213
+ "step": 14500
214
+ },
215
+ {
216
+ "epoch": 4.897232878948657,
217
+ "grad_norm": 0.5432486534118652,
218
+ "learning_rate": 0.00010375705277132426,
219
+ "loss": 4.81570703125,
220
+ "step": 15000
221
+ },
222
+ {
223
+ "epoch": 5.060403232389192,
224
+ "grad_norm": 0.6771642565727234,
225
+ "learning_rate": 0.00010043810155990707,
226
+ "loss": 4.76427978515625,
227
+ "step": 15500
228
+ },
229
+ {
230
+ "epoch": 5.223655211819444,
231
+ "grad_norm": 0.6445333957672119,
232
+ "learning_rate": 9.711915034848989e-05,
233
+ "loss": 4.65180908203125,
234
+ "step": 16000
235
+ },
236
+ {
237
+ "epoch": 5.386907191249694,
238
+ "grad_norm": 0.599590539932251,
239
+ "learning_rate": 9.380019913707268e-05,
240
+ "loss": 4.70791796875,
241
+ "step": 16500
242
+ },
243
+ {
244
+ "epoch": 5.550159170679945,
245
+ "grad_norm": 0.6182076334953308,
246
+ "learning_rate": 9.04812479256555e-05,
247
+ "loss": 4.666,
248
+ "step": 17000
249
+ },
250
+ {
251
+ "epoch": 5.713411150110195,
252
+ "grad_norm": 0.601693332195282,
253
+ "learning_rate": 8.71622967142383e-05,
254
+ "loss": 4.6578359375,
255
+ "step": 17500
256
+ },
257
+ {
258
+ "epoch": 5.876663129540446,
259
+ "grad_norm": 0.6478536128997803,
260
+ "learning_rate": 8.384334550282111e-05,
261
+ "loss": 4.64422509765625,
262
+ "step": 18000
263
+ },
264
+ {
265
+ "epoch": 6.039833482980981,
266
+ "grad_norm": 0.6848897337913513,
267
+ "learning_rate": 8.052439429140392e-05,
268
+ "loss": 4.63602099609375,
269
+ "step": 18500
270
+ },
271
+ {
272
+ "epoch": 6.203085462411232,
273
+ "grad_norm": 0.599690854549408,
274
+ "learning_rate": 7.720544307998673e-05,
275
+ "loss": 4.57735595703125,
276
+ "step": 19000
277
+ },
278
+ {
279
+ "epoch": 6.366337441841482,
280
+ "grad_norm": 0.567436158657074,
281
+ "learning_rate": 7.388649186856955e-05,
282
+ "loss": 4.51408740234375,
283
+ "step": 19500
284
+ },
285
+ {
286
+ "epoch": 6.529589421271733,
287
+ "grad_norm": 0.9956502914428711,
288
+ "learning_rate": 7.056754065715234e-05,
289
+ "loss": 4.52962158203125,
290
+ "step": 20000
291
+ },
292
+ {
293
+ "epoch": 6.692841400701983,
294
+ "grad_norm": 0.6528770923614502,
295
+ "learning_rate": 6.724858944573516e-05,
296
+ "loss": 4.47805078125,
297
+ "step": 20500
298
+ },
299
+ {
300
+ "epoch": 6.856093380132235,
301
+ "grad_norm": 0.5711560249328613,
302
+ "learning_rate": 6.392963823431795e-05,
303
+ "loss": 4.5910791015625,
304
+ "step": 21000
305
+ },
306
+ {
307
+ "epoch": 7.019263733572769,
308
+ "grad_norm": 0.5683479309082031,
309
+ "learning_rate": 6.061068702290077e-05,
310
+ "loss": 4.54874267578125,
311
+ "step": 21500
312
+ },
313
+ {
314
+ "epoch": 7.182515713003021,
315
+ "grad_norm": 0.6191678643226624,
316
+ "learning_rate": 5.729173581148357e-05,
317
+ "loss": 4.418267578125,
318
+ "step": 22000
319
+ },
320
+ {
321
+ "epoch": 7.345767692433271,
322
+ "grad_norm": 0.6200758814811707,
323
+ "learning_rate": 5.3972784600066386e-05,
324
+ "loss": 4.46817431640625,
325
+ "step": 22500
326
+ },
327
+ {
328
+ "epoch": 7.509019671863522,
329
+ "grad_norm": 0.5810480713844299,
330
+ "learning_rate": 5.0653833388649185e-05,
331
+ "loss": 4.4288740234375,
332
+ "step": 23000
333
+ },
334
+ {
335
+ "epoch": 7.672271651293772,
336
+ "grad_norm": 0.6930559277534485,
337
+ "learning_rate": 4.7334882177232e-05,
338
+ "loss": 4.42653369140625,
339
+ "step": 23500
340
+ },
341
+ {
342
+ "epoch": 7.835523630724023,
343
+ "grad_norm": 0.6396164298057556,
344
+ "learning_rate": 4.4015930965814805e-05,
345
+ "loss": 4.49230419921875,
346
+ "step": 24000
347
+ },
348
+ {
349
+ "epoch": 7.998775610154273,
350
+ "grad_norm": 0.5532464981079102,
351
+ "learning_rate": 4.069697975439761e-05,
352
+ "loss": 4.39004931640625,
353
+ "step": 24500
354
+ },
355
+ {
356
+ "epoch": 8.161945963594809,
357
+ "grad_norm": 0.6885871291160583,
358
+ "learning_rate": 3.737802854298042e-05,
359
+ "loss": 4.382765625,
360
+ "step": 25000
361
+ },
362
+ {
363
+ "epoch": 8.32519794302506,
364
+ "grad_norm": 0.6912867426872253,
365
+ "learning_rate": 3.4059077331563225e-05,
366
+ "loss": 4.396109375,
367
+ "step": 25500
368
+ },
369
+ {
370
+ "epoch": 8.48844992245531,
371
+ "grad_norm": 0.6827392578125,
372
+ "learning_rate": 3.074012612014603e-05,
373
+ "loss": 4.40128125,
374
+ "step": 26000
375
+ },
376
+ {
377
+ "epoch": 8.651701901885561,
378
+ "grad_norm": 0.6782070398330688,
379
+ "learning_rate": 2.742117490872884e-05,
380
+ "loss": 4.35358203125,
381
+ "step": 26500
382
+ },
383
+ {
384
+ "epoch": 8.81495388131581,
385
+ "grad_norm": 0.7143053412437439,
386
+ "learning_rate": 2.410222369731165e-05,
387
+ "loss": 4.339625,
388
+ "step": 27000
389
+ },
390
+ {
391
+ "epoch": 8.978205860746062,
392
+ "grad_norm": 0.684012234210968,
393
+ "learning_rate": 2.0783272485894458e-05,
394
+ "loss": 4.3393515625,
395
+ "step": 27500
396
+ },
397
+ {
398
+ "epoch": 9.141376214186597,
399
+ "grad_norm": 0.6413611769676208,
400
+ "learning_rate": 1.7464321274477265e-05,
401
+ "loss": 4.32941748046875,
402
+ "step": 28000
403
+ },
404
+ {
405
+ "epoch": 9.304628193616848,
406
+ "grad_norm": 0.6120012402534485,
407
+ "learning_rate": 1.4145370063060073e-05,
408
+ "loss": 4.31831884765625,
409
+ "step": 28500
410
+ },
411
+ {
412
+ "epoch": 9.467880173047098,
413
+ "grad_norm": 0.6299709677696228,
414
+ "learning_rate": 1.0826418851642881e-05,
415
+ "loss": 4.28150341796875,
416
+ "step": 29000
417
+ },
418
+ {
419
+ "epoch": 9.631132152477349,
420
+ "grad_norm": 0.6235489845275879,
421
+ "learning_rate": 7.5074676402256894e-06,
422
+ "loss": 4.31871484375,
423
+ "step": 29500
424
+ },
425
+ {
426
+ "epoch": 9.794384131907599,
427
+ "grad_norm": 0.7088222503662109,
428
+ "learning_rate": 4.188516428808497e-06,
429
+ "loss": 4.35441064453125,
430
+ "step": 30000
431
+ },
432
+ {
433
+ "epoch": 9.95763611133785,
434
+ "grad_norm": 0.636341392993927,
435
+ "learning_rate": 8.695652173913044e-07,
436
+ "loss": 4.2993779296875,
437
+ "step": 30500
438
+ },
439
+ {
440
+ "epoch": 10.0,
441
+ "step": 30630,
442
+ "total_flos": 1.2426966463650202e+17,
443
+ "train_loss": 5.026907345643823,
444
+ "train_runtime": 19932.1348,
445
+ "train_samples_per_second": 49.168,
446
+ "train_steps_per_second": 1.537
447
+ }
448
+ ],
449
+ "logging_steps": 500,
450
+ "max_steps": 30630,
451
+ "num_input_tokens_seen": 0,
452
+ "num_train_epochs": 10,
453
+ "save_steps": 2000,
454
+ "stateful_callbacks": {
455
+ "TrainerControl": {
456
+ "args": {
457
+ "should_epoch_stop": false,
458
+ "should_evaluate": false,
459
+ "should_log": false,
460
+ "should_save": true,
461
+ "should_training_stop": true
462
+ },
463
+ "attributes": {}
464
+ }
465
+ },
466
+ "total_flos": 1.2426966463650202e+17,
467
+ "train_batch_size": 8,
468
+ "trial_name": null,
469
+ "trial_params": null
470
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1aef21c170dea1ff74d8ca0080f3f628809ef5bc7ac895de7fc736581881577
3
+ size 5329