Hugging Face's logo

error577
/
39d93b8a-13bf-40e9-8ba8-8d338e0337b1

PEFT
Safetensors
bloom
axolotl
Generated from Trainer
Model card Files
xet
 Community
error577 commited on
Commit
da3690c
·
verified ·
1 Parent(s): b48d066

End of training

Browse files
Files changed (3) hide show
  1. README.md +225 -0
  2. adapter_model.bin +3 -0
  3. adapter_model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,225 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: bigscience-bloom-rail-1.0
4
+ base_model: bigscience/bloomz-560m
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: 39d93b8a-13bf-40e9-8ba8-8d338e0337b1
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.4.1`
20
+ ```yaml
21
+ adapter: lora
22
+ auto_resume_from_checkpoints: true
23
+ base_model: bigscience/bloomz-560m
24
+ bf16: auto
25
+ chat_template: llama3
26
+ dataset_prepared_path: null
27
+ dataset_processes: 6
28
+ datasets:
29
+ - data_files:
30
+ - 31f811fb709cc914_train_data.json
31
+ ds_type: json
32
+ format: custom
33
+ path: /workspace/input_data/31f811fb709cc914_train_data.json
34
+ type:
35
+ field_instruction: instruction
36
+ field_output: response
37
+ format: '{instruction}'
38
+ no_input_format: '{instruction}'
39
+ system_format: '{system}'
40
+ system_prompt: ''
41
+ debug: null
42
+ deepspeed: null
43
+ early_stopping_patience: 3
44
+ eval_max_new_tokens: 128
45
+ eval_steps: 200
46
+ eval_table_size: null
47
+ evals_per_epoch: null
48
+ flash_attention: false
49
+ fp16: false
50
+ fsdp: null
51
+ fsdp_config: null
52
+ gradient_accumulation_steps: 4
53
+ gradient_checkpointing: true
54
+ group_by_length: false
55
+ hub_model_id: error577/39d93b8a-13bf-40e9-8ba8-8d338e0337b1
56
+ hub_repo: null
57
+ hub_strategy: checkpoint
58
+ hub_token: null
59
+ learning_rate: 0.0002
60
+ load_in_4bit: false
61
+ load_in_8bit: false
62
+ local_rank: null
63
+ logging_steps: 1
64
+ lora_alpha: 16
65
+ lora_dropout: 0.05
66
+ lora_fan_in_fan_out: null
67
+ lora_model_dir: null
68
+ lora_r: 8
69
+ lora_target_linear: true
70
+ lr_scheduler: cosine
71
+ max_grad_norm: 1.0
72
+ max_steps: null
73
+ micro_batch_size: 2
74
+ mlflow_experiment_name: /tmp/31f811fb709cc914_train_data.json
75
+ model_type: AutoModelForCausalLM
76
+ num_epochs: 3
77
+ optimizer: adamw_bnb_8bit
78
+ output_dir: miner_id_24
79
+ pad_to_sequence_len: true
80
+ resume_from_checkpoint: null
81
+ s2_attention: null
82
+ sample_packing: false
83
+ save_steps: 200
84
+ sequence_len: 512
85
+ strict: false
86
+ tf32: false
87
+ tokenizer_type: AutoTokenizer
88
+ train_on_inputs: false
89
+ trust_remote_code: true
90
+ val_set_size: 0.005
91
+ wandb_entity: null
92
+ wandb_mode: online
93
+ wandb_name: 8a000dd1-5b3f-47db-9e70-f522ce6599ed
94
+ wandb_project: Gradients-On-Demand
95
+ wandb_run: your_name
96
+ wandb_runid: 8a000dd1-5b3f-47db-9e70-f522ce6599ed
97
+ warmup_steps: 30
98
+ weight_decay: 0.0
99
+ xformers_attention: null
100
+
101
+ ```
102
+
103
+ </details><br>
104
+
105
+ # 39d93b8a-13bf-40e9-8ba8-8d338e0337b1
106
+
107
+ This model is a fine-tuned version of [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) on the None dataset.
108
+ It achieves the following results on the evaluation set:
109
+ - Loss: 1.0696
110
+
111
+ ## Model description
112
+
113
+ More information needed
114
+
115
+ ## Intended uses & limitations
116
+
117
+ More information needed
118
+
119
+ ## Training and evaluation data
120
+
121
+ More information needed
122
+
123
+ ## Training procedure
124
+
125
+ ### Training hyperparameters
126
+
127
+ The following hyperparameters were used during training:
128
+ - learning_rate: 0.0002
129
+ - train_batch_size: 2
130
+ - eval_batch_size: 2
131
+ - seed: 42
132
+ - gradient_accumulation_steps: 4
133
+ - total_train_batch_size: 8
134
+ - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
135
+ - lr_scheduler_type: cosine
136
+ - lr_scheduler_warmup_steps: 30
137
+ - num_epochs: 3
138
+
139
+ ### Training results
140
+
141
+ | Training Loss | Epoch | Step | Validation Loss |
142
+ |:-------------:|:------:|:-----:|:---------------:|
143
+ | 9.0252 | 0.0002 | 1 | 2.2089 |
144
+ | 5.8772 | 0.0329 | 200 | 1.5151 |
145
+ | 5.6189 | 0.0658 | 400 | 1.4449 |
146
+ | 6.4043 | 0.0987 | 600 | 1.3977 |
147
+ | 5.6258 | 0.1315 | 800 | 1.3686 |
148
+ | 6.5765 | 0.1644 | 1000 | 1.3478 |
149
+ | 5.3079 | 0.1973 | 1200 | 1.3205 |
150
+ | 5.0516 | 0.2302 | 1400 | 1.3033 |
151
+ | 4.9268 | 0.2631 | 1600 | 1.2868 |
152
+ | 5.4649 | 0.2960 | 1800 | 1.2725 |
153
+ | 4.0741 | 0.3288 | 2000 | 1.2626 |
154
+ | 5.1813 | 0.3617 | 2200 | 1.2504 |
155
+ | 4.7344 | 0.3946 | 2400 | 1.2434 |
156
+ | 4.3783 | 0.4275 | 2600 | 1.2340 |
157
+ | 5.2606 | 0.4604 | 2800 | 1.2265 |
158
+ | 5.4076 | 0.4933 | 3000 | 1.2197 |
159
+ | 4.6141 | 0.5261 | 3200 | 1.2121 |
160
+ | 4.6438 | 0.5590 | 3400 | 1.2015 |
161
+ | 4.9827 | 0.5919 | 3600 | 1.1986 |
162
+ | 6.4062 | 0.6248 | 3800 | 1.1911 |
163
+ | 4.0697 | 0.6577 | 4000 | 1.1877 |
164
+ | 5.0534 | 0.6906 | 4200 | 1.1814 |
165
+ | 4.7874 | 0.7234 | 4400 | 1.1786 |
166
+ | 5.2285 | 0.7563 | 4600 | 1.1764 |
167
+ | 4.7855 | 0.7892 | 4800 | 1.1699 |
168
+ | 4.3143 | 0.8221 | 5000 | 1.1654 |
169
+ | 4.8166 | 0.8550 | 5200 | 1.1595 |
170
+ | 5.2696 | 0.8879 | 5400 | 1.1548 |
171
+ | 4.0906 | 0.9207 | 5600 | 1.1515 |
172
+ | 4.5442 | 0.9536 | 5800 | 1.1503 |
173
+ | 4.3865 | 0.9865 | 6000 | 1.1437 |
174
+ | 3.3439 | 1.0194 | 6200 | 1.1433 |
175
+ | 5.4398 | 1.0523 | 6400 | 1.1440 |
176
+ | 3.1569 | 1.0852 | 6600 | 1.1406 |
177
+ | 4.5091 | 1.1181 | 6800 | 1.1336 |
178
+ | 5.2349 | 1.1509 | 7000 | 1.1311 |
179
+ | 4.2358 | 1.1838 | 7200 | 1.1323 |
180
+ | 4.442 | 1.2167 | 7400 | 1.1288 |
181
+ | 4.3978 | 1.2496 | 7600 | 1.1231 |
182
+ | 3.9429 | 1.2825 | 7800 | 1.1220 |
183
+ | 5.2279 | 1.3154 | 8000 | 1.1214 |
184
+ | 4.7596 | 1.3482 | 8200 | 1.1181 |
185
+ | 4.8692 | 1.3811 | 8400 | 1.1151 |
186
+ | 4.3599 | 1.4140 | 8600 | 1.1113 |
187
+ | 5.431 | 1.4469 | 8800 | 1.1069 |
188
+ | 3.6955 | 1.4798 | 9000 | 1.1041 |
189
+ | 4.7102 | 1.5127 | 9200 | 1.1054 |
190
+ | 4.4714 | 1.5455 | 9400 | 1.1023 |
191
+ | 3.4939 | 1.5784 | 9600 | 1.1004 |
192
+ | 5.278 | 1.6113 | 9800 | 1.0972 |
193
+ | 3.5237 | 1.6442 | 10000 | 1.0961 |
194
+ | 5.3808 | 1.6771 | 10200 | 1.0963 |
195
+ | 4.5247 | 1.7100 | 10400 | 1.0937 |
196
+ | 3.4588 | 1.7428 | 10600 | 1.0912 |
197
+ | 4.9685 | 1.7757 | 10800 | 1.0906 |
198
+ | 4.4331 | 1.8086 | 11000 | 1.0865 |
199
+ | 4.6026 | 1.8415 | 11200 | 1.0863 |
200
+ | 3.8171 | 1.8744 | 11400 | 1.0840 |
201
+ | 3.6165 | 1.9073 | 11600 | 1.0831 |
202
+ | 3.7015 | 1.9402 | 11800 | 1.0842 |
203
+ | 4.3536 | 1.9730 | 12000 | 1.0788 |
204
+ | 4.0382 | 2.0059 | 12200 | 1.0796 |
205
+ | 4.0658 | 2.0388 | 12400 | 1.0780 |
206
+ | 3.0832 | 2.0717 | 12600 | 1.0786 |
207
+ | 4.3379 | 2.1046 | 12800 | 1.0747 |
208
+ | 3.4001 | 2.1375 | 13000 | 1.0760 |
209
+ | 3.5611 | 2.1703 | 13200 | 1.0739 |
210
+ | 3.4944 | 2.2032 | 13400 | 1.0758 |
211
+ | 3.849 | 2.2361 | 13600 | 1.0736 |
212
+ | 5.0364 | 2.2690 | 13800 | 1.0747 |
213
+ | 4.4197 | 2.3019 | 14000 | 1.0696 |
214
+ | 4.6541 | 2.3348 | 14200 | 1.0716 |
215
+ | 5.5559 | 2.3676 | 14400 | 1.0697 |
216
+ | 4.3708 | 2.4005 | 14600 | 1.0696 |
217
+
218
+
219
+ ### Framework versions
220
+
221
+ - PEFT 0.13.2
222
+ - Transformers 4.46.0
223
+ - Pytorch 2.5.0+cu124
224
+ - Datasets 3.0.1
225
+ - Tokenizers 0.20.1
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34c16cef104986f87000ad097d8d243f7eaea8e2250dc7e87c5d41d2884222fe
3
+ size 12653386
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bdcc19c9dee38b4fd3a03818ca5c853ebffa1063983bf61f590ac17a5a12c251
3
  size 12609416
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:14c613ba54e54a21e2b2f3d1f791b2f87dee009217516759f9bed81f138197c8
3
  size 12609416