Hugging Face's logo

Basha738
/
outputs

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files
xet
Metrics Community
Basha738 commited on
Commit
800de26
·
verified ·
1 Parent(s): f4a7058

llama2-13B-supervised-ft-10-epochs-351

Browse files
Files changed (4) hide show
  1. README.md +51 -8
  2. adapter_config.json +5 -5
  3. adapter_model.safetensors +2 -2
  4. training_args.bin +1 -1
README.md CHANGED
@@ -17,12 +17,7 @@ should probably proofread and complete it, then remove this comment. -->
17
 
18
  This model was trained from scratch on the None dataset.
19
  It achieves the following results on the evaluation set:
20
- - eval_loss: 1.6825
21
- - eval_runtime: 99.0641
22
- - eval_samples_per_second: 0.697
23
- - eval_steps_per_second: 0.697
24
- - epoch: 7.53
25
- - step: 128
26
 
27
  ## Model description
28
 
@@ -41,7 +36,7 @@ More information needed
41
  ### Training hyperparameters
42
 
43
  The following hyperparameters were used during training:
44
- - learning_rate: 5e-06
45
  - train_batch_size: 1
46
  - eval_batch_size: 1
47
  - seed: 42
@@ -49,9 +44,57 @@ The following hyperparameters were used during training:
49
  - total_train_batch_size: 16
50
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
51
  - lr_scheduler_type: linear
52
- - num_epochs: 20
53
  - mixed_precision_training: Native AMP
54
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  ### Framework versions
56
 
57
  - PEFT 0.8.2
 
17
 
18
  This model was trained from scratch on the None dataset.
19
  It achieves the following results on the evaluation set:
20
+ - Loss: 1.4471
 
 
 
 
 
21
 
22
  ## Model description
23
 
 
36
  ### Training hyperparameters
37
 
38
  The following hyperparameters were used during training:
39
+ - learning_rate: 5e-05
40
  - train_batch_size: 1
41
  - eval_batch_size: 1
42
  - seed: 42
 
44
  - total_train_batch_size: 16
45
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
  - lr_scheduler_type: linear
47
+ - num_epochs: 10
48
  - mixed_precision_training: Native AMP
49
 
50
+ ### Training results
51
+
52
+ | Training Loss | Epoch | Step | Validation Loss |
53
+ |:-------------:|:-----:|:----:|:---------------:|
54
+ | 2.0121 | 0.24 | 4 | 1.9623 |
55
+ | 2.0559 | 0.47 | 8 | 1.8792 |
56
+ | 1.8364 | 0.71 | 12 | 1.8000 |
57
+ | 1.6852 | 0.94 | 16 | 1.7083 |
58
+ | 1.7672 | 1.18 | 20 | 1.6258 |
59
+ | 1.6344 | 1.41 | 24 | 1.5731 |
60
+ | 1.5026 | 1.65 | 28 | 1.5477 |
61
+ | 1.4473 | 1.88 | 32 | 1.5344 |
62
+ | 1.5297 | 2.12 | 36 | 1.5254 |
63
+ | 1.5623 | 2.35 | 40 | 1.5183 |
64
+ | 1.3866 | 2.59 | 44 | 1.5122 |
65
+ | 1.5727 | 2.82 | 48 | 1.5067 |
66
+ | 1.3696 | 3.06 | 52 | 1.5018 |
67
+ | 1.5159 | 3.29 | 56 | 1.4973 |
68
+ | 1.5107 | 3.53 | 60 | 1.4932 |
69
+ | 1.4175 | 3.76 | 64 | 1.4893 |
70
+ | 1.404 | 4.0 | 68 | 1.4854 |
71
+ | 1.4115 | 4.24 | 72 | 1.4821 |
72
+ | 1.5804 | 4.47 | 76 | 1.4788 |
73
+ | 1.4163 | 4.71 | 80 | 1.4759 |
74
+ | 1.4404 | 4.94 | 84 | 1.4729 |
75
+ | 1.4331 | 5.18 | 88 | 1.4700 |
76
+ | 1.4269 | 5.41 | 92 | 1.4674 |
77
+ | 1.4902 | 5.65 | 96 | 1.4650 |
78
+ | 1.493 | 5.88 | 100 | 1.4627 |
79
+ | 1.4476 | 6.12 | 104 | 1.4608 |
80
+ | 1.4142 | 6.35 | 108 | 1.4592 |
81
+ | 1.4147 | 6.59 | 112 | 1.4573 |
82
+ | 1.3399 | 6.82 | 116 | 1.4558 |
83
+ | 1.4323 | 7.06 | 120 | 1.4543 |
84
+ | 1.2752 | 7.29 | 124 | 1.4531 |
85
+ | 1.4907 | 7.53 | 128 | 1.4524 |
86
+ | 1.3854 | 7.76 | 132 | 1.4510 |
87
+ | 1.3615 | 8.0 | 136 | 1.4498 |
88
+ | 1.5176 | 8.24 | 140 | 1.4492 |
89
+ | 1.3634 | 8.47 | 144 | 1.4490 |
90
+ | 1.3644 | 8.71 | 148 | 1.4486 |
91
+ | 1.4832 | 8.94 | 152 | 1.4481 |
92
+ | 1.4305 | 9.18 | 156 | 1.4476 |
93
+ | 1.5344 | 9.41 | 160 | 1.4473 |
94
+ | 1.4204 | 9.65 | 164 | 1.4471 |
95
+ | 1.3488 | 9.88 | 168 | 1.4471 |
96
+
97
+
98
  ### Framework versions
99
 
100
  - PEFT 0.8.2
adapter_config.json CHANGED
@@ -15,17 +15,17 @@
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
17
  "peft_type": "LORA",
18
- "r": 64,
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
  "k_proj",
 
23
  "gate_proj",
24
- "o_proj",
25
- "down_proj",
26
  "q_proj",
27
- "up_proj",
28
- "v_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_rslora": false
 
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
17
  "peft_type": "LORA",
18
+ "r": 128,
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
  "k_proj",
23
+ "up_proj",
24
  "gate_proj",
25
+ "v_proj",
 
26
  "q_proj",
27
+ "down_proj",
28
+ "o_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_rslora": false
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:270ced53a9714f416bffc8c4ffc60dc86f1e477f69b6173af6e30c63c4d6abd2
3
- size 1001465824
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7270f8b2ce500576568e6d0d99dc89289051a76cac3bfc66b49b84b0c3bdb80
3
+ size 2002857080
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:587b32f730f5908b2afd8b6d94ad08395d97c135b5a409ca2313f96f39a15d5f
3
  size 4664
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2feb3a72004a78345189fddde57ecae17643f1dfe32c31b72068642d138e3242
3
  size 4664