Hugging Face's logo

ninagroot
/
Baby-Llama-58M-RUN2

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
text-generation-inference
Model card Files
xet
Community
ninagroot commited on
Commit
d163f15
·
verified ·
1 Parent(s): fd52a99

ninagroot/babyllamatest

Browse files
Files changed (5) hide show
  1. README.md +136 -0
  2. config.json +28 -0
  3. generation_config.json +7 -0
  4. model.safetensors +3 -0
  5. training_args.bin +3 -0
README.md ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - generated_from_trainer
4
+ model-index:
5
+ - name: Baby-Llama-58M
6
+ results: []
7
+ ---
8
+
9
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
10
+ should probably proofread and complete it, then remove this comment. -->
11
+
12
+ # Baby-Llama-58M
13
+
14
+ This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
15
+ It achieves the following results on the evaluation set:
16
+ - Loss: 6.7221
17
+
18
+ ## Model description
19
+
20
+ More information needed
21
+
22
+ ## Intended uses & limitations
23
+
24
+ More information needed
25
+
26
+ ## Training and evaluation data
27
+
28
+ More information needed
29
+
30
+ ## Training procedure
31
+
32
+ ### Training hyperparameters
33
+
34
+ The following hyperparameters were used during training:
35
+ - learning_rate: 0.00025
36
+ - train_batch_size: 128
37
+ - eval_batch_size: 8
38
+ - seed: 42
39
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
40
+ - lr_scheduler_type: cosine
41
+ - lr_scheduler_warmup_steps: 50
42
+ - num_epochs: 80
43
+ - mixed_precision_training: Native AMP
44
+
45
+ ### Training results
46
+
47
+ | Training Loss | Epoch | Step | Validation Loss |
48
+ |:-------------:|:-----:|:----:|:---------------:|
49
+ | 135.1538 | 1.0 | 8 | 118.8448 |
50
+ | 112.3406 | 2.0 | 16 | 102.1364 |
51
+ | 107.9124 | 3.0 | 24 | 86.8275 |
52
+ | 85.5837 | 4.0 | 32 | 71.8709 |
53
+ | 82.7059 | 5.0 | 40 | 60.4278 |
54
+ | 62.0973 | 6.0 | 48 | 51.7763 |
55
+ | 56.6325 | 7.0 | 56 | 44.4392 |
56
+ | 46.5864 | 8.0 | 64 | 39.5206 |
57
+ | 40.749 | 9.0 | 72 | 36.8323 |
58
+ | 34.1225 | 10.0 | 80 | 30.4178 |
59
+ | 26.3662 | 11.0 | 88 | 25.6518 |
60
+ | 21.4543 | 12.0 | 96 | 21.5034 |
61
+ | 17.4064 | 13.0 | 104 | 18.2917 |
62
+ | 14.5338 | 14.0 | 112 | 16.0543 |
63
+ | 12.8652 | 15.0 | 120 | 14.5666 |
64
+ | 11.1266 | 16.0 | 128 | 13.6536 |
65
+ | 9.5181 | 17.0 | 136 | 12.6228 |
66
+ | 8.0769 | 18.0 | 144 | 11.2297 |
67
+ | 7.3252 | 19.0 | 152 | 10.6871 |
68
+ | 6.7225 | 20.0 | 160 | 10.5576 |
69
+ | 6.1834 | 21.0 | 168 | 9.6600 |
70
+ | 6.0954 | 22.0 | 176 | 9.5832 |
71
+ | 5.715 | 23.0 | 184 | 9.4159 |
72
+ | 5.5297 | 24.0 | 192 | 8.8495 |
73
+ | 5.1538 | 25.0 | 200 | 8.6964 |
74
+ | 5.0472 | 26.0 | 208 | 8.4671 |
75
+ | 5.0581 | 27.0 | 216 | 8.3979 |
76
+ | 4.6914 | 28.0 | 224 | 8.2086 |
77
+ | 4.6117 | 29.0 | 232 | 8.2212 |
78
+ | 4.5157 | 30.0 | 240 | 8.1633 |
79
+ | 4.1918 | 31.0 | 248 | 8.1399 |
80
+ | 4.5274 | 32.0 | 256 | 7.7368 |
81
+ | 4.0493 | 33.0 | 264 | 7.7647 |
82
+ | 4.2799 | 34.0 | 272 | 7.8127 |
83
+ | 4.5331 | 35.0 | 280 | 7.6971 |
84
+ | 4.5937 | 36.0 | 288 | 7.6908 |
85
+ | 3.9957 | 37.0 | 296 | 7.6509 |
86
+ | 4.3035 | 38.0 | 304 | 7.5682 |
87
+ | 4.2626 | 39.0 | 312 | 7.4550 |
88
+ | 3.7238 | 40.0 | 320 | 7.4516 |
89
+ | 3.9562 | 41.0 | 328 | 7.2862 |
90
+ | 3.8612 | 42.0 | 336 | 7.3332 |
91
+ | 3.6178 | 43.0 | 344 | 7.3013 |
92
+ | 3.7672 | 44.0 | 352 | 7.2144 |
93
+ | 3.715 | 45.0 | 360 | 7.2103 |
94
+ | 3.7594 | 46.0 | 368 | 7.2457 |
95
+ | 4.3614 | 47.0 | 376 | 7.1274 |
96
+ | 4.0406 | 48.0 | 384 | 7.0472 |
97
+ | 3.5213 | 49.0 | 392 | 6.9963 |
98
+ | 3.7373 | 50.0 | 400 | 7.0503 |
99
+ | 3.7399 | 51.0 | 408 | 6.9916 |
100
+ | 3.8109 | 52.0 | 416 | 6.9899 |
101
+ | 3.3897 | 53.0 | 424 | 6.9132 |
102
+ | 3.2456 | 54.0 | 432 | 6.9393 |
103
+ | 3.8682 | 55.0 | 440 | 6.9017 |
104
+ | 3.3904 | 56.0 | 448 | 6.8995 |
105
+ | 3.8449 | 57.0 | 456 | 6.8478 |
106
+ | 3.6319 | 58.0 | 464 | 6.8388 |
107
+ | 3.4726 | 59.0 | 472 | 6.8123 |
108
+ | 3.5895 | 60.0 | 480 | 6.8452 |
109
+ | 3.4 | 61.0 | 488 | 6.7875 |
110
+ | 3.6904 | 62.0 | 496 | 6.7963 |
111
+ | 3.3957 | 63.0 | 504 | 6.7976 |
112
+ | 3.4602 | 64.0 | 512 | 6.8317 |
113
+ | 3.2714 | 65.0 | 520 | 6.8063 |
114
+ | 3.5695 | 66.0 | 528 | 6.7709 |
115
+ | 3.1538 | 67.0 | 536 | 6.7849 |
116
+ | 3.5586 | 68.0 | 544 | 6.7565 |
117
+ | 3.194 | 69.0 | 552 | 6.7629 |
118
+ | 3.0488 | 70.0 | 560 | 6.7462 |
119
+ | 3.6931 | 71.0 | 568 | 6.7269 |
120
+ | 3.7324 | 72.0 | 576 | 6.7367 |
121
+ | 3.2075 | 73.0 | 584 | 6.7460 |
122
+ | 3.3394 | 74.0 | 592 | 6.7111 |
123
+ | 3.4074 | 75.0 | 600 | 6.7456 |
124
+ | 3.3679 | 76.0 | 608 | 6.7225 |
125
+ | 3.2689 | 77.0 | 616 | 6.7234 |
126
+ | 3.6886 | 78.0 | 624 | 6.7247 |
127
+ | 3.4587 | 79.0 | 632 | 6.7224 |
128
+ | 3.6444 | 80.0 | 640 | 6.7221 |
129
+
130
+
131
+ ### Framework versions
132
+
133
+ - Transformers 4.39.1
134
+ - Pytorch 2.1.2+cu121
135
+ - Datasets 2.16.1
136
+ - Tokenizers 0.15.0
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 512,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 1024,
13
+ "max_position_embeddings": 256,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 8,
16
+ "num_hidden_layers": 16,
17
+ "num_key_value_heads": 8,
18
+ "pad_token_id": 0,
19
+ "pretraining_tp": 1,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_scaling": null,
22
+ "rope_theta": 10000.0,
23
+ "tie_word_embeddings": false,
24
+ "torch_dtype": "float32",
25
+ "transformers_version": "4.39.1",
26
+ "use_cache": true,
27
+ "vocab_size": 12198
28
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.39.1"
7
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:997eba865b9faab2d4699a3f0fe0c29102613d454821db24be613e290577cfa1
3
+ size 217819016
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4945f6edf65f0e10920d596836c581a302f5b5423e6d65dc766dc067d9f9f291
3
+ size 4984