Hugging Face's logo

jigs97022
/
tinyfeels-1.7b

Text Generation
Transformers
Safetensors
GGUF
English
smollm2
lora
emotional
chat
companion
unsloth
q4_k_m
roleplay
small-model
conversational
Model card Files
xet
 Community
jigs97022 commited on
Commit
4c8f74c
·
verified ·
1 Parent(s): 7347e06

Upload folder using huggingface_hub

Browse files
Files changed (49) hide show
  1. lora-adapter/README.md +59 -0
  2. lora-adapter/checkpoint-100/README.md +210 -0
  3. lora-adapter/checkpoint-100/adapter_config.json +52 -0
  4. lora-adapter/checkpoint-100/adapter_model.safetensors +3 -0
  5. lora-adapter/checkpoint-100/chat_template.jinja +6 -0
  6. lora-adapter/checkpoint-100/optimizer.pt +3 -0
  7. lora-adapter/checkpoint-100/rng_state.pth +3 -0
  8. lora-adapter/checkpoint-100/scaler.pt +3 -0
  9. lora-adapter/checkpoint-100/scheduler.pt +3 -0
  10. lora-adapter/checkpoint-100/tokenizer.json +0 -0
  11. lora-adapter/checkpoint-100/tokenizer_config.json +173 -0
  12. lora-adapter/checkpoint-100/trainer_state.json +112 -0
  13. lora-adapter/checkpoint-100/training_args.bin +3 -0
  14. lora-adapter/checkpoint-200/README.md +210 -0
  15. lora-adapter/checkpoint-200/adapter_config.json +52 -0
  16. lora-adapter/checkpoint-200/adapter_model.safetensors +3 -0
  17. lora-adapter/checkpoint-200/chat_template.jinja +6 -0
  18. lora-adapter/checkpoint-200/optimizer.pt +3 -0
  19. lora-adapter/checkpoint-200/rng_state.pth +3 -0
  20. lora-adapter/checkpoint-200/scaler.pt +3 -0
  21. lora-adapter/checkpoint-200/scheduler.pt +3 -0
  22. lora-adapter/checkpoint-200/tokenizer.json +0 -0
  23. lora-adapter/checkpoint-200/tokenizer_config.json +173 -0
  24. lora-adapter/checkpoint-200/trainer_state.json +190 -0
  25. lora-adapter/checkpoint-200/training_args.bin +3 -0
  26. lora-adapter/checkpoint-300/README.md +210 -0
  27. lora-adapter/checkpoint-300/adapter_config.json +52 -0
  28. lora-adapter/checkpoint-300/adapter_model.safetensors +3 -0
  29. lora-adapter/checkpoint-300/chat_template.jinja +6 -0
  30. lora-adapter/checkpoint-300/optimizer.pt +3 -0
  31. lora-adapter/checkpoint-300/rng_state.pth +3 -0
  32. lora-adapter/checkpoint-300/scaler.pt +3 -0
  33. lora-adapter/checkpoint-300/scheduler.pt +3 -0
  34. lora-adapter/checkpoint-300/tokenizer.json +0 -0
  35. lora-adapter/checkpoint-300/tokenizer_config.json +173 -0
  36. lora-adapter/checkpoint-300/trainer_state.json +268 -0
  37. lora-adapter/checkpoint-300/training_args.bin +3 -0
  38. lora-adapter/checkpoint-339/README.md +210 -0
  39. lora-adapter/checkpoint-339/adapter_config.json +52 -0
  40. lora-adapter/checkpoint-339/adapter_model.safetensors +3 -0
  41. lora-adapter/checkpoint-339/chat_template.jinja +6 -0
  42. lora-adapter/checkpoint-339/optimizer.pt +3 -0
  43. lora-adapter/checkpoint-339/rng_state.pth +3 -0
  44. lora-adapter/checkpoint-339/scaler.pt +3 -0
  45. lora-adapter/checkpoint-339/scheduler.pt +3 -0
  46. lora-adapter/checkpoint-339/tokenizer.json +0 -0
  47. lora-adapter/checkpoint-339/tokenizer_config.json +173 -0
  48. lora-adapter/checkpoint-339/trainer_state.json +297 -0
  49. lora-adapter/checkpoint-339/training_args.bin +3 -0
lora-adapter/README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: unsloth/SmolLM2-1.7B-Instruct
3
+ library_name: transformers
4
+ model_name: smollm2-emotional-lora
5
+ tags:
6
+ - generated_from_trainer
7
+ - sft
8
+ - unsloth
9
+ - trl
10
+ licence: license
11
+ ---
12
+
13
+ # Model Card for smollm2-emotional-lora
14
+
15
+ This model is a fine-tuned version of [unsloth/SmolLM2-1.7B-Instruct](https://huggingface.co/unsloth/SmolLM2-1.7B-Instruct).
16
+ It has been trained using [TRL](https://github.com/huggingface/trl).
17
+
18
+ ## Quick start
19
+
20
+ ```python
21
+ from transformers import pipeline
22
+
23
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
24
+ generator = pipeline("text-generation", model="None", device="cuda")
25
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
26
+ print(output["generated_text"])
27
+ ```
28
+
29
+ ## Training procedure
30
+
31
+
32
+
33
+
34
+ This model was trained with SFT.
35
+
36
+ ### Framework versions
37
+
38
+ - TRL: 0.24.0
39
+ - Transformers: 5.5.0
40
+ - Pytorch: 2.11.0+cu128
41
+ - Datasets: 4.3.0
42
+ - Tokenizers: 0.22.2
43
+
44
+ ## Citations
45
+
46
+
47
+
48
+ Cite TRL as:
49
+
50
+ ```bibtex
51
+ @misc{vonwerra2022trl,
52
+ title = {{TRL: Transformer Reinforcement Learning}},
53
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
54
+ year = 2020,
55
+ journal = {GitHub repository},
56
+ publisher = {GitHub},
57
+ howpublished = {\url{https://github.com/huggingface/trl}}
58
+ }
59
+ ```
lora-adapter/checkpoint-100/README.md ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: unsloth/SmolLM2-1.7B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:unsloth/SmolLM2-1.7B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ - unsloth
12
+ ---
13
+
14
+ # Model Card for Model ID
15
+
16
+ <!-- Provide a quick summary of what the model is/does. -->
17
+
18
+
19
+
20
+ ## Model Details
21
+
22
+ ### Model Description
23
+
24
+ <!-- Provide a longer summary of what this model is. -->
25
+
26
+
27
+
28
+ - **Developed by:** [More Information Needed]
29
+ - **Funded by [optional]:** [More Information Needed]
30
+ - **Shared by [optional]:** [More Information Needed]
31
+ - **Model type:** [More Information Needed]
32
+ - **Language(s) (NLP):** [More Information Needed]
33
+ - **License:** [More Information Needed]
34
+ - **Finetuned from model [optional]:** [More Information Needed]
35
+
36
+ ### Model Sources [optional]
37
+
38
+ <!-- Provide the basic links for the model. -->
39
+
40
+ - **Repository:** [More Information Needed]
41
+ - **Paper [optional]:** [More Information Needed]
42
+ - **Demo [optional]:** [More Information Needed]
43
+
44
+ ## Uses
45
+
46
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
47
+
48
+ ### Direct Use
49
+
50
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
51
+
52
+ [More Information Needed]
53
+
54
+ ### Downstream Use [optional]
55
+
56
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
57
+
58
+ [More Information Needed]
59
+
60
+ ### Out-of-Scope Use
61
+
62
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
63
+
64
+ [More Information Needed]
65
+
66
+ ## Bias, Risks, and Limitations
67
+
68
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
69
+
70
+ [More Information Needed]
71
+
72
+ ### Recommendations
73
+
74
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
75
+
76
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
77
+
78
+ ## How to Get Started with the Model
79
+
80
+ Use the code below to get started with the model.
81
+
82
+ [More Information Needed]
83
+
84
+ ## Training Details
85
+
86
+ ### Training Data
87
+
88
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
89
+
90
+ [More Information Needed]
91
+
92
+ ### Training Procedure
93
+
94
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
95
+
96
+ #### Preprocessing [optional]
97
+
98
+ [More Information Needed]
99
+
100
+
101
+ #### Training Hyperparameters
102
+
103
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
104
+
105
+ #### Speeds, Sizes, Times [optional]
106
+
107
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
108
+
109
+ [More Information Needed]
110
+
111
+ ## Evaluation
112
+
113
+ <!-- This section describes the evaluation protocols and provides the results. -->
114
+
115
+ ### Testing Data, Factors & Metrics
116
+
117
+ #### Testing Data
118
+
119
+ <!-- This should link to a Dataset Card if possible. -->
120
+
121
+ [More Information Needed]
122
+
123
+ #### Factors
124
+
125
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
126
+
127
+ [More Information Needed]
128
+
129
+ #### Metrics
130
+
131
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
132
+
133
+ [More Information Needed]
134
+
135
+ ### Results
136
+
137
+ [More Information Needed]
138
+
139
+ #### Summary
140
+
141
+
142
+
143
+ ## Model Examination [optional]
144
+
145
+ <!-- Relevant interpretability work for the model goes here -->
146
+
147
+ [More Information Needed]
148
+
149
+ ## Environmental Impact
150
+
151
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
152
+
153
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
154
+
155
+ - **Hardware Type:** [More Information Needed]
156
+ - **Hours used:** [More Information Needed]
157
+ - **Cloud Provider:** [More Information Needed]
158
+ - **Compute Region:** [More Information Needed]
159
+ - **Carbon Emitted:** [More Information Needed]
160
+
161
+ ## Technical Specifications [optional]
162
+
163
+ ### Model Architecture and Objective
164
+
165
+ [More Information Needed]
166
+
167
+ ### Compute Infrastructure
168
+
169
+ [More Information Needed]
170
+
171
+ #### Hardware
172
+
173
+ [More Information Needed]
174
+
175
+ #### Software
176
+
177
+ [More Information Needed]
178
+
179
+ ## Citation [optional]
180
+
181
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
182
+
183
+ **BibTeX:**
184
+
185
+ [More Information Needed]
186
+
187
+ **APA:**
188
+
189
+ [More Information Needed]
190
+
191
+ ## Glossary [optional]
192
+
193
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
194
+
195
+ [More Information Needed]
196
+
197
+ ## More Information [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Authors [optional]
202
+
203
+ [More Information Needed]
204
+
205
+ ## Model Card Contact
206
+
207
+ [More Information Needed]
208
+ ### Framework versions
209
+
210
+ - PEFT 0.19.1
lora-adapter/checkpoint-100/adapter_config.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "LlamaForCausalLM",
7
+ "parent_library": "transformers.models.llama.modeling_llama",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "unsloth/SmolLM2-1.7B-Instruct",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "layer_replication": null,
20
+ "layers_pattern": null,
21
+ "layers_to_transform": null,
22
+ "loftq_config": {},
23
+ "lora_alpha": 32,
24
+ "lora_bias": false,
25
+ "lora_dropout": 0.05,
26
+ "lora_ga_config": null,
27
+ "megatron_config": null,
28
+ "megatron_core": "megatron.core",
29
+ "modules_to_save": null,
30
+ "peft_type": "LORA",
31
+ "peft_version": "0.19.1",
32
+ "qalora_group_size": 16,
33
+ "r": 16,
34
+ "rank_pattern": {},
35
+ "revision": null,
36
+ "target_modules": [
37
+ "down_proj",
38
+ "up_proj",
39
+ "k_proj",
40
+ "q_proj",
41
+ "v_proj",
42
+ "o_proj",
43
+ "gate_proj"
44
+ ],
45
+ "target_parameters": null,
46
+ "task_type": "CAUSAL_LM",
47
+ "trainable_token_indices": null,
48
+ "use_bdlora": null,
49
+ "use_dora": false,
50
+ "use_qalora": false,
51
+ "use_rslora": false
52
+ }
lora-adapter/checkpoint-100/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca49b1abf710d89f1815ead8db4315932fd62e3ac2e80db73d0ceb3895c53920
3
+ size 72396376
lora-adapter/checkpoint-100/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
lora-adapter/checkpoint-100/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbd03fd7928dad5f3434f6d3f708945574f93c5b29b731eb020c9d938f3b68d8
3
+ size 37138213
lora-adapter/checkpoint-100/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f441831887b262e704c48a377976cd7b95a1916beb47779992cdefe50e8f5352
3
+ size 14645
lora-adapter/checkpoint-100/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c4393a84a3109995aa1202073b039b12062e3189ed89aa0b94ef0510ba843009
3
+ size 1383
lora-adapter/checkpoint-100/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d7abd561cc312e8726b1c642d5f7e5ab96cceefb95f5236b6b3def7f98c4f3f
3
+ size 1465
lora-adapter/checkpoint-100/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
lora-adapter/checkpoint-100/tokenizer_config.json ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|im_start|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>"
11
+ ],
12
+ "is_local": false,
13
+ "model_max_length": 8192,
14
+ "pad_token": "<|PAD_TOKEN|>",
15
+ "padding_side": "right",
16
+ "tokenizer_class": "GPT2Tokenizer",
17
+ "unk_token": "�",
18
+ "vocab_size": 49152,
19
+ "added_tokens_decoder": {
20
+ "0": {
21
+ "content": "<|endoftext|>",
22
+ "single_word": false,
23
+ "lstrip": false,
24
+ "rstrip": false,
25
+ "normalized": false,
26
+ "special": true
27
+ },
28
+ "1": {
29
+ "content": "<|im_start|>",
30
+ "single_word": false,
31
+ "lstrip": false,
32
+ "rstrip": false,
33
+ "normalized": false,
34
+ "special": true
35
+ },
36
+ "2": {
37
+ "content": "<|im_end|>",
38
+ "single_word": false,
39
+ "lstrip": false,
40
+ "rstrip": false,
41
+ "normalized": false,
42
+ "special": true
43
+ },
44
+ "3": {
45
+ "content": "<repo_name>",
46
+ "single_word": false,
47
+ "lstrip": false,
48
+ "rstrip": false,
49
+ "normalized": false,
50
+ "special": true
51
+ },
52
+ "4": {
53
+ "content": "<reponame>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ "5": {
61
+ "content": "<file_sep>",
62
+ "single_word": false,
63
+ "lstrip": false,
64
+ "rstrip": false,
65
+ "normalized": false,
66
+ "special": true
67
+ },
68
+ "6": {
69
+ "content": "<filename>",
70
+ "single_word": false,
71
+ "lstrip": false,
72
+ "rstrip": false,
73
+ "normalized": false,
74
+ "special": true
75
+ },
76
+ "7": {
77
+ "content": "<gh_stars>",
78
+ "single_word": false,
79
+ "lstrip": false,
80
+ "rstrip": false,
81
+ "normalized": false,
82
+ "special": true
83
+ },
84
+ "8": {
85
+ "content": "<issue_start>",
86
+ "single_word": false,
87
+ "lstrip": false,
88
+ "rstrip": false,
89
+ "normalized": false,
90
+ "special": true
91
+ },
92
+ "9": {
93
+ "content": "<issue_comment>",
94
+ "single_word": false,
95
+ "lstrip": false,
96
+ "rstrip": false,
97
+ "normalized": false,
98
+ "special": true
99
+ },
100
+ "10": {
101
+ "content": "<issue_closed>",
102
+ "single_word": false,
103
+ "lstrip": false,
104
+ "rstrip": false,
105
+ "normalized": false,
106
+ "special": true
107
+ },
108
+ "11": {
109
+ "content": "<jupyter_start>",
110
+ "single_word": false,
111
+ "lstrip": false,
112
+ "rstrip": false,
113
+ "normalized": false,
114
+ "special": true
115
+ },
116
+ "12": {
117
+ "content": "<jupyter_text>",
118
+ "single_word": false,
119
+ "lstrip": false,
120
+ "rstrip": false,
121
+ "normalized": false,
122
+ "special": true
123
+ },
124
+ "13": {
125
+ "content": "<jupyter_code>",
126
+ "single_word": false,
127
+ "lstrip": false,
128
+ "rstrip": false,
129
+ "normalized": false,
130
+ "special": true
131
+ },
132
+ "14": {
133
+ "content": "<jupyter_output>",
134
+ "single_word": false,
135
+ "lstrip": false,
136
+ "rstrip": false,
137
+ "normalized": false,
138
+ "special": true
139
+ },
140
+ "15": {
141
+ "content": "<jupyter_script>",
142
+ "single_word": false,
143
+ "lstrip": false,
144
+ "rstrip": false,
145
+ "normalized": false,
146
+ "special": true
147
+ },
148
+ "16": {
149
+ "content": "<empty_output>",
150
+ "single_word": false,
151
+ "lstrip": false,
152
+ "rstrip": false,
153
+ "normalized": false,
154
+ "special": true
155
+ },
156
+ "24211": {
157
+ "content": "�",
158
+ "single_word": false,
159
+ "lstrip": false,
160
+ "rstrip": false,
161
+ "normalized": false,
162
+ "special": true
163
+ },
164
+ "49152": {
165
+ "content": "<|PAD_TOKEN|>",
166
+ "single_word": false,
167
+ "lstrip": false,
168
+ "rstrip": false,
169
+ "normalized": false,
170
+ "special": true
171
+ }
172
+ }
173
+ }
lora-adapter/checkpoint-100/trainer_state.json ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 0.8888888888888888,
6
+ "eval_steps": 100,
7
+ "global_step": 100,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.08888888888888889,
14
+ "grad_norm": 0.6075066328048706,
15
+ "learning_rate": 3.6e-05,
16
+ "loss": 2.435858154296875,
17
+ "step": 10
18
+ },
19
+ {
20
+ "epoch": 0.17777777777777778,
21
+ "grad_norm": 0.5337297320365906,
22
+ "learning_rate": 7.6e-05,
23
+ "loss": 2.261408805847168,
24
+ "step": 20
25
+ },
26
+ {
27
+ "epoch": 0.26666666666666666,
28
+ "grad_norm": 4.804379463195801,
29
+ "learning_rate": 0.000116,
30
+ "loss": 1.9577768325805665,
31
+ "step": 30
32
+ },
33
+ {
34
+ "epoch": 0.35555555555555557,
35
+ "grad_norm": 0.39172491431236267,
36
+ "learning_rate": 0.00015600000000000002,
37
+ "loss": 1.6207334518432617,
38
+ "step": 40
39
+ },
40
+ {
41
+ "epoch": 0.4444444444444444,
42
+ "grad_norm": 0.31786638498306274,
43
+ "learning_rate": 0.000196,
44
+ "loss": 1.5145363807678223,
45
+ "step": 50
46
+ },
47
+ {
48
+ "epoch": 0.5333333333333333,
49
+ "grad_norm": 0.295754611492157,
50
+ "learning_rate": 0.00019952179661709028,
51
+ "loss": 1.5010628700256348,
52
+ "step": 60
53
+ },
54
+ {
55
+ "epoch": 0.6222222222222222,
56
+ "grad_norm": 0.29678136110305786,
57
+ "learning_rate": 0.00019787461880296965,
58
+ "loss": 1.4191683769226073,
59
+ "step": 70
60
+ },
61
+ {
62
+ "epoch": 0.7111111111111111,
63
+ "grad_norm": 0.304258793592453,
64
+ "learning_rate": 0.00019507200355689026,
65
+ "loss": 1.392717456817627,
66
+ "step": 80
67
+ },
68
+ {
69
+ "epoch": 0.8,
70
+ "grad_norm": 0.33903768658638,
71
+ "learning_rate": 0.00019114703654091961,
72
+ "loss": 1.382735824584961,
73
+ "step": 90
74
+ },
75
+ {
76
+ "epoch": 0.8888888888888888,
77
+ "grad_norm": 0.35391494631767273,
78
+ "learning_rate": 0.00018614605309577137,
79
+ "loss": 1.3719289779663086,
80
+ "step": 100
81
+ },
82
+ {
83
+ "epoch": 0.8888888888888888,
84
+ "eval_loss": 1.350045919418335,
85
+ "eval_runtime": 8.3527,
86
+ "eval_samples_per_second": 23.944,
87
+ "eval_steps_per_second": 5.986,
88
+ "step": 100
89
+ }
90
+ ],
91
+ "logging_steps": 10,
92
+ "max_steps": 339,
93
+ "num_input_tokens_seen": 0,
94
+ "num_train_epochs": 3,
95
+ "save_steps": 100,
96
+ "stateful_callbacks": {
97
+ "TrainerControl": {
98
+ "args": {
99
+ "should_epoch_stop": false,
100
+ "should_evaluate": false,
101
+ "should_log": false,
102
+ "should_save": true,
103
+ "should_training_stop": false
104
+ },
105
+ "attributes": {}
106
+ }
107
+ },
108
+ "total_flos": 1765916751802368.0,
109
+ "train_batch_size": 4,
110
+ "trial_name": null,
111
+ "trial_params": null
112
+ }
lora-adapter/checkpoint-100/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bed2516133b47b8ec32a3f5147a39523682e8fb734ef7e3dafef29759653d26
3
+ size 5713
lora-adapter/checkpoint-200/README.md ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: unsloth/SmolLM2-1.7B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:unsloth/SmolLM2-1.7B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ - unsloth
12
+ ---
13
+
14
+ # Model Card for Model ID
15
+
16
+ <!-- Provide a quick summary of what the model is/does. -->
17
+
18
+
19
+
20
+ ## Model Details
21
+
22
+ ### Model Description
23
+
24
+ <!-- Provide a longer summary of what this model is. -->
25
+
26
+
27
+
28
+ - **Developed by:** [More Information Needed]
29
+ - **Funded by [optional]:** [More Information Needed]
30
+ - **Shared by [optional]:** [More Information Needed]
31
+ - **Model type:** [More Information Needed]
32
+ - **Language(s) (NLP):** [More Information Needed]
33
+ - **License:** [More Information Needed]
34
+ - **Finetuned from model [optional]:** [More Information Needed]
35
+
36
+ ### Model Sources [optional]
37
+
38
+ <!-- Provide the basic links for the model. -->
39
+
40
+ - **Repository:** [More Information Needed]
41
+ - **Paper [optional]:** [More Information Needed]
42
+ - **Demo [optional]:** [More Information Needed]
43
+
44
+ ## Uses
45
+
46
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
47
+
48
+ ### Direct Use
49
+
50
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
51
+
52
+ [More Information Needed]
53
+
54
+ ### Downstream Use [optional]
55
+
56
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
57
+
58
+ [More Information Needed]
59
+
60
+ ### Out-of-Scope Use
61
+
62
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
63
+
64
+ [More Information Needed]
65
+
66
+ ## Bias, Risks, and Limitations
67
+
68
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
69
+
70
+ [More Information Needed]
71
+
72
+ ### Recommendations
73
+
74
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
75
+
76
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
77
+
78
+ ## How to Get Started with the Model
79
+
80
+ Use the code below to get started with the model.
81
+
82
+ [More Information Needed]
83
+
84
+ ## Training Details
85
+
86
+ ### Training Data
87
+
88
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
89
+
90
+ [More Information Needed]
91
+
92
+ ### Training Procedure
93
+
94
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
95
+
96
+ #### Preprocessing [optional]
97
+
98
+ [More Information Needed]
99
+
100
+
101
+ #### Training Hyperparameters
102
+
103
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
104
+
105
+ #### Speeds, Sizes, Times [optional]
106
+
107
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
108
+
109
+ [More Information Needed]
110
+
111
+ ## Evaluation
112
+
113
+ <!-- This section describes the evaluation protocols and provides the results. -->
114
+
115
+ ### Testing Data, Factors & Metrics
116
+
117
+ #### Testing Data
118
+
119
+ <!-- This should link to a Dataset Card if possible. -->
120
+
121
+ [More Information Needed]
122
+
123
+ #### Factors
124
+
125
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
126
+
127
+ [More Information Needed]
128
+
129
+ #### Metrics
130
+
131
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
132
+
133
+ [More Information Needed]
134
+
135
+ ### Results
136
+
137
+ [More Information Needed]
138
+
139
+ #### Summary
140
+
141
+
142
+
143
+ ## Model Examination [optional]
144
+
145
+ <!-- Relevant interpretability work for the model goes here -->
146
+
147
+ [More Information Needed]
148
+
149
+ ## Environmental Impact
150
+
151
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
152
+
153
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
154
+
155
+ - **Hardware Type:** [More Information Needed]
156
+ - **Hours used:** [More Information Needed]
157
+ - **Cloud Provider:** [More Information Needed]
158
+ - **Compute Region:** [More Information Needed]
159
+ - **Carbon Emitted:** [More Information Needed]
160
+
161
+ ## Technical Specifications [optional]
162
+
163
+ ### Model Architecture and Objective
164
+
165
+ [More Information Needed]
166
+
167
+ ### Compute Infrastructure
168
+
169
+ [More Information Needed]
170
+
171
+ #### Hardware
172
+
173
+ [More Information Needed]
174
+
175
+ #### Software
176
+
177
+ [More Information Needed]
178
+
179
+ ## Citation [optional]
180
+
181
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
182
+
183
+ **BibTeX:**
184
+
185
+ [More Information Needed]
186
+
187
+ **APA:**
188
+
189
+ [More Information Needed]
190
+
191
+ ## Glossary [optional]
192
+
193
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
194
+
195
+ [More Information Needed]
196
+
197
+ ## More Information [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Authors [optional]
202
+
203
+ [More Information Needed]
204
+
205
+ ## Model Card Contact
206
+
207
+ [More Information Needed]
208
+ ### Framework versions
209
+
210
+ - PEFT 0.19.1
lora-adapter/checkpoint-200/adapter_config.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "LlamaForCausalLM",
7
+ "parent_library": "transformers.models.llama.modeling_llama",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "unsloth/SmolLM2-1.7B-Instruct",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "layer_replication": null,
20
+ "layers_pattern": null,
21
+ "layers_to_transform": null,
22
+ "loftq_config": {},
23
+ "lora_alpha": 32,
24
+ "lora_bias": false,
25
+ "lora_dropout": 0.05,
26
+ "lora_ga_config": null,
27
+ "megatron_config": null,
28
+ "megatron_core": "megatron.core",
29
+ "modules_to_save": null,
30
+ "peft_type": "LORA",
31
+ "peft_version": "0.19.1",
32
+ "qalora_group_size": 16,
33
+ "r": 16,
34
+ "rank_pattern": {},
35
+ "revision": null,
36
+ "target_modules": [
37
+ "down_proj",
38
+ "up_proj",
39
+ "k_proj",
40
+ "q_proj",
41
+ "v_proj",
42
+ "o_proj",
43
+ "gate_proj"
44
+ ],
45
+ "target_parameters": null,
46
+ "task_type": "CAUSAL_LM",
47
+ "trainable_token_indices": null,
48
+ "use_bdlora": null,
49
+ "use_dora": false,
50
+ "use_qalora": false,
51
+ "use_rslora": false
52
+ }
lora-adapter/checkpoint-200/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:712e1ec776aca02cc0fd090f39729a2f53fa7ae3b457ee3fb8ca9cf084876c7c
3
+ size 72396376
lora-adapter/checkpoint-200/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
lora-adapter/checkpoint-200/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca61da4fbe99c680987b59c751b7c16873e6f394c41648845162cea242772ed5
3
+ size 37138213
lora-adapter/checkpoint-200/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:386d6f2a299e7266f793e6f69aa7e72d835903558495ca678158c0a5b030508d
3
+ size 14645
lora-adapter/checkpoint-200/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:124625e167eb28acbfc793cfcb3e8a08b32e7fea06501462bc9e420a5e1beb2a
3
+ size 1383
lora-adapter/checkpoint-200/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22ce57635b2c45eb8e1f8167e5dc87b749a7080debef60762a49549291d95cb1
3
+ size 1465
lora-adapter/checkpoint-200/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
lora-adapter/checkpoint-200/tokenizer_config.json ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|im_start|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>"
11
+ ],
12
+ "is_local": false,
13
+ "model_max_length": 8192,
14
+ "pad_token": "<|PAD_TOKEN|>",
15
+ "padding_side": "right",
16
+ "tokenizer_class": "GPT2Tokenizer",
17
+ "unk_token": "�",
18
+ "vocab_size": 49152,
19
+ "added_tokens_decoder": {
20
+ "0": {
21
+ "content": "<|endoftext|>",
22
+ "single_word": false,
23
+ "lstrip": false,
24
+ "rstrip": false,
25
+ "normalized": false,
26
+ "special": true
27
+ },
28
+ "1": {
29
+ "content": "<|im_start|>",
30
+ "single_word": false,
31
+ "lstrip": false,
32
+ "rstrip": false,
33
+ "normalized": false,
34
+ "special": true
35
+ },
36
+ "2": {
37
+ "content": "<|im_end|>",
38
+ "single_word": false,
39
+ "lstrip": false,
40
+ "rstrip": false,
41
+ "normalized": false,
42
+ "special": true
43
+ },
44
+ "3": {
45
+ "content": "<repo_name>",
46
+ "single_word": false,
47
+ "lstrip": false,
48
+ "rstrip": false,
49
+ "normalized": false,
50
+ "special": true
51
+ },
52
+ "4": {
53
+ "content": "<reponame>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ "5": {
61
+ "content": "<file_sep>",
62
+ "single_word": false,
63
+ "lstrip": false,
64
+ "rstrip": false,
65
+ "normalized": false,
66
+ "special": true
67
+ },
68
+ "6": {
69
+ "content": "<filename>",
70
+ "single_word": false,
71
+ "lstrip": false,
72
+ "rstrip": false,
73
+ "normalized": false,
74
+ "special": true
75
+ },
76
+ "7": {
77
+ "content": "<gh_stars>",
78
+ "single_word": false,
79
+ "lstrip": false,
80
+ "rstrip": false,
81
+ "normalized": false,
82
+ "special": true
83
+ },
84
+ "8": {
85
+ "content": "<issue_start>",
86
+ "single_word": false,
87
+ "lstrip": false,
88
+ "rstrip": false,
89
+ "normalized": false,
90
+ "special": true
91
+ },
92
+ "9": {
93
+ "content": "<issue_comment>",
94
+ "single_word": false,
95
+ "lstrip": false,
96
+ "rstrip": false,
97
+ "normalized": false,
98
+ "special": true
99
+ },
100
+ "10": {
101
+ "content": "<issue_closed>",
102
+ "single_word": false,
103
+ "lstrip": false,
104
+ "rstrip": false,
105
+ "normalized": false,
106
+ "special": true
107
+ },
108
+ "11": {
109
+ "content": "<jupyter_start>",
110
+ "single_word": false,
111
+ "lstrip": false,
112
+ "rstrip": false,
113
+ "normalized": false,
114
+ "special": true
115
+ },
116
+ "12": {
117
+ "content": "<jupyter_text>",
118
+ "single_word": false,
119
+ "lstrip": false,
120
+ "rstrip": false,
121
+ "normalized": false,
122
+ "special": true
123
+ },
124
+ "13": {
125
+ "content": "<jupyter_code>",
126
+ "single_word": false,
127
+ "lstrip": false,
128
+ "rstrip": false,
129
+ "normalized": false,
130
+ "special": true
131
+ },
132
+ "14": {
133
+ "content": "<jupyter_output>",
134
+ "single_word": false,
135
+ "lstrip": false,
136
+ "rstrip": false,
137
+ "normalized": false,
138
+ "special": true
139
+ },
140
+ "15": {
141
+ "content": "<jupyter_script>",
142
+ "single_word": false,
143
+ "lstrip": false,
144
+ "rstrip": false,
145
+ "normalized": false,
146
+ "special": true
147
+ },
148
+ "16": {
149
+ "content": "<empty_output>",
150
+ "single_word": false,
151
+ "lstrip": false,
152
+ "rstrip": false,
153
+ "normalized": false,
154
+ "special": true
155
+ },
156
+ "24211": {
157
+ "content": "�",
158
+ "single_word": false,
159
+ "lstrip": false,
160
+ "rstrip": false,
161
+ "normalized": false,
162
+ "special": true
163
+ },
164
+ "49152": {
165
+ "content": "<|PAD_TOKEN|>",
166
+ "single_word": false,
167
+ "lstrip": false,
168
+ "rstrip": false,
169
+ "normalized": false,
170
+ "special": true
171
+ }
172
+ }
173
+ }
lora-adapter/checkpoint-200/trainer_state.json ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.7733333333333334,
6
+ "eval_steps": 100,
7
+ "global_step": 200,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.08888888888888889,
14
+ "grad_norm": 0.6075066328048706,
15
+ "learning_rate": 3.6e-05,
16
+ "loss": 2.435858154296875,
17
+ "step": 10
18
+ },
19
+ {
20
+ "epoch": 0.17777777777777778,
21
+ "grad_norm": 0.5337297320365906,
22
+ "learning_rate": 7.6e-05,
23
+ "loss": 2.261408805847168,
24
+ "step": 20
25
+ },
26
+ {
27
+ "epoch": 0.26666666666666666,
28
+ "grad_norm": 4.804379463195801,
29
+ "learning_rate": 0.000116,
30
+ "loss": 1.9577768325805665,
31
+ "step": 30
32
+ },
33
+ {
34
+ "epoch": 0.35555555555555557,
35
+ "grad_norm": 0.39172491431236267,
36
+ "learning_rate": 0.00015600000000000002,
37
+ "loss": 1.6207334518432617,
38
+ "step": 40
39
+ },
40
+ {
41
+ "epoch": 0.4444444444444444,
42
+ "grad_norm": 0.31786638498306274,
43
+ "learning_rate": 0.000196,
44
+ "loss": 1.5145363807678223,
45
+ "step": 50
46
+ },
47
+ {
48
+ "epoch": 0.5333333333333333,
49
+ "grad_norm": 0.295754611492157,
50
+ "learning_rate": 0.00019952179661709028,
51
+ "loss": 1.5010628700256348,
52
+ "step": 60
53
+ },
54
+ {
55
+ "epoch": 0.6222222222222222,
56
+ "grad_norm": 0.29678136110305786,
57
+ "learning_rate": 0.00019787461880296965,
58
+ "loss": 1.4191683769226073,
59
+ "step": 70
60
+ },
61
+ {
62
+ "epoch": 0.7111111111111111,
63
+ "grad_norm": 0.304258793592453,
64
+ "learning_rate": 0.00019507200355689026,
65
+ "loss": 1.392717456817627,
66
+ "step": 80
67
+ },
68
+ {
69
+ "epoch": 0.8,
70
+ "grad_norm": 0.33903768658638,
71
+ "learning_rate": 0.00019114703654091961,
72
+ "loss": 1.382735824584961,
73
+ "step": 90
74
+ },
75
+ {
76
+ "epoch": 0.8888888888888888,
77
+ "grad_norm": 0.35391494631767273,
78
+ "learning_rate": 0.00018614605309577137,
79
+ "loss": 1.3719289779663086,
80
+ "step": 100
81
+ },
82
+ {
83
+ "epoch": 0.8888888888888888,
84
+ "eval_loss": 1.350045919418335,
85
+ "eval_runtime": 8.3527,
86
+ "eval_samples_per_second": 23.944,
87
+ "eval_steps_per_second": 5.986,
88
+ "step": 100
89
+ },
90
+ {
91
+ "epoch": 0.9777777777777777,
92
+ "grad_norm": 0.3610546290874481,
93
+ "learning_rate": 0.00018012809123906228,
94
+ "loss": 1.3875890731811524,
95
+ "step": 110
96
+ },
97
+ {
98
+ "epoch": 1.0622222222222222,
99
+ "grad_norm": 0.3689529001712799,
100
+ "learning_rate": 0.00017316419470489209,
101
+ "loss": 1.3298458099365233,
102
+ "step": 120
103
+ },
104
+ {
105
+ "epoch": 1.1511111111111112,
106
+ "grad_norm": 0.3758569657802582,
107
+ "learning_rate": 0.00016533657425255952,
108
+ "loss": 1.2853054046630858,
109
+ "step": 130
110
+ },
111
+ {
112
+ "epoch": 1.24,
113
+ "grad_norm": 0.4089072048664093,
114
+ "learning_rate": 0.00015673763714540214,
115
+ "loss": 1.302170467376709,
116
+ "step": 140
117
+ },
118
+ {
119
+ "epoch": 1.3288888888888888,
120
+ "grad_norm": 0.47941380739212036,
121
+ "learning_rate": 0.00014746889625703502,
122
+ "loss": 1.2683821678161622,
123
+ "step": 150
124
+ },
125
+ {
126
+ "epoch": 1.4177777777777778,
127
+ "grad_norm": 0.4418100416660309,
128
+ "learning_rate": 0.0001376397716832963,
129
+ "loss": 1.2893789291381836,
130
+ "step": 160
131
+ },
132
+ {
133
+ "epoch": 1.5066666666666668,
134
+ "grad_norm": 0.4881313741207123,
135
+ "learning_rate": 0.0001273662990072083,
136
+ "loss": 1.2691761016845704,
137
+ "step": 170
138
+ },
139
+ {
140
+ "epoch": 1.5955555555555554,
141
+ "grad_norm": 0.4534631371498108,
142
+ "learning_rate": 0.00011676975946624944,
143
+ "loss": 1.2686445236206054,
144
+ "step": 180
145
+ },
146
+ {
147
+ "epoch": 1.6844444444444444,
148
+ "grad_norm": 0.4390407204627991,
149
+ "learning_rate": 0.0001059752481931988,
150
+ "loss": 1.2629841804504394,
151
+ "step": 190
152
+ },
153
+ {
154
+ "epoch": 1.7733333333333334,
155
+ "grad_norm": 0.474439799785614,
156
+ "learning_rate": 9.511019743287242e-05,
157
+ "loss": 1.289689540863037,
158
+ "step": 200
159
+ },
160
+ {
161
+ "epoch": 1.7733333333333334,
162
+ "eval_loss": 1.287023901939392,
163
+ "eval_runtime": 7.9351,
164
+ "eval_samples_per_second": 25.205,
165
+ "eval_steps_per_second": 6.301,
166
+ "step": 200
167
+ }
168
+ ],
169
+ "logging_steps": 10,
170
+ "max_steps": 339,
171
+ "num_input_tokens_seen": 0,
172
+ "num_train_epochs": 3,
173
+ "save_steps": 100,
174
+ "stateful_callbacks": {
175
+ "TrainerControl": {
176
+ "args": {
177
+ "should_epoch_stop": false,
178
+ "should_evaluate": false,
179
+ "should_log": false,
180
+ "should_save": true,
181
+ "should_training_stop": false
182
+ },
183
+ "attributes": {}
184
+ }
185
+ },
186
+ "total_flos": 3526390050582528.0,
187
+ "train_batch_size": 4,
188
+ "trial_name": null,
189
+ "trial_params": null
190
+ }
lora-adapter/checkpoint-200/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bed2516133b47b8ec32a3f5147a39523682e8fb734ef7e3dafef29759653d26
3
+ size 5713
lora-adapter/checkpoint-300/README.md ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: unsloth/SmolLM2-1.7B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:unsloth/SmolLM2-1.7B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ - unsloth
12
+ ---
13
+
14
+ # Model Card for Model ID
15
+
16
+ <!-- Provide a quick summary of what the model is/does. -->
17
+
18
+
19
+
20
+ ## Model Details
21
+
22
+ ### Model Description
23
+
24
+ <!-- Provide a longer summary of what this model is. -->
25
+
26
+
27
+
28
+ - **Developed by:** [More Information Needed]
29
+ - **Funded by [optional]:** [More Information Needed]
30
+ - **Shared by [optional]:** [More Information Needed]
31
+ - **Model type:** [More Information Needed]
32
+ - **Language(s) (NLP):** [More Information Needed]
33
+ - **License:** [More Information Needed]
34
+ - **Finetuned from model [optional]:** [More Information Needed]
35
+
36
+ ### Model Sources [optional]
37
+
38
+ <!-- Provide the basic links for the model. -->
39
+
40
+ - **Repository:** [More Information Needed]
41
+ - **Paper [optional]:** [More Information Needed]
42
+ - **Demo [optional]:** [More Information Needed]
43
+
44
+ ## Uses
45
+
46
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
47
+
48
+ ### Direct Use
49
+
50
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
51
+
52
+ [More Information Needed]
53
+
54
+ ### Downstream Use [optional]
55
+
56
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
57
+
58
+ [More Information Needed]
59
+
60
+ ### Out-of-Scope Use
61
+
62
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
63
+
64
+ [More Information Needed]
65
+
66
+ ## Bias, Risks, and Limitations
67
+
68
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
69
+
70
+ [More Information Needed]
71
+
72
+ ### Recommendations
73
+
74
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
75
+
76
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
77
+
78
+ ## How to Get Started with the Model
79
+
80
+ Use the code below to get started with the model.
81
+
82
+ [More Information Needed]
83
+
84
+ ## Training Details
85
+
86
+ ### Training Data
87
+
88
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
89
+
90
+ [More Information Needed]
91
+
92
+ ### Training Procedure
93
+
94
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
95
+
96
+ #### Preprocessing [optional]
97
+
98
+ [More Information Needed]
99
+
100
+
101
+ #### Training Hyperparameters
102
+
103
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
104
+
105
+ #### Speeds, Sizes, Times [optional]
106
+
107
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
108
+
109
+ [More Information Needed]
110
+
111
+ ## Evaluation
112
+
113
+ <!-- This section describes the evaluation protocols and provides the results. -->
114
+
115
+ ### Testing Data, Factors & Metrics
116
+
117
+ #### Testing Data
118
+
119
+ <!-- This should link to a Dataset Card if possible. -->
120
+
121
+ [More Information Needed]
122
+
123
+ #### Factors
124
+
125
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
126
+
127
+ [More Information Needed]
128
+
129
+ #### Metrics
130
+
131
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
132
+
133
+ [More Information Needed]
134
+
135
+ ### Results
136
+
137
+ [More Information Needed]
138
+
139
+ #### Summary
140
+
141
+
142
+
143
+ ## Model Examination [optional]
144
+
145
+ <!-- Relevant interpretability work for the model goes here -->
146
+
147
+ [More Information Needed]
148
+
149
+ ## Environmental Impact
150
+
151
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
152
+
153
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
154
+
155
+ - **Hardware Type:** [More Information Needed]
156
+ - **Hours used:** [More Information Needed]
157
+ - **Cloud Provider:** [More Information Needed]
158
+ - **Compute Region:** [More Information Needed]
159
+ - **Carbon Emitted:** [More Information Needed]
160
+
161
+ ## Technical Specifications [optional]
162
+
163
+ ### Model Architecture and Objective
164
+
165
+ [More Information Needed]
166
+
167
+ ### Compute Infrastructure
168
+
169
+ [More Information Needed]
170
+
171
+ #### Hardware
172
+
173
+ [More Information Needed]
174
+
175
+ #### Software
176
+
177
+ [More Information Needed]
178
+
179
+ ## Citation [optional]
180
+
181
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
182
+
183
+ **BibTeX:**
184
+
185
+ [More Information Needed]
186
+
187
+ **APA:**
188
+
189
+ [More Information Needed]
190
+
191
+ ## Glossary [optional]
192
+
193
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
194
+
195
+ [More Information Needed]
196
+
197
+ ## More Information [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Authors [optional]
202
+
203
+ [More Information Needed]
204
+
205
+ ## Model Card Contact
206
+
207
+ [More Information Needed]
208
+ ### Framework versions
209
+
210
+ - PEFT 0.19.1
lora-adapter/checkpoint-300/adapter_config.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "LlamaForCausalLM",
7
+ "parent_library": "transformers.models.llama.modeling_llama",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "unsloth/SmolLM2-1.7B-Instruct",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "layer_replication": null,
20
+ "layers_pattern": null,
21
+ "layers_to_transform": null,
22
+ "loftq_config": {},
23
+ "lora_alpha": 32,
24
+ "lora_bias": false,
25
+ "lora_dropout": 0.05,
26
+ "lora_ga_config": null,
27
+ "megatron_config": null,
28
+ "megatron_core": "megatron.core",
29
+ "modules_to_save": null,
30
+ "peft_type": "LORA",
31
+ "peft_version": "0.19.1",
32
+ "qalora_group_size": 16,
33
+ "r": 16,
34
+ "rank_pattern": {},
35
+ "revision": null,
36
+ "target_modules": [
37
+ "down_proj",
38
+ "up_proj",
39
+ "k_proj",
40
+ "q_proj",
41
+ "v_proj",
42
+ "o_proj",
43
+ "gate_proj"
44
+ ],
45
+ "target_parameters": null,
46
+ "task_type": "CAUSAL_LM",
47
+ "trainable_token_indices": null,
48
+ "use_bdlora": null,
49
+ "use_dora": false,
50
+ "use_qalora": false,
51
+ "use_rslora": false
52
+ }
lora-adapter/checkpoint-300/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a02f0c0fb106cec5dbc7c2b2a2af2bd519a9b3f1022361d77cd942a32d5557d
3
+ size 72396376
lora-adapter/checkpoint-300/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
lora-adapter/checkpoint-300/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab4ebb1da3e2781a6efe0818b699571a48ac9c055f433b18e2ec120959d2e8b8
3
+ size 37138533
lora-adapter/checkpoint-300/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9d307e55f3852c3fa8d182849317daceacf7fe27c41d45820ebff2697d0c3f1
3
+ size 14645
lora-adapter/checkpoint-300/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30af866df24edce708e1eb20700878b402fa05707fa9bc5f332496baf440dbbb
3
+ size 1383
lora-adapter/checkpoint-300/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a2a5ab80b29d25c584a7b76208f40add5bb7706f7fde31b418293b813f37addd
3
+ size 1465
lora-adapter/checkpoint-300/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
lora-adapter/checkpoint-300/tokenizer_config.json ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|im_start|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>"
11
+ ],
12
+ "is_local": false,
13
+ "model_max_length": 8192,
14
+ "pad_token": "<|PAD_TOKEN|>",
15
+ "padding_side": "right",
16
+ "tokenizer_class": "GPT2Tokenizer",
17
+ "unk_token": "�",
18
+ "vocab_size": 49152,
19
+ "added_tokens_decoder": {
20
+ "0": {
21
+ "content": "<|endoftext|>",
22
+ "single_word": false,
23
+ "lstrip": false,
24
+ "rstrip": false,
25
+ "normalized": false,
26
+ "special": true
27
+ },
28
+ "1": {
29
+ "content": "<|im_start|>",
30
+ "single_word": false,
31
+ "lstrip": false,
32
+ "rstrip": false,
33
+ "normalized": false,
34
+ "special": true
35
+ },
36
+ "2": {
37
+ "content": "<|im_end|>",
38
+ "single_word": false,
39
+ "lstrip": false,
40
+ "rstrip": false,
41
+ "normalized": false,
42
+ "special": true
43
+ },
44
+ "3": {
45
+ "content": "<repo_name>",
46
+ "single_word": false,
47
+ "lstrip": false,
48
+ "rstrip": false,
49
+ "normalized": false,
50
+ "special": true
51
+ },
52
+ "4": {
53
+ "content": "<reponame>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ "5": {
61
+ "content": "<file_sep>",
62
+ "single_word": false,
63
+ "lstrip": false,
64
+ "rstrip": false,
65
+ "normalized": false,
66
+ "special": true
67
+ },
68
+ "6": {
69
+ "content": "<filename>",
70
+ "single_word": false,
71
+ "lstrip": false,
72
+ "rstrip": false,
73
+ "normalized": false,
74
+ "special": true
75
+ },
76
+ "7": {
77
+ "content": "<gh_stars>",
78
+ "single_word": false,
79
+ "lstrip": false,
80
+ "rstrip": false,
81
+ "normalized": false,
82
+ "special": true
83
+ },
84
+ "8": {
85
+ "content": "<issue_start>",
86
+ "single_word": false,
87
+ "lstrip": false,
88
+ "rstrip": false,
89
+ "normalized": false,
90
+ "special": true
91
+ },
92
+ "9": {
93
+ "content": "<issue_comment>",
94
+ "single_word": false,
95
+ "lstrip": false,
96
+ "rstrip": false,
97
+ "normalized": false,
98
+ "special": true
99
+ },
100
+ "10": {
101
+ "content": "<issue_closed>",
102
+ "single_word": false,
103
+ "lstrip": false,
104
+ "rstrip": false,
105
+ "normalized": false,
106
+ "special": true
107
+ },
108
+ "11": {
109
+ "content": "<jupyter_start>",
110
+ "single_word": false,
111
+ "lstrip": false,
112
+ "rstrip": false,
113
+ "normalized": false,
114
+ "special": true
115
+ },
116
+ "12": {
117
+ "content": "<jupyter_text>",
118
+ "single_word": false,
119
+ "lstrip": false,
120
+ "rstrip": false,
121
+ "normalized": false,
122
+ "special": true
123
+ },
124
+ "13": {
125
+ "content": "<jupyter_code>",
126
+ "single_word": false,
127
+ "lstrip": false,
128
+ "rstrip": false,
129
+ "normalized": false,
130
+ "special": true
131
+ },
132
+ "14": {
133
+ "content": "<jupyter_output>",
134
+ "single_word": false,
135
+ "lstrip": false,
136
+ "rstrip": false,
137
+ "normalized": false,
138
+ "special": true
139
+ },
140
+ "15": {
141
+ "content": "<jupyter_script>",
142
+ "single_word": false,
143
+ "lstrip": false,
144
+ "rstrip": false,
145
+ "normalized": false,
146
+ "special": true
147
+ },
148
+ "16": {
149
+ "content": "<empty_output>",
150
+ "single_word": false,
151
+ "lstrip": false,
152
+ "rstrip": false,
153
+ "normalized": false,
154
+ "special": true
155
+ },
156
+ "24211": {
157
+ "content": "�",
158
+ "single_word": false,
159
+ "lstrip": false,
160
+ "rstrip": false,
161
+ "normalized": false,
162
+ "special": true
163
+ },
164
+ "49152": {
165
+ "content": "<|PAD_TOKEN|>",
166
+ "single_word": false,
167
+ "lstrip": false,
168
+ "rstrip": false,
169
+ "normalized": false,
170
+ "special": true
171
+ }
172
+ }
173
+ }
lora-adapter/checkpoint-300/trainer_state.json ADDED
@@ -0,0 +1,268 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 2.6577777777777776,
6
+ "eval_steps": 100,
7
+ "global_step": 300,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.08888888888888889,
14
+ "grad_norm": 0.6075066328048706,
15
+ "learning_rate": 3.6e-05,
16
+ "loss": 2.435858154296875,
17
+ "step": 10
18
+ },
19
+ {
20
+ "epoch": 0.17777777777777778,
21
+ "grad_norm": 0.5337297320365906,
22
+ "learning_rate": 7.6e-05,
23
+ "loss": 2.261408805847168,
24
+ "step": 20
25
+ },
26
+ {
27
+ "epoch": 0.26666666666666666,
28
+ "grad_norm": 4.804379463195801,
29
+ "learning_rate": 0.000116,
30
+ "loss": 1.9577768325805665,
31
+ "step": 30
32
+ },
33
+ {
34
+ "epoch": 0.35555555555555557,
35
+ "grad_norm": 0.39172491431236267,
36
+ "learning_rate": 0.00015600000000000002,
37
+ "loss": 1.6207334518432617,
38
+ "step": 40
39
+ },
40
+ {
41
+ "epoch": 0.4444444444444444,
42
+ "grad_norm": 0.31786638498306274,
43
+ "learning_rate": 0.000196,
44
+ "loss": 1.5145363807678223,
45
+ "step": 50
46
+ },
47
+ {
48
+ "epoch": 0.5333333333333333,
49
+ "grad_norm": 0.295754611492157,
50
+ "learning_rate": 0.00019952179661709028,
51
+ "loss": 1.5010628700256348,
52
+ "step": 60
53
+ },
54
+ {
55
+ "epoch": 0.6222222222222222,
56
+ "grad_norm": 0.29678136110305786,
57
+ "learning_rate": 0.00019787461880296965,
58
+ "loss": 1.4191683769226073,
59
+ "step": 70
60
+ },
61
+ {
62
+ "epoch": 0.7111111111111111,
63
+ "grad_norm": 0.304258793592453,
64
+ "learning_rate": 0.00019507200355689026,
65
+ "loss": 1.392717456817627,
66
+ "step": 80
67
+ },
68
+ {
69
+ "epoch": 0.8,
70
+ "grad_norm": 0.33903768658638,
71
+ "learning_rate": 0.00019114703654091961,
72
+ "loss": 1.382735824584961,
73
+ "step": 90
74
+ },
75
+ {
76
+ "epoch": 0.8888888888888888,
77
+ "grad_norm": 0.35391494631767273,
78
+ "learning_rate": 0.00018614605309577137,
79
+ "loss": 1.3719289779663086,
80
+ "step": 100
81
+ },
82
+ {
83
+ "epoch": 0.8888888888888888,
84
+ "eval_loss": 1.350045919418335,
85
+ "eval_runtime": 8.3527,
86
+ "eval_samples_per_second": 23.944,
87
+ "eval_steps_per_second": 5.986,
88
+ "step": 100
89
+ },
90
+ {
91
+ "epoch": 0.9777777777777777,
92
+ "grad_norm": 0.3610546290874481,
93
+ "learning_rate": 0.00018012809123906228,
94
+ "loss": 1.3875890731811524,
95
+ "step": 110
96
+ },
97
+ {
98
+ "epoch": 1.0622222222222222,
99
+ "grad_norm": 0.3689529001712799,
100
+ "learning_rate": 0.00017316419470489209,
101
+ "loss": 1.3298458099365233,
102
+ "step": 120
103
+ },
104
+ {
105
+ "epoch": 1.1511111111111112,
106
+ "grad_norm": 0.3758569657802582,
107
+ "learning_rate": 0.00016533657425255952,
108
+ "loss": 1.2853054046630858,
109
+ "step": 130
110
+ },
111
+ {
112
+ "epoch": 1.24,
113
+ "grad_norm": 0.4089072048664093,
114
+ "learning_rate": 0.00015673763714540214,
115
+ "loss": 1.302170467376709,
116
+ "step": 140
117
+ },
118
+ {
119
+ "epoch": 1.3288888888888888,
120
+ "grad_norm": 0.47941380739212036,
121
+ "learning_rate": 0.00014746889625703502,
122
+ "loss": 1.2683821678161622,
123
+ "step": 150
124
+ },
125
+ {
126
+ "epoch": 1.4177777777777778,
127
+ "grad_norm": 0.4418100416660309,
128
+ "learning_rate": 0.0001376397716832963,
129
+ "loss": 1.2893789291381836,
130
+ "step": 160
131
+ },
132
+ {
133
+ "epoch": 1.5066666666666668,
134
+ "grad_norm": 0.4881313741207123,
135
+ "learning_rate": 0.0001273662990072083,
136
+ "loss": 1.2691761016845704,
137
+ "step": 170
138
+ },
139
+ {
140
+ "epoch": 1.5955555555555554,
141
+ "grad_norm": 0.4534631371498108,
142
+ "learning_rate": 0.00011676975946624944,
143
+ "loss": 1.2686445236206054,
144
+ "step": 180
145
+ },
146
+ {
147
+ "epoch": 1.6844444444444444,
148
+ "grad_norm": 0.4390407204627991,
149
+ "learning_rate": 0.0001059752481931988,
150
+ "loss": 1.2629841804504394,
151
+ "step": 190
152
+ },
153
+ {
154
+ "epoch": 1.7733333333333334,
155
+ "grad_norm": 0.474439799785614,
156
+ "learning_rate": 9.511019743287242e-05,
157
+ "loss": 1.289689540863037,
158
+ "step": 200
159
+ },
160
+ {
161
+ "epoch": 1.7733333333333334,
162
+ "eval_loss": 1.287023901939392,
163
+ "eval_runtime": 7.9351,
164
+ "eval_samples_per_second": 25.205,
165
+ "eval_steps_per_second": 6.301,
166
+ "step": 200
167
+ },
168
+ {
169
+ "epoch": 1.8622222222222222,
170
+ "grad_norm": 0.6423256993293762,
171
+ "learning_rate": 8.430287216859443e-05,
172
+ "loss": 1.2817410469055175,
173
+ "step": 210
174
+ },
175
+ {
176
+ "epoch": 1.951111111111111,
177
+ "grad_norm": 0.5383001565933228,
178
+ "learning_rate": 7.368085591795522e-05,
179
+ "loss": 1.228531837463379,
180
+ "step": 220
181
+ },
182
+ {
183
+ "epoch": 2.0355555555555553,
184
+ "grad_norm": 0.5520081520080566,
185
+ "learning_rate": 6.336954457346462e-05,
186
+ "loss": 1.196599292755127,
187
+ "step": 230
188
+ },
189
+ {
190
+ "epoch": 2.1244444444444444,
191
+ "grad_norm": 0.4924458861351013,
192
+ "learning_rate": 5.349066606873525e-05,
193
+ "loss": 1.150006103515625,
194
+ "step": 240
195
+ },
196
+ {
197
+ "epoch": 2.2133333333333334,
198
+ "grad_norm": 0.5595092177391052,
199
+ "learning_rate": 4.416084334595314e-05,
200
+ "loss": 1.1321084022521972,
201
+ "step": 250
202
+ },
203
+ {
204
+ "epoch": 2.3022222222222224,
205
+ "grad_norm": 0.5763599872589111,
206
+ "learning_rate": 3.549021758920955e-05,
207
+ "loss": 1.1533366203308106,
208
+ "step": 260
209
+ },
210
+ {
211
+ "epoch": 2.391111111111111,
212
+ "grad_norm": 0.5541089177131653,
213
+ "learning_rate": 2.758114797681215e-05,
214
+ "loss": 1.142934799194336,
215
+ "step": 270
216
+ },
217
+ {
218
+ "epoch": 2.48,
219
+ "grad_norm": 0.6457899808883667,
220
+ "learning_rate": 2.0527003302365412e-05,
221
+ "loss": 1.1384923934936524,
222
+ "step": 280
223
+ },
224
+ {
225
+ "epoch": 2.568888888888889,
226
+ "grad_norm": 0.5974424481391907,
227
+ "learning_rate": 1.4411059729873765e-05,
228
+ "loss": 1.140573215484619,
229
+ "step": 290
230
+ },
231
+ {
232
+ "epoch": 2.6577777777777776,
233
+ "grad_norm": 0.6066321134567261,
234
+ "learning_rate": 9.305517695178833e-06,
235
+ "loss": 1.175175094604492,
236
+ "step": 300
237
+ },
238
+ {
239
+ "epoch": 2.6577777777777776,
240
+ "eval_loss": 1.2747913599014282,
241
+ "eval_runtime": 7.9917,
242
+ "eval_samples_per_second": 25.026,
243
+ "eval_steps_per_second": 6.257,
244
+ "step": 300
245
+ }
246
+ ],
247
+ "logging_steps": 10,
248
+ "max_steps": 339,
249
+ "num_input_tokens_seen": 0,
250
+ "num_train_epochs": 3,
251
+ "save_steps": 100,
252
+ "stateful_callbacks": {
253
+ "TrainerControl": {
254
+ "args": {
255
+ "should_epoch_stop": false,
256
+ "should_evaluate": false,
257
+ "should_log": false,
258
+ "should_save": true,
259
+ "should_training_stop": false
260
+ },
261
+ "attributes": {}
262
+ }
263
+ },
264
+ "total_flos": 5287322671251456.0,
265
+ "train_batch_size": 4,
266
+ "trial_name": null,
267
+ "trial_params": null
268
+ }
lora-adapter/checkpoint-300/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bed2516133b47b8ec32a3f5147a39523682e8fb734ef7e3dafef29759653d26
3
+ size 5713
lora-adapter/checkpoint-339/README.md ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: unsloth/SmolLM2-1.7B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:unsloth/SmolLM2-1.7B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ - unsloth
12
+ ---
13
+
14
+ # Model Card for Model ID
15
+
16
+ <!-- Provide a quick summary of what the model is/does. -->
17
+
18
+
19
+
20
+ ## Model Details
21
+
22
+ ### Model Description
23
+
24
+ <!-- Provide a longer summary of what this model is. -->
25
+
26
+
27
+
28
+ - **Developed by:** [More Information Needed]
29
+ - **Funded by [optional]:** [More Information Needed]
30
+ - **Shared by [optional]:** [More Information Needed]
31
+ - **Model type:** [More Information Needed]
32
+ - **Language(s) (NLP):** [More Information Needed]
33
+ - **License:** [More Information Needed]
34
+ - **Finetuned from model [optional]:** [More Information Needed]
35
+
36
+ ### Model Sources [optional]
37
+
38
+ <!-- Provide the basic links for the model. -->
39
+
40
+ - **Repository:** [More Information Needed]
41
+ - **Paper [optional]:** [More Information Needed]
42
+ - **Demo [optional]:** [More Information Needed]
43
+
44
+ ## Uses
45
+
46
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
47
+
48
+ ### Direct Use
49
+
50
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
51
+
52
+ [More Information Needed]
53
+
54
+ ### Downstream Use [optional]
55
+
56
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
57
+
58
+ [More Information Needed]
59
+
60
+ ### Out-of-Scope Use
61
+
62
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
63
+
64
+ [More Information Needed]
65
+
66
+ ## Bias, Risks, and Limitations
67
+
68
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
69
+
70
+ [More Information Needed]
71
+
72
+ ### Recommendations
73
+
74
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
75
+
76
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
77
+
78
+ ## How to Get Started with the Model
79
+
80
+ Use the code below to get started with the model.
81
+
82
+ [More Information Needed]
83
+
84
+ ## Training Details
85
+
86
+ ### Training Data
87
+
88
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
89
+
90
+ [More Information Needed]
91
+
92
+ ### Training Procedure
93
+
94
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
95
+
96
+ #### Preprocessing [optional]
97
+
98
+ [More Information Needed]
99
+
100
+
101
+ #### Training Hyperparameters
102
+
103
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
104
+
105
+ #### Speeds, Sizes, Times [optional]
106
+
107
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
108
+
109
+ [More Information Needed]
110
+
111
+ ## Evaluation
112
+
113
+ <!-- This section describes the evaluation protocols and provides the results. -->
114
+
115
+ ### Testing Data, Factors & Metrics
116
+
117
+ #### Testing Data
118
+
119
+ <!-- This should link to a Dataset Card if possible. -->
120
+
121
+ [More Information Needed]
122
+
123
+ #### Factors
124
+
125
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
126
+
127
+ [More Information Needed]
128
+
129
+ #### Metrics
130
+
131
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
132
+
133
+ [More Information Needed]
134
+
135
+ ### Results
136
+
137
+ [More Information Needed]
138
+
139
+ #### Summary
140
+
141
+
142
+
143
+ ## Model Examination [optional]
144
+
145
+ <!-- Relevant interpretability work for the model goes here -->
146
+
147
+ [More Information Needed]
148
+
149
+ ## Environmental Impact
150
+
151
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
152
+
153
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
154
+
155
+ - **Hardware Type:** [More Information Needed]
156
+ - **Hours used:** [More Information Needed]
157
+ - **Cloud Provider:** [More Information Needed]
158
+ - **Compute Region:** [More Information Needed]
159
+ - **Carbon Emitted:** [More Information Needed]
160
+
161
+ ## Technical Specifications [optional]
162
+
163
+ ### Model Architecture and Objective
164
+
165
+ [More Information Needed]
166
+
167
+ ### Compute Infrastructure
168
+
169
+ [More Information Needed]
170
+
171
+ #### Hardware
172
+
173
+ [More Information Needed]
174
+
175
+ #### Software
176
+
177
+ [More Information Needed]
178
+
179
+ ## Citation [optional]
180
+
181
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
182
+
183
+ **BibTeX:**
184
+
185
+ [More Information Needed]
186
+
187
+ **APA:**
188
+
189
+ [More Information Needed]
190
+
191
+ ## Glossary [optional]
192
+
193
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
194
+
195
+ [More Information Needed]
196
+
197
+ ## More Information [optional]
198
+
199
+ [More Information Needed]
200
+
201
+ ## Model Card Authors [optional]
202
+
203
+ [More Information Needed]
204
+
205
+ ## Model Card Contact
206
+
207
+ [More Information Needed]
208
+ ### Framework versions
209
+
210
+ - PEFT 0.19.1
lora-adapter/checkpoint-339/adapter_config.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "LlamaForCausalLM",
7
+ "parent_library": "transformers.models.llama.modeling_llama",
8
+ "unsloth_fixed": true
9
+ },
10
+ "base_model_name_or_path": "unsloth/SmolLM2-1.7B-Instruct",
11
+ "bias": "none",
12
+ "corda_config": null,
13
+ "ensure_weight_tying": false,
14
+ "eva_config": null,
15
+ "exclude_modules": null,
16
+ "fan_in_fan_out": false,
17
+ "inference_mode": true,
18
+ "init_lora_weights": true,
19
+ "layer_replication": null,
20
+ "layers_pattern": null,
21
+ "layers_to_transform": null,
22
+ "loftq_config": {},
23
+ "lora_alpha": 32,
24
+ "lora_bias": false,
25
+ "lora_dropout": 0.05,
26
+ "lora_ga_config": null,
27
+ "megatron_config": null,
28
+ "megatron_core": "megatron.core",
29
+ "modules_to_save": null,
30
+ "peft_type": "LORA",
31
+ "peft_version": "0.19.1",
32
+ "qalora_group_size": 16,
33
+ "r": 16,
34
+ "rank_pattern": {},
35
+ "revision": null,
36
+ "target_modules": [
37
+ "down_proj",
38
+ "up_proj",
39
+ "k_proj",
40
+ "q_proj",
41
+ "v_proj",
42
+ "o_proj",
43
+ "gate_proj"
44
+ ],
45
+ "target_parameters": null,
46
+ "task_type": "CAUSAL_LM",
47
+ "trainable_token_indices": null,
48
+ "use_bdlora": null,
49
+ "use_dora": false,
50
+ "use_qalora": false,
51
+ "use_rslora": false
52
+ }
lora-adapter/checkpoint-339/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e9f5d4a49e19b560a51601ae02e2255f8359cbed8f8f7050a44511d7a0175c61
3
+ size 72396376
lora-adapter/checkpoint-339/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
lora-adapter/checkpoint-339/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c41590b3d34e2e3f8c3e299dbed1fc03340305228683d500d62daa0a20057a2
3
+ size 37138533
lora-adapter/checkpoint-339/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b6a9c6bc12cb0ec2371d47919188be56c4525321c91056c0c8219b96596f802
3
+ size 14645
lora-adapter/checkpoint-339/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:14bd5c75fdea9f94ec829329c0f666fd63edbc684f38f55543ab208602a15702
3
+ size 1383
lora-adapter/checkpoint-339/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3ceeb962b61bec3eb227b3256130b942e361cce57b3a45380d791246e4f6b73
3
+ size 1465
lora-adapter/checkpoint-339/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
lora-adapter/checkpoint-339/tokenizer_config.json ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|im_start|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>"
11
+ ],
12
+ "is_local": false,
13
+ "model_max_length": 8192,
14
+ "pad_token": "<|PAD_TOKEN|>",
15
+ "padding_side": "right",
16
+ "tokenizer_class": "GPT2Tokenizer",
17
+ "unk_token": "�",
18
+ "vocab_size": 49152,
19
+ "added_tokens_decoder": {
20
+ "0": {
21
+ "content": "<|endoftext|>",
22
+ "single_word": false,
23
+ "lstrip": false,
24
+ "rstrip": false,
25
+ "normalized": false,
26
+ "special": true
27
+ },
28
+ "1": {
29
+ "content": "<|im_start|>",
30
+ "single_word": false,
31
+ "lstrip": false,
32
+ "rstrip": false,
33
+ "normalized": false,
34
+ "special": true
35
+ },
36
+ "2": {
37
+ "content": "<|im_end|>",
38
+ "single_word": false,
39
+ "lstrip": false,
40
+ "rstrip": false,
41
+ "normalized": false,
42
+ "special": true
43
+ },
44
+ "3": {
45
+ "content": "<repo_name>",
46
+ "single_word": false,
47
+ "lstrip": false,
48
+ "rstrip": false,
49
+ "normalized": false,
50
+ "special": true
51
+ },
52
+ "4": {
53
+ "content": "<reponame>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ "5": {
61
+ "content": "<file_sep>",
62
+ "single_word": false,
63
+ "lstrip": false,
64
+ "rstrip": false,
65
+ "normalized": false,
66
+ "special": true
67
+ },
68
+ "6": {
69
+ "content": "<filename>",
70
+ "single_word": false,
71
+ "lstrip": false,
72
+ "rstrip": false,
73
+ "normalized": false,
74
+ "special": true
75
+ },
76
+ "7": {
77
+ "content": "<gh_stars>",
78
+ "single_word": false,
79
+ "lstrip": false,
80
+ "rstrip": false,
81
+ "normalized": false,
82
+ "special": true
83
+ },
84
+ "8": {
85
+ "content": "<issue_start>",
86
+ "single_word": false,
87
+ "lstrip": false,
88
+ "rstrip": false,
89
+ "normalized": false,
90
+ "special": true
91
+ },
92
+ "9": {
93
+ "content": "<issue_comment>",
94
+ "single_word": false,
95
+ "lstrip": false,
96
+ "rstrip": false,
97
+ "normalized": false,
98
+ "special": true
99
+ },
100
+ "10": {
101
+ "content": "<issue_closed>",
102
+ "single_word": false,
103
+ "lstrip": false,
104
+ "rstrip": false,
105
+ "normalized": false,
106
+ "special": true
107
+ },
108
+ "11": {
109
+ "content": "<jupyter_start>",
110
+ "single_word": false,
111
+ "lstrip": false,
112
+ "rstrip": false,
113
+ "normalized": false,
114
+ "special": true
115
+ },
116
+ "12": {
117
+ "content": "<jupyter_text>",
118
+ "single_word": false,
119
+ "lstrip": false,
120
+ "rstrip": false,
121
+ "normalized": false,
122
+ "special": true
123
+ },
124
+ "13": {
125
+ "content": "<jupyter_code>",
126
+ "single_word": false,
127
+ "lstrip": false,
128
+ "rstrip": false,
129
+ "normalized": false,
130
+ "special": true
131
+ },
132
+ "14": {
133
+ "content": "<jupyter_output>",
134
+ "single_word": false,
135
+ "lstrip": false,
136
+ "rstrip": false,
137
+ "normalized": false,
138
+ "special": true
139
+ },
140
+ "15": {
141
+ "content": "<jupyter_script>",
142
+ "single_word": false,
143
+ "lstrip": false,
144
+ "rstrip": false,
145
+ "normalized": false,
146
+ "special": true
147
+ },
148
+ "16": {
149
+ "content": "<empty_output>",
150
+ "single_word": false,
151
+ "lstrip": false,
152
+ "rstrip": false,
153
+ "normalized": false,
154
+ "special": true
155
+ },
156
+ "24211": {
157
+ "content": "�",
158
+ "single_word": false,
159
+ "lstrip": false,
160
+ "rstrip": false,
161
+ "normalized": false,
162
+ "special": true
163
+ },
164
+ "49152": {
165
+ "content": "<|PAD_TOKEN|>",
166
+ "single_word": false,
167
+ "lstrip": false,
168
+ "rstrip": false,
169
+ "normalized": false,
170
+ "special": true
171
+ }
172
+ }
173
+ }
lora-adapter/checkpoint-339/trainer_state.json ADDED
@@ -0,0 +1,297 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 3.0,
6
+ "eval_steps": 100,
7
+ "global_step": 339,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.08888888888888889,
14
+ "grad_norm": 0.6075066328048706,
15
+ "learning_rate": 3.6e-05,
16
+ "loss": 2.435858154296875,
17
+ "step": 10
18
+ },
19
+ {
20
+ "epoch": 0.17777777777777778,
21
+ "grad_norm": 0.5337297320365906,
22
+ "learning_rate": 7.6e-05,
23
+ "loss": 2.261408805847168,
24
+ "step": 20
25
+ },
26
+ {
27
+ "epoch": 0.26666666666666666,
28
+ "grad_norm": 4.804379463195801,
29
+ "learning_rate": 0.000116,
30
+ "loss": 1.9577768325805665,
31
+ "step": 30
32
+ },
33
+ {
34
+ "epoch": 0.35555555555555557,
35
+ "grad_norm": 0.39172491431236267,
36
+ "learning_rate": 0.00015600000000000002,
37
+ "loss": 1.6207334518432617,
38
+ "step": 40
39
+ },
40
+ {
41
+ "epoch": 0.4444444444444444,
42
+ "grad_norm": 0.31786638498306274,
43
+ "learning_rate": 0.000196,
44
+ "loss": 1.5145363807678223,
45
+ "step": 50
46
+ },
47
+ {
48
+ "epoch": 0.5333333333333333,
49
+ "grad_norm": 0.295754611492157,
50
+ "learning_rate": 0.00019952179661709028,
51
+ "loss": 1.5010628700256348,
52
+ "step": 60
53
+ },
54
+ {
55
+ "epoch": 0.6222222222222222,
56
+ "grad_norm": 0.29678136110305786,
57
+ "learning_rate": 0.00019787461880296965,
58
+ "loss": 1.4191683769226073,
59
+ "step": 70
60
+ },
61
+ {
62
+ "epoch": 0.7111111111111111,
63
+ "grad_norm": 0.304258793592453,
64
+ "learning_rate": 0.00019507200355689026,
65
+ "loss": 1.392717456817627,
66
+ "step": 80
67
+ },
68
+ {
69
+ "epoch": 0.8,
70
+ "grad_norm": 0.33903768658638,
71
+ "learning_rate": 0.00019114703654091961,
72
+ "loss": 1.382735824584961,
73
+ "step": 90
74
+ },
75
+ {
76
+ "epoch": 0.8888888888888888,
77
+ "grad_norm": 0.35391494631767273,
78
+ "learning_rate": 0.00018614605309577137,
79
+ "loss": 1.3719289779663086,
80
+ "step": 100
81
+ },
82
+ {
83
+ "epoch": 0.8888888888888888,
84
+ "eval_loss": 1.350045919418335,
85
+ "eval_runtime": 8.3527,
86
+ "eval_samples_per_second": 23.944,
87
+ "eval_steps_per_second": 5.986,
88
+ "step": 100
89
+ },
90
+ {
91
+ "epoch": 0.9777777777777777,
92
+ "grad_norm": 0.3610546290874481,
93
+ "learning_rate": 0.00018012809123906228,
94
+ "loss": 1.3875890731811524,
95
+ "step": 110
96
+ },
97
+ {
98
+ "epoch": 1.0622222222222222,
99
+ "grad_norm": 0.3689529001712799,
100
+ "learning_rate": 0.00017316419470489209,
101
+ "loss": 1.3298458099365233,
102
+ "step": 120
103
+ },
104
+ {
105
+ "epoch": 1.1511111111111112,
106
+ "grad_norm": 0.3758569657802582,
107
+ "learning_rate": 0.00016533657425255952,
108
+ "loss": 1.2853054046630858,
109
+ "step": 130
110
+ },
111
+ {
112
+ "epoch": 1.24,
113
+ "grad_norm": 0.4089072048664093,
114
+ "learning_rate": 0.00015673763714540214,
115
+ "loss": 1.302170467376709,
116
+ "step": 140
117
+ },
118
+ {
119
+ "epoch": 1.3288888888888888,
120
+ "grad_norm": 0.47941380739212036,
121
+ "learning_rate": 0.00014746889625703502,
122
+ "loss": 1.2683821678161622,
123
+ "step": 150
124
+ },
125
+ {
126
+ "epoch": 1.4177777777777778,
127
+ "grad_norm": 0.4418100416660309,
128
+ "learning_rate": 0.0001376397716832963,
129
+ "loss": 1.2893789291381836,
130
+ "step": 160
131
+ },
132
+ {
133
+ "epoch": 1.5066666666666668,
134
+ "grad_norm": 0.4881313741207123,
135
+ "learning_rate": 0.0001273662990072083,
136
+ "loss": 1.2691761016845704,
137
+ "step": 170
138
+ },
139
+ {
140
+ "epoch": 1.5955555555555554,
141
+ "grad_norm": 0.4534631371498108,
142
+ "learning_rate": 0.00011676975946624944,
143
+ "loss": 1.2686445236206054,
144
+ "step": 180
145
+ },
146
+ {
147
+ "epoch": 1.6844444444444444,
148
+ "grad_norm": 0.4390407204627991,
149
+ "learning_rate": 0.0001059752481931988,
150
+ "loss": 1.2629841804504394,
151
+ "step": 190
152
+ },
153
+ {
154
+ "epoch": 1.7733333333333334,
155
+ "grad_norm": 0.474439799785614,
156
+ "learning_rate": 9.511019743287242e-05,
157
+ "loss": 1.289689540863037,
158
+ "step": 200
159
+ },
160
+ {
161
+ "epoch": 1.7733333333333334,
162
+ "eval_loss": 1.287023901939392,
163
+ "eval_runtime": 7.9351,
164
+ "eval_samples_per_second": 25.205,
165
+ "eval_steps_per_second": 6.301,
166
+ "step": 200
167
+ },
168
+ {
169
+ "epoch": 1.8622222222222222,
170
+ "grad_norm": 0.6423256993293762,
171
+ "learning_rate": 8.430287216859443e-05,
172
+ "loss": 1.2817410469055175,
173
+ "step": 210
174
+ },
175
+ {
176
+ "epoch": 1.951111111111111,
177
+ "grad_norm": 0.5383001565933228,
178
+ "learning_rate": 7.368085591795522e-05,
179
+ "loss": 1.228531837463379,
180
+ "step": 220
181
+ },
182
+ {
183
+ "epoch": 2.0355555555555553,
184
+ "grad_norm": 0.5520081520080566,
185
+ "learning_rate": 6.336954457346462e-05,
186
+ "loss": 1.196599292755127,
187
+ "step": 230
188
+ },
189
+ {
190
+ "epoch": 2.1244444444444444,
191
+ "grad_norm": 0.4924458861351013,
192
+ "learning_rate": 5.349066606873525e-05,
193
+ "loss": 1.150006103515625,
194
+ "step": 240
195
+ },
196
+ {
197
+ "epoch": 2.2133333333333334,
198
+ "grad_norm": 0.5595092177391052,
199
+ "learning_rate": 4.416084334595314e-05,
200
+ "loss": 1.1321084022521972,
201
+ "step": 250
202
+ },
203
+ {
204
+ "epoch": 2.3022222222222224,
205
+ "grad_norm": 0.5763599872589111,
206
+ "learning_rate": 3.549021758920955e-05,
207
+ "loss": 1.1533366203308106,
208
+ "step": 260
209
+ },
210
+ {
211
+ "epoch": 2.391111111111111,
212
+ "grad_norm": 0.5541089177131653,
213
+ "learning_rate": 2.758114797681215e-05,
214
+ "loss": 1.142934799194336,
215
+ "step": 270
216
+ },
217
+ {
218
+ "epoch": 2.48,
219
+ "grad_norm": 0.6457899808883667,
220
+ "learning_rate": 2.0527003302365412e-05,
221
+ "loss": 1.1384923934936524,
222
+ "step": 280
223
+ },
224
+ {
225
+ "epoch": 2.568888888888889,
226
+ "grad_norm": 0.5974424481391907,
227
+ "learning_rate": 1.4411059729873765e-05,
228
+ "loss": 1.140573215484619,
229
+ "step": 290
230
+ },
231
+ {
232
+ "epoch": 2.6577777777777776,
233
+ "grad_norm": 0.6066321134567261,
234
+ "learning_rate": 9.305517695178833e-06,
235
+ "loss": 1.175175094604492,
236
+ "step": 300
237
+ },
238
+ {
239
+ "epoch": 2.6577777777777776,
240
+ "eval_loss": 1.2747913599014282,
241
+ "eval_runtime": 7.9917,
242
+ "eval_samples_per_second": 25.026,
243
+ "eval_steps_per_second": 6.257,
244
+ "step": 300
245
+ },
246
+ {
247
+ "epoch": 2.7466666666666666,
248
+ "grad_norm": 0.5629175305366516,
249
+ "learning_rate": 5.270649559485907e-06,
250
+ "loss": 1.1434768676757812,
251
+ "step": 310
252
+ },
253
+ {
254
+ "epoch": 2.8355555555555556,
255
+ "grad_norm": 0.5851128697395325,
256
+ "learning_rate": 2.3540880771700803e-06,
257
+ "loss": 1.1377876281738282,
258
+ "step": 320
259
+ },
260
+ {
261
+ "epoch": 2.924444444444444,
262
+ "grad_norm": 0.5346568822860718,
263
+ "learning_rate": 5.902640776996315e-07,
264
+ "loss": 1.1460090637207032,
265
+ "step": 330
266
+ },
267
+ {
268
+ "epoch": 3.0,
269
+ "eval_loss": 1.274646282196045,
270
+ "eval_runtime": 7.7983,
271
+ "eval_samples_per_second": 25.647,
272
+ "eval_steps_per_second": 6.412,
273
+ "step": 339
274
+ }
275
+ ],
276
+ "logging_steps": 10,
277
+ "max_steps": 339,
278
+ "num_input_tokens_seen": 0,
279
+ "num_train_epochs": 3,
280
+ "save_steps": 100,
281
+ "stateful_callbacks": {
282
+ "TrainerControl": {
283
+ "args": {
284
+ "should_epoch_stop": false,
285
+ "should_evaluate": false,
286
+ "should_log": false,
287
+ "should_save": true,
288
+ "should_training_stop": true
289
+ },
290
+ "attributes": {}
291
+ }
292
+ },
293
+ "total_flos": 5962134935494656.0,
294
+ "train_batch_size": 4,
295
+ "trial_name": null,
296
+ "trial_params": null
297
+ }
lora-adapter/checkpoint-339/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bed2516133b47b8ec32a3f5147a39523682e8fb734ef7e3dafef29759653d26
3
+ size 5713