Hugging Face's logo

CharlesLi
/
mistral_cot_simplest_lora

PEFT
TensorBoard
Safetensors
mistral
alignment-handbook
trl
sft
Generated from Trainer
Model card Files
xet
Metrics Community
CharlesLi commited on
Commit
b4b5923
·
verified ·
1 Parent(s): 1d19c9c

Model save

Browse files
Files changed (4) hide show
  1. README.md +71 -0
  2. all_results.json +9 -0
  3. train_results.json +9 -0
  4. trainer_state.json +80 -0
README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: apache-2.0
4
+ base_model: mistralai/Mistral-7B-Instruct-v0.1
5
+ tags:
6
+ - trl
7
+ - sft
8
+ - generated_from_trainer
9
+ datasets:
10
+ - generator
11
+ model-index:
12
+ - name: mistral_cot_simplest_lora
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # mistral_cot_simplest_lora
20
+
21
+ This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) on the generator dataset.
22
+ It achieves the following results on the evaluation set:
23
+ - Loss: 0.8421
24
+
25
+ ## Model description
26
+
27
+ More information needed
28
+
29
+ ## Intended uses & limitations
30
+
31
+ More information needed
32
+
33
+ ## Training and evaluation data
34
+
35
+ More information needed
36
+
37
+ ## Training procedure
38
+
39
+ ### Training hyperparameters
40
+
41
+ The following hyperparameters were used during training:
42
+ - learning_rate: 0.0002
43
+ - train_batch_size: 4
44
+ - eval_batch_size: 8
45
+ - seed: 42
46
+ - distributed_type: multi-GPU
47
+ - num_devices: 2
48
+ - gradient_accumulation_steps: 2
49
+ - total_train_batch_size: 16
50
+ - total_eval_batch_size: 16
51
+ - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
52
+ - lr_scheduler_type: cosine
53
+ - lr_scheduler_warmup_ratio: 0.1
54
+ - num_epochs: 5
55
+
56
+ ### Training results
57
+
58
+ | Training Loss | Epoch | Step | Validation Loss |
59
+ |:-------------:|:------:|:----:|:---------------:|
60
+ | 1.9369 | 1.0 | 2 | 1.0072 |
61
+ | 1.9369 | 2.0 | 4 | 0.8536 |
62
+ | 1.3047 | 2.6667 | 5 | 0.8421 |
63
+
64
+
65
+ ### Framework versions
66
+
67
+ - PEFT 0.14.0
68
+ - Transformers 4.47.1
69
+ - Pytorch 2.1.2
70
+ - Datasets 3.2.0
71
+ - Tokenizers 0.21.0
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 2.6666666666666665,
3
+ "total_flos": 2759020687392768.0,
4
+ "train_loss": 1.4311530351638795,
5
+ "train_runtime": 40.306,
6
+ "train_samples": 100,
7
+ "train_samples_per_second": 2.481,
8
+ "train_steps_per_second": 0.124
9
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 2.6666666666666665,
3
+ "total_flos": 2759020687392768.0,
4
+ "train_loss": 1.4311530351638795,
5
+ "train_runtime": 40.306,
6
+ "train_samples": 100,
7
+ "train_samples_per_second": 2.481,
8
+ "train_steps_per_second": 0.124
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 2.6666666666666665,
5
+ "eval_steps": 500,
6
+ "global_step": 5,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.6666666666666666,
13
+ "grad_norm": 0.5989622834533995,
14
+ "learning_rate": 0.0002,
15
+ "loss": 1.9369,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 1.0,
20
+ "eval_loss": 1.0072091817855835,
21
+ "eval_runtime": 1.4634,
22
+ "eval_samples_per_second": 2.05,
23
+ "eval_steps_per_second": 0.683,
24
+ "step": 2
25
+ },
26
+ {
27
+ "epoch": 2.0,
28
+ "eval_loss": 0.8536226153373718,
29
+ "eval_runtime": 1.5341,
30
+ "eval_samples_per_second": 1.956,
31
+ "eval_steps_per_second": 0.652,
32
+ "step": 4
33
+ },
34
+ {
35
+ "epoch": 2.6666666666666665,
36
+ "grad_norm": 0.4590662786527058,
37
+ "learning_rate": 0.0,
38
+ "loss": 1.3047,
39
+ "step": 5
40
+ },
41
+ {
42
+ "epoch": 2.6666666666666665,
43
+ "eval_loss": 0.8420554995536804,
44
+ "eval_runtime": 1.562,
45
+ "eval_samples_per_second": 1.921,
46
+ "eval_steps_per_second": 0.64,
47
+ "step": 5
48
+ },
49
+ {
50
+ "epoch": 2.6666666666666665,
51
+ "step": 5,
52
+ "total_flos": 2759020687392768.0,
53
+ "train_loss": 1.4311530351638795,
54
+ "train_runtime": 40.306,
55
+ "train_samples_per_second": 2.481,
56
+ "train_steps_per_second": 0.124
57
+ }
58
+ ],
59
+ "logging_steps": 5,
60
+ "max_steps": 5,
61
+ "num_input_tokens_seen": 0,
62
+ "num_train_epochs": 5,
63
+ "save_steps": 1000,
64
+ "stateful_callbacks": {
65
+ "TrainerControl": {
66
+ "args": {
67
+ "should_epoch_stop": false,
68
+ "should_evaluate": false,
69
+ "should_log": false,
70
+ "should_save": true,
71
+ "should_training_stop": true
72
+ },
73
+ "attributes": {}
74
+ }
75
+ },
76
+ "total_flos": 2759020687392768.0,
77
+ "train_batch_size": 4,
78
+ "trial_name": null,
79
+ "trial_params": null
80
+ }