Hugging Face's logo

amphora
/
merged-bench-train-base

Text Generation
Transformers
Safetensors
qwen2
axolotl
Generated from Trainer
conversational
text-generation-inference
Model card Files
xet
Community
1
amphora commited on
Commit
e022fca
·
verified ·
1 Parent(s): 5d17cfe

Model save

Browse files
Files changed (1) hide show
  1. README.md +167 -0
README.md ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: Qwen/Qwen2.5-7B-Instruct
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ datasets:
9
+ - train_base.jsonl
10
+ model-index:
11
+ - name: merged-bench-train-base
12
+ results: []
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
19
+ <details><summary>See axolotl config</summary>
20
+
21
+ axolotl version: `0.8.0.dev0`
22
+ ```yaml
23
+ base_model: Qwen/Qwen2.5-7B-Instruct
24
+ model_type: AutoModelForCausalLM
25
+ tokenizer_type: AutoTokenizer
26
+ trust_remote_code: false
27
+
28
+ load_in_8bit: false
29
+ load_in_4bit: false
30
+ strict: false
31
+
32
+ output_dir: ./outputs/out
33
+ chat_template: qwen_25
34
+ datasets:
35
+ - path: train_base.jsonl
36
+ type: chat_template
37
+ field_messages: messages
38
+ message_field_role: role
39
+ message_field_content: content
40
+ roles:
41
+ system:
42
+ - system
43
+ user:
44
+ - user
45
+ assistant:
46
+ - assistant
47
+
48
+ dataset_prepared_path: last_run_prepared
49
+ val_set_size: 0.005
50
+ output_dir: ./outputs/out
51
+ eval_sample_packing: False
52
+
53
+ sequence_len: 8192
54
+ sample_packing: False
55
+ pad_to_sequence_len: False
56
+
57
+ wandb_project: mergedbench
58
+ wandb_entity:
59
+ wandb_watch:
60
+ wandb_name:
61
+ wandb_log_model:
62
+ hub_model_id: amphora/merged-bench-train-base
63
+
64
+ plugins:
65
+ - axolotl.integrations.liger.LigerPlugin
66
+ liger_rope: true
67
+ liger_rms_norm: true
68
+ liger_swiglu: true
69
+ liger_fused_linear_cross_entropy: true
70
+
71
+ gradient_accumulation_steps: 4
72
+ micro_batch_size: 8
73
+ eval_batch_size: 4
74
+ num_epochs: 3
75
+ optimizer: paged_adamw_8bit
76
+ lr_scheduler: cosine
77
+ learning_rate: 2e-5
78
+
79
+ train_on_inputs: false
80
+ group_by_length: false
81
+ bf16: auto
82
+ fp16:
83
+ tf32: false
84
+
85
+ gradient_checkpointing: true
86
+ gradient_checkpointing_kwargs:
87
+ use_reentrant: false
88
+ early_stopping_patience:
89
+ resume_from_checkpoint:
90
+ logging_steps: 1
91
+ xformers_attention:
92
+ flash_attention: true
93
+
94
+ warmup_steps: 30
95
+ evals_per_epoch: 3
96
+ eval_max_new_tokens: 128
97
+ eval_table_size:
98
+ saves_per_epoch: 1
99
+ debug:
100
+ deepspeed: deepspeed_configs/zero3_bf16.json
101
+ weight_decay: 0.01
102
+ fsdp:
103
+ fsdp_config:
104
+ special_tokens:
105
+ ```
106
+
107
+ </details><br>
108
+
109
+ # merged-bench-train-base
110
+
111
+ This model is a fine-tuned version of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) on the train_base.jsonl dataset.
112
+ It achieves the following results on the evaluation set:
113
+ - Loss: 0.3727
114
+
115
+ ## Model description
116
+
117
+ More information needed
118
+
119
+ ## Intended uses & limitations
120
+
121
+ More information needed
122
+
123
+ ## Training and evaluation data
124
+
125
+ More information needed
126
+
127
+ ## Training procedure
128
+
129
+ ### Training hyperparameters
130
+
131
+ The following hyperparameters were used during training:
132
+ - learning_rate: 2e-05
133
+ - train_batch_size: 8
134
+ - eval_batch_size: 4
135
+ - seed: 42
136
+ - distributed_type: multi-GPU
137
+ - num_devices: 4
138
+ - gradient_accumulation_steps: 4
139
+ - total_train_batch_size: 128
140
+ - total_eval_batch_size: 16
141
+ - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
142
+ - lr_scheduler_type: cosine
143
+ - lr_scheduler_warmup_steps: 30
144
+ - num_epochs: 3.0
145
+
146
+ ### Training results
147
+
148
+ | Training Loss | Epoch | Step | Validation Loss |
149
+ |:-------------:|:------:|:----:|:---------------:|
150
+ | 1.0175 | 0.0118 | 1 | 1.0696 |
151
+ | 0.3467 | 0.3314 | 28 | 0.5233 |
152
+ | 0.3178 | 0.6627 | 56 | 0.4110 |
153
+ | 0.3214 | 0.9941 | 84 | 0.3711 |
154
+ | 0.2103 | 1.3195 | 112 | 0.3646 |
155
+ | 0.2032 | 1.6509 | 140 | 0.3586 |
156
+ | 0.191 | 1.9822 | 168 | 0.3436 |
157
+ | 0.1189 | 2.3077 | 196 | 0.3672 |
158
+ | 0.1051 | 2.6391 | 224 | 0.3723 |
159
+ | 0.1103 | 2.9704 | 252 | 0.3727 |
160
+
161
+
162
+ ### Framework versions
163
+
164
+ - Transformers 4.50.3
165
+ - Pytorch 2.6.0+cu124
166
+ - Datasets 3.5.0
167
+ - Tokenizers 0.21.1