Hugging Face's logo

QuantFactory
/
Crispy_Crab_4B-GGUF

Transformers
GGUF
axolotl
Generated from Trainer
Model card Files
xet
 Community
aashish1904 commited on
Commit
8b06c8f
·
verified ·
1 Parent(s): 07ed58b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +179 -0
README.md ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+
4
+ library_name: transformers
5
+ license: other
6
+ base_model: jeiku/instructered4B
7
+ tags:
8
+ - axolotl
9
+ - generated_from_trainer
10
+ model-index:
11
+ - name: TheBest4B
12
+ results: []
13
+
14
+ ---
15
+
16
+ [![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
17
+
18
+
19
+ # QuantFactory/Crispy_Crab_4B-GGUF
20
+ This is quantized version of [FourOhFour/Crispy_Crab_4B](https://huggingface.co/FourOhFour/Crispy_Crab_4B) created using llama.cpp
21
+
22
+ # Original Model Card
23
+
24
+
25
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
26
+ should probably proofread and complete it, then remove this comment. -->
27
+
28
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
29
+ <details><summary>See axolotl config</summary>
30
+
31
+ axolotl version: `0.4.1`
32
+ ```yaml
33
+ base_model: jeiku/instructered4B
34
+ model_type: AutoModelForCausalLM
35
+ tokenizer_type: AutoTokenizer
36
+
37
+ load_in_8bit: false
38
+ load_in_4bit: false
39
+ strict: false
40
+
41
+ hub_model_id: jeiku/TheBest4B
42
+ hub_strategy: "all_checkpoints"
43
+ push_dataset_to_hub:
44
+ hf_use_auth_token: true
45
+
46
+ datasets:
47
+ - path: FourOhFour/RP_Phase
48
+ type: sharegpt
49
+ conversation: chatml
50
+
51
+ chat_template: chatml
52
+
53
+ shuffle_merged_datasets: true
54
+ val_set_size: 0.0025
55
+ output_dir: ./outputs/out
56
+
57
+ adapter:
58
+ lora_r:
59
+ lora_alpha:
60
+ lora_dropout:
61
+ lora_target_linear:
62
+
63
+ sequence_len: 8192
64
+ sample_packing: true
65
+ eval_sample_packing: false
66
+ pad_to_sequence_len: true
67
+
68
+ plugins:
69
+ - axolotl.integrations.liger.LigerPlugin
70
+ liger_rope: true
71
+ liger_rms_norm: true
72
+ liger_swiglu: true
73
+ liger_fused_linear_cross_entropy: true
74
+
75
+ wandb_project: EXP4B
76
+ wandb_entity:
77
+ wandb_watch:
78
+ wandb_name: EXP4B
79
+ wandb_log_model:
80
+
81
+ gradient_accumulation_steps: 12
82
+ micro_batch_size: 3
83
+ num_epochs: 2
84
+ optimizer: adamw_bnb_8bit
85
+ lr_scheduler: cosine
86
+ learning_rate: 0.00001
87
+ weight_decay: 0.05
88
+
89
+ train_on_inputs: false
90
+ group_by_length: false
91
+ bf16: auto
92
+ fp16:
93
+ tf32: true
94
+
95
+ gradient_checkpointing: true
96
+ early_stopping_patience:
97
+ resume_from_checkpoint:
98
+ local_rank:
99
+ logging_steps: 1
100
+ xformers_attention:
101
+ flash_attention: true
102
+
103
+ warmup_ratio: 0.1
104
+ evals_per_epoch: 4
105
+ eval_table_size:
106
+ eval_max_new_tokens: 128
107
+ saves_per_epoch: 2
108
+
109
+ debug:
110
+ deepspeed: deepspeed_configs/zero3_bf16.json
111
+ fsdp:
112
+ fsdp_config:
113
+
114
+ special_tokens:
115
+ pad_token: <|finetune_right_pad_id|>
116
+
117
+ ```
118
+
119
+ </details><br>
120
+
121
+ # TheBest4B
122
+
123
+ This model is a fine-tuned version of [jeiku/instructered4B](https://huggingface.co/jeiku/instructered4B) on the None dataset.
124
+ It achieves the following results on the evaluation set:
125
+ - Loss: 2.1148
126
+
127
+ ## Model description
128
+
129
+ More information needed
130
+
131
+ ## Intended uses & limitations
132
+
133
+ More information needed
134
+
135
+ ## Training and evaluation data
136
+
137
+ More information needed
138
+
139
+ ## Training procedure
140
+
141
+ ### Training hyperparameters
142
+
143
+ The following hyperparameters were used during training:
144
+ - learning_rate: 1e-05
145
+ - train_batch_size: 3
146
+ - eval_batch_size: 3
147
+ - seed: 42
148
+ - distributed_type: multi-GPU
149
+ - num_devices: 2
150
+ - gradient_accumulation_steps: 12
151
+ - total_train_batch_size: 72
152
+ - total_eval_batch_size: 6
153
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
154
+ - lr_scheduler_type: cosine
155
+ - lr_scheduler_warmup_steps: 22
156
+ - num_epochs: 2
157
+
158
+ ### Training results
159
+
160
+ | Training Loss | Epoch | Step | Validation Loss |
161
+ |:-------------:|:------:|:----:|:---------------:|
162
+ | 1.8805 | 0.0089 | 1 | 2.7425 |
163
+ | 1.7985 | 0.2491 | 28 | 2.2908 |
164
+ | 1.727 | 0.4981 | 56 | 2.1943 |
165
+ | 1.7429 | 0.7472 | 84 | 2.1665 |
166
+ | 1.6867 | 0.9963 | 112 | 2.1309 |
167
+ | 1.6463 | 1.2461 | 140 | 2.1267 |
168
+ | 1.593 | 1.4959 | 168 | 2.1148 |
169
+ | 1.604 | 1.7457 | 196 | 2.1129 |
170
+ | 1.6085 | 1.9955 | 224 | 2.1148 |
171
+
172
+
173
+ ### Framework versions
174
+
175
+ - Transformers 4.45.1
176
+ - Pytorch 2.4.1+cu121
177
+ - Datasets 2.21.0
178
+ - Tokenizers 0.20.0
179
+