mechark/LLama-8b-Instruct-humanizer

README.md

@@ -19,7 +19,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [unsloth/llama-3.1-8b-instruct-unsloth-bnb-4bit](https://huggingface.co/unsloth/llama-3.1-8b-instruct-unsloth-bnb-4bit) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.0004
+- Loss: 1.6516
 
 ## Model description
 
@@ -38,7 +38,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 2e-05
+- learning_rate: 1e-05
 - train_batch_size: 6
 - eval_batch_size: 8
 - seed: 3407
@@ -46,25 +46,28 @@ The following hyperparameters were used during training:
 - total_train_batch_size: 96
 - optimizer: Use adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 10
+- lr_scheduler_warmup_steps: 10
+- num_epochs: 3
 
 ### Training results
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 1.7535        | 1.5475 | 50   | 1.6139          |
-| 1.3319        | 3.1217 | 100  | 1.2796          |
-| 1.0905        | 4.6692 | 150  | 1.0633          |
-| 1.0207        | 6.2433 | 200  | 1.0138          |
-| 0.9656        | 7.7909 | 250  | 1.0022          |
-| 1.0342        | 9.3650 | 300  | 1.0004          |
+| 1.9534        | 0.3042 | 10   | 1.8653          |
+| 1.8372        | 0.6084 | 20   | 1.8373          |
+| 1.8058        | 0.9125 | 30   | 1.8220          |
+| 1.9211        | 1.2433 | 40   | 1.7757          |
+| 1.804         | 1.5475 | 50   | 1.7168          |
+| 1.7052        | 1.8517 | 60   | 1.6860          |
+| 1.6974        | 2.1825 | 70   | 1.6672          |
+| 1.7511        | 2.4867 | 80   | 1.6541          |
+| 1.6929        | 2.7909 | 90   | 1.6516          |
 
 
 ### Framework versions
 
 - PEFT 0.15.1
-- Transformers 4.51.2
+- Transformers 4.51.3
 - Pytorch 2.6.0+cu124
 - Datasets 3.5.0
 - Tokenizers 0.21.1

adapter_config.json

@@ -24,13 +24,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
-    "q_proj",
-    "gate_proj",
     "o_proj",
+    "down_proj",
+    "q_proj",
     "v_proj",
+    "k_proj",
     "up_proj",
-    "down_proj"
+    "gate_proj"
   ],
   "task_type": "CAUSAL_LM",
   "trainable_token_indices": null,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0dfc21935c7f6efe3ad4c66d916c95a21ffccd40090776b70d9a02e4b540c5db
+oid sha256:aae4b0d50da6e41e40ba2d8feb7859f753cb4bb36ac239019c00adaade210a8a
 size 167832240

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:74e0882ffb0d6304eb9de39a737be956d57cab059ffb7e6f8bfaf37bb83911c3
+oid sha256:e41b177df17c65dcf90fa1340952e9e13ce4209db9d0a643a6b1b642df3f5efe
 size 5304