End of training

README.md

@@ -0,0 +1,63 @@
+---
+tags:
+- trl
+- sft
+- generated_from_trainer
+model-index:
+- name: mistral-supervised
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# mistral-supervised
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.4161
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 3
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 5.8982        | 0.5889 | 500  | 4.6761          |
+| 4.2609        | 1.1779 | 1000 | 3.9880          |
+| 3.7225        | 1.7668 | 1500 | 3.6593          |
+| 3.433         | 2.3557 | 2000 | 3.4856          |
+| 3.2793        | 2.9446 | 2500 | 3.4161          |
+
+
+### Framework versions
+
+- Transformers 4.43.3
+- Pytorch 2.3.1+cu121
+- Datasets 2.20.0
+- Tokenizers 0.19.1

config.json

@@ -15,12 +15,13 @@
   "num_attention_heads": 16,
   "num_hidden_layers": 4,
   "num_key_value_heads": 8,
+  "pad_token_id": 2,
   "rms_norm_eps": 1e-06,
   "rope_theta": 10000.0,
   "sliding_window": 768,
   "tie_word_embeddings": false,
   "torch_dtype": "float32",
-  "transformers_version": "4.45.2",
+  "transformers_version": "4.43.3",
   "use_cache": true,
   "vocab_size": 32000
 }

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.43.3"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7281a829822c63c6a67b798b6ce520c0d304874b66b72b7fa32c94fa070acc9
+oid sha256:5885361f0cf75a8b2d9aaed8f0eae858591a4a4ab457d5be38ae2ab5171358b1
 size 338197712

runs/Nov03_15-53-45_shekhar-a100/events.out.tfevents.1730649232.shekhar-a100

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df29c1acd852715fa1abdfdf949613e07755675f056b4be17ee05d2f174f6848
+size 5205

runs/Nov03_15-54-53_shekhar-a100/events.out.tfevents.1730649298.shekhar-a100

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d0125ffd62a2887c0578fa0ba4b0ff43b97b738f80ac045433cb77113e74a2a
+size 5226

runs/Nov03_16-00-01_shekhar-a100/events.out.tfevents.1730649608.shekhar-a100

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65ba91d428d1b02b8e5501e9b632985c90b2cda13b522d87bc19fd7d30d08d47
+size 5226

runs/Nov03_16-14-33_shekhar-a100/events.out.tfevents.1730650482.shekhar-a100

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80a71960e49a17e9199e54f7c9be71b2a40604c4865d75acb2cdc746f8f7cf8d
+size 7855

special_tokens_map.json

@@ -13,7 +13,7 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": "<pad>",
+  "pad_token": "</s>",
   "unk_token": {
     "content": "<unk>",
     "lstrip": false,

tokenizer_config.json

@@ -34,7 +34,7 @@
   "eos_token": "</s>",
   "legacy": false,
   "model_max_length": 1000000000000000019884624838656,
-  "pad_token": "<pad>",
+  "pad_token": "</s>",
   "sp_model_kwargs": {},
   "spaces_between_special_tokens": false,
   "tokenizer_class": "LlamaTokenizer",

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:992d4593f6e88d6fb93a624bbbc75bf5f5656238e5f6c125b03e1b3a743b14d7
+oid sha256:daf56b48d4425fdc37a5eca9bfd12c5b65e9489360f12538d93de9fa04d38ea7
 size 5432