Model save

README.md

@@ -14,7 +14,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 3.3235
+- Loss: 2.7250
 
 ## Model description
 
@@ -44,28 +44,22 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-| Training Loss | Epoch | Step  | Validation Loss |
-|:-------------:|:-----:|:-----:|:---------------:|
-| 3.901         | 0.064 | 5000  | 3.8367          |
-| 3.6772        | 0.128 | 10000 | 3.6281          |
-| 3.5953        | 0.192 | 15000 | 3.5480          |
-| 3.5516        | 0.256 | 20000 | 3.5038          |
-| 3.5219        | 0.32  | 25000 | 3.4746          |
-| 3.5077        | 0.384 | 30000 | 3.4526          |
-| 3.4891        | 0.448 | 35000 | 3.4360          |
-| 3.4705        | 0.512 | 40000 | 3.4208          |
-| 3.464         | 0.576 | 45000 | 3.4114          |
-| 3.4546        | 0.64  | 50000 | 3.3989          |
-| 3.4452        | 0.704 | 55000 | 3.3886          |
-| 3.4383        | 0.768 | 60000 | 3.3772          |
-| 3.4148        | 0.832 | 65000 | 3.3581          |
-| 3.3919        | 0.896 | 70000 | 3.3376          |
-| 3.3815        | 0.96  | 75000 | 3.3235          |
+| Training Loss | Epoch  | Step  | Validation Loss |
+|:-------------:|:------:|:-----:|:---------------:|
+| 3.7833        | 0.1067 | 5000  | 3.6697          |
+| 3.4271        | 0.2133 | 10000 | 3.2796          |
+| 3.2851        | 0.32   | 15000 | 3.1214          |
+| 3.2115        | 0.4267 | 20000 | 3.0228          |
+| 3.166         | 0.5333 | 25000 | 2.9708          |
+| 3.118         | 0.64   | 30000 | 2.9159          |
+| 3.0656        | 0.7467 | 35000 | 2.8546          |
+| 2.9881        | 0.8533 | 40000 | 2.7819          |
+| 2.9405        | 0.96   | 45000 | 2.7250          |
 
 
 ### Framework versions
 
-- Transformers 4.57.3
-- Pytorch 2.8.0+cu128
-- Datasets 4.4.2
+- Transformers 4.57.6
+- Pytorch 2.10.0+cu130
+- Datasets 4.5.0
 - Tokenizers 0.22.2

config.json

@@ -24,11 +24,15 @@
   "num_attention_heads": 8,
   "num_hidden_layers": 12,
   "num_key_value_heads": 2,
+  "num_stack_heads": 4,
   "pad_token_id": 151643,
   "partial_rotary_factor": 0.25,
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
   "rope_theta": 10000.0,
-  "transformers_version": "4.57.3",
+  "stack_d_model": 16,
+  "stack_slots": 24,
+  "transformers_version": "4.57.6",
+  "use_stack": true,
   "vocab_size": 151665
 }

generation_config.json

@@ -4,5 +4,5 @@
     151643
   ],
   "pad_token_id": 151643,
-  "transformers_version": "4.57.3"
+  "transformers_version": "4.57.6"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a91c397dc45b9aa4aacd762e3a9a769470ded360eccb80d66418d32b01de278
-size 251027488
+oid sha256:cc2cbf5ebfaeea4dbc63143d14ca208f4b5531357f7f31c2a4688e3b762c510b
+size 251434000

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bb17f745b950d54af953674a17895dead3a37d42e72e1c2b7c18dbfd8c0a43d1
+oid sha256:22cc0c67d1158f3a68631cff288743112f63360559878c1007d8da0095cfb98f
 size 6033