Model save

README.md

@@ -14,7 +14,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 3.2500
+- Loss: 3.2382
 
 ## Model description
 
@@ -46,29 +46,20 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step  | Validation Loss |
 |:-------------:|:------:|:-----:|:---------------:|
-| 3.8685        | 0.0533 | 5000  | 3.8281          |
-| 3.6486        | 0.1067 | 10000 | 3.6086          |
-| 3.5596        | 0.16   | 15000 | 3.5224          |
-| 3.5262        | 0.2133 | 20000 | 3.4727          |
-| 3.4856        | 0.2667 | 25000 | 3.4395          |
-| 3.4585        | 0.32   | 30000 | 3.4181          |
-| 3.4457        | 0.3733 | 35000 | 3.4013          |
-| 3.4247        | 0.4267 | 40000 | 3.3865          |
-| 3.4283        | 0.48   | 45000 | 3.3743          |
-| 3.4162        | 0.5333 | 50000 | 3.3662          |
-| 3.4076        | 0.5867 | 55000 | 3.3583          |
-| 3.3995        | 0.64   | 60000 | 3.3524          |
-| 3.3996        | 0.6933 | 65000 | 3.3458          |
-| 3.3921        | 0.7467 | 70000 | 3.3410          |
-| 3.3648        | 0.8    | 75000 | 3.3157          |
-| 3.3406        | 0.8533 | 80000 | 3.2877          |
-| 3.3174        | 0.9067 | 85000 | 3.2641          |
-| 3.2925        | 0.96   | 90000 | 3.2500          |
+| 3.812         | 0.1067 | 5000  | 3.7478          |
+| 3.6047        | 0.2133 | 10000 | 3.5369          |
+| 3.5195        | 0.32   | 15000 | 3.4601          |
+| 3.4769        | 0.4267 | 20000 | 3.4073          |
+| 3.4506        | 0.5333 | 25000 | 3.3818          |
+| 3.427         | 0.64   | 30000 | 3.3606          |
+| 3.4171        | 0.7467 | 35000 | 3.3413          |
+| 3.3404        | 0.8533 | 40000 | 3.2827          |
+| 3.3055        | 0.96   | 45000 | 3.2382          |
 
 
 ### Framework versions
 
-- Transformers 4.57.0
+- Transformers 4.57.1
 - Pytorch 2.8.0+cu129
 - Datasets 3.6.0
 - Tokenizers 0.22.1

config.json

@@ -13,6 +13,7 @@
   "dtype": "bfloat16",
   "eos_token_id": 151645,
   "fan_ratio": 0.125,
+  "fan_ratio_ffn": 0.0625,
   "head_dim": 64,
   "hidden_act": "xielu",
   "hidden_size": 512,
@@ -47,6 +48,6 @@
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
   "rope_theta": 10000.0,
-  "transformers_version": "4.57.0",
+  "transformers_version": "4.57.1",
   "vocab_size": 151665
 }

generation_config.json

@@ -4,5 +4,5 @@
     151645
   ],
   "pad_token_id": 151643,
-  "transformers_version": "4.57.0"
+  "transformers_version": "4.57.1"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83957e75447ef4c2b1dcd2dac38119d174aeab7d9075cdab8e1107f2aceebfe4
-size 245234560
+oid sha256:e56bdf216a373272a4846746f394538aadc5a0f7b50a8fafaa4cc631a56ab3b2
+size 253878368

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25c46a2850e36fe6a43353ef94521e9dbda2ddad2b2211a691e88e5726c689b7
+oid sha256:9949eb5243f36e1086fb0786e44c6a297d02ee20dfc49abad1e0cbc17e7249a1
 size 6033