NLP702-bert-base-uncased_finetuning-distillation_hs768-nh32-nl12

README.md

@@ -15,8 +15,8 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.4056
-- Accuracy: 0.8416
+- Loss: 0.4298
+- Accuracy: 0.8332
 
 ## Model description
 
@@ -50,10 +50,10 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch | Step | Validation Loss | Accuracy |
 |:-------------:|:-----:|:----:|:---------------:|:--------:|
-| 1.495         | 1.39  | 500  | 0.6461          | 0.7467   |
-| 0.437         | 2.78  | 1000 | 0.4789          | 0.8131   |
-| 0.2094        | 4.17  | 1500 | 0.4312          | 0.8337   |
-| 0.1147        | 5.56  | 2000 | 0.3926          | 0.8515   |
+| 1.5966        | 1.39  | 500  | 0.7974          | 0.6719   |
+| 0.5528        | 2.78  | 1000 | 0.5465          | 0.7900   |
+| 0.2857        | 4.17  | 1500 | 0.4594          | 0.8264   |
+| 0.1546        | 5.56  | 2000 | 0.4235          | 0.8401   |
 
 
 ### Framework versions

best/config.json

@@ -136,8 +136,8 @@
   "layer_norm_eps": 1e-12,
   "max_position_embeddings": 512,
   "model_type": "bert",
-  "num_attention_heads": 16,
-  "num_hidden_layers": 8,
+  "num_attention_heads": 32,
+  "num_hidden_layers": 12,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
   "torch_dtype": "float32",

best/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a7f3f5f6c30da89ae2f557f15b5d424e5f0a660d0490be1077e2cf04b60a0d8
-size 324723536
+oid sha256:bc07f53d1945a88e44b1167465147924ab30a65231d714f855e4e5a74e19178d
+size 438137056

config.json

@@ -136,8 +136,8 @@
   "layer_norm_eps": 1e-12,
   "max_position_embeddings": 512,
   "model_type": "bert",
-  "num_attention_heads": 16,
-  "num_hidden_layers": 8,
+  "num_attention_heads": 32,
+  "num_hidden_layers": 12,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
   "torch_dtype": "float32",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a7f3f5f6c30da89ae2f557f15b5d424e5f0a660d0490be1077e2cf04b60a0d8
-size 324723536
+oid sha256:bc07f53d1945a88e44b1167465147924ab30a65231d714f855e4e5a74e19178d
+size 438137056