model_15M_small_ds_masking_0.1_predicted_hparamas

README.md

@@ -16,8 +16,8 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.1351
-- Accuracy: 0.9525
+- Loss: 0.3599
+- Accuracy: 0.8744
 
 ## Model description
 
@@ -36,12 +36,10 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.003303
+- learning_rate: 0.003439
 - train_batch_size: 256
 - eval_batch_size: 256
 - seed: 42
-- gradient_accumulation_steps: 16
-- total_train_batch_size: 4096
 - optimizer: Use OptimizerNames.SCHEDULE_FREE_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: constant
 - lr_scheduler_warmup_steps: 1000
@@ -52,42 +50,16 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss | Accuracy |
 |:-------------:|:------:|:----:|:---------------:|:--------:|
-| No log        | 0      | 0    | 4.7921          | 0.0008   |
-| No log        | 0.0044 | 122  | 0.7441          | 0.7665   |
-| No log        | 0.0087 | 244  | 0.4206          | 0.8615   |
-| No log        | 0.0131 | 366  | 0.3098          | 0.8961   |
-| No log        | 0.0175 | 488  | 0.2602          | 0.9122   |
-| 11.7806       | 0.0218 | 610  | 0.2386          | 0.9188   |
-| 11.7806       | 0.0262 | 732  | 0.2316          | 0.9214   |
-| 11.7806       | 0.0306 | 854  | 0.2172          | 0.9259   |
-| 11.7806       | 0.0349 | 976  | 0.2144          | 0.9264   |
-| 4.0915        | 0.0393 | 1098 | 0.2056          | 0.9295   |
-| 4.0915        | 0.0437 | 1220 | 0.1973          | 0.9327   |
-| 4.0915        | 0.0480 | 1342 | 0.1923          | 0.9341   |
-| 4.0915        | 0.0524 | 1464 | 0.1848          | 0.9366   |
-| 3.3969        | 0.0568 | 1586 | 0.1802          | 0.9380   |
-| 3.3969        | 0.0611 | 1708 | 0.1739          | 0.9403   |
-| 3.3969        | 0.0655 | 1830 | 0.1687          | 0.9415   |
-| 3.3969        | 0.0699 | 1952 | 0.1638          | 0.9434   |
-| 2.8823        | 0.0743 | 2074 | 0.1608          | 0.9444   |
-| 2.8823        | 0.0786 | 2196 | 0.1563          | 0.9459   |
-| 2.8823        | 0.0830 | 2318 | 0.1532          | 0.9467   |
-| 2.8823        | 0.0874 | 2440 | 0.1498          | 0.9480   |
-| 2.6025        | 0.0917 | 2562 | 0.1489          | 0.9482   |
-| 2.6025        | 0.0961 | 2684 | 0.1465          | 0.9491   |
-| 2.6025        | 0.1005 | 2806 | 0.1423          | 0.9503   |
-| 2.6025        | 0.1048 | 2928 | 0.1417          | 0.9505   |
-| 2.4119        | 0.1092 | 3050 | 0.1393          | 0.9512   |
-| 2.4119        | 0.1136 | 3172 | 0.1383          | 0.9515   |
-| 2.4119        | 0.1179 | 3294 | 0.1357          | 0.9527   |
-| 2.4119        | 0.1223 | 3416 | 0.1342          | 0.9533   |
-| 2.2796        | 0.1267 | 3538 | 0.1344          | 0.9531   |
-| 2.2796        | 0.1310 | 3660 | 0.1352          | 0.9529   |
+| No log        | 0      | 0    | 4.6089          | 0.0017   |
+| 0.4423        | 0.4302 | 1953 | 0.3749          | 0.8701   |
+| 0.3952        | 0.8604 | 3906 | 0.3602          | 0.8745   |
+| 0.4069        | 1.2905 | 5859 | 0.3640          | 0.8738   |
+| 0.4012        | 1.7207 | 7812 | 0.3674          | 0.8711   |
 
 
 ### Framework versions
 
-- Transformers 4.50.1
-- Pytorch 2.8.0.dev20250325+cu128
-- Datasets 3.4.1
+- Transformers 4.51.2
+- Pytorch 2.8.0.dev20250410+cu128
+- Datasets 3.5.0
 - Tokenizers 0.21.1

config.json

@@ -17,10 +17,10 @@
   "global_attn_every_n_layers": 1,
   "global_rope_theta": 160000.0,
   "hidden_activation": "gelu",
-  "hidden_size": 768,
+  "hidden_size": 384,
   "initializer_cutoff_factor": 2.0,
   "initializer_range": 0.02,
-  "intermediate_size": 1152,
+  "intermediate_size": 576,
   "local_attention": 128,
   "local_rope_theta": 10000.0,
   "max_position_embeddings": 502,
@@ -29,15 +29,14 @@
   "model_type": "modernbert",
   "norm_bias": false,
   "norm_eps": 1e-05,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 22,
+  "num_attention_heads": 6,
+  "num_hidden_layers": 12,
   "pad_token_id": 1,
-  "reference_compile": true,
   "repad_logits_with_grad": false,
   "sep_token_id": 3,
   "sparse_pred_ignore_index": -100,
   "sparse_prediction": false,
   "torch_dtype": "float32",
-  "transformers_version": "4.50.1",
+  "transformers_version": "4.51.2",
   "vocab_size": 82
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2f1a8afcad2257ffdc80b29e011bb4b6938ec1392006e49822d69ab2a4e1f33c
-size 443955224
+oid sha256:de3dea857f6e6cad9d8ee484718c303aaa6109a7bc51f33aa4e590c361b1b73e
+size 60925776

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eda8fc4856bc15beae646fcc0308df8cba95eec480ebb8fbb26821042e287aa8
+oid sha256:3a2b46f80daa977eea7bab37ba38e09ff666b2e9052ba40aad0a4a0380a48b6a
 size 5905