Hugging Face's logo

KitsuVp
/
NeoLLM

Safetensors
English
neollm
causal-lm
research
bf16
attention
normalization
custom_code
Model card Files
xet
 Community
1
KitsuVp commited on
Commit
8cd30bb
·
verified ·
1 Parent(s): f791b90

Model save

Browse files
Files changed (5) hide show
  1. README.md +36 -26
  2. config.json +7 -1
  3. generation_config.json +1 -1
  4. model.safetensors +1 -1
  5. training_args.bin +1 -1
README.md CHANGED
@@ -14,18 +14,34 @@ should probably proofread and complete it, then remove this comment. -->
14
 
15
  This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
  It achieves the following results on the evaluation set:
17
- - Loss: 3.4451
18
- - Ntp Loss: 2.9981
19
- - Tweo Loss: 0.0194
20
- - Nitp Loss: 0.1548
21
- - Total Model Loss: 3.1531
22
- - Pace Step: 46875.0
23
- - Pace Update Due: 1.0
24
- - Pace Beta: 0.0046
25
- - Pace Clip Fraction: 0.0000
26
- - Pace Mean Gain: 0.0626
27
- - Pace Control L2: 0.0937
28
- - Pace Ema Distance L2: 9.3472
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ## Model description
31
 
@@ -55,23 +71,17 @@ The following hyperparameters were used during training:
55
 
56
  ### Training results
57
 
58
- | Training Loss | Epoch | Step | Validation Loss | Loss | Model Loss | Update Due | Beta | Clip Fraction | Mean Gain | Control L2 | Ema Distance L2 |
59
- |:-------------:|:------:|:-------:|:---------------:|:------:|:----------:|:----------:|:------:|:-------------:|:---------:|:----------:|:---------------:|
60
- | 4.3293 | 0.1067 | 5000.0 | 4.1294 | 0.2787 | 3.8522 | 1.0 | 0.0141 | 0.2085 | 0.2833 | 1.4828 | 36.7415 |
61
- | 3.9588 | 0.2133 | 10000.0 | 3.7924 | 0.1655 | 3.5070 | 1.0 | 0.0100 | 0.1817 | 0.2260 | 1.4158 | 56.8642 |
62
- | 3.8245 | 0.32 | 15000.0 | 3.6735 | 0.1547 | 3.3874 | 1.0 | 0.0082 | 0.1703 | 0.2057 | 1.1435 | 67.8131 |
63
- | 3.7585 | 0.4267 | 20000.0 | 3.6112 | 0.1556 | 3.3272 | 1.0 | 0.0071 | 0.1649 | 0.1956 | 1.2229 | 74.0324 |
64
- | 3.7154 | 0.5333 | 25000.0 | 3.5712 | 0.1580 | 3.2889 | 1.0 | 0.0063 | 0.1612 | 0.1906 | 1.0322 | 78.7246 |
65
- | 3.6791 | 0.64 | 30000.0 | 3.5380 | 0.1542 | 3.2499 | 1.0 | 0.0058 | 0.1585 | 0.1869 | 1.2708 | 81.2646 |
66
- | 3.6606 | 0.7467 | 35000.0 | 3.5148 | 0.1540 | 3.2230 | 1.0 | 0.0053 | 0.1595 | 0.1875 | 0.9860 | 83.7974 |
67
- | 3.5787 | 0.8533 | 40000.0 | 3.4789 | 0.1541 | 3.1880 | 1.0 | 0.0050 | 0.1410 | 0.1689 | 0.5123 | 53.5547 |
68
- | 3.5391 | 0.96 | 45000.0 | 3.4501 | 0.1549 | 3.1584 | 1.0 | 0.0047 | 0.0000 | 0.1122 | 0.1787 | 20.1584 |
69
- | 3.5324 | 1.0 | 46875.0 | 3.4451 | 0.1548 | 3.1531 | 1.0 | 0.0046 | 0.0000 | 0.0626 | 0.0937 | 9.3472 |
70
 
71
 
72
  ### Framework versions
73
 
74
- - Transformers 5.12.1
75
- - Pytorch 2.12.1+cu130
76
  - Datasets 5.0.0
77
  - Tokenizers 0.22.2
 
14
 
15
  This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
  It achieves the following results on the evaluation set:
17
+ - Loss: 3.6452
18
+ - Ntp Loss: 3.3628
19
+ - Tweo Loss: 0.0099
20
+ - Nitp Loss: 0.1585
21
+ - Total Model Loss: 3.5214
22
+ - Optimizer Step: 15625.0
23
+ - Optimizer Metrics Due: 0.0
24
+ - Pace Step: 0.0
25
+ - Pace Update Due: 0.0
26
+ - Pace Previous Iterate Active: 0.0
27
+ - Conda Adaptive Scale Active: 1.0
28
+ - Conda Scale Smoothed Metric Exact: 1.0
29
+ - Conda Scale Beta: 0.0080
30
+ - Conda Scale Energy Attenuation: 0.0053
31
+ - Conda Scale Candidate Ratio Mean: 1.2433
32
+ - Conda Scale Smoothed Ratio Mean: 1.2388
33
+ - Conda Scale Applied Ratio Mean: 1.2388
34
+ - Conda Scale Gain Mean: 0.1394
35
+ - Conda Scale Gain Clip Fraction: 0.0
36
+ - Conda Scale Ratio Min: 1.1510
37
+ - Conda Scale Ratio Max: 1.2500
38
+ - Conda Scale Low Limit Fraction: 0.0
39
+ - Conda Scale High Limit Fraction: 0.0
40
+ - Conda Scale Amplified Fraction: 1.0
41
+ - Conda Scale Attenuated Fraction: 0.0
42
+ - Conda Scale Energy Weighted Amplification: 0.2388
43
+ - Conda Scale Energy Weighted Attenuation Applied: 0.0
44
+ - Conda Scale Monitored Matrices: 146.0
45
 
46
  ## Model description
47
 
 
71
 
72
  ### Training results
73
 
74
+ | Training Loss | Epoch | Step | Validation Loss | Loss | Model Loss | Metrics Due | Update Due | Previous Iterate Active | Adaptive Scale Active | Scale Smoothed Metric Exact | Scale Beta | Scale Energy Attenuation | Scale Candidate Ratio Mean | Scale Smoothed Ratio Mean | Scale Applied Ratio Mean | Scale Gain Mean | Scale Gain Clip Fraction | Scale Ratio Min | Scale Ratio Max | Scale Low Limit Fraction | Scale High Limit Fraction | Scale Amplified Fraction | Scale Attenuated Fraction | Scale Energy Weighted Amplification | Scale Energy Weighted Attenuation Applied | Scale Monitored Matrices |
75
+ |:-------------:|:-----:|:----:|:---------------:|:------:|:----------:|:-----------:|:----------:|:-----------------------:|:---------------------:|:---------------------------:|:----------:|:------------------------:|:--------------------------:|:-------------------------:|:------------------------:|:---------------:|:------------------------:|:---------------:|:---------------:|:------------------------:|:-------------------------:|:------------------------:|:-------------------------:|:-----------------------------------:|:-----------------------------------------:|:------------------------:|
76
+ | 4.1674 | 0.32 | 0.0 | 4.1684 | 0.1519 | 4.0475 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0141 | 0.1124 | 1.1095 | 1.1084 | 1.1084 | 0.3022 | 0.2128 | 0.0016 | 1.2497 | 0.0 | 0.0 | 0.8630 | 0.1370 | 0.1385 | 0.0301 | 146.0 |
77
+ | 3.8735 | 0.64 | 0.0 | 3.8442 | 0.1381 | 3.7236 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0100 | 0.0524 | 1.1845 | 1.1793 | 1.1793 | 0.2376 | 0.1869 | 0.7660 | 1.2499 | 0.0 | 0.0 | 0.8904 | 0.1096 | 0.1845 | 0.0052 | 146.0 |
78
+ | 3.6993 | 0.96 | 0.0 | 3.6600 | 0.1609 | 3.5348 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0082 | 0.0197 | 1.2254 | 1.2325 | 1.2325 | 0.1725 | 0.1451 | 1.0918 | 1.2500 | 0.0 | 0.0 | 1.0 | 0.0 | 0.2325 | 0.0 | 146.0 |
79
+ | 3.6818 | 1.0 | 0.0 | 3.6452 | 0.1585 | 3.5214 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0080 | 0.0053 | 1.2433 | 1.2388 | 1.2388 | 0.1394 | 0.0 | 1.1510 | 1.2500 | 0.0 | 0.0 | 1.0 | 0.0 | 0.2388 | 0.0 | 146.0 |
 
 
 
 
 
 
80
 
81
 
82
  ### Framework versions
83
 
84
+ - Transformers 5.13.0
85
+ - Pytorch 2.13.0+cu132
86
  - Datasets 5.0.0
87
  - Tokenizers 0.22.2
config.json CHANGED
@@ -29,6 +29,11 @@
29
  "head_dim": 64,
30
  "hidden_act": "xielu",
31
  "hidden_size": 512,
 
 
 
 
 
32
  "iha_global_layers_use_iha": true,
33
  "iha_local_global_pattern": "LLLLG",
34
  "iha_num_pseudo_heads": 2,
@@ -81,7 +86,7 @@
81
  "stack_memory_cache_size": 2048,
82
  "stack_slots": 16,
83
  "tie_word_embeddings": false,
84
- "transformers_version": "5.12.1",
85
  "tweo_eps": 1e-06,
86
  "tweo_loss_weight": 0.01,
87
  "tweo_power": 4.0,
@@ -95,6 +100,7 @@
95
  "use_fan_residual": false,
96
  "use_gpas": false,
97
  "use_hadamard_o_proj": true,
 
98
  "use_iha": true,
99
  "use_jtokm": false,
100
  "use_laurel": false,
 
29
  "head_dim": 64,
30
  "hidden_act": "xielu",
31
  "hidden_size": 512,
32
+ "hola_chunk_size": 256,
33
+ "hola_logit_bias_init": -2.0,
34
+ "hola_memory_size": 64,
35
+ "hola_rms_gamma_init": 1.0,
36
+ "hola_score_eps": 1e-06,
37
  "iha_global_layers_use_iha": true,
38
  "iha_local_global_pattern": "LLLLG",
39
  "iha_num_pseudo_heads": 2,
 
86
  "stack_memory_cache_size": 2048,
87
  "stack_slots": 16,
88
  "tie_word_embeddings": false,
89
+ "transformers_version": "5.13.0",
90
  "tweo_eps": 1e-06,
91
  "tweo_loss_weight": 0.01,
92
  "tweo_power": 4.0,
 
100
  "use_fan_residual": false,
101
  "use_gpas": false,
102
  "use_hadamard_o_proj": true,
103
+ "use_hola_memory": false,
104
  "use_iha": true,
105
  "use_jtokm": false,
106
  "use_laurel": false,
generation_config.json CHANGED
@@ -7,5 +7,5 @@
7
  "output_attentions": false,
8
  "output_hidden_states": false,
9
  "pad_token_id": 0,
10
- "transformers_version": "5.12.1"
11
  }
 
7
  "output_attentions": false,
8
  "output_hidden_states": false,
9
  "pad_token_id": 0,
10
+ "transformers_version": "5.13.0"
11
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c91b08ab888181d4d563808a8485f5942b35c5a1fc32c418928c3658c3206253
3
  size 240855132
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f5a1ce747e807640c30d998242fd2a410d112ab6ee480ac7f4c2f259f5ad7a7
3
  size 240855132
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b56e21448e5cd1b996aecf663e108757cbe7f9b4b50f8e928bbba2dc4feba375
3
  size 5329
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:301f0b56770ad8de16f47fdcc2651a9d7035e200e0da1f7f4399f68e3dadc4f6
3
  size 5329