Hugging Face's logo

lapp0
/
distily_bench_obj_cross_v2.1

TensorBoard
Safetensors
Distily
gpt_neo
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files
xet
Metrics Community
lapp0 commited on
Commit
c53b74a
·
verified ·
1 Parent(s): 1e50d88

End of training

Browse files
Files changed (2) hide show
  1. README.md +22 -22
  2. logs/attn_loss_fn=None, attn_weight=0, gradient_accumulation_steps=1, hs_loss_fn=0, hs_weight=0, learning_rate=0.004, lr_scheduler_type=constant_with_warmup, max_grad_norm=1.0, num_warmup_steps=1000, optim/events.out.tfevents.1723845165.5f530b1cf724 +3 -0
README.md CHANGED
@@ -15,14 +15,14 @@ This student model is distilled from the teacher model [roneneldan/TinyStories-3
15
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
16
 
17
  It achieves the following results on the evaluation set:
18
- - eval_enwikippl: 181.7250
19
- - eval_frwikippl: 74363.8594
20
- - eval_zhwikippl: 2022567.625
21
- - eval_tinystoriesppl: 9.8853
22
- - eval_loss: 1.1866
23
- - eval_runtime: 6.5139
24
- - eval_samples_per_second: 76.759
25
- - eval_steps_per_second: 9.672
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -62,20 +62,20 @@ Peak GPU Memory: 8.0568 GB
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 |
65
- | 0 | 0 | 28486.4824 | 70517.9062 | 6.3282 | 6.4967 | 76.962 | 9.697 | 14258.5928 | 39903.9922 |
66
- | 500 | 0.0808 | 273.2714 | 228192.9688 | 1.3967 | 6.4879 | 77.066 | 9.71 | 11.1741 | 4322949.5 |
67
- | 1000 | 0.1616 | 243.0954 | 144923.9688 | 1.2256 | 6.4896 | 77.046 | 9.708 | 10.8133 | 7570136.5 |
68
- | 1500 | 0.2424 | 203.6588 | 90912.9609 | 1.1940 | 6.4809 | 77.149 | 9.721 | 10.8402 | 2947090.25 |
69
- | 2000 | 0.3232 | 200.2716 | 83404.1484 | 1.1897 | 6.4988 | 76.937 | 9.694 | 10.7438 | 2592153.75 |
70
- | 2500 | 0.4040 | 184.8414 | 76467.0 | 1.1874 | 6.4917 | 77.021 | 9.705 | 9.9579 | 2185851.5 |
71
- | 3000 | 0.4848 | 181.0154 | 75556.9688 | 1.1875 | 6.4856 | 77.094 | 9.714 | 9.7151 | 2026889.125 |
72
- | 3500 | 0.5656 | 180.5813 | 75770.125 | 1.1868 | 6.4769 | 77.198 | 9.727 | 9.7699 | 2169583.75 |
73
- | 4000 | 0.6464 | 183.1808 | 76985.7891 | 1.1867 | 6.4852 | 77.099 | 9.714 | 9.8759 | 2138550.5 |
74
- | 4500 | 0.7272 | 181.8940 | 75908.9922 | 1.1866 | 6.4932 | 77.004 | 9.703 | 9.7946 | 2117543.5 |
75
- | 5000 | 0.8080 | 182.3313 | 75185.3516 | 1.1865 | 6.4852 | 77.098 | 9.714 | 9.8894 | 2145408.0 |
76
- | 5500 | 0.8888 | 181.7320 | 76262.6484 | 1.1871 | 6.4902 | 77.039 | 9.707 | 9.6759 | 2123201.5 |
77
- | 6000 | 0.9696 | 183.1027 | 76574.7891 | 1.1868 | 6.488 | 77.066 | 9.71 | 9.8169 | 2035559.625 |
78
- | 6188 | 1.0 | 181.7250 | 74363.8594 | 1.1866 | 6.5139 | 76.759 | 9.672 | 9.8853 | 2022567.625 |
79
 
80
  ### Framework versions
81
  - Distily 0.2.0
 
15
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
16
 
17
  It achieves the following results on the evaluation set:
18
+ - eval_enwikippl: 173.9496
19
+ - eval_frwikippl: 51516.1875
20
+ - eval_zhwikippl: 493522.8438
21
+ - eval_tinystoriesppl: 9.7845
22
+ - eval_loss: 1.1894
23
+ - eval_runtime: 6.4922
24
+ - eval_samples_per_second: 77.015
25
+ - eval_steps_per_second: 9.704
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 |
65
+ | 0 | 0 | 21321.3555 | 56774.5312 | 6.6010 | 6.5093 | 76.814 | 9.679 | 11289.9248 | 60744.7383 |
66
+ | 500 | 0.0808 | 274.1302 | 180526.1719 | 1.4071 | 6.5099 | 76.806 | 9.678 | 11.2389 | 1649163.5 |
67
+ | 1000 | 0.1616 | 232.6131 | 105189.0703 | 1.2311 | 6.5042 | 76.873 | 9.686 | 10.8138 | 1187807.125 |
68
+ | 1500 | 0.2424 | 199.5283 | 67338.7891 | 1.1984 | 6.4963 | 76.967 | 9.698 | 10.7981 | 709016.375 |
69
+ | 2000 | 0.3232 | 192.2678 | 58199.6094 | 1.1924 | 6.4957 | 76.973 | 9.699 | 10.6832 | 606559.6875 |
70
+ | 2500 | 0.4040 | 183.1808 | 51215.9492 | 1.1903 | 6.4822 | 77.134 | 9.719 | 10.5633 | 500283.6562 |
71
+ | 3000 | 0.4848 | 184.6982 | 51273.6641 | 1.1903 | 6.4934 | 77.001 | 9.702 | 10.7035 | 478993.8125 |
72
+ | 3500 | 0.5656 | 184.5123 | 51534.3672 | 1.1903 | 6.5069 | 76.842 | 9.682 | 10.6876 | 514633.75 |
73
+ | 4000 | 0.6464 | 173.4584 | 51400.2227 | 1.1900 | 6.4884 | 77.06 | 9.71 | 9.7361 | 485814.9688 |
74
+ | 4500 | 0.7272 | 172.5871 | 50727.7070 | 1.1895 | 6.4895 | 77.047 | 9.708 | 9.6995 | 504438.7812 |
75
+ | 5000 | 0.8080 | 175.0851 | 51407.4805 | 1.1897 | 6.4772 | 77.194 | 9.726 | 9.8356 | 502826.4062 |
76
+ | 5500 | 0.8888 | 172.8748 | 51803.6094 | 1.1903 | 6.486 | 77.089 | 9.713 | 9.5756 | 504438.7812 |
77
+ | 6000 | 0.9696 | 175.4109 | 53070.2930 | 1.1896 | 6.5075 | 76.835 | 9.681 | 9.7760 | 502021.9062 |
78
+ | 6188 | 1.0 | 173.9496 | 51516.1875 | 1.1894 | 6.4922 | 77.015 | 9.704 | 9.7845 | 493522.8438 |
79
 
80
  ### Framework versions
81
  - Distily 0.2.0
logs/attn_loss_fn=None, attn_weight=0, gradient_accumulation_steps=1, hs_loss_fn=0, hs_weight=0, learning_rate=0.004, lr_scheduler_type=constant_with_warmup, max_grad_norm=1.0, num_warmup_steps=1000, optim/events.out.tfevents.1723845165.5f530b1cf724 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4093fdf0e90b956cdb2e946f98f1d631d7b53bb53e233493c0498b5746d97a0c
3
+ size 307