Hugging Face's logo

ojaffe
/
babylm-pythia28b-feature-deep

Safetensors
gpt_neox
Model card Files
xet
 Community
babylm-pythia28b-feature-deep
File size: 13,613 Bytes
55b0d32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
Experiment: 2026-04-13-000000-kd-pythia28b-feature-deep
Teacher: EleutherAI/pythia-2.8b, T=1.5
Loss weights: KD=0.4, Feature=0.2, CE=0.4
Tokenizer: 50277 tokens, teacher vocab_size: 50304
Teacher: 32 layers, 2560 hidden dim
Tokenizing data from /home/coder/code/data/pool.txt...
Training data: 10000000 tokens
Loading teacher: EleutherAI/pythia-2.8b
Teacher parameters: 2,775,208,960
Dataset: 9765 sequences
Micro batch: 1, Steps/epoch: 610, Total steps: 6100
Creating student model...
Student parameters: 190,674,432
Projection head parameters: 31,457,280
Layer mapping (student->teacher): [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30]

=== Training ===
  Epoch 1/10 Step 50/610 | Total: 8.1864 | KD: 8.6463 | Feat: 7.3320 | CE: 8.1537 | LR: 0.000082
  Epoch 1/10 Step 100/610 | Total: 6.2206 | KD: 6.5567 | Feat: 3.9396 | CE: 7.0249 | LR: 0.000164
  Epoch 1/10 Step 150/610 | Total: 5.6501 | KD: 5.3165 | Feat: 3.7812 | CE: 6.9181 | LR: 0.000246
  Epoch 1/10 Step 200/610 | Total: 6.4433 | KD: 6.8477 | Feat: 3.5718 | CE: 7.4747 | LR: 0.000328
  Epoch 1/10 Step 250/610 | Total: 5.0950 | KD: 5.1406 | Feat: 3.3867 | CE: 5.9034 | LR: 0.000410
  Epoch 1/10 Step 300/610 | Total: 5.1363 | KD: 4.9494 | Feat: 3.1440 | CE: 6.3192 | LR: 0.000492
  Epoch 1/10 Step 350/610 | Total: 4.6152 | KD: 4.6498 | Feat: 2.7992 | CE: 5.4887 | LR: 0.000500
  Epoch 1/10 Step 400/610 | Total: 5.7377 | KD: 6.4735 | Feat: 2.8904 | CE: 6.4255 | LR: 0.000500
  Epoch 1/10 Step 450/610 | Total: 4.8944 | KD: 4.8438 | Feat: 2.5622 | CE: 6.1110 | LR: 0.000499
  Epoch 1/10 Step 500/610 | Total: 4.1451 | KD: 3.4578 | Feat: 2.3935 | CE: 5.7083 | LR: 0.000499
  Epoch 1/10 Step 550/610 | Total: 3.9210 | KD: 3.3376 | Feat: 2.4933 | CE: 5.2183 | LR: 0.000498
  Epoch 1/10 Step 600/610 | Total: 4.7073 | KD: 4.7069 | Feat: 2.3479 | CE: 5.8875 | LR: 0.000497
Epoch 1/10 complete | Total: 5.4953 | KD: 5.3816 | Feat: 3.6196 | CE: 6.5468 | PPL: 697.00 | Time: 575.9s
  Epoch 2/10 Step 50/610 | Total: 3.9160 | KD: 3.5029 | Feat: 2.1437 | CE: 5.2153 | LR: 0.000495
  Epoch 2/10 Step 100/610 | Total: 4.0539 | KD: 3.6557 | Feat: 2.1478 | CE: 5.4051 | LR: 0.000494
  Epoch 2/10 Step 150/610 | Total: 3.6320 | KD: 3.2882 | Feat: 2.0075 | CE: 4.7880 | LR: 0.000492
  Epoch 2/10 Step 200/610 | Total: 3.6028 | KD: 2.6042 | Feat: 2.0134 | CE: 5.3962 | LR: 0.000491
  Epoch 2/10 Step 250/610 | Total: 3.8877 | KD: 3.4841 | Feat: 2.0681 | CE: 5.2011 | LR: 0.000489
  Epoch 2/10 Step 300/610 | Total: 3.8197 | KD: 3.1101 | Feat: 1.9286 | CE: 5.4748 | LR: 0.000487
  Epoch 2/10 Step 350/610 | Total: 4.1354 | KD: 3.8985 | Feat: 2.0296 | CE: 5.4252 | LR: 0.000484
  Epoch 2/10 Step 400/610 | Total: 4.1257 | KD: 3.9162 | Feat: 2.0201 | CE: 5.3879 | LR: 0.000482
  Epoch 2/10 Step 450/610 | Total: 3.4743 | KD: 3.0214 | Feat: 1.9018 | CE: 4.7133 | LR: 0.000479
  Epoch 2/10 Step 500/610 | Total: 3.1596 | KD: 3.1626 | Feat: 3.3694 | CE: 3.0516 | LR: 0.000477
  Epoch 2/10 Step 550/610 | Total: 3.4628 | KD: 2.7523 | Feat: 1.8496 | CE: 4.9799 | LR: 0.000474
  Epoch 2/10 Step 600/610 | Total: 3.8569 | KD: 3.3902 | Feat: 2.0341 | CE: 5.2351 | LR: 0.000471
Epoch 2/10 complete | Total: 3.9193 | KD: 3.4748 | Feat: 2.0827 | CE: 5.2820 | PPL: 196.77 | Time: 577.0s
  Epoch 3/10 Step 50/610 | Total: 3.5683 | KD: 2.9345 | Feat: 1.8425 | CE: 5.0651 | LR: 0.000467
  Epoch 3/10 Step 100/610 | Total: 3.4096 | KD: 2.8565 | Feat: 1.9666 | CE: 4.6843 | LR: 0.000463
  Epoch 3/10 Step 150/610 | Total: 3.3918 | KD: 2.4428 | Feat: 1.8824 | CE: 5.0955 | LR: 0.000459
  Epoch 3/10 Step 200/610 | Total: 3.3360 | KD: 2.5212 | Feat: 1.7611 | CE: 4.9383 | LR: 0.000456
  Epoch 3/10 Step 250/610 | Total: 3.2771 | KD: 2.5633 | Feat: 1.7582 | CE: 4.7503 | LR: 0.000452
  Epoch 3/10 Step 300/610 | Total: 3.4962 | KD: 2.8814 | Feat: 1.9220 | CE: 4.8980 | LR: 0.000448
  Epoch 3/10 Step 350/610 | Total: 3.4583 | KD: 3.0312 | Feat: 1.7974 | CE: 4.7158 | LR: 0.000443
  Epoch 3/10 Step 400/610 | Total: 3.3889 | KD: 2.6089 | Feat: 1.7761 | CE: 4.9754 | LR: 0.000439
  Epoch 3/10 Step 450/610 | Total: 3.2733 | KD: 2.6945 | Feat: 1.8134 | CE: 4.5819 | LR: 0.000435
  Epoch 3/10 Step 500/610 | Total: 3.2353 | KD: 2.6182 | Feat: 1.7313 | CE: 4.6044 | LR: 0.000430
  Epoch 3/10 Step 550/610 | Total: 3.4044 | KD: 2.4063 | Feat: 1.8698 | CE: 5.1699 | LR: 0.000425
  Epoch 3/10 Step 600/610 | Total: 3.4470 | KD: 3.0868 | Feat: 1.8285 | CE: 4.6163 | LR: 0.000420
Epoch 3/10 complete | Total: 3.4092 | KD: 2.8156 | Feat: 1.8659 | CE: 4.7743 | PPL: 118.42 | Time: 611.9s
  Epoch 4/10 Step 50/610 | Total: 2.9971 | KD: 2.1861 | Feat: 1.6516 | CE: 4.4810 | LR: 0.000414
  Epoch 4/10 Step 100/610 | Total: 3.3938 | KD: 2.8755 | Feat: 1.7854 | CE: 4.7163 | LR: 0.000409
  Epoch 4/10 Step 150/610 | Total: 2.9963 | KD: 2.4135 | Feat: 1.6882 | CE: 4.2331 | LR: 0.000404
  Epoch 4/10 Step 200/610 | Total: 2.7576 | KD: 2.2535 | Feat: 1.6531 | CE: 3.8139 | LR: 0.000398
  Epoch 4/10 Step 250/610 | Total: 3.0670 | KD: 2.4556 | Feat: 1.7451 | CE: 4.3393 | LR: 0.000393
  Epoch 4/10 Step 300/610 | Total: 2.9537 | KD: 2.1853 | Feat: 1.8698 | CE: 4.2640 | LR: 0.000387
  Epoch 4/10 Step 350/610 | Total: 2.8543 | KD: 2.0759 | Feat: 1.6860 | CE: 4.2167 | LR: 0.000382
  Epoch 4/10 Step 400/610 | Total: 2.8591 | KD: 2.2062 | Feat: 1.6965 | CE: 4.0934 | LR: 0.000376
  Epoch 4/10 Step 450/610 | Total: 2.7729 | KD: 2.0462 | Feat: 1.6157 | CE: 4.0783 | LR: 0.000370
  Epoch 4/10 Step 500/610 | Total: 2.9726 | KD: 2.4038 | Feat: 1.6998 | CE: 4.1777 | LR: 0.000364
  Epoch 4/10 Step 550/610 | Total: 3.1071 | KD: 2.4301 | Feat: 1.7276 | CE: 4.4737 | LR: 0.000358
  Epoch 4/10 Step 600/610 | Total: 3.1766 | KD: 2.3772 | Feat: 1.6459 | CE: 4.7414 | LR: 0.000352
Epoch 4/10 complete | Total: 3.0896 | KD: 2.4215 | Feat: 1.7418 | CE: 4.4317 | PPL: 84.08 | Time: 608.2s
  Epoch 5/10 Step 50/610 | Total: 3.0564 | KD: 2.4257 | Feat: 1.7416 | CE: 4.3445 | LR: 0.000344
  Epoch 5/10 Step 100/610 | Total: 2.8600 | KD: 2.3587 | Feat: 1.6764 | CE: 3.9532 | LR: 0.000338
  Epoch 5/10 Step 150/610 | Total: 2.9286 | KD: 2.2813 | Feat: 1.6254 | CE: 4.2275 | LR: 0.000331
  Epoch 5/10 Step 200/610 | Total: 2.7952 | KD: 2.1602 | Feat: 1.6791 | CE: 3.9883 | LR: 0.000325
  Epoch 5/10 Step 250/610 | Total: 2.7933 | KD: 2.0946 | Feat: 1.7414 | CE: 4.0180 | LR: 0.000319
  Epoch 5/10 Step 300/610 | Total: 2.9690 | KD: 2.1790 | Feat: 1.6916 | CE: 4.3977 | LR: 0.000312
  Epoch 5/10 Step 350/610 | Total: 3.0617 | KD: 2.4966 | Feat: 1.8075 | CE: 4.2538 | LR: 0.000305
  Epoch 5/10 Step 400/610 | Total: 2.8659 | KD: 2.1624 | Feat: 1.6406 | CE: 4.1821 | LR: 0.000299
  Epoch 5/10 Step 450/610 | Total: 2.9925 | KD: 2.5149 | Feat: 1.8050 | CE: 4.0639 | LR: 0.000292
  Epoch 5/10 Step 500/610 | Total: 2.8271 | KD: 2.0077 | Feat: 1.6424 | CE: 4.2390 | LR: 0.000285
  Epoch 5/10 Step 550/610 | Total: 3.0939 | KD: 2.0260 | Feat: 1.7392 | CE: 4.8391 | LR: 0.000279
  Epoch 5/10 Step 600/610 | Total: 2.6065 | KD: 1.8440 | Feat: 1.5509 | CE: 3.8967 | LR: 0.000272
Epoch 5/10 complete | Total: 2.8705 | KD: 2.1690 | Feat: 1.6678 | CE: 4.1734 | PPL: 64.93 | Time: 612.2s
  Epoch 6/10 Step 50/610 | Total: 2.6809 | KD: 2.1161 | Feat: 1.5942 | CE: 3.7891 | LR: 0.000264
  Epoch 6/10 Step 100/610 | Total: 2.9745 | KD: 2.2179 | Feat: 1.7695 | CE: 4.3336 | LR: 0.000257
  Epoch 6/10 Step 150/610 | Total: 2.8580 | KD: 2.0098 | Feat: 1.6832 | CE: 4.2936 | LR: 0.000250
  Epoch 6/10 Step 200/610 | Total: 2.8297 | KD: 2.0033 | Feat: 1.6560 | CE: 4.2428 | LR: 0.000244
  Epoch 6/10 Step 250/610 | Total: 2.7607 | KD: 2.2310 | Feat: 1.8511 | CE: 3.7451 | LR: 0.000237
  Epoch 6/10 Step 300/610 | Total: 2.6993 | KD: 1.7620 | Feat: 1.6660 | CE: 4.1532 | LR: 0.000230
  Epoch 6/10 Step 350/610 | Total: 2.7636 | KD: 2.1197 | Feat: 1.5798 | CE: 3.9995 | LR: 0.000223
  Epoch 6/10 Step 400/610 | Total: 2.6216 | KD: 1.9625 | Feat: 1.5277 | CE: 3.8277 | LR: 0.000217
  Epoch 6/10 Step 450/610 | Total: 2.6010 | KD: 2.3540 | Feat: 1.8340 | CE: 3.2314 | LR: 0.000210
  Epoch 6/10 Step 500/610 | Total: 2.6266 | KD: 1.9309 | Feat: 1.5951 | CE: 3.8382 | LR: 0.000203
  Epoch 6/10 Step 550/610 | Total: 2.6250 | KD: 1.8968 | Feat: 1.6129 | CE: 3.8593 | LR: 0.000197
  Epoch 6/10 Step 600/610 | Total: 2.7286 | KD: 1.8854 | Feat: 1.5825 | CE: 4.1450 | LR: 0.000190
Epoch 6/10 complete | Total: 2.6987 | KD: 1.9844 | Feat: 1.6198 | CE: 3.9523 | PPL: 52.06 | Time: 612.7s
  Epoch 7/10 Step 50/610 | Total: 2.5603 | KD: 1.7247 | Feat: 1.5762 | CE: 3.8880 | LR: 0.000182
  Epoch 7/10 Step 100/610 | Total: 2.4174 | KD: 1.4712 | Feat: 1.4719 | CE: 3.8363 | LR: 0.000176
  Epoch 7/10 Step 150/610 | Total: 2.4575 | KD: 1.5661 | Feat: 1.5924 | CE: 3.7815 | LR: 0.000169
  Epoch 7/10 Step 200/610 | Total: 2.4903 | KD: 1.9422 | Feat: 1.6067 | CE: 3.4803 | LR: 0.000163
  Epoch 7/10 Step 250/610 | Total: 2.4724 | KD: 1.6219 | Feat: 1.4672 | CE: 3.8255 | LR: 0.000156
  Epoch 7/10 Step 300/610 | Total: 2.8538 | KD: 2.5563 | Feat: 1.7433 | CE: 3.7066 | LR: 0.000150
  Epoch 7/10 Step 350/610 | Total: 2.6101 | KD: 1.8997 | Feat: 1.6202 | CE: 3.8154 | LR: 0.000144
  Epoch 7/10 Step 400/610 | Total: 2.5744 | KD: 1.8232 | Feat: 1.5993 | CE: 3.8132 | LR: 0.000138
  Epoch 7/10 Step 450/610 | Total: 2.6047 | KD: 2.0311 | Feat: 1.6986 | CE: 3.6313 | LR: 0.000132
  Epoch 7/10 Step 500/610 | Total: 2.3530 | KD: 1.5324 | Feat: 1.5064 | CE: 3.5969 | LR: 0.000126
  Epoch 7/10 Step 550/610 | Total: 2.5189 | KD: 1.7109 | Feat: 1.4661 | CE: 3.8533 | LR: 0.000120
  Epoch 7/10 Step 600/610 | Total: 2.7000 | KD: 1.9451 | Feat: 1.5581 | CE: 4.0258 | LR: 0.000114
Epoch 7/10 complete | Total: 2.5575 | KD: 1.8442 | Feat: 1.5877 | CE: 3.7556 | PPL: 42.76 | Time: 607.8s
  Epoch 8/10 Step 50/610 | Total: 2.3942 | KD: 1.7681 | Feat: 1.5518 | CE: 3.4413 | LR: 0.000108
  Epoch 8/10 Step 100/610 | Total: 2.4507 | KD: 1.6283 | Feat: 1.4817 | CE: 3.7577 | LR: 0.000102
  Epoch 8/10 Step 150/610 | Total: 2.2769 | KD: 1.5257 | Feat: 1.4964 | CE: 3.4185 | LR: 0.000097
  Epoch 8/10 Step 200/610 | Total: 2.3906 | KD: 1.7458 | Feat: 1.5641 | CE: 3.4487 | LR: 0.000091
  Epoch 8/10 Step 250/610 | Total: 2.3981 | KD: 1.7643 | Feat: 1.5695 | CE: 3.4461 | LR: 0.000086
  Epoch 8/10 Step 300/610 | Total: 2.5373 | KD: 1.9043 | Feat: 1.6074 | CE: 3.6353 | LR: 0.000081
  Epoch 8/10 Step 350/610 | Total: 2.5661 | KD: 2.1185 | Feat: 1.7089 | CE: 3.4422 | LR: 0.000076
  Epoch 8/10 Step 400/610 | Total: 2.0963 | KD: 1.3613 | Feat: 1.3887 | CE: 3.1851 | LR: 0.000071
  Epoch 8/10 Step 450/610 | Total: 2.4184 | KD: 1.7480 | Feat: 1.5969 | CE: 3.4995 | LR: 0.000067
  Epoch 8/10 Step 500/610 | Total: 2.6513 | KD: 2.1528 | Feat: 1.7659 | CE: 3.5925 | LR: 0.000062
  Epoch 8/10 Step 550/610 | Total: 2.2905 | KD: 1.4646 | Feat: 1.4748 | CE: 3.5243 | LR: 0.000058
  Epoch 8/10 Step 600/610 | Total: 2.4812 | KD: 1.8646 | Feat: 1.5226 | CE: 3.5770 | LR: 0.000054
Epoch 8/10 complete | Total: 2.4467 | KD: 1.7424 | Feat: 1.5670 | CE: 3.5910 | PPL: 36.27 | Time: 603.7s
  Epoch 9/10 Step 50/610 | Total: 2.2767 | KD: 1.6577 | Feat: 1.4774 | CE: 3.2953 | LR: 0.000049
  Epoch 9/10 Step 100/610 | Total: 2.3890 | KD: 1.6188 | Feat: 1.5837 | CE: 3.5617 | LR: 0.000045
  Epoch 9/10 Step 150/610 | Total: 2.3482 | KD: 1.3013 | Feat: 1.4864 | CE: 3.8260 | LR: 0.000041
  Epoch 9/10 Step 200/610 | Total: 2.3688 | KD: 1.8216 | Feat: 1.6432 | CE: 3.2787 | LR: 0.000037
  Epoch 9/10 Step 250/610 | Total: 2.4384 | KD: 1.7656 | Feat: 1.6477 | CE: 3.5064 | LR: 0.000034
  Epoch 9/10 Step 300/610 | Total: 2.3326 | KD: 1.5498 | Feat: 1.5596 | CE: 3.5019 | LR: 0.000030
  Epoch 9/10 Step 350/610 | Total: 2.3332 | KD: 1.4673 | Feat: 1.4482 | CE: 3.6417 | LR: 0.000027
  Epoch 9/10 Step 400/610 | Total: 2.4788 | KD: 2.1555 | Feat: 1.7858 | CE: 3.1486 | LR: 0.000024
  Epoch 9/10 Step 450/610 | Total: 2.1478 | KD: 1.5298 | Feat: 1.5330 | CE: 3.0730 | LR: 0.000021
  Epoch 9/10 Step 500/610 | Total: 2.3649 | KD: 1.7084 | Feat: 1.4749 | CE: 3.4664 | LR: 0.000019
  Epoch 9/10 Step 550/610 | Total: 2.4433 | KD: 1.6476 | Feat: 1.4891 | CE: 3.7161 | LR: 0.000016
  Epoch 9/10 Step 600/610 | Total: 2.5170 | KD: 1.9813 | Feat: 1.5330 | CE: 3.5446 | LR: 0.000014
Epoch 9/10 complete | Total: 2.3728 | KD: 1.6794 | Feat: 1.5554 | CE: 3.4748 | PPL: 32.29 | Time: 610.8s
  Epoch 10/10 Step 50/610 | Total: 2.0793 | KD: 1.4996 | Feat: 1.4757 | CE: 2.9607 | LR: 0.000011
  Epoch 10/10 Step 100/610 | Total: 2.3868 | KD: 1.9817 | Feat: 1.8337 | CE: 3.0684 | LR: 0.000009
  Epoch 10/10 Step 150/610 | Total: 2.5163 | KD: 1.8682 | Feat: 1.5748 | CE: 3.6352 | LR: 0.000008
  Epoch 10/10 Step 200/610 | Total: 2.4237 | KD: 1.6199 | Feat: 1.5397 | CE: 3.6695 | LR: 0.000006
  Epoch 10/10 Step 250/610 | Total: 2.2686 | KD: 1.5143 | Feat: 1.5076 | CE: 3.4034 | LR: 0.000005
  Epoch 10/10 Step 300/610 | Total: 2.3278 | KD: 1.7659 | Feat: 1.4888 | CE: 3.3093 | LR: 0.000004
  Epoch 10/10 Step 350/610 | Total: 2.1883 | KD: 1.7437 | Feat: 1.5376 | CE: 2.9581 | LR: 0.000002
  Epoch 10/10 Step 400/610 | Total: 2.6920 | KD: 2.1745 | Feat: 1.6783 | CE: 3.7163 | LR: 0.000002
  Epoch 10/10 Step 450/610 | Total: 2.3675 | KD: 1.6594 | Feat: 1.4961 | CE: 3.5114 | LR: 0.000001
  Epoch 10/10 Step 500/610 | Total: 2.1779 | KD: 1.5680 | Feat: 1.4493 | CE: 3.1522 | LR: 0.000000
  Epoch 10/10 Step 550/610 | Total: 2.1507 | KD: 1.7401 | Feat: 1.5867 | CE: 2.8432 | LR: 0.000000
  Epoch 10/10 Step 600/610 | Total: 2.3170 | KD: 1.6886 | Feat: 1.5340 | CE: 3.3369 | LR: 0.000000
Epoch 10/10 complete | Total: 2.3372 | KD: 1.6512 | Feat: 1.5505 | CE: 3.4166 | PPL: 30.47 | Time: 608.7s

Saving student model...
Model saved to /home/coder/experiments/2026-04-13-000000-kd-pythia28b-feature-deep

Training complete!