| Experiment: 2026-04-13-000000-kd-pythia28b-feature-deep |
| Teacher: EleutherAI/pythia-2.8b, T=1.5 |
| Loss weights: KD=0.4, Feature=0.2, CE=0.4 |
| Tokenizer: 50277 tokens, teacher vocab_size: 50304 |
| Teacher: 32 layers, 2560 hidden dim |
| Tokenizing data from /home/coder/code/data/pool.txt... |
| Training data: 10000000 tokens |
| Loading teacher: EleutherAI/pythia-2.8b |
| Teacher parameters: 2,775,208,960 |
| Dataset: 9765 sequences |
| Micro batch: 1, Steps/epoch: 610, Total steps: 6100 |
| Creating student model... |
| Student parameters: 190,674,432 |
| Projection head parameters: 31,457,280 |
| Layer mapping (student->teacher): [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30] |
|
|
| === Training === |
| Epoch 1/10 Step 50/610 | Total: 8.1864 | KD: 8.6463 | Feat: 7.3320 | CE: 8.1537 | LR: 0.000082 |
| Epoch 1/10 Step 100/610 | Total: 6.2206 | KD: 6.5567 | Feat: 3.9396 | CE: 7.0249 | LR: 0.000164 |
| Epoch 1/10 Step 150/610 | Total: 5.6501 | KD: 5.3165 | Feat: 3.7812 | CE: 6.9181 | LR: 0.000246 |
| Epoch 1/10 Step 200/610 | Total: 6.4433 | KD: 6.8477 | Feat: 3.5718 | CE: 7.4747 | LR: 0.000328 |
| Epoch 1/10 Step 250/610 | Total: 5.0950 | KD: 5.1406 | Feat: 3.3867 | CE: 5.9034 | LR: 0.000410 |
| Epoch 1/10 Step 300/610 | Total: 5.1363 | KD: 4.9494 | Feat: 3.1440 | CE: 6.3192 | LR: 0.000492 |
| Epoch 1/10 Step 350/610 | Total: 4.6152 | KD: 4.6498 | Feat: 2.7992 | CE: 5.4887 | LR: 0.000500 |
| Epoch 1/10 Step 400/610 | Total: 5.7377 | KD: 6.4735 | Feat: 2.8904 | CE: 6.4255 | LR: 0.000500 |
| Epoch 1/10 Step 450/610 | Total: 4.8944 | KD: 4.8438 | Feat: 2.5622 | CE: 6.1110 | LR: 0.000499 |
| Epoch 1/10 Step 500/610 | Total: 4.1451 | KD: 3.4578 | Feat: 2.3935 | CE: 5.7083 | LR: 0.000499 |
| Epoch 1/10 Step 550/610 | Total: 3.9210 | KD: 3.3376 | Feat: 2.4933 | CE: 5.2183 | LR: 0.000498 |
| Epoch 1/10 Step 600/610 | Total: 4.7073 | KD: 4.7069 | Feat: 2.3479 | CE: 5.8875 | LR: 0.000497 |
| Epoch 1/10 complete | Total: 5.4953 | KD: 5.3816 | Feat: 3.6196 | CE: 6.5468 | PPL: 697.00 | Time: 575.9s |
| Epoch 2/10 Step 50/610 | Total: 3.9160 | KD: 3.5029 | Feat: 2.1437 | CE: 5.2153 | LR: 0.000495 |
| Epoch 2/10 Step 100/610 | Total: 4.0539 | KD: 3.6557 | Feat: 2.1478 | CE: 5.4051 | LR: 0.000494 |
| Epoch 2/10 Step 150/610 | Total: 3.6320 | KD: 3.2882 | Feat: 2.0075 | CE: 4.7880 | LR: 0.000492 |
| Epoch 2/10 Step 200/610 | Total: 3.6028 | KD: 2.6042 | Feat: 2.0134 | CE: 5.3962 | LR: 0.000491 |
| Epoch 2/10 Step 250/610 | Total: 3.8877 | KD: 3.4841 | Feat: 2.0681 | CE: 5.2011 | LR: 0.000489 |
| Epoch 2/10 Step 300/610 | Total: 3.8197 | KD: 3.1101 | Feat: 1.9286 | CE: 5.4748 | LR: 0.000487 |
| Epoch 2/10 Step 350/610 | Total: 4.1354 | KD: 3.8985 | Feat: 2.0296 | CE: 5.4252 | LR: 0.000484 |
| Epoch 2/10 Step 400/610 | Total: 4.1257 | KD: 3.9162 | Feat: 2.0201 | CE: 5.3879 | LR: 0.000482 |
| Epoch 2/10 Step 450/610 | Total: 3.4743 | KD: 3.0214 | Feat: 1.9018 | CE: 4.7133 | LR: 0.000479 |
| Epoch 2/10 Step 500/610 | Total: 3.1596 | KD: 3.1626 | Feat: 3.3694 | CE: 3.0516 | LR: 0.000477 |
| Epoch 2/10 Step 550/610 | Total: 3.4628 | KD: 2.7523 | Feat: 1.8496 | CE: 4.9799 | LR: 0.000474 |
| Epoch 2/10 Step 600/610 | Total: 3.8569 | KD: 3.3902 | Feat: 2.0341 | CE: 5.2351 | LR: 0.000471 |
| Epoch 2/10 complete | Total: 3.9193 | KD: 3.4748 | Feat: 2.0827 | CE: 5.2820 | PPL: 196.77 | Time: 577.0s |
| Epoch 3/10 Step 50/610 | Total: 3.5683 | KD: 2.9345 | Feat: 1.8425 | CE: 5.0651 | LR: 0.000467 |
| Epoch 3/10 Step 100/610 | Total: 3.4096 | KD: 2.8565 | Feat: 1.9666 | CE: 4.6843 | LR: 0.000463 |
| Epoch 3/10 Step 150/610 | Total: 3.3918 | KD: 2.4428 | Feat: 1.8824 | CE: 5.0955 | LR: 0.000459 |
| Epoch 3/10 Step 200/610 | Total: 3.3360 | KD: 2.5212 | Feat: 1.7611 | CE: 4.9383 | LR: 0.000456 |
| Epoch 3/10 Step 250/610 | Total: 3.2771 | KD: 2.5633 | Feat: 1.7582 | CE: 4.7503 | LR: 0.000452 |
| Epoch 3/10 Step 300/610 | Total: 3.4962 | KD: 2.8814 | Feat: 1.9220 | CE: 4.8980 | LR: 0.000448 |
| Epoch 3/10 Step 350/610 | Total: 3.4583 | KD: 3.0312 | Feat: 1.7974 | CE: 4.7158 | LR: 0.000443 |
| Epoch 3/10 Step 400/610 | Total: 3.3889 | KD: 2.6089 | Feat: 1.7761 | CE: 4.9754 | LR: 0.000439 |
| Epoch 3/10 Step 450/610 | Total: 3.2733 | KD: 2.6945 | Feat: 1.8134 | CE: 4.5819 | LR: 0.000435 |
| Epoch 3/10 Step 500/610 | Total: 3.2353 | KD: 2.6182 | Feat: 1.7313 | CE: 4.6044 | LR: 0.000430 |
| Epoch 3/10 Step 550/610 | Total: 3.4044 | KD: 2.4063 | Feat: 1.8698 | CE: 5.1699 | LR: 0.000425 |
| Epoch 3/10 Step 600/610 | Total: 3.4470 | KD: 3.0868 | Feat: 1.8285 | CE: 4.6163 | LR: 0.000420 |
| Epoch 3/10 complete | Total: 3.4092 | KD: 2.8156 | Feat: 1.8659 | CE: 4.7743 | PPL: 118.42 | Time: 611.9s |
| Epoch 4/10 Step 50/610 | Total: 2.9971 | KD: 2.1861 | Feat: 1.6516 | CE: 4.4810 | LR: 0.000414 |
| Epoch 4/10 Step 100/610 | Total: 3.3938 | KD: 2.8755 | Feat: 1.7854 | CE: 4.7163 | LR: 0.000409 |
| Epoch 4/10 Step 150/610 | Total: 2.9963 | KD: 2.4135 | Feat: 1.6882 | CE: 4.2331 | LR: 0.000404 |
| Epoch 4/10 Step 200/610 | Total: 2.7576 | KD: 2.2535 | Feat: 1.6531 | CE: 3.8139 | LR: 0.000398 |
| Epoch 4/10 Step 250/610 | Total: 3.0670 | KD: 2.4556 | Feat: 1.7451 | CE: 4.3393 | LR: 0.000393 |
| Epoch 4/10 Step 300/610 | Total: 2.9537 | KD: 2.1853 | Feat: 1.8698 | CE: 4.2640 | LR: 0.000387 |
| Epoch 4/10 Step 350/610 | Total: 2.8543 | KD: 2.0759 | Feat: 1.6860 | CE: 4.2167 | LR: 0.000382 |
| Epoch 4/10 Step 400/610 | Total: 2.8591 | KD: 2.2062 | Feat: 1.6965 | CE: 4.0934 | LR: 0.000376 |
| Epoch 4/10 Step 450/610 | Total: 2.7729 | KD: 2.0462 | Feat: 1.6157 | CE: 4.0783 | LR: 0.000370 |
| Epoch 4/10 Step 500/610 | Total: 2.9726 | KD: 2.4038 | Feat: 1.6998 | CE: 4.1777 | LR: 0.000364 |
| Epoch 4/10 Step 550/610 | Total: 3.1071 | KD: 2.4301 | Feat: 1.7276 | CE: 4.4737 | LR: 0.000358 |
| Epoch 4/10 Step 600/610 | Total: 3.1766 | KD: 2.3772 | Feat: 1.6459 | CE: 4.7414 | LR: 0.000352 |
| Epoch 4/10 complete | Total: 3.0896 | KD: 2.4215 | Feat: 1.7418 | CE: 4.4317 | PPL: 84.08 | Time: 608.2s |
| Epoch 5/10 Step 50/610 | Total: 3.0564 | KD: 2.4257 | Feat: 1.7416 | CE: 4.3445 | LR: 0.000344 |
| Epoch 5/10 Step 100/610 | Total: 2.8600 | KD: 2.3587 | Feat: 1.6764 | CE: 3.9532 | LR: 0.000338 |
| Epoch 5/10 Step 150/610 | Total: 2.9286 | KD: 2.2813 | Feat: 1.6254 | CE: 4.2275 | LR: 0.000331 |
| Epoch 5/10 Step 200/610 | Total: 2.7952 | KD: 2.1602 | Feat: 1.6791 | CE: 3.9883 | LR: 0.000325 |
| Epoch 5/10 Step 250/610 | Total: 2.7933 | KD: 2.0946 | Feat: 1.7414 | CE: 4.0180 | LR: 0.000319 |
| Epoch 5/10 Step 300/610 | Total: 2.9690 | KD: 2.1790 | Feat: 1.6916 | CE: 4.3977 | LR: 0.000312 |
| Epoch 5/10 Step 350/610 | Total: 3.0617 | KD: 2.4966 | Feat: 1.8075 | CE: 4.2538 | LR: 0.000305 |
| Epoch 5/10 Step 400/610 | Total: 2.8659 | KD: 2.1624 | Feat: 1.6406 | CE: 4.1821 | LR: 0.000299 |
| Epoch 5/10 Step 450/610 | Total: 2.9925 | KD: 2.5149 | Feat: 1.8050 | CE: 4.0639 | LR: 0.000292 |
| Epoch 5/10 Step 500/610 | Total: 2.8271 | KD: 2.0077 | Feat: 1.6424 | CE: 4.2390 | LR: 0.000285 |
| Epoch 5/10 Step 550/610 | Total: 3.0939 | KD: 2.0260 | Feat: 1.7392 | CE: 4.8391 | LR: 0.000279 |
| Epoch 5/10 Step 600/610 | Total: 2.6065 | KD: 1.8440 | Feat: 1.5509 | CE: 3.8967 | LR: 0.000272 |
| Epoch 5/10 complete | Total: 2.8705 | KD: 2.1690 | Feat: 1.6678 | CE: 4.1734 | PPL: 64.93 | Time: 612.2s |
| Epoch 6/10 Step 50/610 | Total: 2.6809 | KD: 2.1161 | Feat: 1.5942 | CE: 3.7891 | LR: 0.000264 |
| Epoch 6/10 Step 100/610 | Total: 2.9745 | KD: 2.2179 | Feat: 1.7695 | CE: 4.3336 | LR: 0.000257 |
| Epoch 6/10 Step 150/610 | Total: 2.8580 | KD: 2.0098 | Feat: 1.6832 | CE: 4.2936 | LR: 0.000250 |
| Epoch 6/10 Step 200/610 | Total: 2.8297 | KD: 2.0033 | Feat: 1.6560 | CE: 4.2428 | LR: 0.000244 |
| Epoch 6/10 Step 250/610 | Total: 2.7607 | KD: 2.2310 | Feat: 1.8511 | CE: 3.7451 | LR: 0.000237 |
| Epoch 6/10 Step 300/610 | Total: 2.6993 | KD: 1.7620 | Feat: 1.6660 | CE: 4.1532 | LR: 0.000230 |
| Epoch 6/10 Step 350/610 | Total: 2.7636 | KD: 2.1197 | Feat: 1.5798 | CE: 3.9995 | LR: 0.000223 |
| Epoch 6/10 Step 400/610 | Total: 2.6216 | KD: 1.9625 | Feat: 1.5277 | CE: 3.8277 | LR: 0.000217 |
| Epoch 6/10 Step 450/610 | Total: 2.6010 | KD: 2.3540 | Feat: 1.8340 | CE: 3.2314 | LR: 0.000210 |
| Epoch 6/10 Step 500/610 | Total: 2.6266 | KD: 1.9309 | Feat: 1.5951 | CE: 3.8382 | LR: 0.000203 |
| Epoch 6/10 Step 550/610 | Total: 2.6250 | KD: 1.8968 | Feat: 1.6129 | CE: 3.8593 | LR: 0.000197 |
| Epoch 6/10 Step 600/610 | Total: 2.7286 | KD: 1.8854 | Feat: 1.5825 | CE: 4.1450 | LR: 0.000190 |
| Epoch 6/10 complete | Total: 2.6987 | KD: 1.9844 | Feat: 1.6198 | CE: 3.9523 | PPL: 52.06 | Time: 612.7s |
| Epoch 7/10 Step 50/610 | Total: 2.5603 | KD: 1.7247 | Feat: 1.5762 | CE: 3.8880 | LR: 0.000182 |
| Epoch 7/10 Step 100/610 | Total: 2.4174 | KD: 1.4712 | Feat: 1.4719 | CE: 3.8363 | LR: 0.000176 |
| Epoch 7/10 Step 150/610 | Total: 2.4575 | KD: 1.5661 | Feat: 1.5924 | CE: 3.7815 | LR: 0.000169 |
| Epoch 7/10 Step 200/610 | Total: 2.4903 | KD: 1.9422 | Feat: 1.6067 | CE: 3.4803 | LR: 0.000163 |
| Epoch 7/10 Step 250/610 | Total: 2.4724 | KD: 1.6219 | Feat: 1.4672 | CE: 3.8255 | LR: 0.000156 |
| Epoch 7/10 Step 300/610 | Total: 2.8538 | KD: 2.5563 | Feat: 1.7433 | CE: 3.7066 | LR: 0.000150 |
| Epoch 7/10 Step 350/610 | Total: 2.6101 | KD: 1.8997 | Feat: 1.6202 | CE: 3.8154 | LR: 0.000144 |
| Epoch 7/10 Step 400/610 | Total: 2.5744 | KD: 1.8232 | Feat: 1.5993 | CE: 3.8132 | LR: 0.000138 |
| Epoch 7/10 Step 450/610 | Total: 2.6047 | KD: 2.0311 | Feat: 1.6986 | CE: 3.6313 | LR: 0.000132 |
| Epoch 7/10 Step 500/610 | Total: 2.3530 | KD: 1.5324 | Feat: 1.5064 | CE: 3.5969 | LR: 0.000126 |
| Epoch 7/10 Step 550/610 | Total: 2.5189 | KD: 1.7109 | Feat: 1.4661 | CE: 3.8533 | LR: 0.000120 |
| Epoch 7/10 Step 600/610 | Total: 2.7000 | KD: 1.9451 | Feat: 1.5581 | CE: 4.0258 | LR: 0.000114 |
| Epoch 7/10 complete | Total: 2.5575 | KD: 1.8442 | Feat: 1.5877 | CE: 3.7556 | PPL: 42.76 | Time: 607.8s |
| Epoch 8/10 Step 50/610 | Total: 2.3942 | KD: 1.7681 | Feat: 1.5518 | CE: 3.4413 | LR: 0.000108 |
| Epoch 8/10 Step 100/610 | Total: 2.4507 | KD: 1.6283 | Feat: 1.4817 | CE: 3.7577 | LR: 0.000102 |
| Epoch 8/10 Step 150/610 | Total: 2.2769 | KD: 1.5257 | Feat: 1.4964 | CE: 3.4185 | LR: 0.000097 |
| Epoch 8/10 Step 200/610 | Total: 2.3906 | KD: 1.7458 | Feat: 1.5641 | CE: 3.4487 | LR: 0.000091 |
| Epoch 8/10 Step 250/610 | Total: 2.3981 | KD: 1.7643 | Feat: 1.5695 | CE: 3.4461 | LR: 0.000086 |
| Epoch 8/10 Step 300/610 | Total: 2.5373 | KD: 1.9043 | Feat: 1.6074 | CE: 3.6353 | LR: 0.000081 |
| Epoch 8/10 Step 350/610 | Total: 2.5661 | KD: 2.1185 | Feat: 1.7089 | CE: 3.4422 | LR: 0.000076 |
| Epoch 8/10 Step 400/610 | Total: 2.0963 | KD: 1.3613 | Feat: 1.3887 | CE: 3.1851 | LR: 0.000071 |
| Epoch 8/10 Step 450/610 | Total: 2.4184 | KD: 1.7480 | Feat: 1.5969 | CE: 3.4995 | LR: 0.000067 |
| Epoch 8/10 Step 500/610 | Total: 2.6513 | KD: 2.1528 | Feat: 1.7659 | CE: 3.5925 | LR: 0.000062 |
| Epoch 8/10 Step 550/610 | Total: 2.2905 | KD: 1.4646 | Feat: 1.4748 | CE: 3.5243 | LR: 0.000058 |
| Epoch 8/10 Step 600/610 | Total: 2.4812 | KD: 1.8646 | Feat: 1.5226 | CE: 3.5770 | LR: 0.000054 |
| Epoch 8/10 complete | Total: 2.4467 | KD: 1.7424 | Feat: 1.5670 | CE: 3.5910 | PPL: 36.27 | Time: 603.7s |
| Epoch 9/10 Step 50/610 | Total: 2.2767 | KD: 1.6577 | Feat: 1.4774 | CE: 3.2953 | LR: 0.000049 |
| Epoch 9/10 Step 100/610 | Total: 2.3890 | KD: 1.6188 | Feat: 1.5837 | CE: 3.5617 | LR: 0.000045 |
| Epoch 9/10 Step 150/610 | Total: 2.3482 | KD: 1.3013 | Feat: 1.4864 | CE: 3.8260 | LR: 0.000041 |
| Epoch 9/10 Step 200/610 | Total: 2.3688 | KD: 1.8216 | Feat: 1.6432 | CE: 3.2787 | LR: 0.000037 |
| Epoch 9/10 Step 250/610 | Total: 2.4384 | KD: 1.7656 | Feat: 1.6477 | CE: 3.5064 | LR: 0.000034 |
| Epoch 9/10 Step 300/610 | Total: 2.3326 | KD: 1.5498 | Feat: 1.5596 | CE: 3.5019 | LR: 0.000030 |
| Epoch 9/10 Step 350/610 | Total: 2.3332 | KD: 1.4673 | Feat: 1.4482 | CE: 3.6417 | LR: 0.000027 |
| Epoch 9/10 Step 400/610 | Total: 2.4788 | KD: 2.1555 | Feat: 1.7858 | CE: 3.1486 | LR: 0.000024 |
| Epoch 9/10 Step 450/610 | Total: 2.1478 | KD: 1.5298 | Feat: 1.5330 | CE: 3.0730 | LR: 0.000021 |
| Epoch 9/10 Step 500/610 | Total: 2.3649 | KD: 1.7084 | Feat: 1.4749 | CE: 3.4664 | LR: 0.000019 |
| Epoch 9/10 Step 550/610 | Total: 2.4433 | KD: 1.6476 | Feat: 1.4891 | CE: 3.7161 | LR: 0.000016 |
| Epoch 9/10 Step 600/610 | Total: 2.5170 | KD: 1.9813 | Feat: 1.5330 | CE: 3.5446 | LR: 0.000014 |
| Epoch 9/10 complete | Total: 2.3728 | KD: 1.6794 | Feat: 1.5554 | CE: 3.4748 | PPL: 32.29 | Time: 610.8s |
| Epoch 10/10 Step 50/610 | Total: 2.0793 | KD: 1.4996 | Feat: 1.4757 | CE: 2.9607 | LR: 0.000011 |
| Epoch 10/10 Step 100/610 | Total: 2.3868 | KD: 1.9817 | Feat: 1.8337 | CE: 3.0684 | LR: 0.000009 |
| Epoch 10/10 Step 150/610 | Total: 2.5163 | KD: 1.8682 | Feat: 1.5748 | CE: 3.6352 | LR: 0.000008 |
| Epoch 10/10 Step 200/610 | Total: 2.4237 | KD: 1.6199 | Feat: 1.5397 | CE: 3.6695 | LR: 0.000006 |
| Epoch 10/10 Step 250/610 | Total: 2.2686 | KD: 1.5143 | Feat: 1.5076 | CE: 3.4034 | LR: 0.000005 |
| Epoch 10/10 Step 300/610 | Total: 2.3278 | KD: 1.7659 | Feat: 1.4888 | CE: 3.3093 | LR: 0.000004 |
| Epoch 10/10 Step 350/610 | Total: 2.1883 | KD: 1.7437 | Feat: 1.5376 | CE: 2.9581 | LR: 0.000002 |
| Epoch 10/10 Step 400/610 | Total: 2.6920 | KD: 2.1745 | Feat: 1.6783 | CE: 3.7163 | LR: 0.000002 |
| Epoch 10/10 Step 450/610 | Total: 2.3675 | KD: 1.6594 | Feat: 1.4961 | CE: 3.5114 | LR: 0.000001 |
| Epoch 10/10 Step 500/610 | Total: 2.1779 | KD: 1.5680 | Feat: 1.4493 | CE: 3.1522 | LR: 0.000000 |
| Epoch 10/10 Step 550/610 | Total: 2.1507 | KD: 1.7401 | Feat: 1.5867 | CE: 2.8432 | LR: 0.000000 |
| Epoch 10/10 Step 600/610 | Total: 2.3170 | KD: 1.6886 | Feat: 1.5340 | CE: 3.3369 | LR: 0.000000 |
| Epoch 10/10 complete | Total: 2.3372 | KD: 1.6512 | Feat: 1.5505 | CE: 3.4166 | PPL: 30.47 | Time: 608.7s |
|
|
| Saving student model... |
| Model saved to /home/coder/experiments/2026-04-13-000000-kd-pythia28b-feature-deep |
|
|
| Training complete! |
|
|
