Experiment: 2026-04-12-210000-kd-pythia28b-feature-rdrop
Teacher: EleutherAI/pythia-2.8b, T=1.5
Loss weights: KD=0.4, Feature=0.2, CE=0.3, RDrop=0.1
Tokenizer: 50277 tokens, teacher vocab_size: 50304
Teacher: 32 layers, 2560 hidden dim
Tokenizing data from /home/coder/code/data/pool.txt...
Training data: 10000000 tokens
Loading teacher: EleutherAI/pythia-2.8b
Teacher parameters: 2,775,208,960
Dataset: 9765 sequences
Micro batch: 1, Steps/epoch: 305, Total steps: 3050
Creating student model...
Student parameters: 162,322,944
Projection head parameters: 23,592,960
Layer mapping (student->teacher): [0, 2, 5, 8, 10, 13, 16, 18, 21, 24, 26, 29]

=== Training with R-Drop ===
  Epoch 1/10 Step 50/305 | Total: 6.0214 | KD: 6.9677 | Feat: 5.0412 | CE: 7.4172 | RDrop: 0.0092 | LR: 0.000164
  Epoch 1/10 Step 100/305 | Total: 5.2800 | KD: 5.7856 | Feat: 4.2464 | CE: 7.0461 | RDrop: 0.0266 | LR: 0.000329
  Epoch 1/10 Step 150/305 | Total: 5.0634 | KD: 5.7360 | Feat: 3.5665 | CE: 6.8415 | RDrop: 0.0321 | LR: 0.000493
  Epoch 1/10 Step 200/305 | Total: 4.1893 | KD: 4.5079 | Feat: 3.4622 | CE: 5.6353 | RDrop: 0.0312 | LR: 0.000500
  Epoch 1/10 Step 250/305 | Total: 4.1731 | KD: 4.2357 | Feat: 3.1450 | CE: 6.1520 | RDrop: 0.0417 | LR: 0.000499
  Epoch 1/10 Step 300/305 | Total: 4.4054 | KD: 4.6918 | Feat: 2.8016 | CE: 6.5448 | RDrop: 0.0494 | LR: 0.000497
Epoch 1/10 complete | Total: 5.3027 | KD: 5.8904 | Feat: 4.3536 | CE: 6.9089 | RDrop: 0.0319 | PPL: 1001.10 | Time: 650.0s
  Epoch 2/10 Step 50/305 | Total: 3.8968 | KD: 4.2439 | Feat: 2.6071 | CE: 5.5753 | RDrop: 0.0522 | LR: 0.000494
  Epoch 2/10 Step 100/305 | Total: 3.8220 | KD: 4.2396 | Feat: 2.4365 | CE: 5.4430 | RDrop: 0.0595 | LR: 0.000491
  Epoch 2/10 Step 150/305 | Total: 3.3392 | KD: 3.3337 | Feat: 2.3149 | CE: 5.1213 | RDrop: 0.0639 | LR: 0.000487
  Epoch 2/10 Step 200/305 | Total: 3.3445 | KD: 3.7147 | Feat: 2.1189 | CE: 4.7617 | RDrop: 0.0635 | LR: 0.000482
  Epoch 2/10 Step 250/305 | Total: 3.7643 | KD: 4.1607 | Feat: 2.1857 | CE: 5.5153 | RDrop: 0.0829 | LR: 0.000477
  Epoch 2/10 Step 300/305 | Total: 3.2070 | KD: 3.2016 | Feat: 2.0687 | CE: 5.0159 | RDrop: 0.0784 | LR: 0.000470
Epoch 2/10 complete | Total: 3.7289 | KD: 3.9185 | Feat: 2.3421 | CE: 5.6221 | RDrop: 0.0646 | PPL: 276.46 | Time: 653.9s
  Epoch 3/10 Step 50/305 | Total: 3.2531 | KD: 3.3110 | Feat: 1.9368 | CE: 5.1125 | RDrop: 0.0754 | LR: 0.000463
  Epoch 3/10 Step 100/305 | Total: 2.9885 | KD: 2.9719 | Feat: 1.8368 | CE: 4.7486 | RDrop: 0.0782 | LR: 0.000456
  Epoch 3/10 Step 150/305 | Total: 2.8978 | KD: 2.5139 | Feat: 1.8793 | CE: 5.0260 | RDrop: 0.0855 | LR: 0.000448
  Epoch 3/10 Step 200/305 | Total: 3.1070 | KD: 3.2472 | Feat: 1.8652 | CE: 4.7568 | RDrop: 0.0801 | LR: 0.000439
  Epoch 3/10 Step 250/305 | Total: 3.1707 | KD: 3.0595 | Feat: 1.8389 | CE: 5.2359 | RDrop: 0.0829 | LR: 0.000430
  Epoch 3/10 Step 300/305 | Total: 3.1604 | KD: 3.1890 | Feat: 1.8093 | CE: 5.0432 | RDrop: 0.0999 | LR: 0.000420
Epoch 3/10 complete | Total: 3.2387 | KD: 3.2571 | Feat: 1.9343 | CE: 5.1343 | RDrop: 0.0876 | PPL: 169.74 | Time: 676.1s
  Epoch 4/10 Step 50/305 | Total: 2.9752 | KD: 3.0603 | Feat: 1.9605 | CE: 4.4961 | RDrop: 0.1016 | LR: 0.000409
  Epoch 4/10 Step 100/305 | Total: 3.1861 | KD: 3.3058 | Feat: 1.8652 | CE: 4.9268 | RDrop: 0.1272 | LR: 0.000398
  Epoch 4/10 Step 150/305 | Total: 2.8691 | KD: 2.5714 | Feat: 1.7743 | CE: 4.9160 | RDrop: 0.1083 | LR: 0.000387
  Epoch 4/10 Step 200/305 | Total: 2.9842 | KD: 2.4550 | Feat: 1.8794 | CE: 5.3837 | RDrop: 0.1124 | LR: 0.000376
  Epoch 4/10 Step 250/305 | Total: 2.9845 | KD: 2.4031 | Feat: 1.8831 | CE: 5.4501 | RDrop: 0.1159 | LR: 0.000364
  Epoch 4/10 Step 300/305 | Total: 2.9162 | KD: 2.8203 | Feat: 1.6641 | CE: 4.8119 | RDrop: 0.1166 | LR: 0.000352
Epoch 4/10 complete | Total: 2.9210 | KD: 2.7991 | Feat: 1.7886 | CE: 4.7755 | RDrop: 0.1094 | PPL: 118.57 | Time: 652.7s
  Epoch 5/10 Step 50/305 | Total: 2.6400 | KD: 2.1276 | Feat: 1.6590 | CE: 4.8167 | RDrop: 0.1215 | LR: 0.000338
  Epoch 5/10 Step 100/305 | Total: 2.9993 | KD: 3.0259 | Feat: 1.7573 | CE: 4.7426 | RDrop: 0.1472 | LR: 0.000325
  Epoch 5/10 Step 150/305 | Total: 2.8508 | KD: 2.6582 | Feat: 1.6954 | CE: 4.7853 | RDrop: 0.1287 | LR: 0.000312
  Epoch 5/10 Step 200/305 | Total: 2.5812 | KD: 2.3281 | Feat: 1.6114 | CE: 4.3861 | RDrop: 0.1183 | LR: 0.000299
  Epoch 5/10 Step 250/305 | Total: 2.7241 | KD: 2.2188 | Feat: 1.6133 | CE: 5.0051 | RDrop: 0.1243 | LR: 0.000285
  Epoch 5/10 Step 300/305 | Total: 2.6038 | KD: 2.1958 | Feat: 1.5930 | CE: 4.6510 | RDrop: 0.1151 | LR: 0.000272
Epoch 5/10 complete | Total: 2.6982 | KD: 2.4815 | Feat: 1.7033 | CE: 4.5087 | RDrop: 0.1233 | PPL: 90.81 | Time: 654.7s
  Epoch 6/10 Step 50/305 | Total: 2.7998 | KD: 2.9051 | Feat: 1.7456 | CE: 4.2534 | RDrop: 0.1266 | LR: 0.000257
  Epoch 6/10 Step 100/305 | Total: 2.5711 | KD: 2.2946 | Feat: 1.6035 | CE: 4.3971 | RDrop: 0.1345 | LR: 0.000243
  Epoch 6/10 Step 150/305 | Total: 2.5412 | KD: 2.2438 | Feat: 1.6529 | CE: 4.3312 | RDrop: 0.1373 | LR: 0.000230
  Epoch 6/10 Step 200/305 | Total: 2.2760 | KD: 1.8798 | Feat: 1.6278 | CE: 3.9532 | RDrop: 0.1257 | LR: 0.000216
  Epoch 6/10 Step 250/305 | Total: 2.2424 | KD: 2.0361 | Feat: 1.6052 | CE: 3.6508 | RDrop: 0.1168 | LR: 0.000203
  Epoch 6/10 Step 300/305 | Total: 2.4578 | KD: 2.0573 | Feat: 1.6518 | CE: 4.2996 | RDrop: 0.1468 | LR: 0.000190
Epoch 6/10 complete | Total: 2.5423 | KD: 2.2667 | Feat: 1.6449 | CE: 4.3108 | RDrop: 0.1347 | PPL: 74.50 | Time: 677.9s
  Epoch 7/10 Step 50/305 | Total: 2.4728 | KD: 2.2836 | Feat: 1.6764 | CE: 4.0322 | RDrop: 0.1442 | LR: 0.000176
  Epoch 7/10 Step 100/305 | Total: 2.4739 | KD: 2.0296 | Feat: 1.5273 | CE: 4.4732 | RDrop: 0.1464 | LR: 0.000163
  Epoch 7/10 Step 150/305 | Total: 2.3404 | KD: 2.0472 | Feat: 1.5688 | CE: 3.9795 | RDrop: 0.1393 | LR: 0.000150
  Epoch 7/10 Step 200/305 | Total: 2.1892 | KD: 1.8245 | Feat: 1.5321 | CE: 3.8015 | RDrop: 0.1250 | LR: 0.000138
  Epoch 7/10 Step 250/305 | Total: 2.2750 | KD: 1.7955 | Feat: 1.5471 | CE: 4.1092 | RDrop: 0.1456 | LR: 0.000126
  Epoch 7/10 Step 300/305 | Total: 2.5758 | KD: 2.2461 | Feat: 1.6734 | CE: 4.4213 | RDrop: 0.1630 | LR: 0.000114
Epoch 7/10 complete | Total: 2.4291 | KD: 2.1155 | Feat: 1.6058 | CE: 4.1572 | RDrop: 0.1460 | PPL: 63.89 | Time: 723.9s
  Epoch 8/10 Step 50/305 | Total: 2.3862 | KD: 1.7968 | Feat: 1.5950 | CE: 4.4417 | RDrop: 0.1602 | LR: 0.000102
  Epoch 8/10 Step 100/305 | Total: 2.4481 | KD: 2.1217 | Feat: 1.6270 | CE: 4.1943 | RDrop: 0.1571 | LR: 0.000091
  Epoch 8/10 Step 150/305 | Total: 2.2883 | KD: 2.0677 | Feat: 1.5516 | CE: 3.7873 | RDrop: 0.1472 | LR: 0.000081
  Epoch 8/10 Step 200/305 | Total: 2.4604 | KD: 2.2386 | Feat: 1.6053 | CE: 4.0937 | RDrop: 0.1583 | LR: 0.000071
  Epoch 8/10 Step 250/305 | Total: 2.2566 | KD: 1.7832 | Feat: 1.5026 | CE: 4.0915 | RDrop: 0.1531 | LR: 0.000062
  Epoch 8/10 Step 300/305 | Total: 2.2104 | KD: 1.5839 | Feat: 1.4529 | CE: 4.2391 | RDrop: 0.1455 | LR: 0.000054
Epoch 8/10 complete | Total: 2.3479 | KD: 2.0100 | Feat: 1.5802 | CE: 4.0406 | RDrop: 0.1565 | PPL: 56.86 | Time: 710.3s
  Epoch 9/10 Step 50/305 | Total: 2.3016 | KD: 1.9420 | Feat: 1.5301 | CE: 4.0077 | RDrop: 0.1646 | LR: 0.000045
  Epoch 9/10 Step 100/305 | Total: 2.2020 | KD: 1.9314 | Feat: 1.5711 | CE: 3.6674 | RDrop: 0.1499 | LR: 0.000037
  Epoch 9/10 Step 150/305 | Total: 2.3650 | KD: 1.9143 | Feat: 1.5938 | CE: 4.2121 | RDrop: 0.1690 | LR: 0.000030
  Epoch 9/10 Step 200/305 | Total: 2.5223 | KD: 2.3367 | Feat: 1.6243 | CE: 4.1522 | RDrop: 0.1707 | LR: 0.000024
  Epoch 9/10 Step 250/305 | Total: 2.2730 | KD: 1.8212 | Feat: 1.5544 | CE: 4.0576 | RDrop: 0.1633 | LR: 0.000019
  Epoch 9/10 Step 300/305 | Total: 2.3371 | KD: 2.1022 | Feat: 1.5315 | CE: 3.9084 | RDrop: 0.1744 | LR: 0.000014
Epoch 9/10 complete | Total: 2.2978 | KD: 1.9466 | Feat: 1.5667 | CE: 3.9646 | RDrop: 0.1642 | PPL: 52.70 | Time: 713.1s
  Epoch 10/10 Step 50/305 | Total: 2.3142 | KD: 1.9352 | Feat: 1.5293 | CE: 4.0545 | RDrop: 0.1790 | LR: 0.000009
  Epoch 10/10 Step 100/305 | Total: 2.2459 | KD: 1.8950 | Feat: 1.5121 | CE: 3.8968 | RDrop: 0.1645 | LR: 0.000006
  Epoch 10/10 Step 150/305 | Total: 2.1932 | KD: 1.8486 | Feat: 1.5336 | CE: 3.7680 | RDrop: 0.1658 | LR: 0.000004
  Epoch 10/10 Step 200/305 | Total: 2.3417 | KD: 2.1580 | Feat: 1.6941 | CE: 3.7407 | RDrop: 0.1746 | LR: 0.000002
  Epoch 10/10 Step 250/305 | Total: 2.1040 | KD: 1.5644 | Feat: 1.4711 | CE: 3.8992 | RDrop: 0.1429 | LR: 0.000000
  Epoch 10/10 Step 300/305 | Total: 2.4579 | KD: 2.2141 | Feat: 1.5720 | CE: 4.1276 | RDrop: 0.1957 | LR: 0.000000
Epoch 10/10 complete | Total: 2.2748 | KD: 1.9188 | Feat: 1.5616 | CE: 3.9275 | RDrop: 0.1672 | PPL: 50.78 | Time: 710.3s

Saving student model...
Model saved to /home/coder/experiments/2026-04-12-210000-kd-pythia28b-feature-rdrop

Training complete!