BEN2 pruning test # 1 - distillation (epoch 2)
Original parameters: 94,134,658
Pruned parameters: 36,275,254
Parameter reduction: 61.5%
- Swin Transformer depths: [2, 2, 18, 2] → [2, 2, 6, 2]
- Swin Transformer heads: [4, 8, 16, 32] → [3, 6, 12, 24]
distill loss: 0.0001
