Columbidae/Qwen-27B-Pruned-Retrained

Pruned Qwen (Epoch 1)

This is ToastyPigeon/qwen2.5-32b-unnamed-test-model pruned down from 32b -> 27b.

Using PruneMe to find layers to remove resulted in the removal of layers [25, 29) and [36, 43) for a reduction from 64 -> 52 layers.

Trained on 1 epoch of mixed data from the datasets that went into the pre-pruned model (I'll document that later), totaling about ~10M tokens so far of retraining.

Coherent but a little dumb. Likely needs more than 10M tokens of retraining to re-align the layers.