leoncynn/paper8-cross-model-compression

Structural FFN Decomposition Guides Cross-Model Compression and Quantization

Artifacts for the paper by Yeonseong Cynn (River Lab, May 2026).

Summary

Decomposes transformer FFN layers into structural (format-preserving) and classification-relevant components across BERT and GPT-2.

Key findings:

• Early-layer FFN is 90-200x more structural than classification-relevant; late layers approach 1:1
• Structural pruning: head + FFN neuron removal with layer-wise retraining achieves 19.1% parameter reduction on BERT (SST-2) and 9.1% on GPT-2 with no accuracy loss
• Neuron pruning: removing 8% rarely-active FFN neurons improves BERT accuracy by 0.3%
• Mixed-precision quantization: INT4 on structurally-dominant layers (L1-L3) with STE retraining recovers to -2.1% loss

Files

Weights

• bert_sst2_int4_ste.pt — BERT SST-2 with L1-L3 INT4 quantization + STE retraining. Standard BERT state_dict, loadable directly. Accuracy: 90.1% (original FP32: 92.4%).

Results — BERT (results/bert/)

• bert_structural_prune.json — Per-layer structural pruning results (head/FFN reduction, accuracy)
• bert_sst2_all_prune.json — All-layer simultaneous FFN pruning results
• bert_l8_prune_results.json — L8 FFN correction + pruning (multi-seed)
• bert_quantize_results.json — INT4/INT8 post-training quantization results
• bert_quantize_retrain.json — INT4 STE retraining results

Results — GPT-2 (results/gpt2/)

• gpt2_structural_prune.json — Per-layer structural pruning (head + FFN)
• gpt2_each_layer_prune.json — Individual layer compression results
• gpt2_prune_validate.json — Pruning validation (PPL, accuracy)

Figures

• figures/fig1_ratio.png — FFN dual role ratio: BERT vs GPT-2 (log scale)
• figures/fig2_compression.png — Per-layer compression rates comparison
• figures/fig3_pruning.png — BERT SST-2 FFN neuron pruning curve
• figures/fig4_quantization.png — INT4 quantization results (PTQ vs STE)

Base Models

• BERT: textattack/bert-base-uncased-SST-2
• GPT-2: gpt2 (124M, pre-trained)

License

MIT