# Structural FFN Decomposition Guides Cross-Model Compression and Quantization

Artifacts for the paper by Yeonseong Cynn (River Lab, May 2026).

## Summary

Decomposes transformer FFN layers into structural (format-preserving) and classification-relevant components across BERT and GPT-2.

Key findings:
- Early-layer FFN is 90-200x more structural than classification-relevant; late layers approach 1:1
- **Structural pruning**: head + FFN neuron removal with layer-wise retraining achieves 19.1% parameter reduction on BERT (SST-2) and 9.1% on GPT-2 with no accuracy loss
- **Neuron pruning**: removing 8% rarely-active FFN neurons *improves* BERT accuracy by 0.3%
- **Mixed-precision quantization**: INT4 on structurally-dominant layers (L1-L3) with STE retraining recovers to -2.1% loss

## Files

### Weights
- `bert_sst2_int4_ste.pt` — BERT SST-2 with L1-L3 INT4 quantization + STE retraining. Standard BERT state_dict, loadable directly. Accuracy: 90.1% (original FP32: 92.4%).

### Results — BERT (`results/bert/`)
- `bert_structural_prune.json` — Per-layer structural pruning results (head/FFN reduction, accuracy)
- `bert_sst2_all_prune.json` — All-layer simultaneous FFN pruning results
- `bert_l8_prune_results.json` — L8 FFN correction + pruning (multi-seed)
- `bert_quantize_results.json` — INT4/INT8 post-training quantization results
- `bert_quantize_retrain.json` — INT4 STE retraining results

### Results — GPT-2 (`results/gpt2/`)
- `gpt2_structural_prune.json` — Per-layer structural pruning (head + FFN)
- `gpt2_each_layer_prune.json` — Individual layer compression results
- `gpt2_prune_validate.json` — Pruning validation (PPL, accuracy)

### Figures
- `figures/fig1_ratio.png` — FFN dual role ratio: BERT vs GPT-2 (log scale)
- `figures/fig2_compression.png` — Per-layer compression rates comparison
- `figures/fig3_pruning.png` — BERT SST-2 FFN neuron pruning curve
- `figures/fig4_quantization.png` — INT4 quantization results (PTQ vs STE)

## Base Models

- BERT: [textattack/bert-base-uncased-SST-2](https://huggingface.co/textattack/bert-base-uncased-SST-2)
- GPT-2: [gpt2](https://huggingface.co/gpt2) (124M, pre-trained)

## License

MIT