| # Dissecting BERT Layers: FFN Dual Role, Separability-Guided Layer Skip, and Interpretable Classification |
|
|
| Artifacts for the paper by Yeonseong Cynn (River Lab, May 2026). |
|
|
| ## Summary |
|
|
| Layer-level analysis framework for BERT across five GLUE tasks (SST-2, CoLA, MRPC, QNLI, RTE). |
|
|
| Key findings: |
| - **Separability-guided layer skip**: identifies removable layers via separability delta analysis, validated by actual BERT forward-pass experiments |
| - **FFN dual role**: 92% structural (norm normalization) vs. 8% classification-relevant, explaining why FFN removal collapses models while individual layers appear "harmful" |
| - **Error analysis**: 60-93% of misclassifications are high-confidence errors (margin > 0.3), indicating BERT's CLS representation is the bottleneck |
|
|
| ## Files |
|
|
| ### Weights |
| - `bert_sst2_prune_masks.npz` — Per-layer FFN neuron pruning masks (0/1) for BERT SST-2. Keys: `mask_L0` through `mask_L11`, each shape `(3072,)`. |
|
|
| ### Results (JSON) |
| - `results/{task}_layer_analysis.json` — Layer separability metrics, delta changes, and FFN structural/classification ratio per task |
| - `results/{task}_skip_results.json` — Single and multi-layer skip experiment results per task |
|
|
| Tasks: `sst2`, `cola`, `mrpc`, `qnli`, `rte` |
|
|
| ### Figures |
| - `figures/fig1_separability.png` — Layer separability curves across 5 GLUE tasks |
| - `figures/fig2_ffn_ratio.png` — FFN structural/classification ratio heatmap (log scale) |
| - `figures/fig3_errors.png` — Error direction and confidence analysis |
| - `figures/fig4_skip_prediction.png` — Separability prediction vs actual skip accuracy |
|
|
| ## Usage |
|
|
| ### Loading pruning masks |
| ```python |
| import numpy as np |
| |
| masks = np.load("bert_sst2_prune_masks.npz") |
| for layer in range(12): |
| mask = masks[f"mask_L{layer}"] # (3072,) binary mask |
| kept = mask.sum() |
| print(f"L{layer+1}: {int(kept)}/3072 neurons kept ({kept/3072*100:.0f}%)") |
| ``` |
|
|
| ### Applying masks to BERT |
| ```python |
| import torch |
| from transformers import BertForSequenceClassification |
| |
| model = BertForSequenceClassification.from_pretrained("textattack/bert-base-uncased-SST-2") |
| masks = np.load("bert_sst2_prune_masks.npz") |
| |
| # Zero out pruned neurons in FFN intermediate layer |
| for layer_idx in range(12): |
| mask = torch.tensor(masks[f"mask_L{layer_idx}"], dtype=torch.float32) |
| ffn = model.bert.encoder.layer[layer_idx].intermediate.dense |
| ffn.weight.data *= mask.unsqueeze(1) |
| ffn.bias.data *= mask |
| ``` |
|
|
| ## Base Model |
|
|
| All experiments use [textattack/bert-base-uncased-SST-2](https://huggingface.co/textattack/bert-base-uncased-SST-2) and corresponding task-specific fine-tuned models from [textattack](https://huggingface.co/textattack). |
|
|
| ## License |
|
|
| MIT |
|
|
