leoncynn/paper7-bert-layer-analysis

Dissecting BERT Layers: FFN Dual Role, Separability-Guided Layer Skip, and Interpretable Classification

Artifacts for the paper by Yeonseong Cynn (River Lab, May 2026).

Summary

Layer-level analysis framework for BERT across five GLUE tasks (SST-2, CoLA, MRPC, QNLI, RTE).

Key findings:

• Separability-guided layer skip: identifies removable layers via separability delta analysis, validated by actual BERT forward-pass experiments
• FFN dual role: 92% structural (norm normalization) vs. 8% classification-relevant, explaining why FFN removal collapses models while individual layers appear "harmful"
• Error analysis: 60-93% of misclassifications are high-confidence errors (margin > 0.3), indicating BERT's CLS representation is the bottleneck

Files

Weights

• bert_sst2_prune_masks.npz — Per-layer FFN neuron pruning masks (0/1) for BERT SST-2. Keys: mask_L0 through mask_L11, each shape (3072,).

Results (JSON)

• results/{task}_layer_analysis.json — Layer separability metrics, delta changes, and FFN structural/classification ratio per task
• results/{task}_skip_results.json — Single and multi-layer skip experiment results per task

Tasks: sst2, cola, mrpc, qnli, rte

Figures

• figures/fig1_separability.png — Layer separability curves across 5 GLUE tasks
• figures/fig2_ffn_ratio.png — FFN structural/classification ratio heatmap (log scale)
• figures/fig3_errors.png — Error direction and confidence analysis
• figures/fig4_skip_prediction.png — Separability prediction vs actual skip accuracy

Usage

Loading pruning masks

import numpy as np

masks = np.load("bert_sst2_prune_masks.npz")
for layer in range(12):
    mask = masks[f"mask_L{layer}"]  # (3072,) binary mask
    kept = mask.sum()
    print(f"L{layer+1}: {int(kept)}/3072 neurons kept ({kept/3072*100:.0f}%)")

Applying masks to BERT

import torch
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("textattack/bert-base-uncased-SST-2")
masks = np.load("bert_sst2_prune_masks.npz")

# Zero out pruned neurons in FFN intermediate layer
for layer_idx in range(12):
    mask = torch.tensor(masks[f"mask_L{layer_idx}"], dtype=torch.float32)
    ffn = model.bert.encoder.layer[layer_idx].intermediate.dense
    ffn.weight.data *= mask.unsqueeze(1)
    ffn.bias.data *= mask

Base Model

All experiments use textattack/bert-base-uncased-SST-2 and corresponding task-specific fine-tuned models from textattack.

License

MIT