OCR feature fragility: TopK sparse autoencoders
Sparse autoencoders (SAEs) trained on the hidden states of a frozen historical NER model, used in the paper Collateral Damage: How OCR Errors Spread Through the Features of NER Models (in preparation). They are meant for studying which features break under OCR errors, not as a general-purpose SAE.
Try them in the Space: emanuelaboros/ocr-feature-fragility. Code: EmanuelaBoros/trace-cav, experiments/feature_fragility.
Variants
All are TopK SAEs (Gao et al., 2024): 768 β 2,048 features, exactly 32 active per token, unit-norm decoder columns, AuxK dead-feature loss.
| Folder | NER model | Layer | FVU (dev) | Label agreement when spliced in |
|---|---|---|---|---|
bert-layer8 (main) |
historical-ner-baseline | 8 / 12 | 0.254 | 0.990 (entity tokens 0.951) |
bert-layer4 |
same | 4 / 12 | 0.239 | 0.978 (0.918) |
bert-layer12 |
same | 12 / 12 | 0.155 | 0.996 (0.981) |
gliner-layer8 |
gliner_multi-v2.1 | 8 | 0.210 | 0.966 (0.905) |
"Label agreement when spliced in" is the share of dev tokens whose predicted label is unchanged when the layer output is replaced by the SAE reconstruction. Full diagnostics are in each config.json.
Training
About 1M tokens from the 1,292 official HIPE-OCRepair training references (full documents, aligned 128-token chunks), 10 epochs, batch 1,024, Adam at 3e-4, seed 42. Inputs are centred with the training mean and divided by a scalar RMS (both stored in the checkpoint). No OCR text was used for training, and dev and test documents were held out.
Use
from huggingface_hub import hf_hub_download
import importlib.util, torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
spec = importlib.util.spec_from_file_location("sae", hf_hub_download("emanuelaboros/ocr-feature-fragility-saes", "sae.py"))
sae_mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(sae_mod)
sae = sae_mod.TopKSAE.from_pretrained("emanuelaboros/ocr-feature-fragility-saes", "bert-layer8")
name = "emanuelaboros/historical-ner-baseline"
tok, ner = AutoTokenizer.from_pretrained(name), AutoModelForTokenClassification.from_pretrained(name).eval()
with torch.no_grad():
h = ner(**tok("Zurich, le 3 mai 1887", return_tensors="pt"), output_hidden_states=True).hidden_states[8]
z = sae.encode(h) # (1, tokens, 2048), 32 non-zero per token
hidden_states[8] is the output of block 8 (index 0 is the embeddings). The layer field of config.json gives the matching index.
Limitations
Trained on historical newspapers in English, French and German; behaviour on modern or other-language text is untested. The features are unlabelled. The NER models keep their own licences, and this repository follows the non-commercial licence of the HIPE data and of the BERT tagger.
Model tree for emanuelaboros/ocr-feature-fragility-saes
Base model
dbmdz/bert-base-historic-multilingual-cased