aieng-lab/gradiend-multilingual-bert-features

GRADIEND features in multilingual BERT

45 pretrained GRADIEND models for google-bert/bert-base-multilingual-cased. Each model learns one feature inside the base model, for example he vs she, singular vs plural pronouns, the case of German articles or positive vs negative sentiment. GRADIEND learns a feature by training an encoder-decoder on the gradients of the base model. The learned weights show where the base model encodes the feature, and they can be used to change it, for example to debias it.

The models belong to the paper The GRADIEND Python Package: An End-to-End System for Gradient-Based Feature Learning (Drechsel and Herbold, EMNLP 2026 System Demonstrations), which presents the gradiend Python package. The same features were also trained for another base model, Qwen/Qwen2.5-0.5B: aieng-lab/gradiend-qwen2.5-0.5b-features.

Models

Feature	Models	Datasets	Run ids
German articles (gender x case)	23	aieng-lab/de-gender-case-articles neutral: aieng-lab/wortschatz-leipzig-de-grammar-neutral	gender_de_fem_acc_fem_dat, gender_de_fem_acc_fem_gen, gender_de_fem_acc_masc_acc, ...
English gender (he/she)	1	aieng-lab/genter, aieng-lab/namexact neutral: aieng-lab/geneutral	gender_en
Pronouns: number	1	aieng-lab/en-pronouns neutral: aieng-lab/en-pronoun-neutral	pronoun_number_singular_plural
Pronouns: person	3	aieng-lab/en-pronouns neutral: aieng-lab/en-pronoun-neutral	pronoun_person_1vs2, pronoun_person_1vs3, pronoun_person_2vs3
English pronoun pairs	10	aieng-lab/en-pronouns neutral: aieng-lab/en-pronoun-neutral	pronoun_1PL_2SGPL, pronoun_1PL_3PL, pronoun_1PL_3SG, ...
Sentiment	1	aieng-lab/en-sentiment-nrc neutral: aieng-lab/en-sentiment-nrc-neutral	sentiment_positive_negative
Race	3	aieng-lab/gradiend_race_data neutral: aieng-lab/biasneutral	race_black_asian, race_white_asian, race_white_black
Religion	3	aieng-lab/gradiend_religion_data neutral: aieng-lab/biasneutral	religion_christian_jewish, religion_christian_muslim, religion_muslim_jewish

The run id of a model is the name of its folder in this repository. All run ids:

Show all 45 run ids

gender_de_fem_acc_fem_dat, gender_de_fem_acc_fem_gen, gender_de_fem_acc_masc_acc, gender_de_fem_acc_neut_acc, gender_de_fem_dat_masc_dat, gender_de_fem_dat_neut_dat, gender_de_fem_gen_masc_gen, gender_de_fem_gen_neut_gen, gender_de_fem_nom_fem_dat, gender_de_fem_nom_fem_gen, gender_de_fem_nom_neut_nom, gender_de_masc_acc_masc_dat, gender_de_masc_dat_masc_gen, gender_de_masc_nom_fem_nom, gender_de_masc_nom_masc_acc, gender_de_masc_nom_masc_dat, gender_de_masc_nom_masc_gen, gender_de_masc_nom_neut_nom, gender_de_neut_acc_neut_dat, gender_de_neut_acc_neut_gen, gender_de_neut_dat_neut_gen, gender_de_neut_nom_neut_dat, gender_de_neut_nom_neut_gen, gender_en, pronoun_1PL_2SGPL, pronoun_1PL_3PL, pronoun_1PL_3SG, pronoun_1SG_1PL, pronoun_1SG_2SGPL, pronoun_1SG_3PL, pronoun_1SG_3SG, pronoun_2SGPL_3PL, pronoun_2SGPL_3SG, pronoun_3SG_3PL, pronoun_number_singular_plural, pronoun_person_1vs2, pronoun_person_1vs3, pronoun_person_2vs3, race_black_asian, race_white_asian, race_white_black, religion_christian_jewish, religion_christian_muslim, religion_muslim_jewish, sentiment_positive_negative

or list them from the Hub:

from huggingface_hub import list_repo_files

run_ids = sorted({f.split("/")[0] for f in list_repo_files("aieng-lab/gradiend-multilingual-bert-features") if f.endswith("/model/config.json")})

Feature comparison

Top-k overlap: the share of the 1000 most important decoder weights that two models have in common. Models of related features overlap, unrelated features hardly do.

Cross-encoding: how the model of one feature (row) responds to the data of another feature (column). +100 / -100 means the data is encoded like the model's own feature, in the same / opposite direction; 0 means no response.

Use

pip install gradiend

Load one model (here the singular vs plural pronoun model):

from pathlib import Path

from huggingface_hub import snapshot_download

from gradiend.model import ParamMappedGradiendModel

root = Path(snapshot_download("aieng-lab/gradiend-multilingual-bert-features", allow_patterns=["pronoun_3SG_3PL/model/*"]))
model = ParamMappedGradiendModel.from_pretrained(root / "pronoun_3SG_3PL" / "model")
print(model.base_model_id, model.input_dim, model.latent_dim)

Compare several models by top-k overlap (needs gradiend>=0.2.2):

from gradiend import plot_topk_overlap_heatmap

ids = ["pronoun_3SG_3PL", "pronoun_1SG_3PL", "gender_de_masc_nom_fem_nom"]
root = Path(snapshot_download("aieng-lab/gradiend-multilingual-bert-features", allow_patterns=[f"{i}/model/*" for i in ids]))
models = {i: ParamMappedGradiendModel.from_pretrained(root / i / "model") for i in ids}
plot_topk_overlap_heatmap(models, topk=1000, part="decoder-weight", value="intersection_frac")

The notebook 03_compare_features.ipynb compares all 45 models of this repository.

Files

• <run id>/model/: the GRADIEND model, loaded with ParamMappedGradiendModel.from_pretrained
• <run id>/training_convergence.png: training curve
• <run id>/encoded_values*, cross_task_transition_order.json: cached encoder evaluations, from which experiments/multilingual_gradiend_demo.py redraws the figures above (--plot-only --experiment-dir aieng-lab/gradiend-multilingual-bert-features)

Training

The models were trained with the gradiend Python package (PyPI, documentation) by experiments/multilingual_gradiend_demo.py. Options that are not listed here are the package defaults. Objective: masked language modelling (MLM). Settings per feature (read from each model's model/training.json):

Feature	Learning rate	Steps	Batch size	Pre-/post-pruning (top-k)
German articles (gender x case)	1e-6	1000	16	10% / 1%
English gender (he/she)	1e-4	100	8	10% / 1%
Pronouns: number	1e-5	1000	16	10% / 1%
Pronouns: person	1e-5	1000	16	10% / 1%
English pronoun pairs	1e-4	1000	16	10% / 1%
Sentiment	1e-4	15000	8	1% / 1%
Race	1e-5	1000	16	10% / 1%
Religion	1e-5	1000	16	10% / 1%

Citation

@misc{drechsel2026gradiendpythonpackage,
      title={The {GRADIEND} Python Package: An End-to-End System for Gradient-Based Feature Learning},
      author={Jonathan Drechsel and Steffen Herbold},
      year={2026},
      eprint={2602.23993},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.23993},
}

@inproceedings{drechsel2026gradiend,
    title={{GRADIEND}: Feature Learning within Neural Networks Exemplified through Biases},
    author={Jonathan Drechsel and Steffen Herbold},
    booktitle={The Fourteenth International Conference on Learning Representations},
    year={2026},
    url={https://openreview.net/forum?id=1vBNAnAgCD}
}