README.md

---
library_name: transformers
license: mit
tags:
  - protein-language-model
  - biology
  - AMPLIFY
---

# AMPLIFY-ablations

Pretraining data ablations from **"pLM representations unlock metagenomic space beyond homology"** (Le Breton et al.). They are research artifacts for studying how training data composition shapes protein language model representations. For the final models the paper releases, use [`flair-bio/AMPLIFY-B`](https://huggingface.co/flair-bio/AMPLIFY-B) or [`flair-bio/AMPLIFY-C`](https://huggingface.co/flair-bio/AMPLIFY-C).

Every model is a subfolder, grouped by ablation family. All models share the 120M architecture (24 layers, hidden size 640). Most were trained for 100k steps (about 200B tokens). Names ending in `_1m` were trained for 1M steps (about 2T tokens). The `clustering-fixed-epochs` models are the exception, running a fixed 2 epochs over their clustered set, so their step count varies with dataset size. The exact data mixture behind each name (RED filtering quantile, clustering threshold, metagenomic subsampling rate) is tabulated in the paper's supplementary material. Per-model evaluation scores are in the supplementary data under `in_silico_results.csv`.

## Usage

```python
from transformers import AutoModel, AutoTokenizer

repo = "flair-bio/AMPLIFY-ablations"
model = AutoModel.from_pretrained(repo, subfolder="uniref100-filtering/MILA_U100_0.0", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="uniref100-filtering/MILA_U100_0.0", trust_remote_code=True)

inputs = tokenizer(["MSVVGIDLGFQSCYVAVARAGGIETIANEYSDRCTPACISF"], return_tensors="pt")
out = model(**inputs, output_hidden_states=True)
```

## Models

### `uniref100-filtering/`

UniRef100 at a sweep of RED filtering quantiles. (SI Table S7.)

| Subfolder | Training steps |
|---|---|
| `uniref100-filtering/MILA_U100_0.0` | 100,000 |
| `uniref100-filtering/MILA_U100_0.01` | 100,000 |
| `uniref100-filtering/MILA_U100_0.02` | 100,000 |
| `uniref100-filtering/MILA_U100_0.03` | 100,000 |
| `uniref100-filtering/MILA_U100_0.05` | 100,000 |
| `uniref100-filtering/MILA_U100_0.05_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.10` | 100,000 |
| `uniref100-filtering/MILA_U100_0.10_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.15` | 100,000 |
| `uniref100-filtering/MILA_U100_0.15_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.20` | 100,000 |
| `uniref100-filtering/MILA_U100_0.20_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.25` | 100,000 |
| `uniref100-filtering/MILA_U100_0.25_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.30` | 100,000 |
| `uniref100-filtering/MILA_U100_0.30_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.35` | 100,000 |
| `uniref100-filtering/MILA_U100_0.35_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.40` | 100,000 |
| `uniref100-filtering/MILA_U100_0.40_1m` | 1,000,000 |
| `uniref100-filtering/MILA_U100_0.50` | 100,000 |
| `uniref100-filtering/MILA_U100_0.60` | 100,000 |
| `uniref100-filtering/MILA_U100_0.65` | 100,000 |
| `uniref100-filtering/MILA_U100_0.75` | 100,000 |

### `bfd-filtering/`

BFD at a sweep of RED filtering quantiles. (SI Table S8.)

| Subfolder | Training steps |
|---|---|
| `bfd-filtering/MILA_BFD_0.10` | 100,000 |
| `bfd-filtering/MILA_BFD_0.20` | 100,000 |
| `bfd-filtering/MILA_BFD_0.20_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.30` | 100,000 |
| `bfd-filtering/MILA_BFD_0.30_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.40` | 100,000 |
| `bfd-filtering/MILA_BFD_0.40_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.50` | 100,000 |
| `bfd-filtering/MILA_BFD_0.50_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.60` | 100,000 |
| `bfd-filtering/MILA_BFD_0.60_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.70` | 100,000 |
| `bfd-filtering/MILA_BFD_0.70_1m` | 1,000,000 |
| `bfd-filtering/MILA_BFD_0.80` | 100,000 |
| `bfd-filtering/MILA_BFD_0.80_1m` | 1,000,000 |

### `mgnify-filtering/`

MGnify at a sweep of RED filtering quantiles. (SI Table S9.)

| Subfolder | Training steps |
|---|---|
| `mgnify-filtering/MILA_MGnify_0.10` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.20` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.20_1m` | 1,000,000 |
| `mgnify-filtering/MILA_MGnify_0.30` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.30_1m` | 1,000,000 |
| `mgnify-filtering/MILA_MGnify_0.40` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.40_1m` | 1,000,000 |
| `mgnify-filtering/MILA_MGnify_0.50` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.60` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.70` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.80` | 100,000 |
| `mgnify-filtering/MILA_MGnify_0.90` | 100,000 |

### `combined-filtering/`

UniRef100, BFD and MGnify filtered together at a common RED quantile. (SI Table S10.)

| Subfolder | Training steps |
|---|---|
| `combined-filtering/MILA_ALL_0.30` | 100,000 |
| `combined-filtering/MILA_ALL_0.40` | 100,000 |
| `combined-filtering/MILA_ALL_0.50` | 100,000 |
| `combined-filtering/MILA_ALL_0.50_1m` | 1,000,000 |
| `combined-filtering/MILA_ALL_0.60` | 100,000 |
| `combined-filtering/MILA_ALL_0.60_1m` | 1,000,000 |
| `combined-filtering/MILA_ALL_0.70` | 100,000 |
| `combined-filtering/MILA_ALL_0.70_1m` | 1,000,000 |
| `combined-filtering/MILA_ALL_0.80` | 100,000 |
| `combined-filtering/MILA_ALL_0.80_1m` | 1,000,000 |
| `combined-filtering/MILA_ALL_0.90` | 100,000 |

### `subsampling/`

Filtered metagenomic data mixed with UniRef100 at a sweep of subsampling rates. (SI Table S11.)

| Subfolder | Training steps |
|---|---|
| `subsampling/MILA_OPT_0.10_sub10` | 100,000 |
| `subsampling/MILA_OPT_0.10_sub10_1m` | 1,000,000 |
| `subsampling/MILA_OPT_0.20_sub20` | 100,000 |
| `subsampling/MILA_OPT_0.20_sub20_1m` | 1,000,000 |
| `subsampling/MILA_OPT_0.30_sub30_1m` | 1,000,000 |
| `subsampling/MILA_OPT_0.40_sub40_1m` | 1,000,000 |
| `subsampling/MILA_OPT_0.50_sub50` | 100,000 |
| `subsampling/MILA_OPT_0.60_sub60` | 100,000 |

### `red-subsampling/`

A common RED threshold plus metagenomic subsampling. (SI Table S12.)

| Subfolder | Training steps |
|---|---|
| `red-subsampling/MILA_RED_0.03_0.88` | 100,000 |
| `red-subsampling/MILA_RED_0.05_0.88` | 100,000 |
| `red-subsampling/MILA_RED_0.10_0.88` | 100,000 |
| `red-subsampling/MILA_RED_0.20_0.88` | 100,000 |
| `red-subsampling/MILA_RED_0.30_0.88` | 100,000 |
| `red-subsampling/MILA_RED_0.40_0.88` | 100,000 |

### `clustering/`

UniRef100 clustered at a sweep of identity thresholds, at a fixed 100k-step budget. (SI Table S13.)

| Subfolder | Training steps |
|---|---|
| `clustering/MILA_U100_clust30` | 100,000 |
| `clustering/MILA_U100_clust40` | 100,000 |
| `clustering/MILA_U100_clust50` | 100,000 |
| `clustering/MILA_U100_clust60` | 100,000 |
| `clustering/MILA_U100_clust70` | 100,000 |
| `clustering/MILA_U100_clust80` | 100,000 |
| `clustering/MILA_U100_clust90` | 100,000 |

### `clustering-fixed-epochs/`

The same clustering sweep trained for a fixed 2 epochs instead of a fixed step budget. (SI Table S13.)

Control runs for the clustering sweep, trained for 2 epochs over the clustered set rather than to a fixed step budget, so the step count varies with dataset size. The clustering result reported in the paper (Figure 2E-F) uses the `clustering/` models above.

| Subfolder | Training steps |
|---|---|
| `clustering-fixed-epochs/MILA_U100_clust30_fixed_epochs` | 24,298 |
| `clustering-fixed-epochs/MILA_U100_clust40_fixed_epochs` | 26,534 |
| `clustering-fixed-epochs/MILA_U100_clust50_fixed_epochs` | 32,160 |
| `clustering-fixed-epochs/MILA_U100_clust60_fixed_epochs` | 39,796 |
| `clustering-fixed-epochs/MILA_U100_clust70_fixed_epochs` | 51,284 |
| `clustering-fixed-epochs/MILA_U100_clust80_fixed_epochs` | 68,926 |
| `clustering-fixed-epochs/MILA_U100_clust90_fixed_epochs` | 97,340 |

### `unfiltered/`

Baselines on unfiltered dataset mixtures. (SI Table S6.)

| Subfolder | Training steps |
|---|---|
| `unfiltered/MILA_BFD` | 100,000 |
| `unfiltered/MILA_BFD_no_ambig` | 100,000 |
| `unfiltered/MILA_MGnify` | 100,000 |
| `unfiltered/MILA_MGnify_no_ambig` | 100,000 |
| `unfiltered/MILA_UR100_BFD_MGnify_no_ambig` | 100,000 |
| `unfiltered/MILA_UR100_BFD_no_ambig` | 100,000 |
| `unfiltered/MILA_UR100_MGnify_no_ambig` | 100,000 |

## Citation

```bibtex
@article{lebreton_plm_metagenomic,
  title  = {pLM representations unlock metagenomic space beyond homology},
  author = {Le Breton, Lola and Heurtel-Depeiges, David and Millar, Douglas C. and
            Zetzsche, Lara E. and Vernon, Robert M. and Langmead, Christopher James and
            Chandar, Sarath and Fournier, Quentin}
}
```