README.md

---
license: apache-2.0
tags:
  - protein
  - biology
  - protein-language-model
  - continual-learning
library_name: transformers
---

# CoPeP Continual Learning Checkpoints

This repository contains **90 checkpoints** from continual
learning experiments with the [AMPLIFY](https://huggingface.co/chandar-lab/AMPLIFY_120M)
protein language model (120M parameters).

## Loading a checkpoint

```python
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "chandar-lab/copep-checkpoints",
    subfolder="replay/task_5",
    trust_remote_code=True,
)
```

## Available checkpoints

| Method | Tasks |
|--------|-------|
| `continual` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `gradient_ascent` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `hare_tortoise` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `joint` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `match` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `random_labels` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `replay` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `shrink_perturb` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |
| `single_year` | task_0, task_1, task_2, task_3, task_4, task_5, task_6, task_7, task_8, task_9 |


Each `task_N` subfolder contains a `config.json` and `model.safetensors`.

### Task mapping

- **task_0** : pre-2004 (base model)
- **task_1** – **task_9** : successive temporal splits of UniRef data

For methods that start from task_1 (continual, gradient_ascent, match,
random_labels, replay, shrink_perturb), `task_0` is the same checkpoint as
`single_year/task_0` (the base pre-trained model).

## Model architecture

- **Architecture:** Transformer encoder with RoPE + SwiGLU
- **Parameters:** ~120M
- **Config:** hidden_size=640, num_hidden_layers=24, num_attention_heads=10, intermediate_size=2560
- **Vocab size:** 32 (amino acid tokens + special tokens)
- **Max length:** 512 (training), 50000 (inference with RoPE extrapolation)