---
license: mit
tags:
  - transformer
  - methylation
  - epigenomics
  - pretraining
  - masked-regression
datasets:
  - custom
language: en
library_name: transformers
model_name: MethFormer
pipeline_tag: feature-extraction
---

# 🧚 MethFormer: A Transformer for DNA Methylation

**MethFormer** is a masked regression transformer model trained to learn local and long-range patterns in DNA methylation (5mC and 5hmC) across genomic regions. Pretrained on binned methylation data, it is designed for downstream fine-tuning on tasks such as predicting MLL binding or chromatin state.

---

## 🚀 Overview

* **Inputs**: Binned methylation values (5mC, 5hmC) over 1024bp windows (32 bins × 2 channels)
* **Pretraining objective**: Masked methylation imputation (per-bin regression)
* **Architecture**: Transformer encoder with linear projection head
* **Downstream tasks**: MLL binding prediction, chromatin state inference, or enhancer classification

---

## 📁 Project Structure

```
.
├── config/                       # config
├── data/                         # Binned methylation datasets (HuggingFace format)
├── output/                       # Pretrained models, logs, and checkpoints
├── scripts/                      
│   ├── methformer.py             # Model classes, data collator, 
│   ├── pretrain_methformer.py    # Main training script
│   └── finetune_mll.py           # (optional) downstream fine-tuning
├── requirements.txt
└── README.md
```

---

## 👩‍💻 Pretraining MethFormer

### Step 1: Prepare Dataset

Preprocess 5mC and 5hmC data into 1024bp windows, binned into 32 bins × 2 features. Save using Hugging Face's `datasets.DatasetDict` format:

```
DatasetDict({
  train: Dataset({
    features: ['input_values', 'attention_mask', 'labels']
  }),
  validation: Dataset(...)
})
```

### Step 2: Run Pretraining

```bash
python scripts/pretrain_methformer.py
```

Options can be customized inside the script or modified for sweep tuning. This will:

* Train the model using masked regression loss
* Evaluate on a held-out chromosome (e.g., `chr8`)
* Log metrics to [Weights & Biases](https://wandb.ai)
* Save the best model checkpoint

---

## 📊 Metrics

* `masked_mse`: Mean squared error over unmasked positions
* `masked_mae`: Mean absolute error

---

## 🧪 Fine-tuning on MLL Binding

After pretraining:

1. Replace the regression head with a scalar head for MLL prediction.
2. Use a `Trainer` to fine-tune on log1p-transformed MLL-N RPKM values mean over 1kb regions.

See `scripts/finetune_mll.py` for an example.

---

## 🔍 Visualizations & Interpretability

You can run [Captum](https://captum.ai) or SHAP for:

* Per-bin attribution of 5mC/5hmC to MLL binding
* Visualizing what MethFormer attends to during fine-tuning

---

## 🛠️ Dependencies

Key packages:

* `transformers`
* `datasets`
* `wandb`
* `torch`
* `anndata`
* `scikit-learn`

---

## 🧠 Acknowledgements

* Built with inspiration from DNABERT, Grover, and vision transformers