README.md

metadata

license: cc-by-4.0
pipeline_tag: other
tags:
  - genomics
  - transcription-factor-binding
  - chip-seq
  - sequence-to-function
  - encode

Cherimoya Binding ATlas (BATv1)

This is the first release of the Cherimoya Binding ATlas (BATv1): a collection of 17,145 Cherimoya models trained on ChIP-seq experiments from the ENCODE Project that measure the binding of transcription factors and other DNA-associated proteins. Each model maps a DNA sequence, together with the experiment's control signal, to a base-resolution, strand-specific ChIP-seq profile and its total read count.

This release covers 3,429 ENCODE ChIP-seq experiments of 1,139 targets in 183 biosamples. Every experiment is trained and evaluated across 5 folds (fold_0–fold_4) of a chromosome-held-out cross-validation, with performance reported on both the validation and the test chromosomes of each fold. Each model predicts binding for a single experiment, so download only the model(s) you need.

See https://github.com/jmschrei/cherimoya for more details on the Cherimoya model.

Model description

Each Cherimoya model is a compact residual convolutional network (45,586 parameters; 9 layers, 32 filters) in the BPNet family. It takes two inputs over in_window=2114 bp:

• a one-hot DNA sequence, shape (N, 4, 2114), channels in ACGT order;
• the experiment's control signal, shape (N, 2, 2114): per-base read counts of the control bigWigs on the plus (channel 0) and minus (channel 1) strands over the same window.

It produces two outputs over the central out_window=1000 bp:

• a profile head, logits of shape (N, 2, 1000) for the plus and minus strands. The predicted profile is one softmax taken jointly over both strands and all 1,000 positions, so the two strands share one distribution;
• a count head, shape (N, 1): the predicted natural log of (total reads + 1) over both strands in the 1,000 bp window.

Each model was trained for at least 20 epochs and, when its training set was small, for more epochs to reach about 20,000 optimizer steps (at least 10,000 for the smallest training sets, which drop a partial batch every epoch). The published checkpoints are the exponential-moving-average (EMA) shadow weights at the epoch with the best validation count Pearson correlation.

Training data

Models are trained on the human genome (GRCh38 / hg38) against each experiment's observed ENCODE ChIP-seq signal (the plus- and minus-strand bigWigs in signal_plus_file_accession and signal_minus_file_accession), with the experiment's control bigWigs (control_plus_file_accession, control_minus_file_accession) as the second input. Training loci are the experiment's peak regions plus GC-matched background (non-peak) regions, taken from ENCODE's training-regions file for that experiment (training_regions_file_accession, a .tar.gz of per-fold BED files), with the ENCODE exclusion list ENCFF940NTE removed. The five folds partition the genome by chromosome and are the same for every experiment; each fold trains on the chromosomes not listed here:

fold	validation chromosomes	test chromosomes
fold_0	chr8, chr20	chr1, chr3, chr6
fold_1	chr12, chr17	chr2, chr8, chr9, chr16
fold_2	chr22, chr7	chr4, chr11, chr12, chr15, chrY
fold_3	chr6, chr21	chr5, chr10, chr14, chr18, chr20, chr22
fold_4	chr10, chr18	chr7, chr13, chr17, chr19, chr21, chrX

Repository layout

Checkpoints live under models/, one directory per experiment:

file	description
models/<encid>/cherimoya.fold_<i>.torch	checkpoint for fold i (EMA shadow weights)

Repository-level files:

file	description
performance.validation.tsv	validation-chromosome metrics for every experiment and fold
performance.test.tsv	test-chromosome metrics for every experiment and fold
BATv1-metadata.tsv	experiment metadata, keyed by experiment_accession
manifest.csv	the full list of experiments and folds included, with each checkpoint's md5
fit.json / evaluate.validation.json / evaluate.test.json	a representative example of the per-fold training and evaluation settings (fold 0 of ENCSR353DFU)

Performance tables

performance.validation.tsv and performance.test.tsv have one row per experiment and fold. The first four columns are experiment_accession, fold, n_peaks and checkpoint_epoch (the training epoch, counted from 0, of the published checkpoint), followed by these metrics:

column	measured over	description
profile_mnll, profile_jsd, profile_pearson, profile_spearman	peaks	agreement between the observed and predicted base-resolution profiles
count_pearson, count_spearman, count_mse	peaks	agreement between the observed and predicted log counts
all_count_pearson, all_count_spearman, all_count_mse	peaks and background regions	the same count metrics over both kinds of region
auroc, auprc	peaks and background regions	separation of peaks from background regions by the predicted log count

The headline metrics are profile_pearson (profile shape) and count_pearson (binding strength across peaks). n_peaks is the number of distinct peak regions in that fold's validation or test peak file, before exclusion-list filtering. Some experiments have few peaks, so their metrics are noisy: 142 of the 17,145 folds have fewer than 10 validation peaks and 31 have fewer than 10 test peaks. A low score on a fold with few test peaks therefore does not by itself show that the model is poor. The 506 folds with fewer than 100 test peaks (3.0% of all folds) hold 44 of the 90 folds whose test profile_pearson or count_pearson is at or below 0, and in experiments with such a fold the standard deviation of test count_pearson across folds has a median of 0.126, against 0.023 in the others. Because these experiments also train on few peaks, some of their models are weaker as well; compare an experiment's five folds, and its validation and test scores, before judging it.

Metadata

BATv1-metadata.tsv has one row per experiment. Its first ten columns are the same as CATv1-metadata.tsv in CATv1, except that experiment_peturbed is spelled experiment_perturbed. The remaining columns are specific to ChIP-seq:

column	description
target_label	the ChIP-seq target, as ENCODE names it
target_investigated_as	ENCODE's classes for the target, ;-separated (experiments per class: transcription factor 2,589, chromatin remodeler 271, RNA polymerase complex 200, cofactor 171, RNA binding protein 131, DNA repair 49, cohesin 44, DNA replication 32, other context 9)
experiment_status	the experiment's status on the ENCODE portal
signal_plus_file_accession, signal_minus_file_accession	the observed signal bigWigs the models were trained on
control_plus_file_accession, control_minus_file_accession	the control bigWigs the models take as input
control_plus_md5, control_minus_md5	md5 of the control files the models were trained with
*_portal_status	the status of the annotation or file on the ENCODE portal when this release was built; not_public means the portal does not serve it
controls_on_encode	TRUE when the ENCODE portal serves both control files
training_regions_file_accession	ENCODE's .tar.gz of per-fold peak and background BED files
n_peaks	distinct peak regions across all folds
training_batch_size	the batch size the models were trained with
best_epochs	the training epoch (counted from 0) of each fold's published checkpoint, folds 0-4, ;-separated; 0 or 1 means the checkpoint is from the start of training
quality_flags	;-separated reasons to use an experiment's models with care, none otherwise: small_training_set (trained with a batch size below 64), few_test_peaks (a fold with fewer than 100 test peaks), early_checkpoint (a fold whose checkpoint is from epoch 0 or 1), nonpositive_test_pearson (a fold with test profile_pearson or count_pearson at or below 0)

Each experiment_accession is the source ENCODE experiment (https://www.encodeproject.org/experiments/<experiment_accession>/), and the paired annotation_accession points to the ENCODE BPNet model annotation for that same experiment (https://www.encodeproject.org/annotations/<annotation_accession>/); 819 of these annotations are not yet public (annotation_portal_status). A file whose *_portal_status is released or archived downloads from https://www.encodeproject.org/files/<accession>/@@download/<accession>.bigWig (.tar.gz for the training-regions file).

Which model should I use?

Pick the experiment that matches your target and biosample using target_label and experiment_biosample_term_name in BATv1-metadata.tsv, look at quality_flags and n_peaks (see the note on experiments with few peaks below), and download that experiment's checkpoint(s). To predict the experiment's observed signal, also download its two control bigWigs from ENCODE (controls_on_encode is TRUE when the portal serves both); for predictions of the sequence's contribution alone, use an all-zero control instead (see Predictions without a control). The five folds are cross-validation replicates of the same experiment: use a single fold (e.g. fold_0), or average the predictions of all five folds for a more robust estimate. A model only reflects the experiment it was trained on, so do not use it for a different target or cell type.

Intended uses and limitations

Beyond profile and count prediction, the models support downstream sequence analyses such as attribution, motif marginalization, and in-silico mutagenesis. They are human (GRCh38) only and are specific to the experiment they were trained on. Also note:

• Controls. The models were trained with the experiment's two control tracks as input, so predicting the observed signal needs them. For 211 experiments, at least one control file is not served by the ENCODE portal at the time of this release (controls_on_encode is FALSE); those models can still make predictions with an all-zero control (see Predictions without a control).
• Training regions and signal not yet public for some experiments. For 819 experiments, the ENCODE portal does not serve the training-regions file, the observed signal bigWigs or the BPNet annotation at the time of this release (*_portal_status is not_public). The models can still be run, but their training and evaluation regions and targets cannot yet be downloaded.
• Two training-regions files are cut short. In ENCODE's training-regions files for ENCSR142SQX (ENCFF899EKQ) and ENCSR280SCF (ENCFF283MYH), one fold-2 BED file (test peaks and training background regions, respectively) is filled with NUL bytes after its first rows. The BATv1 models and metrics for those folds use the complete files, rebuilt from the experiment's other folds, which cover the same regions.
• Revoked experiments. 5 experiments (all ZNF143) have experiment_status revoked on the ENCODE portal. Their models are included and flagged.
• Experiments with few peaks. Performance is lower and varies more between folds in experiments with few peaks. In the 471 experiments with fewer than 2,500 peaks (n_peaks), the median test profile_pearson (mean over folds) is 0.097, against 0.211 in the other 2,958, and its standard deviation across folds has a median of 0.026, against 0.005. Scored on the same test regions, ENCODE's BPNet model for the same experiment (annotation_accession) predicted profile shape better than BATv1 in 75% of the experiments with fewer than 2,500 peaks and in 11% of the others, so for small experiments it may be the better choice for profile shape. Checkpoints were chosen on validation count_pearson alone; in 160 folds of 137 experiments (80 of them with fewer than 2,500 peaks) this kept a checkpoint from the first or second epoch (checkpoint_epoch 0 or 1 in the performance tables, best_epochs in the metadata), whose profile predictions are close to uninformative (median test profile_pearson 0.006). quality_flags marks 268 experiments that need this kind of care.
• Small experiments. 8 experiments have too few training peaks in some folds for a batch of 64, so they were trained with a batch size of 32 or 16 (training_batch_size). Their validation metrics rest on 1–37 peaks per fold.
• Control files with a malformed zoom index. The released ENCODE control files ENCFF280PMX, ENCFF451MKO, ENCFF578CCB, ENCFF809CPZ (used by 6 experiments) have one malformed entry in the index of their 40 bp zoom level. The base-resolution data, which is what the models read, is intact, but readers that load zoom levels (for example pybigtools) can fail on these files.

Installation

pip install cherimoya    # PyPI (Python 3.10+)

Install-from-source, Docker images, and GPU/Triton notes: https://github.com/jmschrei/cherimoya

Usage

import torch
import pandas
from huggingface_hub import hf_hub_download
from tangermeme.io import extract_loci
from tangermeme.predict import predict
from cherimoya import Cherimoya

eid = "ENCSR353DFU"
meta = pandas.read_csv(hf_hub_download("programmable-genomics/BATv1", "BATv1-metadata.tsv"), sep="\t")
row = meta.set_index("experiment_accession").loc[eid]

path = hf_hub_download("programmable-genomics/BATv1", "models/{}/cherimoya.fold_0.torch".format(eid))
model = Cherimoya.load(path, device="cuda", compile=False).eval()

# The experiment's control bigWigs, plus strand first, downloaded from
# https://www.encodeproject.org/files/<accession>/@@download/<accession>.bigWig
controls = [row.control_plus_file_accession + ".bigWig",
    row.control_minus_file_accession + ".bigWig"]

# Sequence (N, 4, 2114) and control signal (N, 2, 2114), centered on each
# region of a BED file.
X, X_ctl = extract_loci("regions.bed", "hg38.fa", in_signals=controls, in_window=2114)

profile_logits, log_counts = predict(model, X, args=(X_ctl,), device="cuda", batch_size=64)

# One softmax over both strands and all positions: (N, 2, 1000).
profile = torch.softmax(profile_logits.float().reshape(len(X), -1), dim=-1).reshape(-1, 2, 1000)

Predictions without a control

An all-zero control track gives clean predictions of what the sequence alone contributes, without the biases carried by the experiment's control signal, and it lets you run the models whose control files are not available. ControlWrapper supplies the zero control, so the wrapped model takes the sequence alone, as attribution and design tools expect. Continuing from the example above:

from cherimoya import ControlWrapper

model = ControlWrapper(model)

X = extract_loci("regions.bed", "hg38.fa", in_window=2114)
profile_logits, log_counts = predict(model, X, device="cuda", batch_size=64)

Contact

Jacob Schreiber jacob.schreiber@umassmed.edu — Programmable Genomics Laboratory @ UMass Chan.