--- license: cc-by-4.0 pipeline_tag: other tags: - genomics - transcription-factor-binding - chip-seq - sequence-to-function - encode --- # Cherimoya Binding ATlas (BATv1) This is the first release of the **Cherimoya Binding ATlas (BATv1)**: a collection of 17,145 Cherimoya models trained on ChIP-seq experiments from the ENCODE Project that measure the binding of transcription factors and other DNA-associated proteins. Each model maps a DNA sequence, together with the experiment's control signal, to a base-resolution, strand-specific ChIP-seq profile and its total read count. This release covers **3,429 ENCODE ChIP-seq experiments** of 1,139 targets in 183 biosamples. Every experiment is trained and evaluated across **5 folds** (`fold_0`–`fold_4`) of a chromosome-held-out cross-validation, with performance reported on both the validation and the test chromosomes of each fold. Each model predicts binding for a **single experiment**, so download only the model(s) you need. See https://github.com/jmschrei/cherimoya for more details on the Cherimoya model. ## Model description Each Cherimoya model is a compact residual convolutional network (45,586 parameters; 9 layers, 32 filters) in the BPNet family. It takes two inputs over `in_window=2114` bp: - a one-hot DNA sequence, shape `(N, 4, 2114)`, channels in ACGT order; - the experiment's **control** signal, shape `(N, 2, 2114)`: per-base read counts of the control bigWigs on the plus (channel 0) and minus (channel 1) strands over the same window. It produces two outputs over the central `out_window=1000` bp: - a **profile** head, logits of shape `(N, 2, 1000)` for the plus and minus strands. The predicted profile is one softmax taken jointly over both strands and all 1,000 positions, so the two strands share one distribution; - a **count** head, shape `(N, 1)`: the predicted natural log of (total reads + 1) over both strands in the 1,000 bp window. Each model was trained for at least 20 epochs and, when its training set was small, for more epochs to reach about 20,000 optimizer steps (at least 10,000 for the smallest training sets, which drop a partial batch every epoch). The published checkpoints are the exponential-moving-average (EMA) shadow weights at the epoch with the best validation count Pearson correlation. ## Training data Models are trained on the human genome (**GRCh38 / hg38**) against each experiment's observed ENCODE ChIP-seq signal (the plus- and minus-strand bigWigs in `signal_plus_file_accession` and `signal_minus_file_accession`), with the experiment's control bigWigs (`control_plus_file_accession`, `control_minus_file_accession`) as the second input. Training loci are the experiment's peak regions plus GC-matched background (non-peak) regions, taken from ENCODE's training-regions file for that experiment (`training_regions_file_accession`, a `.tar.gz` of per-fold BED files), with the ENCODE exclusion list ENCFF940NTE removed. The five folds partition the genome by chromosome and are the same for every experiment; each fold trains on the chromosomes not listed here: | fold | validation chromosomes | test chromosomes | |---|---|---| | `fold_0` | chr8, chr20 | chr1, chr3, chr6 | | `fold_1` | chr12, chr17 | chr2, chr8, chr9, chr16 | | `fold_2` | chr22, chr7 | chr4, chr11, chr12, chr15, chrY | | `fold_3` | chr6, chr21 | chr5, chr10, chr14, chr18, chr20, chr22 | | `fold_4` | chr10, chr18 | chr7, chr13, chr17, chr19, chr21, chrX | ## Repository layout Checkpoints live under `models/`, one directory per experiment: | file | description | |---|---| | `models//cherimoya.fold_.torch` | checkpoint for fold `i` (EMA shadow weights) | Repository-level files: | file | description | |---|---| | `performance.validation.tsv` | validation-chromosome metrics for every experiment and fold | | `performance.test.tsv` | test-chromosome metrics for every experiment and fold | | `BATv1-metadata.tsv` | experiment metadata, keyed by `experiment_accession` | | `manifest.csv` | the full list of experiments and folds included, with each checkpoint's md5 | | `fit.json` / `evaluate.validation.json` / `evaluate.test.json` | a representative example of the per-fold training and evaluation settings (fold 0 of ENCSR353DFU) | ### Performance tables `performance.validation.tsv` and `performance.test.tsv` have one row per experiment and fold. The first four columns are `experiment_accession`, `fold`, `n_peaks` and `checkpoint_epoch` (the training epoch, counted from 0, of the published checkpoint), followed by these metrics: | column | measured over | description | |---|---|---| | `profile_mnll`, `profile_jsd`, `profile_pearson`, `profile_spearman` | peaks | agreement between the observed and predicted base-resolution profiles | | `count_pearson`, `count_spearman`, `count_mse` | peaks | agreement between the observed and predicted log counts | | `all_count_pearson`, `all_count_spearman`, `all_count_mse` | peaks and background regions | the same count metrics over both kinds of region | | `auroc`, `auprc` | peaks and background regions | separation of peaks from background regions by the predicted log count | The headline metrics are `profile_pearson` (profile shape) and `count_pearson` (binding strength across peaks). `n_peaks` is the number of distinct peak regions in that fold's validation or test peak file, before exclusion-list filtering. Some experiments have few peaks, so their metrics are noisy: 142 of the 17,145 folds have fewer than 10 validation peaks and 31 have fewer than 10 test peaks. A low score on a fold with few test peaks therefore does not by itself show that the model is poor. The 506 folds with fewer than 100 test peaks (3.0% of all folds) hold 44 of the 90 folds whose test `profile_pearson` or `count_pearson` is at or below 0, and in experiments with such a fold the standard deviation of test `count_pearson` across folds has a median of 0.126, against 0.023 in the others. Because these experiments also train on few peaks, some of their models are weaker as well; compare an experiment's five folds, and its validation and test scores, before judging it. ### Metadata `BATv1-metadata.tsv` has one row per experiment. Its first ten columns are the same as `CATv1-metadata.tsv` in [CATv1](https://huggingface.co/programmable-genomics/CATv1), except that `experiment_peturbed` is spelled `experiment_perturbed`. The remaining columns are specific to ChIP-seq: | column | description | |---|---| | `target_label` | the ChIP-seq target, as ENCODE names it | | `target_investigated_as` | ENCODE's classes for the target, `;`-separated (experiments per class: transcription factor 2,589, chromatin remodeler 271, RNA polymerase complex 200, cofactor 171, RNA binding protein 131, DNA repair 49, cohesin 44, DNA replication 32, other context 9) | | `experiment_status` | the experiment's status on the ENCODE portal | | `signal_plus_file_accession`, `signal_minus_file_accession` | the observed signal bigWigs the models were trained on | | `control_plus_file_accession`, `control_minus_file_accession` | the control bigWigs the models take as input | | `control_plus_md5`, `control_minus_md5` | md5 of the control files the models were trained with | | `*_portal_status` | the status of the annotation or file on the ENCODE portal when this release was built; `not_public` means the portal does not serve it | | `controls_on_encode` | `TRUE` when the ENCODE portal serves both control files | | `training_regions_file_accession` | ENCODE's `.tar.gz` of per-fold peak and background BED files | | `n_peaks` | distinct peak regions across all folds | | `training_batch_size` | the batch size the models were trained with | | `best_epochs` | the training epoch (counted from 0) of each fold's published checkpoint, folds 0-4, `;`-separated; 0 or 1 means the checkpoint is from the start of training | | `quality_flags` | `;`-separated reasons to use an experiment's models with care, `none` otherwise: `small_training_set` (trained with a batch size below 64), `few_test_peaks` (a fold with fewer than 100 test peaks), `early_checkpoint` (a fold whose checkpoint is from epoch 0 or 1), `nonpositive_test_pearson` (a fold with test `profile_pearson` or `count_pearson` at or below 0) | Each `experiment_accession` is the source ENCODE experiment (`https://www.encodeproject.org/experiments//`), and the paired `annotation_accession` points to the ENCODE BPNet model annotation for that same experiment (`https://www.encodeproject.org/annotations//`); 819 of these annotations are not yet public (`annotation_portal_status`). A file whose `*_portal_status` is `released` or `archived` downloads from `https://www.encodeproject.org/files//@@download/.bigWig` (`.tar.gz` for the training-regions file). ## Which model should I use? Pick the experiment that matches your target and biosample using `target_label` and `experiment_biosample_term_name` in `BATv1-metadata.tsv`, look at `quality_flags` and `n_peaks` (see the note on experiments with few peaks below), and download that experiment's checkpoint(s). To predict the experiment's observed signal, also download its two control bigWigs from ENCODE (`controls_on_encode` is `TRUE` when the portal serves both); for predictions of the sequence's contribution alone, use an all-zero control instead (see [Predictions without a control](#predictions-without-a-control)). The five folds are cross-validation replicates of the same experiment: use a single fold (e.g. `fold_0`), or average the predictions of all five folds for a more robust estimate. A model only reflects the experiment it was trained on, so do not use it for a different target or cell type. ## Intended uses and limitations Beyond profile and count prediction, the models support downstream sequence analyses such as attribution, motif marginalization, and in-silico mutagenesis. They are **human (GRCh38) only** and are specific to the experiment they were trained on. Also note: - **Controls.** The models were trained with the experiment's two control tracks as input, so predicting the observed signal needs them. For 211 experiments, at least one control file is not served by the ENCODE portal at the time of this release (`controls_on_encode` is `FALSE`); those models can still make predictions with an all-zero control (see [Predictions without a control](#predictions-without-a-control)). - **Training regions and signal not yet public for some experiments.** For 819 experiments, the ENCODE portal does not serve the training-regions file, the observed signal bigWigs or the BPNet annotation at the time of this release (`*_portal_status` is `not_public`). The models can still be run, but their training and evaluation regions and targets cannot yet be downloaded. - **Two training-regions files are cut short.** In ENCODE's training-regions files for ENCSR142SQX (ENCFF899EKQ) and ENCSR280SCF (ENCFF283MYH), one fold-2 BED file (test peaks and training background regions, respectively) is filled with NUL bytes after its first rows. The BATv1 models and metrics for those folds use the complete files, rebuilt from the experiment's other folds, which cover the same regions. - **Revoked experiments.** 5 experiments (all ZNF143) have `experiment_status` `revoked` on the ENCODE portal. Their models are included and flagged. - **Experiments with few peaks.** Performance is lower and varies more between folds in experiments with few peaks. In the 471 experiments with fewer than 2,500 peaks (`n_peaks`), the median test `profile_pearson` (mean over folds) is 0.097, against 0.211 in the other 2,958, and its standard deviation across folds has a median of 0.026, against 0.005. Scored on the same test regions, ENCODE's BPNet model for the same experiment (`annotation_accession`) predicted profile shape better than BATv1 in 75% of the experiments with fewer than 2,500 peaks and in 11% of the others, so for small experiments it may be the better choice for profile shape. Checkpoints were chosen on validation `count_pearson` alone; in 160 folds of 137 experiments (80 of them with fewer than 2,500 peaks) this kept a checkpoint from the first or second epoch (`checkpoint_epoch` 0 or 1 in the performance tables, `best_epochs` in the metadata), whose profile predictions are close to uninformative (median test `profile_pearson` 0.006). `quality_flags` marks 268 experiments that need this kind of care. - **Small experiments.** 8 experiments have too few training peaks in some folds for a batch of 64, so they were trained with a batch size of 32 or 16 (`training_batch_size`). Their validation metrics rest on 1–37 peaks per fold. - **Control files with a malformed zoom index.** The released ENCODE control files ENCFF280PMX, ENCFF451MKO, ENCFF578CCB, ENCFF809CPZ (used by 6 experiments) have one malformed entry in the index of their 40 bp zoom level. The base-resolution data, which is what the models read, is intact, but readers that load zoom levels (for example pybigtools) can fail on these files. ## Installation ```bash pip install cherimoya # PyPI (Python 3.10+) ``` Install-from-source, Docker images, and GPU/Triton notes: https://github.com/jmschrei/cherimoya ## Usage ```python import torch import pandas from huggingface_hub import hf_hub_download from tangermeme.io import extract_loci from tangermeme.predict import predict from cherimoya import Cherimoya eid = "ENCSR353DFU" meta = pandas.read_csv(hf_hub_download("programmable-genomics/BATv1", "BATv1-metadata.tsv"), sep="\t") row = meta.set_index("experiment_accession").loc[eid] path = hf_hub_download("programmable-genomics/BATv1", "models/{}/cherimoya.fold_0.torch".format(eid)) model = Cherimoya.load(path, device="cuda", compile=False).eval() # The experiment's control bigWigs, plus strand first, downloaded from # https://www.encodeproject.org/files//@@download/.bigWig controls = [row.control_plus_file_accession + ".bigWig", row.control_minus_file_accession + ".bigWig"] # Sequence (N, 4, 2114) and control signal (N, 2, 2114), centered on each # region of a BED file. X, X_ctl = extract_loci("regions.bed", "hg38.fa", in_signals=controls, in_window=2114) profile_logits, log_counts = predict(model, X, args=(X_ctl,), device="cuda", batch_size=64) # One softmax over both strands and all positions: (N, 2, 1000). profile = torch.softmax(profile_logits.float().reshape(len(X), -1), dim=-1).reshape(-1, 2, 1000) ``` ### Predictions without a control An all-zero control track gives clean predictions of what the sequence alone contributes, without the biases carried by the experiment's control signal, and it lets you run the models whose control files are not available. `ControlWrapper` supplies the zero control, so the wrapped model takes the sequence alone, as attribution and design tools expect. Continuing from the example above: ```python from cherimoya import ControlWrapper model = ControlWrapper(model) X = extract_loci("regions.bed", "hg38.fa", in_window=2114) profile_logits, log_counts = predict(model, X, device="cuda", batch_size=64) ``` ## Contact Jacob Schreiber — Programmable Genomics Laboratory @ UMass Chan.