Update README, metadata and performance tables

README.md

@@ -29,7 +29,7 @@ It produces two outputs over the central `out_window=1000` bp:
 - a **profile** head, logits of shape `(N, 2, 1000)` for the plus and minus strands. The predicted profile is one softmax taken jointly over both strands and all 1,000 positions, so the two strands share one distribution;
 - a **count** head, shape `(N, 1)`: the predicted natural log of (total reads + 1) over both strands in the 1,000 bp window.
 
-The published checkpoints are the exponential-moving-average (EMA) shadow weights at the point in training with the best validation count Pearson correlation.
+Each model was trained for at least 20 epochs and, when its training set was small, for more epochs to reach about 20,000 optimizer steps (at least 10,000 for the smallest training sets, which drop a partial batch every epoch). The published checkpoints are the exponential-moving-average (EMA) shadow weights at the epoch with the best validation count Pearson correlation.
 
 ## Training data
 
@@ -91,6 +91,7 @@ The headline metrics are `profile_pearson` (profile shape) and `count_pearson` (
 | `training_regions_file_accession` | ENCODE's `.tar.gz` of per-fold peak and background BED files |
 | `n_peaks` | distinct peak regions across all folds |
 | `training_batch_size` | the batch size the models were trained with |
+| `best_epochs` | the training epoch (counted from 0) of each fold's published checkpoint, folds 0-4, `;`-separated; 0 or 1 means the checkpoint is from the start of training |
 | `quality_flags` | `;`-separated reasons to use an experiment's models with care, `none` otherwise: `small_training_set` (trained with a batch size below 64), `few_test_peaks` (a fold with fewer than 100 test peaks), `early_checkpoint` (a fold whose checkpoint is from epoch 0 or 1), `nonpositive_test_pearson` (a fold with test `profile_pearson` or `count_pearson` at or below 0) |
 
 Each `experiment_accession` is the source ENCODE experiment (`https://www.encodeproject.org/experiments/<experiment_accession>/`), and the paired `annotation_accession` points to the ENCODE BPNet model annotation for that same experiment (`https://www.encodeproject.org/annotations/<annotation_accession>/`); 819 of these annotations are not yet public (`annotation_portal_status`). A file whose `*_portal_status` is `released` or `archived` downloads from `https://www.encodeproject.org/files/<accession>/@@download/<accession>.bigWig` (`.tar.gz` for the training-regions file).
@@ -107,7 +108,7 @@ Beyond profile and count prediction, the models support downstream sequence anal
 - **Training regions and signal not yet public for some experiments.** For 819 experiments, the ENCODE portal does not serve the training-regions file, the observed signal bigWigs or the BPNet annotation at the time of this release (`*_portal_status` is `not_public`). The models can still be run, but their training and evaluation regions and targets cannot yet be downloaded.
 - **Two training-regions files are cut short.** In ENCODE's training-regions files for ENCSR142SQX (ENCFF899EKQ) and ENCSR280SCF (ENCFF283MYH), one fold-2 BED file (test peaks and training background regions, respectively) is filled with NUL bytes after its first rows. The BATv1 models and metrics for those folds use the complete files, rebuilt from the experiment's other folds, which cover the same regions.
 - **Revoked experiments.** 5 experiments (all ZNF143) have `experiment_status` `revoked` on the ENCODE portal. Their models are included and flagged.
-- **Experiments with few peaks.** Performance is lower and varies more between folds in experiments with few peaks. In the 471 experiments with fewer than 2,500 peaks (`n_peaks`), the median test `profile_pearson` (mean over folds) is 0.089, against 0.211 in the other 2,958, and its standard deviation across folds has a median of 0.028, against 0.005. Scored on the same test regions, ENCODE's BPNet model for the same experiment (`annotation_accession`) predicted profile shape better than BATv1 in 74% of the experiments with fewer than 2,500 peaks and in 11% of the others, so for small experiments it may be the better choice for profile shape. Checkpoints were chosen on validation `count_pearson` alone; in 238 folds of 193 experiments (132 of them with fewer than 2,500 peaks) this kept a checkpoint from the first or second epoch (`checkpoint_epoch` 0 or 1), whose profile predictions are close to uninformative (median test `profile_pearson` 0.004). `quality_flags` marks 296 experiments that need this kind of care.
+- **Experiments with few peaks.** Performance is lower and varies more between folds in experiments with few peaks. In the 471 experiments with fewer than 2,500 peaks (`n_peaks`), the median test `profile_pearson` (mean over folds) is 0.097, against 0.211 in the other 2,958, and its standard deviation across folds has a median of 0.026, against 0.005. Scored on the same test regions, ENCODE's BPNet model for the same experiment (`annotation_accession`) predicted profile shape better than BATv1 in 75% of the experiments with fewer than 2,500 peaks and in 11% of the others, so for small experiments it may be the better choice for profile shape. Checkpoints were chosen on validation `count_pearson` alone; in 160 folds of 137 experiments (80 of them with fewer than 2,500 peaks) this kept a checkpoint from the first or second epoch (`checkpoint_epoch` 0 or 1 in the performance tables, `best_epochs` in the metadata), whose profile predictions are close to uninformative (median test `profile_pearson` 0.006). `quality_flags` marks 268 experiments that need this kind of care.
 - **Small experiments.** 8 experiments have too few training peaks in some folds for a batch of 64, so they were trained with a batch size of 32 or 16 (`training_batch_size`). Their validation metrics rest on 1–37 peaks per fold.
 - **Control files with a malformed zoom index.** The released ENCODE control files ENCFF280PMX, ENCFF451MKO, ENCFF578CCB, ENCFF809CPZ (used by 6 experiments) have one malformed entry in the index of their 40 bp zoom level. The base-resolution data, which is what the models read, is intact, but readers that load zoom levels (for example pybigtools) can fail on these files.