README.md

---
license: cc-by-4.0
task_categories:
  - text-generation
tags:
  - biology
  - proteins
  - genomics
  - uniprot
  - ensembl
  - pretraining
size_categories:
  - 10M<n<100M
pretty_name: biocorpus (sequence-first biology pretraining corpus)
---

# biocorpus — a sequence-first biology pretraining corpus

**21.1 M documents · ~9.15 B tokens** (Llama-3 / `marin-community/marin-tokenizer`).
Every document places a biological **sequence first**, then its **annotation** — so an
autoregressive model learns to read structure/function *from* sequence (recognition),
and, for the curated slice emitted in both directions, to generate sequence *from* a
specification (design).

This is **not** a scrape of papers or abstracts. Each record is a real UniProtKB or
Ensembl entry, rendered into plain text with **only substantive, learnable annotation**
kept — no markup tags, no bookkeeping, no uncharacterized filler.

## Composition

| source | documents | tokens | what it is |
|---|--:|--:|---|
| **TrEMBL representatives** | 20,091,178 | 8.49 B | one annotated UniProtKB entry per **UniRef50** cluster — the deduplicated protein universe |
| **Swiss-Prot** (both orderings) | 950,174 | 0.58 B | the full reviewed set, emitted sequence→annotation **and** annotation→sequence |
| **Central dogma + splice** (human) | 96,768 | 0.07 B | verified DNA→RNA→protein transcripts; 5′/3′ splice-site windows |
| **total** | **21,138,120** | **9.15 B** | |

The protein backbone is deduplicated at the **UniRef50** level (50% identity): we keep the
UniProtKB entry of each cluster representative, so the corpus spans the whole protein
universe once, without the ~10× redundancy of raw TrEMBL. Cross-file exact-sequence dedup
removes any residual duplicates (0 found — representatives are unique by construction).

## Quality gate

A protein is included **only** if it carries genuinely learnable annotation — one of:

- **tier A (biological):** function, catalytic activity, subcellular localization, pathway, disease, or GO terms; **or**
- **tier B (structural):** a domain boundary, active/binding site, signal peptide, transmembrane span, PTM, disulfide, repeat, …

Entries whose *only* annotation is a disordered/coiled-coil region — the bulk of
"Uncharacterized protein" TrEMBL entries — are dropped. Genomic records are intrinsically
annotated (exon structure, UTRs, splice motifs) and always kept.

Annotation coverage of the kept set:

| field | TrEMBL reps | Swiss-Prot |
|---|--:|--:|
| per-residue features (domains/sites/PTMs/…) | 92.4% | 84.5% |
| GO terms | 64.4% | 98.7% |
| subcellular location | 21.7% | 65.9% |
| function | 5.9% | 85.2% |
| catalytic activity | 6.9% | 46.1% |

## Document format

```
>tr:Q977Q6 Methionine aminopeptidase [uncultured crenarchaeote 4B7]
MTFDNYIKAGKIAGEIRENVRKTDWVGKTVYEICEYVENEIKKRGAKCAFPVNTSINEVAAHYTAEPNDEIT…

Methionine aminopeptidase — UniProtKB/TrEMBL Q977Q6 — is a 225-residue protein from
uncultured crenarchaeote 4B7 (NCBI taxon 44557).
Catalytic activity: Reaction=Release of N-terminal amino acids, preferentially methionine…; EC=3.4.11.18
GO annotations: cytoplasm (component); initiator methionyl aminopeptidase activity (function);
  metal ion binding (function); metalloexopeptidase activity (function); proteolysis (process)
Sequence features: 1 domain [6-194 (Peptidase M24)]
Keywords: Aminopeptidase; Hydrolase; Metal-binding; Protease
Lineage: Archaea > Nitrososphaerota > Nitrososphaeria > Nitrosopumilales > environmental samples
```

Central-dogma records show one transcript as genomic pre-mRNA (exons uppercase, introns
lowercase) → spliced mRNA (with 5′UTR/CDS/3′UTR boundaries) → translated protein, and are
**verified**: the mRNA equals the spliced exons and `translate(CDS)` equals the protein.

Each line of the JSONL is one document with fields: `id`, `source`, `accession`,
`entity_type`, `seq_type`, `seq_len`, `organism`, `taxid`, `gene`, `name`, `annotations`,
`sequence`, `ordering`, and the rendered `text`. Shards are shuffled.

## Sources, licensing, provenance

- **UniProtKB** (Swiss-Prot + TrEMBL) and **UniRef50** — UniProt Consortium, CC-BY 4.0.
- **Ensembl** (human GRCh38, release 112) gene models and sequence — EMBL-EBI, no restriction.

Built by a local flat-file join (no per-record web requests): UniRef50 FASTA supplies the
representative set; each representative's annotation is rendered straight from the UniProtKB
flat file. Fully reproducible from the builder in the
[biocorpus](https://github.com/timodonnell/biocorpus) repo (`builders/bio_pretrain/`).