metadata
library_name: locus
tags:
- dna
- genomics
- locally-consistent-parsing
LoCUS
Pretrained LoCUS models: DNA language models that read DNA as segments from locally consistent parsing (LCP). Code, fine-tuning recipes and the scripts that reproduce every result of the paper: https://github.com/STORMgroup/locus
| Folder | Model |
|---|---|
locus-0.5m/ |
LoCUS-0.5M: width 64, 4 kb context |
locus-2m/ |
LoCUS-2M: width 128, 4 kb context |
twin-fixed-2m/ |
Tokenizer study: LoCUS-2M with fixed 10 bp segments every 5 bp |
twin-bpe-2m/ |
Tokenizer study: LoCUS-2M with BPE tokens (vocabulary 4,096) as segments |
locus-0.5m-no-chr20-21/ |
Pretraining-split ablation: LoCUS-0.5M pretrained without chromosomes 20 and 21 |
locus-2m-no-chr20-21/ |
Pretraining-split ablation: LoCUS-2M pretrained without chromosomes 20 and 21 |
twin-fixed-2m-no-chr20-21/ |
Pretraining-split ablation: twin-fixed-2m pretrained without chromosomes 20 and 21 |
twin-bpe-2m-no-chr20-21/ |
Pretraining-split ablation: twin-bpe-2m pretrained without chromosomes 20 and 21 (its BPE vocabulary trained on those chromosomes) |
Each folder has config.json (the pretraining configuration and step), model.safetensors (the weights) and a model card.
The first four are pretrained on GRCh38 chromosomes 3-22. The four -no-chr20-21 models are the same models pretrained on
chromosomes 1-19 and 22, without the chromosomes of the Nucleotide Transformer benchmark's test sets, everything else
unchanged; the paper uses them to check that seeing the test chromosomes in pretraining does not inflate the test scores.
import locus
model = locus.load("locus-2m") # downloads locus-2m/ from this repository on first use
vectors = model.embed(["ACGT" * 100])