File size: 1,790 Bytes
83cfd16 4296bdc 83cfd16 4296bdc 83cfd16 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | ---
library_name: locus
tags:
- dna
- genomics
- locally-consistent-parsing
---
# LoCUS
Pretrained LoCUS models: DNA language models that read DNA as segments from locally consistent parsing (LCP).
Code, fine-tuning recipes and the scripts that reproduce every result of the paper: https://github.com/STORMgroup/locus
| Folder | Model |
|---|---|
| `locus-0.5m/` | LoCUS-0.5M: width 64, 4 kb context |
| `locus-2m/` | LoCUS-2M: width 128, 4 kb context |
| `twin-fixed-2m/` | Tokenizer study: LoCUS-2M with fixed 10 bp segments every 5 bp |
| `twin-bpe-2m/` | Tokenizer study: LoCUS-2M with BPE tokens (vocabulary 4,096) as segments |
| `locus-0.5m-no-chr20-21/` | Pretraining-split ablation: LoCUS-0.5M pretrained without chromosomes 20 and 21 |
| `locus-2m-no-chr20-21/` | Pretraining-split ablation: LoCUS-2M pretrained without chromosomes 20 and 21 |
| `twin-fixed-2m-no-chr20-21/` | Pretraining-split ablation: twin-fixed-2m pretrained without chromosomes 20 and 21 |
| `twin-bpe-2m-no-chr20-21/` | Pretraining-split ablation: twin-bpe-2m pretrained without chromosomes 20 and 21 (its BPE vocabulary trained on those chromosomes) |
Each folder has `config.json` (the pretraining configuration and step), `model.safetensors` (the weights) and a model card.
The first four are pretrained on GRCh38 chromosomes 3-22. The four `-no-chr20-21` models are the same models pretrained on
chromosomes 1-19 and 22, without the chromosomes of the Nucleotide Transformer benchmark's test sets, everything else
unchanged; the paper uses them to check that seeing the test chromosomes in pretraining does not inflate the test scores.
```python
import locus
model = locus.load("locus-2m") # downloads locus-2m/ from this repository on first use
vectors = model.embed(["ACGT" * 100])
```
|