---
library_name: locus
tags:
- dna
- genomics
- locally-consistent-parsing
---

# LoCUS

Pretrained LoCUS models: DNA language models that read DNA as segments from locally consistent parsing (LCP).
Code, fine-tuning recipes and the scripts that reproduce every result of the paper: https://github.com/STORMgroup/locus

| Folder | Model |
|---|---|
| `locus-0.5m/` | LoCUS-0.5M: width 64, 4 kb context |
| `locus-2m/` | LoCUS-2M: width 128, 4 kb context |
| `twin-fixed-2m/` | Tokenizer study: LoCUS-2M with fixed 10 bp segments every 5 bp |
| `twin-bpe-2m/` | Tokenizer study: LoCUS-2M with BPE tokens (vocabulary 4,096) as segments |
| `locus-0.5m-no-chr20-21/` | Pretraining-split ablation: LoCUS-0.5M pretrained without chromosomes 20 and 21 |
| `locus-2m-no-chr20-21/` | Pretraining-split ablation: LoCUS-2M pretrained without chromosomes 20 and 21 |
| `twin-fixed-2m-no-chr20-21/` | Pretraining-split ablation: twin-fixed-2m pretrained without chromosomes 20 and 21 |
| `twin-bpe-2m-no-chr20-21/` | Pretraining-split ablation: twin-bpe-2m pretrained without chromosomes 20 and 21 (its BPE vocabulary trained on those chromosomes) |

Each folder has `config.json` (the pretraining configuration and step), `model.safetensors` (the weights) and a model card.
The first four are pretrained on GRCh38 chromosomes 3-22. The four `-no-chr20-21` models are the same models pretrained on
chromosomes 1-19 and 22, without the chromosomes of the Nucleotide Transformer benchmark's test sets, everything else
unchanged; the paper uses them to check that seeing the test chromosomes in pretraining does not inflate the test scores.

```python
import locus

model = locus.load("locus-2m")          # downloads locus-2m/ from this repository on first use
vectors = model.embed(["ACGT" * 100])
```