| --- |
| library_name: locus |
| tags: |
| - dna |
| - genomics |
| - locally-consistent-parsing |
| --- |
| |
| # LoCUS |
|
|
| Pretrained LoCUS models: DNA language models that read DNA as segments from locally consistent parsing (LCP). |
| Code, fine-tuning recipes and the scripts that reproduce every result of the paper: https://github.com/STORMgroup/locus |
|
|
| | Folder | Model | |
| |---|---| |
| | `locus-0.5m/` | LoCUS-0.5M: width 64, 4 kb context | |
| | `locus-2m/` | LoCUS-2M: width 128, 4 kb context | |
| | `twin-fixed-2m/` | Tokenizer study: LoCUS-2M with fixed 10 bp segments every 5 bp | |
| | `twin-bpe-2m/` | Tokenizer study: LoCUS-2M with BPE tokens (vocabulary 4,096) as segments | |
| | `locus-0.5m-no-chr20-21/` | Pretraining-split ablation: LoCUS-0.5M pretrained without chromosomes 20 and 21 | |
| | `locus-2m-no-chr20-21/` | Pretraining-split ablation: LoCUS-2M pretrained without chromosomes 20 and 21 | |
| | `twin-fixed-2m-no-chr20-21/` | Pretraining-split ablation: twin-fixed-2m pretrained without chromosomes 20 and 21 | |
| | `twin-bpe-2m-no-chr20-21/` | Pretraining-split ablation: twin-bpe-2m pretrained without chromosomes 20 and 21 (its BPE vocabulary trained on those chromosomes) | |
|
|
| Each folder has `config.json` (the pretraining configuration and step), `model.safetensors` (the weights) and a model card. |
| The first four are pretrained on GRCh38 chromosomes 3-22. The four `-no-chr20-21` models are the same models pretrained on |
| chromosomes 1-19 and 22, without the chromosomes of the Nucleotide Transformer benchmark's test sets, everything else |
| unchanged; the paper uses them to check that seeing the test chromosomes in pretraining does not inflate the test scores. |
|
|
| ```python |
| import locus |
| |
| model = locus.load("locus-2m") # downloads locus-2m/ from this repository on first use |
| vectors = model.embed(["ACGT" * 100]) |
| ``` |
|
|
