README.md

---
license: other
license_name: lowdown-labs-lovely-license-1.0
license_link: LICENSE
tags:
  - fela
  - fourier-neural-operator
  - fno
  - cpu
  - on-device
  - genomics
  - dna
  - sequence-classification
  - gated-deltanet
library_name: transformers
pipeline_tag: text-classification
---

# DISCLAIMER

This model is a research preview. Its training and evaluation data (Genomic Benchmarks, Apache-2.0,
and the public GRCh38 reference assembly) is permissively licensed, but the released model itself is
under the Lowdown Labs Lovely License 1.0. Lowdown Labs has put together this model in the interest
of advancing public science.

# FELA-DNA: a small, CPU runnable DNA sequence model

FELA-DNA reads raw DNA, the letters A, C, G, T, and labels it. For a biologist that means you
can ask, right on your own laptop or even a Raspberry Pi, things like "is this region a
promoter?", "is this stretch coding or not?", or "is this an enhancer?" No GPU, no cloud.

Two trained pieces ship here. One is a small ready to run classifier that calls a window
coding or intergenomic. The other is a foundation encoder you fine tune for whatever task you
have in mind. Both could run on a CPU system.

# What goes in, what comes out

- Input: a DNA sequence as a string of A, C, G, T (and N for an unknown base). The model
  tokenizes one nucleotide per token (no k-mer vocabulary). A window is up to the variant's
  `seq_len` (256 for `coding_clf`, 1024 for `backbone`); shape after tokenizing is
  `(batch, seq_len)` of integer token ids.
- Output: a class label for the window. The class set depends on the task head you load,
  for example promoter vs not promoter, coding vs intergenomic, enhancer vs not enhancer,
  or human vs worm. The model returns one logit per class; take the argmax for the label,
  or the softmax for a probability.
- This enables on device, no cloud screening of regulatory and coding regions, and a
  per position score track over a long region (slide the window along a chromosome and
  keep the per window probability) that loads into a genome browser.

# What ships (two variants, both run on CPU)

The repo ships two safetensors files, picked out by the `variants` map in `config.json`.

- `coding_clf.safetensors` (the default), 3.12M parameters. This is the ready to run
  classifier. It labels a window coding or intergenomic and reaches 0.918 test accuracy on the
  GenomicBenchmarks demo task. Load it and score a sequence, nothing to train.
- `backbone.safetensors`, 14.64M parameters. This is the foundation encoder. It has no task
  head of its own, it just turns DNA into embeddings, and you fine tune it on top to reproduce
  the benchmark table below. It was pretrained on the human reference genome by learning to
  fill in hidden bases. Under the hood it uses a Gated DeltaNet recall layer, which runs on CPU
  in plain PyTorch (matched exactly against the GPU reference), so the GPU kernel is a
  training time thing only.

Neither variant needs a GPU to run. A GPU only speeds up training for our practical purposes.

# Why we built it this way

The sequence mixer is a Fourier Neural Operator, which is a learned global convolution
done in the frequency domain. It has no all pairs attention matrix, so the memory needed
to score a region does not grow with the square of the region length. In plain terms: a
standard attention transformer scoring a long stretch of DNA has to build an N by N table
that blows up the longer the input gets, while this model keeps a small, fixed working
memory and streams the sequence through. 

That is what lets it score a whole chromosome arm
on a CPU box and run on hardware as small as a Raspberry Pi. We train with
reverse complement augmentation and average predictions over a sequence and its
reverse complement at test time, so the model is approximately strand symmetric, which is
the right inductive bias for DNA.

# Performance

Speed and footprint, measured on CPU (AMD EPYC 9555, batch size 1, median of 20 runs).
Input is one 512 nucleotide window, shape `(1, 512)`.

## Pure FNO (fastest on CPU, the default on device model)

| Format | Size on disk | Peak working RAM | Latency 1 core | Latency 4 core | Device class |
|---|---|---|---|---|---|
| fp32 | 51.96 MB | 0.03 MB | 177.301 ms | 77.006 ms | Raspberry Pi Zero 2 W / Pi 4 class |

Throughput is 5.64 windows per second on one core. int8 disk size is 14.86 MB; per core int8
latency was not separately measured.

## GDN variant (more accurate on some tasks, also CPU, slower)

| Format | Size on disk | Peak working RAM | Latency 1 core | Latency 4 core | Device class |
|---|---|---|---|---|---|
| fp32 | 58.57 MB | 0.03 MB | 185.0 ms | 122.0 ms | Raspberry Pi class (CPU via pure torch recurrence) |

The GDN variant is 14.64M parameters and runs on CPU at 5.41 windows per second on one core.
Its CPU inference uses the pure torch gated delta rule recurrence and matches the fla
reference to 3.0e-08. int8 disk size is 16.13 MB.

## Long inputs (both variants): constant working memory

For long inputs the working RAM stays flat. With the pure FNO model, scoring a whole
200,000 bp region of GRCh38 chr21 took 5.0 s (about 40,000 bp/s) using 18 MB of RAM in a
single streaming pass. 

A standard softmax transformer attending over the same 200,000 bp
would need an N by N attention matrix of about 1,280 GB, which does not fit in commodity RAM,
so the memory advantage at that length is roughly 70,000x.


# Accuracy

We evaluate upon top 1 test accuracy on Genomic Benchmarks (Gresova et al. 2023), single nucleotide input,
with GRCh38 masked nucleotide pretraining, reverse complement augmentation, and test time
reverse complement averaging. We hold a validation split out of the training set for model
selection and never tune on the test set. HyenaDNA numbers are the published Genomic
Benchmarks results (Nguyen et al. 2023). All values are measured on the official test folds (ours).

| Benchmark | Metric | FELA-DNA GDN | HyenaDNA (published) |
|---|---|---|---|
| human nontata promoters | top-1 acc | 96.8 | 96.6 |
| demo coding vs intergenomic seqs | top-1 acc | 93.9 | 91.3 |
| demo human or worm | top-1 acc | 96.8 | 96.6 |
| human enhancers cohn | top-1 acc | 73.6 | 74.2 |
| human enhancers ensembl | top-1 acc | 92.2 | 89.2 |
| human ocr ensembl | top-1 acc | 80.4 | not reported |

The GDN backbone beats HyenaDNA on four tasks (nontata promoters 96.8 vs 96.6, coding vs intergenomic 93.9 vs 91.3,
human or worm 96.8 vs 96.6, and enhancers ensembl 92.2 vs 89.2; nontata and worm by thin
~0.2 point margins, coding and ensembl clearly), and is within about one point on the others.

Both backbones run on CPU, so again the choice is speed vs absolute accuracy. The honest limit is
human enhancers cohn, where both of our backbones trail HyenaDNA (73.6 and 71.5 vs 74.2).
With higher training and research budgets, we could probably close that gap.

# How to run it

See `quickstart/` for a minimal loader and the Hugging Face Space in `space/` for an
interactive playground. Both load through `modeling.py`:

```bash
cd quickstart && python run.py                     # coding_clf classifier (default)
cd quickstart && python run.py --variant backbone  # pretrained GDN encoder
```

For a production serving path, the variants run under the FELA server, a separate CPU native
serving project (`https://github.com/Lowdown-Labs/fela_server`).


## Loading with standard tooling

The repo ships `config.json` (a `variants` map with the architecture hyperparameters and
metadata for each variant) and a self contained `modeling.py` with a `load_model` /
`from_pretrained` entry. The weights ship as safetensors (not pickle): `coding_clf.safetensors`
(the default classifier) and `backbone.safetensors` (the GDN foundation encoder). A few lines
load a variant from a local directory or a Hugging Face repo:

```python
from modeling import load_model, score_sequence
m = load_model("/path/to/weights_dir")                    # default variant: coding_clf
label, prob = score_sequence(m, "ACGTACGT...")

enc = load_model("/path/to/weights_dir", variant="backbone")  # the GDN foundation encoder
emb = enc.embed(ids)                                      # masked mean pooled 384 dimensional embedding
```

## Formats

- fp32: reference and CPU.
- int8: on device deployment format (AVX512-VNNI on x86, NEON dot product on ARM). int8 disk
  size is 14.86 MB (pure FNO) and 16.13 MB (GDN).
- bf16: server inference; most commodity ARM CPUs lack native bf16, so it is not the
  on device format.

# Training data

- Pretraining: GRCh38 (the Genome Reference Consortium human reference genome, build 38),
  masked nucleotide self supervised pretraining on 10 chromosomes, 1024 bp windows, with
  reverse complement augmentation. GRCh38 is a public reference assembly.
- Fine tuning and evaluation: Genomic Benchmarks (Gresova et al. 2023), six tasks
  (human nontata promoters, demo coding vs intergenomic seqs, demo human or worm, human
  enhancers cohn, human enhancers ensembl, human ocr ensembl). Genomic Benchmarks is
  released under the Apache 2.0 license.
- Reference baseline for comparison: HyenaDNA (Nguyen et al. 2023), published Genomic
  Benchmarks results. We did not retrain HyenaDNA.
- The GDN variant uses the GatedDeltaNet layer from the flash-linear-attention (fla)
  library; its Triton kernel is used for training, and CPU inference uses a pure torch
  recurrence that matches the fla reference.

## Training data, splits and licensing

Fine tuning and evaluation dataset:

- Genomic Benchmarks (Gresova, Martinek, Cechak, Simecek, Alexiou. "Genomic Benchmarks:
  a collection of datasets for genomic sequence classification." BMC Genomic Data, 2023).
  Source: https://github.com/ML-Bioinfo-CEITEC/genomic_benchmarks. License: Apache-2.0
  (verified in the repository LICENSE file). Commercial use verdict: permitted (Apache-2.0
  is a permissive commercial friendly license, attribution required).
- Six tasks used: human_nontata_promoters, demo_coding_vs_intergenomic_seqs,
  demo_human_or_worm, human_enhancers_cohn, human_enhancers_ensembl, human_ocr_ensembl.

Pretraining data:

- GRCh38 (Genome Reference Consortium Human Build 38), a public reference assembly, no usage
  restriction for research use. Masked nucleotide self supervised pretraining on 10
  chromosomes, 1024 bp windows, reverse complement augmentation.

Split method and sizes:

- Test set is the OFFICIAL Genomic Benchmarks `test` fold per task (never tuned on). No
  seeded resplit of the test data. Official test sizes: human_nontata_promoters 9034,
  demo_coding_vs_intergenomic_seqs 25000, demo_human_or_worm 25000, human_enhancers_cohn
  6948, human_enhancers_ensembl 30970, human_ocr_ensembl 34952.
- Train/validation split: the official `train` fold is shuffled with `--seed 0` (default)
  and the first `--val_frac 0.1` (10%) is held out as validation for model selection; the
  remaining 90% is training. Example (human_nontata_promoters): train 24388, val 2709.
- Running `python train.py --dataset <task> --smoke` builds the split, asserts the official
  test size, and exits before training.

# Intended use, limitations, and safety

What it is for: research use. Zero infrastructure scoring of regulatory elements, promoters,
and coding regions on a laptop, a CPU box, or a Raspberry Pi, including long regions that do
not fit a transformer's attention matrix. A per position score track for a genome browser, and
simple variant effect scoring (score a reference window and an alternate allele window, take
the difference in the task probability).

What it is not for: this is not a clinical or diagnostic tool. Predictions are statistical.
Do not use FELA-DNA for any clinical, diagnostic, or safety critical decision without
independent validation against established methods and appropriate review.

Privacy: the model runs on the device. DNA sequences do not have to leave your machine, which
matters for sensitive genomic data.

Evaluated conditions and known failure modes:

- On human enhancers cohn, both backbones trail HyenaDNA. Short noisy enhancer windows are the
  weakest case for the global Fourier mixer.
- Pretraining used 10 GRCh38 chromosomes, not the whole genome, so coverage of rare context
  regions is partial.
- Evaluated on single nucleotide windows up to 512 bp for classification and on chr21 for the
  whole region streaming demo. Other species and very different sequence statistics were not
  evaluated.
- Latency and size were measured on an x86 server CPU; the Raspberry Pi claim is the
  size plus compute envelope, realized via the ONNX export.

# How to cite

```bibtex
@misc{lowdownlabs_feladna,
  title  = {FELA-DNA: a small CPU runnable Fourier Neural Operator model for DNA classification},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}
```

You must also cite the datasets, the reference benchmark, and the libraries used:

- GRCh38 reference genome (Genome Reference Consortium).
- Gresova, K., Martinek, V., Cechak, D., Simecek, P., Alexiou, P. (2023). Genomic
  Benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic
  Data.
- Nguyen, E., Poli, M., Faizi, M., et al. (2023). HyenaDNA: Long-Range Genomic Sequence
  Modeling at Single Nucleotide Resolution. NeurIPS.
- Li, Z., Kovachki, N., Azizzadenesheli, K., et al. (2021). Fourier Neural Operator for
  Parametric Partial Differential Equations. ICLR.
- flash-linear-attention (fla) library, for the GatedDeltaNet recall layer used in the GDN
  variant.
- PyTorch (Paszke et al. 2019).

## Acknowledgements and references

- GRCh38: Genome Reference Consortium Human Build 38.
- Genomic Benchmarks: Gresova et al. (2023), BMC Genomic Data.
- HyenaDNA: Nguyen et al. (2023), NeurIPS.
- Fourier Neural Operator: Li et al. (2021), ICLR.
- flash-linear-attention (fla): GatedDeltaNet linear attention kernels.
- PyTorch: Paszke et al. (2019), NeurIPS.

## Citations & licenses

This section gives the formal reference for every dataset and method the model actually
uses, each with a direct hyperlink to the real license text (verified from source). The
released model itself is under the Lowdown Labs Lovely License 1.0 (see License below),
independent of the permissive training data licenses recorded here.

## Datasets

**Genomic Benchmarks (fine tuning + evaluation).**
Gresova, K., Martinek, V., Cechak, D., Simecek, P., Alexiou, P. (2023). "Genomic Benchmarks:
a collection of datasets for genomic sequence classification." *BMC Genomic Data* 24, 25.
DOI: <https://doi.org/10.1186/s12863-023-01123-8>. Code/data repository:
<https://github.com/ML-Bioinfo-CEITEC/genomic_benchmarks>.
License: **Apache-2.0**, verified from the repository LICENSE file
(<https://github.com/ML-Bioinfo-CEITEC/genomic_benchmarks/blob/main/LICENSE>, "Apache License,
Version 2.0"). Commercial use: permitted with attribution.

```bibtex
@article{gresova2023genomic,
  title   = {Genomic Benchmarks: a collection of datasets for genomic sequence classification},
  author  = {Gresova, Katarina and Martinek, Vlastimil and Cechak, David and Simecek, Petr and Alexiou, Panagiotis},
  journal = {BMC Genomic Data},
  volume  = {24},
  number  = {1},
  pages   = {25},
  year    = {2023},
  doi     = {10.1186/s12863-023-01123-8}
}
```

**GRCh38 human reference assembly (self supervised pretraining).**
Genome Reference Consortium Human Build 38 (GRCh38). Assembly page:
<https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_000001405.26/>. GRCh38 is a public reference
assembly; the Genome Reference Consortium places no usage restriction on the reference for
research use (<https://www.ncbi.nlm.nih.gov/grc>). Attribution to the Genome Reference
Consortium is expected.

**HyenaDNA (published comparison baseline only, not training data).**
Nguyen, E., Poli, M., Faizi, M., et al. (2023). "HyenaDNA: Long-Range Genomic Sequence
Modeling at Single Nucleotide Resolution." *NeurIPS*. arXiv:2306.15794
(<https://arxiv.org/abs/2306.15794>). We did not retrain HyenaDNA; the numbers quoted are the
authors' published Genomic Benchmarks results.

## Methods and code

Confirmed against `modeling.py` and `config.json` in this repo: the model is a Fourier
Neural Operator sequence mixer, and the GDN variant adds a Gated DeltaNet recall layer
(`layer_pattern="SSSL"`, `use_gdn=true`) via the `fla` library at training time.

- **PyTorch** (the training and inference framework). Paszke, A., et al. (2019). "PyTorch: An
  Imperative Style, High-Performance Deep Learning Library." *NeurIPS 32*.
  <https://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library>.
- **Fourier Neural Operator** (the sequence mixer). Li, Z., Kovachki, N., Azizzadenesheli, K.,
  Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A. (2021). "Fourier Neural Operator for
  Parametric Partial Differential Equations." *ICLR*. arXiv:2010.08895
  (<https://arxiv.org/abs/2010.08895>).
- **Gated DeltaNet** (the recall layer in the GDN variant). Yang, S., Kautz, J., Hatamizadeh, A.
  (2024/2025). "Gated Delta Networks: Improving Mamba2 with Delta Rule." *ICLR 2025*.
  arXiv:2412.06464 (<https://arxiv.org/abs/2412.06464>). Builds on Gated Linear Attention:
  Yang, S., Wang, B., Shen, Y., Panda, R., Kim, Y. (2023). "Gated Linear Attention Transformers
  with Hardware-Efficient Training." arXiv:2312.06635 (<https://arxiv.org/abs/2312.06635>).
- **flash-linear-attention (`fla`)**, the GatedDeltaNet Triton kernel used for training the
  GDN variant (CPU inference uses the pure torch recurrence in `modeling.py` instead).
  Repository: <https://github.com/fla-org/flash-linear-attention> (MIT license).

# Model family

This is part of the FELA family from Lowdown Labs: one FNO architecture across many
modalities, all CPU native and subquadratic. This repo is pushed as `lowdown-labs/fela-genomics`.
The sibling repos (final Hugging Face names) are:

- `lowdown-labs/fela-genomics` (this repo): DNA sequence classification.
- `lowdown-labs/fela-pdm`: rotating machinery and turbofan health.
- `lowdown-labs/fela-power-grid`: probabilistic solar and wind power forecasting.
- `lowdown-labs/fela-video`: video moment retrieval and temporal grounding.
- `lowdown-labs/fela-streaming-asr`: streaming CPU speech recognition.

These are grouped under the FELA Collection on Hugging Face. The models are independently
trained per modality and do not share weights, so none carries a `base_model` link.

# License

Released under the Lowdown Labs Lovely License 1.0 (CC BY-NC 4.0 plus Hippocratic License 3.0). See LICENSE. For most LL models, a commercial license may be available; contact Lowdown Labs.