README.md

metadata

language:
  - dna
tags:
  - genomics
  - biology
  - dna
  - masked-language-model
  - prokaryotic
  - metagenomics
  - bioinformatics
license: apache-2.0
datasets:
  - arcinstitute/opengenome2
library_name: pytorch
pipeline_tag: fill-mask
model-index:
  - name: seqlens-v2-micro-16k
    results:
      - task:
          type: text-classification
          name: Coding vs Non-coding Classification
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.9111
      - task:
          type: text-classification
          name: Genus Classification (50 held-out genera)
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.7978

SeqLens v2 Micro 16K

A compact genomic language model pre-trained on prokaryotic genomes for microbial bioinformatics tasks.

Model Description

SeqLens v2 is a bidirectional genomic language model built on BiMamba (bidirectional Mamba2 SSM) with interleaved sliding-window attention. It is designed for microbial genomics — taxonomic classification, antimicrobial resistance detection, plasmid identification, and metagenomic analysis.

The Micro variant is the smallest in the SeqLens v2 family, targeting high-throughput, low-latency inference.

Property	Value
Parameters	10.3M
Hidden dimension	256
Layers	8 (BiMamba) + 2 (sliding-window attention at layers 3, 7)
Context length	16,384 tokens (single nucleotide)
Vocabulary	A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens)
Pre-training objective	Masked Language Modeling (MLM), 15% mask rate
Architecture	BiMamba2 + chunked sliding-window attention + SwiGLU FFN

Architecture Details

• BiMamba blocks: Bidirectional Mamba2 SSM — processes sequences in both forward and reverse directions using shared weights. Provides O(L) scaling with sequence length.
• Sliding-window attention: Applied every 4th layer with window size 512. Captures fine-grained local patterns (codons, motifs) that SSMs can miss.
• Attention-weighted pooling: Learned pooling for sequence-level embeddings (superior to mean pooling for downstream tasks).
• SwiGLU FFN: Gated feed-forward with 4× expansion at each layer.

Training

Data

Pre-trained on prokaryotic genomes from OpenGenome2 (Apache 2.0):

• GTDB v220: 113,379 species-cluster representative genomes
• Single-nucleotide tokenization, 16,384 bp chunks
• Quality filtered: sequences with >10% N or low Shannon entropy excluded

Hyperparameters

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.98, ε=1e-8)
Learning rate	1e-3 (cosine decay to 1e-5)
Warmup	500 steps
Weight decay	0.1
Gradient clipping	1.0
Precision	BF16 mixed
Batch size	64 effective (8 × 8 GPUs)
Total steps	10,000
Tokens seen	~1.2B

Compute

Resource	Value
Hardware	8× NVIDIA A100-SXM4-80GB
Training time	57 minutes
Framework	PyTorch 2.6.0 + mamba-ssm 2.2.4

Evaluation

Coding vs Non-coding Classification (linear probe, frozen backbone)

Model	Params	Accuracy	F1
SeqLens v2 Micro	10M	0.911	0.911
SeqLens v1 (89M)	89M	0.687	0.687
4-mer baseline	—	0.588	0.588
Random init	10M	0.596	0.596

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Model	Params	Accuracy	F1
4-mer baseline	—	0.865	0.838
Random init	10M	0.826	0.768
SeqLens v2 Micro	10M	0.798	0.730

Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.

Extended Benchmark Comparison

All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.

ProkBERT Prokaryotic Benchmarks (accuracy)

Task	v2_novel	v2_standard	NTv3-8M	Caduceus	SeqLens_v1	NT-v2	DNABERT-2
Phage ID (L512)	0.751	0.730	0.762	0.653	0.690	0.626	0.652
Phage ID (L1024)	0.805	0.775	0.808	0.650	0.713	0.659	0.700
Phage ID (L2048)	0.838	0.815	0.834	0.675	0.770	0.713	0.723
Promoter (sigma70)	0.653	0.657	0.674	0.665	0.628	0.594	0.591
Promoter (multispecies)	0.582	0.584	0.630	0.608	0.588	0.572	0.545
Lifestyle (BASEL)	0.705	0.737	0.753	0.702	0.735	0.669	0.700
Lifestyle (E. coli, held-out)	0.648	0.667	0.706	0.627	0.660	0.607	0.638
Lifestyle (Extremophile)	0.813	0.871	0.835	0.852	0.775	0.823	0.797
Average	0.724	0.729	0.751	0.679	0.695	0.658	0.668

NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Our v2_standard (this model) is the best of our own three recipes here — the opposite ranking from CDS/GenomicBenchmarks, where the novel recipe wins. See PROJECT_STATE.md §3.4 for a per-task recipe breakdown.

GenomicBenchmarks — Eukaryotic Tasks (accuracy)

Task	v2_novel	v2_standard	NTv3-8M	Caduceus	SeqLens_v1	NT-v2	DNABERT-2
Mouse Enhancers	0.822	0.826	0.810	0.744	0.806	0.802	0.727
Coding vs Intergenic	0.895	0.904	0.927	0.936	0.929	0.886	0.948
Human vs Worm	0.906	0.928	0.955	0.967	0.969	0.941	0.981
Enhancers (Cohn)	0.730	0.726	0.734	0.746	0.754	0.721	0.809
Enhancers (Ensembl)	0.715	0.729	0.735	0.747	0.771	0.749	0.758
NonTATA Promoters	0.830	0.842	0.852	0.865	0.864	0.831	0.891
OCR (Ensembl)	0.661	0.676	0.670	0.682	0.692	0.670	0.677
Average (excl. Regulatory)	0.794	0.805	0.812	0.813	0.826	0.800	0.827

The human_ensembl_regulatory task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.

Full results: experiments/eval_reports/genomic_benchmarks/comparison.json and experiments/eval_reports/prokbench/comparison.json.

Usage

import torch
from model import SeqLensForMLM
from config import SeqLensConfig, MICRO_CONFIG
from tokenizer import NucleotideTokenizer

# Load model
device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

# Tokenize a DNA sequence
tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCGATCG..." # your DNA sequence
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Get sequence-level embeddings (for classification tasks)
with torch.no_grad():
    embeddings = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Or get per-position predictions (MLM)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Variant	Params	Layers	Dim	Context	Status
Micro	10M	8	256	16K	✅ Released
Base	~100M	12	512	32K	In development
Large	~400M	24	768	64K	Planned

Limitations

• Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
• 16K context may truncate long contigs; longer variants planned
• Current model trained for ~1.2B tokens; extended training may improve performance
• The human_ensembl_regulatory GenomicBenchmarks task is excluded from our averages — it's a confirmed sequence-length artifact of single-nucleotide tokenization, not a genuine capability (see Extended Benchmark Comparison above)

Citation

@misc{seqlens-v2-2026,
  title={SeqLens v2: Compact Genomic Language Models for Microbial Bioinformatics},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-16k}
}

License

Apache 2.0