PlantHelixSeek

A Hybrid Linear Attention-Transformer Mixture-of-Experts (MoE) DNA Language Model for Plants

Single-nucleotide resolution · Bidirectional MLM · 8,192 bp context · 470M params

PlantHelixSeek is a plant DNA foundation model pretrained with masked language modeling (MLM) at single-nucleotide resolution. It is the base model for all PlantHelixSeek fine-tuned checkpoints (sequence classification, token classification, gene structure prediction, CRE prediction).

Model Details

Property	Value
Total parameters	470M
Active parameters per token	~235M (MoE routing)
Hidden size	512
Layers	39 (hybrid: Transformer + KDA + MLA)
Attention heads	16
Experts	8 (top-1 + 1 shared)
Max context	8,192 bp
Vocabulary	11 tokens (single-nucleotide: A/T/C/G/N + specials)
Architecture	HelixSeek (Transformer + KDA + MLA + MoE)

Architecture

HelixSeek interleaves three complementary layer types for efficient long-context bidirectional DNA modeling:

• Transformer layers — full attention for precise positional reasoning
• HelixSeek-Delta (KDA) — Delta Attention: linear attention with gated delta rule for O(n) long-range modeling
• HelixSeek-MLA — Multi-head Latent Attention for KV-cache compression

The FFN uses a sparse Mixture-of-Experts with 8 experts (top-1 routing + 1 shared expert), achieving high capacity at constant inference cost.

Downstream Models

All downstream checkpoints are fine-tuned from this base:

Sequence Classification (binary)

• PlantHelixSeek-core_promoter — Core promoter identification
• PlantHelixSeek-lncRNA — lncRNA classification
• PlantHelixSeek-open_chromatin — Open chromatin region
• PlantHelixSeek-sequence_conservation — Sequence conservation
• PlantHelixSeek-H3K27ac / -H3K27me3 / -H3K4me3 — Histone marks
• PlantHelixSeek-translation_onoff — Translation on/off

Regression

• PlantHelixSeek-promoter_strength — Promoter strength
• PlantHelixSeek-terminator_strength — Terminator strength
• PlantHelixSeek-translation_absolute — Translation level

Paper Downstream Tasks

• PlantHelixSeek-Anno — Gene structure prediction (17 BILOU labels, token classification)
• PlantHelixSeek-CRE — Cis-regulatory element prediction

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)

# Mask recovery
seq = "ATCGGCTAAGCCT<mask>GTCAGTTACCGATCG"
inputs = tokenizer(seq, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

Always pass trust_remote_code=True — the HelixSeek architecture is registered via HuggingFace auto_map.

License

CC-BY-NC 4.0 (research use; contact authors for commercial licensing)