FunDLM-Small-1024bp-Centromere

LoRA adapter that classifies 1,024-bp DNA windows as centromeric (1) or non-centromeric (0). It was fine-tuned on top of the fungal DNA language model FunDLM-Small-1024bp and released with the FunDLM paper.

Centromeres are among the fastest-evolving genomic regions and are defined epigenetically by the centromere-specific histone CENP-A rather than by a conserved DNA sequence. This adapter recognizes centromere-associated sequence in a chromosome that was withheld from fine-tuning.

Model summary

Base architecture	FunDLM-Small-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 24 layers, d_model 768, 88M unique parameters)
Max input length	1,024 bp
Task	Sequence classification (softmax over two classes)
Number of classes	2
Labels	0 = non-centromeric, 1 = centromeric
Adapter	LoRA, r = 8, alpha = 32, dropout = 0.1
Training checkpoint	checkpoint-25008 (25,008 steps, 192 windows per step)

Repository contents

Path	Contents
adapter/adapter_config.json, adapter/adapter_model.safetensors	LoRA adapter weights
config.json, configuration_caduceus.py, modeling_caduceus.py, modeling_rcps.py	base FunDLM-Small-1024bp configuration and custom model code
pytorch_model.bin	base FunDLM-Small-1024bp weights
tokenizer.json, tokenizer_config.json, special_tokens_map.json	base tokenizer

The base model files are included so the repository is self-contained. The same base model is also available as gnikesh/FunDLM-Small-1024bp.

Installation

Environment setup follows the same steps as PlantCAD2. This model uses a Mamba-based state-space backbone, so the Mamba kernels are required:

# Core dependencies (versions used for fine-tuning)
pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2
pip install torch  # choose the build that matches your CUDA version

# Mamba/Caduceus-specific dependencies
pip install mamba_ssm causal_conv1d --no-build-isolation

Running the classifier in Python

The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the model with model.float() and run the forward pass without the autocast block.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base_id = "gnikesh/FunDLM-Small-1024bp"
adapter_id = "PlantG3/FunDLM-Small-1024bp-Centromere"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=2,
    id2label={0: "non-centromeric", 1: "centromeric"},
    label2id={"non-centromeric": 0, "centromeric": 1},
)
# Use the base classifier with the adapter applied, without merging it.
model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model
model = model.cuda().eval()

sequence = "ACGT" * 256  # replace with a 1,024-bp window
encoded = tokenizer(
    sequence,
    add_special_tokens=False,
    padding="max_length",
    truncation=True,
    max_length=1024,
)
input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits = model(input_ids=input_ids).logits
prob_centromere = torch.softmax(logits.float(), dim=-1)[0, 1].item()
print(f"centromere probability: {prob_centromere:.4f}")

Do not call merge_and_unload() on this adapter. The forward and reverse-complement Mamba branches carry separate LoRA weights, and merging folds both into the shared base weights and changes the predictions.

Batch prediction with the FunDLM code

The fine-tuning and prediction CLI is in the FunDLM repository. Clone this model repository and point --checkpoint_dir at its adapter/ directory:

git clone https://huggingface.co/PlantG3/FunDLM-Small-1024bp-Centromere

python src/finetune/lora_fine_tune.py predict \
    --checkpoint_dir FunDLM-Small-1024bp-Centromere/adapter \
    --data_dir windows.parquet \
    --output_file predictions.csv \
    --model_name gnikesh/FunDLM-Small-1024bp \
    --task_type classification \
    --batch_size 32 \
    --bf16 True \
    --seed 42

windows.parquet is the tokenized input produced by the tokenize subcommand of the same script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps non-ACGT characters to [UNK].

Training data

The adapter was trained on 1,024-bp windows from three Pyricularia assemblies: Guy11, Br36 and Br62. Chromosome 6 of Br62 was withheld entirely from training because its centromere is less similar to the others. The held-out test set contains 68,168 windows from that chromosome, of which 33,166 are centromeric and 35,002 are non-centromeric. The centromeric test windows are 3-bp tiles of a single, roughly 100.5-kb centromere, so they overlap almost entirely and window-level confidence intervals understate the uncertainty of this test.

Training settings

• LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba in_proj and out_proj projections
• AdamW, learning rate 1e-5 with cosine decay, 2,501 warm-up steps
• Weight decay 0.01, gradient clipping 0.5, bfloat16, seed 42
• 24 windows per device x 8 gradient-accumulation steps = 192 windows per optimization step
• 25,008 optimization steps
• Reported checkpoint: checkpoint-25008

Evaluation

Held-out test set: the 68,168 windows of Br62 chromosome 6.

Metric	Value
AUROC	0.904
Average precision (AP)	0.916
Accuracy at probability 0.5	0.826
Precision at probability 0.5	0.874
Recall at probability 0.5	0.750
F1 at probability 0.5	0.808

This is the adapter behind the paper's centromere results for the Small model. In the paper the Medium-1024bp adapter localized the annotated centromere of every core chromosome as a single high-scoring region in a genome-wide B71 scan; the two adapters reached similar ranking performance, but the Small scan contained more high-scoring regions outside the annotated centromeres.

Limitations

The Br62 chromosome-6 test windows tile one centromere at 3-bp steps and are therefore not independent observations. Centromeres and mini-chromosomes are both repeat-rich and AT-rich, and the B71 genome-wide scan showed that high-scoring windows also occur in the mini-chromosome, so a high probability should be interpreted at the level of regions rather than individual windows. The adapter was trained on three closely related Pyricularia genomes; validate it on the target lineage before applying it broadly.

Contact

PlantG3 lab, Kansas State University. Please open an issue at PlantG3/funDLM for questions about the model.