---
base_model: gnikesh/FunDLM-Small-1024bp
pipeline_tag: text-classification
tags:
  - dna
  - genomics
  - fungi
  - caduceus
  - lora
  - sequence-classification
---

# FunDLM-Small-1024bp-Mini

LoRA adapter that classifies 1,024-bp DNA windows as mini-chromosomal (1) or
core-chromosomal (0). It was fine-tuned on top of the fungal DNA language model
[FunDLM-Small-1024bp](https://huggingface.co/gnikesh/FunDLM-Small-1024bp) and released with
the FunDLM paper.

Mini-chromosomes (also called supernumerary or accessory chromosomes) are small, repeat-rich,
gene-poor chromosomes that can carry effector genes in blast fungi. This adapter recognizes
mini-chromosomal sequence in isolates that were not used for fine-tuning.

## Model summary

| | |
|---|---|
| Base architecture | FunDLM-Small-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 24 layers, d_model 768, 88M unique parameters) |
| Max input length | 1,024 bp |
| Task | Sequence classification (softmax over two classes) |
| Number of classes | 2 |
| Labels | 0 = core chromosome, 1 = mini-chromosome |
| Adapter | LoRA, r = 8, alpha = 32, dropout = 0.1 |
| Training checkpoint | `checkpoint-27000` (27,000 steps, 512 windows per step) |

## Repository contents

| Path | Contents |
|---|---|
| `adapter/adapter_config.json`, `adapter/adapter_model.safetensors` | LoRA adapter weights |
| `config.json`, `configuration_caduceus.py`, `modeling_caduceus.py`, `modeling_rcps.py` | base FunDLM-Small-1024bp configuration and custom model code |
| `pytorch_model.bin` | base FunDLM-Small-1024bp weights |
| `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` | base tokenizer |

The base model files are included so the repository is self-contained. The same base model is
also available as [`gnikesh/FunDLM-Small-1024bp`](https://huggingface.co/gnikesh/FunDLM-Small-1024bp).

## Installation

Environment setup follows the same steps as
[PlantCAD2](https://github.com/plantcad/plantcad/blob/main/docs/local-install.md). This model
uses a Mamba-based state-space backbone, so the Mamba kernels are required:

```bash
# Core dependencies (versions used for fine-tuning)
pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2
pip install torch  # choose the build that matches your CUDA version

# Mamba/Caduceus-specific dependencies
pip install mamba_ssm causal_conv1d --no-build-isolation
```

## Running the classifier in Python

The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the
model with `model.float()` and run the forward pass without the autocast block.

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base_id = "gnikesh/FunDLM-Small-1024bp"
adapter_id = "PlantG3/FunDLM-Small-1024bp-Mini"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=2,
    id2label={0: "core", 1: "mini"},
    label2id={"core": 0, "mini": 1},
)
# Use the base classifier with the adapter applied, without merging it.
model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model
model = model.cuda().eval()

sequence = "ACGT" * 256  # replace with a 1,024-bp window
encoded = tokenizer(
    sequence,
    add_special_tokens=False,
    padding="max_length",
    truncation=True,
    max_length=1024,
)
input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits = model(input_ids=input_ids).logits
prob_mini = torch.softmax(logits.float(), dim=-1)[0, 1].item()
print(f"mini-chromosome probability: {prob_mini:.4f}")
```

Do not call `merge_and_unload()` on this adapter. The forward and reverse-complement Mamba
branches carry separate LoRA weights, and merging folds both into the shared base weights and
changes the predictions.

## Batch prediction with the FunDLM code

The fine-tuning and prediction CLI is in the
[FunDLM repository](https://github.com/PlantG3/funDLM/blob/main/src/finetune/lora_fine_tune.py).
Clone this model repository and point `--checkpoint_dir` at its `adapter/` directory:

```bash
git clone https://huggingface.co/PlantG3/FunDLM-Small-1024bp-Mini

python src/finetune/lora_fine_tune.py predict \
    --checkpoint_dir FunDLM-Small-1024bp-Mini/adapter \
    --data_dir windows.parquet \
    --output_file predictions.csv \
    --model_name gnikesh/FunDLM-Small-1024bp \
    --task_type classification \
    --batch_size 32 \
    --bf16 True \
    --seed 42
```

`windows.parquet` is the tokenized input produced by the `tokenize` subcommand of the same
script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps
non-ACGT characters to `[UNK]`.

## Training data

The adapter was trained on 1,024-bp windows from the six *Pyricularia* assemblies used to train
MiniC: 70-15, B71Ref2, MZ5-1-6, LpKY97, O135 and TF05. Core and mini-chromosomal sequences were
tiled with different step sizes to balance the classes, and windows with a confident alignment
to sequence shared between the two compartments were removed. This gave 13,655,061 training and
718,688 validation windows.

## Training settings

- LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba `in_proj` and `out_proj` projections
- AdamW, learning rate 1e-4 with cosine decay, 2,700 warm-up steps
- Weight decay 0.01, gradient clipping 1.0, bfloat16, seed 42
- 16 windows per device x 32 gradient-accumulation steps = 512 windows per optimization step
- 27,000 optimization steps (one pass over the training windows)
- Reported checkpoint: `checkpoint-27000`

## Evaluation

Held-out test set: 162,080 non-overlapping 1,024-bp windows from four isolates that were not
used for fine-tuning (Br62, G239, O137 and JP37), of which 4,551 (2.8%) are mini-chromosomal.

| Metric | Value |
|---|---|
| AUROC | 0.986 |
| Average precision (AP) | 0.884 |
| Precision at probability 0.5 | 0.886 |
| Recall at probability 0.5 | 0.828 |
| F1 at probability 0.5 | 0.856 |

Average precision by isolate: Br62 0.883, G239 0.996, O137 0.799, JP37 0.557. This is the
adapter behind the paper's mini-chromosome results for the Small model; it performs slightly
better than the Medium-1024bp adapter (AP 0.866).

## Limitations

The held-out-isolate test largely measures recognition of sequence related to the training
isolates: 76% of the test windows are at least 99.9% identical to training sequence over their
best alignment. On windows without a close homolog in the training genomes, performance drops
for this adapter and for the baselines tested in the paper. The adapter is intended for
fungal mini-chromosome discovery and should not be interpreted as a general repeat or
accessory-chromosome detector without validation on the target lineage.

## Contact

PlantG3 lab, Kansas State University. Please open an issue at
[PlantG3/funDLM](https://github.com/PlantG3/funDLM/issues) for questions about the model.