| --- |
| base_model: gnikesh/FunDLM-Small-1024bp |
| pipeline_tag: text-classification |
| tags: |
| - dna |
| - genomics |
| - fungi |
| - caduceus |
| - lora |
| - sequence-classification |
| --- |
| |
| # FunDLM-Small-1024bp-Mini |
|
|
| LoRA adapter that classifies 1,024-bp DNA windows as mini-chromosomal (1) or |
| core-chromosomal (0). It was fine-tuned on top of the fungal DNA language model |
| [FunDLM-Small-1024bp](https://huggingface.co/gnikesh/FunDLM-Small-1024bp) and released with |
| the FunDLM paper. |
|
|
| Mini-chromosomes (also called supernumerary or accessory chromosomes) are small, repeat-rich, |
| gene-poor chromosomes that can carry effector genes in blast fungi. This adapter recognizes |
| mini-chromosomal sequence in isolates that were not used for fine-tuning. |
|
|
| ## Model summary |
|
|
| | | | |
| |---|---| |
| | Base architecture | FunDLM-Small-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 24 layers, d_model 768, 88M unique parameters) | |
| | Max input length | 1,024 bp | |
| | Task | Sequence classification (softmax over two classes) | |
| | Number of classes | 2 | |
| | Labels | 0 = core chromosome, 1 = mini-chromosome | |
| | Adapter | LoRA, r = 8, alpha = 32, dropout = 0.1 | |
| | Training checkpoint | `checkpoint-27000` (27,000 steps, 512 windows per step) | |
| |
| ## Repository contents |
| |
| | Path | Contents | |
| |---|---| |
| | `adapter/adapter_config.json`, `adapter/adapter_model.safetensors` | LoRA adapter weights | |
| | `config.json`, `configuration_caduceus.py`, `modeling_caduceus.py`, `modeling_rcps.py` | base FunDLM-Small-1024bp configuration and custom model code | |
| | `pytorch_model.bin` | base FunDLM-Small-1024bp weights | |
| | `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` | base tokenizer | |
|
|
| The base model files are included so the repository is self-contained. The same base model is |
| also available as [`gnikesh/FunDLM-Small-1024bp`](https://huggingface.co/gnikesh/FunDLM-Small-1024bp). |
|
|
| ## Installation |
|
|
| Environment setup follows the same steps as |
| [PlantCAD2](https://github.com/plantcad/plantcad/blob/main/docs/local-install.md). This model |
| uses a Mamba-based state-space backbone, so the Mamba kernels are required: |
|
|
| ```bash |
| # Core dependencies (versions used for fine-tuning) |
| pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2 |
| pip install torch # choose the build that matches your CUDA version |
| |
| # Mamba/Caduceus-specific dependencies |
| pip install mamba_ssm causal_conv1d --no-build-isolation |
| ``` |
|
|
| ## Running the classifier in Python |
|
|
| The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the |
| model with `model.float()` and run the forward pass without the autocast block. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| from peft import PeftModel |
| |
| base_id = "gnikesh/FunDLM-Small-1024bp" |
| adapter_id = "PlantG3/FunDLM-Small-1024bp-Mini" |
| |
| tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True) |
| base = AutoModelForSequenceClassification.from_pretrained( |
| base_id, |
| trust_remote_code=True, |
| num_labels=2, |
| id2label={0: "core", 1: "mini"}, |
| label2id={"core": 0, "mini": 1}, |
| ) |
| # Use the base classifier with the adapter applied, without merging it. |
| model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model |
| model = model.cuda().eval() |
| |
| sequence = "ACGT" * 256 # replace with a 1,024-bp window |
| encoded = tokenizer( |
| sequence, |
| add_special_tokens=False, |
| padding="max_length", |
| truncation=True, |
| max_length=1024, |
| ) |
| input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda() |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): |
| logits = model(input_ids=input_ids).logits |
| prob_mini = torch.softmax(logits.float(), dim=-1)[0, 1].item() |
| print(f"mini-chromosome probability: {prob_mini:.4f}") |
| ``` |
|
|
| Do not call `merge_and_unload()` on this adapter. The forward and reverse-complement Mamba |
| branches carry separate LoRA weights, and merging folds both into the shared base weights and |
| changes the predictions. |
|
|
| ## Batch prediction with the FunDLM code |
|
|
| The fine-tuning and prediction CLI is in the |
| [FunDLM repository](https://github.com/PlantG3/funDLM/blob/main/src/finetune/lora_fine_tune.py). |
| Clone this model repository and point `--checkpoint_dir` at its `adapter/` directory: |
|
|
| ```bash |
| git clone https://huggingface.co/PlantG3/FunDLM-Small-1024bp-Mini |
| |
| python src/finetune/lora_fine_tune.py predict \ |
| --checkpoint_dir FunDLM-Small-1024bp-Mini/adapter \ |
| --data_dir windows.parquet \ |
| --output_file predictions.csv \ |
| --model_name gnikesh/FunDLM-Small-1024bp \ |
| --task_type classification \ |
| --batch_size 32 \ |
| --bf16 True \ |
| --seed 42 |
| ``` |
|
|
| `windows.parquet` is the tokenized input produced by the `tokenize` subcommand of the same |
| script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps |
| non-ACGT characters to `[UNK]`. |
|
|
| ## Training data |
|
|
| The adapter was trained on 1,024-bp windows from the six *Pyricularia* assemblies used to train |
| MiniC: 70-15, B71Ref2, MZ5-1-6, LpKY97, O135 and TF05. Core and mini-chromosomal sequences were |
| tiled with different step sizes to balance the classes, and windows with a confident alignment |
| to sequence shared between the two compartments were removed. This gave 13,655,061 training and |
| 718,688 validation windows. |
|
|
| ## Training settings |
|
|
| - LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba `in_proj` and `out_proj` projections |
| - AdamW, learning rate 1e-4 with cosine decay, 2,700 warm-up steps |
| - Weight decay 0.01, gradient clipping 1.0, bfloat16, seed 42 |
| - 16 windows per device x 32 gradient-accumulation steps = 512 windows per optimization step |
| - 27,000 optimization steps (one pass over the training windows) |
| - Reported checkpoint: `checkpoint-27000` |
|
|
| ## Evaluation |
|
|
| Held-out test set: 162,080 non-overlapping 1,024-bp windows from four isolates that were not |
| used for fine-tuning (Br62, G239, O137 and JP37), of which 4,551 (2.8%) are mini-chromosomal. |
|
|
| | Metric | Value | |
| |---|---| |
| | AUROC | 0.986 | |
| | Average precision (AP) | 0.884 | |
| | Precision at probability 0.5 | 0.886 | |
| | Recall at probability 0.5 | 0.828 | |
| | F1 at probability 0.5 | 0.856 | |
|
|
| Average precision by isolate: Br62 0.883, G239 0.996, O137 0.799, JP37 0.557. This is the |
| adapter behind the paper's mini-chromosome results for the Small model; it performs slightly |
| better than the Medium-1024bp adapter (AP 0.866). |
|
|
| ## Limitations |
|
|
| The held-out-isolate test largely measures recognition of sequence related to the training |
| isolates: 76% of the test windows are at least 99.9% identical to training sequence over their |
| best alignment. On windows without a close homolog in the training genomes, performance drops |
| for this adapter and for the baselines tested in the paper. The adapter is intended for |
| fungal mini-chromosome discovery and should not be interpreted as a general repeat or |
| accessory-chromosome detector without validation on the target lineage. |
|
|
| ## Contact |
|
|
| PlantG3 lab, Kansas State University. Please open an issue at |
| [PlantG3/funDLM](https://github.com/PlantG3/funDLM/issues) for questions about the model. |
|
|
