README.md

---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: fill-mask
tags:
- DNA
- genomics
- fish
- Caduceus
- masked-language-model
- nucleotide-modeling
- foundation-model
- reverse-complement
- custom-code
- FishCaduceus
---

# FishCaduceus-28L-1024

`FishCaduceus-28L-1024` is a fish-specific DNA language model in the **FishCaduceus** family. It was pretrained on fish genomic sequences using a masked language modeling objective and is intended for nucleotide-level representation learning, transfer learning, and sequence-based analyses in fish genomics.

This repository contains the pretrained `FishCaduceus-28L-1024` checkpoint.

## Model description

FishCaduceus is a family of Caduceus-based DNA language models developed for fish genomes. The models operate at single-nucleotide resolution and use bidirectional state-space sequence modeling together with reverse-complement-aware components.

The model learns to recover masked nucleotides from their surrounding genomic context. Its hidden representations can also be used as input features for downstream fish genomics tasks.

## Model family

| Model | Number of layers | Hidden dimension | Pretraining context length |
|---|---:|---:|---:|
| FishCaduceus-20L-512 | 20 | 384 | 512 nt |
| FishCaduceus-28L-512 | 28 | 768 | 512 nt |
| **FishCaduceus-28L-1024** | **28** | **768** | **1024 nt** |

## Architecture

- **Backbone**: Caduceus-based bidirectional state-space DNA language model
- **Training objective**: masked language modeling
- **Tokenization**: single-nucleotide character-level tokenization
- **Vocabulary size**: 8
- **Number of layers**: 28
- **Hidden dimension (`d_model`)**: 768
- **Pretraining context length**: 1024 nucleotides
- **Selected training checkpoint**: step 45,000
- **Framework**: PyTorch and Hugging Face Transformers
- **Model type**: custom Transformers model (`trust_remote_code=True`)

The context length above describes the sequence length used during pretraining. Users should apply truncation or windowing appropriate for the selected model variant.

## Training data

The model was pretrained on curated fish genomic sequences obtained from publicly available genome resources. The genomes were processed through a unified quality-control and sequence-preparation workflow before construction of the masked-language-modeling corpus.

Detailed information about the included fish species, genome assemblies, preprocessing criteria, and dataset splits will be provided in the accompanying FishCaduceus dataset card and manuscript.

## Training objective

FishCaduceus was pretrained using masked language modeling. Selected nucleotide positions were masked, and the model was optimized to predict the original nucleotide from the surrounding sequence context.

Because the tokenizer operates at single-nucleotide resolution, model outputs can be used for nucleotide-level probability estimation and sequence representation learning.

## Intended uses

The model is intended for research applications including:

- nucleotide-level genomic sequence representation learning
- masked nucleotide prediction
- extraction of frozen sequence embeddings
- transfer learning and downstream fine-tuning
- fish genomic annotation tasks
- comparative and evolutionary genomics
- exploratory variant-effect and regulatory-sequence analyses

## Out-of-scope uses

The model is not intended to:

- provide clinical or diagnostic decisions
- replace experimental validation
- establish biological causality from sequence scores alone
- be used as the sole basis for decisions affecting human or animal health

## Limitations

- The model was developed primarily from fish genomic sequences, so performance may be lower for distant taxonomic groups.
- Predictions reflect patterns learned from the pretraining corpus and do not by themselves demonstrate biological function or causality.
- Performance may vary across species, genomic regions, sequence lengths, and downstream task definitions.
- Long genomic sequences should be divided into windows compatible with the selected model's pretraining context length.
- This repository contains custom modeling code. Review the repository code before loading it with `trust_remote_code=True`.

## How to use

Install the dependencies listed in `requirements.txt` before loading the model.

### Load the tokenizer and model

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()
```

### Masked nucleotide prediction

```python
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"
sequence = "ACGTACGT[MASK]ACGTACGT"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model.eval()

inputs = tokenizer(sequence, return_tensors="pt")

with torch.inference_mode():
    outputs = model(**inputs)

mask_rows, mask_cols = (
    inputs["input_ids"] == tokenizer.mask_token_id
).nonzero(as_tuple=True)

base_tokens = ["A", "C", "G", "T"]
base_ids = tokenizer.convert_tokens_to_ids(base_tokens)

mask_logits = outputs.logits[mask_rows[0], mask_cols[0], base_ids]
base_probabilities = torch.softmax(mask_logits, dim=-1)

print(dict(zip(base_tokens, base_probabilities.tolist())))
```

### Extract sequence representations

```python
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"
sequence = "ACGTACGTACGTACGT"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model.eval()

inputs = tokenizer(
    sequence,
    return_tensors="pt",
    truncation=True,
    max_length=1024,
)

with torch.inference_mode():
    outputs = model(
        **inputs,
        output_hidden_states=True,
    )

token_embeddings = outputs.hidden_states[-1]
sequence_embedding = token_embeddings.mean(dim=1)

print("Token embeddings:", token_embeddings.shape)
print("Sequence embedding:", sequence_embedding.shape)
```

## Repository files

The repository includes the files required to load the custom FishCaduceus model and tokenizer:

- `config.json`
- model weights (`pytorch_model.bin` or `model.safetensors`)
- `configuration_caduceus.py`
- `modeling_caduceus.py`
- `modeling_rcps.py`
- `tokenization_caduceus.py`
- `tokenizer.json`
- `tokenizer_config.json`
- `special_tokens_map.json`
- `requirements.txt`
- `README.md`

## Citation

The FishCaduceus manuscript is in preparation. Citation information will be added after publication.

Users of the model should also acknowledge the original Caduceus architecture and other upstream software components where appropriate.

## Acknowledgements

FishCaduceus builds on the Caduceus architecture and related open-source sequence-modeling software. The model was developed for research on fish genomes at the Institute of Hydrobiology, Chinese Academy of Sciences.

## License

This repository is released under the Apache License 2.0. Users are responsible for complying with the licenses of all upstream software dependencies.

## Contact

For questions, please contact: xqxia@ihb.ac.cn