Upload FishCaduceus-28L-512 pretrained model files

README.md

@@ -1,3 +1,228 @@
 ---
-license: cc-by-nc-sa-4.0
+language:
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: fill-mask
+tags:
+- DNA
+- genomics
+- fish
+- Caduceus
+- masked-language-model
+- nucleotide-modeling
+- foundation-model
+- reverse-complement
+- custom-code
+- FishCaduceus
 ---
+
+# FishCaduceus-28L-512
+
+`FishCaduceus-28L-512` is a fish-specific DNA language model in the **FishCaduceus** family. It was pretrained on fish genomic sequences using a masked language modeling objective and is intended for nucleotide-level representation learning, transfer learning, and sequence-based analyses in fish genomics.
+
+This repository contains the pretrained `FishCaduceus-28L-512` checkpoint.
+
+## Model description
+
+FishCaduceus is a family of Caduceus-based DNA language models developed for fish genomes. The models operate at single-nucleotide resolution and use bidirectional state-space sequence modeling together with reverse-complement-aware components.
+
+The model learns to recover masked nucleotides from their surrounding genomic context. Its hidden representations can also be used as input features for downstream fish genomics tasks.
+
+## Model family
+
+| Model | Number of layers | Hidden dimension | Pretraining context length |
+|---|---:|---:|---:|
+| FishCaduceus-20L-512 | 20 | 384 | 512 nt |
+| **FishCaduceus-28L-512** | **28** | **768** | **512 nt** |
+| FishCaduceus-28L-1024 | 28 | 768 | 1024 nt |
+
+## Architecture
+
+- **Backbone**: Caduceus-based bidirectional state-space DNA language model
+- **Training objective**: masked language modeling
+- **Tokenization**: single-nucleotide character-level tokenization
+- **Vocabulary size**: 8
+- **Number of layers**: 28
+- **Hidden dimension (`d_model`)**: 768
+- **Pretraining context length**: 512 nucleotides
+- **Selected training checkpoint**: step 40,000
+- **Framework**: PyTorch and Hugging Face Transformers
+- **Model type**: custom Transformers model (`trust_remote_code=True`)
+
+The context length above describes the sequence length used during pretraining. Users should apply truncation or windowing appropriate for the selected model variant.
+
+## Training data
+
+The model was pretrained on curated fish genomic sequences obtained from publicly available genome resources. The genomes were processed through a unified quality-control and sequence-preparation workflow before construction of the masked-language-modeling corpus.
+
+Detailed information about the included fish species, genome assemblies, preprocessing criteria, and dataset splits will be provided in the accompanying FishCaduceus dataset card and manuscript.
+
+## Training objective
+
+FishCaduceus was pretrained using masked language modeling. Selected nucleotide positions were masked, and the model was optimized to predict the original nucleotide from the surrounding sequence context.
+
+Because the tokenizer operates at single-nucleotide resolution, model outputs can be used for nucleotide-level probability estimation and sequence representation learning.
+
+## Intended uses
+
+The model is intended for research applications including:
+
+- nucleotide-level genomic sequence representation learning
+- masked nucleotide prediction
+- extraction of frozen sequence embeddings
+- transfer learning and downstream fine-tuning
+- fish genomic annotation tasks
+- comparative and evolutionary genomics
+- exploratory variant-effect and regulatory-sequence analyses
+
+## Out-of-scope uses
+
+The model is not intended to:
+
+- provide clinical or diagnostic decisions
+- replace experimental validation
+- establish biological causality from sequence scores alone
+- be used as the sole basis for decisions affecting human or animal health
+
+## Limitations
+
+- The model was developed primarily from fish genomic sequences, so performance may be lower for distant taxonomic groups.
+- Predictions reflect patterns learned from the pretraining corpus and do not by themselves demonstrate biological function or causality.
+- Performance may vary across species, genomic regions, sequence lengths, and downstream task definitions.
+- Long genomic sequences should be divided into windows compatible with the selected model's pretraining context length.
+- This repository contains custom modeling code. Review the repository code before loading it with `trust_remote_code=True`.
+
+## How to use
+
+Install the dependencies listed in `requirements.txt` before loading the model.
+
+### Load the tokenizer and model
+
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model_id = "FishCaduceus/FishCaduceus-28L-512"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model = AutoModelForMaskedLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model.eval()
+```
+
+### Masked nucleotide prediction
+
+```python
+import torch
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model_id = "FishCaduceus/FishCaduceus-28L-512"
+sequence = "ACGTACGT[MASK]ACGTACGT"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model = AutoModelForMaskedLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model.eval()
+
+inputs = tokenizer(sequence, return_tensors="pt")
+
+with torch.inference_mode():
+    outputs = model(**inputs)
+
+mask_rows, mask_cols = (
+    inputs["input_ids"] == tokenizer.mask_token_id
+).nonzero(as_tuple=True)
+
+base_tokens = ["A", "C", "G", "T"]
+base_ids = tokenizer.convert_tokens_to_ids(base_tokens)
+
+mask_logits = outputs.logits[mask_rows[0], mask_cols[0], base_ids]
+base_probabilities = torch.softmax(mask_logits, dim=-1)
+
+print(dict(zip(base_tokens, base_probabilities.tolist())))
+```
+
+### Extract sequence representations
+
+```python
+import torch
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model_id = "FishCaduceus/FishCaduceus-28L-512"
+sequence = "ACGTACGTACGTACGT"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model = AutoModelForMaskedLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+model.eval()
+
+inputs = tokenizer(
+    sequence,
+    return_tensors="pt",
+    truncation=True,
+    max_length=512,
+)
+
+with torch.inference_mode():
+    outputs = model(
+        **inputs,
+        output_hidden_states=True,
+    )
+
+token_embeddings = outputs.hidden_states[-1]
+sequence_embedding = token_embeddings.mean(dim=1)
+
+print("Token embeddings:", token_embeddings.shape)
+print("Sequence embedding:", sequence_embedding.shape)
+```
+
+## Repository files
+
+The repository includes the files required to load the custom FishCaduceus model and tokenizer:
+
+- `config.json`
+- model weights (`pytorch_model.bin` or `model.safetensors`)
+- `configuration_caduceus.py`
+- `modeling_caduceus.py`
+- `modeling_rcps.py`
+- `tokenization_caduceus.py`
+- `tokenizer.json`
+- `tokenizer_config.json`
+- `special_tokens_map.json`
+- `requirements.txt`
+- `README.md`
+
+## Citation
+
+The FishCaduceus manuscript is in preparation. Citation information will be added after publication.
+
+Users of the model should also acknowledge the original Caduceus architecture and other upstream software components where appropriate.
+
+## Acknowledgements
+
+FishCaduceus builds on the Caduceus architecture and related open-source sequence-modeling software. The model was developed for research on fish genomes at the Institute of Hydrobiology, Chinese Academy of Sciences.
+
+## License
+
+This repository is released under the Apache License 2.0. Users are responsible for complying with the licenses of all upstream software dependencies.
+
+## Contact
+
+For questions, please contact: xqxia@ihb.ac.cn

config.json

@@ -0,0 +1,57 @@
+{
+  "_name_or_path": "FishCaduceus-28L-512",
+  "architectures": [
+    "CaduceusForMaskedLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_caduceus.CaduceusConfig",
+    "AutoModel": "modeling_caduceus.Caduceus",
+    "AutoModelForMaskedLM": "modeling_caduceus.CaduceusForMaskedLM",
+    "AutoModelForSequenceClassification": "modeling_caduceus.CaduceusForSequenceClassification"
+  },
+  "bidirectional": true,
+  "bidirectional_strategy": "add",
+  "bidirectional_weight_tie": true,
+  "complement_map": {
+    "0": 0,
+    "1": 1,
+    "2": 2,
+    "3": 6,
+    "4": 5,
+    "5": 4,
+    "6": 3,
+    "7": 7
+  },
+  "d_model": 768,
+  "fused_add_norm": true,
+  "initializer_cfg": {
+    "initializer_range": 0.02,
+    "n_residuals_per_layer": 1,
+    "rescale_prenorm_residual": true
+  },
+  "model_type": "caduceus",
+  "n_layer": 28,
+  "norm_epsilon": 1e-05,
+  "pad_token_id": -100,
+  "pad_vocab_size_multiple": 8,
+  "rcps": true,
+  "residual_in_fp32": true,
+  "rms_norm": true,
+  "ssm_cfg": {
+    "bias": false,
+    "conv_bias": true,
+    "d_conv": 4,
+    "d_state": 16,
+    "dt_init": "random",
+    "dt_init_floor": 0.0001,
+    "dt_max": 0.1,
+    "dt_min": 0.001,
+    "dt_rank": "auto",
+    "dt_scale": 1.0,
+    "expand": 2,
+    "use_fast_path": true
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.3",
+  "vocab_size": 8
+}

configuration_caduceus.py

@@ -0,0 +1,55 @@
+"""Caduceus config for Hugging Face.
+
+"""
+
+from typing import Optional, Union
+
+from transformers import PretrainedConfig
+
+
+class CaduceusConfig(PretrainedConfig):
+    """Config that extends the original MambaConfig with params relevant to bi-directionality and RC equivariance."""
+    model_type = "caduceus"
+
+    def __init__(
+            self,
+            # From original MambaConfig
+            d_model: int = 2560,
+            n_layer: int = 64,
+            vocab_size: int = 50277,
+            ssm_cfg: Optional[dict] = None,
+            rms_norm: bool = True,
+            residual_in_fp32: bool = True,
+            fused_add_norm: bool = True,
+            pad_vocab_size_multiple: int = 8,
+
+            # Not in original MambaConfig, but default arg in create_block in mamba_ssm repo; used in layer norm
+            norm_epsilon: float = 1e-5,
+
+            # Used in init_weights
+            initializer_cfg: Optional[dict] = None,
+
+            # Caduceus-specific params
+            bidirectional: bool = True,
+            bidirectional_strategy: Union[str, None] = "add",
+            bidirectional_weight_tie: bool = True,
+            rcps: bool = False,
+            complement_map: Optional[dict] = None,  # used for RCPSEmbedding / RCPSLMHead
+            **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.d_model = d_model
+        self.n_layer = n_layer
+        self.vocab_size = vocab_size
+        self.ssm_cfg = ssm_cfg
+        self.rms_norm = rms_norm
+        self.residual_in_fp32 = residual_in_fp32
+        self.fused_add_norm = fused_add_norm
+        self.pad_vocab_size_multiple = pad_vocab_size_multiple
+        self.norm_epsilon = norm_epsilon
+        self.initializer_cfg = initializer_cfg
+        self.bidirectional = bidirectional
+        self.bidirectional_strategy = bidirectional_strategy
+        self.bidirectional_weight_tie = bidirectional_weight_tie
+        self.rcps = rcps
+        self.complement_map = complement_map

modeling_caduceus.py

@@ -0,0 +1,664 @@
+"""Caduceus model for Hugging Face.
+
+"""
+
+import inspect
+import math
+from functools import partial
+from typing import Optional, Tuple, Union
+
+import torch
+from mamba_ssm.modules.mamba_simple import Mamba
+try:
+    from mamba_ssm.modules.mamba_simple import Block  # Legacy mambav1 file structure
+except ImportError:
+    from mamba_ssm.modules.block import Block  # mambav2 file structure
+from torch import nn
+from torch.nn import functional as F
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import BaseModelOutputWithNoAttention, MaskedLMOutput, SequenceClassifierOutput
+
+try:
+    from mamba_ssm.ops.triton.layernorm import RMSNorm, layer_norm_fn, rms_norm_fn  # Legacy mambav1 file structure
+except ImportError:
+    try:
+        from mamba_ssm.ops.triton.layer_norm import RMSNorm, layer_norm_fn, rms_norm_fn  # mambav2 file structure
+    except ImportError:
+        RMSNorm, layer_norm_fn, rms_norm_fn = None, None, None
+
+from .configuration_caduceus import CaduceusConfig
+from .modeling_rcps import RCPSAddNormWrapper, RCPSEmbedding, RCPSLMHead, RCPSMambaBlock
+
+
+def create_block(
+        d_model,
+        ssm_cfg=None,
+        norm_epsilon=1e-5,
+        rms_norm=False,
+        residual_in_fp32=False,
+        fused_add_norm=False,
+        layer_idx=None,
+        bidirectional=True,
+        bidirectional_strategy="add",
+        bidirectional_weight_tie=True,
+        rcps=False,
+        device=None,
+        dtype=None,
+):
+    """Create Caduceus block.
+
+    Adapted from: https://github.com/state-spaces/mamba/blob/main/mamba_ssm/models/mixer_seq_simple.py
+    """
+    if ssm_cfg is None:
+        ssm_cfg = {}
+    factory_kwargs = {"device": device, "dtype": dtype}
+    bidirectional_kwargs = {
+        "bidirectional": bidirectional,
+        "bidirectional_strategy": bidirectional_strategy,
+        "bidirectional_weight_tie": bidirectional_weight_tie,
+    }
+    mixer_cls = partial(BiMambaWrapper, layer_idx=layer_idx, **ssm_cfg, **bidirectional_kwargs, **factory_kwargs)
+    norm_cls = partial(
+        nn.LayerNorm if not rms_norm else RMSNorm, eps=norm_epsilon, **factory_kwargs
+    )
+    block_cls = RCPSMambaBlock if rcps else Block
+    # mambav2 compatibility
+    if "mlp_cls" in inspect.signature(block_cls.__init__).parameters:
+        block = block_cls(
+            d_model,
+            mixer_cls,
+            mlp_cls=nn.Identity,
+            norm_cls=norm_cls,
+            fused_add_norm=fused_add_norm,
+            residual_in_fp32=residual_in_fp32,
+        )
+    else:
+        block = block_cls(
+            d_model,
+            mixer_cls,
+            norm_cls=norm_cls,
+            fused_add_norm=fused_add_norm,
+            residual_in_fp32=residual_in_fp32,
+        )
+    block.layer_idx = layer_idx
+    return block
+
+
+class BiMambaWrapper(nn.Module):
+    """Thin wrapper around Mamba to support bi-directionality."""
+
+    def __init__(
+            self,
+            d_model: int,
+            bidirectional: bool = True,
+            bidirectional_strategy: Optional[str] = "add",
+            bidirectional_weight_tie: bool = True,
+            **mamba_kwargs,
+    ):
+        super().__init__()
+        if bidirectional and bidirectional_strategy is None:
+            bidirectional_strategy = "add"  # Default strategy: `add`
+        if bidirectional and bidirectional_strategy not in ["add", "ew_multiply"]:
+            raise NotImplementedError(f"`{bidirectional_strategy}` strategy for bi-directionality is not implemented!")
+        self.bidirectional = bidirectional
+        self.bidirectional_strategy = bidirectional_strategy
+        self.mamba_fwd = Mamba(
+            d_model=d_model,
+            **mamba_kwargs
+        )
+        if bidirectional:
+            self.mamba_rev = Mamba(
+                d_model=d_model,
+                **mamba_kwargs
+            )
+            if bidirectional_weight_tie:  # Tie in and out projections (where most of param count lies)
+                self.mamba_rev.in_proj.weight = self.mamba_fwd.in_proj.weight
+                self.mamba_rev.in_proj.bias = self.mamba_fwd.in_proj.bias
+                self.mamba_rev.out_proj.weight = self.mamba_fwd.out_proj.weight
+                self.mamba_rev.out_proj.bias = self.mamba_fwd.out_proj.bias
+        else:
+            self.mamba_rev = None
+
+    def forward(self, hidden_states, inference_params=None):
+        """Bidirectional-enabled forward pass
+
+        hidden_states: (B, L, D)
+        Returns: same shape as hidden_states
+        """
+        out = self.mamba_fwd(hidden_states, inference_params=inference_params)
+        if self.bidirectional:
+            out_rev = self.mamba_rev(
+                hidden_states.flip(dims=(1,)),  # Flip along the sequence length dimension
+                inference_params=inference_params
+            ).flip(dims=(1,))  # Flip back for combining with forward hidden states
+            if self.bidirectional_strategy == "add":
+                out = out + out_rev
+            elif self.bidirectional_strategy == "ew_multiply":
+                out = out * out_rev
+            else:
+                raise NotImplementedError(f"`{self.bidirectional_strategy}` for bi-directionality not implemented!")
+        return out
+
+
+class CaduceusEmbeddings(nn.Module):
+    def __init__(
+            self,
+            config: CaduceusConfig,
+            device=None,
+            dtype=None,
+    ):
+        super().__init__()
+        factory_kwargs = {"device": device, "dtype": dtype}
+        if config.rcps:
+            self.word_embeddings = RCPSEmbedding(
+                config.vocab_size, config.d_model, config.complement_map, **factory_kwargs
+            )
+        else:
+            self.word_embeddings = nn.Embedding(config.vocab_size, config.d_model, **factory_kwargs)
+
+    def forward(self, input_ids):
+        """
+            input_ids: (batch, seqlen)
+        """
+        return self.word_embeddings(input_ids)
+
+
+class CaduceusMixerModel(nn.Module):
+    def __init__(
+            self,
+            config: CaduceusConfig,
+            device=None,
+            dtype=None,
+    ) -> None:
+        super().__init__()
+        factory_kwargs = {"device": device, "dtype": dtype}
+
+        self.fused_add_norm = config.fused_add_norm
+        self.rcps = config.rcps
+        self.residual_in_fp32 = config.residual_in_fp32
+
+        self.embeddings = CaduceusEmbeddings(config, **factory_kwargs)
+
+        # Mamba changes the order of residual and layer norm:
+        # Instead of LN -> Attn / MLP -> Add, we do:
+        # Add -> LN -> Attn / MLP / Mixer, returning both the residual branch (output of Add) and
+        # the main branch (output of MLP / Mixer). The model definition is unchanged.
+        # This is for performance reason: we can fuse add + layer_norm.
+        if config.fused_add_norm:
+            if layer_norm_fn is None or rms_norm_fn is None:
+                raise ImportError("Failed to import Triton LayerNorm / RMSNorm kernels")
+
+        self.layers = nn.ModuleList(
+            [
+                create_block(
+                    config.d_model,
+                    ssm_cfg=config.ssm_cfg,
+                    norm_epsilon=config.norm_epsilon,
+                    rms_norm=config.rms_norm,
+                    residual_in_fp32=config.residual_in_fp32,
+                    fused_add_norm=config.fused_add_norm,
+                    layer_idx=i,
+                    bidirectional=config.bidirectional,
+                    bidirectional_strategy=config.bidirectional_strategy,
+                    bidirectional_weight_tie=config.bidirectional_weight_tie,
+                    rcps=config.rcps,
+                    **factory_kwargs,
+                )
+                for i in range(config.n_layer)
+            ]
+        )
+
+        norm_f = (nn.LayerNorm if not config.rms_norm else RMSNorm)(
+            config.d_model, eps=config.norm_epsilon, **factory_kwargs
+        )
+        self.norm_f = norm_f if (config.fused_add_norm or not config.rcps) else RCPSAddNormWrapper(norm_f)
+
+    def forward(self, input_ids, inputs_embeds=None, output_hidden_states=False):
+        """Mixer forward."""
+        all_hidden_states = []
+        if inputs_embeds is not None:
+            hidden_states = inputs_embeds
+        else:
+            hidden_states = self.embeddings(input_ids)
+
+        residual = None
+        for layer in self.layers:
+            if output_hidden_states:
+                all_hidden_states.append(hidden_states)
+            # TODO: Add support for gradient checkpointing
+            hidden_states, residual = layer(
+                hidden_states, residual, inference_params=None
+            )
+
+        if not self.fused_add_norm:
+            if self.rcps:
+                # Set prenorm=False here since we don't need the residual
+                hidden_states = self.norm_f(hidden_states, residual=residual, prenorm=False)
+            else:
+                residual = (hidden_states + residual) if residual is not None else hidden_states
+                hidden_states = self.norm_f(residual.to(dtype=self.norm_f.weight.dtype))
+        else:
+            fused_add_norm_fn = rms_norm_fn if isinstance(self.norm_f, RMSNorm) else layer_norm_fn
+            if self.rcps:
+                # Set prenorm=False here since we don't need the residual
+                hidden_states_fwd = fused_add_norm_fn(
+                    hidden_states[..., :hidden_states.shape[-1] // 2],
+                    self.norm_f.weight,
+                    self.norm_f.bias,
+                    eps=self.norm_f.eps,
+                    residual=residual[..., :hidden_states.shape[-1] // 2],
+                    prenorm=False,
+                    residual_in_fp32=self.residual_in_fp32,
+                )
+                hidden_states_rc = fused_add_norm_fn(
+                    hidden_states[..., hidden_states.shape[-1] // 2:].flip(dims=[-2, -1]),
+                    self.norm_f.weight,
+                    self.norm_f.bias,
+                    eps=self.norm_f.eps,
+                    residual=residual[..., hidden_states.shape[-1] // 2:].flip(dims=[-2, -1]),
+                    prenorm=False,
+                    residual_in_fp32=self.residual_in_fp32,
+                )
+                hidden_states = torch.cat([hidden_states_fwd, hidden_states_rc.flip(dims=[-2, -1])], dim=-1)
+            else:
+                # Set prenorm=False here since we don't need the residual
+                hidden_states = fused_add_norm_fn(
+                    hidden_states,
+                    self.norm_f.weight,
+                    self.norm_f.bias,
+                    eps=self.norm_f.eps,
+                    residual=residual,
+                    prenorm=False,
+                    residual_in_fp32=self.residual_in_fp32,
+                )
+            if output_hidden_states:
+                all_hidden_states.append(hidden_states)
+        return hidden_states, all_hidden_states
+
+
+def cross_entropy(logits, y, ignore_index=-100):
+    """Cross entropy loss."""
+    logits = logits.view(-1, logits.shape[-1])
+    y = y.view(-1)
+    return F.cross_entropy(logits, y, ignore_index=ignore_index)
+
+
+def weighted_cross_entropy(logits, y, loss_weights, ignore_index=-100):
+    """Weighted cross entropy loss (discounts certain tokens, e.g., repeated base pairs in genome)."""
+    logits = logits.view(-1, logits.shape[-1])
+    y = y.view(-1)
+    ce = F.cross_entropy(logits, y, ignore_index=ignore_index, reduction="none")
+    loss_weights = loss_weights.view(-1)
+    loss_weights[y == ignore_index] = 0.0
+    # TODO: Follows GPN implementation, but should we remove weight normalization?
+    return (ce * (loss_weights / loss_weights.sum())).sum()
+
+
+class CaduceusPreTrainedModel(PreTrainedModel):
+    """PreTrainedModel wrapper for Caduceus backbone."""
+    config_class = CaduceusConfig
+    base_model_prefix = "caduceus"
+    supports_gradient_checkpointing = False
+    _no_split_modules = ["BiMambaWrapper"]
+
+    def _init_weights(
+            self,
+            module,
+            initializer_range=0.02,  # Now only used for embedding layer.
+            **kwargs,
+    ):
+        """Adapted from: https://github.com/state-spaces/mamba/blob/main/mamba_ssm/models/mixer_seq_simple.py"""
+
+        n_layer = self.config.n_layer
+        initialized_cfg = self.config.initializer_cfg if self.config.initializer_cfg is not None else {}
+        rescale_prenorm_residual = initialized_cfg.get("rescale_prenorm_residual", True)
+        initializer_range = initialized_cfg.get("initializer_range", initializer_range)
+        n_residuals_per_layer = initialized_cfg.get("n_residuals_per_layer", 1)
+
+        if isinstance(module, nn.Linear):
+            if module.bias is not None:
+                if not getattr(module.bias, "_no_reinit", False):
+                    nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, std=initializer_range)
+
+        if rescale_prenorm_residual:
+            # Reinitialize selected weights subject to the OpenAI GPT-2 Paper Scheme:
+            #   > A modified initialization which accounts for the accumulation on the residual path with model depth.
+            #   > Scale the weights of residual layers at initialization by a factor of 1/√N where N is the # of
+            #   residual layers.
+            #   >   -- GPT-2 :: https://openai.com/blog/better-language-models/
+            #
+            # Reference (Megatron-LM): https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/model/gpt_model.py
+            for name, p in module.named_parameters():
+                if name in ["out_proj.weight", "fc2.weight"]:
+                    # Special Scaled Initialization --> There are 2 Layer Norms per Transformer Block
+                    # Following Pytorch init, except scale by 1/sqrt(2 * n_layer)
+                    # We need to reinit p since this code could be called multiple times
+                    # Having just p *= scale would repeatedly scale it down
+                    nn.init.kaiming_uniform_(p, a=math.sqrt(5))
+                    with torch.no_grad():
+                        p /= math.sqrt(n_residuals_per_layer * n_layer)
+
+
+class Caduceus(CaduceusPreTrainedModel):
+    """Caduceus model that can be instantiated using HF patterns."""
+    def __init__(self, config: CaduceusConfig, device=None, dtype=None, **kwargs):
+        super().__init__(config)
+
+        if config.rcps:
+            assert config.complement_map is not None, "Complement map must be provided for RCPS."
+
+        # Adjust vocab size and complement maps if vocab padding is set.
+        if config.vocab_size % config.pad_vocab_size_multiple != 0:
+            config.vocab_size += config.pad_vocab_size_multiple - (config.vocab_size % config.pad_vocab_size_multiple)
+        if config.complement_map is not None and config.vocab_size > len(config.complement_map):
+            for i in range(len(config.complement_map), config.vocab_size):
+                config.complement_map[i] = i
+
+        self.config = config
+        factory_kwargs = {"device": device, "dtype": dtype}
+        self.backbone = CaduceusMixerModel(config, **factory_kwargs, **kwargs)
+
+    def maybe_weight_tie_mamba(self):
+        if getattr(self.config, 'bidirectional', False) and getattr(self.config, 'bidirectional_weight_tie', False):
+            if getattr(self.config, 'rcps', False):
+                for layer in self.backbone.layers:                    
+                    layer.mixer.submodule.mamba_rev.in_proj.weight = layer.mixer.submodule.mamba_fwd.in_proj.weight
+                    layer.mixer.submodule.mamba_rev.in_proj.bias = layer.mixer.submodule.mamba_fwd.in_proj.bias
+                    layer.mixer.submodule.mamba_rev.out_proj.weight = layer.mixer.submodule.mamba_fwd.out_proj.weight
+                    layer.mixer.submodule.mamba_rev.out_proj.bias = layer.mixer.submodule.mamba_fwd.out_proj.bias
+            else:
+                for layer in self.backbone.layers:                    
+                    layer.mixer.mamba_rev.in_proj.weight = layer.mixer.mamba_fwd.in_proj.weight
+                    layer.mixer.mamba_rev.in_proj.bias = layer.mixer.mamba_fwd.in_proj.bias
+                    layer.mixer.mamba_rev.out_proj.weight = layer.mixer.mamba_fwd.out_proj.weight
+                    layer.mixer.mamba_rev.out_proj.bias = layer.mixer.mamba_fwd.out_proj.bias
+
+    def tie_weights(self):
+        self.maybe_weight_tie_mamba()
+
+    def forward(
+            self,
+            input_ids: torch.LongTensor = None,
+            inputs_embeds: Optional[torch.FloatTensor] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[torch.Tensor, Tuple, BaseModelOutputWithNoAttention]:
+        """HF-compatible forward method."""
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        hidden_states, all_hidden_states = self.backbone(
+            input_ids,
+            inputs_embeds=inputs_embeds,
+            output_hidden_states=output_hidden_states
+        )
+        if return_dict:
+            return BaseModelOutputWithNoAttention(
+                last_hidden_state=hidden_states,
+                hidden_states=all_hidden_states if output_hidden_states else None
+            )
+        elif output_hidden_states:
+            return hidden_states, all_hidden_states
+        else:
+            return hidden_states
+
+
+class CaduceusForMaskedLM(CaduceusPreTrainedModel):
+    """HF-compatible Caduceus model for masked language modeling."""
+
+    def __init__(self, config: CaduceusConfig, device=None, dtype=None, **kwargs):
+        super().__init__(config, **kwargs)
+        factory_kwargs = {"device": device, "dtype": dtype}
+        self.caduceus = Caduceus(config, **factory_kwargs, **kwargs)
+        if config.rcps:
+            self.lm_head = RCPSLMHead(
+                complement_map=self.config.complement_map,  # Use caduceus config as it might have been updated
+                vocab_size=self.config.vocab_size,  # Use caduceus config as it might have been updated
+                true_dim=config.d_model,
+                dtype=dtype
+            )
+        else:
+            self.lm_head = nn.Linear(
+                config.d_model,
+                self.config.vocab_size,  # Use caduceus config as it might have been updated
+                bias=False,
+                **factory_kwargs
+            )
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.caduceus.backbone.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        if self.config.rcps:
+            raise NotImplementedError("Setting input embeddings for RCPS LM is not supported.")
+        self.caduceus.backbone.embeddings.word_embeddings = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        """Overrides output embeddings."""
+        if self.config.rcps:
+            raise NotImplementedError("Setting output embeddings for RCPS LM is not supported.")
+        self.lm_head = new_embeddings
+
+    def maybe_weight_tie_mamba(self):
+        self.caduceus.maybe_weight_tie_mamba()
+    
+    def tie_weights(self):
+        """Tie weights, accounting for RCPS."""
+        self.maybe_weight_tie_mamba()
+        if self.config.rcps:
+            self.lm_head.set_weight(self.get_input_embeddings().weight)
+        else:
+            super().tie_weights()
+
+    def get_decoder(self):
+        """Get decoder (backbone) for the model."""
+        return self.caduceus
+
+    def set_decoder(self, decoder):
+        """Set decoder (backbone) for the model."""
+        self.caduceus = decoder
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        loss_weights: Optional[torch.FloatTensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, MaskedLMOutput]:
+        """HF-compatible forward method."""
+
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.caduceus(
+            input_ids=input_ids,
+            inputs_embeds=inputs_embeds,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        logits = self.lm_head(hidden_states)
+        logits = logits.float()
+
+        loss = None
+        if labels is not None:
+            if loss_weights is not None:
+                loss = weighted_cross_entropy(logits, labels, loss_weights, ignore_index=self.config.pad_token_id)
+            else:
+                loss = cross_entropy(logits, labels, ignore_index=self.config.pad_token_id)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return MaskedLMOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+        )
+
+
+class CaduceusForSequenceClassification(CaduceusPreTrainedModel):
+    def __init__(
+            self,
+            config: CaduceusConfig,
+            pooling_strategy: str = "mean",
+            conjoin_train: bool = False,
+            conjoin_eval: bool = False,
+            device=None,
+            dtype=None,
+            **kwargs):
+        super().__init__(config, **kwargs)
+        if pooling_strategy not in ["mean", "max", "first", "last"]:
+            raise NotImplementedError(f"Pooling strategy `{pooling_strategy}` not implemented.")
+        self.pooling_strategy = pooling_strategy
+        factory_kwargs = {"device": device, "dtype": dtype}
+        self.num_labels = kwargs.get("num_labels", config.num_labels)
+        self.caduceus = Caduceus(config, **factory_kwargs, **kwargs)
+        self.score = nn.Linear(config.d_model, self.num_labels, bias=False)
+
+        self.conjoin_train = conjoin_train
+        self.conjoin_eval = conjoin_eval
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.caduceus.backbone.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        if self.config.rcps:
+            raise NotImplementedError("Setting input embeddings for RCPS LM is not supported.")
+        self.caduceus.backbone.embeddings.word_embeddings = value
+
+    def pool_hidden_states(self, hidden_states, sequence_length_dim=1):
+        """Pools hidden states along sequence length dimension."""
+        if self.pooling_strategy == "mean":  # Mean pooling along sequence length dimension
+            return hidden_states.mean(dim=sequence_length_dim)
+        if self.pooling_strategy == "max":  # Max pooling along sequence length dimension
+            return hidden_states.max(dim=sequence_length_dim).values
+        if self.pooling_strategy == "last":  # Use embedding of last token in the sequence
+            return hidden_states.moveaxis(hidden_states, sequence_length_dim, 0)[-1, ...]
+        if self.pooling_strategy == "first":  # Use embedding of first token in the sequence
+            return hidden_states.moveaxis(hidden_states, sequence_length_dim, 0)[0, ...]
+
+    def maybe_weight_tie_mamba(self):
+        self.caduceus.maybe_weight_tie_mamba()
+
+    def tie_weights(self):
+        self.maybe_weight_tie_mamba()
+        super().tie_weights()    
+    
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> Union[Tuple, SequenceClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # Get hidden representations from the backbone
+        if self.config.rcps:  # Hidden states have 2 * d_model channels for RCPS
+            transformer_outputs = self.caduceus(
+                input_ids,
+                inputs_embeds=inputs_embeds,
+                output_hidden_states=output_hidden_states,
+                return_dict=return_dict,
+            )
+            hidden_states = torch.stack(
+                [
+                    transformer_outputs[0][..., :self.config.d_model],
+                    torch.flip(transformer_outputs[0][..., self.config.d_model:], dims=[1, 2])
+                 ],
+                dim=-1
+            )
+        elif self.conjoin_train or (self.conjoin_eval and not self.training):  # For conjoining / post-hoc conjoining
+            assert input_ids is not None, "`input_ids` must be provided for conjoining."
+            assert input_ids.ndim == 3, "`input_ids` must be 3D tensor: channels corresponds to forward and rc strands."
+            transformer_outputs = self.caduceus(
+                input_ids[..., 0],
+                inputs_embeds=None,
+                output_hidden_states=output_hidden_states,
+                return_dict=return_dict,
+            )
+            transformer_outputs_rc = self.caduceus(
+                input_ids[..., 1],
+                inputs_embeds=None,
+                output_hidden_states=output_hidden_states,
+                return_dict=return_dict,
+            )
+            # Stack along channel dimension (dim=-1)
+            hidden_states = torch.stack([transformer_outputs[0], transformer_outputs_rc[0]], dim=-1)
+        else:
+            transformer_outputs = self.caduceus(
+                input_ids,
+                inputs_embeds=None,
+                output_hidden_states=output_hidden_states,
+                return_dict=return_dict,
+            )
+            hidden_states = transformer_outputs[0]
+
+        # Pool and get logits
+        pooled_hidden_states = self.pool_hidden_states(hidden_states)
+        # Potentially run `score` twice (with parameters shared) for conjoining
+        if hidden_states.ndim == 4:  # bsz, seq_len, hidden_dim, 2 where last channel has the stacked fwd and rc reps
+            logits_fwd = self.score(pooled_hidden_states[..., 0])
+            logits_rc = self.score(pooled_hidden_states[..., 1])
+            logits = (logits_fwd + logits_rc) / 2
+        else:
+            logits = self.score(pooled_hidden_states)
+
+        loss = None
+        if labels is not None:
+            labels = labels.to(logits.device)
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = "regression"
+                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
+                    self.config.problem_type = "single_label_classification"
+                else:
+                    self.config.problem_type = "multi_label_classification"
+
+            if self.config.problem_type == "regression":
+                if self.num_labels == 1:
+                    loss = F.mse_loss(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = F.mse_loss(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss = F.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss = F.binary_cross_entropy_with_logits(logits, labels)
+        if not return_dict:
+            output = (logits,) + transformer_outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=transformer_outputs.hidden_states,
+        )

modeling_rcps.py

@@ -0,0 +1,246 @@
+"""Reverse-complement equivariant modules.
+
+"""
+from collections import OrderedDict
+from typing import Optional
+
+import torch
+from torch import Tensor
+from torch import nn
+from torch.nn import functional as F
+
+try:
+    from mamba_ssm.ops.triton.layernorm import RMSNorm, layer_norm_fn, rms_norm_fn  # Legacy mambav1 file structure
+except ImportError:
+    try:
+        from mamba_ssm.ops.triton.layer_norm import RMSNorm, layer_norm_fn, rms_norm_fn  # mambav2 file structure
+    except ImportError:
+        RMSNorm, layer_norm_fn, rms_norm_fn = None, None, None
+
+
+class RCPSEmbedding(nn.Module):
+    """Embedding layer that supports reverse-complement equivariance."""
+    def __init__(self, vocab_size: int, d_model: int, complement_map: dict, **factory_kwargs):
+        """
+        Args:
+            vocab_size: Size of vocabulary.
+            d_model: Dimensionality of embedding (actual embedding matrix will have 1/2 the output dim).
+            complement_map: Dictionary mapping each token id to its complement.
+        """
+        super().__init__()
+        self.register_buffer(
+            "complement_map",
+            torch.tensor(list(OrderedDict(complement_map).values()), dtype=torch.long)
+        )
+        self.embedding = nn.Embedding(vocab_size, d_model, **factory_kwargs)
+
+    @property
+    def weight(self):
+        """Embedding weights."""
+        return self.embedding.weight
+
+    def set_weight(self, value):
+        """Set embedding weights."""
+        self.embedding.weight = value
+
+    def rc(self, x):
+        """Reverse-complement a tensor of input_ids by flipping along length dimension and complementing the ids."""
+        return torch.gather(
+            self.complement_map.unsqueeze(0).expand(x.shape[0], -1),
+            dim=1,
+            index=torch.flip(x, dims=[-1])
+        )
+
+    def forward(self, input_ids):
+        """Reverse-complement equivariant forward pass.
+
+        This embedding module doubles the output dimensionality to support reverse-complement equivariance.
+
+        Args:
+            input_ids: Input tensor of shape (batch_size, seq_len)
+        Returns:
+            Embedding tensor of shape (batch_size, seq_len, d_model * 2)
+        """
+        fwd_out = self.embedding(input_ids)
+        rc_out = torch.flip(self.embedding(self.rc(input_ids)), dims=[-2, -1])
+
+        return torch.cat([fwd_out, rc_out], dim=-1)
+
+
+class RCPSWrapper(nn.Module):
+    """Wrapper to convert arbitrary nn.Module into a reverse-complement equivariant module.
+
+    See ref. "Towards a Better Understanding of Reverse-Complement Equivariance for Deep Learning Models in Regulatory
+    Genomics", Zhou et al. (2022), https://proceedings.mlr.press/v165/zhou22a.html for more details.
+    """
+    def __init__(self, submodule: nn.Module):
+        super().__init__()
+        self.submodule = submodule
+
+    @staticmethod
+    def rc(x):
+        """Reverse-complement a tensor by flipping the length (dim=-2) and channel (dim=-1) dimensions."""
+        return torch.flip(x, dims=[-2, -1])
+
+    def forward(self, x, **kwargs):
+        """Reverse-complement equivariant forward pass.
+
+        Args:
+            x: Input tensor of shape (batch_size, seq_len, channels)
+        Returns:
+            Output tensor of shape (batch_size, seq_len, channels * 2)
+        """
+        n_channels = x.shape[-1]
+        # Run submodule along sequence
+        fwd_out = self.submodule(x[..., :n_channels // 2], **kwargs)
+        # Run submodule along rc-sequence
+        rc_out = self.submodule(self.rc(x[..., n_channels // 2:]), **kwargs)
+        # Concatenate along channel dimension (dim=-1)
+        return torch.cat([fwd_out, self.rc(rc_out)], dim=-1)
+
+
+class RCPSAddNormWrapper(RCPSWrapper):
+    """RC equivariant AddNorm layer."""
+    def __init__(self, submodule: nn.Module):
+        super().__init__(submodule)
+
+    def forward(self, x, residual=None, prenorm=False):
+        """
+        Args:
+            x: Input tensor of shape (batch_size, seq_len, channels)
+            residual: Residual tensor of shape (batch_size, seq_len, channels) or None.
+            prenorm: Whether to return residual.
+        """
+        n_channels = x.shape[-1]
+        if residual is None:
+            residual = x
+            x_fwd = self.submodule(x[..., :n_channels // 2].to(dtype=self.submodule.weight.dtype))
+            x_rc = self.submodule(self.rc(x[..., n_channels // 2:]).to(dtype=self.submodule.weight.dtype))
+            x = torch.cat([x_fwd, self.rc(x_rc)], dim=-1)
+        else:
+            residual_fwd = x[..., :n_channels // 2] + residual[..., :n_channels // 2]
+            x_fwd = self.submodule(residual_fwd.to(dtype=self.submodule.weight.dtype))
+
+            residual_rc = self.rc(x[..., n_channels // 2:]) + self.rc(residual[..., n_channels // 2:])
+            x_rc = self.submodule(residual_rc.to(dtype=self.submodule.weight.dtype))
+
+            residual = torch.cat([residual_fwd, self.rc(residual_rc)], dim=-1)
+            x = torch.cat([x_fwd, self.rc(x_rc)], dim=-1)
+
+        return x if not prenorm else (x, residual)
+
+
+class RCPSMambaBlock(nn.Module):
+    def __init__(
+            self,
+            dim,
+            mixer_cls,
+            norm_cls=nn.LayerNorm,
+            fused_add_norm=False,
+            residual_in_fp32=False,
+            device=None,  # Keep for consistency with original Mamba Block
+            dtype=None,  # Keep for consistency with original Mamba Block
+    ):
+        """RCPS version of simple block wrapping a mixer class with LayerNorm/RMSNorm and residual connection.
+
+        Adapted from: https://github.com/state-spaces/mamba/blob/main/mamba_ssm/modules/mamba_simple.py
+        """
+        super().__init__()
+        self.residual_in_fp32 = residual_in_fp32
+        self.fused_add_norm = fused_add_norm
+        self.mixer = RCPSWrapper(mixer_cls(dim))
+        norm_f = norm_cls(dim)
+        self.norm = norm_f if fused_add_norm else RCPSAddNormWrapper(norm_f)
+        if self.fused_add_norm:
+            assert RMSNorm is not None, "RMSNorm import fails"
+            assert isinstance(
+                self.norm, (nn.LayerNorm, RMSNorm)
+            ), "Only LayerNorm and RMSNorm are supported for fused_add_norm"
+
+    def forward(
+        self, hidden_states: Tensor, residual: Optional[Tensor] = None, inference_params=None
+    ):
+        r"""Pass the input through the encoder layer.
+
+        Args:
+            hidden_states: the sequence to the encoder layer (required).
+            residual: hidden_states = Mixer(LN(residual)).
+            inference_params: inference parameters for mixer.
+        """
+        if not self.fused_add_norm:
+            hidden_states, residual = self.norm(hidden_states, residual=residual, prenorm=True)
+            if self.residual_in_fp32:
+                residual = residual.to(torch.float32)
+        else:
+            fused_add_norm_fn = rms_norm_fn if isinstance(self.norm, RMSNorm) else layer_norm_fn
+
+            hidden_states_fwd, residual_fwd = fused_add_norm_fn(
+                hidden_states[..., hidden_states.shape[-1] // 2:],
+                self.norm.weight,
+                self.norm.bias,
+                residual=residual[..., hidden_states.shape[-1] // 2:] if residual is not None else None,
+                prenorm=True,
+                residual_in_fp32=self.residual_in_fp32,
+                eps=self.norm.eps,
+            )
+
+            hidden_states_rc, residual_rc = fused_add_norm_fn(
+                hidden_states[..., :hidden_states.shape[-1] // 2].flip(dims=[-2, -1]),
+                self.norm.weight,
+                self.norm.bias,
+                residual=residual[..., :hidden_states.shape[-1] // 2].flip(dims=[-2, -1]) if residual is not None else None,
+                prenorm=True,
+                residual_in_fp32=self.residual_in_fp32,
+                eps=self.norm.eps,
+            )
+            hidden_states = torch.cat([hidden_states_fwd, hidden_states_rc.flip(dims=[-2, -1])], dim=-1)
+            residual = torch.cat([residual_fwd, residual_rc.flip(dims=[-2, -1])], dim=-1)
+        hidden_states = self.mixer(hidden_states, inference_params=inference_params)
+        return hidden_states, residual
+
+    def allocate_inference_cache(self, batch_size, max_seqlen, dtype=None, **kwargs):
+        """Allocate inference cache for mixer.
+
+        Keep for compatibility with original Mamba Block.
+        """
+        return self.mixer.allocate_inference_cache(batch_size, max_seqlen, dtype=dtype, **kwargs)
+
+
+class RCPSLMHead(nn.Module):
+    """LM Head for reverse-complement equivariant inputs, which have dim * 2 relative to standard inputs."""
+    def __init__(self, true_dim: int, vocab_size: int, complement_map: dict, **factory_kwargs):
+        """
+        `true_dim` corresponds to the actual dimensionality of the input were it not reverse-complement
+        equivariant, i.e. 0.5 times the actual input dim.
+        """
+        super().__init__()
+        self.register_buffer(
+            "complement_map",
+            torch.tensor(list(OrderedDict(complement_map).values()), dtype=torch.long)
+        )
+        self.true_dim = true_dim
+        self.lm_head = nn.Linear(true_dim, vocab_size, bias=False, **factory_kwargs)
+
+    @property
+    def weight(self):
+        """LM head weights."""
+        return self.lm_head.weight
+
+    def set_weight(self, value):
+        """Set LM head weights."""
+        self.lm_head.weight = value
+
+    def forward(self, x):
+        """
+        Args:
+            x: Input tensor of shape (batch_size, seq_len, dim), where dim = 2 * true_dim.
+        """
+        n_channels = x.shape[-1]
+        assert n_channels == 2 * self.true_dim, "Input must have 2 * true_dim channels."
+        fwd_logits = F.linear(x[..., :n_channels // 2], self.weight, bias=self.lm_head.bias)
+        rc_logits = F.linear(
+            torch.flip(x[..., n_channels // 2:], dims=[-1]),
+            self.weight[self.complement_map, :],
+            bias=self.lm_head.bias
+        )
+        return fwd_logits + rc_logits

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68c3d0a6dfd644e9c3d2568040801a5d4acb8e14d1ef442c1e4febe1cb1c4eba
+size 448611530

requirements.txt

@@ -0,0 +1,5 @@
+torch
+transformers
+tokenizers
+mamba-ssm
+causal-conv1d

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenization_caduceus.py

@@ -0,0 +1,135 @@
+"""Character tokenizer for Hugging Face.
+
+"""
+
+from typing import List, Optional, Dict, Sequence, Tuple
+
+from transformers import PreTrainedTokenizer
+
+
+class CaduceusTokenizer(PreTrainedTokenizer):
+    model_input_names = ["input_ids"]
+
+    def __init__(self,
+                 model_max_length: int,
+                 characters: Sequence[str] = ("A", "C", "G", "T", "N"),
+                 complement_map=None,
+                 bos_token="[BOS]",
+                 eos_token="[SEP]",
+                 sep_token="[SEP]",
+                 cls_token="[CLS]",
+                 pad_token="[PAD]",
+                 mask_token="[MASK]",
+                 unk_token="[UNK]",
+                 **kwargs):
+        """Character tokenizer for Hugging Face transformers.
+
+        Adapted from https://huggingface.co/LongSafari/hyenadna-tiny-1k-seqlen-hf/blob/main/tokenization_hyena.py
+        Args:
+            model_max_length (int): Model maximum sequence length.
+            characters (Sequence[str]): List of desired characters. Any character which
+                is not included in this list will be replaced by a special token called
+                [UNK] with id=6. Following is a list of the special tokens with
+                their corresponding ids:
+                    "[CLS]": 0
+                    "[SEP]": 1
+                    "[BOS]": 2
+                    "[MASK]": 3
+                    "[PAD]": 4
+                    "[RESERVED]": 5
+                    "[UNK]": 6
+                an id (starting at 7) will be assigned to each character.
+            complement_map (Optional[Dict[str, str]]): Dictionary with string complements for each character.
+        """
+        if complement_map is None:
+            complement_map = {"A": "T", "C": "G", "G": "C", "T": "A"}
+        self.characters = characters
+        self.model_max_length = model_max_length
+
+        self._vocab_str_to_int = {
+            "[CLS]": 0,
+            "[SEP]": 1,
+            "[BOS]": 2,
+            "[MASK]": 3,
+            "[PAD]": 4,
+            "[RESERVED]": 5,
+            "[UNK]": 6,
+            **{ch: i + 7 for i, ch in enumerate(self.characters)},
+        }
+        self._vocab_int_to_str = {v: k for k, v in self._vocab_str_to_int.items()}
+        add_prefix_space = kwargs.pop("add_prefix_space", False)
+        padding_side = kwargs.pop("padding_side", "left")
+
+        self._complement_map = {}
+        for k, v in self._vocab_str_to_int.items():
+            complement_id = self._vocab_str_to_int[complement_map[k]] if k in complement_map.keys() else v
+            self._complement_map[self._vocab_str_to_int[k]] = complement_id
+
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            sep_token=sep_token,
+            cls_token=cls_token,
+            pad_token=pad_token,
+            mask_token=mask_token,
+            unk_token=unk_token,
+            add_prefix_space=add_prefix_space,
+            model_max_length=model_max_length,
+            padding_side=padding_side,
+            **kwargs,
+        )
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self._vocab_str_to_int)
+
+    @property
+    def complement_map(self) -> Dict[int, int]:
+        return self._complement_map
+
+    def _tokenize(self, text: str, **kwargs) -> List[str]:
+        return list(text.upper())  # Convert all base pairs to uppercase
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return self._vocab_str_to_int.get(token, self._vocab_str_to_int["[UNK]"])
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return self._vocab_int_to_str[index]
+
+    def convert_tokens_to_string(self, tokens):
+        return "".join(tokens)  # Note: this operation has lost info about which base pairs were originally lowercase
+
+    def get_special_tokens_mask(
+        self,
+        token_ids_0: List[int],
+        token_ids_1: Optional[List[int]] = None,
+        already_has_special_tokens: bool = False,
+    ) -> List[int]:
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=True,
+            )
+
+        result = ([0] * len(token_ids_0)) + [1]
+        if token_ids_1 is not None:
+            result += ([0] * len(token_ids_1)) + [1]
+        return result
+
+    def build_inputs_with_special_tokens(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        sep = [self.sep_token_id]
+        # cls = [self.cls_token_id]
+        result = token_ids_0 + sep
+        if token_ids_1 is not None:
+            result += token_ids_1 + sep
+        return result
+
+    def get_vocab(self) -> Dict[str, int]:
+        return self._vocab_str_to_int
+
+    # Fixed vocabulary with no vocab file
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple:
+        return ()

tokenizer.json

@@ -0,0 +1,62 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "[PAD]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "[MASK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "[UNK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": {
+    "type": "Lowercase"
+  },
+  "pre_tokenizer": {
+    "type": "Whitespace"
+  },
+  "post_processor": null,
+  "decoder": null,
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": "[UNK]",
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "[PAD]": 0,
+      "[MASK]": 1,
+      "[UNK]": 2,
+      "a": 3,
+      "c": 4,
+      "g": 5,
+      "t": 6
+    },
+    "merges": []
+  }
+}

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}