Upload folder using huggingface_hub

README.md

@@ -0,0 +1,181 @@
+---
+language:
+- rna
+library_name: transformers
+tags:
+- RNA
+- mRNA
+- bert
+- language-model
+- flash-attention
+license: apache-2.0
+---
+
+# mRNABERT
+
+An updated HuggingFace implementation of [mRNABERT](https://huggingface.co/YYLY66/mRNABERT)
+(Xiong et al., Nature Communications 2025) with the three bugs fixed from
+[MosaicBERT-updated](https://huggingface.co/Taykhoom/MosaicBERT-updated) and full
+`attn_implementation` dispatch support (`eager`, `sdpa`, `flash_attention_2`).
+
+mRNABERT is a language model pre-trained on 18 million mRNA sequences incorporating
+contrastive learning to integrate semantic features of amino acids.
+
+## Architecture
+
+mRNABERT uses the MosaicBERT architecture with an mRNA-specific vocabulary.
+
+| Parameter | Value |
+|---|---|
+| Layers | 12 |
+| Attention heads | 12 |
+| Embedding dimension | 768 |
+| Vocabulary size | 74 (5 special + 5 single-nt + 64 codons) |
+| Positional encoding | ALiBi (no position embeddings) |
+| Attention | Flash Attention (packed QKV) |
+| FFN | Gated Linear Units (GeGLU) |
+| Padding | Unpadding (tokens concatenated, no padding overhead) |
+| Max sequence length | 1024 tokens |
+| Parameters | ~114M |
+
+### Vocabulary
+
+The tokenizer uses `BertTokenizer` with a hybrid vocabulary:
+
+| Range | Tokens | Use |
+|---|---|---|
+| 0-4 | `[PAD]` `[UNK]` `[CLS]` `[SEP]` `[MASK]` | Special tokens |
+| 5-9 | `A` `T` `C` `G` `N` | Single nucleotides (UTR regions) |
+| 10-73 | `AAA` ... `GGG` | All 64 codons (CDS regions) |
+
+## Bugs Fixed
+
+This port applies the same three fixes as
+[Taykhoom/MosaicBERT-updated](https://huggingface.co/Taykhoom/MosaicBERT-updated):
+
+1. `attn_implementation` dispatch reads `config._attn_implementation` (underscore prefix) instead
+   of `config.attn_implementation` (no underscore, always `None`, silently fell back to eager).
+2. `extended_attention_mask` cast to `hidden_states.dtype` instead of `torch.float32`
+   (broke bfloat16 inference).
+3. `_supports_sdpa = True` and `_supports_flash_attn_2 = True` flags added to all model
+   classes so HF dispatch machinery activates correctly.
+
+## Parity Verification
+
+Hidden states verified max abs diff < 2.4e-05 at all 13 representation levels
+(embedding + 12 transformer layers) relative to the original implementation.
+Both models use `flash_attn_varlen_qkvpacked_func` in this environment; the small
+numerical differences are flash attention rounding, not a correctness issue.
+SDPA vs eager max diff = 1.81e-05. Verified on GPU with PyTorch 2.7 / CUDA 11.8.
+
+## Pretraining
+
+- **Objective:** Masked Language Modeling + contrastive learning (amino-acid semantic features)
+- **Data:** 18 million curated mRNA sequences
+- **Source checkpoint:** `pytorch_model.bin` from [YYLY66/mRNABERT](https://huggingface.co/YYLY66/mRNABERT)
+
+## Usage
+
+Sequences must be pre-tokenized with spaces: single nucleotides for UTR regions,
+three-letter codons for CDS regions.
+
+### Embedding generation
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True)
+model = AutoModel.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True)
+model.eval()
+
+# Space-separated: single nt for UTRs, codons for CDS
+sequences = [
+    "A T C G G A GGG CCC TTT AAA",   # mixed UTR + CDS
+    "ATG TTT CCC GAC TAA",            # CDS only
+]
+enc = tokenizer(sequences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    out = model(**enc)
+
+# Mean pooling over non-padding tokens
+mask = enc["attention_mask"].unsqueeze(-1).float()
+mean_emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1)  # (batch, 768)
+```
+
+### MLM logits
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True)
+model = AutoModelForMaskedLM.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True)
+model.eval()
+
+enc = tokenizer(["A T C G [MASK] CCC TTT"], return_tensors="pt")
+with torch.no_grad():
+    logits = model(**enc).logits  # (1, seq_len, 74)
+```
+
+### Attention implementation
+
+```python
+# SDPA (default on PyTorch >= 2.0)
+model = AutoModel.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True,
+                                   attn_implementation="sdpa")
+
+# Flash Attention 2 (requires: pip install flash-attn --no-build-isolation)
+model = AutoModel.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True,
+                                   attn_implementation="flash_attention_2")
+```
+
+### Fine-tuning
+
+```python
+from transformers import AutoModel
+import torch.nn as nn
+
+class mRNABERTClassifier(nn.Module):
+    def __init__(self, num_labels):
+        super().__init__()
+        self.encoder = AutoModel.from_pretrained("Taykhoom/mRNABERT", trust_remote_code=True)
+        self.head = nn.Linear(768, num_labels)
+
+    def forward(self, input_ids, attention_mask):
+        out = self.encoder(input_ids, attention_mask=attention_mask)
+        mask = attention_mask.unsqueeze(-1).float()
+        pooled = (out.last_hidden_state * mask).sum(1) / mask.sum(1)
+        return self.head(pooled)
+```
+
+## Implementation Notes
+
+The original implementation uses `flash_attn_varlen_qkvpacked_func`. This HF port
+adds `attn_implementation="sdpa"` and `attn_implementation="flash_attention_2"` support,
+which were not part of the original codebase.
+
+## Citation
+
+```bibtex
+@article{xiong2025_mrnabert,
+  title   = {{mRNABERT}: advancing {mRNA} sequence design with a universal language model and comprehensive dataset},
+  author  = {Xiong, Ying and Wang, Aowen and Kang, Yu and Shen, Chao and Hsieh, Chang-Yu and Hou, Tingjun},
+  journal = {Nature Communications},
+  volume  = {16},
+  number  = {1},
+  pages   = {10371},
+  year    = {2025},
+  doi     = {10.1038/s41467-025-65340-8}
+}
+```
+
+## Credits
+
+Original mRNABERT model and weights by Xiong et al. Source: [GitHub](https://github.com/yyly6/mRNABERT).
+The HF conversion code was authored primarily by [Claude Code](https://claude.ai/code)
+and reviewed manually by Taykhoom Dalal.
+
+## License
+
+Apache 2.0, following the original repository.

bert_layers.py

@@ -0,0 +1,783 @@
+# Copyright 2022 MosaicML Examples authors
+# SPDX-License-Identifier: Apache-2.0
+
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018-2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2022, Tri Dao.
+
+import copy
+import logging
+import math
+import warnings
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from einops import rearrange
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import (BaseModelOutputWithPooling,
+                                           MaskedLMOutput,
+                                           SequenceClassifierOutput)
+from transformers.models.bert.modeling_bert import BertPreTrainedModel
+
+from .bert_padding import (index_first_axis,
+                                            index_put_first_axis, pad_input,
+                                            unpad_input, unpad_input_only)
+from .configuration_bert import BertConfig
+
+try:
+    from flash_attn import flash_attn_varlen_qkvpacked_func
+except ImportError:
+    flash_attn_varlen_qkvpacked_func = None
+
+logger = logging.getLogger(__name__)
+
+
+class BertEmbeddings(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        self.word_embeddings = nn.Embedding(config.vocab_size,
+                                            config.hidden_size,
+                                            padding_idx=config.pad_token_id)
+        # ALiBi doesn't use position embeddings
+        self.token_type_embeddings = nn.Embedding(config.type_vocab_size,
+                                                  config.hidden_size)
+
+        self.LayerNorm = nn.LayerNorm(config.hidden_size,
+                                      eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.register_buffer('token_type_ids',
+                             torch.zeros(config.max_position_embeddings,
+                                         dtype=torch.long),
+                             persistent=False)
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        past_key_values_length: int = 0,
+    ) -> torch.Tensor:
+        if (input_ids is not None) == (inputs_embeds is not None):
+            raise ValueError('Must specify either input_ids or input_embeds!')
+        if input_ids is not None:
+            input_shape = input_ids.size()
+        else:
+            assert inputs_embeds is not None
+            input_shape = inputs_embeds.size()[:-1]
+
+        seq_length = input_shape[1]
+
+        if token_type_ids is None:
+            if hasattr(self, 'token_type_ids'):
+                assert isinstance(self.token_type_ids, torch.LongTensor)
+                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
+                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(
+                    input_shape[0], seq_length)
+                token_type_ids = buffered_token_type_ids_expanded
+            else:
+                token_type_ids = torch.zeros(input_shape,
+                                             dtype=torch.long,
+                                             device=self.word_embeddings.device)
+
+        if inputs_embeds is None:
+            inputs_embeds = self.word_embeddings(input_ids)
+        token_type_embeddings = self.token_type_embeddings(token_type_ids)
+
+        embeddings = inputs_embeds + token_type_embeddings
+        # no position embeddings -- ALiBi
+        embeddings = self.LayerNorm(embeddings)
+        embeddings = self.dropout(embeddings)
+        return embeddings
+
+
+class BertUnpadSelfAttention(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(
+                config, 'embedding_size'):
+            raise ValueError(
+                f'The hidden size ({config.hidden_size}) is not a multiple of the number of attention '
+                f'heads ({config.num_attention_heads})')
+
+        self.num_attention_heads = config.num_attention_heads
+        self.attention_head_size = int(config.hidden_size /
+                                       config.num_attention_heads)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size
+        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+        self.p_dropout = config.attention_probs_dropout_prob
+        self.Wqkv = nn.Linear(self.all_head_size, 3 * config.hidden_size)
+        # Read via HF's underscore convention (_attn_implementation is set by
+        # from_pretrained before __init__ when _supports_* flags are True).
+        self.attn_implementation = getattr(config, '_attn_implementation', 'eager')
+
+        if self.attn_implementation == 'flash_attention_2' and flash_attn_varlen_qkvpacked_func is None:
+            warnings.warn(
+                'flash-attn not installed; falling back to eager attention. '
+                'Install flash-attn to use flash_attention_2.'
+            )
+            self.attn_implementation = 'eager'
+
+    def forward(self, hidden_states: torch.Tensor, cu_seqlens: torch.Tensor,
+                max_seqlen_in_batch: int, indices: torch.Tensor,
+                attn_mask: torch.Tensor, bias: torch.Tensor,
+                alibi_slopes: Optional[torch.Tensor] = None,
+                return_attn_weights: bool = False) -> torch.Tensor:
+        qkv = self.Wqkv(hidden_states)  # (nnz, 3 * hidden)
+
+        # flash_attention_2: work on unpadded tokens directly, skip pad/unpad
+        if self.attn_implementation == 'flash_attention_2' and not return_attn_weights:
+            qkv = rearrange(qkv, 'nnz (t h d) -> nnz t h d', t=3,
+                            h=self.num_attention_heads)
+            orig_dtype = qkv.dtype
+            if orig_dtype not in (torch.float16, torch.bfloat16):
+                qkv = qkv.to(torch.bfloat16)
+            max_s_actual = int((cu_seqlens[1:] - cu_seqlens[:-1]).max())
+            attention = flash_attn_varlen_qkvpacked_func(
+                qkv,
+                cu_seqlens,
+                max_s_actual,
+                dropout_p=self.p_dropout if self.training else 0.0,
+                alibi_slopes=alibi_slopes,
+            ).to(orig_dtype)  # (nnz, H, D)
+            return rearrange(attention, 'nnz h d -> nnz (h d)')
+
+        # eager and sdpa: pad back to (B, T, 3, H, D), compute, then unpad
+        batch = cu_seqlens.shape[0] - 1
+        qkv = pad_input(qkv, indices, batch, max_seqlen_in_batch)
+        qkv = rearrange(qkv, 'b s (t h d) -> b s t h d', t=3,
+                        h=self.num_attention_heads)
+
+        if self.attn_implementation == 'sdpa' and not return_attn_weights:
+            q = qkv[:, :, 0].permute(0, 2, 1, 3)  # B H T D
+            k = qkv[:, :, 1].permute(0, 2, 1, 3)
+            v = qkv[:, :, 2].permute(0, 2, 1, 3)
+            attention = F.scaled_dot_product_attention(
+                q, k, v, attn_mask=bias,
+                dropout_p=self.p_dropout if self.training else 0.0,
+            ).permute(0, 2, 1, 3)  # B T H D
+            attention_probs = None
+        else:
+            # eager (also fallback when return_attn_weights=True)
+            q = qkv[:, :, 0, :, :].permute(0, 2, 1, 3)  # b h s d
+            k = qkv[:, :, 1, :, :].permute(0, 2, 3, 1)  # b h d s
+            v = qkv[:, :, 2, :, :].permute(0, 2, 1, 3)  # b h s d
+            attention_scores = torch.matmul(q, k) / math.sqrt(
+                self.attention_head_size)
+            attention_scores = attention_scores + bias
+            attention_probs = nn.functional.softmax(attention_scores, dim=-1)
+            attention_probs = self.dropout(attention_probs)
+            attention = torch.matmul(attention_probs, v).permute(0, 2, 1, 3)  # b s h d
+
+        # attn_mask is 1 for attend and 0 for don't
+        attention = unpad_input_only(attention, torch.squeeze(attn_mask) == 1)
+        out = rearrange(attention, 'nnz h d -> nnz (h d)')
+        if return_attn_weights:
+            return out, attention_probs
+        return out
+
+
+class BertSelfOutput(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.LayerNorm = nn.LayerNorm(config.hidden_size,
+                                      eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+    def forward(self, hidden_states: torch.Tensor,
+                input_tensor: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+        return hidden_states
+
+
+class BertUnpadAttention(nn.Module):
+    """Chains attention, Dropout, and LayerNorm for Mosaic BERT."""
+
+    def __init__(self, config):
+        super().__init__()
+        self.self = BertUnpadSelfAttention(config)
+        self.output = BertSelfOutput(config)
+
+    def forward(
+        self,
+        input_tensor: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        max_s: int,
+        subset_idx: Optional[torch.Tensor] = None,
+        indices: Optional[torch.Tensor] = None,
+        attn_mask: Optional[torch.Tensor] = None,
+        bias: Optional[torch.Tensor] = None,
+        alibi_slopes: Optional[torch.Tensor] = None,
+        return_attn_weights: bool = False,
+    ) -> torch.Tensor:
+        if return_attn_weights:
+            self_output, attn_probs = self.self(
+                input_tensor, cu_seqlens, max_s, indices, attn_mask, bias,
+                alibi_slopes=alibi_slopes, return_attn_weights=True)
+        else:
+            self_output = self.self(input_tensor, cu_seqlens, max_s, indices,
+                                    attn_mask, bias, alibi_slopes=alibi_slopes)
+            attn_probs = None
+        if subset_idx is not None:
+            output = self.output(index_first_axis(self_output, subset_idx),
+                                 index_first_axis(input_tensor, subset_idx))
+        else:
+            output = self.output(self_output, input_tensor)
+        if return_attn_weights:
+            return output, attn_probs
+        return output
+
+
+class BertGatedLinearUnitMLP(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.gated_layers = nn.Linear(config.hidden_size,
+                                      config.intermediate_size * 2,
+                                      bias=False)
+        self.act = nn.GELU(approximate='none')
+        self.wo = nn.Linear(config.intermediate_size, config.hidden_size)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.layernorm = nn.LayerNorm(config.hidden_size,
+                                      eps=config.layer_norm_eps)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        residual_connection = hidden_states
+        hidden_states = self.gated_layers(hidden_states)
+        gated = hidden_states[:, :self.config.intermediate_size]
+        non_gated = hidden_states[:, self.config.intermediate_size:]
+        hidden_states = self.act(gated) * non_gated
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.wo(hidden_states)
+        hidden_states = self.layernorm(hidden_states + residual_connection)
+        return hidden_states
+
+
+class BertLayer(nn.Module):
+
+    def __init__(self, config):
+        super(BertLayer, self).__init__()
+        self.attention = BertUnpadAttention(config)
+        self.mlp = BertGatedLinearUnitMLP(config)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        seqlen: int,
+        subset_idx: Optional[torch.Tensor] = None,
+        indices: Optional[torch.Tensor] = None,
+        attn_mask: Optional[torch.Tensor] = None,
+        bias: Optional[torch.Tensor] = None,
+        alibi_slopes: Optional[torch.Tensor] = None,
+        return_attn_weights: bool = False,
+    ) -> torch.Tensor:
+        if return_attn_weights:
+            attention_output, attn_probs = self.attention(
+                hidden_states, cu_seqlens, seqlen, subset_idx, indices,
+                attn_mask, bias, alibi_slopes=alibi_slopes, return_attn_weights=True)
+        else:
+            attention_output = self.attention(hidden_states, cu_seqlens, seqlen,
+                                              subset_idx, indices, attn_mask, bias,
+                                              alibi_slopes=alibi_slopes)
+            attn_probs = None
+        layer_output = self.mlp(attention_output)
+        if return_attn_weights:
+            return layer_output, attn_probs
+        return layer_output
+
+
+class BertEncoder(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        layer = BertLayer(config)
+        self.layer = nn.ModuleList(
+            [copy.deepcopy(layer) for _ in range(config.num_hidden_layers)])
+
+        self.num_attention_heads = config.num_attention_heads
+        # Read via HF's underscore convention.
+        self.attn_implementation = getattr(config, '_attn_implementation', 'eager')
+
+        self._current_alibi_size = int(config.alibi_starting_size)
+        self.alibi = torch.zeros(
+            (1, self.num_attention_heads, self._current_alibi_size,
+             self._current_alibi_size))
+        self.alibi_slopes = torch.zeros(self.num_attention_heads)
+        self.rebuild_alibi_tensor(size=config.alibi_starting_size)
+
+    def rebuild_alibi_tensor(self,
+                             size: int,
+                             device: Optional[Union[torch.device, str]] = None):
+        n_heads = self.num_attention_heads
+
+        def _get_alibi_head_slopes(n_heads: int) -> List[float]:
+
+            def get_slopes_power_of_2(n_heads: int) -> List[float]:
+                start = (2**(-2**-(math.log2(n_heads) - 3)))
+                ratio = start
+                return [start * ratio**i for i in range(n_heads)]
+
+            if math.log2(n_heads).is_integer():
+                return get_slopes_power_of_2(n_heads)
+
+            closest_power_of_2 = 2**math.floor(math.log2(n_heads))
+            slopes_a = get_slopes_power_of_2(closest_power_of_2)
+            slopes_b = _get_alibi_head_slopes(2 * closest_power_of_2)
+            slopes_b = slopes_b[0::2][:n_heads - closest_power_of_2]
+            return slopes_a + slopes_b
+
+        context_position = torch.arange(size, device=device)[:, None]
+        memory_position = torch.arange(size, device=device)[None, :]
+        relative_position = torch.abs(memory_position - context_position)
+        relative_position = relative_position.unsqueeze(0).expand(n_heads, -1, -1)
+        slopes = torch.Tensor(_get_alibi_head_slopes(n_heads)).to(device)
+        alibi = slopes.unsqueeze(1).unsqueeze(1) * -relative_position
+        alibi = alibi.unsqueeze(0)
+        assert alibi.shape == torch.Size([1, n_heads, size, size])
+
+        self._current_alibi_size = size
+        self.alibi = alibi
+        self.alibi_slopes = slopes
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.Tensor,
+        output_all_encoded_layers: Optional[bool] = True,
+        subset_mask: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+    ) -> Tuple[List[torch.Tensor], Optional[Tuple[torch.Tensor, ...]]]:
+
+        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
+        # Cast to match hidden_states dtype for SDPA/eager compatibility.
+        extended_attention_mask = extended_attention_mask.to(dtype=hidden_states.dtype)
+        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
+
+        attention_mask_bool = attention_mask.bool()
+        batch, seqlen = hidden_states.shape[:2]
+
+        # Capture padded embedding (B, T, D) before unpadding for HF
+        # hidden_states convention: index 0 = embedding, index i+1 = layer i.
+        padded_embedding = hidden_states
+
+        hidden_states, indices, cu_seqlens, _ = unpad_input(
+            hidden_states, attention_mask_bool)
+
+        if self._current_alibi_size < seqlen:
+            warnings.warn(
+                f'Increasing alibi size from {self._current_alibi_size} to {seqlen}'
+            )
+            self.rebuild_alibi_tensor(size=seqlen, device=hidden_states.device)
+        elif self.alibi.device != hidden_states.device:
+            self.alibi = self.alibi.to(hidden_states.device)
+            self.alibi_slopes = self.alibi_slopes.to(hidden_states.device)
+
+        # Cast ALiBi bias to match hidden_states dtype.
+        alibi_bias = self.alibi[:, :, :seqlen, :seqlen].to(dtype=hidden_states.dtype)
+        attn_bias = extended_attention_mask[:, :, :seqlen, :seqlen]
+        alibi_attn_mask = attn_bias + alibi_bias
+        alibi_slopes = (
+            self.alibi_slopes if self.attn_implementation == 'flash_attention_2'
+            else None
+        )
+
+        all_encoder_layers = []
+        all_attention_probs: List[torch.Tensor] = []
+        if subset_mask is None:
+            for layer_module in self.layer:
+                if output_attentions:
+                    hidden_states, attn_probs = layer_module(
+                        hidden_states, cu_seqlens, seqlen, None, indices,
+                        attn_mask=attention_mask, bias=alibi_attn_mask,
+                        alibi_slopes=alibi_slopes, return_attn_weights=True)
+                    all_attention_probs.append(attn_probs)
+                else:
+                    hidden_states = layer_module(hidden_states,
+                                                 cu_seqlens,
+                                                 seqlen,
+                                                 None,
+                                                 indices,
+                                                 attn_mask=attention_mask,
+                                                 bias=alibi_attn_mask,
+                                                 alibi_slopes=alibi_slopes)
+                if output_all_encoded_layers:
+                    all_encoder_layers.append(
+                        pad_input(hidden_states, indices, batch, seqlen))
+            hidden_states = pad_input(hidden_states, indices, batch, seqlen)
+        else:
+            for i in range(len(self.layer) - 1):
+                layer_module = self.layer[i]
+                hidden_states = layer_module(hidden_states,
+                                             cu_seqlens,
+                                             seqlen,
+                                             None,
+                                             indices,
+                                             attn_mask=attention_mask,
+                                             bias=alibi_attn_mask,
+                                             alibi_slopes=alibi_slopes)
+                if output_all_encoded_layers:
+                    all_encoder_layers.append(hidden_states)
+            subset_idx = torch.nonzero(subset_mask[attention_mask_bool],
+                                       as_tuple=False).flatten()
+            hidden_states = self.layer[-1](hidden_states,
+                                           cu_seqlens,
+                                           seqlen,
+                                           subset_idx=subset_idx,
+                                           indices=indices,
+                                           attn_mask=attention_mask,
+                                           bias=alibi_attn_mask,
+                                           alibi_slopes=alibi_slopes)
+
+        if not output_all_encoded_layers:
+            all_encoder_layers.append(hidden_states)
+        else:
+            # Prepend padded embedding as index 0 (HF convention).
+            all_encoder_layers.insert(0, padded_embedding)
+
+        attn_out = tuple(all_attention_probs) if output_attentions else None
+        return all_encoder_layers, attn_out
+
+
+class BertPooler(nn.Module):
+
+    def __init__(self, config):
+        super(BertPooler, self).__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.activation = nn.Tanh()
+
+    def forward(self,
+                hidden_states: torch.Tensor,
+                pool: Optional[bool] = True) -> torch.Tensor:
+        first_token_tensor = hidden_states[:, 0] if pool else hidden_states
+        pooled_output = self.dense(first_token_tensor)
+        pooled_output = self.activation(pooled_output)
+        return pooled_output
+
+
+class BertPredictionHeadTransform(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        if isinstance(config.hidden_act, str):
+            self.transform_act_fn = ACT2FN[config.hidden_act]
+        else:
+            self.transform_act_fn = config.hidden_act
+        self.LayerNorm = torch.nn.LayerNorm(config.hidden_size, eps=1e-12)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.transform_act_fn(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states)
+        return hidden_states
+
+
+class BertModel(BertPreTrainedModel):
+    config_class = BertConfig
+    _supports_sdpa = True
+    _supports_flash_attn_2 = True
+
+    def __init__(self, config, add_pooling_layer=True):
+        super(BertModel, self).__init__(config)
+        self.embeddings = BertEmbeddings(config)
+        self.encoder = BertEncoder(config)
+        self.pooler = BertPooler(config) if add_pooling_layer else None
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.embeddings.word_embeddings = value
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        token_type_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        output_all_encoded_layers: Optional[bool] = False,
+        masked_tokens_mask: Optional[torch.Tensor] = None,
+        output_hidden_states: bool = False,
+        output_attentions: bool = False,
+        **kwargs
+    ) -> BaseModelOutputWithPooling:
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+        if token_type_ids is None:
+            token_type_ids = torch.zeros_like(input_ids)
+
+        embedding_output = self.embeddings(input_ids, token_type_ids,
+                                           position_ids)
+
+        subset_mask = None
+        if masked_tokens_mask is not None:
+            first_col_mask = torch.zeros_like(masked_tokens_mask)
+            first_col_mask[:, 0] = True
+            subset_mask = masked_tokens_mask | first_col_mask
+
+        encoder_outputs, all_attentions = self.encoder(
+            embedding_output,
+            attention_mask,
+            output_all_encoded_layers=output_hidden_states,
+            subset_mask=subset_mask,
+            output_attentions=output_attentions)
+
+        if masked_tokens_mask is None:
+            sequence_output = encoder_outputs[-1]
+            pooled_output = self.pooler(
+                sequence_output) if self.pooler is not None else None
+        else:
+            attention_mask_bool = attention_mask.bool()
+            subset_idx = subset_mask[attention_mask_bool]
+            sequence_output = encoder_outputs[-1][
+                masked_tokens_mask[attention_mask_bool][subset_idx]]
+            if self.pooler is not None:
+                first_col_mask = torch.zeros_like(masked_tokens_mask)
+                first_col_mask[:, 0] = True
+                pool_input = encoder_outputs[-1][
+                    first_col_mask[attention_mask_bool][subset_idx]]
+                pooled_output = self.pooler(pool_input, pool=False)
+            else:
+                pooled_output = None
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=sequence_output,
+            pooler_output=pooled_output,
+            hidden_states=tuple(encoder_outputs) if output_hidden_states else None,
+            attentions=all_attentions,
+        )
+
+
+###################
+# Bert Heads
+###################
+class BertLMPredictionHead(nn.Module):
+
+    def __init__(self, config, bert_model_embedding_weights):
+        super().__init__()
+        self.transform = BertPredictionHeadTransform(config)
+        self.decoder = nn.Linear(bert_model_embedding_weights.size(1),
+                                 bert_model_embedding_weights.size(0))
+        self.decoder.weight = bert_model_embedding_weights
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.transform(hidden_states)
+        hidden_states = self.decoder(hidden_states)
+        return hidden_states
+
+
+class BertOnlyMLMHead(nn.Module):
+
+    def __init__(self, config, bert_model_embedding_weights):
+        super().__init__()
+        self.predictions = BertLMPredictionHead(config,
+                                                bert_model_embedding_weights)
+
+    def forward(self, sequence_output: torch.Tensor) -> torch.Tensor:
+        prediction_scores = self.predictions(sequence_output)
+        return prediction_scores
+
+
+class BertForMaskedLM(BertPreTrainedModel):
+    config_class = BertConfig
+    _supports_sdpa = True
+    _supports_flash_attn_2 = True
+
+    def __init__(self, config):
+        super().__init__(config)
+
+        if config.is_decoder:
+            warnings.warn(
+                'If you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for '
+                'bi-directional self-attention.')
+
+        self.bert = BertModel(config, add_pooling_layer=False)
+        self.cls = BertOnlyMLMHead(config,
+                                   self.bert.embeddings.word_embeddings.weight)
+        self.post_init()
+
+    def get_output_embeddings(self):
+        return self.cls.predictions.decoder
+
+    def set_output_embeddings(self, new_embeddings):
+        self.cls.predictions.decoder = new_embeddings
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], MaskedLMOutput]:
+        if (input_ids is not None) == (inputs_embeds is not None):
+            raise ValueError('Must specify either input_ids or input_embeds!')
+
+        if labels is None:
+            masked_tokens_mask = None
+        else:
+            masked_tokens_mask = labels > 0
+
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.bert(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            masked_tokens_mask=masked_tokens_mask,
+        )
+
+        sequence_output = outputs.last_hidden_state
+        prediction_scores = self.cls(sequence_output)
+
+        loss = None
+        if labels is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            masked_token_idx = torch.nonzero(labels.flatten() > 0,
+                                             as_tuple=False).flatten()
+            loss = loss_fct(prediction_scores,
+                            labels.flatten()[masked_token_idx])
+
+            assert input_ids is not None, 'Coding error; please open an issue'
+            batch, seqlen = input_ids.shape[:2]
+            prediction_scores = rearrange(index_put_first_axis(
+                prediction_scores, masked_token_idx, batch * seqlen),
+                                          '(b s) d -> b s d',
+                                          b=batch)
+
+        if not return_dict:
+            output = (prediction_scores,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return MaskedLMOutput(
+            loss=loss,
+            logits=prediction_scores,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(self, input_ids: torch.Tensor,
+                                      attention_mask: torch.Tensor,
+                                      **model_kwargs):
+        input_shape = input_ids.shape
+        effective_batch_size = input_shape[0]
+
+        if self.config.pad_token_id is None:
+            raise ValueError('The PAD token should be defined for generation')
+
+        attention_mask = torch.cat([
+            attention_mask,
+            attention_mask.new_zeros((attention_mask.shape[0], 1))
+        ], dim=-1)
+        dummy_token = torch.full((effective_batch_size, 1),
+                                 self.config.pad_token_id,
+                                 dtype=torch.long,
+                                 device=input_ids.device)
+        input_ids = torch.cat([input_ids, dummy_token], dim=1)
+
+        return {'input_ids': input_ids, 'attention_mask': attention_mask}
+
+
+class BertForSequenceClassification(BertPreTrainedModel):
+    config_class = BertConfig
+    _supports_sdpa = True
+    _supports_flash_attn_2 = True
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.config = config
+
+        self.bert = BertModel(config)
+        classifier_dropout = (config.classifier_dropout
+                              if config.classifier_dropout is not None else
+                              config.hidden_dropout_prob)
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], SequenceClassifierOutput]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.bert(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+        )
+
+        pooled_output = outputs.pooler_output
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+
+        loss = None
+        if labels is not None:
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = 'regression'
+                elif self.num_labels > 1 and (labels.dtype == torch.long or
+                                              labels.dtype == torch.int):
+                    self.config.problem_type = 'single_label_classification'
+                else:
+                    self.config.problem_type = 'multi_label_classification'
+
+            if self.config.problem_type == 'regression':
+                loss_fct = nn.MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == 'single_label_classification':
+                loss_fct = nn.CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == 'multi_label_classification':
+                loss_fct = nn.BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

bert_padding.py

@@ -0,0 +1,154 @@
+# Copyright 2022 MosaicML Examples authors
+# SPDX-License-Identifier: Apache-2.0
+
+# Adapted from https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/bert_padding.py
+# Which was adapted from https://github.com/mlcommons/training_results_v1.1/blob/main/NVIDIA/benchmarks/bert/implementations/pytorch/padding.py
+
+
+from typing import Tuple, cast
+
+import torch
+import torch.nn.functional as F
+from einops import rearrange, repeat
+
+
+class IndexFirstAxis(torch.autograd.Function):
+
+    @staticmethod
+    def forward(ctx, input: torch.Tensor,
+                indices: torch.Tensor) -> torch.Tensor:
+        """Get just the values of `input` which are at `indices`.
+
+        Arguments:
+            ctx: the autograd context object
+            input: (b, ...) 2+ dimensional tensor
+            indices: (num_idx) 1D tensor
+        """
+        ctx.save_for_backward(indices)
+        assert input.ndim >= 2
+        ctx.first_axis_dim, other_shape = input.shape[0], input.shape[
+            1:]  # type: ignore
+        second_dim = other_shape.numel(
+        )  # product of sizes of all but first dimension
+        # TD [2022-03-04] For some reason torch.gather is a bit faster than indexing.
+        return torch.gather(
+            rearrange(input, 'b ... -> b (...)'),  # (b, ...) -> (b, second_dim)
+            0,
+            repeat(indices, 'z -> z d',
+                   d=second_dim)  # (indices,) -> (indices, second_dim)
+        ).reshape(-1, *other_shape)  # (num_idx, ...)
+
+    @staticmethod
+    def backward(ctx, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None]:
+        indices, = ctx.saved_tensors
+        assert grad_output.ndim >= 2
+        other_shape = grad_output.shape[1:]
+        grad_output = rearrange(grad_output, 'b ... -> b (...)')
+        grad_input = torch.zeros([ctx.first_axis_dim, grad_output.shape[1]],
+                                 device=grad_output.device,
+                                 dtype=grad_output.dtype)
+        # TD [2022-03-04] For some reason torch.scatter is a bit faster than indexing.
+        # grad_input[indices] = grad_output
+        grad_input.scatter_(0,
+                            repeat(indices, 'z -> z d', d=grad_output.shape[1]),
+                            grad_output)
+        return grad_input.reshape(ctx.first_axis_dim, *other_shape), None
+
+
+index_first_axis = IndexFirstAxis.apply
+
+
+class IndexPutFirstAxis(torch.autograd.Function):
+
+    @staticmethod
+    def forward(ctx, values: torch.Tensor, indices: torch.Tensor,
+                first_axis_dim) -> torch.Tensor:
+        ctx.save_for_backward(indices)
+        assert indices.ndim == 1
+        assert values.ndim >= 2
+        output = torch.zeros(first_axis_dim,
+                             *values.shape[1:],
+                             device=values.device,
+                             dtype=values.dtype)
+        output[indices] = values
+        return output
+
+    @staticmethod
+    def backward(ctx,
+                 grad_output: torch.Tensor) -> Tuple[torch.Tensor, None, None]:
+        indices, = ctx.saved_tensors
+        grad_values = grad_output[indices]
+        return grad_values, None, None
+
+
+index_put_first_axis = IndexPutFirstAxis.apply
+
+
+def unpad_input(
+    hidden_states: torch.Tensor,
+    attention_mask: torch.Tensor,
+) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, int]:
+    """Remove padding from input sequences.
+
+    Arguments:
+        hidden_states: (batch, seqlen, ...)
+        attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid.
+
+    Returns:
+        hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+        indices: (total_nnz)
+        cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states.
+        max_seqlen_in_batch: int ()
+    """
+    seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
+    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+    max_seqlen_in_batch = int(seqlens_in_batch.max().item())
+    cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32),
+                       (1, 0))
+    # TD [2022-03-04] We don't want to index with a bool mask, because Pytorch will expand the
+    # bool mask, then call nonzero to get the indices, then index with those. The indices is @dim
+    # times larger than it needs to be, wasting memory. It's faster and more memory-efficient to
+    # index with integer indices. Moreover, torch's index is a bit slower than it needs to be,
+    # so we write custom forward and backward to make it a bit faster.
+    hidden_states = cast(
+        torch.Tensor,
+        index_first_axis(rearrange(hidden_states, 'b s ... -> (b s) ...'),
+                         indices))
+    return hidden_states, indices, cu_seqlens, max_seqlen_in_batch
+
+
+def unpad_input_only(
+    hidden_states: torch.Tensor,
+    attention_mask: torch.Tensor,
+) -> torch.Tensor:
+    """Like unpad_input, but only return the unpadded first tensor.
+
+    Save a small amount of overhead.
+
+    Arguments:
+        hidden_states: (batch, seqlen, ...)
+        attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid.
+
+    Returns:
+        hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+    """
+    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+    return index_first_axis(rearrange(hidden_states, 'b s ... -> (b s) ...'),
+                            indices)
+
+
+def pad_input(hidden_states: torch.Tensor, indices: torch.Tensor, batch: int,
+              seqlen: int) -> torch.Tensor:
+    """Add padding to sequences.
+
+    Arguments:
+        hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+        indices: (total_nnz)
+        batch: int batch_size
+        seqlen: int max sequence length
+
+    Returns:
+        hidden_states: (batch, seqlen, ...)
+    """
+    output = index_put_first_axis(hidden_states, indices, batch * seqlen)
+    return rearrange(output, '(b s) ... -> b s ...', b=batch)

config.json

@@ -0,0 +1,26 @@
+{
+  "alibi_starting_size": 1024,
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_max_length": 1024,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 74
+}

configuration_bert.py

@@ -0,0 +1,26 @@
+# Copyright 2022 MosaicML Examples authors
+# SPDX-License-Identifier: Apache-2.0
+
+from transformers import BertConfig as TransformersBertConfig
+
+
+class BertConfig(TransformersBertConfig):
+
+    auto_map = {
+        "AutoConfig": "configuration_bert.BertConfig",
+        "AutoModel": "bert_layers.BertModel",
+        "AutoModelForMaskedLM": "bert_layers.BertForMaskedLM",
+        "AutoModelForSequenceClassification": "bert_layers.BertForSequenceClassification",
+    }
+
+    def __init__(
+        self,
+        alibi_starting_size: int = 1024,
+        attention_probs_dropout_prob: float = 0.0,
+        **kwargs,
+    ):
+        super().__init__(
+            attention_probs_dropout_prob=attention_probs_dropout_prob,
+            **kwargs,
+        )
+        self.alibi_starting_size = alibi_starting_size

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44f40d1c7df2a140860df1647ce087313b84406e5694ca66c21df90302bf057b
+size 456168656

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "max_len": 1024,
+  "max_length": 1024,
+  "model_max_length": 1024,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

vocab.txt

@@ -0,0 +1,74 @@
+[PAD]
+[UNK]
+[CLS]
+[SEP]
+[MASK]
+A
+T
+C
+G
+N
+AAA
+AAT
+AAC
+AAG
+ATA
+ATT
+ATC
+ATG
+ACA
+ACT
+ACC
+ACG
+AGA
+AGT
+AGC
+AGG
+TAA
+TAT
+TAC
+TAG
+TTA
+TTT
+TTC
+TTG
+TCA
+TCT
+TCC
+TCG
+TGA
+TGT
+TGC
+TGG
+CAA
+CAT
+CAC
+CAG
+CTA
+CTT
+CTC
+CTG
+CCA
+CCT
+CCC
+CCG
+CGA
+CGT
+CGC
+CGG
+GAA
+GAT
+GAC
+GAG
+GTA
+GTT
+GTC
+GTG
+GCA
+GCT
+GCC
+GCG
+GGA
+GGT
+GGC
+GGG