---
language: en
license: apache-2.0
tags:
- biology
- genomics
- dnabert
- sequence-analysis
---

# Genomic DNA Sequence Transformer

## Overview
This model is a BERT-based encoder pre-trained on the human reference genome (GRCh38). It utilizes a k-mer tokenization approach to learn the underlying semantics of DNA, enabling high-accuracy downstream tasks such as promoter identification, splice site prediction, and variant effect scoring.



## Model Architecture
Based on the **DNABERT** framework:
- **Tokenization**: Sequences are converted into 6-mer tokens (e.g., `ATGCGT`).
- **Pre-training**: Masked Language Modeling (MLM) was performed on over 3 billion base pairs.
- **Encoding**: The bidirectional attention mechanism allows each nucleotide position to attend to the entire sequence context, capturing complex regulatory motifs.
- **Metric**: The pre-training objective minimizes the negative log-likelihood:
$$\mathcal{L}_{MLM} = -\mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{i \in \text{masked}} \log p(x_i | x_{\setminus i}) \right]$$

## Intended Use
- **Motif Discovery**: Locating transcription factor binding sites.
- **Functional Annotation**: Predicting the biological function of non-coding regions.
- **Comparative Genomics**: Evaluating evolutionary conservation at a sequence level.

## Limitations
- **Sequence Length**: Restricted to 512 tokens (~517 base pairs including overlaps), making it unsuitable for analyzing whole chromosomes without sliding windows.
- **Species Specificity**: Performance may vary on non-human genomes (e.g., extremophile bacteria or complex plant genomes) without further fine-tuning.
- **Structural Variants**: Primarily focused on single-nucleotide patterns rather than large-scale structural re-arrangements.