Create README.md

README.md

@@ -0,0 +1,34 @@
+---
+language: en
+license: apache-2.0
+tags:
+- biology
+- genomics
+- dnabert
+- sequence-analysis
+---
+
+# Genomic DNA Sequence Transformer
+
+## Overview
+This model is a BERT-based encoder pre-trained on the human reference genome (GRCh38). It utilizes a k-mer tokenization approach to learn the underlying semantics of DNA, enabling high-accuracy downstream tasks such as promoter identification, splice site prediction, and variant effect scoring.
+
+
+
+## Model Architecture
+Based on the **DNABERT** framework:
+- **Tokenization**: Sequences are converted into 6-mer tokens (e.g., `ATGCGT`).
+- **Pre-training**: Masked Language Modeling (MLM) was performed on over 3 billion base pairs.
+- **Encoding**: The bidirectional attention mechanism allows each nucleotide position to attend to the entire sequence context, capturing complex regulatory motifs.
+- **Metric**: The pre-training objective minimizes the negative log-likelihood:
+$$\mathcal{L}_{MLM} = -\mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{i \in \text{masked}} \log p(x_i | x_{\setminus i}) \right]$$
+
+## Intended Use
+- **Motif Discovery**: Locating transcription factor binding sites.
+- **Functional Annotation**: Predicting the biological function of non-coding regions.
+- **Comparative Genomics**: Evaluating evolutionary conservation at a sequence level.
+
+## Limitations
+- **Sequence Length**: Restricted to 512 tokens (~517 base pairs including overlaps), making it unsuitable for analyzing whole chromosomes without sliding windows.
+- **Species Specificity**: Performance may vary on non-human genomes (e.g., extremophile bacteria or complex plant genomes) without further fine-tuning.
+- **Structural Variants**: Primarily focused on single-nucleotide patterns rather than large-scale structural re-arrangements.