| --- |
| license: apache-2.0 |
| tags: |
| - biology |
| --- |
| # Gengram-10B |
|
|
|
|
| This repository hosts the model weights for Gengram-10B. For instructions and details, please refer to the **[Gengram GitHub](https://github.com/zhejianglab/Gengram)**. |
|
|
| Gengram is a novel conditional memory module designed for genomic foundation models (GFMs) that introduces explicit motif memory retrieval to enhance Transformer-based DNA sequence modeling. Unlike traditional GFMs that rely on dense computation to implicitly infer multi-nucleotide motifs, Gengram provides an efficient lookup mechanism for biological patterns through a genomic-specific hashing scheme. |
|
|
| ### ✨ Key Features |
|
|
| - **🎯 Explicit Motif Memory**: Stores and retrieves k-mers (k=1-6) via hash-based lookup tables |
| - **🧬 Local Window Aggregation**: 21bp window mechanism aligned with DNA helical structure |
| - **⚡ Computational Efficiency**: Linear time complexity with minimal overhead |
| - **🔧 Architecture Agnostic**: Compatible with various attention mechanisms (MHA, GQA, MLA) |
| - **⚖️ Stable Training**: Improves load balancing in Mixture-of-Experts models |
| - **🔍 Biological Interpretability**: Learns meaningful motif representations |
|
|
| ### ✨ Biological Interpretability |
|
|
| - **Reverse-complement symmetry** in memory embeddings |
| - **Context-dependent gating** aligned with functional regions |
| - **Hierarchical representation** from shallow to deep layers |
|
|
| For full documentation, training details, and usage instructions, please visit the [GitHub](https://github.com/zhejianglab/Gengram) repository. |
