|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- biology |
|
|
--- |
|
|
# Gengram-10B-torch |
|
|
|
|
|
|
|
|
This repository hosts the model weights for Gengram-10B-torch. For instructions and details, please refer to the **[Gengram GitHub](https://github.com/zhejianglab/Gengram)**. |
|
|
|
|
|
Gengram is a novel conditional memory module designed for genomic foundation models (GFMs) that introduces explicit motif memory retrieval to enhance Transformer-based DNA sequence modeling. Unlike traditional GFMs that rely on dense computation to implicitly infer multi-nucleotide motifs, Gengram provides an efficient lookup mechanism for biological patterns through a genomic-specific hashing scheme. |
|
|
|
|
|
### ✨ Key Features |
|
|
|
|
|
- **🎯 Explicit Motif Memory**: Stores and retrieves k-mers (k=1-6) via hash-based lookup tables |
|
|
- **🧬 Local Window Aggregation**: 21bp window mechanism aligned with DNA helical structure |
|
|
- **⚡ Computational Efficiency**: Linear time complexity with minimal overhead |
|
|
- **🔧 Architecture Agnostic**: Compatible with various attention mechanisms (MHA, GQA, MLA) |
|
|
- **⚖️ Stable Training**: Improves load balancing in Mixture-of-Experts models |
|
|
- **🔍 Biological Interpretability**: Learns meaningful motif representations |
|
|
|
|
|
### ✨ Biological Interpretability |
|
|
|
|
|
- **Reverse-complement symmetry** in memory embeddings |
|
|
- **Context-dependent gating** aligned with functional regions |
|
|
- **Hierarchical representation** from shallow to deep layers |
|
|
|
|
|
For full documentation, training details, and usage instructions, please visit the [GitHub]((https://github.com/zhejianglab/Gengram)) repository. |
