File size: 2,725 Bytes
e0d39c7 b48d661 e0d39c7 f78d3f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | ---
license: mit
---
<div align="center">
<img src="https://raw.githubusercontent.com/BGI-HangzhouAI/Genos/main/images/Genos_model.png" width="100%" />
</div>
# Genos
Genos, as a foundational model in the field of human genomics, trained on hundreds of high-quality genome reference data, has achieved the ability to contextually model human genome sequences up to millions of base pairs. Through single-base resolution learning, this model possesses the capability to identify hidden deep sequence patterns and functional features within genomes, providing scientists with a new research method that connects genetic information with life activities.
For instructions, details, and examples, please refer to the [Genos GitHub](https://github.com/BGI-HangzhouAI/Genos).
Below are the data volume of our model training and related parameters.
<table align="center">
<tr>
<th>Model Specification</th>
<th>Genos 1.2B</th>
<th>Genos 10B</th>
</tr>
<!-- Model Scale category title - span 3 columns -->
<tr>
<td colspan="3" align="center"><b>Model Scale</b></td>
</tr>
<tr>
<td>Total Parameters</td>
<td>1.2B</td>
<td>10B</td>
</tr>
<tr>
<td>Activated Parameters</td>
<td>0.33B</td>
<td>2.87B</td>
</tr>
<tr>
<td>Trained Tokens</td>
<td>1600 B</td>
<td>2200 B</td>
</tr>
<!-- Architecture category title - span 3 columns -->
<tr>
<td colspan="3" align="center"><b>Architecture</b></td>
</tr>
<tr>
<td>Architecture Type</td>
<td>MoE</td>
<td>MoE</td>
</tr>
<tr>
<td>Number of Experts</td>
<td>8</td>
<td>8</td>
</tr>
<tr>
<td>Selected Experts per Token</td>
<td>2</td>
<td>2</td>
</tr>
<tr>
<td>Number of Layers</td>
<td>12</td>
<td>12</td>
</tr>
<tr>
<td>Attention Hidden Dimension</td>
<td>1024</td>
<td>4096</td>
</tr>
<tr>
<td>Number of Attention Heads</td>
<td>16</td>
<td>16</td>
</tr>
<tr>
<td>MoE Hidden Dimension (per Expert)</td>
<td>4096</td>
<td>8192</td>
</tr>
<tr>
<td>Vocabulary Size</td>
<td>128 (padded)</td>
<td>256 (padded)</td>
</tr>
<tr>
<td>Context Length</td>
<td>up to 1M</td>
<td>up to 1M</td>
</tr>
</table>
Genos 1.2B and 10B checkpoints are available here:
- [Genos-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-1.2B)
- [Genos-10B](https://huggingface.co/BGI-HangzhouAI/Genos-10B)
We also provide checkpoints trained under the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) framework:
- [Genos-Megatron-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-1.2B)
- [Genos-Megatron-10B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-10B)
|