Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,106 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- biology
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
<div align="center">
|
| 8 |
+
<img src="https://raw.githubusercontent.com/BGI-HangzhouAI/Genos/main/images/Genos_model.png" width="100%" />
|
| 9 |
+
</div>
|
| 10 |
+
|
| 11 |
+
# Genos
|
| 12 |
+
|
| 13 |
+
Genos, as a foundational model in the field of human genomics, trained on hundreds of high-quality genome reference data, has achieved the ability to contextually model human genome sequences up to millions of base pairs. Through single-base resolution learning, this model possesses the capability to identify hidden deep sequence patterns and functional features within genomes, providing scientists with a new research method that connects genetic information with life activities.
|
| 14 |
+
|
| 15 |
+
For instructions, details, and examples, please refer to the [Genos GitHub](https://github.com/BGI-HangzhouAI/Genos).
|
| 16 |
+
|
| 17 |
+
Below are the data volume of our model training and related parameters.
|
| 18 |
+
|
| 19 |
+
<table align="center">
|
| 20 |
+
<tr>
|
| 21 |
+
<th>Model Specification</th>
|
| 22 |
+
<th>Genos 1.2B</th>
|
| 23 |
+
<th>Genos 10B</th>
|
| 24 |
+
</tr>
|
| 25 |
+
|
| 26 |
+
<!-- Model Scale category title - span 3 columns -->
|
| 27 |
+
<tr>
|
| 28 |
+
<td colspan="3" align="center"><b>Model Scale</b></td>
|
| 29 |
+
</tr>
|
| 30 |
+
<tr>
|
| 31 |
+
<td>Total Parameters</td>
|
| 32 |
+
<td>1.2B</td>
|
| 33 |
+
<td>10B</td>
|
| 34 |
+
</tr>
|
| 35 |
+
<tr>
|
| 36 |
+
<td>Activated Parameters</td>
|
| 37 |
+
<td>0.33B</td>
|
| 38 |
+
<td>2.87B</td>
|
| 39 |
+
</tr>
|
| 40 |
+
<tr>
|
| 41 |
+
<td>Trained Tokens</td>
|
| 42 |
+
<td>1600 B</td>
|
| 43 |
+
<td>2200 B</td>
|
| 44 |
+
</tr>
|
| 45 |
+
|
| 46 |
+
<!-- Architecture category title - span 3 columns -->
|
| 47 |
+
<tr>
|
| 48 |
+
<td colspan="3" align="center"><b>Architecture</b></td>
|
| 49 |
+
</tr>
|
| 50 |
+
<tr>
|
| 51 |
+
<td>Architecture Type</td>
|
| 52 |
+
<td>MoE</td>
|
| 53 |
+
<td>MoE</td>
|
| 54 |
+
</tr>
|
| 55 |
+
<tr>
|
| 56 |
+
<td>Number of Experts</td>
|
| 57 |
+
<td>8</td>
|
| 58 |
+
<td>8</td>
|
| 59 |
+
</tr>
|
| 60 |
+
<tr>
|
| 61 |
+
<td>Selected Experts per Token</td>
|
| 62 |
+
<td>2</td>
|
| 63 |
+
<td>2</td>
|
| 64 |
+
</tr>
|
| 65 |
+
<tr>
|
| 66 |
+
<td>Number of Layers</td>
|
| 67 |
+
<td>12</td>
|
| 68 |
+
<td>12</td>
|
| 69 |
+
</tr>
|
| 70 |
+
<tr>
|
| 71 |
+
<td>Attention Hidden Dimension</td>
|
| 72 |
+
<td>1024</td>
|
| 73 |
+
<td>4096</td>
|
| 74 |
+
</tr>
|
| 75 |
+
<tr>
|
| 76 |
+
<td>Number of Attention Heads</td>
|
| 77 |
+
<td>16</td>
|
| 78 |
+
<td>16</td>
|
| 79 |
+
</tr>
|
| 80 |
+
<tr>
|
| 81 |
+
<td>MoE Hidden Dimension (per Expert)</td>
|
| 82 |
+
<td>4096</td>
|
| 83 |
+
<td>8192</td>
|
| 84 |
+
</tr>
|
| 85 |
+
<tr>
|
| 86 |
+
<td>Vocabulary Size</td>
|
| 87 |
+
<td>128 (padded)</td>
|
| 88 |
+
<td>256 (padded)</td>
|
| 89 |
+
</tr>
|
| 90 |
+
<tr>
|
| 91 |
+
<td>Context Length</td>
|
| 92 |
+
<td>up to 1M</td>
|
| 93 |
+
<td>up to 1M</td>
|
| 94 |
+
</tr>
|
| 95 |
+
</table>
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
Genos 1.2B and 10B checkpoints are available here:
|
| 99 |
+
|
| 100 |
+
- [Genos-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-1.2B)
|
| 101 |
+
- [Genos-10B](https://huggingface.co/BGI-HangzhouAI/Genos-10B)
|
| 102 |
+
|
| 103 |
+
We also provide checkpoints trained under the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) framework:
|
| 104 |
+
|
| 105 |
+
- [Genos-Megatron-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-1.2B)
|
| 106 |
+
- [Genos-Megatron-10B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-10B)
|