loubnabnl HF Staff commited on
Commit
b2b5137
·
verified ·
1 Parent(s): 3743563

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -257,7 +257,7 @@ Sample sizes: Carbon & GENERator n=500. Evo2-7B n=150 at 16k, n=100 at 32k, n=20
257
  - **Architecture:** decoder-only Transformer (Llama-style), 30 layers, hidden 3072, FFN 8448, 32 attention heads with GQA (4 KV groups), SwiGLU, RMSNorm.
258
  - **Tokenizer:** Carbon 6-mer hybrid (vocab ≈ 156 k including DNA tags and metadata tokens and BPE tokens for future English & DNA continual pretraining).
259
  - **Precision:** bfloat16
260
- - **Positional embedding:** RoPE, base $\theta = 5 \times 10^{6}$, max position 32 768.
261
 
262
  ### Pre-training
263
 
 
257
  - **Architecture:** decoder-only Transformer (Llama-style), 30 layers, hidden 3072, FFN 8448, 32 attention heads with GQA (4 KV groups), SwiGLU, RMSNorm.
258
  - **Tokenizer:** Carbon 6-mer hybrid (vocab ≈ 156 k including DNA tags and metadata tokens and BPE tokens for future English & DNA continual pretraining).
259
  - **Precision:** bfloat16
260
+ - **Positional embedding:** RoPE, base θ = 5 × 10^6, max position 32,768.
261
 
262
  ### Pre-training
263