FlameF0X commited on
Commit
8cc5a91
·
verified ·
1 Parent(s): d961ad2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -3
README.md CHANGED
@@ -5,6 +5,15 @@ datasets:
5
  - HuggingFaceFW/fineweb-edu
6
  ---
7
 
8
- Checkpoint:
9
- - Model Parameters: 433.49M
10
- - PPL: 75
 
 
 
 
 
 
 
 
 
 
5
  - HuggingFaceFW/fineweb-edu
6
  ---
7
 
8
+ ```py
9
+ config = {
10
+ "d_model": 1180,
11
+ "n_rwkv_layers": 16,
12
+ "n_attn_layers": 4,
13
+ "n_heads": 10,
14
+ "seq_len": 1024,
15
+ "batch_size": 4,
16
+ "accum_steps": 8,
17
+ "lr": 4e-4,
18
+ }
19
+ ```