QianweiJoker commited on
Commit
3d3ec15
·
verified ·
1 Parent(s): d2326f3

Upload configs/model/small-110M.yaml with huggingface_hub

Browse files
Files changed (1) hide show
  1. configs/model/small-110M.yaml +12 -0
configs/model/small-110M.yaml ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ name: small-110M
2
+ type: ddit
3
+ hidden_size: 768
4
+ cond_dim: 128
5
+ length: 512
6
+ n_blocks: 15
7
+ n_heads: 12
8
+ scale_by_sigma: True
9
+ dropout: 0.1
10
+ tie_word_embeddings: True
11
+ adaln: False
12
+ attn_backend: flash_attn