netsrik commited on
Commit
f11e2ac
·
verified ·
1 Parent(s): cbe7387

Clarify hidden size and split model details

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -14,7 +14,9 @@ An experimental 1,301,120-parameter language model trained with DeepJ. On 100 de
14
 
15
  ## Model details
16
 
17
- This compact language model was created with [DeepJ](https://github.com/KirstenAli/DeepJ) and uses a DeepSeek-style Transformer architecture with 4 layers, a width of 128, 4 attention heads, Q rank 64, KV rank 32 and a 2,048-token vocabulary. It is not an exact implementation of DeepSeek V2, V3 or R1 and does not currently use an incremental KV cache.
 
 
18
  The `model.dj` checkpoint and `tokenizer.bpe` files use DeepJ's versioned binary formats.
19
  They are not directly loadable by the Python Transformers library.
20
 
 
14
 
15
  ## Model details
16
 
17
+ This compact language model was created with [DeepJ](https://github.com/KirstenAli/DeepJ) and uses a DeepSeek-style Transformer architecture with 4 layers, a hidden size of 128, 4 attention heads, Q rank 64, KV rank 32 and a 2,048-token vocabulary.
18
+
19
+ It is not an exact implementation of DeepSeek V2, V3 or R1 and does not currently use an incremental KV cache.
20
  The `model.dj` checkpoint and `tokenizer.bpe` files use DeepJ's versioned binary formats.
21
  They are not directly loadable by the Python Transformers library.
22