docs: add Neo Ar model specification
#2
by assemsabry - opened
- docs/MODEL_SPECIFICATION.md +21 -0
docs/MODEL_SPECIFICATION.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Neo Ar Model Specification
|
| 2 |
+
|
| 3 |
+
Neo Ar is an Arabic-only System One decision model with a larger encoder than
|
| 4 |
+
the English Neo model.
|
| 5 |
+
|
| 6 |
+
- 12 Transformer layers.
|
| 7 |
+
- Hidden size 768.
|
| 8 |
+
- 12 attention heads.
|
| 9 |
+
- Intermediate size 3,072.
|
| 10 |
+
- Maximum context 1,024 tokens.
|
| 11 |
+
- Up to 64 choice options.
|
| 12 |
+
- Up to 7 score levels.
|
| 13 |
+
- Binary `noul` decision.
|
| 14 |
+
- Custom Arabic WordPiece tokenizer.
|
| 15 |
+
|
| 16 |
+
The released architecture contains 86,393,160 trainable parameters with a
|
| 17 |
+
644-token Arabic vocabulary.
|
| 18 |
+
|
| 19 |
+
The final parameter count is measured from the released checkpoint after the
|
| 20 |
+
Arabic tokenizer vocabulary is finalized. The full source and training process
|
| 21 |
+
are documented at https://github.com/tokenaii/Neo-Ar.
|