Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Markov 5-gram LM (500M tokens)
|
| 2 |
+
Classical 5-gram language model with Modified Kneser-Ney smoothing.
|
| 3 |
+
- Architecture: N-gram with GPU hash tables (sorted int64 + searchsorted)
|
| 4 |
+
- Training data: 500M tokens from OpenTransformer web crawl datasets
|
| 5 |
+
- Tokenizer: GPT-2 (50257 vocab)
|
| 6 |
+
- Model size: 61.6M n-gram entries, 1.83GB GPU memory
|
| 7 |
+
- Eval (Pile): Perplexity 46047, Top-1 accuracy 15.14%
|
| 8 |
+
- Inference: 176K tok/s eval throughput on RTX 3060
|
| 9 |
+
Trained by OpenTransformers Ltd. Part of AGILLM research.
|