Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Traum Tokenizer
|
| 2 |
|
| 3 |
Traum Tokenizer is a high-performance, specialized tokenizer designed for next-generation Large Language Models (LLMs) and specifically optimized for the Flash - SLM project. Developed after extensive research into existing tokenizers like GPT-2 and BERT, Traum Tokenizer addresses the critical need for a balanced approach between compression efficiency, training speed, and linguistic understanding.
|
|
@@ -69,4 +77,4 @@ print(f"Decoded text: {tokenizer.decode(tokens)}")
|
|
| 69 |
**Assem Sabry** is an Egyptian AI Engineer & Researcher and the founder of Token AI (founded in 2025).
|
| 70 |
|
| 71 |
- Website: https://assem.cloud/
|
| 72 |
-
- LinkedIn: https://www.linkedin.com/in/assem7/
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
pipeline_tag: token-classification
|
| 6 |
+
tags:
|
| 7 |
+
- tokenizer
|
| 8 |
+
---
|
| 9 |
# Traum Tokenizer
|
| 10 |
|
| 11 |
Traum Tokenizer is a high-performance, specialized tokenizer designed for next-generation Large Language Models (LLMs) and specifically optimized for the Flash - SLM project. Developed after extensive research into existing tokenizers like GPT-2 and BERT, Traum Tokenizer addresses the critical need for a balanced approach between compression efficiency, training speed, and linguistic understanding.
|
|
|
|
| 77 |
**Assem Sabry** is an Egyptian AI Engineer & Researcher and the founder of Token AI (founded in 2025).
|
| 78 |
|
| 79 |
- Website: https://assem.cloud/
|
| 80 |
+
- LinkedIn: https://www.linkedin.com/in/assem7/
|