Upload folder using huggingface_hub
Browse files
README.md
CHANGED
|
@@ -20,6 +20,8 @@ tags:
|
|
| 20 |
|
| 21 |
**The first grammar-first Sanskrit tokenizer based on Pāṇinian morphological analysis.**
|
| 22 |
|
|
|
|
|
|
|
| 23 |
## 🚨 The Problem
|
| 24 |
|
| 25 |
Statistical tokenizers (BPE/WordPiece) systematically underperform on Sanskrit because they do not model **Sandhi**(phonetic fusion).
|
|
|
|
| 20 |
|
| 21 |
**The first grammar-first Sanskrit tokenizer based on Pāṇinian morphological analysis.**
|
| 22 |
|
| 23 |
+
[](https://huggingface.co/spaces/ArthaLabs/panini-tokenizer-demo)
|
| 24 |
+
|
| 25 |
## 🚨 The Problem
|
| 26 |
|
| 27 |
Statistical tokenizers (BPE/WordPiece) systematically underperform on Sanskrit because they do not model **Sandhi**(phonetic fusion).
|