Instructions to use ArthaLabs/panini-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ArthaLabs/panini-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ArthaLabs/panini-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload src/splitter.py with huggingface_hub
Browse files- src/splitter.py +3 -3
src/splitter.py
CHANGED
|
@@ -6,9 +6,9 @@ Detects and splits Sanskrit compound words at their boundaries.
|
|
| 6 |
from typing import List, Tuple, Optional
|
| 7 |
from dataclasses import dataclass
|
| 8 |
|
| 9 |
-
# Import analyzer for Kosha access
|
| 10 |
-
from
|
| 11 |
-
from
|
| 12 |
|
| 13 |
|
| 14 |
@dataclass
|
|
|
|
| 6 |
from typing import List, Tuple, Optional
|
| 7 |
from dataclasses import dataclass
|
| 8 |
|
| 9 |
+
# Import analyzer for Kosha access (absolute imports for HF compatibility)
|
| 10 |
+
from analyzer import VidyutAnalyzer, MorphParse
|
| 11 |
+
from sandhi_engine import SandhiEngine
|
| 12 |
|
| 13 |
|
| 14 |
@dataclass
|