File size: 379 Bytes
e0ceb5c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | """
Curriculum Tokenizer - Fast C++ BPE inference with Hugging Face compatibility.
Usage:
from curriculum_tokenizer import CurriculumTokenizer
tok = CurriculumTokenizer.from_pretrained("./artifacts")
ids = tok.encode("Hello world!")
text = tok.decode(ids)
"""
from .tokenizer import CurriculumTokenizer
__version__ = "0.3.0"
__all__ = ["CurriculumTokenizer"]
|