File size: 161 Bytes
c8dfeef | 1 2 3 4 5 6 7 | class CustomTokenizer:
def __init__(self):
pass
def __call__(self, text):
# No tokenization, return input text as-is
return text |
c8dfeef | 1 2 3 4 5 6 7 | class CustomTokenizer:
def __init__(self):
pass
def __call__(self, text):
# No tokenization, return input text as-is
return text |