intentgraph-lm / src /tokenizer.py
hmnshudhmn24's picture
Upload 23 files
602b5b1 verified
class SimpleTokenizer:
def tokenize(self, text: str):
return text.lower().split()