Viraj0112's picture
Upload folder using huggingface_hub
03a907a verified
import re
def tokenize(text: str) -> list[str]:
return re.findall(r"[A-Za-z']+", text.lower())