feat: add custom tokenizer with multi-char Chinese token splitting (#8)

bffb3df

EasonLiu linyueqian commited on Apr 16