Multilingual tokenizer
#2
by
martin-nguyen
- opened
Hi Jmica,
Have you tried combining tokens for new languages (JA, DE, FR) and preserving the existing tokens from Chatterbox?
Thanks.
martin-nguyen
changed discussion title from
Script to create JA tokenizer.json
to Multilingual tokenizer