Multilingual tokenizer
#2
by martin-nguyen - opened
Hi Jmica,
Have you tried combining tokens for new languages (JA, DE, FR) and preserving the existing tokens from Chatterbox?
Thanks.
martin-nguyen changed discussion title from Script to create JA tokenizer.json to Multilingual tokenizer