| This model has not been trained on any Cantonese material. |
|
|
| It is simply a base model in which the embeddings and tokenizer were patched with Cantonese characters. One can find the original model [gpt2-tiny-chinese](https://huggingface.co/ckiplab/gpt2-tiny-chinese). |
|
|
|
|
|
|
|
|
|
|
|
|
| I used this [repo](https://github.com/ayaka14732/bert-tokenizer-cantonese) to identify missing Cantonese characters |
|
|
| [My forked and modified version](https://github.com/jedcheng/bert-tokenizer-cantonese) |
|
|
| After identifying the missing characters, the Huggingface library provides very high level API to modify the tokenizer and embeddings. |
|
|
| ``` |
| Download a tokenizer and a model from the Huggingface library. Then: |
| |
| tokenizer.add_tokens("your new tokens") |
| model.resize_token_embeddings(len(tokenizer)) |
| |
| tokenizer.push_to_hub("your model name") |
| ``` |
