|
|
--- |
|
|
language: |
|
|
- ko |
|
|
tags: |
|
|
- bert |
|
|
- tokenizer only |
|
|
license: |
|
|
- mit |
|
|
--- |
|
|
|
|
|
## ๋ผ์ด๋ธ๋ฌ๋ฆฌ ๋ฒ์ |
|
|
|
|
|
- transformers: 4.21.1 |
|
|
- datasets: 2.4.0 |
|
|
- tokenizers: 0.12.1 |
|
|
|
|
|
[Bingsu/ko_BBPE_tokenizer_roberta](https://huggingface.co/Bingsu/ko_BBPE_tokenizer_roberta)์์ post-processor๋ฅผ BertProcessing๋ก ๋ณ๊ฒฝํ๊ณ ํ ํฌ๋์ด์ ํด๋์ค๋ฅผ `BertTokenizerFast`๋ก ๋ณ๊ฒฝํ ๊ฒ์
๋๋ค. |
|
|
|
|
|
๋ ๋ฌธ์ฅ์ ํ ํฌ๋์ด์ ์ ์
๋ ฅํ์ ๋, sep ํ ํฐ์ด ๋ค๋ฅด๊ฒ ์ฒ๋ฆฌ๋๋ค๋ ๊ฒ์ ์ ์ธํ๊ณ ๊ฒฐ๊ณผ๋ ๋์ผํฉ๋๋ค. |