Bingsu commited on
Commit
e3b51e1
ยท
1 Parent(s): 05edf11

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -16,4 +16,4 @@ license:
16
 
17
  [Bingsu/ko_BBPE_tokenizer_roberta](https://huggingface.co/Bingsu/ko_BBPE_tokenizer_roberta)์—์„œ post-processor๋ฅผ BertProcessing๋กœ ๋ณ€๊ฒฝํ•˜๊ณ  ํ† ํฌ๋‚˜์ด์ € ํด๋ž˜์Šค๋ฅผ `BertTokenizerFast`๋กœ ๋ณ€๊ฒฝํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค.
18
 
19
- ๋‘ ๋ฌธ์žฅ์„ ํ† ํฌ๋‚˜์ด์ €์— ์ž…๋ ฅํ–ˆ์„ ๋•Œ, sep ํ† ํฐ์ด ๋‹ค๋ฅด๊ฒŒ ์ฒ˜๋ฆฌ๋œ๋‹ค๋Š” ๊ฒƒ์„ ์ œ์™ธํ•˜๋ฉด ๊ฒฐ๊ณผ๋Š” ๋™์ผํ•ฉ๋‹ˆ๋‹ค.
 
16
 
17
  [Bingsu/ko_BBPE_tokenizer_roberta](https://huggingface.co/Bingsu/ko_BBPE_tokenizer_roberta)์—์„œ post-processor๋ฅผ BertProcessing๋กœ ๋ณ€๊ฒฝํ•˜๊ณ  ํ† ํฌ๋‚˜์ด์ € ํด๋ž˜์Šค๋ฅผ `BertTokenizerFast`๋กœ ๋ณ€๊ฒฝํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค.
18
 
19
+ ๋‘ ๋ฌธ์žฅ์„ ํ† ํฌ๋‚˜์ด์ €์— ์ž…๋ ฅํ–ˆ์„ ๋•Œ, sep ํ† ํฐ์ด ๋‹ค๋ฅด๊ฒŒ ์ฒ˜๋ฆฌ๋œ๋‹ค๋Š” ๊ฒƒ์„ ์ œ์™ธํ•˜๊ณ  ๊ฒฐ๊ณผ๋Š” ๋™์ผํ•ฉ๋‹ˆ๋‹ค.