Update README.md
Browse files
README.md
CHANGED
|
@@ -172,8 +172,8 @@ Python,但估计听说过这门语言的读者很少。
|
|
| 172 |
"builtins" 。
|
| 173 |
'''
|
| 174 |
# Chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
|
| 175 |
-
# Therefore adjust it to your need, when prob_threshold is small like 0.000001,
|
| 176 |
-
# when it is set to 1,
|
| 177 |
chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
|
| 178 |
|
| 179 |
# print chunks
|
|
|
|
| 172 |
"builtins" 。
|
| 173 |
'''
|
| 174 |
# Chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
|
| 175 |
+
# Therefore adjust it to your need, when prob_threshold is small like 0.000001, each token is one chunk,
|
| 176 |
+
# when it is set to 1, the whole text is one chunk.
|
| 177 |
chunks, token_pos = chunk_text(model, doc, tokenizer, prob_threshold=0.5)
|
| 178 |
|
| 179 |
# print chunks
|