Update README.md
Browse files
README.md
CHANGED
|
@@ -55,8 +55,8 @@ text="In the heart of the bustling city, where towering skyscrapers touch the cl
|
|
| 55 |
With each passing light year, the anticipation of unraveling secrets that could alter humanity's\
|
| 56 |
understanding of life in the universe grew ever stronger."
|
| 57 |
|
| 58 |
-
# chunk the text
|
| 59 |
-
chunks=model.chunk_text(text, tokenizer)
|
| 60 |
|
| 61 |
# print chunks
|
| 62 |
for i, c in enumerate(chunks):
|
|
|
|
| 55 |
With each passing light year, the anticipation of unraveling secrets that could alter humanity's\
|
| 56 |
understanding of life in the universe grew ever stronger."
|
| 57 |
|
| 58 |
+
# chunk the text. The lower threshold is, the more chunks will be generated.
|
| 59 |
+
chunks=model.chunk_text(text, tokenizer, threshold=0)
|
| 60 |
|
| 61 |
# print chunks
|
| 62 |
for i, c in enumerate(chunks):
|