Update README.md
Browse files
README.md
CHANGED
|
@@ -49,13 +49,13 @@ state_dict = safetensors.torch.load_file("./model.safetensors")
|
|
| 49 |
model.load_state_dict(state_dict)
|
| 50 |
|
| 51 |
# text to be chunked
|
| 52 |
-
text=
|
| 53 |
-
of honking cars never ceases, Sarah, an aspiring novelist, found solace in the quiet corners of the ancient library.
|
| 54 |
-
Surrounded by shelves that whispered stories of centuries past, she crafted her own world with words, oblivious to the rush outside
|
| 55 |
-
Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7.
|
| 56 |
-
As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves.
|
| 57 |
-
With each passing light year, the anticipation of unraveling secrets that could alter humanity's
|
| 58 |
-
understanding of life in the universe grew ever stronger.
|
| 59 |
|
| 60 |
# chunk the text. The lower threshold is, the more chunks will be generated. Can be (-inf, +inf).
|
| 61 |
chunks=model.chunk_text(text, tokenizer, threshold=0)
|
|
|
|
| 49 |
model.load_state_dict(state_dict)
|
| 50 |
|
| 51 |
# text to be chunked
|
| 52 |
+
text='''In the heart of the bustling city, where towering skyscrapers touch the clouds and the symphony
|
| 53 |
+
of honking cars never ceases, Sarah, an aspiring novelist, found solace in the quiet corners of the ancient library.
|
| 54 |
+
Surrounded by shelves that whispered stories of centuries past, she crafted her own world with words, oblivious to the rush outside.
|
| 55 |
+
Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7.
|
| 56 |
+
As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves.
|
| 57 |
+
With each passing light year, the anticipation of unraveling secrets that could alter humanity's
|
| 58 |
+
understanding of life in the universe grew ever stronger.'''
|
| 59 |
|
| 60 |
# chunk the text. The lower threshold is, the more chunks will be generated. Can be (-inf, +inf).
|
| 61 |
chunks=model.chunk_text(text, tokenizer, threshold=0)
|