tim1900 commited on
Commit
adbf8b7
·
verified ·
1 Parent(s): ffb9545

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -6
README.md CHANGED
@@ -48,12 +48,13 @@ model.load_state_dict(state_dict)
48
 
49
  # text to be chunked, can be any size.
50
  text='''In the heart of the bustling city, where towering skyscrapers touch the clouds and the symphony
51
- of honking cars never ceases, Sarah, an aspiring novelist, found solace in the quiet corners of the ancient library.
52
- Surrounded by shelves that whispered stories of centuries past, she crafted her own world with words, oblivious to the rush outside.
53
- Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7.
54
- As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves.
55
- With each passing light year, the anticipation of unraveling secrets that could alter humanity's
56
- understanding of life in the universe grew ever stronger.'''
 
57
 
58
  # chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
59
  chunks=model.chunk_text(text, tokenizer, prob_threshold=0.5)
 
48
 
49
  # text to be chunked, can be any size.
50
  text='''In the heart of the bustling city, where towering skyscrapers touch the clouds and the symphony
51
+ of honking cars never ceases, Sarah, an aspiring
52
+ novelist, found solace in the quiet corners of the ancient library.
53
+ Surrounded by shelves that whispered stories of centuries
54
+
55
+ past, she crafted her own world with words, oblivious to the rush outside. Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7.
56
+ As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves. With each passing light year, the anticipation of unraveling secrets that could alter humanity's
57
+ understanding of life in the universe grew ever stronger.'''
58
 
59
  # chunk the text. The prob_threshold should be between (0, 1). The lower it is, the more chunks will be generated.
60
  chunks=model.chunk_text(text, tokenizer, prob_threshold=0.5)