tim1900 commited on
Commit
ed656a3
·
verified ·
1 Parent(s): 00092d6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -7
README.md CHANGED
@@ -49,13 +49,13 @@ state_dict = safetensors.torch.load_file("./model.safetensors")
49
  model.load_state_dict(state_dict)
50
 
51
  # text to be chunked
52
- text="In the heart of the bustling city, where towering skyscrapers touch the clouds and the symphony \
53
- of honking cars never ceases, Sarah, an aspiring novelist, found solace in the quiet corners of the ancient library. \
54
- Surrounded by shelves that whispered stories of centuries past, she crafted her own world with words, oblivious to the rush outside.\
55
- Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7. \
56
- As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves. \
57
- With each passing light year, the anticipation of unraveling secrets that could alter humanity's\
58
- understanding of life in the universe grew ever stronger."
59
 
60
  # chunk the text. The lower threshold is, the more chunks will be generated. Can be (-inf, +inf).
61
  chunks=model.chunk_text(text, tokenizer, threshold=0)
 
49
  model.load_state_dict(state_dict)
50
 
51
  # text to be chunked
52
+ text='''In the heart of the bustling city, where towering skyscrapers touch the clouds and the symphony
53
+ of honking cars never ceases, Sarah, an aspiring novelist, found solace in the quiet corners of the ancient library.
54
+ Surrounded by shelves that whispered stories of centuries past, she crafted her own world with words, oblivious to the rush outside.
55
+ Dr. Alexander Thompson, aboard the spaceship 'Pandora's Venture', was en route to the newly discovered exoplanet Zephyr-7.
56
+ As the lead astrobiologist of the expedition, his mission was to uncover signs of microbial life within the planet's subterranean ice caves.
57
+ With each passing light year, the anticipation of unraveling secrets that could alter humanity's
58
+ understanding of life in the universe grew ever stronger.'''
59
 
60
  # chunk the text. The lower threshold is, the more chunks will be generated. Can be (-inf, +inf).
61
  chunks=model.chunk_text(text, tokenizer, threshold=0)