Instructions to use tim1900/bert-chunker with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tim1900/bert-chunker with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="tim1900/bert-chunker")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("tim1900/bert-chunker") model = AutoModel.from_pretrained("tim1900/bert-chunker", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -57,7 +57,7 @@ text="In the heart of the bustling city, where towering skyscrapers touch the cl
|
|
| 57 |
With each passing light year, the anticipation of unraveling secrets that could alter humanity's\
|
| 58 |
understanding of life in the universe grew ever stronger."
|
| 59 |
|
| 60 |
-
# chunk the text. The lower threshold is, the more chunks will be generated. Can be
|
| 61 |
chunks=model.chunk_text(text, tokenizer, threshold=0)
|
| 62 |
|
| 63 |
# print chunks
|
|
|
|
| 57 |
With each passing light year, the anticipation of unraveling secrets that could alter humanity's\
|
| 58 |
understanding of life in the universe grew ever stronger."
|
| 59 |
|
| 60 |
+
# chunk the text. The lower threshold is, the more chunks will be generated. Can be (-inf, +inf).
|
| 61 |
chunks=model.chunk_text(text, tokenizer, threshold=0)
|
| 62 |
|
| 63 |
# print chunks
|