Instructions to use tim1900/bert-chunker with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tim1900/bert-chunker with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="tim1900/bert-chunker")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("tim1900/bert-chunker") model = AutoModel.from_pretrained("tim1900/bert-chunker", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -63,12 +63,4 @@ for i, c in enumerate(chunks):
|
|
| 63 |
print(f'-----chunk: {i}------------')
|
| 64 |
print(c)
|
| 65 |
|
| 66 |
-
# chunk the text faster but compromising performance a lot, by using a fixed context window, batchsize is the number of windows run per batch.
|
| 67 |
-
print('----->Here is the result of fast chunk method<------:')
|
| 68 |
-
chunks=model.chunk_text_fast(text, tokenizer, batchsize=20, prob_threshold=0.5)
|
| 69 |
-
|
| 70 |
-
# print chunks
|
| 71 |
-
for i, c in enumerate(chunks):
|
| 72 |
-
print(f'-----chunk: {i}------------')
|
| 73 |
-
print(c)
|
| 74 |
```
|
|
|
|
| 63 |
print(f'-----chunk: {i}------------')
|
| 64 |
print(c)
|
| 65 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
```
|