Update README.md
Browse files
README.md
CHANGED
|
@@ -96,16 +96,4 @@ chunks, token_pos=chunk_text(model,text, tokenizer, prob_threshold=0.5)
|
|
| 96 |
for i, (c,t) in enumerate(zip(chunks,token_pos)):
|
| 97 |
print(f'-----chunk: {i}----token_idx: {t}--------')
|
| 98 |
print(c)
|
| 99 |
-
```
|
| 100 |
-
## Citation
|
| 101 |
-
|
| 102 |
-
If this work is helpful, please kindly cite as:
|
| 103 |
-
|
| 104 |
-
```bibtex
|
| 105 |
-
@article{bert-chunker,
|
| 106 |
-
title={bert-chunker: Efficient and Trained Chunking for Unstructured Documents},
|
| 107 |
-
author={Yannan Luo},
|
| 108 |
-
year={2024},
|
| 109 |
-
url={https://github.com/jackfsuia/bert-chunker}
|
| 110 |
-
}
|
| 111 |
```
|
|
|
|
| 96 |
for i, (c,t) in enumerate(zip(chunks,token_pos)):
|
| 97 |
print(f'-----chunk: {i}----token_idx: {t}--------')
|
| 98 |
print(c)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
```
|