Update README.md
Browse files
README.md
CHANGED
|
@@ -181,6 +181,8 @@ bert-chunker-3 (prob_threshold=0.50543) | N/A | 0 | 90.4 ± 28.7 | 3.3 ± 3.1 |
|
|
| 181 |
★ bert-chunker-3.5 | <= 800 | 0 | 89.5 ± 27.7 | 7.6 ± 5.8 | 29.2 ± 17.9 | 7.6 ± 5.8 |**O(N)** | **Yes**
|
| 182 |
★ bert-chunker-3.5 | <= 400 | 0 | **94.1 ± 22.5**| 4.3 ± 3.5 | 18.1 ± 13.3 | 4.3 ± 3.5 |**O(N)** | **Yes**
|
| 183 |
★ bert-chunker-3.5 | <= 200 | 0 | 90.4 ± 26.2 | 7.7 ± 5.7 | 29.2 ± 17.9 | 7.6 ± 5.7 |**O(N)** | **Yes**
|
|
|
|
|
|
|
| 184 |
## Citation
|
| 185 |
```bibtex
|
| 186 |
@article{bert-chunker,
|
|
|
|
| 181 |
★ bert-chunker-3.5 | <= 800 | 0 | 89.5 ± 27.7 | 7.6 ± 5.8 | 29.2 ± 17.9 | 7.6 ± 5.8 |**O(N)** | **Yes**
|
| 182 |
★ bert-chunker-3.5 | <= 400 | 0 | **94.1 ± 22.5**| 4.3 ± 3.5 | 18.1 ± 13.3 | 4.3 ± 3.5 |**O(N)** | **Yes**
|
| 183 |
★ bert-chunker-3.5 | <= 200 | 0 | 90.4 ± 26.2 | 7.7 ± 5.7 | 29.2 ± 17.9 | 7.6 ± 5.7 |**O(N)** | **Yes**
|
| 184 |
+
## Future
|
| 185 |
+
This model is undertrained due to laziness and lack of money. I observed that the model is undertrained because the outputs from two non-overlapping windows show poor comparability in split point probabilities. It particularly undermines the performance when max_tokens_per_chunk is large. If the data were sufficient, I think cross-window probability comparability would be improved. This is corroborated by my preliminary experiments.
|
| 186 |
## Citation
|
| 187 |
```bibtex
|
| 188 |
@article{bert-chunker,
|