tim1900 commited on
Commit
fb160fb
·
verified ·
1 Parent(s): 351e717

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -0
README.md CHANGED
@@ -181,6 +181,8 @@ bert-chunker-3 (prob_threshold=0.50543) | N/A | 0 | 90.4 ± 28.7 | 3.3 ± 3.1 |
181
  ★ bert-chunker-3.5 | <= 800 | 0 | 89.5 ± 27.7 | 7.6 ± 5.8 | 29.2 ± 17.9 | 7.6 ± 5.8 |**O(N)** | **Yes**
182
  ★ bert-chunker-3.5 | <= 400 | 0 | **94.1 ± 22.5**| 4.3 ± 3.5 | 18.1 ± 13.3 | 4.3 ± 3.5 |**O(N)** | **Yes**
183
  ★ bert-chunker-3.5 | <= 200 | 0 | 90.4 ± 26.2 | 7.7 ± 5.7 | 29.2 ± 17.9 | 7.6 ± 5.7 |**O(N)** | **Yes**
 
 
184
  ## Citation
185
  ```bibtex
186
  @article{bert-chunker,
 
181
  ★ bert-chunker-3.5 | <= 800 | 0 | 89.5 ± 27.7 | 7.6 ± 5.8 | 29.2 ± 17.9 | 7.6 ± 5.8 |**O(N)** | **Yes**
182
  ★ bert-chunker-3.5 | <= 400 | 0 | **94.1 ± 22.5**| 4.3 ± 3.5 | 18.1 ± 13.3 | 4.3 ± 3.5 |**O(N)** | **Yes**
183
  ★ bert-chunker-3.5 | <= 200 | 0 | 90.4 ± 26.2 | 7.7 ± 5.7 | 29.2 ± 17.9 | 7.6 ± 5.7 |**O(N)** | **Yes**
184
+ ## Future
185
+ This model is undertrained due to laziness and lack of money. I observed that the model is undertrained because the outputs from two non-overlapping windows show poor comparability in split point probabilities. It particularly undermines the performance when max_tokens_per_chunk is large. If the data were sufficient, I think cross-window probability comparability would be improved. This is corroborated by my preliminary experiments.
186
  ## Citation
187
  ```bibtex
188
  @article{bert-chunker,