Update README.md
Browse filesAdd metrics on ROG test data
README.md
CHANGED
|
@@ -16,7 +16,32 @@ metrics:
|
|
| 16 |
|
| 17 |
This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
|
| 18 |
|
| 19 |
-
It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
|
| 22 |
|
|
|
|
| 16 |
|
| 17 |
This model classifies individual 20ms frames of audio based on presence of filled pauses ("eee", "errm", ...).
|
| 18 |
|
| 19 |
+
It was trained on human-annotated Slovenian speech corpus ROG-Artur and achieves F1 of 0.95 for the positive class on
|
| 20 |
+
te test split of the same dataset.
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
# Evaluation
|
| 24 |
+
|
| 25 |
+
Although the output of the model is a series 0 or 1, describing their 20ms frames, the evaluation was done on
|
| 26 |
+
event level; spans of consecutive outputs 1 were bundled together into one event. When the true and predicted
|
| 27 |
+
events partially overlap, this is counted as a true positive.
|
| 28 |
+
|
| 29 |
+
## Evaluation on ROG corpus
|
| 30 |
+
|
| 31 |
+
The train and test data were obtained by resegmenting ROG corpus and using only segments with filled pauses. As a result,
|
| 32 |
+
no true negatives are present in the data and the behaviour of the negative class (i.e. no filled pause detected) is unpredictable.
|
| 33 |
+
```
|
| 34 |
+
precision recall f1-score support
|
| 35 |
+
|
| 36 |
+
0 0.531 0.123 0.200 211
|
| 37 |
+
1 0.907 0.987 0.946 1834
|
| 38 |
+
|
| 39 |
+
accuracy 0.898 2045
|
| 40 |
+
macro avg 0.719 0.555 0.573 2045
|
| 41 |
+
weighted avg 0.868 0.898 0.869 2045
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
## Evaluation on ParlaSpeech [HR](https://huggingface.co/datasets/classla/ParlaSpeech-HR) and [RS](https://huggingface.co/datasets/classla/ParlaSpeech-RS) corpora
|
| 45 |
|
| 46 |
Evaluation on 800 human-annotated instances ParlaSpeech-HR and ParlaSpeech-RS produced the following metrics:
|
| 47 |
|