Replace fold-0 numbers with the pooled 5-fold cross-validation
Browse files
README.md
CHANGED
|
@@ -49,17 +49,22 @@ Script: `scripts/multitask_train.py` in the narrative-lens repository; provenanc
|
|
| 49 |
|
| 50 |
## Measurements
|
| 51 |
|
| 52 |
-
Unseen authors (
|
|
|
|
| 53 |
|
| 54 |
-
| | student | teacher |
|
| 55 |
| --- | ---: | ---: |
|
| 56 |
-
| Proper person spans, exact F1 | 0.
|
| 57 |
-
| Nominal mentions, exact F1 | 0.
|
| 58 |
-
| Antecedent accuracy,
|
| 59 |
-
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
-
The
|
| 62 |
-
|
|
|
|
|
|
|
| 63 |
narrative-lens pipeline (the model trained on these works, so this shows the integration, not generalization):
|
| 64 |
supplied-cast pronoun resolution F1 0.848 (precision 0.861), explicit-mode speaker F1 0.699 (precision 0.787),
|
| 65 |
against 0.786 and 0.657 for v1.
|
|
|
|
| 49 |
|
| 50 |
## Measurements
|
| 51 |
|
| 52 |
+
Unseen authors (pooled over an author-disjoint 5-fold split over LitBank, PDNC, and the PDNC alias-list
|
| 53 |
+
documents; each fold's teacher also structures ~29,200 windows from the 100 unlabelled Gutenberg novels):
|
| 54 |
|
| 55 |
+
| | student (pooled) | teacher (range across folds) |
|
| 56 |
| --- | ---: | ---: |
|
| 57 |
+
| Proper person spans, exact F1 | 0.822 | 0.633-0.906 |
|
| 58 |
+
| Nominal mentions, exact F1 | 0.708 | 0.604-0.686 |
|
| 59 |
+
| Antecedent accuracy, proper mentions (within window) | 0.937 | - |
|
| 60 |
+
| Antecedent accuracy, nominal mentions (within window) | 0.859 | - |
|
| 61 |
+
| Antecedent accuracy, pronouns (within window) | 0.935 | 0.925-0.954 |
|
| 62 |
+
| Quotation speaker accuracy | 0.854 | 0.849-0.898 |
|
| 63 |
|
| 64 |
+
The student is close to or ahead of its own teacher on every head except speaker accuracy, where the
|
| 65 |
+
teacher stays a few points ahead. The teacher's token-head scores vary more across folds because
|
| 66 |
+
LitBank's entity and nominal labels are its only supervision, trained on one window grid for two
|
| 67 |
+
epochs against the student's two grids with LitBank repeated four times. LitBank development split through the
|
| 68 |
narrative-lens pipeline (the model trained on these works, so this shows the integration, not generalization):
|
| 69 |
supplied-cast pronoun resolution F1 0.848 (precision 0.861), explicit-mode speaker F1 0.699 (precision 0.787),
|
| 70 |
against 0.786 and 0.657 for v1.
|