davidcondrey commited on
Commit
28ea356
·
verified ·
1 Parent(s): 8fd1c33

Replace fold-0 numbers with the pooled 5-fold cross-validation

Browse files
Files changed (1) hide show
  1. README.md +13 -8
README.md CHANGED
@@ -49,17 +49,22 @@ Script: `scripts/multitask_train.py` in the narrative-lens repository; provenanc
49
 
50
  ## Measurements
51
 
52
- Unseen authors (fold 0 of an author-disjoint 5-fold split over LitBank and PDNC):
 
53
 
54
- | | student | teacher |
55
  | --- | ---: | ---: |
56
- | Proper person spans, exact F1 | 0.812 | 0.829 |
57
- | Nominal mentions, exact F1 | 0.704 | 0.641 |
58
- | Antecedent accuracy, pronouns (within window) | 0.908 | 0.923 |
59
- | Quotation speaker accuracy | 0.814 | 0.863 |
 
 
60
 
61
- The fold-0 figures above are for the v1 student (same recipe without alias lists and Gutenberg self-training);
62
- the v2 five-fold cross-validation is in progress and will replace them. LitBank development split through the
 
 
63
  narrative-lens pipeline (the model trained on these works, so this shows the integration, not generalization):
64
  supplied-cast pronoun resolution F1 0.848 (precision 0.861), explicit-mode speaker F1 0.699 (precision 0.787),
65
  against 0.786 and 0.657 for v1.
 
49
 
50
  ## Measurements
51
 
52
+ Unseen authors (pooled over an author-disjoint 5-fold split over LitBank, PDNC, and the PDNC alias-list
53
+ documents; each fold's teacher also structures ~29,200 windows from the 100 unlabelled Gutenberg novels):
54
 
55
+ | | student (pooled) | teacher (range across folds) |
56
  | --- | ---: | ---: |
57
+ | Proper person spans, exact F1 | 0.822 | 0.633-0.906 |
58
+ | Nominal mentions, exact F1 | 0.708 | 0.604-0.686 |
59
+ | Antecedent accuracy, proper mentions (within window) | 0.937 | - |
60
+ | Antecedent accuracy, nominal mentions (within window) | 0.859 | - |
61
+ | Antecedent accuracy, pronouns (within window) | 0.935 | 0.925-0.954 |
62
+ | Quotation speaker accuracy | 0.854 | 0.849-0.898 |
63
 
64
+ The student is close to or ahead of its own teacher on every head except speaker accuracy, where the
65
+ teacher stays a few points ahead. The teacher's token-head scores vary more across folds because
66
+ LitBank's entity and nominal labels are its only supervision, trained on one window grid for two
67
+ epochs against the student's two grids with LitBank repeated four times. LitBank development split through the
68
  narrative-lens pipeline (the model trained on these works, so this shows the integration, not generalization):
69
  supplied-cast pronoun resolution F1 0.848 (precision 0.861), explicit-mode speaker F1 0.699 (precision 0.787),
70
  against 0.786 and 0.657 for v1.