HavelockAI
/

bert-marker-subtype

@@ -25,10 +25,10 @@ model-index:
       name: Marker Subtype Classification
     metrics:
     - type: f1
-      value: 0.4704
       name: F1 (macro)
     - type: accuracy
-      value: 0.515
       name: Accuracy
 ---
@@ -46,8 +46,8 @@ This is the finest level of the Havelock span classification hierarchy. Given a
 | Architecture | `BertForSequenceClassification` |
 | Task | Multi-class classification (71 classes) |
 | Max sequence length | 128 tokens |
-| Best F1 (macro) | **0.4704** |
-| Best Accuracy | **0.515** |
 | Parameters | ~109M |
 ## Usage
@@ -107,24 +107,31 @@ A stratified 80/20 train/test split was used (random seed 42). The test set cont
 | Parameter | Value |
 |-----------|-------|
-| Epochs | 3 |
-| Batch size | 8 |
-| Learning rate | 2e-5 |
 | Optimizer | AdamW |
 | LR schedule | Linear warmup (10% of total steps) |
 | Gradient clipping | 1.0 |
-| Loss | Cross-entropy |
 | Min examples per class | 15 |
 ### Training Metrics
 | Epoch | Loss | Accuracy | F1 (macro) |
 |-------|------|----------|------------|
-| 1 | 3.2554 | 0.4210 | 0.3060 |
-| 2 | 2.0844 | 0.5033 | 0.4345 |
-| 3 | 1.5922 | 0.5154 | 0.4704 |
-Best checkpoint selected by F1 at epoch 3. Loss still declining steeply.
 ### Test Set Classification Report
@@ -132,88 +139,88 @@ Best checkpoint selected by F1 at epoch 3. Loss still declining steeply.
 ```
                         precision    recall  f1-score   support
-         abstract_noun      0.262     0.333     0.294       144
-       additive_formal      0.250     0.038     0.067        26
-         agent_demoted      0.944     0.548     0.694        31
-     agentless_passive      0.458     0.619     0.526       105
-          alliteration      0.400     0.133     0.200        30
-              anaphora      0.468     0.659     0.547        88
-            antithesis      0.575     0.742     0.648        31
-                 aside      0.467     0.127     0.200        55
-             assonance      0.744     0.970     0.842        33
-             asyndeton      0.867     0.433     0.578        30
-     audience_response      0.800     0.533     0.640        30
- categorical_statement      0.362     0.388     0.374        98
-          causal_chain      0.472     0.625     0.538        80
-       causal_explicit      0.400     0.406     0.403        69
-              citation      0.494     0.612     0.547        67
-   conceptual_metaphor      0.235     0.055     0.089        73
-            concessive      0.677     0.739     0.707        88
   concessive_connector      0.920     0.742     0.821        31
-           conditional      0.627     0.671     0.648       155
-        conflict_frame      0.800     0.774     0.787        31
-           contrastive      0.390     0.595     0.471       116
-       cross_reference      0.429     0.353     0.387        34
-     definitional_move      0.429     0.077     0.130        39
-     discourse_formula      0.499     0.703     0.583       276
-        dramatic_pause      0.833     0.806     0.820        31
-       embodied_action      0.286     0.377     0.325        69
-           enumeration      0.504     0.694     0.584        85
-       epistemic_hedge      0.429     0.624     0.508       101
-            epistrophe      0.763     0.906     0.829        32
-               epithet      0.429     0.444     0.436        27
-      everyday_example      0.432     0.390     0.410        41
-            evidential      0.608     0.574     0.590        54
-    footnote_reference      1.000     0.133     0.235        15
-            imperative      0.617     0.760     0.681       146
-          inclusive_we      0.579     0.700     0.634       120
- institutional_subject      0.586     0.548     0.567        31
-  intensifier_doubling      0.792     0.633     0.704        30
-    lexical_repetition      0.535     0.649     0.587        94
-        list_structure      0.300     0.167     0.214        36
-         metadiscourse      0.310     0.310     0.310        87
-methodological_framing      0.000     0.000     0.000        32
-      named_individual      0.446     0.527     0.483        55
-        nested_clauses      0.375     0.172     0.236        87
-        nominalization      0.336     0.333     0.335       120
-   objectifying_stance      0.250     0.023     0.043        43
-           parallelism      0.250     0.052     0.086        58
-          phatic_check      1.000     0.286     0.444        21
-         phatic_filler      0.529     0.300     0.383        30
-          polysyndeton      0.675     0.844     0.750        32
-           probability      0.571     0.327     0.416        49
-               proverb      0.222     0.065     0.100        31
-   qualified_assertion      0.286     0.100     0.148        60
-               refrain      0.895     0.567     0.694        30
-        relative_chain      0.504     0.600     0.548       115
-     religious_formula      0.917     0.688     0.786        32
-   rhetorical_question      0.614     0.820     0.702       161
-                 rhyme      0.545     0.562     0.554        32
-                rhythm      0.839     0.812     0.825        32
-         second_person      0.557     0.600     0.578       235
-       self_correction      0.895     0.567     0.694        30
-        sensory_detail      0.000     0.000     0.000        37
-    simple_conjunction      0.667     0.049     0.091        41
-        specific_place      1.000     0.038     0.074        26
-technical_abbreviation      1.000     0.053     0.100        19
-        technical_term      0.489     0.571     0.527       161
-       temporal_anchor      0.471     0.490     0.480        49
-    temporal_embedding      0.448     0.481     0.464        81
-third_person_reference      0.917     0.710     0.800        31
-              tricolon      0.656     0.700     0.677        30
-               us_them      0.882     0.484     0.625        31
-              vocative      0.593     0.603     0.598        58
-              accuracy                          0.515      4608
-             macro avg      0.561     0.465     0.470      4608
-          weighted avg      0.512     0.515     0.490      4608
 ```
 </details>
-**Top performing subtypes (F1 > 0.75):** `assonance` (0.842), `epistrophe` (0.829), `rhythm` (0.825), `concessive_connector` (0.821), `dramatic_pause` (0.820), `third_person_reference` (0.800), `conflict_frame` (0.787), `religious_formula` (0.786), `polysyndeton` (0.750).
-**Near-zero F1 subtypes:** `methodological_framing` (0.000), `sensory_detail` (0.000), `specific_place` (0.074), `parallelism` (0.086), `conceptual_metaphor` (0.089), `objectifying_stance` (0.043), `simple_conjunction` (0.091), `technical_abbreviation` (0.100), `proverb` (0.100). These tend to be either semantically diffuse classes or classes with very low support.
 ## Class Distribution
@@ -228,10 +235,10 @@ The test set exhibits significant imbalance across 71 classes:
 ## Limitations
-- **Severely undertrained**: 3 epochs with loss at 1.59 and still falling steeply. This model has the most headroom for improvement of the three span classifiers.
 - **71-way classification on ~23k spans**: The data budget per class is thin, particularly for classes near the 15-example minimum. More data or class consolidation would help.
 - **Semantic overlap**: Some subtypes are difficult to distinguish from surface text alone (e.g., `parallelism` vs `anaphora` vs `tricolon`; `epistemic_hedge` vs `qualified_assertion` vs `probability`). The model may benefit from hierarchical classification that conditions on type-level predictions.
-- **Recall-precision tradeoff**: Many rare classes show high precision but very low recall (e.g., `footnote_reference`: P=1.000, R=0.133), suggesting the model learns narrow prototypes but misses variation.
 - **Span-level only**: Requires pre-extracted spans. Does not detect boundaries.
 - **128-token context window**: Longer spans are truncated.
@@ -245,9 +252,9 @@ The 71 subtypes represent the full granularity of the Havelock taxonomy, operati
 |-------|------|---------|-----|
 | [`HavelockAI/bert-marker-category`](https://huggingface.co/HavelockAI/bert-marker-category) | Binary (oral/literate) | 2 | 0.875 |
 | [`HavelockAI/bert-marker-type`](https://huggingface.co/HavelockAI/bert-marker-type) | Functional type | 25 | 0.449 |
-| **This model** | Fine-grained subtype | 71 | 0.470 |
 | [`HavelockAI/bert-orality-regressor`](https://huggingface.co/HavelockAI/bert-orality-regressor) | Document-level score | Regression | MAE 0.079 |
-| [`HavelockAI/bert-token-classifier`](https://huggingface.co/HavelockAI/bert-token-classifier) | Span detection (BIO) | 145 | 0.461 |
 ## Citation
 ```bibtex
@@ -265,5 +272,4 @@ The 71 subtypes represent the full granularity of the Havelock taxonomy, operati
 ---
-*Trained: February 2026*
-*Model version: da931b4a · Trained: February 2026*

       name: Marker Subtype Classification
     metrics:
     - type: f1
+      value: 0.5320
       name: F1 (macro)
     - type: accuracy
+      value: 0.517
       name: Accuracy
 ---
 | Architecture | `BertForSequenceClassification` |
 | Task | Multi-class classification (71 classes) |
 | Max sequence length | 128 tokens |
+| Best F1 (macro) | **0.5320** |
+| Best Accuracy | **0.517** |
 | Parameters | ~109M |
 ## Usage
 | Parameter | Value |
 |-----------|-------|
+| Epochs | 10 |
+| Batch size | 256 |
+| Learning rate | 1.5e-4 |
 | Optimizer | AdamW |
 | LR schedule | Linear warmup (10% of total steps) |
 | Gradient clipping | 1.0 |
+| Loss | Cross-entropy with class weights (range 0.23–4.33) |
 | Min examples per class | 15 |
 ### Training Metrics
 | Epoch | Loss | Accuracy | F1 (macro) |
 |-------|------|----------|------------|
+| 1 | 3.7795 | 0.3249 | 0.1618 |
+| 2 | 2.3703 | 0.4918 | 0.4254 |
+| 3 | 1.5864 | 0.5139 | 0.4964 |
+| 4 | 1.0582 | 0.5195 | 0.5238 |
+| 5 | 0.6955 | 0.5189 | 0.5196 |
+| 6 | 0.4761 | 0.5148 | 0.5227 |
+| 7 | 0.3279 | 0.5178 | **0.5320** |
+| 8 | 0.2419 | 0.5119 | 0.5213 |
+| 9 | 0.1885 | 0.5206 | 0.5283 |
+| 10 | 0.1454 | 0.5169 | 0.5250 |
+Best checkpoint selected by F1 at epoch 7. Accuracy plateaus from epoch 3 onward while F1 continues improving through rare-class gains.
 ### Test Set Classification Report
 ```
                         precision    recall  f1-score   support
+         abstract_noun      0.315     0.312     0.314       144
+       additive_formal      0.478     0.423     0.449        26
+         agent_demoted      0.909     0.645     0.755        31
+     agentless_passive      0.533     0.543     0.538       105
+          alliteration      0.632     0.400     0.490        30
+              anaphora      0.526     0.466     0.494        88
+            antithesis      0.641     0.806     0.714        31
+                 aside      0.261     0.218     0.238        55
+             assonance      0.917     1.000     0.957        33
+             asyndeton      0.677     0.700     0.689        30
+     audience_response      0.808     0.700     0.750        30
+ categorical_statement      0.329     0.245     0.281        98
+          causal_chain      0.442     0.425     0.433        80
+       causal_explicit      0.406     0.406     0.406        69
+              citation      0.646     0.627     0.636        67
+   conceptual_metaphor      0.298     0.233     0.262        73
+            concessive      0.690     0.659     0.674        88
   concessive_connector      0.920     0.742     0.821        31
+           conditional      0.620     0.684     0.650       155
+        conflict_frame      0.833     0.806     0.820        31
+           contrastive      0.463     0.543     0.500       116
+       cross_reference      0.538     0.412     0.467        34
+     definitional_move      0.300     0.308     0.304        39
+     discourse_formula      0.559     0.565     0.562       276
+        dramatic_pause      0.781     0.806     0.794        31
+       embodied_action      0.333     0.362     0.347        69
+           enumeration      0.607     0.600     0.604        85
+       epistemic_hedge      0.491     0.554     0.521       101
+            epistrophe      0.867     0.812     0.839        32
+               epithet      0.424     0.519     0.467        27
+      everyday_example      0.361     0.317     0.338        41
+            evidential      0.526     0.556     0.541        54
+    footnote_reference      0.615     0.533     0.571        15
+            imperative      0.659     0.753     0.703       146
+          inclusive_we      0.613     0.608     0.611       120
+ institutional_subject      0.600     0.581     0.590        31
+  intensifier_doubling      0.833     0.667     0.741        30
+    lexical_repetition      0.486     0.564     0.522        94
+        list_structure      0.286     0.278     0.282        36
+         metadiscourse      0.320     0.276     0.296        87
+methodological_framing      0.269     0.219     0.241        32
+      named_individual      0.364     0.436     0.397        55
+        nested_clauses      0.370     0.310     0.338        87
+        nominalization      0.377     0.433     0.403       120
+   objectifying_stance      0.125     0.233     0.163        43
+           parallelism      0.218     0.293     0.250        58
+          phatic_check      0.636     0.667     0.651        21
+         phatic_filler      0.333     0.400     0.364        30
+          polysyndeton      0.964     0.844     0.900        32
+           probability      0.574     0.551     0.562        49
+               proverb      0.304     0.226     0.259        31
+   qualified_assertion      0.219     0.233     0.226        60
+               refrain      0.818     0.600     0.692        30
+        relative_chain      0.558     0.504     0.530       115
+     religious_formula      0.840     0.656     0.737        32
+   rhetorical_question      0.686     0.745     0.714       161
+                 rhyme      0.480     0.375     0.421        32
+                rhythm      0.778     0.875     0.824        32
+         second_person      0.543     0.596     0.568       235
+       self_correction      0.826     0.633     0.717        30
+        sensory_detail      0.387     0.324     0.353        37
+    simple_conjunction      0.222     0.195     0.208        41
+        specific_place      0.526     0.385     0.444        26
+technical_abbreviation      0.278     0.263     0.270        19
+        technical_term      0.615     0.466     0.530       161
+       temporal_anchor      0.404     0.429     0.416        49
+    temporal_embedding      0.438     0.519     0.475        81
+third_person_reference      0.788     0.839     0.812        31
+              tricolon      0.607     0.567     0.586        30
+               us_them      0.606     0.645     0.625        31
+              vocative      0.643     0.621     0.632        58
+              accuracy                          0.517      4608
+             macro avg      0.540     0.517     0.525      4608
+          weighted avg      0.522     0.517     0.517      4608
 ```
 </details>
+**Top performing subtypes (F1 > 0.75):** `assonance` (0.957), `polysyndeton` (0.900), `epistrophe` (0.839), `rhythm` (0.824), `concessive_connector` (0.821), `conflict_frame` (0.820), `third_person_reference` (0.812), `dramatic_pause` (0.794), `agent_demoted` (0.755), `audience_response` (0.750).
+**Weakest subtypes (F1 < 0.25):** `objectifying_stance` (0.163), `simple_conjunction` (0.208), `qualified_assertion` (0.226), `aside` (0.238), `methodological_framing` (0.241), `parallelism` (0.250). These tend to be semantically diffuse classes that overlap heavily with neighbouring subtypes.
 ## Class Distribution
 ## Limitations
+- **Accuracy plateau with F1 headroom**: Accuracy saturated around 0.52 from epoch 3 while F1 continued climbing through epoch 7, suggesting the model is still finding better decision boundaries for rare classes. Further training with LR decay or curriculum strategies may help.
 - **71-way classification on ~23k spans**: The data budget per class is thin, particularly for classes near the 15-example minimum. More data or class consolidation would help.
 - **Semantic overlap**: Some subtypes are difficult to distinguish from surface text alone (e.g., `parallelism` vs `anaphora` vs `tricolon`; `epistemic_hedge` vs `qualified_assertion` vs `probability`). The model may benefit from hierarchical classification that conditions on type-level predictions.
+- **Recall-precision tradeoff**: Many rare classes show high precision but lower recall (e.g., `polysyndeton`: P=0.964, R=0.844; `agent_demoted`: P=0.909, R=0.645), suggesting the model learns narrow prototypes but misses variation.
 - **Span-level only**: Requires pre-extracted spans. Does not detect boundaries.
 - **128-token context window**: Longer spans are truncated.
 |-------|------|---------|-----|
 | [`HavelockAI/bert-marker-category`](https://huggingface.co/HavelockAI/bert-marker-category) | Binary (oral/literate) | 2 | 0.875 |
 | [`HavelockAI/bert-marker-type`](https://huggingface.co/HavelockAI/bert-marker-type) | Functional type | 25 | 0.449 |
+| **This model** | Fine-grained subtype | 71 | 0.532 |
 | [`HavelockAI/bert-orality-regressor`](https://huggingface.co/HavelockAI/bert-orality-regressor) | Document-level score | Regression | MAE 0.079 |
+| [`HavelockAI/bert-token-classifier`](https://huggingface.co/HavelockAI/bert-token-classifier) | Span detection (BIO) | 145 | 0.500 |
 ## Citation
 ```bibtex
 ---
+*Model version: da931b4a · Trained: February 2026*

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5a1f8420254999b58763469bd26ef2ba803a70ee980986aca9881f290dd9bb4
 size 438170868

 version https://git-lfs.github.com/spec/v1
+oid sha256:908751e3d1db4b122a3c05ea81d50dfbef7cacda40e601e6b09905b8aa7fb99f
 size 438170868