| | --- |
| | license: mit |
| | tags: |
| | - token-classification |
| | - modernbert |
| | - orality |
| | - linguistics |
| | - multi-label |
| | language: |
| | - en |
| | metrics: |
| | - f1 |
| | base_model: |
| | - answerdotai/ModernBERT-base |
| | pipeline_tag: token-classification |
| | library_name: transformers |
| | datasets: |
| | - custom |
| | --- |
| | |
| | # Havelock Orality Token Classifier |
| |
|
| | ModernBERT-based token classifier for detecting **oral and literate markers** in text, based on Walter Ong's "Orality and Literacy" (1982). |
| |
|
| | This model performs multi-label span-level detection of 53 rhetorical marker types, where each token independently carries B/I/O labels per type β allowing overlapping spans (e.g. a token that is simultaneously part of a concessive and a nested clause). |
| |
|
| | ## Model Details |
| |
|
| | | Property | Value | |
| | |----------|-------| |
| | | Base model | `answerdotai/ModernBERT-base` | |
| | | Task | Multi-label token classification (independent B/I/O per type) | |
| | | Marker types | 53 (22 oral, 31 literate) | |
| | | Test macro F1 | **0.378** (per-type detection, binary positive = B or I) | |
| | | Training | 20 epochs, fp16 | |
| | | Regularization | Mixout (p=0.1) β stochastic L2 anchor to pretrained weights | |
| | | Loss | Per-type focal loss (Ξ³=2.0) with inverse-frequency OBI and type weights | |
| | | Min examples | 150 (types below this threshold excluded) | |
| |
|
| | ## Usage |
| | ```python |
| | import json |
| | import torch |
| | from transformers import AutoModel, AutoTokenizer |
| | from huggingface_hub import hf_hub_download |
| | |
| | model_name = "HavelockAI/bert-token-classifier" |
| | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| | model = AutoModel.from_pretrained(model_name, trust_remote_code=True) |
| | model.eval() |
| | |
| | # Load marker type map |
| | type_map_path = hf_hub_download(model_name, "type_to_idx.json") |
| | type_to_idx = json.loads(open(type_map_path).read()) |
| | idx_to_type = {v: k for k, v in type_to_idx.items()} |
| | |
| | text = "Tell me, O Muse, of that ingenious hero who travelled far and wide" |
| | inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128) |
| | |
| | with torch.no_grad(): |
| | logits = model(**inputs) # (1, seq_len, num_types, 3) |
| | preds = logits.argmax(dim=-1) # (1, seq_len, num_types) |
| | |
| | tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0]) |
| | for i, token in enumerate(tokens): |
| | active = [ |
| | f"{idx_to_type[t]}={'OBI'[v]}" |
| | for t, v in enumerate(preds[0, i].tolist()) |
| | if v > 0 |
| | ] |
| | if active: |
| | print(f"{token:15} {', '.join(active)}") |
| | ``` |
| |
|
| | > **Note:** This model uses a custom architecture (`HavelockTokenClassifier`) with independent B/I/O heads per marker type, enabling overlapping span detection. Loading requires `trust_remote_code=True`. |
| |
|
| | ## Training Data |
| |
|
| | - Sources: Project Gutenberg, textfiles.com, Reddit, Wikipedia talk pages |
| | - Types with fewer than 150 annotated spans are excluded from training |
| | - Multi-label BIO annotation: tokens can carry labels for multiple overlapping marker types simultaneously |
| |
|
| | ## Marker Types (53) |
| |
|
| | ### Oral Markers (22 types) |
| |
|
| | Characteristics of oral tradition and spoken discourse: |
| |
|
| | | Category | Markers | |
| | |----------|---------| |
| | | **Address & Interaction** | vocative, imperative, second_person, inclusive_we, rhetorical_question, phatic_check, phatic_filler | |
| | | **Repetition & Pattern** | anaphora, parallelism, tricolon, lexical_repetition, antithesis | |
| | | **Conjunction** | simple_conjunction | |
| | | **Formulas** | discourse_formula, intensifier_doubling | |
| | | **Narrative** | named_individual, specific_place, temporal_anchor, sensory_detail, embodied_action, everyday_example | |
| | | **Performance** | self_correction | |
| |
|
| | ### Literate Markers (31 types) |
| |
|
| | Characteristics of written, analytical discourse: |
| |
|
| | | Category | Markers | |
| | |----------|---------| |
| | | **Abstraction** | nominalization, abstract_noun, conceptual_metaphor, categorical_statement | |
| | | **Syntax** | nested_clauses, relative_chain, conditional, concessive, temporal_embedding, causal_explicit | |
| | | **Hedging** | epistemic_hedge, probability, evidential, qualified_assertion, concessive_connector | |
| | | **Impersonality** | agentless_passive, agent_demoted, institutional_subject, objectifying_stance | |
| | | **Scholarly apparatus** | citation, cross_reference, metadiscourse, definitional_move | |
| | | **Technical** | technical_term, technical_abbreviation, enumeration, list_structure | |
| | | **Connectives** | contrastive, additive_formal | |
| | | **Setting** | concrete_setting, aside | |
| | |
| | ## Evaluation |
| | |
| | Per-type detection F1 on test set (binary: B or I = positive, O = negative): |
| | |
| | <details><summary>Click to show per-marker precision/recall/F1/support</summary> |
| | ``` |
| | Type Prec Rec F1 Sup |
| | ======================================================================== |
| | literate_abstract_noun 0.190 0.325 0.240 381 |
| | literate_additive_formal 0.246 0.556 0.341 27 |
| | literate_agent_demoted 0.404 0.368 0.386 304 |
| | literate_agentless_passive 0.575 0.607 0.591 1133 |
| | literate_aside 0.379 0.429 0.403 436 |
| | literate_categorical_statement 0.267 0.146 0.189 514 |
| | literate_causal_explicit 0.227 0.279 0.251 190 |
| | literate_citation 0.639 0.556 0.595 372 |
| | literate_conceptual_metaphor 0.310 0.364 0.335 415 |
| | literate_concessive 0.499 0.470 0.484 502 |
| | literate_concessive_connector 0.455 0.408 0.430 49 |
| | literate_concrete_setting 0.241 0.125 0.165 407 |
| | literate_conditional 0.369 0.630 0.466 760 |
| | literate_contrastive 0.310 0.428 0.360 341 |
| | literate_cross_reference 0.386 0.524 0.444 42 |
| | literate_definitional_move 0.395 0.185 0.252 81 |
| | literate_enumeration 0.495 0.483 0.489 775 |
| | literate_epistemic_hedge 0.421 0.481 0.449 445 |
| | literate_evidential 0.625 0.360 0.457 472 |
| | literate_institutional_subject 0.332 0.326 0.329 282 |
| | literate_list_structure 0.338 0.523 0.411 86 |
| | literate_metadiscourse 0.140 0.393 0.206 135 |
| | literate_nested_clauses 0.091 0.246 0.133 1169 |
| | literate_nominalization 0.499 0.612 0.549 991 |
| | literate_objectifying_stance 0.635 0.365 0.464 167 |
| | literate_probability 0.432 0.593 0.500 27 |
| | literate_qualified_assertion 0.143 0.100 0.118 40 |
| | literate_relative_chain 0.382 0.507 0.436 1424 |
| | literate_technical_abbreviation 0.667 0.711 0.688 225 |
| | literate_technical_term 0.280 0.375 0.321 715 |
| | literate_temporal_embedding 0.228 0.259 0.242 526 |
| | oral_anaphora 0.800 0.028 0.054 287 |
| | oral_antithesis 0.249 0.238 0.243 412 |
| | oral_discourse_formula 0.340 0.408 0.371 557 |
| | oral_embodied_action 0.280 0.391 0.326 425 |
| | oral_everyday_example 0.333 0.156 0.212 404 |
| | oral_imperative 0.591 0.662 0.625 293 |
| | oral_inclusive_we 0.516 0.632 0.568 622 |
| | oral_intensifier_doubling 0.680 0.200 0.309 85 |
| | oral_lexical_repetition 0.404 0.254 0.312 173 |
| | oral_named_individual 0.441 0.749 0.556 770 |
| | oral_parallelism 0.741 0.110 0.191 182 |
| | oral_phatic_check 0.611 0.733 0.667 30 |
| | oral_phatic_filler 0.174 0.409 0.244 93 |
| | oral_rhetorical_question 0.509 0.692 0.586 905 |
| | oral_second_person 0.576 0.552 0.564 811 |
| | oral_self_correction 0.158 0.235 0.189 51 |
| | oral_sensory_detail 0.285 0.169 0.212 461 |
| | oral_simple_conjunction 0.179 0.102 0.130 98 |
| | oral_specific_place 0.556 0.705 0.622 424 |
| | oral_temporal_anchor 0.410 0.559 0.473 546 |
| | oral_tricolon 0.299 0.119 0.171 553 |
| | oral_vocative 0.652 0.747 0.696 158 |
| | ======================================================================== |
| | Macro avg (types w/ support) 0.378 |
| | ``` |
| | |
| | </details> |
| | |
| | **Missing labels (test set):** 0/53 β all types detected at least once. |
| | |
| | Notable patterns: |
| | - **Strong performers** (F1 > 0.5): vocative (0.696), technical_abbreviation (0.688), phatic_check (0.667), imperative (0.625), specific_place (0.622), citation (0.595), agentless_passive (0.591), rhetorical_question (0.586), inclusive_we (0.568), second_person (0.564), named_individual (0.556), nominalization (0.549), probability (0.500) |
| | - **Weak performers** (F1 < 0.2): anaphora (0.054), qualified_assertion (0.118), simple_conjunction (0.130), nested_clauses (0.133), concrete_setting (0.165), tricolon (0.171), categorical_statement (0.189), self_correction (0.189), parallelism (0.191) |
| | - **Precision-recall tradeoff**: Most types show balanced precision/recall. Notable exceptions include `anaphora` (0.800 precision / 0.028 recall), `parallelism` (0.741 / 0.110), and `intensifier_doubling` (0.680 / 0.200), which remain high-precision but very low-recall. |
| | |
| | ## Architecture |
| | |
| | Custom `MultiLabelTokenClassifier` with independent B/I/O heads per marker type: |
| | ``` |
| | ModernBERT (answerdotai/ModernBERT-base) |
| | βββ Dropout (p=0.1) |
| | βββ Linear (hidden_size β num_types Γ 3) |
| | βββ Reshape to (batch, seq, num_types, 3) |
| | ``` |
| | |
| | Each marker type gets an independent 3-way O/B/I classification, so a token can simultaneously carry labels for multiple overlapping marker types. Types share the full backbone representation but make independent predictions. |
| | |
| | ### Regularization |
| | |
| | - **Mixout** (p=0.1): During training, each backbone weight element has a 10% chance of being replaced by its pretrained value per forward pass, acting as a stochastic L2 anchor that prevents representation drift (Lee et al., 2019) |
| | - **Per-type focal loss** (Ξ³=2.0): Focuses learning on hard examples, reducing the contribution of easy negatives |
| | - **Inverse-frequency type weights**: Rare marker types receive higher loss weighting |
| | - **Inverse-frequency OBI weights**: B and I classes upweighted relative to dominant O class |
| | - **Weighted random sampling**: Examples containing rarer markers sampled more frequently |
| | |
| | ### Initialization |
| | |
| | Fine-tuned from `answerdotai/ModernBERT-base`. Backbone linear layers wrapped with Mixout during training (frozen pretrained copy used as anchor). The classification head is randomly initialized: |
| | ``` |
| | backbone.* layers β loaded from pretrained, anchored via Mixout |
| | classifier.weight β randomly initialized |
| | classifier.bias β randomly initialized |
| | ``` |
| | |
| | ## Limitations |
| | |
| | - **Near-zero recall types**: `anaphora` (0.028 recall), `simple_conjunction` (0.102), `parallelism` (0.110), and `tricolon` (0.119) are rarely detected despite being present in training data |
| | - **Low-precision types**: `nested_clauses` (0.091), `metadiscourse` (0.140), and `qualified_assertion` (0.143) have precision below 0.15, meaning most predictions for those types are false positives |
| | - **Context window**: 128 tokens max; longer spans may be truncated |
| | - **Domain**: Trained primarily on historical/literary texts; may underperform on modern social media |
| | - **Subjectivity**: Some marker boundaries are inherently ambiguous |
| | |
| | ## Citation |
| | ```bibtex |
| | @misc{havelock2026token, |
| | title={Havelock Orality Token Classifier}, |
| | author={Havelock AI}, |
| | year={2026}, |
| | url={https://huggingface.co/HavelockAI/bert-token-classifier} |
| | } |
| | ``` |
| | |
| | ## References |
| | |
| | - Ong, Walter J. *Orality and Literacy: The Technologizing of the Word*. Routledge, 1982. |
| | - Lee, C. et al. "Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models." ICLR 2020. |
| | - Warner, A. et al. "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." 2024. |
| | |
| | --- |
| |
|
| | *Trained: February 2026* |
