--- license: mit tags: - token-classification - modernbert - orality - linguistics - multi-label language: - en metrics: - f1 base_model: - answerdotai/ModernBERT-base pipeline_tag: token-classification library_name: transformers datasets: - custom --- # Havelock Orality Token Classifier ModernBERT-based token classifier for detecting **oral and literate markers** in text, based on Walter Ong's "Orality and Literacy" (1982). This model performs multi-label span-level detection of 53 rhetorical marker types, where each token independently carries B/I/O labels per type — allowing overlapping spans (e.g. a token that is simultaneously part of a concessive and a nested clause). ## Model Details | Property | Value | |----------|-------| | Base model | `answerdotai/ModernBERT-base` | | Task | Multi-label token classification (independent B/I/O per type) | | Marker types | 53 (22 oral, 31 literate) | | Test macro F1 | **0.378** (per-type detection, binary positive = B or I) | | Training | 20 epochs, fp16 | | Regularization | Mixout (p=0.1) — stochastic L2 anchor to pretrained weights | | Loss | Per-type focal loss (γ=2.0) with inverse-frequency OBI and type weights | | Min examples | 150 (types below this threshold excluded) | ## Usage ```python import json import torch from transformers import AutoModel, AutoTokenizer from huggingface_hub import hf_hub_download model_name = "HavelockAI/bert-token-classifier" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModel.from_pretrained(model_name, trust_remote_code=True) model.eval() # Load marker type map type_map_path = hf_hub_download(model_name, "type_to_idx.json") type_to_idx = json.loads(open(type_map_path).read()) idx_to_type = {v: k for k, v in type_to_idx.items()} text = "Tell me, O Muse, of that ingenious hero who travelled far and wide" inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128) with torch.no_grad(): logits = model(**inputs) # (1, seq_len, num_types, 3) preds = logits.argmax(dim=-1) # (1, seq_len, num_types) tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0]) for i, token in enumerate(tokens): active = [ f"{idx_to_type[t]}={'OBI'[v]}" for t, v in enumerate(preds[0, i].tolist()) if v > 0 ] if active: print(f"{token:15} {', '.join(active)}") ``` > **Note:** This model uses a custom architecture (`HavelockTokenClassifier`) with independent B/I/O heads per marker type, enabling overlapping span detection. Loading requires `trust_remote_code=True`. ## Training Data - Sources: Project Gutenberg, textfiles.com, Reddit, Wikipedia talk pages - Types with fewer than 150 annotated spans are excluded from training - Multi-label BIO annotation: tokens can carry labels for multiple overlapping marker types simultaneously ## Marker Types (53) ### Oral Markers (22 types) Characteristics of oral tradition and spoken discourse: | Category | Markers | |----------|---------| | **Address & Interaction** | vocative, imperative, second_person, inclusive_we, rhetorical_question, phatic_check, phatic_filler | | **Repetition & Pattern** | anaphora, parallelism, tricolon, lexical_repetition, antithesis | | **Conjunction** | simple_conjunction | | **Formulas** | discourse_formula, intensifier_doubling | | **Narrative** | named_individual, specific_place, temporal_anchor, sensory_detail, embodied_action, everyday_example | | **Performance** | self_correction | ### Literate Markers (31 types) Characteristics of written, analytical discourse: | Category | Markers | |----------|---------| | **Abstraction** | nominalization, abstract_noun, conceptual_metaphor, categorical_statement | | **Syntax** | nested_clauses, relative_chain, conditional, concessive, temporal_embedding, causal_explicit | | **Hedging** | epistemic_hedge, probability, evidential, qualified_assertion, concessive_connector | | **Impersonality** | agentless_passive, agent_demoted, institutional_subject, objectifying_stance | | **Scholarly apparatus** | citation, cross_reference, metadiscourse, definitional_move | | **Technical** | technical_term, technical_abbreviation, enumeration, list_structure | | **Connectives** | contrastive, additive_formal | | **Setting** | concrete_setting, aside | ## Evaluation Per-type detection F1 on test set (binary: B or I = positive, O = negative):
Click to show per-marker precision/recall/F1/support ``` Type Prec Rec F1 Sup ======================================================================== literate_abstract_noun 0.190 0.325 0.240 381 literate_additive_formal 0.246 0.556 0.341 27 literate_agent_demoted 0.404 0.368 0.386 304 literate_agentless_passive 0.575 0.607 0.591 1133 literate_aside 0.379 0.429 0.403 436 literate_categorical_statement 0.267 0.146 0.189 514 literate_causal_explicit 0.227 0.279 0.251 190 literate_citation 0.639 0.556 0.595 372 literate_conceptual_metaphor 0.310 0.364 0.335 415 literate_concessive 0.499 0.470 0.484 502 literate_concessive_connector 0.455 0.408 0.430 49 literate_concrete_setting 0.241 0.125 0.165 407 literate_conditional 0.369 0.630 0.466 760 literate_contrastive 0.310 0.428 0.360 341 literate_cross_reference 0.386 0.524 0.444 42 literate_definitional_move 0.395 0.185 0.252 81 literate_enumeration 0.495 0.483 0.489 775 literate_epistemic_hedge 0.421 0.481 0.449 445 literate_evidential 0.625 0.360 0.457 472 literate_institutional_subject 0.332 0.326 0.329 282 literate_list_structure 0.338 0.523 0.411 86 literate_metadiscourse 0.140 0.393 0.206 135 literate_nested_clauses 0.091 0.246 0.133 1169 literate_nominalization 0.499 0.612 0.549 991 literate_objectifying_stance 0.635 0.365 0.464 167 literate_probability 0.432 0.593 0.500 27 literate_qualified_assertion 0.143 0.100 0.118 40 literate_relative_chain 0.382 0.507 0.436 1424 literate_technical_abbreviation 0.667 0.711 0.688 225 literate_technical_term 0.280 0.375 0.321 715 literate_temporal_embedding 0.228 0.259 0.242 526 oral_anaphora 0.800 0.028 0.054 287 oral_antithesis 0.249 0.238 0.243 412 oral_discourse_formula 0.340 0.408 0.371 557 oral_embodied_action 0.280 0.391 0.326 425 oral_everyday_example 0.333 0.156 0.212 404 oral_imperative 0.591 0.662 0.625 293 oral_inclusive_we 0.516 0.632 0.568 622 oral_intensifier_doubling 0.680 0.200 0.309 85 oral_lexical_repetition 0.404 0.254 0.312 173 oral_named_individual 0.441 0.749 0.556 770 oral_parallelism 0.741 0.110 0.191 182 oral_phatic_check 0.611 0.733 0.667 30 oral_phatic_filler 0.174 0.409 0.244 93 oral_rhetorical_question 0.509 0.692 0.586 905 oral_second_person 0.576 0.552 0.564 811 oral_self_correction 0.158 0.235 0.189 51 oral_sensory_detail 0.285 0.169 0.212 461 oral_simple_conjunction 0.179 0.102 0.130 98 oral_specific_place 0.556 0.705 0.622 424 oral_temporal_anchor 0.410 0.559 0.473 546 oral_tricolon 0.299 0.119 0.171 553 oral_vocative 0.652 0.747 0.696 158 ======================================================================== Macro avg (types w/ support) 0.378 ```
**Missing labels (test set):** 0/53 — all types detected at least once. Notable patterns: - **Strong performers** (F1 > 0.5): vocative (0.696), technical_abbreviation (0.688), phatic_check (0.667), imperative (0.625), specific_place (0.622), citation (0.595), agentless_passive (0.591), rhetorical_question (0.586), inclusive_we (0.568), second_person (0.564), named_individual (0.556), nominalization (0.549), probability (0.500) - **Weak performers** (F1 < 0.2): anaphora (0.054), qualified_assertion (0.118), simple_conjunction (0.130), nested_clauses (0.133), concrete_setting (0.165), tricolon (0.171), categorical_statement (0.189), self_correction (0.189), parallelism (0.191) - **Precision-recall tradeoff**: Most types show balanced precision/recall. Notable exceptions include `anaphora` (0.800 precision / 0.028 recall), `parallelism` (0.741 / 0.110), and `intensifier_doubling` (0.680 / 0.200), which remain high-precision but very low-recall. ## Architecture Custom `MultiLabelTokenClassifier` with independent B/I/O heads per marker type: ``` ModernBERT (answerdotai/ModernBERT-base) └── Dropout (p=0.1) └── Linear (hidden_size → num_types × 3) └── Reshape to (batch, seq, num_types, 3) ``` Each marker type gets an independent 3-way O/B/I classification, so a token can simultaneously carry labels for multiple overlapping marker types. Types share the full backbone representation but make independent predictions. ### Regularization - **Mixout** (p=0.1): During training, each backbone weight element has a 10% chance of being replaced by its pretrained value per forward pass, acting as a stochastic L2 anchor that prevents representation drift (Lee et al., 2019) - **Per-type focal loss** (γ=2.0): Focuses learning on hard examples, reducing the contribution of easy negatives - **Inverse-frequency type weights**: Rare marker types receive higher loss weighting - **Inverse-frequency OBI weights**: B and I classes upweighted relative to dominant O class - **Weighted random sampling**: Examples containing rarer markers sampled more frequently ### Initialization Fine-tuned from `answerdotai/ModernBERT-base`. Backbone linear layers wrapped with Mixout during training (frozen pretrained copy used as anchor). The classification head is randomly initialized: ``` backbone.* layers → loaded from pretrained, anchored via Mixout classifier.weight → randomly initialized classifier.bias → randomly initialized ``` ## Limitations - **Near-zero recall types**: `anaphora` (0.028 recall), `simple_conjunction` (0.102), `parallelism` (0.110), and `tricolon` (0.119) are rarely detected despite being present in training data - **Low-precision types**: `nested_clauses` (0.091), `metadiscourse` (0.140), and `qualified_assertion` (0.143) have precision below 0.15, meaning most predictions for those types are false positives - **Context window**: 128 tokens max; longer spans may be truncated - **Domain**: Trained primarily on historical/literary texts; may underperform on modern social media - **Subjectivity**: Some marker boundaries are inherently ambiguous ## Citation ```bibtex @misc{havelock2026token, title={Havelock Orality Token Classifier}, author={Havelock AI}, year={2026}, url={https://huggingface.co/HavelockAI/bert-token-classifier} } ``` ## References - Ong, Walter J. *Orality and Literacy: The Technologizing of the Word*. Routledge, 1982. - Lee, C. et al. "Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models." ICLR 2020. - Warner, A. et al. "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." 2024. --- *Trained: February 2026*