This model performs multi-label span-level detection of 53 rhetorical marker types, where each token independently carries B/I/O labels per type — allowing overlapping spans (e.g. a token that is simultaneously part of a concessive and a nested clause).

Model Details

Property	Value
Base model	`answerdotai/ModernBERT-base`
Task	Multi-label token classification (independent B/I/O per type)
Marker types	53 (22 oral, 31 literate)
Test macro F1	0.378 (per-type detection, binary positive = B or I)
Training	20 epochs, fp16
Regularization	Mixout (p=0.1) — stochastic L2 anchor to pretrained weights
Loss	Per-type focal loss (γ=2.0) with inverse-frequency OBI and type weights
Min examples	150 (types below this threshold excluded)

Usage

python

import json
import torch
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download

model_name = "HavelockAI/bert-token-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
model.eval()

# Load marker type map
type_map_path = hf_hub_download(model_name, "type_to_idx.json")
type_to_idx = json.loads(open(type_map_path).read())
idx_to_type = {v: k for k, v in type_to_idx.items()}

text = "Tell me, O Muse, of that ingenious hero who travelled far and wide"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs)  # (1, seq_len, num_types, 3)
    preds = logits.argmax(dim=-1)  # (1, seq_len, num_types)

tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
for i, token in enumerate(tokens):
    active = [
        f"{idx_to_type[t]}={'OBI'[v]}"
        for t, v in enumerate(preds[0, i].tolist())
        if v > 0
    ]
    if active:
        print(f"{token:15} {', '.join(active)}")

Note: This model uses a custom architecture (HavelockTokenClassifier) with independent B/I/O heads per marker type, enabling overlapping span detection. Loading requires trust_remote_code=True.

Training Data

Sources: Project Gutenberg, textfiles.com, Reddit, Wikipedia talk pages
Types with fewer than 150 annotated spans are excluded from training
Multi-label BIO annotation: tokens can carry labels for multiple overlapping marker types simultaneously

Marker Types (53)

Oral Markers (22 types)

Characteristics of oral tradition and spoken discourse:

Category	Markers
Address & Interaction	vocative, imperative, second_person, inclusive_we, rhetorical_question, phatic_check, phatic_filler
Repetition & Pattern	anaphora, parallelism, tricolon, lexical_repetition, antithesis
Conjunction	simple_conjunction
Formulas	discourse_formula, intensifier_doubling
Narrative	named_individual, specific_place, temporal_anchor, sensory_detail, embodied_action, everyday_example
Performance	self_correction

Literate Markers (31 types)

Characteristics of written, analytical discourse:

Category	Markers
Abstraction	nominalization, abstract_noun, conceptual_metaphor, categorical_statement
Syntax	nested_clauses, relative_chain, conditional, concessive, temporal_embedding, causal_explicit
Hedging	epistemic_hedge, probability, evidential, qualified_assertion, concessive_connector
Impersonality	agentless_passive, agent_demoted, institutional_subject, objectifying_stance
Scholarly apparatus	citation, cross_reference, metadiscourse, definitional_move
Technical	technical_term, technical_abbreviation, enumeration, list_structure
Connectives	contrastive, additive_formal
Setting	concrete_setting, aside

Evaluation

Per-type detection F1 on test set (binary: B or I = positive, O = negative):

Click to show per-marker precision/recall/F1/support

``` Type Prec Rec F1 Sup ======================================================================== literate_abstract_noun 0.190 0.325 0.240 381 literate_additive_formal 0.246 0.556 0.341 27 literate_agent_demoted 0.404 0.368 0.386 304 literate_agentless_passive 0.575 0.607 0.591 1133 literate_aside 0.379 0.429 0.403 436 literate_categorical_statement 0.267 0.146 0.189 514 literate_causal_explicit 0.227 0.279 0.251 190 literate_citation 0.639 0.556 0.595 372 literate_conceptual_metaphor 0.310 0.364 0.335 415 literate_concessive 0.499 0.470 0.484 502 literate_concessive_connector 0.455 0.408 0.430 49 literate_concrete_setting 0.241 0.125 0.165 407 literate_conditional 0.369 0.630 0.466 760 literate_contrastive 0.310 0.428 0.360 341 literate_cross_reference 0.386 0.524 0.444 42 literate_definitional_move 0.395 0.185 0.252 81 literate_enumeration 0.495 0.483 0.489 775 literate_epistemic_hedge 0.421 0.481 0.449 445 literate_evidential 0.625 0.360 0.457 472 literate_institutional_subject 0.332 0.326 0.329 282 literate_list_structure 0.338 0.523 0.411 86 literate_metadiscourse 0.140 0.393 0.206 135 literate_nested_clauses 0.091 0.246 0.133 1169 literate_nominalization 0.499 0.612 0.549 991 literate_objectifying_stance 0.635 0.365 0.464 167 literate_probability 0.432 0.593 0.500 27 literate_qualified_assertion 0.143 0.100 0.118 40 literate_relative_chain 0.382 0.507 0.436 1424 literate_technical_abbreviation 0.667 0.711 0.688 225 literate_technical_term 0.280 0.375 0.321 715 literate_temporal_embedding 0.228 0.259 0.242 526 oral_anaphora 0.800 0.028 0.054 287 oral_antithesis 0.249 0.238 0.243 412 oral_discourse_formula 0.340 0.408 0.371 557 oral_embodied_action 0.280 0.391 0.326 425 oral_everyday_example 0.333 0.156 0.212 404 oral_imperative 0.591 0.662 0.625 293 oral_inclusive_we 0.516 0.632 0.568 622 oral_intensifier_doubling 0.680 0.200 0.309 85 oral_lexical_repetition 0.404 0.254 0.312 173 oral_named_individual 0.441 0.749 0.556 770 oral_parallelism 0.741 0.110 0.191 182 oral_phatic_check 0.611 0.733 0.667 30 oral_phatic_filler 0.174 0.409 0.244 93 oral_rhetorical_question 0.509 0.692 0.586 905 oral_second_person 0.576 0.552 0.564 811 oral_self_correction 0.158 0.235 0.189 51 oral_sensory_detail 0.285 0.169 0.212 461 oral_simple_conjunction 0.179 0.102 0.130 98 oral_specific_place 0.556 0.705 0.622 424 oral_temporal_anchor 0.410 0.559 0.473 546 oral_tricolon 0.299 0.119 0.171 553 oral_vocative 0.652 0.747 0.696 158 ======================================================================== Macro avg (types w/ support) 0.378 ```

Missing labels (test set): 0/53 — all types detected at least once.

Notable patterns:

Strong performers (F1 > 0.5): vocative (0.696), technical_abbreviation (0.688), phatic_check (0.667), imperative (0.625), specific_place (0.622), citation (0.595), agentless_passive (0.591), rhetorical_question (0.586), inclusive_we (0.568), second_person (0.564), named_individual (0.556), nominalization (0.549), probability (0.500)
Weak performers (F1 < 0.2): anaphora (0.054), qualified_assertion (0.118), simple_conjunction (0.130), nested_clauses (0.133), concrete_setting (0.165), tricolon (0.171), categorical_statement (0.189), self_correction (0.189), parallelism (0.191)
Precision-recall tradeoff: Most types show balanced precision/recall. Notable exceptions include anaphora (0.800 precision / 0.028 recall), parallelism (0.741 / 0.110), and intensifier_doubling (0.680 / 0.200), which remain high-precision but very low-recall.

Architecture

Custom MultiLabelTokenClassifier with independent B/I/O heads per marker type:

ModernBERT (answerdotai/ModernBERT-base)
    └── Dropout (p=0.1)
        └── Linear (hidden_size → num_types × 3)
            └── Reshape to (batch, seq, num_types, 3)

Each marker type gets an independent 3-way O/B/I classification, so a token can simultaneously carry labels for multiple overlapping marker types. Types share the full backbone representation but make independent predictions.

Regularization

Mixout (p=0.1): During training, each backbone weight element has a 10% chance of being replaced by its pretrained value per forward pass, acting as a stochastic L2 anchor that prevents representation drift (Lee et al., 2019)
Per-type focal loss (γ=2.0): Focuses learning on hard examples, reducing the contribution of easy negatives
Inverse-frequency type weights: Rare marker types receive higher loss weighting
Inverse-frequency OBI weights: B and I classes upweighted relative to dominant O class
Weighted random sampling: Examples containing rarer markers sampled more frequently

Initialization

Fine-tuned from answerdotai/ModernBERT-base. Backbone linear layers wrapped with Mixout during training (frozen pretrained copy used as anchor). The classification head is randomly initialized:

backbone.* layers  → loaded from pretrained, anchored via Mixout
classifier.weight  → randomly initialized
classifier.bias    → randomly initialized

Limitations

Near-zero recall types: anaphora (0.028 recall), simple_conjunction (0.102), parallelism (0.110), and tricolon (0.119) are rarely detected despite being present in training data
Low-precision types: nested_clauses (0.091), metadiscourse (0.140), and qualified_assertion (0.143) have precision below 0.15, meaning most predictions for those types are false positives
Context window: 128 tokens max; longer spans may be truncated
Domain: Trained primarily on historical/literary texts; may underperform on modern social media
Subjectivity: Some marker boundaries are inherently ambiguous

Citation

bibtex

@misc{havelock2026token,
  title={Havelock Orality Token Classifier},
  author={Havelock AI},
  year={2026},
  url={https://huggingface.co/HavelockAI/bert-token-classifier}
}

References

Ong, Walter J. Orality and Literacy: The Technologizing of the Word. Routledge, 1982.
Lee, C. et al. "Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models." ICLR 2020.
Warner, A. et al. "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." 2024.

Trained: February 2026