Binary classifier for detecting anachronisms in mathematical and scientific texts from pre-modern traditions. Part of the ABACUS project - AGI verification via pre-modern mathematical reasoning.

Model Description

abacus-cheat-tell-v3 detects whether a passage from pre-modern mathematical texts contains anachronistic language or concepts (i.e., ideas that could not have existed at the time of writing). This is a core component of the ABACUS "no-cheating" protocol: any model trained on pre-modern corpora that produces post-1930 concepts is flagged.

Architecture: ModernBertForSequenceClassification (answerdotai/ModernBERT-base, 149M params) Labels: 0 = authentic, 1 = anachronism Warm-started from: idirectships/abacus-cheat-tell-v2

v2 vs v3 Metrics

Metric	v2 (baseline)	v3	Delta
F1	0.6522	0.7368	+0.0846
Accuracy	0.5429	0.7143	+0.1714
Precision	n/a	0.6667	-
Recall	n/a	0.8235	-

v3 was trained on the abacus-cheat-tell-eval-v3 train split (140 balanced examples) and uses explicit anachronism metadata with insertion position. v2 was trained on an unknown dataset of similar size over 4 epochs.

Training Details

Dataset: idirectships/abacus-cheat-tell-eval-v3 train split

140 examples, perfectly balanced: 70 authentic / 70 anachronism
Traditions: Greek, Chinese, Japanese, Indian, Islamic, Babylonian, Egyptian, Mayan

Hyperparameters:

Base model: idirectships/abacus-cheat-tell-v2 (warm-start from v2 weights)
Learning rate: 1e-5
Batch size: 8 (train), 16 (eval)
Max epochs: 10 (best at epoch 7, early stopping patience=3)
Warmup steps: 30
Weight decay: 0.05
Optimizer: AdamW fused
Training hardware: MacBook M4 Max (Apple MPS backend)
Wall clock: 13.0 minutes (782s)

Intended Use

Primary use: ABACUS provenance pipeline - anachronism filter for pre-modern mathematical TEAs (Translation-Era Artifacts). A text scoring anachronism (label=1) likely contains post-1930 mathematical terminology embedded in a historical context, indicating contaminated training data.

Limitations

Small eval set (35 examples) - F1 has high variance at this scale
May flag legitimate retrospective commentary (e.g., "this anticipates Fermat's Last Theorem") as anachronism
Specialized on mathematical/scientific domains - not intended for general anachronism detection
Limited coverage of Indigenous mathematical traditions in training data

License

Apache-2.0 (matching base model and v2)

Production Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("idirectships/abacus-cheat-tell-v3")
mdl = AutoModelForSequenceClassification.from_pretrained("idirectships/abacus-cheat-tell-v3")
mdl.train(False)

def classify(text):
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        logits = mdl(**inputs).logits
    label_id = logits.argmax(-1).item()
    score = torch.softmax(logits, dim=-1)[0][label_id].item()
    return {"label": ["authentic", "anachronism"][label_id], "score": round(score, 4)}

print(classify("Archimedes calculated pi using a polygon approximation method."))
# -> {'label': 'authentic', 'score': ...}

print(classify("Newton's discovery of quantum entanglement in 1687 led to..."))
# -> {'label': 'anachronism', 'score': ...}