HuggingFace-compatible packaging of the Latin BERT model from:

The original model and training code are available at github.com/dbamman/latin-bert. This repo repackages the same weights for use with HuggingFace transformers.

Note: This is an experimental repackaging. If you encounter any issues, please open a thread in the Discussion tab.

The original Latin BERT uses a tensor2tensor SubwordTextEncoder, not standard WordPiece. This repo includes a faithful reimplementation as a HuggingFace PreTrainedTokenizer — this is why trust_remote_code=True is required.

Verified against the original case studies from the paper:

POS tagging (Table 1)

Masked word prediction (Table 3)

Works with spacy-transformers:

The HF tokenizer's full-string path did character-class splitting on the entire string. Unlike the original tensor2tensor tokenizer.encode (and the GitHub gen_berts.py LatinTokenizer, which splits on whitespace first), it did not drop inter-word spaces, so it escaped every space to \32;_ and injected spurious subtokens between every word. This corrupted masked-word predictions and broke subword→word alignment when calling tokenizer("full sentence") (the documented usage). Both the slow and fast tokenizers now whitespace-split first, so the native string API reproduces the original per-word tokenization with no is_split_into_words needed. The fast tokenizer also gained the do_lower_case normalizer it was previously missing (it had ignored the flag, escaping capitals to codepoints exactly as the slow tokenizer did before v1.1.1). Decode now renders word boundaries as spaces (BERT-style; "tres." → "tres .").

Verified: the native full-string path reproduces the paper's infilling metrics exactly (P@1 33.1% / P@10 62.2% / P@50 74.0%, n=1161), identical to the per-word path, and recovers the reported masked words (e.g. Caesar aliam, Livy primo). The full case-study suite reproduces through the native path — POS tagging (95.2 / 98.2 / 99.2%), WSD (0.764), and the §4.4 contextual nearest neighbors (Tables 6–7: top cosine 0.835 for in and 0.919 for audentes, with the paper's lexical variants audentis / audaces / audentem retrieved).

Responds to discussion #3 — thanks to aimgo for the report.

The original Latin BERT vocabulary was trained on lowercased text. All original case studies (POS tagging, WSD, infilling) explicitly called .lower() before tokenizing. The HF PreTrainedTokenizer wrapper was missing this step, causing uppercase characters to be escaped to their ASCII codepoints (e.g. C → \67;), inflating token counts ~4x and producing embeddings the model was never trained on. The tokenizer now lowercases input by default (do_lower_case=True), matching the original pipeline behavior.

Repackaged the original tensor2tensor SubwordTextEncoder tokenizer and PyTorch weights as a HuggingFace PreTrainedTokenizer + safetensors model.

Bamman & Burns (2020) Latin BERT weights and tensor2tensor tokenizer.