The model starts from bert-base-uncased and adds a tiny learned capitalization embedding channel. The lexical token ID still follows the uncased BERT vocabulary, while a parallel capitalization_ids tensor carries case information:

text

= lowercase / no capitalization feature / punctuation / special token
= first-cap, e.g. Tom
= all-caps, e.g. NASA
= mixed-case, e.g. iPhone

This checkpoint is a masked-language-model checkpoint, not a downstream fine-tuned classifier.

Checkpoint

Best project checkpoint:

mixed_case_dropout/capitalized_from_3class_steps3000_lr2e5_drop01/final

Training recipe:

text

base model: bert-base-uncased
capitalization vocab size: 4
capitalization loss weight: 0.25
capitalization class weights: [1, 2, 8, 4]
capitalization embedding dropout: 0.1
continued pretraining: 3,000 steps on the real-acronym mix

Loading

This model uses a custom architecture, so it is not directly loadable with plain AutoModelForMaskedLM. Install the project package first:

pip install git+https://github.com/Santosh-Gupta/CapitalizationEmbeddings.git

Then load:

python

from transformers import AutoTokenizer

from capitalization_embeddings import (
    CapitalizedBertForMaskedLM,
    tokenize_with_capitalization,
)

repo_id = "Santosh-Gupta/capitalized-bert-base-uncased-mlm"

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = CapitalizedBertForMaskedLM.from_pretrained(repo_id)

encoding = tokenize_with_capitalization(
    tokenizer,
    "Tom works at NASA.",
    return_tensors="pt",
    use_mixed_case=True,
)

outputs = model(**encoding)
print(outputs.logits.shape)
print(outputs.capitalization_logits.shape)

Intended Use

This checkpoint is mainly intended for research and continued experimentation:

continued MLM pretraining;
probing capitalization-aware BERT representations;
initializing token/entity classification fine-tuning experiments;
reproducing the blog-post results in the repository.

It is not intended as a general replacement for bert-base-cased.

Results Summary

The project found that capitalization embeddings are most useful on token and entity-heavy tasks where case behaves like a reusable feature. Sequence-task results were mixed.

Headline examples from the repository:

Benchmark	Metric	Uncased	Cased	Capitalized
CoNLL-2003 NER	entity F1	0.9040 +/- 0.0025	0.9119 +/- 0.0035	0.9165 +/- 0.0018
WNUT-17 NER	entity F1	0.4424 +/- 0.0152	0.4426 +/- 0.0100	0.4495 +/- 0.0103
SST-5	accuracy	0.5410 +/- 0.0039	0.5283 +/- 0.0084	0.5407 +/- 0.0068

See the repository README for the full writeup and limitations.

Limitations

This is a custom BERT architecture requiring the project package.
This is an MLM checkpoint, not an instruction model or text generator.
The method did not universally dominate cased or uncased BERT.
Some benchmark families favored cased BERT or uncased BERT for reasons that a small case embedding did not fully recover.

Citation

If you use this checkpoint, cite the GitHub repository:

text

Santosh Gupta. CapitalizationEmbeddings.
https://github.com/Santosh-Gupta/CapitalizationEmbeddings