CoBERTa is a 24.5 million parameter RoBERTa-style masked language model specifically trained on synthetic academic data. It demonstrates that high-quality synthetic data can compensate for model scale, enabling domain specialization, formal logic and reasoning on consumer hardware. It achieves academic language proficiency with 5× fewer parameters than comparable models on certain tasks, trained in around 6 hours on consumer hardware. It was trained on HuggingfaceTB's Cosmopedia dataset.

Intended uses & limitations

You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions on a task that interests you.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at a model like GPT2.

Basic Usage

python

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("appleroll/coberta-base")
tokenizer = AutoTokenizer.from_pretrained("appleroll/coberta-base")

text = "The key to effective communication is to [MASK] clearly and listen actively."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits

# Get top predictions for [MASK]
mask_token_index = torch.where(inputs.input_ids == tokenizer.mask_token_id)[1]
mask_token_logits = predictions[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()

for token in top_5_tokens:
    print(f"{tokenizer.decode([token])}")

Citation

If you use CoBERTa in your research, please cite:

bibtex

@misc{coberta,
    title     = {CoBERTa: Consumer-friendly Models that Punch Above Weight},
    url       = {https://huggingface.co/appleroll/coberta-base},
    author    = {Zhang, Ethan},
    year      = {2025}
}

Additional Information

Authors:

Ethan Zhang (Independent Researcher)

Contact

For questions, feedback, or collaboration: ethanzhangyixuan@gmail.com

Acknowledgements

Cosmopedia dataset creators (HuggingfaceTB)
Apple MLX development team
Hugging Face for model hosting
Open-source ML community

Version

Current: v0.5 (Experimental Release)

Previous: None

License

This model is licensed under the MIT License. See the LICENSE file for details.